Commit Graph
94 Commits
Author SHA1 Message Date
PhilippandGitHub 2d80298aa2 Dependabot: Ignoring all major updates. (#241) 2023-10-19 19:24:58 +02:00
PhilippandGitHub 36a0bab6ff Add relations from finanical reports to SQL (#216) 2023-10-19 19:21:33 +02:00
PhilippandGitHub a358ebe9c1 Dependabot/coverage (#240) 2023-10-19 18:47:32 +02:00
PhilippandGitHub b92be60789 Automaticlly updating pre-commits (#220) 2023-10-19 18:44:13 +02:00
PhilippandGitHub 0441a3482b Create dependabot.yml (#227) 2023-10-19 18:05:36 +02:00
PhilippandGitHub fdba24ef11 Updates including security-updates (#226) 2023-10-19 17:59:15 +02:00
PhilippandGitHub c0d42a22d7 General updates & linting (#218) 2023-10-14 17:43:45 +02:00
PhilippandGitHub 6365e252b9 Added location to person (#185) 2023-10-14 15:27:19 +00:00
PhilippandGitHub 411f1053dd Set home title (#217) 2023-10-14 17:16:54 +02:00
PhilippandGitHub f8c111d7e2 Resolve mismatch between staging and prod db data for financials (#211)
SQL Creation is now done dynamicly by the definition of the enumeration
type.
2023-10-14 17:16:14 +02:00
PhilippandGitHub c8d3c7395b Update docker-compose.yml (#210) 2023-10-12 18:09:52 +02:00
PhilippandGitHub 68346aff15 Removed error suppression from sql data transformation. (#205) 2023-10-09 17:58:20 +02:00
PhilippandGitHub 63325e7faa Add constraints to the SQL entities (#186) 2023-10-06 18:48:58 +02:00
PhilippandGitHub ea9c777217 Enabeling data processing in docker container. (#195) 2023-10-06 18:36:29 +02:00
PhilippandGitHub b1ca268a62 SQL fixes after new mongo ingest (#199) 2023-10-06 18:22:19 +02:00
PhilippandGitHub 8bb27f5195 Set the DASH_URL_BASE_PATHNAME in the docker container to be used as a default (#192)
Set the DASH_URL_BASE_PATHNAME in the docker container to be used as a
prefix.

Added a Dash Title
2023-10-06 18:13:25 +02:00
Philipp ab96c5a01a Switched two docker container off. 2023-10-05 22:16:45 +02:00
PhilippandGitHub 09c36960e3 Add an list of missing relation partners to be searched (#171)
- [x] Add a new table
- [x] Add a field to the table that can register if the company was
already queried
- [x] Add a field to the table that counts how many times a relation
partner was missing
- [x] Add a function that restets the counter

Also:
- Reworked the get_company function to use the location dict as kwargs
2023-10-05 19:57:30 +02:00
PhilippandGitHub c6f2c7467c Rework the transfer of company data to fit the new data in the mongodb (#188)
This adds the additional company data as proposed to the sql db.

- [x] @TrisNol Is everything included or did I miss a feature. Relations
are in another issue.
- [x] @KM-R New DB features for the Dashbord for your review.
2023-10-05 19:47:46 +02:00
PhilippandGitHub ea89a64a80 Linter update to ruff==0.0.292 (#191) 2023-10-04 20:04:59 +02:00
PhilippandGitHub 41f2c9f995 Executing black over all jupyter notebook (#190)
Reverting black for the jupyter notebooks gets old. Can we just run
black over all of them?
2023-10-04 20:03:47 +02:00
PhilippandGitHub 030ad00c7d Testing speedup with in memory SQLite (#189)
If no SQLite File is written and deleted testing is MUCH faster.
2023-10-04 19:36:57 +02:00
01b4ce00c1 Spellchecking with PyCharm (#133)
Co-authored-by: KM-R <129882581+KM-R@users.noreply.github.com>
2023-10-02 20:47:42 +02:00
PhilippandGitHub d2d4a436f8 Add a cli interface to choose a configuration (#163)
- [x] add a cli to the webserver to take env variables into account 
- [x] add a cli to the data processing that takes enviromental variable
as a valid source into account
- [x] rework the cli for the reset sql command
- [x] rework the cli for the copying of sql data from one db to another
2023-10-02 20:31:42 +02:00
PhilippandGitHub 2abe12f027 Add a function to convert DM to EUR (#168)
The function is ment to transform the captial dict into a format that
can be added as a kwarg (**norm_capital(capital_dict) to the company
entities.
This PR only contains the function itself.
2023-10-02 19:46:17 +02:00
PhilippandGitHub 05472cc16a Added longitude/latitude and positional accuracy to the company data (#180) 2023-10-02 17:18:04 +02:00
PhilippandGitHub c96462532b Added a pre-commit hook that disallows committing to main. (#181)
Since I committed to main again I added an pre-commit hook that prevents
that. Since pushing to main is disallowed this should prevent someone to
be stuck with a mess.
2023-10-01 21:40:39 +02:00
PhilippandGitHub b46a52e019 Added a weekly counter similar to all other weeklies to weekly 10. (#165) 2023-10-01 15:08:45 +02:00
PhilippandGitHub ebf30da778 Removed some superfluous tools from the docker image (#157) 2023-09-28 19:06:58 +02:00
PhilippandGitHub 4d99601fc6 Optional docs Feature: Changelog (#155)
Added the ability to generate a changelog of the main branch in the
Documenation.
2023-09-28 19:05:05 +02:00
552e1bdf96 Added some metadata to the package (#144)
Filled out the mainter section.
Added a new description.
Added the path to the repo to the package.
Added the path to the documentation.

Co-authored-by: KM-R <129882581+KM-R@users.noreply.github.com>
Co-authored-by: Tristan Nolde <tristan.nolde@yahoo.de>
2023-09-28 17:17:01 +02:00
PhilippandGitHub 7d6bb8b6f5 Optional: Add auto-walrus (#158)
To learn the use of the Walrus opperator this hook integrates it in your
code if it finds a reason for it.
This branch is a suggestion of mine to force the use of the walrus
operaor as an autofix to keep it in mind when programming.
This should not increase the workload for us. It just lets us confront a
new programming tool. It also isn't a rull to be follwed. Just a hook
that looks at your code with an autofix.
2023-09-28 17:13:42 +02:00
Philipp f54c140d98 Update after rebasing 2023-09-25 20:39:13 +02:00
Philipp 54dd574fb8 Added the ability to include drawio diagrams into the sphinx docs. (#107) 2023-09-25 20:37:59 +02:00
PhilippandGitHub 0fdfcc938a Added 2 Seminararbeiten to sphinx (#154)
Added 2 Seminararbeiten to sphinx
2023-09-25 20:30:43 +02:00
PhilippandGitHub 9cc0654f56 Delete untagged images (#151)
This change to the pipline deletes untagged / overwritten tags once a
week to keep the memory consumption in the GH registry reasonable.
2023-09-25 20:28:49 +02:00
PhilippandGitHub 5e716801c0 Change usage of black to the official black action (#152) 2023-09-25 20:26:02 +02:00
PhilippandGitHub 5bbdf046d2 Ruff fix after merge (#149) 2023-09-24 18:38:05 +02:00
PhilippandGitHub 091e67de79 build first set of docker container in pipline and place them in the the github registry (#142)
- added a Dockerfile for the thre containers
- added a workflow step to build and placing the container in the
registry
- added a docker-compose.yaml to use the build images
- added a docker compose to build the images locally and a script for
prebuild steps
2023-09-24 16:32:52 +00:00
PhilippandGitHub 820fb3e52b Repaired the SQL copy and reduced the log volume a bit (#141)
- Added a cli interface to the SQL copy
- Repaired the SQL copy function
- Added the SQL copy function to the scripts
- Reduced the logging verbosity
2023-09-24 15:11:49 +02:00
PhilippandGitHub d9ca808efa Mypy does not validate if type: ignore comments are needed. (#137) 2023-09-23 12:44:27 +02:00
PhilippandGitHub c5a383e903 Pipline / dependency updates (#145) 2023-09-23 10:49:14 +02:00
PhilippandGitHub 286636dfa4 Reduced the execution of tests and lint actions on pull requests (#143)
This change will change the pipline to limit duplicat execution of tests
and lint actions in pull requests to save computing time.
2023-09-23 09:22:14 +02:00
PhilippandGitHub 495cd90d85 Reduced required code coverage to 70% (#140) 2023-09-21 18:09:16 +02:00
PhilippandGitHub 80f077ee7a Version ups (#117)
Rutine version ups for
- python-depdencies
- pre-commits
- piplines
2023-09-17 13:50:53 +02:00
PhilippandGitHub 06fef5361d Lightend black in the pipline building (#128)
Pipline should now run now 20 -40s faster.
2023-09-17 13:47:20 +02:00
PhilippandGitHub 56b6280264 Transfer financials from staging to sql (#129) 2023-09-17 13:45:08 +02:00
PhilippandGitHub fea31e543b Bugfix update for transfer of company data from mongo to sql (#121)
Fixed the following errors:

- Typo in readme.md
- Mongo delivers the last_update of an company as a string not as date.

Added:
- Entrypoint description in readme.md
2023-09-13 19:14:36 +02:00
PhilippandGitHub 507647d164 Added custom 404 error code page in the sphinx docs (#115) 2023-09-13 19:01:09 +02:00
ba44b082b8 Data transfer script (#114)
Transfers data betwenn two sql instances.
Limited in data volume. Should be good enough for now.

---------

Co-authored-by: Tim <tim.ronneburg@outlook.de>
2023-09-11 21:10:36 +02:00
PhilippandGitHub d64f53eca9 Script for the transfer of data from Mongo to SQL (#80) 2023-09-11 20:21:35 +02:00
PhilippandGitHub 2ea99c8c17 Removed unneded lines ☺ (#113) 2023-09-11 19:49:08 +02:00
PhilippandGitHub cee1ef901a Updatest vor everything (#106)
- direct dependencies
- other depdenencies
- pre-commit
2023-09-09 16:47:48 +00:00
PhilippandGitHub 23a193a4fb Added a small test ☺ (#108) 2023-09-09 18:35:50 +02:00
PhilippandGitHub 2233b12468 Add sql lite session (#71)
I added an sql lite session generator.

Changes
The function `def get_session() -> Session` 
has changed to 
`def get_session(connect_to: ConfigProvider | str) -> Session`
If a JsonFileConfig Is given the postgress conection is checked.
If a string is given that starts with `sqlite:///` an sql db is created.
The use should otherwise be the same.
2023-09-07 18:41:10 +02:00
Philipp ecf94d4b61 Added an import smoke test. 2023-09-06 19:08:44 +02:00
PhilippandGitHub d04255a7a8 Old date corrected in meeting notes fixed (#76)
The date in the meeting nots was from the meeting two weeks before.
2023-09-04 19:49:17 +02:00
PhilippandGitHub 17e89da0d8 Repo struktur dokumentieren (#75) 2023-09-04 19:48:14 +02:00
PhilippandGitHub f20823a613 Version updates for pre-commit/poetry/linter (#74)
Just a rutine update to keep ruff up to date and sync.
2023-09-03 17:56:13 +02:00
PhilippandGitHub 011184b898 Small fixes to the documentation. (#70) 2023-08-31 16:57:50 +02:00
PhilippandGitHub 69039706ac Version updates (#69)
Version ups for poetry and pre-commit to keep them in sync with the
pipline.
2023-08-31 16:57:09 +02:00
PhilippandGitHub 06e59b8061 Changes to the database (#67)
In this PR are pruposed changed that come up when i started the data
migration.
Plese review them an approve and disaprove.
2023-08-24 17:23:19 +02:00
Philipp 82979b1981 Expanded a bit on the installation of the project. 2023-08-20 13:03:06 +02:00
PhilippandGitHub e4a57c9136 Chore/rework workflow (#52)
* Reworked the lint action
* Removed the file change requirement
* Repaired mypy
* Repaired pip-audit
2023-08-17 22:05:12 +02:00
Philipp faca0a0af6 Repaired a mypy option 2023-08-16 22:10:52 +02:00
PhilippandGitHub 9c237402be Excluded ipynb files from json pretty formatter + poetry update (#50)
* Excluded ipynb files from json pretty formatter +  poetry update
* Reworked pip audit
* Relocated dependencies
* Reworked the tests a bit.
* Poetry update
* Excluded lint and test workflows against the target branch
2023-08-16 20:25:15 +02:00
Philipp 67c94cfb7e Changes sql-alchemy to version 1.4.49 2023-08-13 15:00:29 +02:00
PhilippandGitHub 50bf7811ef Div. dev ops updates (#43)
* Pipline rework to limit mypy and balck to src and tests
* Poetry update
* pre-commit update
2023-08-10 19:20:12 +02:00
PhilippandGitHub f9b1f1e5e4 Ruff update and lint version definition (#36) 2023-07-11 14:23:52 +02:00
PhilippandGitHub aea4b450c8 Pre commit update (#35) 2023-07-11 14:23:15 +02:00
Philipp 1c621f46a7 Added selenium to poetry 2023-07-11 01:59:11 +02:00
PhilippandGitHub e29133f509 Never executing jupyter for sphinx (#31) 2023-06-29 17:54:23 +02:00
PhilippandGitHub a9304201af (chore): Initilised devops tools (#29)
* Added a first action

* Repaired a typo

* Repaired a typo2

* Repaired a typo2

* Added flake8 action

* Repaired a typo in the flake8 action.

* Added a first bandit action

* Added a first batch

* Added a first batch

* Added a first batch

* Added a first batch

* Added a first batch

* Added the flake8-prebuild as a need to flake8

* Added the flake8-prebuild as a need to flake8

* Added the flake8-prebuild as a need to flake8

* Added the docker socket to the volume.

* Added the flake8-prebuild as a need to flake8

* Removed latest part from container.

* Removed latest part from container.

* Removed latest part from container.

* Reworked flake8

* Reworked flake8

* Reworked flake8

* Reworked flake8

* Reworked flake8

* Reworked flake8

* Reworked flake8

* Reworked flake8

* Reworked flake8

* Reworked flake8

* Reworked flake8

* Reworked flake8

* Reworked flake8 poetry

* Reworked flake8 poetry

* Changed to 64bit

* Some edits to the runner

* Added python setup

* Added python -m to python docker image.

* Added python -m to python docker image.

* Added python -m to python docker image.

* Added python -m to python docker image.

* Added python -m to python docker image.

* Added python -m to python docker image.

* Added ra run linter

* Added ra run linter

* Added ra run linter

* Added ra run linter

* Removed redundant version

* Removed redundant version

* Added isort

* Added isort

* Added isort

* Added poetry install

* Added poetry install

* Added flake8 as lint.

* Added flake8 as lint.

* Added flake8 as lint.

* Added flake8 as lint.

* Added flake8 as lint.

* Added flake8 as lint.

* Added flake8 as lint.

* Uses nodejs and python image

* Added flake8 as lint.

* Added flake8 as lint.

* Added flake8 as lint.

* Added flake8 as lint.

* Removed selfhosted runner

* Removed self hosted runner

* Removed self hosted runner

* Removed self hosted runner

* Added black and flake8 tests

* Removed self hosted runner

* Removed self hosted runner

* Removed unneded actions

* Added a mypy error.

* Removed poetry call before boetry setup

* Removed poetry call before poetry setup

* Added a test to understand the poetry action better

* Added a test to understand the poetry action better

* Added a test to understand the poetry action better

* Added a test to understand the poetry action better

* Added a test to understand the poetry action better

* Added a test to understand the poetry action better

* Added the snook poetry builder

* Reworked the repo a bit

* Removed unneeded poetry installation

* Added the isort action

* Added isort test

* Added ruff

* Added full ruff configuration

* Added full ruff configuration2

* Added full ruff configuration2

* Removed duplicat configurations

* Removed some redundant pre-commit hooks

* Removed unneeded actions.

* Removed unneeded actions.

* Repaired ruff

* Added tests.

* Removed

* Removed

* Removed a missing file

* Removed a missing file

* Removed a missing file

* Removed a missing file

* Removed a missing file

* Added reports as artifacts

* Added reports as artifacts

* Added reports as artifacts

* Removed the unneded poetry test

* Added a license checker.

* Added a license checker.

* Removed some unneeded configuration.

* Removed the import reformatted.

* Added doc generation.

* Added doc generation.

* Added license summary.

* Add

* Add lint

* Switched pip-licenses to poetry.

* Switched pip-licenses to poetry.

* Switched pip-licenses to poetry.

* Remove some more packages.

* Remove some more packages.

* Added a make file

* Added a make file

* Added a make file

* Added a make file

* Added a make file

* Added a make file

* Added a make file

* Added a make file

* Added a make file

* Added a make file

* Added a make file

* Added a make file

* Added a make file

* Added a make file

* Added a make file

* Added version codes to the main package

* Changed the format of the md files

* Presentation first draft

* Version up and added extensions

* Version up and added extensions

* Version up and added extensions

* Removed the venv path from docbuild

* Actions version up

* Actions version up

* Actions version up

* Actions version up

* Actions version up

* Actions version up

* Experiements with sphinx

* Experiments with sphinx

* Experiments with sphinx

* Experiments with sphinx

* Experiments with sphinx

* Experiments with sphinx

* Experiments with sphinx

* Experiments with sphinx

* First draft of the sphinx documentation.

* Added the protocol to the time series.

* Added the protocol to the time series.

* First draft ot a first build pipline

* Added mermaid version support

* Added documentations pull and branch request requirements.

* Added documentations pull and branch request requirements.

* Added documentations pull and branch request requirements.

* Added documentations pull and branch request requirements.

* Tests should now be passing

* Tests should now be passing

* Tests should now be passing

* Tests should now be passing

* Tests should now be passing

* Tests should now be passing

* Tests should now be passing

* Tests should now be passing

* Add safety

* Add safety

* Add safety

* Added the action on pull_request_target

* Added the action on pull_request_target

* Added the action on pull_request_target

* Added a pytest coverage report

* Added a pytest coverage report

* Added a pytest coverage report

* Added a pytest coverage report

* Added a pytest coverage report

* Added a build step

* Added a build step

* Added a build step

* Added a build step

* Changed the lint action to work only on python changes.

* Changed the lint action to work only on python changes.

* Changed the lint action to work only on python changes.

* Added the ability to compile a html report

* Added the ability to compile a html report

* Added the ability to compile a html report

* Added the ability to compile a html report

* Added the ability to compile a html report

* Added the ability to compile a html report

* Added the ability to compile a html report

* Added the ability to compile a html report

* Added the ability to compile a html report

* Added the ability to compile a html report

* Added the ability to compile a html report

* Added the ability to compile a html report

* Added the ability to compile a html report

* Added the ability to compile a html report

* Added the ability to compile a html report

* Coverage

* Finished test and build workflow

* Finished test and build workflow

* Finished test and build workflow

* Finished test and build workflow

* Finished test and build workflow

* Finished test and build workflow

* Finished test and build workflow

* Finished test and build workflow

* Finished test and build workflow

* Finished test and build workflow

* Finished test and build workflow

* Finished test and build workflow

* Finished test and build workflow

* Finished test and build workflow

* Finished test and build workflow

* Finished test and build workflow

* Finished test and build workflow

* Finished test and build workflow

* Finished test and build workflow

* Finished test and build workflow

* Repaired a bug.

* Repaired a bug.

* Repaired a bug.

* Repaired a bug.

* Repaired a bug.

* Added a github branch.ref

* Removed a poetry install

* Docbuild now excludes templates

* Added the seminarpräsentation to the documentation build

* Added the seminarpräsentation to the documentation build

* Added the seminarpräsentation to the documentation build

* dded a few images

* Changed the pre-commit image

* Changed the pre-commit image

* Presentation done

* Never executing jupyter for sphinx

* Never executing jupyter for sphinx

* Never executing jupyter for sphinx

* Never executing jupyter for sphinx

* Never executing jupyter for sphinx
2023-06-23 18:47:04 +02:00
PhilippandGitHub a9101aef2f Meeting notes 2023_05_25 (#26)
* New meeting notes
* Some small changes to the other protokolls.
* Spelling error changes
2023-06-05 01:25:46 +02:00
Philipp a8d4810337 Added the DGID env variable to the documentation. 2023-06-01 22:58:15 +02:00
PhilippandGitHub 928a88ba80 Feature/issue#13 (#25)
* Frist draft of what is planned for the coursework

feature/issue#13

* Some small changes to the draft.
2023-05-29 20:12:36 +02:00
Philipp be2ba08b77 Add a new github runner infrastructure 2023-05-27 16:55:25 +02:00
Philipp d92561f745 Added a first draft of a github runner 2023-05-27 16:31:37 +02:00
PhilippandGitHub 3633594156 Moved contribution rules and format documentation to CONTRIBUTING.md (#19)
* Moved contribution rules and format documentation to CONTRIBUTING.md
2023-05-11 17:16:17 +02:00
Philipp 1ad1931c7c Added a folder for the seminar arbeiten. 2023-05-06 10:41:19 +02:00
Philipp f1e1a05fe8 Removed empty cell. 2023-05-01 13:21:58 +02:00
Philipp ccfb1e7899 Added a translation prior to the use of FinBert. 2023-05-01 13:19:52 +02:00
Philipp 9810091f3a Pre-commit update 2023-05-01 12:22:59 +02:00
Philipp d3b14e78ea Merge remote-tracking branch 'github/main' into main 2023-05-01 12:21:37 +02:00
Philipp 29347ce7fb Added a text in research central. 2023-05-01 12:20:56 +02:00
Philipp 141070ea16 Added another example 2023-04-30 23:32:13 +02:00
Philipp 5a52f61c1c Added a first trail with FinBert. 2023-04-30 23:23:02 +02:00
Philipp 983c06eead Merge branch 'main' of github.com:fhswf/aki_prj23_transparenzregister into main 2023-04-27 00:05:33 +02:00
Philipp 6da2ba6a4f Revert "OpenRegister DB Schema hinzugefügt."
This reverts commit d0803397ce.
2023-04-27 00:05:07 +02:00
Philipp d0803397ce OpenRegister DB Schema hinzugefügt. 2023-04-13 16:26:46 +02:00
Philipp 1a8c6bcb95 Merge branch 'hotfix/pre-commit' into main 2023-04-13 00:53:16 +02:00
Philipp b1bfbe9c2c Kleine änderungen an pre-commits und readme
Einige pre-commits hinzugefügt und in der README dokumentiert.
Eine Pre-commit installationsanleitung wurde zur readme hinzugefügt.
Spellchecker and Markdown whitespace table format
2023-04-13 00:07:35 +02:00
Philipp 322a5a8c26 Ignoring .idea 2023-04-06 19:03:23 +02:00
Philipp 262476bd87 Added a frist structure 2023-04-06 19:00:15 +02:00