Philipp/aki_prj23_transparenzregister
1
0
Fork 0
mirror of https://github.com/fhswf/aki_prj23_transparenzregister.git synced 2026-02-13 22:17:38 +01:00
Code Issues Packages Projects Releases Wiki Activity
452 Commits 7 Branches 0 Tags
5f4f732eb2437d9ee6dd14802cc51c35d02a6a71
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Philipp 4716eb9fe2 Added a frist rollback function on a pending Rollback error. (#346) 
Possible fix for #341.
Rolls back stuck SQL sessions.
This only helps against the symptomps not the cause.
 2023-11-09 19:48:26 +00:00
Philipp 97fc7f01b5 Added some redirects to home page. (#348) 2023-11-09 19:06:36 +01:00
Philipp 3ce49c9e19 Added requirements.txt to gitignore (#347) 2023-11-09 18:38:42 +01:00
Philipp fe02b0f6c5 Added an error handling when the sql session crashes in process on network connection lost. (#352) 2023-11-09 18:38:04 +01:00
Philipp 12f40af538 Added pagination, sorting and filtering for the table of related persons. (#333) 
Added pagination, sorting and filtering for the table of related
persons.
 2023-11-07 22:19:36 +01:00
Philipp 16b975a0f4 Hide the plotly logo in graph objects (#339) 2023-11-07 22:16:27 +01:00
Philipp 97b05bacf4 Added a custom favicon. (#338) 2023-11-07 22:15:08 +01:00
Philipp aec7414e69 Removed the background color of plotly plot. (#334) 
Removed the background color of the plot to fully use the plotly_white
layout.
 2023-11-07 15:55:50 +01:00
Philipp d79bc0f267 Added poetry install to the pre-commit hooks. (#332) 
Limiting most pre-commit hooks to the pre-commit stage.
 2023-11-06 21:58:33 +01:00
Philipp 4323522b96 Added poetry install to the pre-commit hooks. (#329) 2023-11-06 20:02:29 +01:00
Philipp 2fddc9149a Reduced the number of times the dependabot auto merge is triggered. (#318) 2023-11-03 21:25:55 +01:00
Philipp ad36c68993 Moved the AI tests into the AI folder. (#315) 2023-11-03 13:45:24 +01:00
Philipp a638949644 Added a types package and upgraded poetry. (#313) 2023-11-02 22:13:32 +01:00
Philipp d40746c69b Used black on jupyter. (#312) 2023-11-02 20:52:46 +00:00
Philipp 8d9981d967 Moved AI files in the AI module. (#308) 2023-11-02 20:30:04 +01:00
Philipp 389810c9bc Update and rename dependabot-approve.yaml to dependabot-auto-merge.yaml 2023-11-01 18:21:56 +01:00
Philipp 3c2394d2d5 Sets auto merge for PRs opened by dependabot to true. (#293) 
Sets auto merge for PRs opened by dependabot to true.
Review is still needed.
 2023-10-30 19:20:54 +01:00
Philipp 4362463f3c Added types-six to the linting packages and unlocked all the linter versions. (#286) 
Added the from mypy demanded types-six package as a lint dependency.
This should allow for a faster execution.
 2023-10-29 20:40:57 +01:00
Philipp f72d606d18 Added base-path support in URL generating features (#288) 
Add the basepath dash url to the path generation for dynamicly generated
links.
 2023-10-29 20:40:40 +01:00
Philipp 6526549bdd Reworked the about page for a nicer version of the git hash (#283) 
The git hash looks currently like this:

`Build from GiT Revision: ('d3189f78c47d812f447739f97876ced01227e1fd',)`

After this change it should look like this:
`Build from GiT Revision: d3189f78c47d812f447739f97876ced01227e1fd`
 2023-10-29 12:56:08 +01:00
Philipp fca53c5e0c Dash security-fix against DoS. (#284) 
Dash has currently a security vulnerability.
This updated should fix this.
Dependabot is currently not allowed to upgrade main-version.
 2023-10-29 12:55:28 +01:00
Philipp 0e388eb927 Using actor to identify dependabot and skipp some actions. (#273) 2023-10-26 19:29:25 +02:00
Philipp e7a778e4cb Changed the plotly default theme (#269) 
I think the current blueish plotly default theme bits itself quit beadly
with @KM-R s great work.
I propose to set the deault plotly template to either:

- plotly_white
- ggplot2
- seaborn

Simply change the constant i have set will change this.
Heren you can find the catalog of themes:
https://plotly.com/python/templates/#disable-default-theming

I would not suggest to do more than changing the default layout.
 2023-10-26 19:22:44 +02:00
Philipp 7953ba9291 Mixed typo fixes (#270) 2023-10-26 19:06:45 +02:00
Philipp ab97d77d2d Made pip-audit pass on every build. (#272) 
Since dependabot is activated this can be reduced as an reporting tool.
 2023-10-26 16:50:24 +00:00
Philipp 4c56431bff Update dependabot.yml 2023-10-26 18:19:20 +02:00
Philipp 896136dcee Added an about page (#251) 
This page was added since it is sometimes difficult to say which version
was deployed on an server. This should allow an easy lookup on the
server and make it comparable with what is expected.
 2023-10-26 17:32:17 +02:00
Philipp 6d77867eb5 Using the python action and the inbuild poetry cache (#261) 
I have found out that the python setup action supports the use of the
poetry cache which increases the build speed.
 2023-10-24 21:49:09 +02:00
Philipp 1eb972b7ff Adds the transfer of sentiments into the sql db (#253) 
Transfers the sentimenes from the mongodb int the sql db.
 2023-10-24 17:50:40 +02:00
Philippgithub-actions[bot]philipp-horstenkamp
 		b250b67343 Auto maintance push (#258) 
Added that the maintance action allso runs on changes of itself.

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: philipp-horstenkamp <philipp-horstenkamp@users.noreply.github.com>
 2023-10-23 17:50:51 +00:00
Philipp 77a0faa257 Reduced build time (#260) 
The building and pushing of artifacts takes to long. With those changes
build experimental images aren't pushed anymore.
This saves build time and disc space.
 2023-10-23 19:43:53 +02:00
Philipp f0d763cc5b Fixing the pre-commit upgrade (#256) 
The action allso executes over all files. This is ok when using
pre-conmit.ci. We only execute file fixes once.
 2023-10-23 17:15:39 +02:00
Philipp ad7aa3e53b Filled Docker Labels and ENVs to identify the git revision it was build from (#250) 
Added labels and env vars to identify the git revion a container was
build from to have a more transparent build flow of untaged images.
 2023-10-21 18:13:03 +02:00
Philipp 88f0673470 Slimmed down the env vars access and added logging in case the password protection was disabled. (#252) 
Slimmed down the env vars access and added logging in case the password
protection was disabled.
 2023-10-21 17:57:56 +02:00
Philipp 55ebb4c17d Typo fixes (#249) 2023-10-21 10:58:54 +02:00
Philipp 640faeccec Adding auth back in. (#248) 2023-10-20 14:56:54 +00:00
Philipp 86daa9a781 Renamed a few jobs (#246) 2023-10-19 19:43:55 +02:00
Philipp 2d80298aa2 Dependabot: Ignoring all major updates. (#241) 2023-10-19 19:24:58 +02:00
Philipp 36a0bab6ff Add relations from finanical reports to SQL (#216) 2023-10-19 19:21:33 +02:00
Philipp a358ebe9c1 Dependabot/coverage (#240) 2023-10-19 18:47:32 +02:00
Philipp b92be60789 Automaticlly updating pre-commits (#220) 2023-10-19 18:44:13 +02:00
Philipp 0441a3482b Create dependabot.yml (#227) 2023-10-19 18:05:36 +02:00
Philipp fdba24ef11 Updates including security-updates (#226) 2023-10-19 17:59:15 +02:00
Philipp c0d42a22d7 General updates & linting (#218) 2023-10-14 17:43:45 +02:00
Philipp 6365e252b9 Added location to person (#185) 2023-10-14 15:27:19 +00:00
Philipp 411f1053dd Set home title (#217) 2023-10-14 17:16:54 +02:00
Philipp f8c111d7e2 Resolve mismatch between staging and prod db data for financials (#211) 
SQL Creation is now done dynamicly by the definition of the enumeration
type.
 2023-10-14 17:16:14 +02:00
Philipp c8d3c7395b Update docker-compose.yml (#210) 2023-10-12 18:09:52 +02:00
Philipp 68346aff15 Removed error suppression from sql data transformation. (#205) 2023-10-09 17:58:20 +02:00
Philipp 63325e7faa Add constraints to the SQL entities (#186) 2023-10-06 18:48:58 +02:00
Philipp ea9c777217 Enabeling data processing in docker container. (#195) 2023-10-06 18:36:29 +02:00
Philipp b1ca268a62 SQL fixes after new mongo ingest (#199) 2023-10-06 18:22:19 +02:00
Philipp 8bb27f5195 Set the DASH_URL_BASE_PATHNAME in the docker container to be used as a default (#192) 
Set the DASH_URL_BASE_PATHNAME in the docker container to be used as a
prefix.

Added a Dash Title
 2023-10-06 18:13:25 +02:00
Philipp ab96c5a01a Switched two docker container off. 2023-10-05 22:16:45 +02:00
Philipp 09c36960e3 Add an list of missing relation partners to be searched (#171) 
- [x] Add a new table
- [x] Add a field to the table that can register if the company was
already queried
- [x] Add a field to the table that counts how many times a relation
partner was missing
- [x] Add a function that restets the counter

Also:
- Reworked the get_company function to use the location dict as kwargs
 2023-10-05 19:57:30 +02:00
Philipp c6f2c7467c Rework the transfer of company data to fit the new data in the mongodb (#188) 
This adds the additional company data as proposed to the sql db.

- [x] @TrisNol Is everything included or did I miss a feature. Relations
are in another issue.
- [x] @KM-R New DB features for the Dashbord for your review.
 2023-10-05 19:47:46 +02:00
Philipp ea89a64a80 Linter update to ruff==0.0.292 (#191) 2023-10-04 20:04:59 +02:00
Philipp 41f2c9f995 Executing black over all jupyter notebook (#190) 
Reverting black for the jupyter notebooks gets old. Can we just run
black over all of them?
 2023-10-04 20:03:47 +02:00
Philipp 030ad00c7d Testing speedup with in memory SQLite (#189) 
If no SQLite File is written and deleted testing is MUCH faster.
 2023-10-04 19:36:57 +02:00
PhilippandKM-R 01b4ce00c1 Spellchecking with PyCharm (#133) 
Co-authored-by: KM-R <129882581+KM-R@users.noreply.github.com>
 2023-10-02 20:47:42 +02:00
Philipp d2d4a436f8 Add a cli interface to choose a configuration (#163) 
- [x] add a cli to the webserver to take env variables into account 
- [x] add a cli to the data processing that takes enviromental variable
as a valid source into account
- [x] rework the cli for the reset sql command
- [x] rework the cli for the copying of sql data from one db to another
 2023-10-02 20:31:42 +02:00
Philipp 2abe12f027 Add a function to convert DM to EUR (#168) 
The function is ment to transform the captial dict into a format that
can be added as a kwarg (**norm_capital(capital_dict) to the company
entities.
This PR only contains the function itself.
 2023-10-02 19:46:17 +02:00
Philipp 05472cc16a Added longitude/latitude and positional accuracy to the company data (#180) 2023-10-02 17:18:04 +02:00
Philipp c96462532b Added a pre-commit hook that disallows committing to main. (#181) 
Since I committed to main again I added an pre-commit hook that prevents
that. Since pushing to main is disallowed this should prevent someone to
be stuck with a mess.
 2023-10-01 21:40:39 +02:00
Philipp b46a52e019 Added a weekly counter similar to all other weeklies to weekly 10. (#165) 2023-10-01 15:08:45 +02:00
Philipp ebf30da778 Removed some superfluous tools from the docker image (#157) 2023-09-28 19:06:58 +02:00
Philipp 4d99601fc6 Optional docs Feature: Changelog (#155) 
Added the ability to generate a changelog of the main branch in the
Documenation.
 2023-09-28 19:05:05 +02:00
PhilippKM-RTristan Nolde
 		552e1bdf96 Added some metadata to the package (#144) 
Filled out the mainter section.
Added a new description.
Added the path to the repo to the package.
Added the path to the documentation.

Co-authored-by: KM-R <129882581+KM-R@users.noreply.github.com>
Co-authored-by: Tristan Nolde <tristan.nolde@yahoo.de>
 2023-09-28 17:17:01 +02:00
Philipp 7d6bb8b6f5 Optional: Add auto-walrus (#158) 
To learn the use of the Walrus opperator this hook integrates it in your
code if it finds a reason for it.
This branch is a suggestion of mine to force the use of the walrus
operaor as an autofix to keep it in mind when programming.
This should not increase the workload for us. It just lets us confront a
new programming tool. It also isn't a rull to be follwed. Just a hook
that looks at your code with an autofix.
 2023-09-28 17:13:42 +02:00
Philipp f54c140d98 Update after rebasing 2023-09-25 20:39:13 +02:00
Philipp 54dd574fb8 Added the ability to include drawio diagrams into the sphinx docs. (#107) 2023-09-25 20:37:59 +02:00
Philipp 0fdfcc938a Added 2 Seminararbeiten to sphinx (#154) 
Added 2 Seminararbeiten to sphinx
 2023-09-25 20:30:43 +02:00
Philipp 9cc0654f56 Delete untagged images (#151) 
This change to the pipline deletes untagged / overwritten tags once a
week to keep the memory consumption in the GH registry reasonable.
 2023-09-25 20:28:49 +02:00
Philipp 5e716801c0 Change usage of black to the official black action (#152) 2023-09-25 20:26:02 +02:00
Philipp 5bbdf046d2 Ruff fix after merge (#149) 2023-09-24 18:38:05 +02:00
Philipp 091e67de79 build first set of docker container in pipline and place them in the the github registry (#142) 
- added a Dockerfile for the thre containers
- added a workflow step to build and placing the container in the
registry
- added a docker-compose.yaml to use the build images
- added a docker compose to build the images locally and a script for
prebuild steps
 2023-09-24 16:32:52 +00:00
Philipp 820fb3e52b Repaired the SQL copy and reduced the log volume a bit (#141) 
- Added a cli interface to the SQL copy
- Repaired the SQL copy function
- Added the SQL copy function to the scripts
- Reduced the logging verbosity
 2023-09-24 15:11:49 +02:00
Philipp d9ca808efa Mypy does not validate if type: ignore comments are needed. (#137) 2023-09-23 12:44:27 +02:00
Philipp c5a383e903 Pipline / dependency updates (#145) 2023-09-23 10:49:14 +02:00
Philipp 286636dfa4 Reduced the execution of tests and lint actions on pull requests (#143) 
This change will change the pipline to limit duplicat execution of tests
and lint actions in pull requests to save computing time.
 2023-09-23 09:22:14 +02:00
Philipp 495cd90d85 Reduced required code coverage to 70% (#140) 2023-09-21 18:09:16 +02:00
Philipp 80f077ee7a Version ups (#117) 
Rutine version ups for
- python-depdencies
- pre-commits
- piplines
 2023-09-17 13:50:53 +02:00
Philipp 06fef5361d Lightend black in the pipline building (#128) 
Pipline should now run now 20 -40s faster.
 2023-09-17 13:47:20 +02:00
Philipp 56b6280264 Transfer financials from staging to sql (#129) 2023-09-17 13:45:08 +02:00
Philipp fea31e543b Bugfix update for transfer of company data from mongo to sql (#121) 
Fixed the following errors:

- Typo in readme.md
- Mongo delivers the last_update of an company as a string not as date.

Added:
- Entrypoint description in readme.md
 2023-09-13 19:14:36 +02:00
Philipp 507647d164 Added custom 404 error code page in the sphinx docs (#115) 2023-09-13 19:01:09 +02:00
PhilippandTim ba44b082b8 Data transfer script (#114) 
Transfers data betwenn two sql instances.
Limited in data volume. Should be good enough for now.

---------

Co-authored-by: Tim <tim.ronneburg@outlook.de>
 2023-09-11 21:10:36 +02:00
Philipp d64f53eca9 Script for the transfer of data from Mongo to SQL (#80) 2023-09-11 20:21:35 +02:00
Philipp 2ea99c8c17 Removed unneded lines ☺ (#113) 2023-09-11 19:49:08 +02:00
Philipp cee1ef901a Updatest vor everything (#106) 
- direct dependencies
- other depdenencies
- pre-commit
 2023-09-09 16:47:48 +00:00
Philipp 23a193a4fb Added a small test ☺ (#108) 2023-09-09 18:35:50 +02:00
Philipp 2233b12468 Add sql lite session (#71) 
I added an sql lite session generator.

Changes
The function `def get_session() -> Session` 
has changed to 
`def get_session(connect_to: ConfigProvider | str) -> Session`
If a JsonFileConfig Is given the postgress conection is checked.
If a string is given that starts with `sqlite:///` an sql db is created.
The use should otherwise be the same.
 2023-09-07 18:41:10 +02:00
Philipp ecf94d4b61 Added an import smoke test. 2023-09-06 19:08:44 +02:00
Philipp d04255a7a8 Old date corrected in meeting notes fixed (#76) 
The date in the meeting nots was from the meeting two weeks before.
 2023-09-04 19:49:17 +02:00
Philipp 17e89da0d8 Repo struktur dokumentieren (#75) 2023-09-04 19:48:14 +02:00
Philipp f20823a613 Version updates for pre-commit/poetry/linter (#74) 
Just a rutine update to keep ruff up to date and sync.
 2023-09-03 17:56:13 +02:00
Philipp 011184b898 Small fixes to the documentation. (#70) 2023-08-31 16:57:50 +02:00
Philipp 69039706ac Version updates (#69) 
Version ups for poetry and pre-commit to keep them in sync with the
pipline.
 2023-08-31 16:57:09 +02:00
Philipp 06e59b8061 Changes to the database (#67) 
In this PR are pruposed changed that come up when i started the data
migration.
Plese review them an approve and disaprove.
 2023-08-24 17:23:19 +02:00
Philipp 82979b1981 Expanded a bit on the installation of the project. 2023-08-20 13:03:06 +02:00