Tristan Nolde
ac1d5760d0
Bugfix/yearly results with string as keys ( #153 )
...
All entries you found have been updated on prod.
2023-09-25 20:35:23 +02:00
0fdfcc938a
Added 2 Seminararbeiten to sphinx ( #154 )
...
Added 2 Seminararbeiten to sphinx
2023-09-25 20:30:43 +02:00
9cc0654f56
Delete untagged images ( #151 )
...
This change to the pipline deletes untagged / overwritten tags once a
week to keep the memory consumption in the GH registry reasonable.
2023-09-25 20:28:49 +02:00
5e716801c0
Change usage of black to the official black action ( #152 )
2023-09-25 20:26:02 +02:00
TrisNol
77711d8a2f
feat: Add simple wrapper to update particual financial entries
2023-09-25 19:34:10 +02:00
TrisNol
7b5cf16e49
feat: Add simple wrapper to update particual financial entries
2023-09-25 19:33:23 +02:00
TrisNol
2050b49fde
fix(data-extraction): Resolve issue in different Bundesanzeiger formats
2023-09-25 18:37:39 +02:00
5bbdf046d2
Ruff fix after merge ( #149 )
2023-09-24 18:38:05 +02:00
091e67de79
build first set of docker container in pipline and place them in the the github registry ( #142 )
...
- added a Dockerfile for the thre containers
- added a workflow step to build and placing the container in the
registry
- added a docker-compose.yaml to use the build images
- added a docker compose to build the images locally and a script for
prebuild steps
2023-09-24 16:32:52 +00:00
Tristan Nolde
5c8d20f4c2
Feature/additional stammdaten ( #132 )
...
Feature/additional stammdaten
2023-09-24 15:31:17 +02:00
820fb3e52b
Repaired the SQL copy and reduced the log volume a bit ( #141 )
...
- Added a cli interface to the SQL copy
- Repaired the SQL copy function
- Added the SQL copy function to the scripts
- Reduced the logging verbosity
2023-09-24 15:11:49 +02:00
TrisNol
81b8752abf
Merge branch 'main' into feature/additional-stammdaten
2023-09-24 13:49:02 +02:00
TrisNol
282d638c11
refactor: Implement PR feedback
2023-09-24 13:46:19 +02:00
d9ca808efa
Mypy does not validate if type: ignore comments are needed. ( #137 )
2023-09-23 12:44:27 +02:00
TrisNol
5a7472cd3c
checkpoint(data-extraction): Adapt load to update exisitng entries in order to keep yearly_results
2023-09-23 12:07:07 +02:00
TrisNol
1e23a8d5a3
refactor(data-extraction): Move date_to_iso function to string_tools
2023-09-23 10:51:54 +02:00
c5a383e903
Pipline / dependency updates ( #145 )
2023-09-23 10:49:14 +02:00
TrisNol
77f08cd901
Merge branch 'main' into feature/additional-stammdaten
2023-09-23 10:32:09 +02:00
TrisNol
d6223b4192
refactor(data-extraction): Improve variable naming and exception handling
2023-09-23 10:21:26 +02:00
TrisNol
4e25be5466
test(data-extraction): Introduct load.py test and scrape test
2023-09-23 10:07:15 +02:00
286636dfa4
Reduced the execution of tests and lint actions on pull requests ( #143 )
...
This change will change the pipline to limit duplicat execution of tests
and lint actions in pull requests to save computing time.
2023-09-23 09:22:14 +02:00
495cd90d85
Reduced required code coverage to 70% ( #140 )
2023-09-21 18:09:16 +02:00
TrisNol
d7f167a868
ignore types mypy
2023-09-21 18:08:20 +02:00
TrisNol
3a11a92687
test(data-extraction): Ignore mypy args errors
2023-09-21 17:55:20 +02:00
TrisNol
365f74edc1
test(data-extraction): Replace hard-coded seperator with OS specific
2023-09-21 17:47:30 +02:00
TrisNol
e6af96ea6d
test(data-extraction): Host temporary_dir in local env
2023-09-21 17:25:41 +02:00
TrisNol
535c31fc9f
test(data-extraction): Change use of TemporaryDirection
2023-09-21 17:16:25 +02:00
TrisNol
56c2ed55ec
test(data-extraction): Delay file creation in test_rename_latest_file to avoid same timestamps
2023-09-21 16:54:23 +02:00
KM-R
487b2f42d1
update data based on selected company ( #122 )
...
Added UI elements to select a company and update shown data depending on chosen company
---------
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de>
2023-09-19 23:45:10 +02:00
TrisNol
7269e7e6a4
test(data-extraction): Cover transform.py
2023-09-18 20:23:19 +02:00
TrisNol
92ea72249c
refactor(data-extraction): Bind company type to an enum
2023-09-17 19:43:22 +02:00
TrisNol
d54c1cd17d
test(data-extraction): Adapt to extended enum
2023-09-17 19:25:30 +02:00
TrisNol
febcd59e39
test(data-extraction): Include first unit tests
2023-09-17 19:20:28 +02:00
80f077ee7a
Version ups ( #117 )
...
Rutine version ups for
- python-depdencies
- pre-commits
- piplines
2023-09-17 13:50:53 +02:00
06fef5361d
Lightend black in the pipline building ( #128 )
...
Pipline should now run now 20 -40s faster.
2023-09-17 13:47:20 +02:00
56b6280264
Transfer financials from staging to sql ( #129 )
2023-09-17 13:45:08 +02:00
TrisNol
bfe50ac76d
checkpoint(data-ingestion): Move Unternehmensregister code to .py
2023-09-15 17:22:54 +02:00
TrisNol
8be192e1de
checkpoint(data-ingestion): Include type in company relations, fix issue in capital for KGs
2023-09-15 15:39:42 +02:00
TrisNol
0c7216e105
checkpoint
2023-09-14 18:17:02 +02:00
TrisNol
413b43c615
checkpoint(data-ingestion): Unify date format in data
2023-09-14 16:47:11 +02:00
fea31e543b
Bugfix update for transfer of company data from mongo to sql ( #121 )
...
Fixed the following errors:
- Typo in readme.md
- Mongo delivers the last_update of an company as a string not as date.
Added:
- Entrypoint description in readme.md
2023-09-13 19:14:36 +02:00
507647d164
Added custom 404 error code page in the sphinx docs ( #115 )
2023-09-13 19:01:09 +02:00
TrisNol
cf92cb61cc
checkpoint(data-ingestion): Extract founding_date and other stats
2023-09-12 19:07:23 +02:00
ba44b082b8
Data transfer script ( #114 )
...
Transfers data betwenn two sql instances.
Limited in data volume. Should be good enough for now.
---------
Co-authored-by: Tim <tim.ronneburg@outlook.de>
2023-09-11 21:10:36 +02:00
d64f53eca9
Script for the transfer of data from Mongo to SQL ( #80 )
2023-09-11 20:21:35 +02:00
2ea99c8c17
Removed unneded lines ☺ ( #113 )
2023-09-11 19:49:08 +02:00
cee1ef901a
Updatest vor everything ( #106 )
...
- direct dependencies
- other depdenencies
- pre-commit
2023-09-09 16:47:48 +00:00
Tristan Nolde
0cca5f429e
feat(config): Read secrets from .env file and environemnt variables ( #109 )
...
Config variables used to connect to our two databased can now also be
ingested from the local environment variables. Such variables can also
be passed in using a `.env` file as described in the README.md
2023-09-09 18:41:56 +02:00
23a193a4fb
Added a small test ☺ ( #108 )
2023-09-09 18:35:50 +02:00
TrisNol
2c8805e12f
checkpoint: Implement PR feedback
2023-09-09 18:22:20 +02:00
