01b4ce00c1
Spellchecking with PyCharm ( #133 )
...
Co-authored-by: KM-R <129882581+KM-R@users.noreply.github.com>
2023-10-02 20:47:42 +02:00
d2d4a436f8
Add a cli interface to choose a configuration ( #163 )
...
- [x] add a cli to the webserver to take env variables into account
- [x] add a cli to the data processing that takes enviromental variable
as a valid source into account
- [x] rework the cli for the reset sql command
- [x] rework the cli for the copying of sql data from one db to another
2023-10-02 20:31:42 +02:00
2abe12f027
Add a function to convert DM to EUR ( #168 )
...
The function is ment to transform the captial dict into a format that
can be added as a kwarg (**norm_capital(capital_dict) to the company
entities.
This PR only contains the function itself.
2023-10-02 19:46:17 +02:00
05472cc16a
Added longitude/latitude and positional accuracy to the company data ( #180 )
2023-10-02 17:18:04 +02:00
TrisNol
ab26a7a01e
fix(data-extraction): Parse house-number from street field if possible, write Straße in full
2023-10-01 21:24:17 +02:00
TrisNol
2050b49fde
fix(data-extraction): Resolve issue in different Bundesanzeiger formats
2023-09-25 18:37:39 +02:00
Tristan Nolde
5c8d20f4c2
Feature/additional stammdaten ( #132 )
...
Feature/additional stammdaten
2023-09-24 15:31:17 +02:00
820fb3e52b
Repaired the SQL copy and reduced the log volume a bit ( #141 )
...
- Added a cli interface to the SQL copy
- Repaired the SQL copy function
- Added the SQL copy function to the scripts
- Reduced the logging verbosity
2023-09-24 15:11:49 +02:00
TrisNol
282d638c11
refactor: Implement PR feedback
2023-09-24 13:46:19 +02:00
TrisNol
5a7472cd3c
checkpoint(data-extraction): Adapt load to update exisitng entries in order to keep yearly_results
2023-09-23 12:07:07 +02:00
TrisNol
1e23a8d5a3
refactor(data-extraction): Move date_to_iso function to string_tools
2023-09-23 10:51:54 +02:00
TrisNol
77f08cd901
Merge branch 'main' into feature/additional-stammdaten
2023-09-23 10:32:09 +02:00
TrisNol
d6223b4192
refactor(data-extraction): Improve variable naming and exception handling
2023-09-23 10:21:26 +02:00
TrisNol
4e25be5466
test(data-extraction): Introduct load.py test and scrape test
2023-09-23 10:07:15 +02:00
TrisNol
d7f167a868
ignore types mypy
2023-09-21 18:08:20 +02:00
TrisNol
e6af96ea6d
test(data-extraction): Host temporary_dir in local env
2023-09-21 17:25:41 +02:00
TrisNol
535c31fc9f
test(data-extraction): Change use of TemporaryDirection
2023-09-21 17:16:25 +02:00
TrisNol
56c2ed55ec
test(data-extraction): Delay file creation in test_rename_latest_file to avoid same timestamps
2023-09-21 16:54:23 +02:00
KM-R
487b2f42d1
update data based on selected company ( #122 )
...
Added UI elements to select a company and update shown data depending on chosen company
---------
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de>
2023-09-19 23:45:10 +02:00
TrisNol
7269e7e6a4
test(data-extraction): Cover transform.py
2023-09-18 20:23:19 +02:00
TrisNol
febcd59e39
test(data-extraction): Include first unit tests
2023-09-17 19:20:28 +02:00
56b6280264
Transfer financials from staging to sql ( #129 )
2023-09-17 13:45:08 +02:00
fea31e543b
Bugfix update for transfer of company data from mongo to sql ( #121 )
...
Fixed the following errors:
- Typo in readme.md
- Mongo delivers the last_update of an company as a string not as date.
Added:
- Entrypoint description in readme.md
2023-09-13 19:14:36 +02:00
ba44b082b8
Data transfer script ( #114 )
...
Transfers data betwenn two sql instances.
Limited in data volume. Should be good enough for now.
---------
Co-authored-by: Tim <tim.ronneburg@outlook.de>
2023-09-11 21:10:36 +02:00
d64f53eca9
Script for the transfer of data from Mongo to SQL ( #80 )
2023-09-11 20:21:35 +02:00
Tristan Nolde
2cd8def200
Feat/fetch financials ( #79 )
2023-09-09 17:28:35 +02:00
TrisNol
b91ecdfe42
test(data-ingestion): Fix myPy complaints regarding tests
2023-09-09 09:57:43 +02:00
TrisNol
2eaa1c5a65
test(utils): Cover added functiosnn in CompanyMongoService
2023-09-08 19:09:27 +02:00
2233b12468
Add sql lite session ( #71 )
...
I added an sql lite session generator.
Changes
The function `def get_session() -> Session`
has changed to
`def get_session(connect_to: ConfigProvider | str) -> Session`
If a JsonFileConfig Is given the postgress conection is checked.
If a string is given that starts with `sqlite:///` an sql db is created.
The use should otherwise be the same.
2023-09-07 18:41:10 +02:00
TrisNol
0a374a10f6
test(data-ingestion): Increase coverage on Bundesanzeiger wrapper
2023-09-06 18:11:47 +02:00
TrisNol
f4a4cbc4f2
test(data-ingestion): Test Bundesanzeiger wrapper
2023-09-06 17:48:00 +02:00
TrisNol
00a5e9ec25
Merge branch 'main' into feat/fetch-financials
2023-09-06 17:16:48 +02:00
06e59b8061
Changes to the database ( #67 )
...
In this PR are pruposed changed that come up when i started the data
migration.
Plese review them an approve and disaprove.
2023-08-24 17:23:19 +02:00
TrisNol
1e15656028
refactor: Pull Auditor extraction into Bundesanzeiger utils
2023-08-18 16:21:52 +02:00
e4a57c9136
Chore/rework workflow ( #52 )
...
* Reworked the lint action
* Removed the file change requirement
* Repaired mypy
* Repaired pip-audit
2023-08-17 22:05:12 +02:00
KM-R
1e7e55e649
Visualization first running dashboard ( #51 )
...
* added session maker
* Update prebuild psycopg-build2
* added table dash
* Update company_stats_dash
* Repaired a test.
* update connector_test
---------
Co-authored-by: Tim <tim.ronneburg@outlook.de>
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de>
2023-08-15 21:30:58 +02:00
TrisNol
775b360ff7
test: Fix unit test of connector.py
2023-08-11 16:50:22 +02:00
TrisNol
46cdaa625f
test: Add Unit test to connector.py
2023-08-11 16:45:05 +02:00
TrisNol
b2b3ab2f04
test: Add test for entities file
2023-08-11 16:35:57 +02:00
TrisNol
4e7f6bca1d
test: Add missing unit tests
2023-08-11 16:29:08 +02:00
TrisNol
d565770b99
checkpoint(db): Refactor mongo utils, extract postgres entities from Juptyer
2023-08-11 15:12:18 +02:00
TrisNol
b05088c966
test: utils.mongo
2023-07-16 15:53:01 +02:00
TrisNol
4b5f63ee62
test: Adding unit tests for news_mongo_service
2023-07-16 15:33:54 +02:00
TrisNol
b788ee3659
test: Adding unit tests
2023-07-16 11:25:21 +02:00
