TrisNol
61f94fa3b9
test: Unit tests
2023-11-04 11:24:36 +01:00
TrisNol
7605858234
checkpoint: Apply missing docstrings
2023-11-04 11:24:36 +01:00
TrisNol
1121f26052
refactor: Apply linter feedback
2023-11-04 11:24:36 +01:00
TrisNol
d6b07431e7
test: Adapt existing unit tests to refactored imports
2023-11-04 11:24:36 +01:00
TrisNol
042a019628
checkpoint: Refactoring, first working version of processing
2023-11-04 11:24:27 +01:00
TrisNol
2458ad98ff
checkpoint: Refactoring data-extraction from unternehmensregister to handle v1 and v3
2023-11-04 11:24:27 +01:00
TrisNol
b7f977138d
checkpoint: Manual role mapping via ID
2023-11-04 11:24:27 +01:00
TrisNol
2d9e3f19f9
checkpoint: First iteration of fixed mapping
2023-11-04 11:24:27 +01:00
TrisNol
9d7bb07989
checkpoint: Adapt data transformation to new structure
2023-11-04 11:24:27 +01:00
TrisNol
7f8511c9d6
checkpoint: Core data fetch routine
2023-11-04 11:24:27 +01:00
TrisNol
d5dbf553c7
lint: Format changes
2023-11-02 16:32:05 +01:00
TrisNol
de689f164a
lint: Format changes
2023-11-02 16:26:13 +01:00
TrisNol
534a9ca809
fix(data-extraction): Resolve wait function issue using lambda
2023-11-02 16:18:42 +01:00
Tristan Nolde and GitHub
d4267c1860
Update docker-compose.yml
2023-10-20 16:36:54 +02:00
Tristan Nolde and GitHub
f236b91ae3
fix(data-extraction): self-referencing auditors ( #224 )
...
fix(data-extraction): self-referencing auditors
2023-10-19 07:16:19 +02:00
TrisNol
83d313150c
test: Update to new functions
2023-10-17 18:47:25 +02:00
TrisNol
35d54301e8
fix(data-extraction): Fix data on staging DB using custom function
2023-10-17 18:42:03 +02:00
TrisNol
600039207d
test(data-extraction): Adapt unit tests to new behaviour
2023-10-17 18:16:44 +02:00
TrisNol
b7d877ef81
fix(data-extraction): Resolve self referencing auditor companies
2023-10-17 18:08:20 +02:00
TrisNol
4058824f15
fix(data-extraction): Resolve regex issue in detecting auditors
2023-10-17 17:56:26 +02:00
Tristan Nolde and GitHub
99b61e7c2e
feat(data-extraction): Extract c/o relation from street ( #222 )
2023-10-15 13:46:10 +02:00
TrisNol
f1474feaf8
refactor: Adapt to extended unit tests
2023-10-15 13:21:41 +02:00
fd47487367
Update tests/utils/data_extraction/unternehmensregister/transform_test.py
...
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de >
2023-10-15 13:07:34 +02:00
f6f4c99147
Update src/aki_prj23_transparenzregister/models/company.py
...
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de >
2023-10-15 13:07:24 +02:00
TrisNol
8db04177be
feat(data-extraction): Extract c/o relation from street in company relation
2023-10-15 13:06:32 +02:00
Tristan Nolde and GitHub
7e54ab98c5
fix(data-extraction): Parse date from Gesellschaftsvertrag entry ( #221 )
2023-10-15 13:06:04 +02:00
TrisNol
eba5235dff
refactor: Implement PR feedback
2023-10-15 12:05:25 +02:00
15ace5382d
Update src/aki_prj23_transparenzregister/utils/data_extraction/unternehmensregister/transform.py
...
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de >
2023-10-15 11:56:06 +02:00
d34a0ffeac
Update src/aki_prj23_transparenzregister/utils/data_extraction/unternehmensregister/transform.py
...
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de >
2023-10-15 11:55:58 +02:00
39c13ac74a
Update tests/utils/data_extraction/unternehmensregister/transform_test.py
...
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de >
2023-10-15 11:51:11 +02:00
TrisNol
b972acee7a
fix(data-extraction): Parse date from Gesellschaftsvertrag entry
2023-10-14 18:22:41 +02:00
Tristan Nolde and GitHub
f65a377d73
fix(data-extraction): Handle malformed date_of_birth fields ( #204 )
...
fix(data-extraction): Handle malformed date_of_birth fields
2023-10-09 19:15:37 +02:00
TrisNol
84d0139531
fix(data-extraction): Handle malformed date_of_birth fields
2023-10-07 17:01:34 +02:00
Tristan Nolde and GitHub
7500895982
fix: Add script to fix malformed yearly_result entries ( #202 )
2023-10-07 12:35:29 +02:00
TrisNol
9cc58ba8be
fix: Add script to fix malformed yearly_result entries
2023-10-07 09:11:43 +02:00
Tristan Nolde and GitHub
5137026fab
hotfix(jupiter): Missing port forwarding of mongo container to host ( #198 )
2023-10-06 18:04:21 +02:00
5e737162a7
Update docker-compose.yml
...
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de >
2023-10-06 17:59:14 +02:00
TrisNol
54151c5041
hotfix(jupiter): Missing port forwarding of mongo container to host
2023-10-06 17:30:18 +02:00
Tristan Nolde and GitHub
bf7c072e87
Fix/company names with quotes ( #187 )
2023-10-04 20:07:51 +02:00
TrisNol
38d19a835d
refactor: Replace last replace with call of dedicated function
2023-10-04 19:39:50 +02:00
TrisNol
c2f1670032
refactor: Implement MyPy feedback
2023-10-03 16:43:23 +02:00
TrisNol
0dfe95652d
refactor: Implement MyPy feedback
2023-10-03 16:42:09 +02:00
TrisNol
259259953e
refactor: Move quote removal funtion to string utils, adapt to requirements
2023-10-03 16:37:54 +02:00
TrisNol
2a446a9937
checkpoint: Remove quotes from company names in relations
2023-10-03 14:33:46 +02:00
TrisNol
49498ad7c0
checkpoint: Remove quotes from company name
2023-10-03 14:33:45 +02:00
Tristan Nolde and GitHub
aa6c0efc87
docs: Meeting notes for 2023-09-28 ( #164 )
...
Zum Review
2023-10-03 14:33:08 +02:00
Tristan Nolde and GitHub
7e9cff046a
fix(data-extraction): Parse house-number from street field if possibl… ( #179 )
2023-10-03 14:26:21 +02:00
TrisNol
231513d45d
docs: Adding Seminararbeit on Data Extraction
2023-10-03 12:38:17 +02:00
TrisNol
0ea233e610
Merge branch 'meetings/2023-09-28' of github.com:fhswf/aki_prj23_transparenzregister into meetings/2023-09-28
2023-10-03 12:17:57 +02:00
TrisNol
f58b6dfc23
docs: PR feedback implemented
2023-10-03 12:16:21 +02:00
TrisNol
dd55b5184c
refactor: Initialiased target docs structure
2023-10-03 12:16:21 +02:00
TrisNol
0d779db1e8
docs: Update meeting notes
2023-10-03 12:16:20 +02:00
Tristan Nolde and Philipp
1c0b23b85d
Update documentations/meeting-notes/Meeting_2023-09-28.md
...
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de >
2023-10-03 12:16:20 +02:00
Tristan Nolde and Philipp
e470316373
Update documentations/meeting-notes/Meeting_2023-09-28.md
...
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de >
2023-10-03 12:16:20 +02:00
Tristan Nolde and Philipp
3eeeaeaba9
Update documentations/meeting-notes/Meeting_2023-09-28.md
...
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de >
2023-10-03 12:16:20 +02:00
TrisNol
63644e56ae
docs: Meeting notes for 2023-09-28
2023-10-03 12:16:20 +02:00
84fb50bad1
Update src/aki_prj23_transparenzregister/utils/data_extraction/unternehmensregister/transform.py
...
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de >
2023-10-02 19:23:57 +02:00
TrisNol
56bfc03ac6
docs: PR feedback implemented
2023-10-02 19:17:35 +02:00
TrisNol
2aec7ce115
refactor: Initialiased target docs structure
2023-10-02 19:17:29 +02:00
TrisNol
e94684050b
docs: Update meeting notes
2023-10-02 19:16:25 +02:00
Tristan Nolde and Philipp
469f8bdd41
Update documentations/meeting-notes/Meeting_2023-09-28.md
...
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de >
2023-10-02 19:16:25 +02:00
Tristan Nolde and Philipp
e551c97130
Update documentations/meeting-notes/Meeting_2023-09-28.md
...
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de >
2023-10-02 19:16:25 +02:00
Tristan Nolde and Philipp
71f442fdc7
Update documentations/meeting-notes/Meeting_2023-09-28.md
...
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de >
2023-10-02 19:16:25 +02:00
TrisNol
8151a18efb
docs: Meeting notes for 2023-09-28
2023-10-02 19:16:24 +02:00
TrisNol
ab26a7a01e
fix(data-extraction): Parse house-number from street field if possible, write Straße in full
2023-10-01 21:24:17 +02:00
Tristan Nolde and GitHub
5fa7cd230a
docs: Adding Draw.io on Solution Arch and timeline ( #105 )
2023-09-25 20:55:27 +02:00
TrisNol
9f3f867ff6
fix(pipeline): Add snapd and install drawio in pipeline
2023-09-25 20:47:17 +02:00
692c8b15cd
Update .github/workflows/documentation.yaml
...
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de >
2023-09-25 20:40:41 +02:00
fa272e6491
Update .github/workflows/documentation.yaml
...
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de >
2023-09-25 20:40:29 +02:00
Tristan Nolde and Philipp
2f8942288e
Update documentations/index.rst
...
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de >
2023-09-25 20:38:04 +02:00
TrisNol and Philipp
dfb93113ac
docs: Show relevant pages of draw.io document
2023-09-25 20:38:00 +02:00
TrisNol and Philipp
3fad05309f
docs: Adding Draw.io on Solution Arch and timeline
2023-09-25 20:37:59 +02:00
Tristan Nolde and GitHub
ac1d5760d0
Bugfix/yearly results with string as keys ( #153 )
...
All entries you found have been updated on prod.
2023-09-25 20:35:23 +02:00
TrisNol
77711d8a2f
feat: Add simple wrapper to update particual financial entries
2023-09-25 19:34:10 +02:00
TrisNol
7b5cf16e49
feat: Add simple wrapper to update particual financial entries
2023-09-25 19:33:23 +02:00
TrisNol
2050b49fde
fix(data-extraction): Resolve issue in different Bundesanzeiger formats
2023-09-25 18:37:39 +02:00
Tristan Nolde and GitHub
5c8d20f4c2
Feature/additional stammdaten ( #132 )
...
Feature/additional stammdaten
2023-09-24 15:31:17 +02:00
TrisNol
81b8752abf
Merge branch 'main' into feature/additional-stammdaten
2023-09-24 13:49:02 +02:00
TrisNol
282d638c11
refactor: Implement PR feedback
2023-09-24 13:46:19 +02:00
TrisNol
5a7472cd3c
checkpoint(data-extraction): Adapt load to update exisitng entries in order to keep yearly_results
2023-09-23 12:07:07 +02:00
TrisNol
1e23a8d5a3
refactor(data-extraction): Move date_to_iso function to string_tools
2023-09-23 10:51:54 +02:00
TrisNol
77f08cd901
Merge branch 'main' into feature/additional-stammdaten
2023-09-23 10:32:09 +02:00
TrisNol
d6223b4192
refactor(data-extraction): Improve variable naming and exception handling
2023-09-23 10:21:26 +02:00
TrisNol
4e25be5466
test(data-extraction): Introduct load.py test and scrape test
2023-09-23 10:07:15 +02:00
TrisNol
d7f167a868
ignore types mypy
2023-09-21 18:08:20 +02:00
TrisNol
3a11a92687
test(data-extraction): Ignore mypy args errors
2023-09-21 17:55:20 +02:00
TrisNol
365f74edc1
test(data-extraction): Replace hard-coded seperator with OS specific
2023-09-21 17:47:30 +02:00
TrisNol
e6af96ea6d
test(data-extraction): Host temporary_dir in local env
2023-09-21 17:25:41 +02:00
TrisNol
535c31fc9f
test(data-extraction): Change use of TemporaryDirection
2023-09-21 17:16:25 +02:00
TrisNol
56c2ed55ec
test(data-extraction): Delay file creation in test_rename_latest_file to avoid same timestamps
2023-09-21 16:54:23 +02:00
TrisNol
7269e7e6a4
test(data-extraction): Cover transform.py
2023-09-18 20:23:19 +02:00
TrisNol
92ea72249c
refactor(data-extraction): Bind company type to an enum
2023-09-17 19:43:22 +02:00
TrisNol
d54c1cd17d
test(data-extraction): Adapt to extended enum
2023-09-17 19:25:30 +02:00
TrisNol
febcd59e39
test(data-extraction): Include first unit tests
2023-09-17 19:20:28 +02:00
TrisNol
bfe50ac76d
checkpoint(data-ingestion): Move Unternehmensregister code to .py
2023-09-15 17:22:54 +02:00
TrisNol
8be192e1de
checkpoint(data-ingestion): Include type in company relations, fix issue in capital for KGs
2023-09-15 15:39:42 +02:00
TrisNol
0c7216e105
checkpoint
2023-09-14 18:17:02 +02:00
TrisNol
413b43c615
checkpoint(data-ingestion): Unify date format in data
2023-09-14 16:47:11 +02:00
TrisNol
cf92cb61cc
checkpoint(data-ingestion): Extract founding_date and other stats
2023-09-12 19:07:23 +02:00
Tristan Nolde and GitHub
0cca5f429e
feat(config): Read secrets from .env file and environemnt variables ( #109 )
...
Config variables used to connect to our two databased can now also be
ingested from the local environment variables. Such variables can also
be passed in using a `.env` file as described in the README.md
2023-09-09 18:41:56 +02:00
TrisNol
