Philipp/aki_prj23_transparenzregister
1
0
Fork 0
mirror of https://github.com/fhswf/aki_prj23_transparenzregister.git synced 2026-02-13 22:17:38 +01:00
Code Issues Packages Projects Releases Wiki Activity
501 Commits 7 Branches 0 Tags
ae41cf61bcc8555047f1f929723df79530b184a1
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
TrisNol ae41cf61bc checkpoint: Resolve error in handelsblatt text fetch 2023-11-11 14:30:00 +01:00
TrisNol a428eb4432 checkpoint: Init news extraction components and main app 2023-11-11 14:30:00 +01:00
Tristan NoldeandGitHub a6d486209a Introduce extended_financial_data code (#357) 
Introducing the previously developed method to fetch the financial data
via table parsing (aka "data lake like solution") in a non-destructive
manner by defaulting to the current RegEx-based behaviour.
 2023-11-11 14:10:20 +01:00
TrisNol b0bcdc6fe1 refactor: PR feedback implemented 2023-11-11 11:18:23 +01:00
Tristan NoldeandPhilipp 834f93a26e Update src/aki_prj23_transparenzregister/utils/data_extraction/bundesanzeiger.py 
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de>
 2023-11-11 11:03:36 +01:00
TrisNol e1b8397f9e feat: Introduce switch for different financial extraction routines 2023-11-11 11:03:36 +01:00
TrisNol 9edf5b1dce test: Increase coverage for multi-column headers 2023-11-11 11:03:36 +01:00
TrisNol 3ba8c0abea refactor: Remove debugging statements 2023-11-11 11:03:36 +01:00
TrisNol 3b1f0425cf deps: Adding html5lib for table parsing via Pandas 2023-11-11 11:03:36 +01:00
TrisNol 801f945c59 temp: Print exception for test debuggin 2023-11-11 11:01:17 +01:00
Tristan NoldeandPhilipp c19697c7f8 Update src/aki_prj23_transparenzregister/utils/data_extraction/bundesanzeiger.py 
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de>
 2023-11-11 11:01:17 +01:00
Tristan NoldeandPhilipp fe7690620a Update src/aki_prj23_transparenzregister/utils/data_extraction/bundesanzeiger.py 
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de>
 2023-11-11 11:01:17 +01:00
TrisNol fecf42d75a test: Unit test new KPI extraction 2023-11-11 11:01:17 +01:00
TrisNol f8a0d58314 feat(data-extraction): Provide KPI table analysis in bundesanzeiger wrapper 2023-11-11 11:01:17 +01:00
TrisNol 815e08a8f1 checkpoint: Transform values to € and normalize column names 2023-11-11 11:01:17 +01:00
TrisNol ec11ae13aa checkpoint: Parse table into dict of financial data 2023-11-11 11:01:17 +01:00
TrisNol 972fcd155e checkpoint: Normalize HTML tables fetched from Bundesanzeiger 2023-11-11 11:01:17 +01:00
Tristan NoldeandGitHub 8781d746e7 hotfix: Add missing networkx dependency (#361) 
Depyloment on Jupiter is currently broken due to missing `networkx`
dependency:

![image](https://github.com/fhswf/aki_prj23_transparenzregister/assets/56017812/8601fb23-e16b-4986-a52a-36a02c6bea29)

Should be fixed by the changes included
 2023-11-10 22:52:18 +01:00
TrisNol c333ad70c5 hotfix: Add missing networkx dependency 2023-11-10 21:47:34 +01:00
TrisNol da72c3d0a8 lint: Format company_elements.py 2023-11-10 19:21:32 +01:00
TrisNol a1d8e942a9 test: Adapt home.py to run unit tests 2023-11-10 19:20:49 +01:00
TrisNolandTim 5cd03306d6 feat: NetworkX in Plotly Dash 2023-11-10 18:48:17 +01:00
TrisNolandTim af0578bb00 feat: NetworkX in Plotly Dash 2023-11-10 18:45:26 +01:00
Tristan NoldeandGitHub f4998a6fae Feat/233 incremental company extension (#322) 
1. New app capable of processing the `missing_company` table has been
added
2. Data transformation for Unternehmensregister data has been extended
to handle v1 and v3 data with a generic layer in between selecting the
right API upon request

Sorry for the big PR, if preferred I can give a quick tour through the
code rather than having you review every line of code
 2023-11-08 14:36:44 +01:00
TrisNol 982cbd7ad4 refactor: Resolve leftover todos 2023-11-05 19:59:13 +01:00
TrisNol f9d3f0eb76 test: Cover apps/find_missing_companies.py 2023-11-05 13:47:06 +01:00
TrisNol f7ec3eaf24 test: Increase test coverage and refactor v3 2023-11-05 12:55:47 +01:00
TrisNol e8d1a37cff test: Extend unit tests 2023-11-04 14:19:41 +01:00
TrisNol 61f94fa3b9 test: Unit tests 2023-11-04 11:24:36 +01:00
TrisNol 7605858234 checkpoint: Apply missing docstrings 2023-11-04 11:24:36 +01:00
TrisNol 1121f26052 refactor: Apply linter feedback 2023-11-04 11:24:36 +01:00
TrisNol d6b07431e7 test: Adapt existing unit tests to refactored imports 2023-11-04 11:24:36 +01:00
TrisNol 042a019628 checkpoint: Refactoring, first working version of processing 2023-11-04 11:24:27 +01:00
TrisNol 2458ad98ff checkpoint: Refactoring data-extraction from unternehmensregister to handle v1 and v3 2023-11-04 11:24:27 +01:00
TrisNol b7f977138d checkpoint: Manual role mapping via ID 2023-11-04 11:24:27 +01:00
TrisNol 2d9e3f19f9 checkpoint: First iteration of fixed mapping 2023-11-04 11:24:27 +01:00
TrisNol 9d7bb07989 checkpoint: Adapt data transformation to new structure 2023-11-04 11:24:27 +01:00
TrisNol 7f8511c9d6 checkpoint: Core data fetch routine 2023-11-04 11:24:27 +01:00
TrisNol d5dbf553c7 lint: Format changes 2023-11-02 16:32:05 +01:00
TrisNol de689f164a lint: Format changes 2023-11-02 16:26:13 +01:00
TrisNol 534a9ca809 fix(data-extraction): Resolve wait function issue using lambda 2023-11-02 16:18:42 +01:00
Tristan NoldeandGitHub d4267c1860 Update docker-compose.yml 2023-10-20 16:36:54 +02:00
Tristan NoldeandGitHub f236b91ae3 fix(data-extraction): self-referencing auditors (#224) 
fix(data-extraction): self-referencing auditors
 2023-10-19 07:16:19 +02:00
TrisNol 83d313150c test: Update to new functions 2023-10-17 18:47:25 +02:00
TrisNol 35d54301e8 fix(data-extraction): Fix data on staging DB using custom function 2023-10-17 18:42:03 +02:00
TrisNol 600039207d test(data-extraction): Adapt unit tests to new behaviour 2023-10-17 18:16:44 +02:00
TrisNol b7d877ef81 fix(data-extraction): Resolve self referencing auditor companies 2023-10-17 18:08:20 +02:00
TrisNol 4058824f15 fix(data-extraction): Resolve regex issue in detecting auditors 2023-10-17 17:56:26 +02:00
Tristan NoldeandGitHub 99b61e7c2e feat(data-extraction): Extract c/o relation from street (#222) 2023-10-15 13:46:10 +02:00
TrisNol f1474feaf8 refactor: Adapt to extended unit tests 2023-10-15 13:21:41 +02:00
Tristan NoldeGitHubPhilipp
 		fd47487367 Update tests/utils/data_extraction/unternehmensregister/transform_test.py 
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de>
 2023-10-15 13:07:34 +02:00
Tristan NoldeGitHubPhilipp
 		f6f4c99147 Update src/aki_prj23_transparenzregister/models/company.py 
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de>
 2023-10-15 13:07:24 +02:00
TrisNol 8db04177be feat(data-extraction): Extract c/o relation from street in company relation 2023-10-15 13:06:32 +02:00
Tristan NoldeandGitHub 7e54ab98c5 fix(data-extraction): Parse date from Gesellschaftsvertrag entry (#221) 2023-10-15 13:06:04 +02:00
TrisNol eba5235dff refactor: Implement PR feedback 2023-10-15 12:05:25 +02:00
Tristan NoldeGitHubPhilipp
 		15ace5382d Update src/aki_prj23_transparenzregister/utils/data_extraction/unternehmensregister/transform.py 
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de>
 2023-10-15 11:56:06 +02:00
Tristan NoldeGitHubPhilipp
 		d34a0ffeac Update src/aki_prj23_transparenzregister/utils/data_extraction/unternehmensregister/transform.py 
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de>
 2023-10-15 11:55:58 +02:00
Tristan NoldeGitHubPhilipp
 		39c13ac74a Update tests/utils/data_extraction/unternehmensregister/transform_test.py 
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de>
 2023-10-15 11:51:11 +02:00
TrisNol b972acee7a fix(data-extraction): Parse date from Gesellschaftsvertrag entry 2023-10-14 18:22:41 +02:00
Tristan NoldeandGitHub f65a377d73 fix(data-extraction): Handle malformed date_of_birth fields (#204) 
fix(data-extraction): Handle malformed date_of_birth fields
 2023-10-09 19:15:37 +02:00
TrisNol 84d0139531 fix(data-extraction): Handle malformed date_of_birth fields 2023-10-07 17:01:34 +02:00
Tristan NoldeandGitHub 7500895982 fix: Add script to fix malformed yearly_result entries (#202) 2023-10-07 12:35:29 +02:00
TrisNol 9cc58ba8be fix: Add script to fix malformed yearly_result entries 2023-10-07 09:11:43 +02:00
Tristan NoldeandGitHub 5137026fab hotfix(jupiter): Missing port forwarding of mongo container to host (#198) 2023-10-06 18:04:21 +02:00
Tristan NoldeGitHubPhilipp
 		5e737162a7 Update docker-compose.yml 
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de>
 2023-10-06 17:59:14 +02:00
TrisNol 54151c5041 hotfix(jupiter): Missing port forwarding of mongo container to host 2023-10-06 17:30:18 +02:00
Tristan NoldeandGitHub bf7c072e87 Fix/company names with quotes (#187) 2023-10-04 20:07:51 +02:00
TrisNol 38d19a835d refactor: Replace last replace with call of dedicated function 2023-10-04 19:39:50 +02:00
TrisNol c2f1670032 refactor: Implement MyPy feedback 2023-10-03 16:43:23 +02:00
TrisNol 0dfe95652d refactor: Implement MyPy feedback 2023-10-03 16:42:09 +02:00
TrisNol 259259953e refactor: Move quote removal funtion to string utils, adapt to requirements 2023-10-03 16:37:54 +02:00
TrisNol 2a446a9937 checkpoint: Remove quotes from company names in relations 2023-10-03 14:33:46 +02:00
TrisNol 49498ad7c0 checkpoint: Remove quotes from company name 2023-10-03 14:33:45 +02:00
Tristan NoldeandGitHub aa6c0efc87 docs: Meeting notes for 2023-09-28 (#164) 
Zum Review
 2023-10-03 14:33:08 +02:00
Tristan NoldeandGitHub 7e9cff046a fix(data-extraction): Parse house-number from street field if possibl… (#179) 2023-10-03 14:26:21 +02:00
TrisNol 231513d45d docs: Adding Seminararbeit on Data Extraction 2023-10-03 12:38:17 +02:00
TrisNol 0ea233e610 Merge branch 'meetings/2023-09-28' of github.com:fhswf/aki_prj23_transparenzregister into meetings/2023-09-28 2023-10-03 12:17:57 +02:00
TrisNol f58b6dfc23 docs: PR feedback implemented 2023-10-03 12:16:21 +02:00
TrisNol dd55b5184c refactor: Initialiased target docs structure 2023-10-03 12:16:21 +02:00
TrisNol 0d779db1e8 docs: Update meeting notes 2023-10-03 12:16:20 +02:00
Tristan NoldeandPhilipp 1c0b23b85d Update documentations/meeting-notes/Meeting_2023-09-28.md 
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de>
 2023-10-03 12:16:20 +02:00
Tristan NoldeandPhilipp e470316373 Update documentations/meeting-notes/Meeting_2023-09-28.md 
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de>
 2023-10-03 12:16:20 +02:00
Tristan NoldeandPhilipp 3eeeaeaba9 Update documentations/meeting-notes/Meeting_2023-09-28.md 
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de>
 2023-10-03 12:16:20 +02:00
TrisNol 63644e56ae docs: Meeting notes for 2023-09-28 2023-10-03 12:16:20 +02:00
Tristan NoldeGitHubPhilipp
 		84fb50bad1 Update src/aki_prj23_transparenzregister/utils/data_extraction/unternehmensregister/transform.py 
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de>
 2023-10-02 19:23:57 +02:00
TrisNol 56bfc03ac6 docs: PR feedback implemented 2023-10-02 19:17:35 +02:00
TrisNol 2aec7ce115 refactor: Initialiased target docs structure 2023-10-02 19:17:29 +02:00
TrisNol e94684050b docs: Update meeting notes 2023-10-02 19:16:25 +02:00
Tristan NoldeandPhilipp 469f8bdd41 Update documentations/meeting-notes/Meeting_2023-09-28.md 
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de>
 2023-10-02 19:16:25 +02:00
Tristan NoldeandPhilipp e551c97130 Update documentations/meeting-notes/Meeting_2023-09-28.md 
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de>
 2023-10-02 19:16:25 +02:00
Tristan NoldeandPhilipp 71f442fdc7 Update documentations/meeting-notes/Meeting_2023-09-28.md 
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de>
 2023-10-02 19:16:25 +02:00
TrisNol 8151a18efb docs: Meeting notes for 2023-09-28 2023-10-02 19:16:24 +02:00
TrisNol ab26a7a01e fix(data-extraction): Parse house-number from street field if possible, write Straße in full 2023-10-01 21:24:17 +02:00
Tristan NoldeandGitHub 5fa7cd230a docs: Adding Draw.io on Solution Arch and timeline (#105) 2023-09-25 20:55:27 +02:00
TrisNol 9f3f867ff6 fix(pipeline): Add snapd and install drawio in pipeline 2023-09-25 20:47:17 +02:00
Tristan NoldeGitHubPhilipp
 		692c8b15cd Update .github/workflows/documentation.yaml 
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de>
 2023-09-25 20:40:41 +02:00
Tristan NoldeGitHubPhilipp
 		fa272e6491 Update .github/workflows/documentation.yaml 
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de>
 2023-09-25 20:40:29 +02:00
Tristan NoldeandPhilipp 2f8942288e Update documentations/index.rst 
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de>
 2023-09-25 20:38:04 +02:00
TrisNolandPhilipp dfb93113ac docs: Show relevant pages of draw.io document 2023-09-25 20:38:00 +02:00
TrisNolandPhilipp 3fad05309f docs: Adding Draw.io on Solution Arch and timeline 2023-09-25 20:37:59 +02:00