Philipp/aki_prj23_transparenzregister
1
0
Fork 0
mirror of https://github.com/fhswf/aki_prj23_transparenzregister.git synced 2026-02-13 22:17:38 +01:00
Code Issues Packages Projects Releases Wiki Activity
744 Commits 7 Branches 0 Tags
f78d4d83a75ce7e564116618bf22ad52c87cb649
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Tristan NoldeGitHubPhilipp
 		e57ef07a3d hotfix: Improve UI (#569) 
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de>
 2024-01-17 16:22:46 +01:00
Tristan NoldeandGitHub f7832a9656 hotfix: Replace remote-debugging-port with remote-debugging-pipe (#563) 2024-01-14 12:11:17 +01:00
Tristan NoldeGitHubPhilipp
 		d55eeef87e feat(ui): Include graph on person page and fix neighbor calculation (#533) 
Includes the company/person relation graph on the persondetails page
incl. the number of relations on lvl 1,2 and 3, the calculation of which
was also fixed as part of the development.

The graph displayed on the company details page now also includes
**all** relations (company + person) until the 3rd lvl.

---------

Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de>
 2024-01-07 13:53:32 +01:00
Tristan NoldeGitHubPhilipp
 		4d10f8a076 hotfix: Resolve missing env settings on ingest container in deployment (#536) 
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de>
 2024-01-07 13:33:04 +01:00
Tristan NoldeGitHubPhilipp
 		8eb622fef2 docs: Describe find-missing-companies and ingest entrypoints (#535) 
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de>
 2024-01-07 13:24:26 +01:00
Tristan NoldeGitHubPhilipp
 		824e9e48ca docs/dockerfile (#529) 
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de>
 2024-01-07 12:23:54 +01:00
Tristan NoldeGitHubPhilipp
 		98a189b94c docs: Abschlussarbeit on data ingestion (#500) 
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de>
 2024-01-02 15:50:58 +01:00
Tristan NoldeandGitHub a29fa74a40 fix/mermaid in sphinx (#505) 
Solution retrieved from
https://github.com/mgaitan/sphinxcontrib-mermaid/issues/113#issuecomment-1549876281
 2024-01-02 11:57:56 +01:00
Tristan NoldeandGitHub 5a2f2b0e2c fix(docs): Resolve missing images in Sphinx output (#499) 2024-01-01 14:31:56 +01:00
Tristan NoldeGitHubPhilipp
 		c917cac58a build: Include Mongo dump with init in local deployment (#448) 
Added the latest data from the internet-facing staging DB as json files
to the repo (Git LFS) and added another container to the local
deployment that ingests the files upon creation.

---------

Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de>
 2023-12-10 11:43:15 +01:00
Tristan NoldeandGitHub 043501a784 docs: Meeting Notes 2023-12-07 (#447) 2023-12-07 20:50:11 +01:00
Tristan NoldeandGitHub 92d14b5824 Ingest schedule (#391) 
Includes a new "app" running the ingestion jobs (aka fetch_news and
find_missing_companies + enrich_company_financials) on a schedule.

This also fixes an issue with the previous schedule implementation by
persisting the schedule in a file that survives new deployment and
continues where it left off.
 2023-12-04 19:05:52 +01:00
Tristan NoldeandGitHub 9fa46aac29 hotfix: Resolve issue in scrape process (#415) 2023-11-20 20:09:11 +01:00
Tristan NoldeandGitHub dd511af3a5 Adapt ingest schedule (#381) 
Adapt ingest schedule
 2023-11-13 21:33:43 +01:00
TrisNol c1faa2cba6 feat(deployment): Enable setting of schedule for ingest container using ENV var 2023-11-13 20:08:26 +01:00
TrisNol 92489e7ef6 checkpoint: Set schedule interval via ENV variable 2023-11-13 19:54:29 +01:00
Tristan NoldeandGitHub bbc15bc7a2 Feat/116 scheduling tools (#358) 
Init `ingestion` container with `fetch_news` target to retrieve latest
news articles from Tagesschau and Handelsblatt twice a day.

Integration of the `find_missing_companies.py` will follow once this is
merged.
 2023-11-11 14:34:51 +01:00
TrisNol 05ea0fbb33 refactor: Include logger.catch with reraise 2023-11-11 14:30:00 +01:00
TrisNol 5dcf8ecf55 build: Dockerize apps/fetch_news.py as ingestor 2023-11-11 14:30:00 +01:00
TrisNol 170056bf58 test: Cover apps/fetch_news.py with unit tests 2023-11-11 14:30:00 +01:00
TrisNol ac6ca3547b test: Add unit test for news api wrapper 2023-11-11 14:30:00 +01:00
TrisNol ae41cf61bc checkpoint: Resolve error in handelsblatt text fetch 2023-11-11 14:30:00 +01:00
TrisNol a428eb4432 checkpoint: Init news extraction components and main app 2023-11-11 14:30:00 +01:00
Tristan NoldeandGitHub a6d486209a Introduce extended_financial_data code (#357) 
Introducing the previously developed method to fetch the financial data
via table parsing (aka "data lake like solution") in a non-destructive
manner by defaulting to the current RegEx-based behaviour.
 2023-11-11 14:10:20 +01:00
TrisNol b0bcdc6fe1 refactor: PR feedback implemented 2023-11-11 11:18:23 +01:00
Tristan NoldeandPhilipp 834f93a26e Update src/aki_prj23_transparenzregister/utils/data_extraction/bundesanzeiger.py 
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de>
 2023-11-11 11:03:36 +01:00
TrisNol e1b8397f9e feat: Introduce switch for different financial extraction routines 2023-11-11 11:03:36 +01:00
TrisNol 9edf5b1dce test: Increase coverage for multi-column headers 2023-11-11 11:03:36 +01:00
TrisNol 3ba8c0abea refactor: Remove debugging statements 2023-11-11 11:03:36 +01:00
TrisNol 3b1f0425cf deps: Adding html5lib for table parsing via Pandas 2023-11-11 11:03:36 +01:00
TrisNol 801f945c59 temp: Print exception for test debuggin 2023-11-11 11:01:17 +01:00
Tristan NoldeandPhilipp c19697c7f8 Update src/aki_prj23_transparenzregister/utils/data_extraction/bundesanzeiger.py 
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de>
 2023-11-11 11:01:17 +01:00
Tristan NoldeandPhilipp fe7690620a Update src/aki_prj23_transparenzregister/utils/data_extraction/bundesanzeiger.py 
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de>
 2023-11-11 11:01:17 +01:00
TrisNol fecf42d75a test: Unit test new KPI extraction 2023-11-11 11:01:17 +01:00
TrisNol f8a0d58314 feat(data-extraction): Provide KPI table analysis in bundesanzeiger wrapper 2023-11-11 11:01:17 +01:00
TrisNol 815e08a8f1 checkpoint: Transform values to € and normalize column names 2023-11-11 11:01:17 +01:00
TrisNol ec11ae13aa checkpoint: Parse table into dict of financial data 2023-11-11 11:01:17 +01:00
TrisNol 972fcd155e checkpoint: Normalize HTML tables fetched from Bundesanzeiger 2023-11-11 11:01:17 +01:00
Tristan NoldeandGitHub 8781d746e7 hotfix: Add missing networkx dependency (#361) 
Depyloment on Jupiter is currently broken due to missing `networkx`
dependency:

![image](https://github.com/fhswf/aki_prj23_transparenzregister/assets/56017812/8601fb23-e16b-4986-a52a-36a02c6bea29)

Should be fixed by the changes included
 2023-11-10 22:52:18 +01:00
TrisNol c333ad70c5 hotfix: Add missing networkx dependency 2023-11-10 21:47:34 +01:00
TrisNol da72c3d0a8 lint: Format company_elements.py 2023-11-10 19:21:32 +01:00
TrisNol a1d8e942a9 test: Adapt home.py to run unit tests 2023-11-10 19:20:49 +01:00
TrisNolandTim 5cd03306d6 feat: NetworkX in Plotly Dash 2023-11-10 18:48:17 +01:00
TrisNolandTim af0578bb00 feat: NetworkX in Plotly Dash 2023-11-10 18:45:26 +01:00
Tristan NoldeandGitHub f4998a6fae Feat/233 incremental company extension (#322) 
1. New app capable of processing the `missing_company` table has been
added
2. Data transformation for Unternehmensregister data has been extended
to handle v1 and v3 data with a generic layer in between selecting the
right API upon request

Sorry for the big PR, if preferred I can give a quick tour through the
code rather than having you review every line of code
 2023-11-08 14:36:44 +01:00
TrisNol 982cbd7ad4 refactor: Resolve leftover todos 2023-11-05 19:59:13 +01:00
TrisNol f9d3f0eb76 test: Cover apps/find_missing_companies.py 2023-11-05 13:47:06 +01:00
TrisNol f7ec3eaf24 test: Increase test coverage and refactor v3 2023-11-05 12:55:47 +01:00
TrisNol e8d1a37cff test: Extend unit tests 2023-11-04 14:19:41 +01:00
TrisNol 61f94fa3b9 test: Unit tests 2023-11-04 11:24:36 +01:00
TrisNol 7605858234 checkpoint: Apply missing docstrings 2023-11-04 11:24:36 +01:00
TrisNol 1121f26052 refactor: Apply linter feedback 2023-11-04 11:24:36 +01:00
TrisNol d6b07431e7 test: Adapt existing unit tests to refactored imports 2023-11-04 11:24:36 +01:00
TrisNol 042a019628 checkpoint: Refactoring, first working version of processing 2023-11-04 11:24:27 +01:00
TrisNol 2458ad98ff checkpoint: Refactoring data-extraction from unternehmensregister to handle v1 and v3 2023-11-04 11:24:27 +01:00
TrisNol b7f977138d checkpoint: Manual role mapping via ID 2023-11-04 11:24:27 +01:00
TrisNol 2d9e3f19f9 checkpoint: First iteration of fixed mapping 2023-11-04 11:24:27 +01:00
TrisNol 9d7bb07989 checkpoint: Adapt data transformation to new structure 2023-11-04 11:24:27 +01:00
TrisNol 7f8511c9d6 checkpoint: Core data fetch routine 2023-11-04 11:24:27 +01:00
TrisNol d5dbf553c7 lint: Format changes 2023-11-02 16:32:05 +01:00
TrisNol de689f164a lint: Format changes 2023-11-02 16:26:13 +01:00
TrisNol 534a9ca809 fix(data-extraction): Resolve wait function issue using lambda 2023-11-02 16:18:42 +01:00
Tristan NoldeandGitHub d4267c1860 Update docker-compose.yml 2023-10-20 16:36:54 +02:00
Tristan NoldeandGitHub f236b91ae3 fix(data-extraction): self-referencing auditors (#224) 
fix(data-extraction): self-referencing auditors
 2023-10-19 07:16:19 +02:00
TrisNol 83d313150c test: Update to new functions 2023-10-17 18:47:25 +02:00
TrisNol 35d54301e8 fix(data-extraction): Fix data on staging DB using custom function 2023-10-17 18:42:03 +02:00
TrisNol 600039207d test(data-extraction): Adapt unit tests to new behaviour 2023-10-17 18:16:44 +02:00
TrisNol b7d877ef81 fix(data-extraction): Resolve self referencing auditor companies 2023-10-17 18:08:20 +02:00
TrisNol 4058824f15 fix(data-extraction): Resolve regex issue in detecting auditors 2023-10-17 17:56:26 +02:00
Tristan NoldeandGitHub 99b61e7c2e feat(data-extraction): Extract c/o relation from street (#222) 2023-10-15 13:46:10 +02:00
TrisNol f1474feaf8 refactor: Adapt to extended unit tests 2023-10-15 13:21:41 +02:00
Tristan NoldeGitHubPhilipp
 		fd47487367 Update tests/utils/data_extraction/unternehmensregister/transform_test.py 
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de>
 2023-10-15 13:07:34 +02:00
Tristan NoldeGitHubPhilipp
 		f6f4c99147 Update src/aki_prj23_transparenzregister/models/company.py 
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de>
 2023-10-15 13:07:24 +02:00
TrisNol 8db04177be feat(data-extraction): Extract c/o relation from street in company relation 2023-10-15 13:06:32 +02:00
Tristan NoldeandGitHub 7e54ab98c5 fix(data-extraction): Parse date from Gesellschaftsvertrag entry (#221) 2023-10-15 13:06:04 +02:00
TrisNol eba5235dff refactor: Implement PR feedback 2023-10-15 12:05:25 +02:00
Tristan NoldeGitHubPhilipp
 		15ace5382d Update src/aki_prj23_transparenzregister/utils/data_extraction/unternehmensregister/transform.py 
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de>
 2023-10-15 11:56:06 +02:00
Tristan NoldeGitHubPhilipp
 		d34a0ffeac Update src/aki_prj23_transparenzregister/utils/data_extraction/unternehmensregister/transform.py 
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de>
 2023-10-15 11:55:58 +02:00
Tristan NoldeGitHubPhilipp
 		39c13ac74a Update tests/utils/data_extraction/unternehmensregister/transform_test.py 
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de>
 2023-10-15 11:51:11 +02:00
TrisNol b972acee7a fix(data-extraction): Parse date from Gesellschaftsvertrag entry 2023-10-14 18:22:41 +02:00
Tristan NoldeandGitHub f65a377d73 fix(data-extraction): Handle malformed date_of_birth fields (#204) 
fix(data-extraction): Handle malformed date_of_birth fields
 2023-10-09 19:15:37 +02:00
TrisNol 84d0139531 fix(data-extraction): Handle malformed date_of_birth fields 2023-10-07 17:01:34 +02:00
Tristan NoldeandGitHub 7500895982 fix: Add script to fix malformed yearly_result entries (#202) 2023-10-07 12:35:29 +02:00
TrisNol 9cc58ba8be fix: Add script to fix malformed yearly_result entries 2023-10-07 09:11:43 +02:00
Tristan NoldeandGitHub 5137026fab hotfix(jupiter): Missing port forwarding of mongo container to host (#198) 2023-10-06 18:04:21 +02:00
Tristan NoldeGitHubPhilipp
 		5e737162a7 Update docker-compose.yml 
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de>
 2023-10-06 17:59:14 +02:00
TrisNol 54151c5041 hotfix(jupiter): Missing port forwarding of mongo container to host 2023-10-06 17:30:18 +02:00
Tristan NoldeandGitHub bf7c072e87 Fix/company names with quotes (#187) 2023-10-04 20:07:51 +02:00
TrisNol 38d19a835d refactor: Replace last replace with call of dedicated function 2023-10-04 19:39:50 +02:00
TrisNol c2f1670032 refactor: Implement MyPy feedback 2023-10-03 16:43:23 +02:00
TrisNol 0dfe95652d refactor: Implement MyPy feedback 2023-10-03 16:42:09 +02:00
TrisNol 259259953e refactor: Move quote removal funtion to string utils, adapt to requirements 2023-10-03 16:37:54 +02:00
TrisNol 2a446a9937 checkpoint: Remove quotes from company names in relations 2023-10-03 14:33:46 +02:00
TrisNol 49498ad7c0 checkpoint: Remove quotes from company name 2023-10-03 14:33:45 +02:00
Tristan NoldeandGitHub aa6c0efc87 docs: Meeting notes for 2023-09-28 (#164) 
Zum Review
 2023-10-03 14:33:08 +02:00
Tristan NoldeandGitHub 7e9cff046a fix(data-extraction): Parse house-number from street field if possibl… (#179) 2023-10-03 14:26:21 +02:00
TrisNol 231513d45d docs: Adding Seminararbeit on Data Extraction 2023-10-03 12:38:17 +02:00
TrisNol 0ea233e610 Merge branch 'meetings/2023-09-28' of github.com:fhswf/aki_prj23_transparenzregister into meetings/2023-09-28 2023-10-03 12:17:57 +02:00
TrisNol f58b6dfc23 docs: PR feedback implemented 2023-10-03 12:16:21 +02:00
TrisNol dd55b5184c refactor: Initialiased target docs structure 2023-10-03 12:16:21 +02:00