Tristan Nolde and Philipp
3eeeaeaba9
Update documentations/meeting-notes/Meeting_2023-09-28.md
...
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de >
2023-10-03 12:16:20 +02:00
TrisNol
63644e56ae
docs: Meeting notes for 2023-09-28
2023-10-03 12:16:20 +02:00
Tristan Nolde and GitHub
5fa7cd230a
docs: Adding Draw.io on Solution Arch and timeline ( #105 )
2023-09-25 20:55:27 +02:00
TrisNol
9f3f867ff6
fix(pipeline): Add snapd and install drawio in pipeline
2023-09-25 20:47:17 +02:00
692c8b15cd
Update .github/workflows/documentation.yaml
...
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de >
2023-09-25 20:40:41 +02:00
fa272e6491
Update .github/workflows/documentation.yaml
...
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de >
2023-09-25 20:40:29 +02:00
Tristan Nolde and Philipp
2f8942288e
Update documentations/index.rst
...
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de >
2023-09-25 20:38:04 +02:00
TrisNol and Philipp
dfb93113ac
docs: Show relevant pages of draw.io document
2023-09-25 20:38:00 +02:00
TrisNol and Philipp
3fad05309f
docs: Adding Draw.io on Solution Arch and timeline
2023-09-25 20:37:59 +02:00
Tristan Nolde and GitHub
ac1d5760d0
Bugfix/yearly results with string as keys ( #153 )
...
All entries you found have been updated on prod.
2023-09-25 20:35:23 +02:00
TrisNol
77711d8a2f
feat: Add simple wrapper to update particual financial entries
2023-09-25 19:34:10 +02:00
TrisNol
7b5cf16e49
feat: Add simple wrapper to update particual financial entries
2023-09-25 19:33:23 +02:00
TrisNol
2050b49fde
fix(data-extraction): Resolve issue in different Bundesanzeiger formats
2023-09-25 18:37:39 +02:00
Tristan Nolde and GitHub
5c8d20f4c2
Feature/additional stammdaten ( #132 )
...
Feature/additional stammdaten
2023-09-24 15:31:17 +02:00
TrisNol
81b8752abf
Merge branch 'main' into feature/additional-stammdaten
2023-09-24 13:49:02 +02:00
TrisNol
282d638c11
refactor: Implement PR feedback
2023-09-24 13:46:19 +02:00
TrisNol
5a7472cd3c
checkpoint(data-extraction): Adapt load to update exisitng entries in order to keep yearly_results
2023-09-23 12:07:07 +02:00
TrisNol
1e23a8d5a3
refactor(data-extraction): Move date_to_iso function to string_tools
2023-09-23 10:51:54 +02:00
TrisNol
77f08cd901
Merge branch 'main' into feature/additional-stammdaten
2023-09-23 10:32:09 +02:00
TrisNol
d6223b4192
refactor(data-extraction): Improve variable naming and exception handling
2023-09-23 10:21:26 +02:00
TrisNol
4e25be5466
test(data-extraction): Introduct load.py test and scrape test
2023-09-23 10:07:15 +02:00
TrisNol
d7f167a868
ignore types mypy
2023-09-21 18:08:20 +02:00
TrisNol
3a11a92687
test(data-extraction): Ignore mypy args errors
2023-09-21 17:55:20 +02:00
TrisNol
365f74edc1
test(data-extraction): Replace hard-coded seperator with OS specific
2023-09-21 17:47:30 +02:00
TrisNol
e6af96ea6d
test(data-extraction): Host temporary_dir in local env
2023-09-21 17:25:41 +02:00
TrisNol
535c31fc9f
test(data-extraction): Change use of TemporaryDirection
2023-09-21 17:16:25 +02:00
TrisNol
56c2ed55ec
test(data-extraction): Delay file creation in test_rename_latest_file to avoid same timestamps
2023-09-21 16:54:23 +02:00
TrisNol
7269e7e6a4
test(data-extraction): Cover transform.py
2023-09-18 20:23:19 +02:00
TrisNol
92ea72249c
refactor(data-extraction): Bind company type to an enum
2023-09-17 19:43:22 +02:00
TrisNol
d54c1cd17d
test(data-extraction): Adapt to extended enum
2023-09-17 19:25:30 +02:00
TrisNol
febcd59e39
test(data-extraction): Include first unit tests
2023-09-17 19:20:28 +02:00
TrisNol
bfe50ac76d
checkpoint(data-ingestion): Move Unternehmensregister code to .py
2023-09-15 17:22:54 +02:00
TrisNol
8be192e1de
checkpoint(data-ingestion): Include type in company relations, fix issue in capital for KGs
2023-09-15 15:39:42 +02:00
TrisNol
0c7216e105
checkpoint
2023-09-14 18:17:02 +02:00
TrisNol
413b43c615
checkpoint(data-ingestion): Unify date format in data
2023-09-14 16:47:11 +02:00
TrisNol
cf92cb61cc
checkpoint(data-ingestion): Extract founding_date and other stats
2023-09-12 19:07:23 +02:00
Tristan Nolde and GitHub
0cca5f429e
feat(config): Read secrets from .env file and environemnt variables ( #109 )
...
Config variables used to connect to our two databased can now also be
ingested from the local environment variables. Such variables can also
be passed in using a `.env` file as described in the README.md
2023-09-09 18:41:56 +02:00
TrisNol
2c8805e12f
checkpoint: Implement PR feedback
2023-09-09 18:22:20 +02:00
TrisNol
330eb466e3
test(config): Fix test with os.environ mocking
2023-09-09 18:13:59 +02:00
TrisNol
0f6ea8a1ec
feat(config): Read secrets from .env file and environemnt variables
2023-09-09 17:59:30 +02:00
Tristan Nolde and GitHub
2cd8def200
Feat/fetch financials ( #79 )
2023-09-09 17:28:35 +02:00
TrisNol
b91ecdfe42
test(data-ingestion): Fix myPy complaints regarding tests
2023-09-09 09:57:43 +02:00
TrisNol
2eaa1c5a65
test(utils): Cover added functiosnn in CompanyMongoService
2023-09-08 19:09:27 +02:00
TrisNol
4944372ebc
test(date-ingestion): Cover enrich_company_financials
2023-09-08 18:56:10 +02:00
TrisNol
cfb4459a16
checkpoint
2023-09-07 17:46:17 +02:00
TrisNol
0a374a10f6
test(data-ingestion): Increase coverage on Bundesanzeiger wrapper
2023-09-06 18:11:47 +02:00
TrisNol
f4a4cbc4f2
test(data-ingestion): Test Bundesanzeiger wrapper
2023-09-06 17:48:00 +02:00
TrisNol
d34bb1f3be
checkpoint(data-ingestion): Replace print statements with loguru
2023-09-06 17:19:19 +02:00
TrisNol
00a5e9ec25
Merge branch 'main' into feat/fetch-financials
2023-09-06 17:16:48 +02:00
TrisNol
b970416641
checkpoint(data-ingestion): Add method to mongo service to fetch companies with yearly results
2023-08-29 18:50:33 +02:00
TrisNol
7a2bc5db2b
multi-thread company financial ingestion
2023-08-29 18:38:25 +02:00
TrisNol
ac07bd8e8e
improve financial fetch, bind financial KPIs to enum
2023-08-29 17:10:42 +02:00
TrisNol
d1c09d51a2
checkpoint: Update Mongo Company entries with yearly_results
2023-08-19 18:06:33 +02:00
TrisNol
6514db2954
checkpoint: Enrich Company entity with yearly results
2023-08-19 15:12:41 +02:00
TrisNol
e8e354932c
feat(data-extraction): Minimal Financial data fetch
2023-08-18 16:44:49 +02:00
TrisNol
1e15656028
refactor: Pull Auditor extraction into Bundesanzeiger utils
2023-08-18 16:21:52 +02:00
TrisNol
f64e0dd96e
fix: Install deutschland via HTTPS link
2023-08-18 14:27:32 +02:00
TrisNol
eb0962e1be
refactor: Move Auditor dataclass to models
2023-08-18 14:20:34 +02:00
TrisNol
309755383e
Install deutschland package
2023-08-18 14:15:05 +02:00
Tristan Nolde and GitHub
9a25d3ad0f
Merge pull request #49 from fhswf/visualization
...
fixed sqlAlchemy and db connection
2023-08-14 21:23:57 +02:00
TrisNol
775b360ff7
test: Fix unit test of connector.py
2023-08-11 16:50:22 +02:00
TrisNol
46cdaa625f
test: Add Unit test to connector.py
2023-08-11 16:45:05 +02:00
TrisNol
b2b3ab2f04
test: Add test for entities file
2023-08-11 16:35:57 +02:00
TrisNol
e7627aa2b7
refactor: Remove wrong typing from entities
2023-08-11 16:31:58 +02:00
TrisNol
4e7f6bca1d
test: Add missing unit tests
2023-08-11 16:29:08 +02:00
TrisNol
86001a19e9
refactor: Add missing type declaration to ORM entities
2023-08-11 15:59:16 +02:00
TrisNol
db2da67ee4
docs: Add reference on how to use secrets.json
2023-08-11 15:55:54 +02:00
TrisNol
1ca1985f57
Merge branch 'main' into feature/refactor-utils
2023-08-11 15:40:33 +02:00
TrisNol
0203de98a7
feat(config): Read connection from secrets.json file
2023-08-11 15:39:08 +02:00
TrisNol
d565770b99
checkpoint(db): Refactor mongo utils, extract postgres entities from Juptyer
2023-08-11 15:12:18 +02:00
Tristan Nolde and GitHub
d493fd0978
Merge pull request #42 from fhswf/feature/abstract-data-extraction
...
Feature/abstract data extraction
2023-08-10 17:59:37 +02:00
Tristan Nolde and GitHub
2d4933c7f8
Merge pull request #41 from fhswf/pr-helm-deploy-kicluster
...
add current deployed charts (without pw)
2023-08-10 17:58:44 +02:00
TrisNol
8905a7f203
docs: Ignore .ipynb in Sphinx docs
2023-08-10 17:39:35 +02:00
TrisNol
42ef18fc58
refactor(docs): Restructure folders
2023-08-10 17:27:22 +02:00
TrisNol
f1f867c6ae
Merge branch 'main' into feature/abstract-data-extraction
2023-08-10 17:12:01 +02:00
Tristan Nolde and GitHub
ebedf7c630
Merge pull request #39 from fhswf/feature/data-extraktion
...
Feature/data extraktion
2023-07-20 17:11:50 +02:00
TrisNol
1e44197199
add types-tqdm
2023-07-20 16:46:53 +02:00
Tristan Nolde and GitHub
398fac4c43
Update values.yml
2023-07-20 08:33:49 +02:00
Tristan Nolde and GitHub
6a46d58037
Merge pull request #40 from fhswf/feature/k8s
...
k8s deployment values
2023-07-20 08:16:38 +02:00
TrisNol
dcd2a0b8af
k8s deployment values
2023-07-20 08:14:20 +02:00
TrisNol
a81d3638c5
checkpoint
2023-07-20 08:05:07 +02:00
TrisNol
b05088c966
test: utils.mongo
2023-07-16 15:53:01 +02:00
TrisNol
4b5f63ee62
test: Adding unit tests for news_mongo_service
2023-07-16 15:33:54 +02:00
TrisNol
b788ee3659
test: Adding unit tests
2023-07-16 11:25:21 +02:00
TrisNol
9da3e4adb9
test: Introducing first unit tests of new Company and News model
2023-07-11 18:03:23 +02:00
TrisNol
52d7f62bbe
Merge branch 'main' into feature/data-extraktion
2023-07-11 14:35:20 +02:00
TrisNol
ed681d7c47
refactor: Implement linter feedback
2023-07-11 14:20:16 +02:00
TrisNol
4c95550dbf
feat(data-extraction): MongoWrapper, DataClasses and services for News and Company data
2023-07-10 18:58:31 +02:00
TrisNol
4c65d37816
Merge main into feature/data-extraction
2023-07-10 17:15:43 +02:00
TrisNol
e44385ce3a
style: Refactoring imports, adapting MongoConnector to different connection_strings
2023-06-30 20:36:03 +02:00
TrisNol
3cd8860312
adding distric court location to export
2023-06-27 19:49:23 +02:00
TrisNol
421b1e8c87
Bundesanzeiger preparation, Handeslblatt RSS feed export
2023-06-27 19:17:54 +02:00
TrisNol
37fb1b1da3
multi-process scraping, transforming unternehmensregister output
2023-06-25 15:58:53 +02:00
Tristan Nolde and GitHub
f691cac9e8
Merge pull request #27 from fhswf/feature/meeting-notes-09-06-2023
...
docs(meeting-notes): Meeting 09.06.2023
2023-06-19 18:10:59 +02:00
TrisNol
c9c7b0cf7a
code cleanup, presentation on data extraction
2023-06-19 18:02:34 +02:00
TrisNol
6e31bc62bd
mongodb wrapper for managing News objects
2023-06-16 18:50:19 +02:00
TrisNol
5b96bb7e3e
adding company ID as well as compatible dataclasses
2023-06-16 18:00:11 +02:00
TrisNol
d3d8adabad
dockerized mongodb as staging DB
2023-06-15 20:24:39 +02:00
TrisNol
3e737fbac5
first news article data extraction from tagesschau api
2023-06-15 18:04:23 +02:00
TrisNol
058c16b3ff
Bulk process Unternehmensregister .xmls
2023-06-11 13:11:44 +02:00
Tristan NoldeandPhilipp
