Tristan Nolde and GitHub
0cca5f429e
feat(config): Read secrets from .env file and environemnt variables ( #109 )
...
Config variables used to connect to our two databased can now also be
ingested from the local environment variables. Such variables can also
be passed in using a `.env` file as described in the README.md
2023-09-09 18:41:56 +02:00
TrisNol
2c8805e12f
checkpoint: Implement PR feedback
2023-09-09 18:22:20 +02:00
TrisNol
330eb466e3
test(config): Fix test with os.environ mocking
2023-09-09 18:13:59 +02:00
TrisNol
0f6ea8a1ec
feat(config): Read secrets from .env file and environemnt variables
2023-09-09 17:59:30 +02:00
Tristan Nolde and GitHub
2cd8def200
Feat/fetch financials ( #79 )
2023-09-09 17:28:35 +02:00
TrisNol
b91ecdfe42
test(data-ingestion): Fix myPy complaints regarding tests
2023-09-09 09:57:43 +02:00
TrisNol
2eaa1c5a65
test(utils): Cover added functiosnn in CompanyMongoService
2023-09-08 19:09:27 +02:00
TrisNol
4944372ebc
test(date-ingestion): Cover enrich_company_financials
2023-09-08 18:56:10 +02:00
TrisNol
cfb4459a16
checkpoint
2023-09-07 17:46:17 +02:00
TrisNol
0a374a10f6
test(data-ingestion): Increase coverage on Bundesanzeiger wrapper
2023-09-06 18:11:47 +02:00
TrisNol
f4a4cbc4f2
test(data-ingestion): Test Bundesanzeiger wrapper
2023-09-06 17:48:00 +02:00
TrisNol
d34bb1f3be
checkpoint(data-ingestion): Replace print statements with loguru
2023-09-06 17:19:19 +02:00
TrisNol
00a5e9ec25
Merge branch 'main' into feat/fetch-financials
2023-09-06 17:16:48 +02:00
TrisNol
b970416641
checkpoint(data-ingestion): Add method to mongo service to fetch companies with yearly results
2023-08-29 18:50:33 +02:00
TrisNol
7a2bc5db2b
multi-thread company financial ingestion
2023-08-29 18:38:25 +02:00
TrisNol
ac07bd8e8e
improve financial fetch, bind financial KPIs to enum
2023-08-29 17:10:42 +02:00
TrisNol
d1c09d51a2
checkpoint: Update Mongo Company entries with yearly_results
2023-08-19 18:06:33 +02:00
TrisNol
6514db2954
checkpoint: Enrich Company entity with yearly results
2023-08-19 15:12:41 +02:00
TrisNol
e8e354932c
feat(data-extraction): Minimal Financial data fetch
2023-08-18 16:44:49 +02:00
TrisNol
1e15656028
refactor: Pull Auditor extraction into Bundesanzeiger utils
2023-08-18 16:21:52 +02:00
TrisNol
f64e0dd96e
fix: Install deutschland via HTTPS link
2023-08-18 14:27:32 +02:00
TrisNol
eb0962e1be
refactor: Move Auditor dataclass to models
2023-08-18 14:20:34 +02:00
TrisNol
309755383e
Install deutschland package
2023-08-18 14:15:05 +02:00
Tristan Nolde and GitHub
9a25d3ad0f
Merge pull request #49 from fhswf/visualization
...
fixed sqlAlchemy and db connection
2023-08-14 21:23:57 +02:00
TrisNol
775b360ff7
test: Fix unit test of connector.py
2023-08-11 16:50:22 +02:00
TrisNol
46cdaa625f
test: Add Unit test to connector.py
2023-08-11 16:45:05 +02:00
TrisNol
b2b3ab2f04
test: Add test for entities file
2023-08-11 16:35:57 +02:00
TrisNol
e7627aa2b7
refactor: Remove wrong typing from entities
2023-08-11 16:31:58 +02:00
TrisNol
4e7f6bca1d
test: Add missing unit tests
2023-08-11 16:29:08 +02:00
TrisNol
86001a19e9
refactor: Add missing type declaration to ORM entities
2023-08-11 15:59:16 +02:00
TrisNol
db2da67ee4
docs: Add reference on how to use secrets.json
2023-08-11 15:55:54 +02:00
TrisNol
1ca1985f57
Merge branch 'main' into feature/refactor-utils
2023-08-11 15:40:33 +02:00
TrisNol
0203de98a7
feat(config): Read connection from secrets.json file
2023-08-11 15:39:08 +02:00
TrisNol
d565770b99
checkpoint(db): Refactor mongo utils, extract postgres entities from Juptyer
2023-08-11 15:12:18 +02:00
Tristan Nolde and GitHub
d493fd0978
Merge pull request #42 from fhswf/feature/abstract-data-extraction
...
Feature/abstract data extraction
2023-08-10 17:59:37 +02:00
Tristan Nolde and GitHub
2d4933c7f8
Merge pull request #41 from fhswf/pr-helm-deploy-kicluster
...
add current deployed charts (without pw)
2023-08-10 17:58:44 +02:00
TrisNol
8905a7f203
docs: Ignore .ipynb in Sphinx docs
2023-08-10 17:39:35 +02:00
TrisNol
42ef18fc58
refactor(docs): Restructure folders
2023-08-10 17:27:22 +02:00
TrisNol
f1f867c6ae
Merge branch 'main' into feature/abstract-data-extraction
2023-08-10 17:12:01 +02:00
Tristan Nolde and GitHub
ebedf7c630
Merge pull request #39 from fhswf/feature/data-extraktion
...
Feature/data extraktion
2023-07-20 17:11:50 +02:00
TrisNol
1e44197199
add types-tqdm
2023-07-20 16:46:53 +02:00
Tristan Nolde and GitHub
398fac4c43
Update values.yml
2023-07-20 08:33:49 +02:00
Tristan Nolde and GitHub
6a46d58037
Merge pull request #40 from fhswf/feature/k8s
...
k8s deployment values
2023-07-20 08:16:38 +02:00
TrisNol
dcd2a0b8af
k8s deployment values
2023-07-20 08:14:20 +02:00
TrisNol
a81d3638c5
checkpoint
2023-07-20 08:05:07 +02:00
TrisNol
b05088c966
test: utils.mongo
2023-07-16 15:53:01 +02:00
TrisNol
4b5f63ee62
test: Adding unit tests for news_mongo_service
2023-07-16 15:33:54 +02:00
TrisNol
b788ee3659
test: Adding unit tests
2023-07-16 11:25:21 +02:00
TrisNol
9da3e4adb9
test: Introducing first unit tests of new Company and News model
2023-07-11 18:03:23 +02:00
TrisNol
52d7f62bbe
Merge branch 'main' into feature/data-extraktion
2023-07-11 14:35:20 +02:00
TrisNol
ed681d7c47
refactor: Implement linter feedback
2023-07-11 14:20:16 +02:00
TrisNol
4c95550dbf
feat(data-extraction): MongoWrapper, DataClasses and services for News and Company data
2023-07-10 18:58:31 +02:00
TrisNol
4c65d37816
Merge main into feature/data-extraction
2023-07-10 17:15:43 +02:00
TrisNol
e44385ce3a
style: Refactoring imports, adapting MongoConnector to different connection_strings
2023-06-30 20:36:03 +02:00
TrisNol
3cd8860312
adding distric court location to export
2023-06-27 19:49:23 +02:00
TrisNol
421b1e8c87
Bundesanzeiger preparation, Handeslblatt RSS feed export
2023-06-27 19:17:54 +02:00
TrisNol
37fb1b1da3
multi-process scraping, transforming unternehmensregister output
2023-06-25 15:58:53 +02:00
Tristan Nolde and GitHub
f691cac9e8
Merge pull request #27 from fhswf/feature/meeting-notes-09-06-2023
...
docs(meeting-notes): Meeting 09.06.2023
2023-06-19 18:10:59 +02:00
TrisNol
c9c7b0cf7a
code cleanup, presentation on data extraction
2023-06-19 18:02:34 +02:00
TrisNol
6e31bc62bd
mongodb wrapper for managing News objects
2023-06-16 18:50:19 +02:00
TrisNol
5b96bb7e3e
adding company ID as well as compatible dataclasses
2023-06-16 18:00:11 +02:00
TrisNol
d3d8adabad
dockerized mongodb as staging DB
2023-06-15 20:24:39 +02:00
TrisNol
3e737fbac5
first news article data extraction from tagesschau api
2023-06-15 18:04:23 +02:00
TrisNol
058c16b3ff
Bulk process Unternehmensregister .xmls
2023-06-11 13:11:44 +02:00
TrisNol
1010b43a5f
Extract first stakeholder informationen from Unternehmensregister export
2023-06-09 14:23:56 +02:00
TrisNol
e2ad2d475a
Traverse all pages
2023-06-09 13:51:36 +02:00
TrisNol
d69368318f
Download Unternehmensregister export via Selenium
2023-06-09 13:01:46 +02:00
TrisNol
8fce1ffc7e
docs(meeting-notes): Meeting 09.06.2023
2023-06-09 10:47:56 +02:00
Tristan Nolde and GitHub
78e5ecb7e4
Add files via upload
2023-05-11 16:09:59 +02:00
TrisNol
e6446ac0ad
feat(docs): Abstract on automated data extraction
2023-05-11 16:09:15 +02:00
Tristan Nolde and GitHub
bd9f2369d2
Merge pull request #6 from fhswf/feature/docs-on-timeseries
...
Feature/docs on timeseries
2023-05-01 12:01:18 +02:00
TrisNol
878e95f5ba
resolve merge conflicts
2023-05-01 11:34:50 +02:00
TrisNol
0b4d955d26
feat(data extraction): Scraping data from Bundesanzeiger and parsing the results
2023-05-01 11:33:50 +02:00
TrisNol
0d2ead8a17
docs: Timeseries
2023-05-01 11:33:31 +02:00
TrisNol
dd06451e6f
feat(data extraction): Scraping data from Bundesanzeiger and parsing the results
2023-05-01 11:30:37 +02:00
TrisNol
7de8b09a11
docs: Timeseries
2023-04-13 18:42:29 +02:00
TrisNol
edb5e02378
hotfix(pre-commit): Rename to .yaml, remove default python version
2023-04-12 11:07:27 +02:00
TrisNol
a79c5b6560
extended bundesanzeiger API test
2023-04-12 10:58:39 +02:00
TrisNol
1762f41cb1
api tests: Adding previous test results for different apis
2023-04-07 11:51:55 +02:00
TrisNol
09333c00dc
docs: Meeting note template created, Tasks for first weekly
2023-04-06 19:11:29 +02:00
Tristan NoldeandGitHub