TrisNol
259259953e
refactor: Move quote removal funtion to string utils, adapt to requirements
2023-10-03 16:37:54 +02:00
TrisNol
2a446a9937
checkpoint: Remove quotes from company names in relations
2023-10-03 14:33:46 +02:00
TrisNol
49498ad7c0
checkpoint: Remove quotes from company name
2023-10-03 14:33:45 +02:00
Tristan Nolde
7e9cff046a
fix(data-extraction): Parse house-number from street field if possibl… ( #179 )
2023-10-03 14:26:21 +02:00
01b4ce00c1
Spellchecking with PyCharm ( #133 )
...
Co-authored-by: KM-R <129882581+KM-R@users.noreply.github.com>
2023-10-02 20:47:42 +02:00
d2d4a436f8
Add a cli interface to choose a configuration ( #163 )
...
- [x] add a cli to the webserver to take env variables into account
- [x] add a cli to the data processing that takes enviromental variable
as a valid source into account
- [x] rework the cli for the reset sql command
- [x] rework the cli for the copying of sql data from one db to another
2023-10-02 20:31:42 +02:00
2abe12f027
Add a function to convert DM to EUR ( #168 )
...
The function is ment to transform the captial dict into a format that
can be added as a kwarg (**norm_capital(capital_dict) to the company
entities.
This PR only contains the function itself.
2023-10-02 19:46:17 +02:00
05472cc16a
Added longitude/latitude and positional accuracy to the company data ( #180 )
2023-10-02 17:18:04 +02:00
TrisNol
ab26a7a01e
fix(data-extraction): Parse house-number from street field if possible, write Straße in full
2023-10-01 21:24:17 +02:00
KM-R
9566276047
Create multi page layout ( #147 )
...
Created two pages (home and company), page reloads after company
selection in dropdown or clicking the home button.
2023-09-26 18:38:40 +02:00
TrisNol
77711d8a2f
feat: Add simple wrapper to update particual financial entries
2023-09-25 19:34:10 +02:00
TrisNol
2050b49fde
fix(data-extraction): Resolve issue in different Bundesanzeiger formats
2023-09-25 18:37:39 +02:00
091e67de79
build first set of docker container in pipline and place them in the the github registry ( #142 )
...
- added a Dockerfile for the thre containers
- added a workflow step to build and placing the container in the
registry
- added a docker-compose.yaml to use the build images
- added a docker compose to build the images locally and a script for
prebuild steps
2023-09-24 16:32:52 +00:00
Tristan Nolde
5c8d20f4c2
Feature/additional stammdaten ( #132 )
...
Feature/additional stammdaten
2023-09-24 15:31:17 +02:00
820fb3e52b
Repaired the SQL copy and reduced the log volume a bit ( #141 )
...
- Added a cli interface to the SQL copy
- Repaired the SQL copy function
- Added the SQL copy function to the scripts
- Reduced the logging verbosity
2023-09-24 15:11:49 +02:00
TrisNol
282d638c11
refactor: Implement PR feedback
2023-09-24 13:46:19 +02:00
TrisNol
5a7472cd3c
checkpoint(data-extraction): Adapt load to update exisitng entries in order to keep yearly_results
2023-09-23 12:07:07 +02:00
TrisNol
1e23a8d5a3
refactor(data-extraction): Move date_to_iso function to string_tools
2023-09-23 10:51:54 +02:00
TrisNol
77f08cd901
Merge branch 'main' into feature/additional-stammdaten
2023-09-23 10:32:09 +02:00
TrisNol
d6223b4192
refactor(data-extraction): Improve variable naming and exception handling
2023-09-23 10:21:26 +02:00
TrisNol
4e25be5466
test(data-extraction): Introduct load.py test and scrape test
2023-09-23 10:07:15 +02:00
TrisNol
d7f167a868
ignore types mypy
2023-09-21 18:08:20 +02:00
TrisNol
e6af96ea6d
test(data-extraction): Host temporary_dir in local env
2023-09-21 17:25:41 +02:00
TrisNol
535c31fc9f
test(data-extraction): Change use of TemporaryDirection
2023-09-21 17:16:25 +02:00
TrisNol
56c2ed55ec
test(data-extraction): Delay file creation in test_rename_latest_file to avoid same timestamps
2023-09-21 16:54:23 +02:00
KM-R
487b2f42d1
update data based on selected company ( #122 )
...
Added UI elements to select a company and update shown data depending on chosen company
---------
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de>
2023-09-19 23:45:10 +02:00
TrisNol
7269e7e6a4
test(data-extraction): Cover transform.py
2023-09-18 20:23:19 +02:00
TrisNol
febcd59e39
test(data-extraction): Include first unit tests
2023-09-17 19:20:28 +02:00
56b6280264
Transfer financials from staging to sql ( #129 )
2023-09-17 13:45:08 +02:00
fea31e543b
Bugfix update for transfer of company data from mongo to sql ( #121 )
...
Fixed the following errors:
- Typo in readme.md
- Mongo delivers the last_update of an company as a string not as date.
Added:
- Entrypoint description in readme.md
2023-09-13 19:14:36 +02:00
ba44b082b8
Data transfer script ( #114 )
...
Transfers data betwenn two sql instances.
Limited in data volume. Should be good enough for now.
---------
Co-authored-by: Tim <tim.ronneburg@outlook.de>
2023-09-11 21:10:36 +02:00
d64f53eca9
Script for the transfer of data from Mongo to SQL ( #80 )
2023-09-11 20:21:35 +02:00
Tristan Nolde
0cca5f429e
feat(config): Read secrets from .env file and environemnt variables ( #109 )
...
Config variables used to connect to our two databased can now also be
ingested from the local environment variables. Such variables can also
be passed in using a `.env` file as described in the README.md
2023-09-09 18:41:56 +02:00
23a193a4fb
Added a small test ☺ ( #108 )
2023-09-09 18:35:50 +02:00
TrisNol
330eb466e3
test(config): Fix test with os.environ mocking
2023-09-09 18:13:59 +02:00
TrisNol
0f6ea8a1ec
feat(config): Read secrets from .env file and environemnt variables
2023-09-09 17:59:30 +02:00
Tristan Nolde
2cd8def200
Feat/fetch financials ( #79 )
2023-09-09 17:28:35 +02:00
TrisNol
b91ecdfe42
test(data-ingestion): Fix myPy complaints regarding tests
2023-09-09 09:57:43 +02:00
TrisNol
2eaa1c5a65
test(utils): Cover added functiosnn in CompanyMongoService
2023-09-08 19:09:27 +02:00
TrisNol
4944372ebc
test(date-ingestion): Cover enrich_company_financials
2023-09-08 18:56:10 +02:00
2233b12468
Add sql lite session ( #71 )
...
I added an sql lite session generator.
Changes
The function `def get_session() -> Session`
has changed to
`def get_session(connect_to: ConfigProvider | str) -> Session`
If a JsonFileConfig Is given the postgress conection is checked.
If a string is given that starts with `sqlite:///` an sql db is created.
The use should otherwise be the same.
2023-09-07 18:41:10 +02:00
ecf94d4b61
Added an import smoke test.
2023-09-06 19:08:44 +02:00
TrisNol
0a374a10f6
test(data-ingestion): Increase coverage on Bundesanzeiger wrapper
2023-09-06 18:11:47 +02:00
TrisNol
f4a4cbc4f2
test(data-ingestion): Test Bundesanzeiger wrapper
2023-09-06 17:48:00 +02:00
TrisNol
00a5e9ec25
Merge branch 'main' into feat/fetch-financials
2023-09-06 17:16:48 +02:00
KM-R
80faf57c77
54 get data from multiple tables postgres ( #72 )
...
Create first dash board including header, basic company information and
finance plot
---------
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de>
2023-09-03 16:18:19 +02:00
06e59b8061
Changes to the database ( #67 )
...
In this PR are pruposed changed that come up when i started the data
migration.
Plese review them an approve and disaprove.
2023-08-24 17:23:19 +02:00
TrisNol
1e15656028
refactor: Pull Auditor extraction into Bundesanzeiger utils
2023-08-18 16:21:52 +02:00
e4a57c9136
Chore/rework workflow ( #52 )
...
* Reworked the lint action
* Removed the file change requirement
* Repaired mypy
* Repaired pip-audit
2023-08-17 22:05:12 +02:00
KM-R
1e7e55e649
Visualization first running dashboard ( #51 )
...
* added session maker
* Update prebuild psycopg-build2
* added table dash
* Update company_stats_dash
* Repaired a test.
* update connector_test
---------
Co-authored-by: Tim <tim.ronneburg@outlook.de>
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de>
2023-08-15 21:30:58 +02:00
