mirror of
https://github.com/fhswf/aki_prj23_transparenzregister.git
synced 2025-05-14 17:58:45 +02:00
Merge branch 'main' into feat/fetch-financials
This commit is contained in:
commit
00a5e9ec25
2
.github/workflows/lint-actions.yaml
vendored
2
.github/workflows/lint-actions.yaml
vendored
@ -53,7 +53,7 @@ jobs:
|
|||||||
- uses: actions/checkout@v3
|
- uses: actions/checkout@v3
|
||||||
- uses: chartboost/ruff-action@v1
|
- uses: chartboost/ruff-action@v1
|
||||||
with:
|
with:
|
||||||
version: 0.0.277
|
version: 0.0.287
|
||||||
|
|
||||||
python-requirements:
|
python-requirements:
|
||||||
name: Check Python Requirements
|
name: Check Python Requirements
|
||||||
|
@ -26,7 +26,7 @@ repos:
|
|||||||
|
|
||||||
- repo: https://github.com/astral-sh/ruff-pre-commit
|
- repo: https://github.com/astral-sh/ruff-pre-commit
|
||||||
# Ruff version.
|
# Ruff version.
|
||||||
rev: v0.0.284
|
rev: v0.0.287
|
||||||
hooks:
|
hooks:
|
||||||
- id: ruff
|
- id: ruff
|
||||||
args: [--fix, --exit-non-zero-on-fix]
|
args: [--fix, --exit-non-zero-on-fix]
|
||||||
@ -53,7 +53,7 @@ repos:
|
|||||||
|
|
||||||
|
|
||||||
- repo: https://github.com/pre-commit/mirrors-mypy
|
- repo: https://github.com/pre-commit/mirrors-mypy
|
||||||
rev: v1.5.0
|
rev: v1.5.1
|
||||||
hooks:
|
hooks:
|
||||||
- id: mypy
|
- id: mypy
|
||||||
additional_dependencies:
|
additional_dependencies:
|
||||||
@ -68,7 +68,7 @@ repos:
|
|||||||
- id: md-toc
|
- id: md-toc
|
||||||
|
|
||||||
- repo: https://github.com/python-poetry/poetry
|
- repo: https://github.com/python-poetry/poetry
|
||||||
rev: 1.5.0
|
rev: 1.6.0
|
||||||
hooks:
|
hooks:
|
||||||
- id: poetry-check
|
- id: poetry-check
|
||||||
|
|
||||||
@ -78,6 +78,6 @@ repos:
|
|||||||
- id: validate-html
|
- id: validate-html
|
||||||
|
|
||||||
- repo: https://github.com/python-jsonschema/check-jsonschema
|
- repo: https://github.com/python-jsonschema/check-jsonschema
|
||||||
rev: 0.24.1
|
rev: 0.26.3
|
||||||
hooks:
|
hooks:
|
||||||
- id: check-github-workflows
|
- id: check-github-workflows
|
||||||
|
114
CONTRIBUTING.md
114
CONTRIBUTING.md
@ -1,70 +1,72 @@
|
|||||||
# Contribution guidelines
|
# Contribution guidelines
|
||||||
|
|
||||||
|
## Dev Setup
|
||||||
|
|
||||||
|
- [Install Python 3.11](https://www.python.org/downloads/release/python-3111/)
|
||||||
|
- [Install Poetry](https://python-poetry.org/docs/#installation)
|
||||||
|
- [Install GiT](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)
|
||||||
|
- [Configure GiT](https://support.atlassian.com/bitbucket-cloud/docs/configure-your-dvcs-username-for-commits/)
|
||||||
|
- [Generate an SSH Key](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent) & [Add SSH Key to GitHub](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/adding-a-new-ssh-key-to-your-github-account)
|
||||||
|
- [Clone the Project (IDE Specific)](https://docs.github.com/de/repositories/creating-and-managing-repositories/cloning-a-repository)
|
||||||
|
- [Install the python Project](https://python-poetry.org/docs/basic-usage/#installing-dependencies)
|
||||||
|
- [Install pre-commit](https://pre-commit.com/#install)
|
||||||
|
|
||||||
|
## Repository structure
|
||||||
|
|
||||||
|
- **src/`aki_prj23_transparenzregister`**:
|
||||||
|
- This subfolder/`package` contains several subdirectories:
|
||||||
|
- `ai`:
|
||||||
|
- Houses AI models and pipelines, including NER and sentiment analysis.
|
||||||
|
- `config`:
|
||||||
|
- Contains configuration files such as database connection strings and model files.
|
||||||
|
- `models`:
|
||||||
|
- Stores data models.
|
||||||
|
- `ui`:
|
||||||
|
- Manages the user interface and dash interface.
|
||||||
|
- `utils`:
|
||||||
|
- Contains general tooling functions, including database access and other miscellaneous functions.
|
||||||
|
|
||||||
|
- **tests**:
|
||||||
|
- Test files organized in a mirrored structure of the 'src' folder. Please at least import every python file you
|
||||||
|
add.
|
||||||
|
Please Try to test every function with a test that compares with an example result. If that is not possible it
|
||||||
|
would be best to find a big consensus that only limited testing is required.
|
||||||
|
|
||||||
## Code style
|
## Code style
|
||||||
|
|
||||||
We defined to use the following formats:
|
We defined to use the following formats:
|
||||||
|
|
||||||
- Whitespace Formatting by [Black](https://github.com/psf/black)
|
- Whitespace Formatting by [Black](https://github.com/psf/black)
|
||||||
- Docstrings Style see the examples provided by [sphinx napoleon](https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html) Types in the docstrings are optional but should be consistent per module.
|
- Docstrings Style see the examples provided
|
||||||
|
by [sphinx napoleon](https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html) Types in the
|
||||||
|
docstrings are optional but should be consistent per module.
|
||||||
- Import Order by [isort](https://pycqa.github.io/isort/)
|
- Import Order by [isort](https://pycqa.github.io/isort/)
|
||||||
- Strict Typing of function headers see the examples provided by the [mypy](https://mypy.readthedocs.io/en/stable/cheat_sheet_py3.html) documentation.
|
- Strict Typing of function headers see the examples provided by
|
||||||
|
the [mypy](https://mypy.readthedocs.io/en/stable/cheat_sheet_py3.html) documentation.
|
||||||
|
- The generell linting is done by ruff but since ruff is a reimplementation of ruff of many different linters the old
|
||||||
|
documentations are still valid.
|
||||||
|
- Ruff can be executed with the `ruff .` command. May errors are auto-fixable with `ruff --fix .` if they are straight
|
||||||
|
forward changes.
|
||||||
|
|
||||||
## Language
|
## Language
|
||||||
|
|
||||||
We decided to use english on everything close to code but will write longer texts that are not close to the code in german.
|
We decided to use english on everything close to code but will write longer texts that are not close to the code in
|
||||||
|
german.
|
||||||
|
|
||||||
## Pre-Commit installation
|
## Pre-Commit Usage
|
||||||
|
|
||||||
- clone the project
|
- the following code format rules are so enforced:
|
||||||
- install the following python package:
|
- Whitespace formatting:
|
||||||
- `pre-commit`
|
- Python-[Black](https://github.com/psf/black)
|
||||||
|
- leading whitespace in all files
|
||||||
- execute the command `pre-commit install` to insert the git githook into your local repository.
|
- Auto format: yaml, toml, ini
|
||||||
- for more information read [Python Pre-Commit](https://pre-commit.com/)
|
|
||||||
- the following code format rules are so enforced:
|
|
||||||
- Whitespace formatierung:
|
|
||||||
- Python-[Black](https://github.com/psf/black)
|
|
||||||
- leading whitespace in all files
|
|
||||||
- Auto format: yaml, toml, ini
|
|
||||||
- Import norm by:
|
- Import norm by:
|
||||||
- [isort](https://pycqa.github.io/isort/)
|
- [isort](https://pycqa.github.io/isort/)
|
||||||
- absolut module path
|
- absolut module path
|
||||||
- Validierung:
|
- Validation:
|
||||||
- yaml, toml, json, xml
|
- yaml, toml, json, xml
|
||||||
- Secret detection
|
- Secret detection
|
||||||
- python normen ([pep8](https://peps.python.org/pep-0008/)) with [flake8](https://flake8.pycqa.org/en/latest/)
|
- python norms ([pep8](https://peps.python.org/pep-0008/))
|
||||||
- type checking with ([mypy](https://github.com/python/mypy))
|
with [ruff](https://github.com/astral-sh/ruff)
|
||||||
|
- type checking with ([mypy](https://github.com/python/mypy))
|
||||||
- Install [Python 3.11](https://www.python.org/downloads/release/python-3111/)
|
- for more information read [Python Pre-Commit](https://pre-commit.com/)
|
||||||
- Install Poetry
|
|
||||||
```
|
|
||||||
pip install poetry
|
|
||||||
```
|
|
||||||
- Install dependencies
|
|
||||||
```
|
|
||||||
poetry install
|
|
||||||
```
|
|
||||||
|
|
||||||
## Setup
|
|
||||||
|
|
||||||
### Connection strings
|
|
||||||
Create a `secrets.json` in the root of this repo with the following structure (values to be replaces by desired config):
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"postgres": {
|
|
||||||
"username": "postgres",
|
|
||||||
"password": "postgres",
|
|
||||||
"host": "localhost",
|
|
||||||
"database": "postgres",
|
|
||||||
"port": 5432
|
|
||||||
},
|
|
||||||
"mongo": {
|
|
||||||
"username": "username",
|
|
||||||
"password": "password",
|
|
||||||
"host": "localhost",
|
|
||||||
"database": "transparenzregister",
|
|
||||||
"port": 27017
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
Example usage see [connector.py](./src/aki_prj23_transparenzregister/utils/postgres/connector.py)
|
|
||||||
|
34
README.md
34
README.md
@ -1,13 +1,39 @@
|
|||||||
# aki_prj23_transparenzregister
|
# aki_prj23_transparenzregister
|
||||||
|
|
||||||
[](https://github.com/astral-sh/ruff)
|
[](https://www.python.org)
|
||||||
[](https://github.com/astral-sh/ruff/actions)
|
[](https://github.com/astral-sh/ruff/actions)
|
||||||
[](https://github.com/fhswf/aki_prj23_transparenzregister/actions/workflows/test-action.yaml)
|
[](https://github.com/astral-sh/ruff)
|
||||||
[](https://github.com/fhswf/aki_prj23_transparenzregister/actions/workflows/lint-actions.yaml)
|
[](https://github.com/pre-commit/pre-commit)
|
||||||
|
[](http://mypy-lang.org/)
|
||||||
|
[](https://mypy.readthedocs.io/en/stable/?badge=stable)
|
||||||
[](https://github.com/psf/black)
|
[](https://github.com/psf/black)
|
||||||
|
|
||||||
## Contributions
|
## Contributions
|
||||||
|
|
||||||
See the [CONTRIBUTING.md](CONTRIBUTING.md) about how code should be formatted and what kind of rules we set ourselves.
|
See the [CONTRIBUTING.md](CONTRIBUTING.md) about how code should be formatted and what kind of rules we set ourselves.
|
||||||
|
|
||||||
[](https://github.com/fhswf/aki_prj23_transparenzregister/actions/workflows/bandit-action.yaml)
|
## DB Connection settings
|
||||||
|
|
||||||
|
To connect to the SQL db see [sql/connector.py](./src/aki_prj23_transparenzregister/utils/postgres/connector.py)
|
||||||
|
To connect to the Mongo db see [connect]
|
||||||
|
|
||||||
|
Create a `secrets.json` in the root of this repo with the following structure (values to be replaces by desired config):
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"postgres": {
|
||||||
|
"username": "postgres",
|
||||||
|
"password": "postgres",
|
||||||
|
"host": "localhost",
|
||||||
|
"database": "postgres",
|
||||||
|
"port": 5432
|
||||||
|
},
|
||||||
|
"mongo": {
|
||||||
|
"username": "username",
|
||||||
|
"password": "password",
|
||||||
|
"host": "localhost",
|
||||||
|
"database": "transparenzregister",
|
||||||
|
"port": 27017
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
@ -4,6 +4,11 @@ Transparenzregister Dokumentation
|
|||||||
=================================
|
=================================
|
||||||
This is the documentation for the AKI project group on the german transparenzregister and an Analysis there of.
|
This is the documentation for the AKI project group on the german transparenzregister and an Analysis there of.
|
||||||
|
|
||||||
|
.. include:: ../README.md
|
||||||
|
:parser: myst_parser.sphinx_
|
||||||
|
.. include:: ../CONTRIBUTING.md
|
||||||
|
:parser: myst_parser.sphinx_
|
||||||
|
|
||||||
.. toctree::
|
.. toctree::
|
||||||
:maxdepth: 3
|
:maxdepth: 3
|
||||||
:caption: Project Management
|
:caption: Project Management
|
||||||
|
@ -1,6 +1,7 @@
|
|||||||
# Weekly *5*: 09.06.2023
|
# Weekly *6*: 09.06.2023
|
||||||
|
|
||||||
## Teilnehmer
|
## Teilnehmer
|
||||||
|
|
||||||
- Prof. Arinir
|
- Prof. Arinir
|
||||||
- Tristan Nolde
|
- Tristan Nolde
|
||||||
- Tim Ronneburg
|
- Tim Ronneburg
|
||||||
@ -12,79 +13,86 @@
|
|||||||
## Themen
|
## Themen
|
||||||
|
|
||||||
- Stepstone Projekt:
|
- Stepstone Projekt:
|
||||||
- Gewünscht wird ein initialer Austausch mit Stepstone
|
- Gewünscht wird ein initialer Austausch mit Stepstone
|
||||||
- Befürchtung ist, dass es zu einem Hinderniss wird
|
- Befürchtung ist, dass es zu einem Hinderniss wird
|
||||||
- Entscheidung liegt daher beim Projekt-Team
|
- Entscheidung liegt daher beim Projekt-Team
|
||||||
- Weitere Informationen sind nicht aktuell nicht vorhanden
|
- Weitere Informationen sind nicht aktuell nicht vorhanden
|
||||||
- Vorschlag Prof. Arinir: Sollte das Thema nochmal zum Team getragen werden, wird der aktuelle Stand vorgestellt und der Link zum Repo wird geteilt. Darüber hinaus werden keine Ressourcen zugesprochen.
|
- Vorschlag Prof. Arinir: Sollte das Thema nochmal zum Team getragen werden, wird der aktuelle Stand vorgestellt und
|
||||||
- Vorstellung [vorheriger Absprache](https://github.com/orgs/fhswf/projects/17?pane=issue&itemId=29707639) und Feedback:
|
der Link zum Repo wird geteilt. Darüber hinaus werden keine Ressourcen zugesprochen.
|
||||||
- Ändert sich der Scope - Nein
|
- Vorstellung [vorheriger Absprache](https://github.com/orgs/fhswf/projects/17?pane=issue&itemId=29707639) und
|
||||||
- NDA - Nein
|
Feedback:
|
||||||
- Veröffentlichung - maximal Impressionen
|
- Ändert sich der Scope - Nein
|
||||||
- Was muss geleistet werden - nicht direkt an Stepstone sondern über FH als Mediator
|
- NDA - Nein
|
||||||
|
- Veröffentlichung - maximal Impressionen
|
||||||
|
- Was muss geleistet werden - nicht direkt an Stepstone sondern über FH als Mediator
|
||||||
- Sollen Präsentationen vorab zur Verfügung gestellt werden?
|
- Sollen Präsentationen vorab zur Verfügung gestellt werden?
|
||||||
- Einige Tage vorher in das Git Repo. hochladen und Prof. Arinir benachrichtigen
|
- Einige Tage vorher in das Git Repo. hochladen und Prof. Arinir benachrichtigen
|
||||||
- Rücksprache Seminarpräsentationen
|
- Rücksprache Seminarpräsentationen
|
||||||
- Verflechtungsanalyse:
|
- Verflechtungsanalyse:
|
||||||
- Graphen Theorie
|
- Graphen Theorie
|
||||||
- Social Network Analyse
|
- Social Network Analyse
|
||||||
- Erweiterung über Graphen Theorie hinaus
|
- Erweiterung über Graphen Theorie hinaus
|
||||||
- Fokus auf Anwendung und Mehrwert, weniger genauer mathematischer Lösung
|
- Fokus auf Anwendung und Mehrwert, weniger genauer mathematischer Lösung
|
||||||
- Feedback:
|
- Feedback:
|
||||||
- Präsentation scheint sehr umfangreich; Wunsch nach Reduzierung der Folienanzahl
|
- Präsentation scheint sehr umfangreich; Wunsch nach Reduzierung der Folienanzahl
|
||||||
- Formeln hinter den Analysen spannend, ggf. doch drauf eingehen, um Kennzahl in Kontext zu setzen
|
- Formeln hinter den Analysen spannend, ggf. doch drauf eingehen, um Kennzahl in Kontext zu setzen
|
||||||
- Visualiserung:
|
- Visualiserung:
|
||||||
- Prinzipien
|
- Prinzipien
|
||||||
- Vorteile
|
- Vorteile
|
||||||
- Bibliotheken für Umsetzung (Network X, PyViz, ...)
|
- Bibliotheken für Umsetzung (Network X, PyViz, ...)
|
||||||
- Effekt von Farbwahl
|
- Effekt von Farbwahl
|
||||||
- Erste Umsetzung im Jupyter Notebook
|
- Erste Umsetzung im Jupyter Notebook
|
||||||
- Feedback:
|
- Feedback:
|
||||||
- Es werden extem viele Datenpunkte angezeigt werden müssen, wie wird dies in den Bibliotheken umgesetzt? Kann dort gefiltert werden?
|
- Es werden extem viele Datenpunkte angezeigt werden müssen, wie wird dies in den Bibliotheken umgesetzt?
|
||||||
- Wenn nicht direkt am Graphen (der Darstellung) gefiltert werden kann, dann frühzeitig filtern, bevor der Graph gebaut wird
|
Kann dort gefiltert werden?
|
||||||
- Datenspeicherung
|
- Wenn nicht direkt am Graphen (der Darstellung) gefiltert werden kann, dann frühzeitig filtern, bevor
|
||||||
- Erste Integration von Visualisierung mit Datenspeicherung
|
der Graph gebaut wird
|
||||||
- Vorstellung der "Datencluster"
|
- Datenspeicherung
|
||||||
- Stammdaten
|
- Erste Integration von Visualisierung mit Datenspeicherung
|
||||||
- Stimmungsdaten
|
- Vorstellung der "Datencluster"
|
||||||
- Social Graph
|
- Stammdaten
|
||||||
- Zeitseriendaten
|
- Stimmungsdaten
|
||||||
- Relationales DB Modell
|
- Social Graph
|
||||||
- Fokus ebenfalls auf Abfrage der Daten für Folge-Projekte wie Visualiserung und Mehrwert fürs Team, weniger Theorie
|
- Zeitseriendaten
|
||||||
- Feedback:
|
- Relationales DB Modell
|
||||||
- Es müssen Erfahrungen mit der Library und Darstellung gesammelt werden, um den Mehrwert der Lösung hervorzuheben
|
- Fokus ebenfalls auf Abfrage der Daten für Folge-Projekte wie Visualiserung und Mehrwert fürs Team, weniger
|
||||||
- Modellierung der Finzanz-Kennzahlen
|
Theorie
|
||||||
- Spaltennamen sollen sprechend sein, z.B. "value" statt "sum"
|
- Feedback:
|
||||||
- Präferenz zum Modell mit einzelnem Eintrag mit mehren Kennzahl Spalten stallt generischer Lösung über Enum
|
- Es müssen Erfahrungen mit der Library und Darstellung gesammelt werden, um den Mehrwert der Lösung
|
||||||
- Text Mining
|
hervorzuheben
|
||||||
- Fokus auf Sentiment Analyse
|
- Modellierung der Finzanz-Kennzahlen
|
||||||
- Vergleich verschiedener Lösungen und ML Modelle
|
- Spaltennamen sollen sprechend sein, z.B. "value" statt "sum"
|
||||||
- Abschließendes Fazit, welches Tool am besten geeignet ist
|
- Präferenz zum Modell mit einzelnem Eintrag mit mehren Kennzahl Spalten stallt generischer Lösung über
|
||||||
- Daten Extraktion
|
Enum
|
||||||
- Fokus auf Web Mining/Scraping im Rahmen des Transparenzregisters
|
- Text Mining
|
||||||
- Datenquellen
|
- Fokus auf Sentiment Analyse
|
||||||
- API
|
- Vergleich verschiedener Lösungen und ML Modelle
|
||||||
- Websites (HTML)
|
- Abschließendes Fazit, welches Tool am besten geeignet ist
|
||||||
- PDF
|
- Daten Extraktion
|
||||||
- Datenextraktion aus diesen Quellen
|
- Fokus auf Web Mining/Scraping im Rahmen des Transparenzregisters
|
||||||
- Orchestrierung mit Airflow
|
- Datenquellen
|
||||||
- DevOps
|
- API
|
||||||
- Dependency Management in Python
|
- Websites (HTML)
|
||||||
- Standard requirements.txt
|
- PDF
|
||||||
- pip-tools
|
- Datenextraktion aus diesen Quellen
|
||||||
- poetry
|
- Orchestrierung mit Airflow
|
||||||
- Vorteile von Lintern
|
- DevOps
|
||||||
- GitHub
|
- Dependency Management in Python
|
||||||
- Actions
|
- Standard requirements.txt
|
||||||
- Security
|
- pip-tools
|
||||||
- etc.
|
- poetry
|
||||||
- Feedback:
|
- Vorteile von Lintern
|
||||||
- Git wird als State-of-the-Art angesehen und muss nicht näher erläutert werden
|
- GitHub
|
||||||
|
- Actions
|
||||||
|
- Security
|
||||||
|
- etc.
|
||||||
|
- Feedback:
|
||||||
|
- Git wird als State-of-the-Art angesehen und muss nicht näher erläutert werden
|
||||||
|
|
||||||
## Abgeleitete Action Items
|
## Abgeleitete Action Items
|
||||||
|
|
||||||
| Action Item | Verantwortlicher | Deadline |
|
| Action Item | Verantwortlicher | Deadline |
|
||||||
|-------------|------------------|-----------------|
|
|------------------------------------------------------------------|------------------|-------------------------|
|
||||||
| Folien hochladen | Projekt Team | vor Präsentationstermin |
|
| Folien hochladen | Projekt Team | vor Präsentationstermin |
|
||||||
| Absprache Abgrenzung von Verflechtungsanalyse und Visualisierung | Tim und Kim | nächster Abgleich |
|
| Absprache Abgrenzung von Verflechtungsanalyse und Visualisierung | Tim und Kim | nächster Abgleich |
|
||||||
| Deployment Plan aufstellen | Projekt Team | nach Seminararbeiten |
|
| Deployment Plan aufstellen | Projekt Team | nach Seminararbeiten |
|
||||||
|
@ -1,6 +1,7 @@
|
|||||||
# Weekly *X*: 03.08.2023
|
# Weekly *7*: 03.08.2023
|
||||||
|
|
||||||
## Teilnehmer
|
## Teilnehmer
|
||||||
|
|
||||||
- Prof. Arinir
|
- Prof. Arinir
|
||||||
- Tristan Nolde
|
- Tristan Nolde
|
||||||
- Tim Ronneburg (Protokollant)
|
- Tim Ronneburg (Protokollant)
|
||||||
@ -9,19 +10,19 @@
|
|||||||
## Themen
|
## Themen
|
||||||
|
|
||||||
- Präsentieren der Ergebnisse der letzten Wochen:
|
- Präsentieren der Ergebnisse der letzten Wochen:
|
||||||
- Named Entity Recognition
|
- Named Entity Recognition
|
||||||
- Vorstellung Datenbank auf dem FH-Cluster:
|
- Vorstellung Datenbank auf dem FH-Cluster:
|
||||||
- Mongo Connector
|
- Mongo Connector
|
||||||
- Datenspeicherung auf dem Cluster
|
- Datenspeicherung auf dem Cluster
|
||||||
- Weitere Vorgehensweise:
|
- Weitere Vorgehensweise:
|
||||||
- Idee: Kleine Workshops/Teams
|
- Idee: Kleine Workshops/Teams
|
||||||
- In 2er Teams die einzelnen Funktionen über Feature Branches erstellen
|
- In 2er Teams die einzelnen Funktionen über Feature Branches erstellen
|
||||||
|
|
||||||
## Abgeleitete Action Items
|
## Abgeleitete Action Items
|
||||||
|
|
||||||
| Action Item | Verantwortlicher | Deadline |
|
| Action Item | Verantwortlicher | Deadline |
|
||||||
|-------------|------------------|-----------------|
|
|--------------------------------------------------------------------|----------------------------|-----------------|
|
||||||
| Mergen aller Branches zu jedem neuen Termin mit Herrn Arinir | Jeder | jedes Weekly |
|
| Mergen aller Branches zu jedem neuen Termin mit Herrn Arinir | Jeder | jedes Weekly |
|
||||||
| Erstellen der Pipelines | Sebastian, Tristan und Tim | nächstes Weekly |
|
| Erstellen der Pipelines | Sebastian, Tristan und Tim | nächstes Weekly |
|
||||||
| Erstellen der Development Datenbank-Instanzen je Entwickler | Sebastian, Tristan und Tim | nächstes Weekly |
|
| Erstellen der Development Datenbank-Instanzen je Entwickler | Sebastian, Tristan und Tim | nächstes Weekly |
|
||||||
| Anlegen der relationalen Postgres DB via Script auf den FH-Cluster | Sebastian, Tristan und Tim | nächstes Weekly |
|
| Anlegen der relationalen Postgres DB via Script auf den FH-Cluster | Sebastian, Tristan und Tim | nächstes Weekly |
|
||||||
|
44
documentations/meeting-notes/Meeting_2023-08-17.md
Normal file
44
documentations/meeting-notes/Meeting_2023-08-17.md
Normal file
@ -0,0 +1,44 @@
|
|||||||
|
# Weekly *8*: 17.08.2023
|
||||||
|
|
||||||
|
## Teilnehmer
|
||||||
|
|
||||||
|
- Prof. Arinir
|
||||||
|
- Tristan Nolde
|
||||||
|
- Tim Ronneburg
|
||||||
|
- Kim Mesewinkel-Risse
|
||||||
|
- Phillip Horstenkamp
|
||||||
|
- Sebastian Zeleny (Protokollant)
|
||||||
|
|
||||||
|
## Themen
|
||||||
|
|
||||||
|
- Welche Services laufen aktuell auf dem Uni-Cluster?
|
||||||
|
- MongoDB und Postgres mit personalisiertem Zugang über VPN
|
||||||
|
- **Requirement:** Das Frontend bzw. Visualisierung soll auf dem Cluster laufen, wofür ein Login erwünscht ist, wie
|
||||||
|
z.B. SSO
|
||||||
|
- die Services (Text Mining, NER, Sentiment) sollten auch dem Cluster laufen
|
||||||
|
- Wo sollen CI/CD laufen?
|
||||||
|
- benötigt werden 2-3 Container für Worker und Services
|
||||||
|
- ProductionDB:
|
||||||
|
- DB "transparenzregister" für Produktiveinsatz
|
||||||
|
- persönliche DB "DB_*Name*" für Development
|
||||||
|
- Erklärung des Postgres Connectors, welcher die SQL-Alchemy Klassen verwendet, um Tabellen zu erstellen
|
||||||
|
- Erklärung wie der Connection-String mit dem JsonConfigProvider und der secret.json erzeugt wird
|
||||||
|
- UI:
|
||||||
|
- Vorstellung der ersten Visualisierung mit Plotly und Anbindung an ProductionDB
|
||||||
|
- Dash startet im Hintergrund einen http-Server, welcher über den Port 8050 erreichbar ist
|
||||||
|
- Dash wird für das Dashboarding verwendet und wird (bei Bedarf) durch weitere Komponenten erweitert
|
||||||
|
- Abschluß des PoC und Umsetzung der bestehenden Architektur
|
||||||
|
|
||||||
|
## Abgeleitete Action Items
|
||||||
|
|
||||||
|
| Action Item | Verantwortlicher | Deadline |
|
||||||
|
|------------------------------------------------------------------|------------------|-----------------|
|
||||||
|
| Anfrage nach Serverressourcen für 2-3 Container bei Prof. Gawron | Prof. Arinir | nächstes Weekly |
|
||||||
|
| Repo-Struktur dokumentieren | Phillip, Tristan | nächstes Weekly |
|
||||||
|
| Anlegen von MongoDB-Instanzen für NER und Sentiment | Tristan | nächstes Weekly |
|
||||||
|
| NER für News | Sebastian | nächstes Weekly |
|
||||||
|
| Beispiel für MongoConnector | Sebastian | nächstes Weekly |
|
||||||
|
| Script um News auf Uni-Cluster zu dumpen | Tristan | nächstes Weekly |
|
||||||
|
| Finanzdaten aus Bundesanzeiger | Tristan | nächstes Weekly |
|
||||||
|
| Plotly Frontend weiterentwickeln | Kim, Tim | nächstes Weekly |
|
||||||
|
| Refactoring der SQL-Alchemy Klassen | Phillip | nächstes Weekly |
|
40
documentations/meeting-notes/Meeting_2023-08-31.md
Normal file
40
documentations/meeting-notes/Meeting_2023-08-31.md
Normal file
@ -0,0 +1,40 @@
|
|||||||
|
# Weekly *9*: 31.08.2023
|
||||||
|
|
||||||
|
## Teilnehmer
|
||||||
|
|
||||||
|
- Prof. Arinir
|
||||||
|
- Tristan Nolde
|
||||||
|
- Philipp Horstenkamp
|
||||||
|
- Sebastian Zeleny
|
||||||
|
- Kim Mesewinkel-Risse (Protokoll)
|
||||||
|
|
||||||
|
## Themen
|
||||||
|
|
||||||
|
- Rückmeldung von Herrn Gawron bzgl. mehr Ressourcen steht noch aus, ggfs. persönliche Absprache nächste Woche möglich
|
||||||
|
|
||||||
|
- Rückfrage von Herrn Arinir bezüglich Aufbau der Software und Architektur
|
||||||
|
- Gerade werden einzelne Funktionen erstellt, Daten werden ungefiltert in die Mongo DB geschrieben, anschließend
|
||||||
|
Bereinigung und Übertragung in die Postgres
|
||||||
|
- Vorstellung aktueller Repo-Struktur durch Tristan, relevanter Code befindet sich im src-Ordner
|
||||||
|
|
||||||
|
- Wie kann sichergestellt werden, dass unsere Ziele erreicht werden?
|
||||||
|
- Zeitplan/Meilensteinplan gewünscht
|
||||||
|
- Wann soll was erreicht werden?
|
||||||
|
- Burndown-Diagramm
|
||||||
|
-> Umsetzung durch Team beim Präsenzmeeting am 09.09.2023
|
||||||
|
|
||||||
|
- Kurze Vorstellung der bearbeiteten Themen: NER + Sentiment (Sebastian), Finanzdaten (Tristan), UI (Kim),
|
||||||
|
Datentransfer (Philipp)
|
||||||
|
|
||||||
|
## Abgeleitete Action Items
|
||||||
|
|
||||||
|
| Action Item | Verantwortlicher | Deadline |
|
||||||
|
|------------------------------------------------------|------------------|-----------------|
|
||||||
|
| Festlegung Zeitplan für Präsenztreffen | Alle | 07.09.2023 |
|
||||||
|
| Zeitplan bzw. Meilensteinplan | Alle | nächstes Weekly |
|
||||||
|
| Erstellen einer Übersicht aller bestehenden Services | Alle | nächstes Weekly |
|
||||||
|
| Update bzgl. Cluster und Ressourcen | Herr Arinir | nächstes Weekly |
|
||||||
|
| Finanzdaten finalisieren | Tristan | nächstes Weekly |
|
||||||
|
| NER/Sentiment Aggregation | Sebastian | nächstes Weekly |
|
||||||
|
| Füllen der UI mit Echtdaten | Kim | nächstes Weekly |
|
||||||
|
| Teststruktur für SQL hinzufügen | Philipp | nächstes Weekly |
|
1059
poetry.lock
generated
1059
poetry.lock
generated
File diff suppressed because it is too large
Load Diff
@ -37,6 +37,7 @@ version = "0.1.0"
|
|||||||
[tool.poetry.dependencies]
|
[tool.poetry.dependencies]
|
||||||
SQLAlchemy = {version = "^1.4.46", extras = ["mypy"]}
|
SQLAlchemy = {version = "^1.4.46", extras = ["mypy"]}
|
||||||
dash = "^2.11.1"
|
dash = "^2.11.1"
|
||||||
|
dash-bootstrap-components = "^1.4.2"
|
||||||
deutschland = {git = "https://github.com/TrisNol/deutschland.git", branch = "hotfix/python-3.11-support"}
|
deutschland = {git = "https://github.com/TrisNol/deutschland.git", branch = "hotfix/python-3.11-support"}
|
||||||
loguru = "^0.7.0"
|
loguru = "^0.7.0"
|
||||||
matplotlib = "^3.7.1"
|
matplotlib = "^3.7.1"
|
||||||
@ -89,7 +90,7 @@ pytest-cov = "^4.1.0"
|
|||||||
pytest-mock = "^3.10.0"
|
pytest-mock = "^3.10.0"
|
||||||
pytest-repeat = "^0.9.1"
|
pytest-repeat = "^0.9.1"
|
||||||
|
|
||||||
# TODO Add enrich_company_financials hinzufügen
|
# TODO Add enrich_company_financials hinzufügen
|
||||||
[tool.poetry.scripts]
|
[tool.poetry.scripts]
|
||||||
mein_test = "aki_prj23_transparenzregister.utils.postgres.connector:init_db"
|
mein_test = "aki_prj23_transparenzregister.utils.postgres.connector:init_db"
|
||||||
|
|
||||||
|
1
src/aki_prj23_transparenzregister/ai/__init__.py
Normal file
1
src/aki_prj23_transparenzregister/ai/__init__.py
Normal file
@ -0,0 +1 @@
|
|||||||
|
"""This module contains all the ai pipelines."""
|
@ -1 +1 @@
|
|||||||
"""App configuration."""
|
"""App configuration tools."""
|
||||||
|
@ -0,0 +1,3 @@
|
|||||||
|
body {
|
||||||
|
margin: 0;
|
||||||
|
}
|
392
src/aki_prj23_transparenzregister/ui/company_finance_dash.py
Normal file
392
src/aki_prj23_transparenzregister/ui/company_finance_dash.py
Normal file
@ -0,0 +1,392 @@
|
|||||||
|
"""Dash."""
|
||||||
|
|
||||||
|
import dash_bootstrap_components as dbc
|
||||||
|
import pandas as pd
|
||||||
|
import plotly.graph_objs as go
|
||||||
|
from dash import Dash, Input, Output, callback, dash_table, dcc, html
|
||||||
|
from dash.exceptions import PreventUpdate
|
||||||
|
from sqlalchemy.engine import Engine
|
||||||
|
|
||||||
|
from aki_prj23_transparenzregister.utils.postgres import entities
|
||||||
|
from aki_prj23_transparenzregister.utils.postgres.connector import (
|
||||||
|
get_session,
|
||||||
|
)
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
session = get_session()
|
||||||
|
query_finance = session.query(
|
||||||
|
entities.AnnualFinanceStatement, entities.Company.name, entities.Company.id
|
||||||
|
).join(entities.Company)
|
||||||
|
|
||||||
|
query_company = session.query(entities.Company, entities.DistrictCourt.name).join(
|
||||||
|
entities.DistrictCourt
|
||||||
|
)
|
||||||
|
engine = session.bind
|
||||||
|
if not isinstance(engine, Engine):
|
||||||
|
raise TypeError
|
||||||
|
|
||||||
|
finance_df: pd.DataFrame = pd.read_sql(str(query_finance), engine)
|
||||||
|
company_df: pd.DataFrame = pd.read_sql(str(query_company), engine)
|
||||||
|
|
||||||
|
select_company_df = company_df[["company_id", "company_name"]]
|
||||||
|
select_company_dropdown = select_company_df.to_dict("records")
|
||||||
|
options = [
|
||||||
|
{"label": i["company_name"], "value": i["company_id"]}
|
||||||
|
for i in select_company_dropdown
|
||||||
|
]
|
||||||
|
|
||||||
|
colors = {
|
||||||
|
"light": "#edefef",
|
||||||
|
"lavender-blush": "#f3e8ee",
|
||||||
|
"ash-gray": "#bacdb0",
|
||||||
|
"cambridge-blue": "#729b79",
|
||||||
|
"paynes-gray": "#475b63",
|
||||||
|
"raisin-black": "#2e2c2f",
|
||||||
|
}
|
||||||
|
|
||||||
|
def financials_figure(
|
||||||
|
finance_df: pd.DataFrame, company: str, metric: str
|
||||||
|
) -> go.Figure:
|
||||||
|
"""Creates plotly line chart for a specific company and a metric."""
|
||||||
|
finance_df = finance_df.loc[finance_df["company_name"] == company]
|
||||||
|
# create figure
|
||||||
|
fig_line = go.Figure()
|
||||||
|
# add trace for company 1
|
||||||
|
fig_line.add_trace(
|
||||||
|
go.Scatter(
|
||||||
|
x=finance_df["annual_finance_statement_date"],
|
||||||
|
y=finance_df[metric],
|
||||||
|
name=company,
|
||||||
|
line_color=colors["raisin-black"],
|
||||||
|
marker_color=colors["raisin-black"],
|
||||||
|
)
|
||||||
|
)
|
||||||
|
# set title and labels
|
||||||
|
fig_line.update_layout(
|
||||||
|
title=metric,
|
||||||
|
xaxis_title="Jahr",
|
||||||
|
yaxis_title="in Mio.€",
|
||||||
|
plot_bgcolor=colors["light"],
|
||||||
|
)
|
||||||
|
return fig_line
|
||||||
|
|
||||||
|
tab_style = {
|
||||||
|
"borderBottom": "1px solid #d6d6d6",
|
||||||
|
"padding": "6px",
|
||||||
|
"backgroundColor": "white",
|
||||||
|
"color": colors["paynes-gray"],
|
||||||
|
"fontWeight": "bold",
|
||||||
|
}
|
||||||
|
|
||||||
|
tab_selected_style = {
|
||||||
|
"borderTop": "1px solid #d6d6d6",
|
||||||
|
"borderBottom": "1px solid #d6d6d6",
|
||||||
|
"padding": "6px",
|
||||||
|
"backgroundColor": colors["paynes-gray"],
|
||||||
|
"color": "white",
|
||||||
|
"fontWeight": "bold",
|
||||||
|
}
|
||||||
|
|
||||||
|
# TBD: get data from database instead of mock data
|
||||||
|
company = 1 # selected company id
|
||||||
|
selected_company = company_df.loc[company_df["company_id"] == company]
|
||||||
|
|
||||||
|
turnover = 123456
|
||||||
|
stock = "1,23"
|
||||||
|
company_data = {
|
||||||
|
"col1": ["Unternehmen", "Straße", "Stadt"],
|
||||||
|
"col2": [
|
||||||
|
selected_company["company_name"][0],
|
||||||
|
selected_company["company_street"][0],
|
||||||
|
str(
|
||||||
|
selected_company["company_zip_code"][0]
|
||||||
|
+ " "
|
||||||
|
+ selected_company["company_city"][0]
|
||||||
|
),
|
||||||
|
],
|
||||||
|
"col3": ["Branche", "Amtsgericht", "Gründungsjahr"],
|
||||||
|
"col4": [
|
||||||
|
selected_company["company_sector"][0],
|
||||||
|
selected_company["district_court_name"][0],
|
||||||
|
"xxx",
|
||||||
|
],
|
||||||
|
}
|
||||||
|
df_company_data = pd.DataFrame(data=company_data)
|
||||||
|
|
||||||
|
app = Dash(
|
||||||
|
__name__, external_stylesheets=[dbc.icons.BOOTSTRAP]
|
||||||
|
) # use dbc for icons
|
||||||
|
|
||||||
|
kennzahlen_layout = html.Div(
|
||||||
|
[
|
||||||
|
dcc.Graph(
|
||||||
|
figure=financials_figure(
|
||||||
|
finance_df, str(company), "annual_finance_statement_ebit"
|
||||||
|
)
|
||||||
|
)
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
app.layout = html.Div(
|
||||||
|
[
|
||||||
|
# title header of page
|
||||||
|
html.Div(
|
||||||
|
style={
|
||||||
|
"backgroundColor": colors["raisin-black"],
|
||||||
|
"border": "1px solid",
|
||||||
|
},
|
||||||
|
children=[
|
||||||
|
html.I(
|
||||||
|
className="bi bi-house-door-fill",
|
||||||
|
style={
|
||||||
|
"fontSize": 24,
|
||||||
|
"paddingLeft": "10px",
|
||||||
|
"color": "white",
|
||||||
|
"display": "inline-block",
|
||||||
|
"verticalAlign": "middle",
|
||||||
|
},
|
||||||
|
),
|
||||||
|
html.H1(
|
||||||
|
children="Transparenzregister für Kapitalgesellschaften",
|
||||||
|
style={
|
||||||
|
"color": "white",
|
||||||
|
"textAlign": "left",
|
||||||
|
"margin": "0",
|
||||||
|
"paddingLeft": "10px",
|
||||||
|
"paddingBottom": "20px",
|
||||||
|
"paddingTop": "20px",
|
||||||
|
"display": "inline-block",
|
||||||
|
"verticalAlign": "middle",
|
||||||
|
},
|
||||||
|
),
|
||||||
|
html.Div(
|
||||||
|
dcc.Dropdown(
|
||||||
|
id="select_company",
|
||||||
|
placeholder="Suche nach Unternehmen oder Person",
|
||||||
|
),
|
||||||
|
style={
|
||||||
|
"float": "right",
|
||||||
|
"width": "30%",
|
||||||
|
"margin": "0",
|
||||||
|
"paddingRight": "10px",
|
||||||
|
"paddingBottom": "20px",
|
||||||
|
"paddingTop": "20px",
|
||||||
|
"display": "inline-block",
|
||||||
|
"verticalAlign": "middle",
|
||||||
|
},
|
||||||
|
),
|
||||||
|
],
|
||||||
|
),
|
||||||
|
# header company name
|
||||||
|
html.Div(
|
||||||
|
style={"backgroundColor": colors["paynes-gray"], "border": "1px solid"},
|
||||||
|
children=[
|
||||||
|
html.H1(
|
||||||
|
children=selected_company["company_name"][0],
|
||||||
|
style={
|
||||||
|
"color": "white",
|
||||||
|
"fontSize": 30,
|
||||||
|
"textAlign": "left",
|
||||||
|
"margin": "0",
|
||||||
|
"paddingLeft": "20px",
|
||||||
|
"paddingBottom": "20px",
|
||||||
|
"paddingTop": "20px",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
],
|
||||||
|
),
|
||||||
|
html.Div(style={"height": "20px"}),
|
||||||
|
html.Div(style={"width": "2%", "display": "inline-block"}),
|
||||||
|
# table basic company information
|
||||||
|
html.Div(
|
||||||
|
style={
|
||||||
|
"backgroundColor": colors["ash-gray"],
|
||||||
|
"border": "1px solid",
|
||||||
|
"border-radius": 10,
|
||||||
|
"width": "45%",
|
||||||
|
"height": "150px",
|
||||||
|
"display": "inline-block",
|
||||||
|
"vertical-align": "top",
|
||||||
|
},
|
||||||
|
children=[
|
||||||
|
html.H5(
|
||||||
|
children="Stammdaten",
|
||||||
|
style={
|
||||||
|
"color": colors["raisin-black"],
|
||||||
|
"fontSize": 16,
|
||||||
|
"textAlign": "center",
|
||||||
|
"margin": "0",
|
||||||
|
"paddingBottom": "10px",
|
||||||
|
"paddingTop": "10px",
|
||||||
|
},
|
||||||
|
),
|
||||||
|
dash_table.DataTable(
|
||||||
|
df_company_data.to_dict("records"),
|
||||||
|
[{"name": i, "id": i} for i in df_company_data.columns],
|
||||||
|
style_table={
|
||||||
|
"width": "80%",
|
||||||
|
"overflowX": "auto",
|
||||||
|
"marginLeft": "auto",
|
||||||
|
"marginRight": "auto",
|
||||||
|
"paddingBottom": "20px",
|
||||||
|
"color": colors["raisin-black"],
|
||||||
|
},
|
||||||
|
# hide header of table
|
||||||
|
css=[
|
||||||
|
{
|
||||||
|
"selector": "tr:first-child",
|
||||||
|
"rule": "display: none",
|
||||||
|
},
|
||||||
|
],
|
||||||
|
style_cell={"textAlign": "center"},
|
||||||
|
style_cell_conditional=[
|
||||||
|
{"if": {"column_id": c}, "fontWeight": "bold"}
|
||||||
|
for c in ["col1", "col3"]
|
||||||
|
],
|
||||||
|
),
|
||||||
|
],
|
||||||
|
),
|
||||||
|
html.Div(style={"width": "2%", "display": "inline-block"}),
|
||||||
|
html.Div(
|
||||||
|
style={
|
||||||
|
"backgroundColor": colors["ash-gray"],
|
||||||
|
"border": "1px solid",
|
||||||
|
"border-radius": 10,
|
||||||
|
"width": "15%",
|
||||||
|
"height": "150px",
|
||||||
|
"display": "inline-block",
|
||||||
|
"vertical-align": "top",
|
||||||
|
},
|
||||||
|
children=[
|
||||||
|
html.H5(
|
||||||
|
children="Stimmung",
|
||||||
|
style={
|
||||||
|
"color": colors["raisin-black"],
|
||||||
|
"fontSize": 16,
|
||||||
|
"textAlign": "center",
|
||||||
|
"margin": "0",
|
||||||
|
"paddingBottom": "10px",
|
||||||
|
"paddingTop": "10px",
|
||||||
|
},
|
||||||
|
),
|
||||||
|
],
|
||||||
|
),
|
||||||
|
html.Div(style={"width": "2%", "display": "inline-block"}),
|
||||||
|
html.Div(
|
||||||
|
style={
|
||||||
|
"backgroundColor": colors["ash-gray"],
|
||||||
|
"border": "1px solid",
|
||||||
|
"border-radius": 10,
|
||||||
|
"width": "15%",
|
||||||
|
"height": "150px",
|
||||||
|
"display": "inline-block",
|
||||||
|
"vertical-align": "top",
|
||||||
|
},
|
||||||
|
children=[
|
||||||
|
html.H5(
|
||||||
|
children="Aktienkurs",
|
||||||
|
style={
|
||||||
|
"color": colors["raisin-black"],
|
||||||
|
"fontSize": 16,
|
||||||
|
"textAlign": "center",
|
||||||
|
"margin": "0",
|
||||||
|
"paddingBottom": "10px",
|
||||||
|
"paddingTop": "10px",
|
||||||
|
},
|
||||||
|
),
|
||||||
|
html.H1(
|
||||||
|
children=stock,
|
||||||
|
style={
|
||||||
|
"color": colors["raisin-black"],
|
||||||
|
"textAlign": "center",
|
||||||
|
},
|
||||||
|
),
|
||||||
|
],
|
||||||
|
),
|
||||||
|
html.Div(style={"width": "2%", "display": "inline-block"}),
|
||||||
|
html.Div(
|
||||||
|
style={
|
||||||
|
"backgroundColor": colors["ash-gray"],
|
||||||
|
"border": "1px solid",
|
||||||
|
"border-radius": 10,
|
||||||
|
"width": "15%",
|
||||||
|
"height": "150px",
|
||||||
|
"display": "inline-block",
|
||||||
|
"vertical-align": "top",
|
||||||
|
},
|
||||||
|
children=[
|
||||||
|
html.H5(
|
||||||
|
children="Umsatz",
|
||||||
|
style={
|
||||||
|
"color": colors["raisin-black"],
|
||||||
|
"fontSize": 16,
|
||||||
|
"textAlign": "center",
|
||||||
|
"margin": "0",
|
||||||
|
"paddingBottom": "10px",
|
||||||
|
"paddingTop": "10px",
|
||||||
|
},
|
||||||
|
),
|
||||||
|
html.H1(
|
||||||
|
children=turnover,
|
||||||
|
style={
|
||||||
|
"color": colors["raisin-black"],
|
||||||
|
"textAlign": "center",
|
||||||
|
},
|
||||||
|
),
|
||||||
|
],
|
||||||
|
),
|
||||||
|
html.Div(style={"width": "2%", "display": "inline-block"}),
|
||||||
|
# ]),
|
||||||
|
html.Div(
|
||||||
|
style={
|
||||||
|
"marginTop": "20px",
|
||||||
|
"border": "1px solid",
|
||||||
|
},
|
||||||
|
children=[
|
||||||
|
dcc.Tabs(
|
||||||
|
id="tabs",
|
||||||
|
value="tab-1",
|
||||||
|
children=[
|
||||||
|
dcc.Tab(
|
||||||
|
label="Kennzahlen",
|
||||||
|
value="tab-1",
|
||||||
|
style=tab_style,
|
||||||
|
selected_style=tab_selected_style,
|
||||||
|
children=[kennzahlen_layout],
|
||||||
|
),
|
||||||
|
dcc.Tab(
|
||||||
|
label="Beteiligte Personen",
|
||||||
|
value="tab-2",
|
||||||
|
style=tab_style,
|
||||||
|
selected_style=tab_selected_style,
|
||||||
|
),
|
||||||
|
dcc.Tab(
|
||||||
|
label="Stimmung",
|
||||||
|
value="tab-3",
|
||||||
|
style=tab_style,
|
||||||
|
selected_style=tab_selected_style,
|
||||||
|
),
|
||||||
|
dcc.Tab(
|
||||||
|
label="Verflechtungen",
|
||||||
|
value="tab-4",
|
||||||
|
style=tab_style,
|
||||||
|
selected_style=tab_selected_style,
|
||||||
|
),
|
||||||
|
],
|
||||||
|
),
|
||||||
|
html.Div(id="tabs-example-content-1"),
|
||||||
|
],
|
||||||
|
),
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
@callback(
|
||||||
|
Output("select_company", "options"), Input("select_company", "search_value")
|
||||||
|
)
|
||||||
|
def update_options(search_value: str) -> list:
|
||||||
|
"""Update page based on selected company."""
|
||||||
|
if not search_value:
|
||||||
|
raise PreventUpdate
|
||||||
|
return [o for o in options if search_value in o["label"]]
|
||||||
|
|
||||||
|
app.run_server(debug=True)
|
@ -1,9 +1,8 @@
|
|||||||
"""Dash."""
|
"""Dash."""
|
||||||
|
|
||||||
import pandas as pd
|
import pandas as pd
|
||||||
from dash import Dash, dash_table
|
from dash import Dash, Input, Output, callback, dash_table, dcc, html
|
||||||
|
|
||||||
# from ..utils.postgres.connector import get_engine, init_db
|
|
||||||
from aki_prj23_transparenzregister.utils.postgres import entities
|
from aki_prj23_transparenzregister.utils.postgres import entities
|
||||||
from aki_prj23_transparenzregister.utils.postgres.connector import (
|
from aki_prj23_transparenzregister.utils.postgres.connector import (
|
||||||
get_session,
|
get_session,
|
||||||
@ -16,9 +15,29 @@ if __name__ == "__main__":
|
|||||||
companies_df: pd.DataFrame = pd.read_sql(str(query), session.bind) # type: ignore
|
companies_df: pd.DataFrame = pd.read_sql(str(query), session.bind) # type: ignore
|
||||||
app = Dash(__name__)
|
app = Dash(__name__)
|
||||||
|
|
||||||
app.layout = dash_table.DataTable(
|
app.layout = html.Div(
|
||||||
companies_df.to_dict("records"),
|
[
|
||||||
[{"name": i, "id": i} for i in companies_df.columns],
|
html.H1(children="Company Data", style={"textAlign": "center"}),
|
||||||
|
html.Div(
|
||||||
|
[
|
||||||
|
dcc.Dropdown(
|
||||||
|
companies_df.company_name.unique(),
|
||||||
|
"Firma 1",
|
||||||
|
id="dropdown-selection",
|
||||||
|
),
|
||||||
|
]
|
||||||
|
),
|
||||||
|
html.Div(id="data_table"),
|
||||||
|
]
|
||||||
)
|
)
|
||||||
|
|
||||||
|
@callback(Output("data_table", "children"), Input("dropdown-selection", "value"))
|
||||||
|
def display_table(value: str) -> dash_table:
|
||||||
|
"""Output table with company stats based on dropdown value."""
|
||||||
|
dff = companies_df[companies_df.company_name == value]
|
||||||
|
return dash_table.DataTable(
|
||||||
|
data=dff.to_dict("records"),
|
||||||
|
columns=[{"id": c, "name": c} for c in companies_df.columns],
|
||||||
|
)
|
||||||
|
|
||||||
app.run(debug=True)
|
app.run(debug=True)
|
||||||
|
@ -2,15 +2,20 @@
|
|||||||
import enum
|
import enum
|
||||||
|
|
||||||
|
|
||||||
class RelationTypeEnum(enum.Enum):
|
class RelationTypeEnum(enum.IntEnum):
|
||||||
"""RelationTypeEnum."""
|
"""RelationTypeEnum."""
|
||||||
|
|
||||||
executive = "Executive"
|
EXECUTIVE = enum.auto()
|
||||||
auditor = "Auditor"
|
AUDITOR = enum.auto()
|
||||||
supervisory_board = "Supervisory_Board"
|
SUPERVISORY_BOARD = enum.auto()
|
||||||
managing_director = "Managing_Directory"
|
MANAGING_DIRECTOR = enum.auto()
|
||||||
authorized_representative = "Authorized_Representative"
|
AUTHORIZED_REPRESENTATIVE = enum.auto()
|
||||||
final_auditor = "Final_Auditor"
|
FINAL_AUDITOR = enum.auto()
|
||||||
|
|
||||||
|
PARTICIPATES_WITH = enum.auto()
|
||||||
|
HAS_SHARES_OF = enum.auto()
|
||||||
|
IS_SUPPLIED_BY = enum.auto()
|
||||||
|
WORKS_WITH = enum.auto()
|
||||||
|
|
||||||
|
|
||||||
class SentimentTypeEnum(enum.Enum):
|
class SentimentTypeEnum(enum.Enum):
|
||||||
@ -20,12 +25,3 @@ class SentimentTypeEnum(enum.Enum):
|
|||||||
sustainability = "sustainability"
|
sustainability = "sustainability"
|
||||||
environmental_aspects = "environmental_aspects"
|
environmental_aspects = "environmental_aspects"
|
||||||
perception = "perception"
|
perception = "perception"
|
||||||
|
|
||||||
|
|
||||||
class RelationTypeCompanyEnum(enum.Enum):
|
|
||||||
"""RelationTypeCompanyEnum."""
|
|
||||||
|
|
||||||
participates_with = "participates_with"
|
|
||||||
has_shares_of = "has_shares_of"
|
|
||||||
is_supplied_by = "is_supplied_by"
|
|
||||||
works_with = "works_with"
|
|
||||||
|
@ -1,11 +1,10 @@
|
|||||||
"""Module containing connection utils for PostgreSQL DB."""
|
"""Module containing connection utils for PostgreSQL DB."""
|
||||||
from sqlalchemy import create_engine
|
from sqlalchemy import create_engine
|
||||||
from sqlalchemy.engine import URL, Engine
|
from sqlalchemy.engine import URL, Engine
|
||||||
from sqlalchemy.orm import Session, sessionmaker
|
from sqlalchemy.orm import Session, declarative_base, sessionmaker
|
||||||
|
|
||||||
from aki_prj23_transparenzregister.config.config_providers import JsonFileConfigProvider
|
from aki_prj23_transparenzregister.config.config_providers import JsonFileConfigProvider
|
||||||
from aki_prj23_transparenzregister.config.config_template import PostgreConnectionString
|
from aki_prj23_transparenzregister.config.config_template import PostgreConnectionString
|
||||||
from aki_prj23_transparenzregister.utils.postgres.entities import Base
|
|
||||||
|
|
||||||
|
|
||||||
def get_engine(conn_args: PostgreConnectionString) -> Engine:
|
def get_engine(conn_args: PostgreConnectionString) -> Engine:
|
||||||
@ -34,6 +33,9 @@ def get_session() -> Session: # pragma: no cover
|
|||||||
return session()
|
return session()
|
||||||
|
|
||||||
|
|
||||||
|
Base = declarative_base()
|
||||||
|
|
||||||
|
|
||||||
def init_db() -> None:
|
def init_db() -> None:
|
||||||
"""Initialize DB with all defined entities."""
|
"""Initialize DB with all defined entities."""
|
||||||
config_provider = JsonFileConfigProvider("./secrets.json")
|
config_provider = JsonFileConfigProvider("./secrets.json")
|
||||||
|
@ -1,20 +1,15 @@
|
|||||||
"""ORM entities for Prod. DB."""
|
"""ORM entities for Prod. DB."""
|
||||||
import uuid
|
|
||||||
from datetime import datetime
|
from datetime import datetime
|
||||||
|
|
||||||
import sqlalchemy as sa
|
import sqlalchemy as sa
|
||||||
from sqlalchemy.orm import (
|
|
||||||
declarative_base,
|
|
||||||
)
|
|
||||||
|
|
||||||
from aki_prj23_transparenzregister.utils.enumy_types import (
|
from aki_prj23_transparenzregister.utils.enumy_types import (
|
||||||
RelationTypeCompanyEnum,
|
|
||||||
RelationTypeEnum,
|
RelationTypeEnum,
|
||||||
SentimentTypeEnum,
|
SentimentTypeEnum,
|
||||||
)
|
)
|
||||||
|
from aki_prj23_transparenzregister.utils.postgres.connector import Base
|
||||||
|
|
||||||
# # create an object *district_court* which inherits attributes from Base-class
|
# # create an object *district_court* which inherits attributes from Base-class
|
||||||
Base = declarative_base()
|
|
||||||
|
|
||||||
|
|
||||||
class DistrictCourt(Base):
|
class DistrictCourt(Base):
|
||||||
@ -22,9 +17,9 @@ class DistrictCourt(Base):
|
|||||||
|
|
||||||
__tablename__ = "district_court"
|
__tablename__ = "district_court"
|
||||||
|
|
||||||
id = sa.Column(sa.Integer, primary_key=True)
|
id = sa.Column(sa.Integer, primary_key=True, autoincrement=True)
|
||||||
city = sa.Column(sa.String(100), nullable=False)
|
city = sa.Column(sa.String(100), nullable=False)
|
||||||
name = sa.Column(sa.String(100), nullable=False)
|
name = sa.Column(sa.String(100), nullable=False, unique=True)
|
||||||
|
|
||||||
|
|
||||||
class Company(Base):
|
class Company(Base):
|
||||||
@ -33,32 +28,48 @@ class Company(Base):
|
|||||||
__tablename__ = "company"
|
__tablename__ = "company"
|
||||||
|
|
||||||
__table_args__ = (
|
__table_args__ = (
|
||||||
sa.PrimaryKeyConstraint("id"),
|
|
||||||
sa.UniqueConstraint("hr", "court_id"),
|
sa.UniqueConstraint("hr", "court_id"),
|
||||||
|
sa.UniqueConstraint("name", "city"),
|
||||||
|
sa.UniqueConstraint("name", "zip_code"),
|
||||||
|
sa.UniqueConstraint("name"),
|
||||||
)
|
)
|
||||||
|
|
||||||
id = sa.Column(sa.String, primary_key=True, default=uuid.uuid4, unique=True)
|
id = sa.Column(sa.Integer, primary_key=True, autoincrement=True)
|
||||||
hr = sa.Column(sa.Integer, nullable=False)
|
hr = sa.Column(sa.String, nullable=False)
|
||||||
court_id = sa.Column(
|
court_id = sa.Column(
|
||||||
sa.Integer,
|
sa.Integer,
|
||||||
sa.ForeignKey("district_court.id"),
|
sa.ForeignKey("district_court.id"),
|
||||||
nullable=False,
|
nullable=False,
|
||||||
)
|
)
|
||||||
name = sa.Column(sa.String(100), nullable=False)
|
name = sa.Column(sa.String(150), nullable=False)
|
||||||
street = sa.Column(sa.String(100), nullable=False)
|
street = sa.Column(sa.String(100), nullable=True)
|
||||||
zip_code = sa.Column(sa.String(5), nullable=False)
|
zip_code = sa.Column(sa.String(5), nullable=True)
|
||||||
city = sa.Column(sa.String(100), nullable=False)
|
city = sa.Column(sa.String(100), nullable=True)
|
||||||
sector = sa.Column(sa.String(100), nullable=False)
|
last_update = sa.Column(sa.Date, nullable=False)
|
||||||
|
sector = sa.Column(sa.String(100), nullable=True)
|
||||||
|
|
||||||
|
|
||||||
class Finance(Base):
|
class Person(Base):
|
||||||
"""Finance."""
|
"""Person."""
|
||||||
|
|
||||||
__tablename__ = "finance"
|
__tablename__ = "person"
|
||||||
|
__table_args__ = (sa.UniqueConstraint("name", "surname", "date_of_birth"),)
|
||||||
|
|
||||||
id = sa.Column(sa.Integer, primary_key=True)
|
id = sa.Column(sa.Integer, primary_key=True)
|
||||||
company_id = sa.Column(sa.String, sa.ForeignKey("company.id"))
|
name = sa.Column(sa.String(100), nullable=False)
|
||||||
date = sa.Column(sa.DateTime(timezone=True), default=datetime.now)
|
surname = sa.Column(sa.String(100), nullable=False)
|
||||||
|
date_of_birth = sa.Column(sa.Date, nullable=False)
|
||||||
|
works_for = sa.Column(sa.String(100), nullable=True)
|
||||||
|
|
||||||
|
|
||||||
|
class AnnualFinanceStatement(Base):
|
||||||
|
"""Finance."""
|
||||||
|
|
||||||
|
__tablename__ = "annual_finance_statement"
|
||||||
|
|
||||||
|
id = sa.Column(sa.Integer, primary_key=True)
|
||||||
|
company_id = sa.Column(sa.Integer, sa.ForeignKey("company.id"))
|
||||||
|
date = sa.Column(sa.DateTime(timezone=True), nullable=False)
|
||||||
total_volume = sa.Column(sa.Float)
|
total_volume = sa.Column(sa.Float)
|
||||||
ebit = sa.Column(sa.Float)
|
ebit = sa.Column(sa.Float)
|
||||||
ebitda = sa.Column(sa.Float)
|
ebitda = sa.Column(sa.Float)
|
||||||
@ -68,7 +79,6 @@ class Finance(Base):
|
|||||||
debt = sa.Column(sa.Float)
|
debt = sa.Column(sa.Float)
|
||||||
return_on_equity = sa.Column(sa.Float)
|
return_on_equity = sa.Column(sa.Float)
|
||||||
capital_turnover_rate = sa.Column(sa.Float)
|
capital_turnover_rate = sa.Column(sa.Float)
|
||||||
|
|
||||||
# company: Mapped[Company] = relationship(Company)
|
# company: Mapped[Company] = relationship(Company)
|
||||||
|
|
||||||
|
|
||||||
@ -79,7 +89,7 @@ class Sentiment(Base):
|
|||||||
__tablename__ = "sentiment"
|
__tablename__ = "sentiment"
|
||||||
|
|
||||||
id = sa.Column(sa.Integer, primary_key=True)
|
id = sa.Column(sa.Integer, primary_key=True)
|
||||||
company_id = sa.Column(sa.String, sa.ForeignKey("company.id"))
|
company_id = sa.Column(sa.Integer, sa.ForeignKey("company.id"))
|
||||||
date = sa.Column(sa.DateTime(timezone=True), default=datetime.now)
|
date = sa.Column(sa.DateTime(timezone=True), default=datetime.now)
|
||||||
sentiment_type = sa.Column(
|
sentiment_type = sa.Column(
|
||||||
sa.Enum(SentimentTypeEnum),
|
sa.Enum(SentimentTypeEnum),
|
||||||
@ -87,53 +97,48 @@ class Sentiment(Base):
|
|||||||
)
|
)
|
||||||
value = sa.Column(sa.Float(), nullable=False)
|
value = sa.Column(sa.Float(), nullable=False)
|
||||||
source = sa.Column(sa.String(100))
|
source = sa.Column(sa.String(100))
|
||||||
|
|
||||||
# sentiment = relationship(Company)
|
# sentiment = relationship(Company)
|
||||||
|
|
||||||
|
|
||||||
# create person object
|
# create person object
|
||||||
class Person(Base):
|
|
||||||
"""Person."""
|
|
||||||
|
|
||||||
__tablename__ = "person"
|
|
||||||
|
|
||||||
|
class Relation(Base):
|
||||||
|
"""A super table containing all relations."""
|
||||||
|
|
||||||
|
__tablename__ = "relation"
|
||||||
id = sa.Column(sa.Integer, primary_key=True)
|
id = sa.Column(sa.Integer, primary_key=True)
|
||||||
name = sa.Column(sa.String(100), nullable=False)
|
company_id = sa.Column(sa.Integer, sa.ForeignKey("company.id"))
|
||||||
surname = sa.Column(sa.String(100), nullable=False)
|
|
||||||
works_for = sa.Column(sa.String(100))
|
date_from = sa.Column(sa.DateTime(timezone=True), nullable=True)
|
||||||
|
date_to = sa.Column(sa.DateTime(timezone=True), nullable=True)
|
||||||
|
|
||||||
|
relation = sa.Column(sa.Enum(RelationTypeEnum), nullable=False)
|
||||||
|
|
||||||
|
|
||||||
# create own relation type and person_relation object
|
# create own relation type and person_relation object
|
||||||
class PersonRelation(Base):
|
class PersonRelation(Relation):
|
||||||
"""PersonRelation."""
|
"""PersonRelation."""
|
||||||
|
|
||||||
__tablename__ = "person_relation"
|
__tablename__ = "person_relation"
|
||||||
|
|
||||||
id = sa.Column(sa.Integer, primary_key=True)
|
id = sa.Column(sa.Integer, sa.ForeignKey("relation.id"), primary_key=True)
|
||||||
company_id = sa.Column(sa.String, sa.ForeignKey("company.id"))
|
person_id = sa.Column(sa.Integer, sa.ForeignKey("person.id"))
|
||||||
person_id = sa.Column(sa.Integer, sa.ForeignKey(Person.id))
|
|
||||||
date_from = sa.Column(sa.DateTime(timezone=True), default=datetime.now)
|
|
||||||
date_to = sa.Column(sa.DateTime(timezone=True), default=datetime.now)
|
|
||||||
relation = sa.Column(sa.Enum(RelationTypeEnum), nullable=False)
|
|
||||||
|
|
||||||
# company = relationship("Company")
|
# company = relationship("Company")
|
||||||
# person = relationship("Person", foreign_keys=[person_id])
|
# person = relationship("Person", foreign_keys=[person_id])
|
||||||
# company = relationship('Company', foreign_keys=[company_hr,company_court])
|
# company = relationship('Company', foreign_keys=[company_hr,company_court])
|
||||||
|
__table_args__ = {"extend_existing": True}
|
||||||
|
|
||||||
|
|
||||||
# create own relation type and company_relation object
|
# create own relation type and company_relation object
|
||||||
class CompanyRelation(Base):
|
class CompanyRelation(Relation):
|
||||||
"""CompanyRelation."""
|
"""CompanyRelation."""
|
||||||
|
|
||||||
__tablename__ = "company_relation"
|
__tablename__ = "company_relation"
|
||||||
|
|
||||||
id = sa.Column(sa.Integer, primary_key=True)
|
id = sa.Column(sa.Integer, sa.ForeignKey("relation.id"), primary_key=True)
|
||||||
company1_id = sa.Column(sa.String, sa.ForeignKey("company.id"), nullable=False)
|
company2_id = sa.Column(sa.Integer, sa.ForeignKey("company.id"), nullable=False)
|
||||||
company2_id = sa.Column(sa.String, sa.ForeignKey("company.id"), nullable=False)
|
|
||||||
date_from = sa.Column(sa.DateTime(timezone=True), default=datetime.now)
|
|
||||||
date_to = sa.Column(sa.DateTime(timezone=True), default=datetime.now)
|
|
||||||
relation = sa.Column(sa.Enum(RelationTypeCompanyEnum), nullable=False)
|
|
||||||
|
|
||||||
# company = relationship("Company")
|
# company = relationship("Company")
|
||||||
|
|
||||||
__table_args__ = {"extend_existing": True}
|
__table_args__ = {"extend_existing": True}
|
||||||
|
@ -1 +0,0 @@
|
|||||||
"""Tests for config module."""
|
|
7
tests/ui/company_finance_dash_test.py
Normal file
7
tests/ui/company_finance_dash_test.py
Normal file
@ -0,0 +1,7 @@
|
|||||||
|
"""Test for the company stats dashboard."""
|
||||||
|
from aki_prj23_transparenzregister.ui import company_finance_dash
|
||||||
|
|
||||||
|
|
||||||
|
def test_import() -> None:
|
||||||
|
"""Checks if an import co company_stats_dash can be made."""
|
||||||
|
assert company_finance_dash is not None
|
@ -1 +0,0 @@
|
|||||||
"""Mongo utils module."""
|
|
@ -1 +0,0 @@
|
|||||||
"""Tests for utils.postgres module."""
|
|
@ -18,7 +18,7 @@ def test_init_db() -> None:
|
|||||||
with patch(
|
with patch(
|
||||||
"aki_prj23_transparenzregister.utils.postgres.connector.get_engine"
|
"aki_prj23_transparenzregister.utils.postgres.connector.get_engine"
|
||||||
) as mock_get_engine, patch(
|
) as mock_get_engine, patch(
|
||||||
"aki_prj23_transparenzregister.utils.postgres.entities.declarative_base"
|
"aki_prj23_transparenzregister.utils.postgres.connector.declarative_base"
|
||||||
) as mock_declarative_base, patch(
|
) as mock_declarative_base, patch(
|
||||||
"aki_prj23_transparenzregister.utils.postgres.connector.JsonFileConfigProvider"
|
"aki_prj23_transparenzregister.utils.postgres.connector.JsonFileConfigProvider"
|
||||||
) as mock_provider:
|
) as mock_provider:
|
||||||
@ -33,4 +33,3 @@ def test_init_db() -> None:
|
|||||||
mock_value.get_postgre_connection_string.return_value = ""
|
mock_value.get_postgre_connection_string.return_value = ""
|
||||||
|
|
||||||
init_db()
|
init_db()
|
||||||
assert True
|
|
||||||
|
@ -1,4 +1,4 @@
|
|||||||
def test_import() -> None:
|
def test_import() -> None:
|
||||||
from aki_prj23_transparenzregister.utils.postgres import entities
|
from aki_prj23_transparenzregister.utils.postgres import entities
|
||||||
|
|
||||||
assert entities is not None
|
assert entities
|
||||||
|
Loading…
x
Reference in New Issue
Block a user