Merge branch 'main' into feat/fetch-financials

This commit is contained in:
TrisNol 2023-09-06 17:16:48 +02:00
commit 00a5e9ec25
25 changed files with 1346 additions and 713 deletions

View File

@ -53,7 +53,7 @@ jobs:
- uses: actions/checkout@v3
- uses: chartboost/ruff-action@v1
with:
version: 0.0.277
version: 0.0.287
python-requirements:
name: Check Python Requirements

View File

@ -26,7 +26,7 @@ repos:
- repo: https://github.com/astral-sh/ruff-pre-commit
# Ruff version.
rev: v0.0.284
rev: v0.0.287
hooks:
- id: ruff
args: [--fix, --exit-non-zero-on-fix]
@ -53,7 +53,7 @@ repos:
- repo: https://github.com/pre-commit/mirrors-mypy
rev: v1.5.0
rev: v1.5.1
hooks:
- id: mypy
additional_dependencies:
@ -68,7 +68,7 @@ repos:
- id: md-toc
- repo: https://github.com/python-poetry/poetry
rev: 1.5.0
rev: 1.6.0
hooks:
- id: poetry-check
@ -78,6 +78,6 @@ repos:
- id: validate-html
- repo: https://github.com/python-jsonschema/check-jsonschema
rev: 0.24.1
rev: 0.26.3
hooks:
- id: check-github-workflows

View File

@ -1,70 +1,72 @@
# Contribution guidelines
## Dev Setup
- [Install Python 3.11](https://www.python.org/downloads/release/python-3111/)
- [Install Poetry](https://python-poetry.org/docs/#installation)
- [Install GiT](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)
- [Configure GiT](https://support.atlassian.com/bitbucket-cloud/docs/configure-your-dvcs-username-for-commits/)
- [Generate an SSH Key](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent) & [Add SSH Key to GitHub](https://docs.github.com/en/authentication/connecting-to-github-with-ssh/adding-a-new-ssh-key-to-your-github-account)
- [Clone the Project (IDE Specific)](https://docs.github.com/de/repositories/creating-and-managing-repositories/cloning-a-repository)
- [Install the python Project](https://python-poetry.org/docs/basic-usage/#installing-dependencies)
- [Install pre-commit](https://pre-commit.com/#install)
## Repository structure
- **src/`aki_prj23_transparenzregister`**:
- This subfolder/`package` contains several subdirectories:
- `ai`:
- Houses AI models and pipelines, including NER and sentiment analysis.
- `config`:
- Contains configuration files such as database connection strings and model files.
- `models`:
- Stores data models.
- `ui`:
- Manages the user interface and dash interface.
- `utils`:
- Contains general tooling functions, including database access and other miscellaneous functions.
- **tests**:
- Test files organized in a mirrored structure of the 'src' folder. Please at least import every python file you
add.
Please Try to test every function with a test that compares with an example result. If that is not possible it
would be best to find a big consensus that only limited testing is required.
## Code style
We defined to use the following formats:
- Whitespace Formatting by [Black](https://github.com/psf/black)
- Docstrings Style see the examples provided by [sphinx napoleon](https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html) Types in the docstrings are optional but should be consistent per module.
- Docstrings Style see the examples provided
by [sphinx napoleon](https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html) Types in the
docstrings are optional but should be consistent per module.
- Import Order by [isort](https://pycqa.github.io/isort/)
- Strict Typing of function headers see the examples provided by the [mypy](https://mypy.readthedocs.io/en/stable/cheat_sheet_py3.html) documentation.
- Strict Typing of function headers see the examples provided by
the [mypy](https://mypy.readthedocs.io/en/stable/cheat_sheet_py3.html) documentation.
- The generell linting is done by ruff but since ruff is a reimplementation of ruff of many different linters the old
documentations are still valid.
- Ruff can be executed with the `ruff .` command. May errors are auto-fixable with `ruff --fix .` if they are straight
forward changes.
## Language
We decided to use english on everything close to code but will write longer texts that are not close to the code in german.
We decided to use english on everything close to code but will write longer texts that are not close to the code in
german.
## Pre-Commit installation
## Pre-Commit Usage
- clone the project
- install the following python package:
- `pre-commit`
- execute the command `pre-commit install` to insert the git githook into your local repository.
- for more information read [Python Pre-Commit](https://pre-commit.com/)
- the following code format rules are so enforced:
- Whitespace formatierung:
- Whitespace formatting:
- Python-[Black](https://github.com/psf/black)
- leading whitespace in all files
- Auto format: yaml, toml, ini
- Import norm by:
- [isort](https://pycqa.github.io/isort/)
- absolut module path
- Validierung:
- Validation:
- yaml, toml, json, xml
- Secret detection
- python normen ([pep8](https://peps.python.org/pep-0008/)) with [flake8](https://flake8.pycqa.org/en/latest/)
- python norms ([pep8](https://peps.python.org/pep-0008/))
with [ruff](https://github.com/astral-sh/ruff)
- type checking with ([mypy](https://github.com/python/mypy))
- Install [Python 3.11](https://www.python.org/downloads/release/python-3111/)
- Install Poetry
```
pip install poetry
```
- Install dependencies
```
poetry install
```
## Setup
### Connection strings
Create a `secrets.json` in the root of this repo with the following structure (values to be replaces by desired config):
```json
{
"postgres": {
"username": "postgres",
"password": "postgres",
"host": "localhost",
"database": "postgres",
"port": 5432
},
"mongo": {
"username": "username",
"password": "password",
"host": "localhost",
"database": "transparenzregister",
"port": 27017
}
}
```
Example usage see [connector.py](./src/aki_prj23_transparenzregister/utils/postgres/connector.py)
- for more information read [Python Pre-Commit](https://pre-commit.com/)

View File

@ -1,13 +1,39 @@
# aki_prj23_transparenzregister
[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/charliermarsh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
[![python](https://img.shields.io/badge/Python-3.11-3776AB.svg?style=flat&logo=python&logoColor=white)](https://www.python.org)
[![Actions status](https://github.com/astral-sh/ruff/workflows/CI/badge.svg)](https://github.com/astral-sh/ruff/actions)
[![Pytest](https://github.com/fhswf/aki_prj23_transparenzregister/actions/workflows/test-action.yaml/badge.svg?branch=main)](https://github.com/fhswf/aki_prj23_transparenzregister/actions/workflows/test-action.yaml)
[![Python-Lint-Push-Action](https://github.com/fhswf/aki_prj23_transparenzregister/actions/workflows/lint-actions.yaml/badge.svg)](https://github.com/fhswf/aki_prj23_transparenzregister/actions/workflows/lint-actions.yaml)
[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/charliermarsh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit&logoColor=white)](https://github.com/pre-commit/pre-commit)
[![Checked with mypy](http://www.mypy-lang.org/static/mypy_badge.svg)](http://mypy-lang.org/)
[![Documentation Status](https://readthedocs.org/projects/mypy/badge/?version=stable)](https://mypy.readthedocs.io/en/stable/?badge=stable)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
## Contributions
See the [CONTRIBUTING.md](CONTRIBUTING.md) about how code should be formatted and what kind of rules we set ourselves.
[![bandit](https://github.com/fhswf/aki_prj23_transparenzregister/actions/workflows/bandit-action.yaml/badge.svg)](https://github.com/fhswf/aki_prj23_transparenzregister/actions/workflows/bandit-action.yaml)
## DB Connection settings
To connect to the SQL db see [sql/connector.py](./src/aki_prj23_transparenzregister/utils/postgres/connector.py)
To connect to the Mongo db see [connect]
Create a `secrets.json` in the root of this repo with the following structure (values to be replaces by desired config):
```json
{
"postgres": {
"username": "postgres",
"password": "postgres",
"host": "localhost",
"database": "postgres",
"port": 5432
},
"mongo": {
"username": "username",
"password": "password",
"host": "localhost",
"database": "transparenzregister",
"port": 27017
}
}
```

View File

@ -4,6 +4,11 @@ Transparenzregister Dokumentation
=================================
This is the documentation for the AKI project group on the german transparenzregister and an Analysis there of.
.. include:: ../README.md
:parser: myst_parser.sphinx_
.. include:: ../CONTRIBUTING.md
:parser: myst_parser.sphinx_
.. toctree::
:maxdepth: 3
:caption: Project Management

View File

@ -1,6 +1,7 @@
# Weekly *5*: 09.06.2023
# Weekly *6*: 09.06.2023
## Teilnehmer
- Prof. Arinir
- Tristan Nolde
- Tim Ronneburg
@ -16,8 +17,10 @@
- Befürchtung ist, dass es zu einem Hinderniss wird
- Entscheidung liegt daher beim Projekt-Team
- Weitere Informationen sind nicht aktuell nicht vorhanden
- Vorschlag Prof. Arinir: Sollte das Thema nochmal zum Team getragen werden, wird der aktuelle Stand vorgestellt und der Link zum Repo wird geteilt. Darüber hinaus werden keine Ressourcen zugesprochen.
- Vorstellung [vorheriger Absprache](https://github.com/orgs/fhswf/projects/17?pane=issue&itemId=29707639) und Feedback:
- Vorschlag Prof. Arinir: Sollte das Thema nochmal zum Team getragen werden, wird der aktuelle Stand vorgestellt und
der Link zum Repo wird geteilt. Darüber hinaus werden keine Ressourcen zugesprochen.
- Vorstellung [vorheriger Absprache](https://github.com/orgs/fhswf/projects/17?pane=issue&itemId=29707639) und
Feedback:
- Ändert sich der Scope - Nein
- NDA - Nein
- Veröffentlichung - maximal Impressionen
@ -40,8 +43,10 @@
- Effekt von Farbwahl
- Erste Umsetzung im Jupyter Notebook
- Feedback:
- Es werden extem viele Datenpunkte angezeigt werden müssen, wie wird dies in den Bibliotheken umgesetzt? Kann dort gefiltert werden?
- Wenn nicht direkt am Graphen (der Darstellung) gefiltert werden kann, dann frühzeitig filtern, bevor der Graph gebaut wird
- Es werden extem viele Datenpunkte angezeigt werden müssen, wie wird dies in den Bibliotheken umgesetzt?
Kann dort gefiltert werden?
- Wenn nicht direkt am Graphen (der Darstellung) gefiltert werden kann, dann frühzeitig filtern, bevor
der Graph gebaut wird
- Datenspeicherung
- Erste Integration von Visualisierung mit Datenspeicherung
- Vorstellung der "Datencluster"
@ -50,12 +55,15 @@
- Social Graph
- Zeitseriendaten
- Relationales DB Modell
- Fokus ebenfalls auf Abfrage der Daten für Folge-Projekte wie Visualiserung und Mehrwert fürs Team, weniger Theorie
- Fokus ebenfalls auf Abfrage der Daten für Folge-Projekte wie Visualiserung und Mehrwert fürs Team, weniger
Theorie
- Feedback:
- Es müssen Erfahrungen mit der Library und Darstellung gesammelt werden, um den Mehrwert der Lösung hervorzuheben
- Es müssen Erfahrungen mit der Library und Darstellung gesammelt werden, um den Mehrwert der Lösung
hervorzuheben
- Modellierung der Finzanz-Kennzahlen
- Spaltennamen sollen sprechend sein, z.B. "value" statt "sum"
- Präferenz zum Modell mit einzelnem Eintrag mit mehren Kennzahl Spalten stallt generischer Lösung über Enum
- Präferenz zum Modell mit einzelnem Eintrag mit mehren Kennzahl Spalten stallt generischer Lösung über
Enum
- Text Mining
- Fokus auf Sentiment Analyse
- Vergleich verschiedener Lösungen und ML Modelle
@ -84,7 +92,7 @@
## Abgeleitete Action Items
| Action Item | Verantwortlicher | Deadline |
|-------------|------------------|-----------------|
|------------------------------------------------------------------|------------------|-------------------------|
| Folien hochladen | Projekt Team | vor Präsentationstermin |
| Absprache Abgrenzung von Verflechtungsanalyse und Visualisierung | Tim und Kim | nächster Abgleich |
| Deployment Plan aufstellen | Projekt Team | nach Seminararbeiten |

View File

@ -1,6 +1,7 @@
# Weekly *X*: 03.08.2023
# Weekly *7*: 03.08.2023
## Teilnehmer
- Prof. Arinir
- Tristan Nolde
- Tim Ronneburg (Protokollant)
@ -20,7 +21,7 @@
## Abgeleitete Action Items
| Action Item | Verantwortlicher | Deadline |
|-------------|------------------|-----------------|
|--------------------------------------------------------------------|----------------------------|-----------------|
| Mergen aller Branches zu jedem neuen Termin mit Herrn Arinir | Jeder | jedes Weekly |
| Erstellen der Pipelines | Sebastian, Tristan und Tim | nächstes Weekly |
| Erstellen der Development Datenbank-Instanzen je Entwickler | Sebastian, Tristan und Tim | nächstes Weekly |

View File

@ -0,0 +1,44 @@
# Weekly *8*: 17.08.2023
## Teilnehmer
- Prof. Arinir
- Tristan Nolde
- Tim Ronneburg
- Kim Mesewinkel-Risse
- Phillip Horstenkamp
- Sebastian Zeleny (Protokollant)
## Themen
- Welche Services laufen aktuell auf dem Uni-Cluster?
- MongoDB und Postgres mit personalisiertem Zugang über VPN
- **Requirement:** Das Frontend bzw. Visualisierung soll auf dem Cluster laufen, wofür ein Login erwünscht ist, wie
z.B. SSO
- die Services (Text Mining, NER, Sentiment) sollten auch dem Cluster laufen
- Wo sollen CI/CD laufen?
- benötigt werden 2-3 Container für Worker und Services
- ProductionDB:
- DB "transparenzregister" für Produktiveinsatz
- persönliche DB "DB_*Name*" für Development
- Erklärung des Postgres Connectors, welcher die SQL-Alchemy Klassen verwendet, um Tabellen zu erstellen
- Erklärung wie der Connection-String mit dem JsonConfigProvider und der secret.json erzeugt wird
- UI:
- Vorstellung der ersten Visualisierung mit Plotly und Anbindung an ProductionDB
- Dash startet im Hintergrund einen http-Server, welcher über den Port 8050 erreichbar ist
- Dash wird für das Dashboarding verwendet und wird (bei Bedarf) durch weitere Komponenten erweitert
- Abschluß des PoC und Umsetzung der bestehenden Architektur
## Abgeleitete Action Items
| Action Item | Verantwortlicher | Deadline |
|------------------------------------------------------------------|------------------|-----------------|
| Anfrage nach Serverressourcen für 2-3 Container bei Prof. Gawron | Prof. Arinir | nächstes Weekly |
| Repo-Struktur dokumentieren | Phillip, Tristan | nächstes Weekly |
| Anlegen von MongoDB-Instanzen für NER und Sentiment | Tristan | nächstes Weekly |
| NER für News | Sebastian | nächstes Weekly |
| Beispiel für MongoConnector | Sebastian | nächstes Weekly |
| Script um News auf Uni-Cluster zu dumpen | Tristan | nächstes Weekly |
| Finanzdaten aus Bundesanzeiger | Tristan | nächstes Weekly |
| Plotly Frontend weiterentwickeln | Kim, Tim | nächstes Weekly |
| Refactoring der SQL-Alchemy Klassen | Phillip | nächstes Weekly |

View File

@ -0,0 +1,40 @@
# Weekly *9*: 31.08.2023
## Teilnehmer
- Prof. Arinir
- Tristan Nolde
- Philipp Horstenkamp
- Sebastian Zeleny
- Kim Mesewinkel-Risse (Protokoll)
## Themen
- Rückmeldung von Herrn Gawron bzgl. mehr Ressourcen steht noch aus, ggfs. persönliche Absprache nächste Woche möglich
- Rückfrage von Herrn Arinir bezüglich Aufbau der Software und Architektur
- Gerade werden einzelne Funktionen erstellt, Daten werden ungefiltert in die Mongo DB geschrieben, anschließend
Bereinigung und Übertragung in die Postgres
- Vorstellung aktueller Repo-Struktur durch Tristan, relevanter Code befindet sich im src-Ordner
- Wie kann sichergestellt werden, dass unsere Ziele erreicht werden?
- Zeitplan/Meilensteinplan gewünscht
- Wann soll was erreicht werden?
- Burndown-Diagramm
-> Umsetzung durch Team beim Präsenzmeeting am 09.09.2023
- Kurze Vorstellung der bearbeiteten Themen: NER + Sentiment (Sebastian), Finanzdaten (Tristan), UI (Kim),
Datentransfer (Philipp)
## Abgeleitete Action Items
| Action Item | Verantwortlicher | Deadline |
|------------------------------------------------------|------------------|-----------------|
| Festlegung Zeitplan für Präsenztreffen | Alle | 07.09.2023 |
| Zeitplan bzw. Meilensteinplan | Alle | nächstes Weekly |
| Erstellen einer Übersicht aller bestehenden Services | Alle | nächstes Weekly |
| Update bzgl. Cluster und Ressourcen | Herr Arinir | nächstes Weekly |
| Finanzdaten finalisieren | Tristan | nächstes Weekly |
| NER/Sentiment Aggregation | Sebastian | nächstes Weekly |
| Füllen der UI mit Echtdaten | Kim | nächstes Weekly |
| Teststruktur für SQL hinzufügen | Philipp | nächstes Weekly |

1059
poetry.lock generated

File diff suppressed because it is too large Load Diff

View File

@ -37,6 +37,7 @@ version = "0.1.0"
[tool.poetry.dependencies]
SQLAlchemy = {version = "^1.4.46", extras = ["mypy"]}
dash = "^2.11.1"
dash-bootstrap-components = "^1.4.2"
deutschland = {git = "https://github.com/TrisNol/deutschland.git", branch = "hotfix/python-3.11-support"}
loguru = "^0.7.0"
matplotlib = "^3.7.1"
@ -89,7 +90,7 @@ pytest-cov = "^4.1.0"
pytest-mock = "^3.10.0"
pytest-repeat = "^0.9.1"
# TODO Add enrich_company_financials hinzufügen
# TODO Add enrich_company_financials hinzufügen
[tool.poetry.scripts]
mein_test = "aki_prj23_transparenzregister.utils.postgres.connector:init_db"

View File

@ -0,0 +1 @@
"""This module contains all the ai pipelines."""

View File

@ -1 +1 @@
"""App configuration."""
"""App configuration tools."""

View File

@ -0,0 +1,3 @@
body {
margin: 0;
}

View File

@ -0,0 +1,392 @@
"""Dash."""
import dash_bootstrap_components as dbc
import pandas as pd
import plotly.graph_objs as go
from dash import Dash, Input, Output, callback, dash_table, dcc, html
from dash.exceptions import PreventUpdate
from sqlalchemy.engine import Engine
from aki_prj23_transparenzregister.utils.postgres import entities
from aki_prj23_transparenzregister.utils.postgres.connector import (
get_session,
)
if __name__ == "__main__":
session = get_session()
query_finance = session.query(
entities.AnnualFinanceStatement, entities.Company.name, entities.Company.id
).join(entities.Company)
query_company = session.query(entities.Company, entities.DistrictCourt.name).join(
entities.DistrictCourt
)
engine = session.bind
if not isinstance(engine, Engine):
raise TypeError
finance_df: pd.DataFrame = pd.read_sql(str(query_finance), engine)
company_df: pd.DataFrame = pd.read_sql(str(query_company), engine)
select_company_df = company_df[["company_id", "company_name"]]
select_company_dropdown = select_company_df.to_dict("records")
options = [
{"label": i["company_name"], "value": i["company_id"]}
for i in select_company_dropdown
]
colors = {
"light": "#edefef",
"lavender-blush": "#f3e8ee",
"ash-gray": "#bacdb0",
"cambridge-blue": "#729b79",
"paynes-gray": "#475b63",
"raisin-black": "#2e2c2f",
}
def financials_figure(
finance_df: pd.DataFrame, company: str, metric: str
) -> go.Figure:
"""Creates plotly line chart for a specific company and a metric."""
finance_df = finance_df.loc[finance_df["company_name"] == company]
# create figure
fig_line = go.Figure()
# add trace for company 1
fig_line.add_trace(
go.Scatter(
x=finance_df["annual_finance_statement_date"],
y=finance_df[metric],
name=company,
line_color=colors["raisin-black"],
marker_color=colors["raisin-black"],
)
)
# set title and labels
fig_line.update_layout(
title=metric,
xaxis_title="Jahr",
yaxis_title="in Mio.€",
plot_bgcolor=colors["light"],
)
return fig_line
tab_style = {
"borderBottom": "1px solid #d6d6d6",
"padding": "6px",
"backgroundColor": "white",
"color": colors["paynes-gray"],
"fontWeight": "bold",
}
tab_selected_style = {
"borderTop": "1px solid #d6d6d6",
"borderBottom": "1px solid #d6d6d6",
"padding": "6px",
"backgroundColor": colors["paynes-gray"],
"color": "white",
"fontWeight": "bold",
}
# TBD: get data from database instead of mock data
company = 1 # selected company id
selected_company = company_df.loc[company_df["company_id"] == company]
turnover = 123456
stock = "1,23"
company_data = {
"col1": ["Unternehmen", "Straße", "Stadt"],
"col2": [
selected_company["company_name"][0],
selected_company["company_street"][0],
str(
selected_company["company_zip_code"][0]
+ " "
+ selected_company["company_city"][0]
),
],
"col3": ["Branche", "Amtsgericht", "Gründungsjahr"],
"col4": [
selected_company["company_sector"][0],
selected_company["district_court_name"][0],
"xxx",
],
}
df_company_data = pd.DataFrame(data=company_data)
app = Dash(
__name__, external_stylesheets=[dbc.icons.BOOTSTRAP]
) # use dbc for icons
kennzahlen_layout = html.Div(
[
dcc.Graph(
figure=financials_figure(
finance_df, str(company), "annual_finance_statement_ebit"
)
)
]
)
app.layout = html.Div(
[
# title header of page
html.Div(
style={
"backgroundColor": colors["raisin-black"],
"border": "1px solid",
},
children=[
html.I(
className="bi bi-house-door-fill",
style={
"fontSize": 24,
"paddingLeft": "10px",
"color": "white",
"display": "inline-block",
"verticalAlign": "middle",
},
),
html.H1(
children="Transparenzregister für Kapitalgesellschaften",
style={
"color": "white",
"textAlign": "left",
"margin": "0",
"paddingLeft": "10px",
"paddingBottom": "20px",
"paddingTop": "20px",
"display": "inline-block",
"verticalAlign": "middle",
},
),
html.Div(
dcc.Dropdown(
id="select_company",
placeholder="Suche nach Unternehmen oder Person",
),
style={
"float": "right",
"width": "30%",
"margin": "0",
"paddingRight": "10px",
"paddingBottom": "20px",
"paddingTop": "20px",
"display": "inline-block",
"verticalAlign": "middle",
},
),
],
),
# header company name
html.Div(
style={"backgroundColor": colors["paynes-gray"], "border": "1px solid"},
children=[
html.H1(
children=selected_company["company_name"][0],
style={
"color": "white",
"fontSize": 30,
"textAlign": "left",
"margin": "0",
"paddingLeft": "20px",
"paddingBottom": "20px",
"paddingTop": "20px",
},
)
],
),
html.Div(style={"height": "20px"}),
html.Div(style={"width": "2%", "display": "inline-block"}),
# table basic company information
html.Div(
style={
"backgroundColor": colors["ash-gray"],
"border": "1px solid",
"border-radius": 10,
"width": "45%",
"height": "150px",
"display": "inline-block",
"vertical-align": "top",
},
children=[
html.H5(
children="Stammdaten",
style={
"color": colors["raisin-black"],
"fontSize": 16,
"textAlign": "center",
"margin": "0",
"paddingBottom": "10px",
"paddingTop": "10px",
},
),
dash_table.DataTable(
df_company_data.to_dict("records"),
[{"name": i, "id": i} for i in df_company_data.columns],
style_table={
"width": "80%",
"overflowX": "auto",
"marginLeft": "auto",
"marginRight": "auto",
"paddingBottom": "20px",
"color": colors["raisin-black"],
},
# hide header of table
css=[
{
"selector": "tr:first-child",
"rule": "display: none",
},
],
style_cell={"textAlign": "center"},
style_cell_conditional=[
{"if": {"column_id": c}, "fontWeight": "bold"}
for c in ["col1", "col3"]
],
),
],
),
html.Div(style={"width": "2%", "display": "inline-block"}),
html.Div(
style={
"backgroundColor": colors["ash-gray"],
"border": "1px solid",
"border-radius": 10,
"width": "15%",
"height": "150px",
"display": "inline-block",
"vertical-align": "top",
},
children=[
html.H5(
children="Stimmung",
style={
"color": colors["raisin-black"],
"fontSize": 16,
"textAlign": "center",
"margin": "0",
"paddingBottom": "10px",
"paddingTop": "10px",
},
),
],
),
html.Div(style={"width": "2%", "display": "inline-block"}),
html.Div(
style={
"backgroundColor": colors["ash-gray"],
"border": "1px solid",
"border-radius": 10,
"width": "15%",
"height": "150px",
"display": "inline-block",
"vertical-align": "top",
},
children=[
html.H5(
children="Aktienkurs",
style={
"color": colors["raisin-black"],
"fontSize": 16,
"textAlign": "center",
"margin": "0",
"paddingBottom": "10px",
"paddingTop": "10px",
},
),
html.H1(
children=stock,
style={
"color": colors["raisin-black"],
"textAlign": "center",
},
),
],
),
html.Div(style={"width": "2%", "display": "inline-block"}),
html.Div(
style={
"backgroundColor": colors["ash-gray"],
"border": "1px solid",
"border-radius": 10,
"width": "15%",
"height": "150px",
"display": "inline-block",
"vertical-align": "top",
},
children=[
html.H5(
children="Umsatz",
style={
"color": colors["raisin-black"],
"fontSize": 16,
"textAlign": "center",
"margin": "0",
"paddingBottom": "10px",
"paddingTop": "10px",
},
),
html.H1(
children=turnover,
style={
"color": colors["raisin-black"],
"textAlign": "center",
},
),
],
),
html.Div(style={"width": "2%", "display": "inline-block"}),
# ]),
html.Div(
style={
"marginTop": "20px",
"border": "1px solid",
},
children=[
dcc.Tabs(
id="tabs",
value="tab-1",
children=[
dcc.Tab(
label="Kennzahlen",
value="tab-1",
style=tab_style,
selected_style=tab_selected_style,
children=[kennzahlen_layout],
),
dcc.Tab(
label="Beteiligte Personen",
value="tab-2",
style=tab_style,
selected_style=tab_selected_style,
),
dcc.Tab(
label="Stimmung",
value="tab-3",
style=tab_style,
selected_style=tab_selected_style,
),
dcc.Tab(
label="Verflechtungen",
value="tab-4",
style=tab_style,
selected_style=tab_selected_style,
),
],
),
html.Div(id="tabs-example-content-1"),
],
),
]
)
@callback(
Output("select_company", "options"), Input("select_company", "search_value")
)
def update_options(search_value: str) -> list:
"""Update page based on selected company."""
if not search_value:
raise PreventUpdate
return [o for o in options if search_value in o["label"]]
app.run_server(debug=True)

View File

@ -1,9 +1,8 @@
"""Dash."""
import pandas as pd
from dash import Dash, dash_table
from dash import Dash, Input, Output, callback, dash_table, dcc, html
# from ..utils.postgres.connector import get_engine, init_db
from aki_prj23_transparenzregister.utils.postgres import entities
from aki_prj23_transparenzregister.utils.postgres.connector import (
get_session,
@ -16,9 +15,29 @@ if __name__ == "__main__":
companies_df: pd.DataFrame = pd.read_sql(str(query), session.bind) # type: ignore
app = Dash(__name__)
app.layout = dash_table.DataTable(
companies_df.to_dict("records"),
[{"name": i, "id": i} for i in companies_df.columns],
app.layout = html.Div(
[
html.H1(children="Company Data", style={"textAlign": "center"}),
html.Div(
[
dcc.Dropdown(
companies_df.company_name.unique(),
"Firma 1",
id="dropdown-selection",
),
]
),
html.Div(id="data_table"),
]
)
@callback(Output("data_table", "children"), Input("dropdown-selection", "value"))
def display_table(value: str) -> dash_table:
"""Output table with company stats based on dropdown value."""
dff = companies_df[companies_df.company_name == value]
return dash_table.DataTable(
data=dff.to_dict("records"),
columns=[{"id": c, "name": c} for c in companies_df.columns],
)
app.run(debug=True)

View File

@ -2,15 +2,20 @@
import enum
class RelationTypeEnum(enum.Enum):
class RelationTypeEnum(enum.IntEnum):
"""RelationTypeEnum."""
executive = "Executive"
auditor = "Auditor"
supervisory_board = "Supervisory_Board"
managing_director = "Managing_Directory"
authorized_representative = "Authorized_Representative"
final_auditor = "Final_Auditor"
EXECUTIVE = enum.auto()
AUDITOR = enum.auto()
SUPERVISORY_BOARD = enum.auto()
MANAGING_DIRECTOR = enum.auto()
AUTHORIZED_REPRESENTATIVE = enum.auto()
FINAL_AUDITOR = enum.auto()
PARTICIPATES_WITH = enum.auto()
HAS_SHARES_OF = enum.auto()
IS_SUPPLIED_BY = enum.auto()
WORKS_WITH = enum.auto()
class SentimentTypeEnum(enum.Enum):
@ -20,12 +25,3 @@ class SentimentTypeEnum(enum.Enum):
sustainability = "sustainability"
environmental_aspects = "environmental_aspects"
perception = "perception"
class RelationTypeCompanyEnum(enum.Enum):
"""RelationTypeCompanyEnum."""
participates_with = "participates_with"
has_shares_of = "has_shares_of"
is_supplied_by = "is_supplied_by"
works_with = "works_with"

View File

@ -1,11 +1,10 @@
"""Module containing connection utils for PostgreSQL DB."""
from sqlalchemy import create_engine
from sqlalchemy.engine import URL, Engine
from sqlalchemy.orm import Session, sessionmaker
from sqlalchemy.orm import Session, declarative_base, sessionmaker
from aki_prj23_transparenzregister.config.config_providers import JsonFileConfigProvider
from aki_prj23_transparenzregister.config.config_template import PostgreConnectionString
from aki_prj23_transparenzregister.utils.postgres.entities import Base
def get_engine(conn_args: PostgreConnectionString) -> Engine:
@ -34,6 +33,9 @@ def get_session() -> Session: # pragma: no cover
return session()
Base = declarative_base()
def init_db() -> None:
"""Initialize DB with all defined entities."""
config_provider = JsonFileConfigProvider("./secrets.json")

View File

@ -1,20 +1,15 @@
"""ORM entities for Prod. DB."""
import uuid
from datetime import datetime
import sqlalchemy as sa
from sqlalchemy.orm import (
declarative_base,
)
from aki_prj23_transparenzregister.utils.enumy_types import (
RelationTypeCompanyEnum,
RelationTypeEnum,
SentimentTypeEnum,
)
from aki_prj23_transparenzregister.utils.postgres.connector import Base
# # create an object *district_court* which inherits attributes from Base-class
Base = declarative_base()
class DistrictCourt(Base):
@ -22,9 +17,9 @@ class DistrictCourt(Base):
__tablename__ = "district_court"
id = sa.Column(sa.Integer, primary_key=True)
id = sa.Column(sa.Integer, primary_key=True, autoincrement=True)
city = sa.Column(sa.String(100), nullable=False)
name = sa.Column(sa.String(100), nullable=False)
name = sa.Column(sa.String(100), nullable=False, unique=True)
class Company(Base):
@ -33,32 +28,48 @@ class Company(Base):
__tablename__ = "company"
__table_args__ = (
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint("hr", "court_id"),
sa.UniqueConstraint("name", "city"),
sa.UniqueConstraint("name", "zip_code"),
sa.UniqueConstraint("name"),
)
id = sa.Column(sa.String, primary_key=True, default=uuid.uuid4, unique=True)
hr = sa.Column(sa.Integer, nullable=False)
id = sa.Column(sa.Integer, primary_key=True, autoincrement=True)
hr = sa.Column(sa.String, nullable=False)
court_id = sa.Column(
sa.Integer,
sa.ForeignKey("district_court.id"),
nullable=False,
)
name = sa.Column(sa.String(100), nullable=False)
street = sa.Column(sa.String(100), nullable=False)
zip_code = sa.Column(sa.String(5), nullable=False)
city = sa.Column(sa.String(100), nullable=False)
sector = sa.Column(sa.String(100), nullable=False)
name = sa.Column(sa.String(150), nullable=False)
street = sa.Column(sa.String(100), nullable=True)
zip_code = sa.Column(sa.String(5), nullable=True)
city = sa.Column(sa.String(100), nullable=True)
last_update = sa.Column(sa.Date, nullable=False)
sector = sa.Column(sa.String(100), nullable=True)
class Finance(Base):
"""Finance."""
class Person(Base):
"""Person."""
__tablename__ = "finance"
__tablename__ = "person"
__table_args__ = (sa.UniqueConstraint("name", "surname", "date_of_birth"),)
id = sa.Column(sa.Integer, primary_key=True)
company_id = sa.Column(sa.String, sa.ForeignKey("company.id"))
date = sa.Column(sa.DateTime(timezone=True), default=datetime.now)
name = sa.Column(sa.String(100), nullable=False)
surname = sa.Column(sa.String(100), nullable=False)
date_of_birth = sa.Column(sa.Date, nullable=False)
works_for = sa.Column(sa.String(100), nullable=True)
class AnnualFinanceStatement(Base):
"""Finance."""
__tablename__ = "annual_finance_statement"
id = sa.Column(sa.Integer, primary_key=True)
company_id = sa.Column(sa.Integer, sa.ForeignKey("company.id"))
date = sa.Column(sa.DateTime(timezone=True), nullable=False)
total_volume = sa.Column(sa.Float)
ebit = sa.Column(sa.Float)
ebitda = sa.Column(sa.Float)
@ -68,7 +79,6 @@ class Finance(Base):
debt = sa.Column(sa.Float)
return_on_equity = sa.Column(sa.Float)
capital_turnover_rate = sa.Column(sa.Float)
# company: Mapped[Company] = relationship(Company)
@ -79,7 +89,7 @@ class Sentiment(Base):
__tablename__ = "sentiment"
id = sa.Column(sa.Integer, primary_key=True)
company_id = sa.Column(sa.String, sa.ForeignKey("company.id"))
company_id = sa.Column(sa.Integer, sa.ForeignKey("company.id"))
date = sa.Column(sa.DateTime(timezone=True), default=datetime.now)
sentiment_type = sa.Column(
sa.Enum(SentimentTypeEnum),
@ -87,53 +97,48 @@ class Sentiment(Base):
)
value = sa.Column(sa.Float(), nullable=False)
source = sa.Column(sa.String(100))
# sentiment = relationship(Company)
# create person object
class Person(Base):
"""Person."""
__tablename__ = "person"
class Relation(Base):
"""A super table containing all relations."""
__tablename__ = "relation"
id = sa.Column(sa.Integer, primary_key=True)
name = sa.Column(sa.String(100), nullable=False)
surname = sa.Column(sa.String(100), nullable=False)
works_for = sa.Column(sa.String(100))
company_id = sa.Column(sa.Integer, sa.ForeignKey("company.id"))
date_from = sa.Column(sa.DateTime(timezone=True), nullable=True)
date_to = sa.Column(sa.DateTime(timezone=True), nullable=True)
relation = sa.Column(sa.Enum(RelationTypeEnum), nullable=False)
# create own relation type and person_relation object
class PersonRelation(Base):
class PersonRelation(Relation):
"""PersonRelation."""
__tablename__ = "person_relation"
id = sa.Column(sa.Integer, primary_key=True)
company_id = sa.Column(sa.String, sa.ForeignKey("company.id"))
person_id = sa.Column(sa.Integer, sa.ForeignKey(Person.id))
date_from = sa.Column(sa.DateTime(timezone=True), default=datetime.now)
date_to = sa.Column(sa.DateTime(timezone=True), default=datetime.now)
relation = sa.Column(sa.Enum(RelationTypeEnum), nullable=False)
id = sa.Column(sa.Integer, sa.ForeignKey("relation.id"), primary_key=True)
person_id = sa.Column(sa.Integer, sa.ForeignKey("person.id"))
# company = relationship("Company")
# person = relationship("Person", foreign_keys=[person_id])
# company = relationship('Company', foreign_keys=[company_hr,company_court])
__table_args__ = {"extend_existing": True}
# create own relation type and company_relation object
class CompanyRelation(Base):
class CompanyRelation(Relation):
"""CompanyRelation."""
__tablename__ = "company_relation"
id = sa.Column(sa.Integer, primary_key=True)
company1_id = sa.Column(sa.String, sa.ForeignKey("company.id"), nullable=False)
company2_id = sa.Column(sa.String, sa.ForeignKey("company.id"), nullable=False)
date_from = sa.Column(sa.DateTime(timezone=True), default=datetime.now)
date_to = sa.Column(sa.DateTime(timezone=True), default=datetime.now)
relation = sa.Column(sa.Enum(RelationTypeCompanyEnum), nullable=False)
id = sa.Column(sa.Integer, sa.ForeignKey("relation.id"), primary_key=True)
company2_id = sa.Column(sa.Integer, sa.ForeignKey("company.id"), nullable=False)
# company = relationship("Company")
__table_args__ = {"extend_existing": True}

View File

@ -1 +0,0 @@
"""Tests for config module."""

View File

@ -0,0 +1,7 @@
"""Test for the company stats dashboard."""
from aki_prj23_transparenzregister.ui import company_finance_dash
def test_import() -> None:
"""Checks if an import co company_stats_dash can be made."""
assert company_finance_dash is not None

View File

@ -1 +0,0 @@
"""Mongo utils module."""

View File

@ -1 +0,0 @@
"""Tests for utils.postgres module."""

View File

@ -18,7 +18,7 @@ def test_init_db() -> None:
with patch(
"aki_prj23_transparenzregister.utils.postgres.connector.get_engine"
) as mock_get_engine, patch(
"aki_prj23_transparenzregister.utils.postgres.entities.declarative_base"
"aki_prj23_transparenzregister.utils.postgres.connector.declarative_base"
) as mock_declarative_base, patch(
"aki_prj23_transparenzregister.utils.postgres.connector.JsonFileConfigProvider"
) as mock_provider:
@ -33,4 +33,3 @@ def test_init_db() -> None:
mock_value.get_postgre_connection_string.return_value = ""
init_db()
assert True

View File

@ -1,4 +1,4 @@
def test_import() -> None:
from aki_prj23_transparenzregister.utils.postgres import entities
assert entities is not None
assert entities