716 Commits

Author SHA1 Message Date
31221a1055
Fixed spagetti connection by adding breaks. (#366) 2023-11-13 19:27:03 +01:00
d0677287b6
Added a filter for financial reports. (#372)
Finanical reports are now filtered before beeing added to the SQL
database to only added knwon keys.
Some matching is also done.
The most importend missing reports are printed to be implemented later
on.
Rapidfuzz could be used.
2023-11-13 18:52:12 +01:00
github-actions[bot]
da340d5ec5
build(deps): bump dash-cytoscape from 0.2.0 to 0.3.0 (#378)
Bumps [dash-cytoscape](https://github.com/plotly/dash-cytoscape) from
0.2.0 to 0.3.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/plotly/dash-cytoscape/releases">dash-cytoscape's
releases</a>.</em></p>
<blockquote>
<h2>v0.3.0 - Add support for Julia, published to CRAN, changes in
documentations</h2>
<h3>Added</h3>
<ul>
<li>Contributed initial build of Julia package.</li>
<li>R package now includes an example application for
<code>cytoCytoscape</code>.</li>
</ul>
<h3>Changed</h3>
<ul>
<li>Dash has been upgraded to 1.* in requirements.txt and
tests/requirements.txt (<a
href="https://redirect.github.com/plotly/dash-cytoscape/issues/123">#123</a>)</li>
<li>React/react-dom have been upgraded to 16.14+ (<a
href="https://redirect.github.com/plotly/dash-cytoscape/issues/117">#117</a>)</li>
<li>Docgen upgraded to 5.3.0 (<a
href="https://redirect.github.com/plotly/dash-cytoscape/issues/117">#117</a>)</li>
<li>Improved prop typing (<a
href="https://redirect.github.com/plotly/dash-cytoscape/issues/117">#117</a>)</li>
</ul>
<h3>Fixed</h3>
<ul>
<li>Various security fixes</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li>See full diff in <a
href="https://github.com/plotly/dash-cytoscape/compare/v0.2.0...v0.3.0">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=dash-cytoscape&package-manager=pip&previous-version=0.2.0&new-version=0.3.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>
2023-11-13 18:48:50 +01:00
dependabot[bot]
b3f9d177f3
build(deps): bump dash-cytoscape from 0.2.0 to 0.3.0
Bumps [dash-cytoscape](https://github.com/plotly/dash-cytoscape) from 0.2.0 to 0.3.0.
- [Release notes](https://github.com/plotly/dash-cytoscape/releases)
- [Commits](https://github.com/plotly/dash-cytoscape/compare/v0.2.0...v0.3.0)

---
updated-dependencies:
- dependency-name: dash-cytoscape
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-11-13 15:54:51 +00:00
41bee38bf7
Print the actor in the maintenance action (#377) 2023-11-13 13:07:45 +01:00
af8a907cf9
Stop table reset of better persistent tables. (#373) 2023-11-12 14:27:44 +01:00
Tim Ronneburg
d66e4e2b67
Added Hausarbeit 2023-11-12 14:22:56 +01:00
19a4460d90
Added a small stability fix. (#374)
The current code has problems with an empty db.
2023-11-12 14:14:15 +01:00
24c55c68b7
Removed docstring ruins. (#367) 2023-11-12 13:58:00 +01:00
github-actions[bot]
3b2f9b98f2
Update pre-commit hooks (#376)
Update versions of pre-commit hooks to latest version.

Co-authored-by: philipp-horstenkamp <philipp-horstenkamp@users.noreply.github.com>
2023-11-12 10:12:17 +01:00
Tristan Nolde
bbc15bc7a2
Feat/116 scheduling tools (#358)
Init `ingestion` container with `fetch_news` target to retrieve latest
news articles from Tagesschau and Handelsblatt twice a day.

Integration of the `find_missing_companies.py` will follow once this is
merged.
2023-11-11 14:34:51 +01:00
TrisNol
05ea0fbb33 refactor: Include logger.catch with reraise 2023-11-11 14:30:00 +01:00
TrisNol
5dcf8ecf55 build: Dockerize apps/fetch_news.py as ingestor 2023-11-11 14:30:00 +01:00
TrisNol
170056bf58 test: Cover apps/fetch_news.py with unit tests 2023-11-11 14:30:00 +01:00
TrisNol
ac6ca3547b test: Add unit test for news api wrapper 2023-11-11 14:30:00 +01:00
TrisNol
ae41cf61bc checkpoint: Resolve error in handelsblatt text fetch 2023-11-11 14:30:00 +01:00
TrisNol
a428eb4432 checkpoint: Init news extraction components and main app 2023-11-11 14:30:00 +01:00
905021af14
Experimental caching (#285)
Added some caching decoraterors to speedup page delivery.
2023-11-11 14:28:25 +01:00
066800123d
Created pipeline to run ner sentiment and sql ingest (#314)
Created a dataprocessing pipline that enhances the raw mined data with
Organsiation extractions and sentiment analysis prio to moving the data
to the sql db.
The transfer of matched data is done afterword.

---------

Co-authored-by: SeZett <zeleny.sebastian@fh-swf.de>
2023-11-11 13:28:12 +00:00
Tristan Nolde
a6d486209a
Introduce extended_financial_data code (#357)
Introducing the previously developed method to fetch the financial data
via table parsing (aka "data lake like solution") in a non-destructive
manner by defaulting to the current RegEx-based behaviour.
2023-11-11 14:10:20 +01:00
e5b61bc19c
Added multi relation dropdowns to dashbord (#363)
This change allows for a more complete combination of relation
combinations to be filtered.
2023-11-11 13:47:46 +01:00
ad8f5d0fb1
Added github actions automerge for pre-commit updates. (#362) 2023-11-11 13:30:16 +01:00
TrisNol
b0bcdc6fe1 refactor: PR feedback implemented 2023-11-11 11:18:23 +01:00
Tristan Nolde
834f93a26e Update src/aki_prj23_transparenzregister/utils/data_extraction/bundesanzeiger.py
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de>
2023-11-11 11:03:36 +01:00
TrisNol
e1b8397f9e feat: Introduce switch for different financial extraction routines 2023-11-11 11:03:36 +01:00
TrisNol
9edf5b1dce test: Increase coverage for multi-column headers 2023-11-11 11:03:36 +01:00
TrisNol
3ba8c0abea refactor: Remove debugging statements 2023-11-11 11:03:36 +01:00
TrisNol
3b1f0425cf deps: Adding html5lib for table parsing via Pandas 2023-11-11 11:03:36 +01:00
TrisNol
801f945c59 temp: Print exception for test debuggin 2023-11-11 11:01:17 +01:00
Tristan Nolde
c19697c7f8 Update src/aki_prj23_transparenzregister/utils/data_extraction/bundesanzeiger.py
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de>
2023-11-11 11:01:17 +01:00
Tristan Nolde
fe7690620a Update src/aki_prj23_transparenzregister/utils/data_extraction/bundesanzeiger.py
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de>
2023-11-11 11:01:17 +01:00
TrisNol
fecf42d75a test: Unit test new KPI extraction 2023-11-11 11:01:17 +01:00
TrisNol
f8a0d58314 feat(data-extraction): Provide KPI table analysis in bundesanzeiger wrapper 2023-11-11 11:01:17 +01:00
TrisNol
815e08a8f1 checkpoint: Transform values to € and normalize column names 2023-11-11 11:01:17 +01:00
TrisNol
ec11ae13aa checkpoint: Parse table into dict of financial data 2023-11-11 11:01:17 +01:00
TrisNol
972fcd155e checkpoint: Normalize HTML tables fetched from Bundesanzeiger 2023-11-11 11:01:17 +01:00
Tristan Nolde
8781d746e7
hotfix: Add missing networkx dependency (#361)
Depyloment on Jupiter is currently broken due to missing `networkx`
dependency:

![image](https://github.com/fhswf/aki_prj23_transparenzregister/assets/56017812/8601fb23-e16b-4986-a52a-36a02c6bea29)

Should be fixed by the changes included
2023-11-10 22:52:18 +01:00
TrisNol
c333ad70c5 hotfix: Add missing networkx dependency 2023-11-10 21:47:34 +01:00
Tim Ronneburg
247719c76f
Feature/visualize verflechtungen (#324) 2023-11-10 19:33:19 +01:00
TrisNol
da72c3d0a8 lint: Format company_elements.py 2023-11-10 19:21:32 +01:00
TrisNol
a1d8e942a9 test: Adapt home.py to run unit tests 2023-11-10 19:20:49 +01:00
Tim
fdbb6b5fd4 Added Graph to Company page again 2023-11-10 18:57:11 +01:00
Tim
e5769b3c25 Added Tests
Co-authored-by: Tristan Nolde <TrisNol@users.noreply.github.com>
2023-11-10 18:56:51 +01:00
Tim
410b690873 Added test 2023-11-10 18:56:51 +01:00
Tim
41af7e2d18 Added test behaviour 2023-11-10 18:56:51 +01:00
Tim
4d2ca3b3e7 Refactored Session handling for Network analysis 2023-11-10 18:56:51 +01:00
Tim
ac46348cc8 Added Dash DAQ 2023-11-10 18:55:13 +01:00
Tim
c38460c740 fixed mypy errors 2023-11-10 18:54:30 +01:00
Tim
f38728450d now ruff confirm 2023-11-10 18:53:47 +01:00
Tim
f2ac0eda91 Added Realtion_count MEthod 2023-11-10 18:53:47 +01:00