Commit Graph

737 Commits

Author SHA1 Message Date
9fa46aac29 hotfix: Resolve issue in scrape process (#415) 2023-11-20 20:09:11 +01:00
cf1c8ea508 Fixed the layout dropdown for the graph. (#393) 2023-11-16 17:43:54 +01:00
e257799f77 Fixed initial layout of homepage. (#388) 2023-11-16 17:26:02 +01:00
ce1598c42e Removed double execution of Layouting in 2 and 3d. (#385)
Prior to layouting the sping layout was allways calculated and later
overwritten. (Double execution)
2023-11-16 17:25:15 +01:00
96d216fb74 Added a web-frontend to add companies to the search queue. (#375)
This webfrontent adds a company to the search que or resets the search
que entry to be searched again.
This will allow for directed growth or an initiallisation.

---------

Co-authored-by: Tristan Nolde <tristan.nolde@yahoo.de>
2023-11-16 17:24:43 +01:00
8e6b28b311 build(deps): bump transformers from 4.35.1 to 4.35.2 (#392)
Bumps [transformers](https://github.com/huggingface/transformers) from
4.35.1 to 4.35.2.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/huggingface/transformers/releases">transformers's
releases</a>.</em></p>
<blockquote>
<h2>Patch release: v4.35.2</h2>
<p>A patch release was made for the following commit:</p>
<ul>
<li>[<code>tokenizers</code>] update tokenizers version pin <a
href="https://redirect.github.com/huggingface/transformers/issues/27494">#27494</a></li>
</ul>
<p>to fix all the issues with versioning regarding
<code>tokenizers</code> and <code>huggingface_hub</code></p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="514de24abf"><code>514de24</code></a>
Release: v4.35.2</li>
<li><a
href="7983bca630"><code>7983bca</code></a>
[<code>tokenizers</code>] update <code>tokenizers</code> version pin (<a
href="https://redirect.github.com/huggingface/transformers/issues/27494">#27494</a>)</li>
<li>See full diff in <a
href="https://github.com/huggingface/transformers/compare/v4.35.1...v4.35.2">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=transformers&package-manager=pip&previous-version=4.35.1&new-version=4.35.2)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>
2023-11-16 16:21:23 +00:00
fea130be11 build(deps): bump transformers from 4.35.1 to 4.35.2
Bumps [transformers](https://github.com/huggingface/transformers) from 4.35.1 to 4.35.2.
- [Release notes](https://github.com/huggingface/transformers/releases)
- [Commits](https://github.com/huggingface/transformers/compare/v4.35.1...v4.35.2)

---
updated-dependencies:
- dependency-name: transformers
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-11-16 16:15:05 +00:00
0d4d1324d0 Simplification of layout selection (#384)
Removed the 2d/3d toggle and added that option into the layout dropdown.

---------

Co-authored-by: KM-R <129882581+KM-R@users.noreply.github.com>
2023-11-14 21:31:30 +00:00
e7278c047e Fixed edge thickness (#370) 2023-11-14 19:32:32 +01:00
4df55f2914 build(deps): bump transformers from 4.35.0 to 4.35.1 (#389)
Bumps [transformers](https://github.com/huggingface/transformers) from
4.35.0 to 4.35.1.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/huggingface/transformers/releases">transformers's
releases</a>.</em></p>
<blockquote>
<h2>Patch release: v4.35.1</h2>
<p>A patch release was made for the following three commits:</p>
<ul>
<li>Fix FA2 import + deprecation cycle (<a
href="https://redirect.github.com/huggingface/transformers/issues/27330">#27330</a>)</li>
<li>Fix from_pt flag when loading with safetensors (<a
href="https://redirect.github.com/huggingface/transformers/issues/27394">#27394</a>)</li>
<li>Default to msgpack for safetensors (<a
href="https://redirect.github.com/huggingface/transformers/issues/27460">#27460</a>)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="10f3e7b31b"><code>10f3e7b</code></a>
Release: v4.35.1</li>
<li><a
href="0b2e2de723"><code>0b2e2de</code></a>
fix copies</li>
<li><a
href="f04737086a"><code>f047370</code></a>
Fix FA2 import + deprecation cycle (<a
href="https://redirect.github.com/huggingface/transformers/issues/27330">#27330</a>)</li>
<li><a
href="d8fffbe4a3"><code>d8fffbe</code></a>
Fix <code>from_pt</code> flag when loading with <code>safetensors</code>
(<a
href="https://redirect.github.com/huggingface/transformers/issues/27394">#27394</a>)</li>
<li><a
href="757171dfcf"><code>757171d</code></a>
Default to msgpack for safetensors (<a
href="https://redirect.github.com/huggingface/transformers/issues/27460">#27460</a>)</li>
<li>See full diff in <a
href="https://github.com/huggingface/transformers/compare/v4.35.0...v4.35.1">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=transformers&package-manager=pip&previous-version=4.35.0&new-version=4.35.1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>
2023-11-14 17:53:05 +01:00
d68e340155 build(deps): bump datetime from 5.2 to 5.3 (#390)
Bumps [datetime](https://github.com/zopefoundation/DateTime) from 5.2 to
5.3.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/zopefoundation/DateTime/blob/master/CHANGES.rst">datetime's
changelog</a>.</em></p>
<blockquote>
<h2>5.3 (2023-11-14)</h2>
<ul>
<li>
<p>Add support for Python 3.12.</p>
</li>
<li>
<p>Add preliminary support for Python 3.13a2.</p>
</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="6e2681aecf"><code>6e2681a</code></a>
Preparing release 5.3</li>
<li><a
href="28124d7778"><code>28124d7</code></a>
Add support for Python 3.12. (<a
href="https://redirect.github.com/zopefoundation/DateTime/issues/57">#57</a>)</li>
<li><a
href="5a434fd341"><code>5a434fd</code></a>
Fix hopefully temporary problem with Python 3.12 on Windows.</li>
<li><a
href="c22954a741"><code>c22954a</code></a>
Merge pull request <a
href="https://redirect.github.com/zopefoundation/DateTime/issues/55">#55</a>
from zopefoundation/config-with-zope-product-template-...</li>
<li><a
href="009cd8b809"><code>009cd8b</code></a>
- Add preliminary support for Python 3.12b4</li>
<li><a
href="1442a44b09"><code>1442a44</code></a>
- documentation and docstring cleanups</li>
<li><a
href="f4af7c61f3"><code>f4af7c6</code></a>
Back to development: 5.3</li>
<li>See full diff in <a
href="https://github.com/zopefoundation/DateTime/compare/5.2...5.3">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=datetime&package-manager=pip&previous-version=5.2&new-version=5.3)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>
2023-11-14 17:52:35 +01:00
4e63c6e907 Added rounding on Metric table to 6 digits to limit the noise (#386)
Currently to many digits are shown in the Metrics table. Rounding to 6
digits should be a bit better.
2023-11-14 17:49:48 +01:00
119a3edfca Fixed sql alchemy reset order. (#383)
There was a problem on the PG variant.
The SQLite variant worked. Now resetting properly also on PG.
2023-11-14 17:44:24 +01:00
7c1fac28fd Fixed tqdm typos. (#382) 2023-11-14 17:42:38 +01:00
62355a3ec2 build(deps): bump datetime from 5.2 to 5.3
Bumps [datetime](https://github.com/zopefoundation/DateTime) from 5.2 to 5.3.
- [Changelog](https://github.com/zopefoundation/DateTime/blob/master/CHANGES.rst)
- [Commits](https://github.com/zopefoundation/DateTime/compare/5.2...5.3)

---
updated-dependencies:
- dependency-name: datetime
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-11-14 15:39:53 +00:00
b56520b7f0 build(deps): bump transformers from 4.35.0 to 4.35.1
Bumps [transformers](https://github.com/huggingface/transformers) from 4.35.0 to 4.35.1.
- [Release notes](https://github.com/huggingface/transformers/releases)
- [Commits](https://github.com/huggingface/transformers/compare/v4.35.0...v4.35.1)

---
updated-dependencies:
- dependency-name: transformers
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-11-14 15:38:25 +00:00
dd511af3a5 Adapt ingest schedule (#381)
Adapt ingest schedule
2023-11-13 21:33:43 +01:00
0c4a2d7a74 Moved transparenzregister url prefix to docker compose. (#380) 2023-11-13 20:08:50 +01:00
c1faa2cba6 feat(deployment): Enable setting of schedule for ingest container using ENV var 2023-11-13 20:08:26 +01:00
92489e7ef6 checkpoint: Set schedule interval via ENV variable 2023-11-13 19:54:29 +01:00
2da3e6cea0 Removed the unused colorscale legend and other unused code fragements (#371) 2023-11-13 19:28:57 +01:00
31221a1055 Fixed spagetti connection by adding breaks. (#366) 2023-11-13 19:27:03 +01:00
d0677287b6 Added a filter for financial reports. (#372)
Finanical reports are now filtered before beeing added to the SQL
database to only added knwon keys.
Some matching is also done.
The most importend missing reports are printed to be implemented later
on.
Rapidfuzz could be used.
2023-11-13 18:52:12 +01:00
da340d5ec5 build(deps): bump dash-cytoscape from 0.2.0 to 0.3.0 (#378)
Bumps [dash-cytoscape](https://github.com/plotly/dash-cytoscape) from
0.2.0 to 0.3.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/plotly/dash-cytoscape/releases">dash-cytoscape's
releases</a>.</em></p>
<blockquote>
<h2>v0.3.0 - Add support for Julia, published to CRAN, changes in
documentations</h2>
<h3>Added</h3>
<ul>
<li>Contributed initial build of Julia package.</li>
<li>R package now includes an example application for
<code>cytoCytoscape</code>.</li>
</ul>
<h3>Changed</h3>
<ul>
<li>Dash has been upgraded to 1.* in requirements.txt and
tests/requirements.txt (<a
href="https://redirect.github.com/plotly/dash-cytoscape/issues/123">#123</a>)</li>
<li>React/react-dom have been upgraded to 16.14+ (<a
href="https://redirect.github.com/plotly/dash-cytoscape/issues/117">#117</a>)</li>
<li>Docgen upgraded to 5.3.0 (<a
href="https://redirect.github.com/plotly/dash-cytoscape/issues/117">#117</a>)</li>
<li>Improved prop typing (<a
href="https://redirect.github.com/plotly/dash-cytoscape/issues/117">#117</a>)</li>
</ul>
<h3>Fixed</h3>
<ul>
<li>Various security fixes</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li>See full diff in <a
href="https://github.com/plotly/dash-cytoscape/compare/v0.2.0...v0.3.0">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=dash-cytoscape&package-manager=pip&previous-version=0.2.0&new-version=0.3.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>
2023-11-13 18:48:50 +01:00
b3f9d177f3 build(deps): bump dash-cytoscape from 0.2.0 to 0.3.0
Bumps [dash-cytoscape](https://github.com/plotly/dash-cytoscape) from 0.2.0 to 0.3.0.
- [Release notes](https://github.com/plotly/dash-cytoscape/releases)
- [Commits](https://github.com/plotly/dash-cytoscape/compare/v0.2.0...v0.3.0)

---
updated-dependencies:
- dependency-name: dash-cytoscape
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-11-13 15:54:51 +00:00
41bee38bf7 Print the actor in the maintenance action (#377) 2023-11-13 13:07:45 +01:00
af8a907cf9 Stop table reset of better persistent tables. (#373) 2023-11-12 14:27:44 +01:00
d66e4e2b67 Added Hausarbeit 2023-11-12 14:22:56 +01:00
19a4460d90 Added a small stability fix. (#374)
The current code has problems with an empty db.
2023-11-12 14:14:15 +01:00
24c55c68b7 Removed docstring ruins. (#367) 2023-11-12 13:58:00 +01:00
3b2f9b98f2 Update pre-commit hooks (#376)
Update versions of pre-commit hooks to latest version.

Co-authored-by: philipp-horstenkamp <philipp-horstenkamp@users.noreply.github.com>
2023-11-12 10:12:17 +01:00
bbc15bc7a2 Feat/116 scheduling tools (#358)
Init `ingestion` container with `fetch_news` target to retrieve latest
news articles from Tagesschau and Handelsblatt twice a day.

Integration of the `find_missing_companies.py` will follow once this is
merged.
2023-11-11 14:34:51 +01:00
05ea0fbb33 refactor: Include logger.catch with reraise 2023-11-11 14:30:00 +01:00
5dcf8ecf55 build: Dockerize apps/fetch_news.py as ingestor 2023-11-11 14:30:00 +01:00
170056bf58 test: Cover apps/fetch_news.py with unit tests 2023-11-11 14:30:00 +01:00
ac6ca3547b test: Add unit test for news api wrapper 2023-11-11 14:30:00 +01:00
ae41cf61bc checkpoint: Resolve error in handelsblatt text fetch 2023-11-11 14:30:00 +01:00
a428eb4432 checkpoint: Init news extraction components and main app 2023-11-11 14:30:00 +01:00
905021af14 Experimental caching (#285)
Added some caching decoraterors to speedup page delivery.
2023-11-11 14:28:25 +01:00
066800123d Created pipeline to run ner sentiment and sql ingest (#314)
Created a dataprocessing pipline that enhances the raw mined data with
Organsiation extractions and sentiment analysis prio to moving the data
to the sql db.
The transfer of matched data is done afterword.

---------

Co-authored-by: SeZett <zeleny.sebastian@fh-swf.de>
2023-11-11 13:28:12 +00:00
a6d486209a Introduce extended_financial_data code (#357)
Introducing the previously developed method to fetch the financial data
via table parsing (aka "data lake like solution") in a non-destructive
manner by defaulting to the current RegEx-based behaviour.
2023-11-11 14:10:20 +01:00
e5b61bc19c Added multi relation dropdowns to dashbord (#363)
This change allows for a more complete combination of relation
combinations to be filtered.
2023-11-11 13:47:46 +01:00
ad8f5d0fb1 Added github actions automerge for pre-commit updates. (#362) 2023-11-11 13:30:16 +01:00
b0bcdc6fe1 refactor: PR feedback implemented 2023-11-11 11:18:23 +01:00
834f93a26e Update src/aki_prj23_transparenzregister/utils/data_extraction/bundesanzeiger.py
Co-authored-by: Philipp Horstenkamp <philipp@horstenkamp.de>
2023-11-11 11:03:36 +01:00
e1b8397f9e feat: Introduce switch for different financial extraction routines 2023-11-11 11:03:36 +01:00
9edf5b1dce test: Increase coverage for multi-column headers 2023-11-11 11:03:36 +01:00
3ba8c0abea refactor: Remove debugging statements 2023-11-11 11:03:36 +01:00
3b1f0425cf deps: Adding html5lib for table parsing via Pandas 2023-11-11 11:03:36 +01:00
801f945c59 temp: Print exception for test debuggin 2023-11-11 11:01:17 +01:00