Philipp/aki_prj23_transparenzregister
1
0
Fork 0
mirror of https://github.com/fhswf/aki_prj23_transparenzregister.git synced 2026-02-13 22:17:38 +01:00
Code Issues Packages Projects Releases Wiki Activity
588 Commits 7 Branches 0 Tags
b1ca35c53fca4dbe6c3310febe47edab49a20b6b
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Philipp 319ca3b8ad Added redirection callbacks to the person and the company page. (#423) 2023-12-04 19:11:22 +01:00
Philipp 5969929f79 Potential fix. (#427) 
I have looked up the error and found an potential fix.
I could not reproduce the error afterwords so it should work.
 2023-11-30 17:22:31 +01:00
Philipp 9c60d01985 Simplifications (#416) 2023-11-30 17:21:19 +01:00
Philipp 6890562a18 Simplified some functions. (#414) 2023-11-26 14:01:12 +00:00
Philipp 0672773551 Ner simplifications (#419) 2023-11-25 16:53:10 +01:00
Philipp 65c41bb20c Added translations to the search que page and the about page. (#407) 2023-11-22 21:48:35 +01:00
Philipp 762d3f9091 Add flask-compress (#409) 
Added flask compression for compressed responsed to limit the download
time of plots.
I switch this on by default. It could also be crontoled via
DASH_COMPRESS as an env variable.
 2023-11-20 21:12:07 +01:00
Philipp 36f2d653b1 Added links (#406) 
Added links into the plot.
When clicking on the plot on the main page a redirect is implemente the
the matching detailpage.
 2023-11-20 20:06:51 +00:00
Philipp cf1c8ea508 Fixed the layout dropdown for the graph. (#393) 2023-11-16 17:43:54 +01:00
Philipp e257799f77 Fixed initial layout of homepage. (#388) 2023-11-16 17:26:02 +01:00
Philipp ce1598c42e Removed double execution of Layouting in 2 and 3d. (#385) 
Prior to layouting the sping layout was allways calculated and later
overwritten. (Double execution)
 2023-11-16 17:25:15 +01:00
PhilippandTristan Nolde 96d216fb74 Added a web-frontend to add companies to the search queue. (#375) 
This webfrontent adds a company to the search que or resets the search
que entry to be searched again.
This will allow for directed growth or an initiallisation.

---------

Co-authored-by: Tristan Nolde <tristan.nolde@yahoo.de>
 2023-11-16 17:24:43 +01:00
PhilippandKM-R 0d4d1324d0 Simplification of layout selection (#384) 
Removed the 2d/3d toggle and added that option into the layout dropdown.

---------

Co-authored-by: KM-R <129882581+KM-R@users.noreply.github.com>
 2023-11-14 21:31:30 +00:00
Philipp e7278c047e Fixed edge thickness (#370) 2023-11-14 19:32:32 +01:00
Philipp 4e63c6e907 Added rounding on Metric table to 6 digits to limit the noise (#386) 
Currently to many digits are shown in the Metrics table. Rounding to 6
digits should be a bit better.
 2023-11-14 17:49:48 +01:00
Philipp 119a3edfca Fixed sql alchemy reset order. (#383) 
There was a problem on the PG variant.
The SQLite variant worked. Now resetting properly also on PG.
 2023-11-14 17:44:24 +01:00
Philipp 7c1fac28fd Fixed tqdm typos. (#382) 2023-11-14 17:42:38 +01:00
Philipp 0c4a2d7a74 Moved transparenzregister url prefix to docker compose. (#380) 2023-11-13 20:08:50 +01:00
Philipp 2da3e6cea0 Removed the unused colorscale legend and other unused code fragements (#371) 2023-11-13 19:28:57 +01:00
Philipp 31221a1055 Fixed spagetti connection by adding breaks. (#366) 2023-11-13 19:27:03 +01:00
Philipp d0677287b6 Added a filter for financial reports. (#372) 
Finanical reports are now filtered before beeing added to the SQL
database to only added knwon keys.
Some matching is also done.
The most importend missing reports are printed to be implemented later
on.
Rapidfuzz could be used.
 2023-11-13 18:52:12 +01:00
Philipp 41bee38bf7 Print the actor in the maintenance action (#377) 2023-11-13 13:07:45 +01:00
Philipp af8a907cf9 Stop table reset of better persistent tables. (#373) 2023-11-12 14:27:44 +01:00
Philipp 19a4460d90 Added a small stability fix. (#374) 
The current code has problems with an empty db.
 2023-11-12 14:14:15 +01:00
Philipp 24c55c68b7 Removed docstring ruins. (#367) 2023-11-12 13:58:00 +01:00
Philipp 905021af14 Experimental caching (#285) 
Added some caching decoraterors to speedup page delivery.
 2023-11-11 14:28:25 +01:00
PhilippandSeZett 066800123d Created pipeline to run ner sentiment and sql ingest (#314) 
Created a dataprocessing pipline that enhances the raw mined data with
Organsiation extractions and sentiment analysis prio to moving the data
to the sql db.
The transfer of matched data is done afterword.

---------

Co-authored-by: SeZett <zeleny.sebastian@fh-swf.de>
 2023-11-11 13:28:12 +00:00
Philipp e5b61bc19c Added multi relation dropdowns to dashbord (#363) 
This change allows for a more complete combination of relation
combinations to be filtered.
 2023-11-11 13:47:46 +01:00
Philipp ad8f5d0fb1 Added github actions automerge for pre-commit updates. (#362) 2023-11-11 13:30:16 +01:00
Philipp 4716eb9fe2 Added a frist rollback function on a pending Rollback error. (#346) 
Possible fix for #341.
Rolls back stuck SQL sessions.
This only helps against the symptomps not the cause.
 2023-11-09 19:48:26 +00:00
Philipp 97fc7f01b5 Added some redirects to home page. (#348) 2023-11-09 19:06:36 +01:00
Philipp 3ce49c9e19 Added requirements.txt to gitignore (#347) 2023-11-09 18:38:42 +01:00
Philipp fe02b0f6c5 Added an error handling when the sql session crashes in process on network connection lost. (#352) 2023-11-09 18:38:04 +01:00
Philipp 12f40af538 Added pagination, sorting and filtering for the table of related persons. (#333) 
Added pagination, sorting and filtering for the table of related
persons.
 2023-11-07 22:19:36 +01:00
Philipp 16b975a0f4 Hide the plotly logo in graph objects (#339) 2023-11-07 22:16:27 +01:00
Philipp 97b05bacf4 Added a custom favicon. (#338) 2023-11-07 22:15:08 +01:00
Philipp aec7414e69 Removed the background color of plotly plot. (#334) 
Removed the background color of the plot to fully use the plotly_white
layout.
 2023-11-07 15:55:50 +01:00
Philipp d79bc0f267 Added poetry install to the pre-commit hooks. (#332) 
Limiting most pre-commit hooks to the pre-commit stage.
 2023-11-06 21:58:33 +01:00
Philipp 4323522b96 Added poetry install to the pre-commit hooks. (#329) 2023-11-06 20:02:29 +01:00
Philipp 2fddc9149a Reduced the number of times the dependabot auto merge is triggered. (#318) 2023-11-03 21:25:55 +01:00
Philipp ad36c68993 Moved the AI tests into the AI folder. (#315) 2023-11-03 13:45:24 +01:00
Philipp a638949644 Added a types package and upgraded poetry. (#313) 2023-11-02 22:13:32 +01:00
Philipp d40746c69b Used black on jupyter. (#312) 2023-11-02 20:52:46 +00:00
Philipp 8d9981d967 Moved AI files in the AI module. (#308) 2023-11-02 20:30:04 +01:00
Philipp 389810c9bc Update and rename dependabot-approve.yaml to dependabot-auto-merge.yaml 2023-11-01 18:21:56 +01:00
Philipp 3c2394d2d5 Sets auto merge for PRs opened by dependabot to true. (#293) 
Sets auto merge for PRs opened by dependabot to true.
Review is still needed.
 2023-10-30 19:20:54 +01:00
Philipp 4362463f3c Added types-six to the linting packages and unlocked all the linter versions. (#286) 
Added the from mypy demanded types-six package as a lint dependency.
This should allow for a faster execution.
 2023-10-29 20:40:57 +01:00
Philipp f72d606d18 Added base-path support in URL generating features (#288) 
Add the basepath dash url to the path generation for dynamicly generated
links.
 2023-10-29 20:40:40 +01:00
Philipp 6526549bdd Reworked the about page for a nicer version of the git hash (#283) 
The git hash looks currently like this:

`Build from GiT Revision: ('d3189f78c47d812f447739f97876ced01227e1fd',)`

After this change it should look like this:
`Build from GiT Revision: d3189f78c47d812f447739f97876ced01227e1fd`
 2023-10-29 12:56:08 +01:00
Philipp fca53c5e0c Dash security-fix against DoS. (#284) 
Dash has currently a security vulnerability.
This updated should fix this.
Dependabot is currently not allowed to upgrade main-version.
 2023-10-29 12:55:28 +01:00
Philipp 0e388eb927 Using actor to identify dependabot and skipp some actions. (#273) 2023-10-26 19:29:25 +02:00
Philipp e7a778e4cb Changed the plotly default theme (#269) 
I think the current blueish plotly default theme bits itself quit beadly
with @KM-R s great work.
I propose to set the deault plotly template to either:

- plotly_white
- ggplot2
- seaborn

Simply change the constant i have set will change this.
Heren you can find the catalog of themes:
https://plotly.com/python/templates/#disable-default-theming

I would not suggest to do more than changing the default layout.
 2023-10-26 19:22:44 +02:00
Philipp 7953ba9291 Mixed typo fixes (#270) 2023-10-26 19:06:45 +02:00
Philipp ab97d77d2d Made pip-audit pass on every build. (#272) 
Since dependabot is activated this can be reduced as an reporting tool.
 2023-10-26 16:50:24 +00:00
Philipp 4c56431bff Update dependabot.yml 2023-10-26 18:19:20 +02:00
Philipp 896136dcee Added an about page (#251) 
This page was added since it is sometimes difficult to say which version
was deployed on an server. This should allow an easy lookup on the
server and make it comparable with what is expected.
 2023-10-26 17:32:17 +02:00
Philipp 6d77867eb5 Using the python action and the inbuild poetry cache (#261) 
I have found out that the python setup action supports the use of the
poetry cache which increases the build speed.
 2023-10-24 21:49:09 +02:00
Philipp 1eb972b7ff Adds the transfer of sentiments into the sql db (#253) 
Transfers the sentimenes from the mongodb int the sql db.
 2023-10-24 17:50:40 +02:00
PhilippSchilder-Reiskephilipp-horstenkamp
 		b250b67343 Auto maintance push (#258) 
Added that the maintance action allso runs on changes of itself.

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: philipp-horstenkamp <philipp-horstenkamp@users.noreply.github.com>
 2023-10-23 17:50:51 +00:00
Philipp 77a0faa257 Reduced build time (#260) 
The building and pushing of artifacts takes to long. With those changes
build experimental images aren't pushed anymore.
This saves build time and disc space.
 2023-10-23 19:43:53 +02:00
Philipp f0d763cc5b Fixing the pre-commit upgrade (#256) 
The action allso executes over all files. This is ok when using
pre-conmit.ci. We only execute file fixes once.
 2023-10-23 17:15:39 +02:00
Philipp ad7aa3e53b Filled Docker Labels and ENVs to identify the git revision it was build from (#250) 
Added labels and env vars to identify the git revion a container was
build from to have a more transparent build flow of untaged images.
 2023-10-21 18:13:03 +02:00
Philipp 88f0673470 Slimmed down the env vars access and added logging in case the password protection was disabled. (#252) 
Slimmed down the env vars access and added logging in case the password
protection was disabled.
 2023-10-21 17:57:56 +02:00
Philipp 55ebb4c17d Typo fixes (#249) 2023-10-21 10:58:54 +02:00
Philipp 640faeccec Adding auth back in. (#248) 2023-10-20 14:56:54 +00:00
Philipp 86daa9a781 Renamed a few jobs (#246) 2023-10-19 19:43:55 +02:00
Philipp 2d80298aa2 Dependabot: Ignoring all major updates. (#241) 2023-10-19 19:24:58 +02:00
Philipp 36a0bab6ff Add relations from finanical reports to SQL (#216) 2023-10-19 19:21:33 +02:00
Philipp a358ebe9c1 Dependabot/coverage (#240) 2023-10-19 18:47:32 +02:00
Philipp b92be60789 Automaticlly updating pre-commits (#220) 2023-10-19 18:44:13 +02:00
Philipp 0441a3482b Create dependabot.yml (#227) 2023-10-19 18:05:36 +02:00
Philipp fdba24ef11 Updates including security-updates (#226) 2023-10-19 17:59:15 +02:00
Philipp c0d42a22d7 General updates & linting (#218) 2023-10-14 17:43:45 +02:00
Philipp 6365e252b9 Added location to person (#185) 2023-10-14 15:27:19 +00:00
Philipp 411f1053dd Set home title (#217) 2023-10-14 17:16:54 +02:00
Philipp f8c111d7e2 Resolve mismatch between staging and prod db data for financials (#211) 
SQL Creation is now done dynamicly by the definition of the enumeration
type.
 2023-10-14 17:16:14 +02:00
Philipp c8d3c7395b Update docker-compose.yml (#210) 2023-10-12 18:09:52 +02:00
Philipp 68346aff15 Removed error suppression from sql data transformation. (#205) 2023-10-09 17:58:20 +02:00
Philipp 63325e7faa Add constraints to the SQL entities (#186) 2023-10-06 18:48:58 +02:00
Philipp ea9c777217 Enabeling data processing in docker container. (#195) 2023-10-06 18:36:29 +02:00
Philipp b1ca268a62 SQL fixes after new mongo ingest (#199) 2023-10-06 18:22:19 +02:00
Philipp 8bb27f5195 Set the DASH_URL_BASE_PATHNAME in the docker container to be used as a default (#192) 
Set the DASH_URL_BASE_PATHNAME in the docker container to be used as a
prefix.

Added a Dash Title
 2023-10-06 18:13:25 +02:00
Philipp ab96c5a01a Switched two docker container off. 2023-10-05 22:16:45 +02:00
Philipp 09c36960e3 Add an list of missing relation partners to be searched (#171) 
- [x] Add a new table
- [x] Add a field to the table that can register if the company was
already queried
- [x] Add a field to the table that counts how many times a relation
partner was missing
- [x] Add a function that restets the counter

Also:
- Reworked the get_company function to use the location dict as kwargs
 2023-10-05 19:57:30 +02:00
Philipp c6f2c7467c Rework the transfer of company data to fit the new data in the mongodb (#188) 
This adds the additional company data as proposed to the sql db.

- [x] @TrisNol Is everything included or did I miss a feature. Relations
are in another issue.
- [x] @KM-R New DB features for the Dashbord for your review.
 2023-10-05 19:47:46 +02:00
Philipp ea89a64a80 Linter update to ruff==0.0.292 (#191) 2023-10-04 20:04:59 +02:00
Philipp 41f2c9f995 Executing black over all jupyter notebook (#190) 
Reverting black for the jupyter notebooks gets old. Can we just run
black over all of them?
 2023-10-04 20:03:47 +02:00
Philipp 030ad00c7d Testing speedup with in memory SQLite (#189) 
If no SQLite File is written and deleted testing is MUCH faster.
 2023-10-04 19:36:57 +02:00
PhilippandKM-R 01b4ce00c1 Spellchecking with PyCharm (#133) 
Co-authored-by: KM-R <129882581+KM-R@users.noreply.github.com>
 2023-10-02 20:47:42 +02:00
Philipp d2d4a436f8 Add a cli interface to choose a configuration (#163) 
- [x] add a cli to the webserver to take env variables into account 
- [x] add a cli to the data processing that takes enviromental variable
as a valid source into account
- [x] rework the cli for the reset sql command
- [x] rework the cli for the copying of sql data from one db to another
 2023-10-02 20:31:42 +02:00
Philipp 2abe12f027 Add a function to convert DM to EUR (#168) 
The function is ment to transform the captial dict into a format that
can be added as a kwarg (**norm_capital(capital_dict) to the company
entities.
This PR only contains the function itself.
 2023-10-02 19:46:17 +02:00
Philipp 05472cc16a Added longitude/latitude and positional accuracy to the company data (#180) 2023-10-02 17:18:04 +02:00
Philipp c96462532b Added a pre-commit hook that disallows committing to main. (#181) 
Since I committed to main again I added an pre-commit hook that prevents
that. Since pushing to main is disallowed this should prevent someone to
be stuck with a mess.
 2023-10-01 21:40:39 +02:00
Philipp b46a52e019 Added a weekly counter similar to all other weeklies to weekly 10. (#165) 2023-10-01 15:08:45 +02:00
Philipp ebf30da778 Removed some superfluous tools from the docker image (#157) 2023-09-28 19:06:58 +02:00
Philipp 4d99601fc6 Optional docs Feature: Changelog (#155) 
Added the ability to generate a changelog of the main branch in the
Documenation.
 2023-09-28 19:05:05 +02:00
PhilippKM-RTristan Nolde
 		552e1bdf96 Added some metadata to the package (#144) 
Filled out the mainter section.
Added a new description.
Added the path to the repo to the package.
Added the path to the documentation.

Co-authored-by: KM-R <129882581+KM-R@users.noreply.github.com>
Co-authored-by: Tristan Nolde <tristan.nolde@yahoo.de>
 2023-09-28 17:17:01 +02:00
Philipp 7d6bb8b6f5 Optional: Add auto-walrus (#158) 
To learn the use of the Walrus opperator this hook integrates it in your
code if it finds a reason for it.
This branch is a suggestion of mine to force the use of the walrus
operaor as an autofix to keep it in mind when programming.
This should not increase the workload for us. It just lets us confront a
new programming tool. It also isn't a rull to be follwed. Just a hook
that looks at your code with an autofix.
 2023-09-28 17:13:42 +02:00
Philipp f54c140d98 Update after rebasing 2023-09-25 20:39:13 +02:00
Philipp 54dd574fb8 Added the ability to include drawio diagrams into the sphinx docs. (#107) 2023-09-25 20:37:59 +02:00