Philipp/aki_prj23_transparenzregister
1
0
Fork 0
mirror of https://github.com/fhswf/aki_prj23_transparenzregister.git synced 2025-06-21 23:53:55 +02:00
Code Issues Packages Projects Releases Wiki Activity

Created pipeline to run ner sentiment and sql ingest (#314)

Browse Source 
Created a dataprocessing pipline that enhances the raw mined data with
Organsiation extractions and sentiment analysis prio to moving the data
to the sql db.
The transfer of matched data is done afterword.

---------

Co-authored-by: SeZett <zeleny.sebastian@fh-swf.de>
This commit is contained in:
Philipp Horstenkamp
2023-11-11 14:28:12 +01:00
committed by GitHub
parent a6d486209a
commit 066800123d
12 changed files with 206 additions and 132 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
tests/utils/transfer_news_test.py
View File

@ -130,7 +130,7 @@ def test_transfer_news_to_sql(full_db: Session, monkeypatch: MonkeyPatch) -> Non
"aki_prj23_transparenzregister.utils.transfer_news.get_all_news",
lambda _: NEWS_TEXTS,
)
transfer_news.transfer_news_to_sql(None, full_db) # type: ignore
transfer_news._transfer_news_to_sql(None, full_db) # type: ignore
articles = pd.read_sql_table(entities.News.__tablename__, full_db.bind) # type: ignore
assert "text" in articles.columns
del articles["text"]