From f645e2ec4b2d0931c382d8ed8b5fed52a9ede67c Mon Sep 17 00:00:00 2001 From: Philipp Horstenkamp Date: Wed, 3 Jan 2024 18:31:01 +0100 Subject: [PATCH] Added a section on startup in the README.md (#507) Reworked the index.md - removed commented out text Minor restructuring of the README.md --------- Co-authored-by: Tristan Nolde --- CONTRIBUTING.md | 2 +- README.md | 116 ++++++++++++++++++++++++++------------- docker-compose.yml | 6 +- documentations/index.rst | 20 ------- local-docker-compose.yml | 5 +- 5 files changed, 85 insertions(+), 64 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index eaae33f..bb45244 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,4 +1,4 @@ -# Contribution guidelines +# CONTRIBUTING.md: Contribution guidelines ## Dev Setup diff --git a/README.md b/README.md index 60fce14..23aa0cb 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,10 @@ -# aki_prj23_transparenzregister +# README.md of the aki_prj23_transparenzregister ## Contributions See the [CONTRIBUTING.md](CONTRIBUTING.md) about how code should be formatted and what kind of rules we set ourselves. -## Available entrypoints +## Defined entrypoints The project has currently the following entrypoint available: @@ -13,15 +13,82 @@ The project has currently the following entrypoint available: - **reset-sql** > Resets all sql tables in the connected db. - **copy-sql** > Copys the content of a db to another db. - **webserver** > Starts the webserver showing the analysis results. +- **find-missing-companies** > +- **ingest** > -## DB Connection settings +All entrypoints support the `-h` argument and show a short help. + +## Applikation startup -To connect to the SQL db see [sql/connector.py](./src/aki_prj23_transparenzregister/utils/sql/connector.py) -To connect to the Mongo db see [connect] +### Central Build -Create a `secrets.json` in the root of this repo with the following structure (values to be replaces by desired config): +The packages / container built by GitHub are accessible for users that are logged into the GitHub Container Registry (GHCR) with a Personal Access token via Docker. +Run `docker login ghcr.io` to start that process. [The complete docs on logging in can be found here.](https://docs.github.com/de/packages/working-with-a-github-packages-registry/working-with-the-container-registry) +The application can than be simply started with `docker compose up --pull`. +Please note that some configuration with a `.env` is necessary. + +### Local Build + +The application can be locally build by starting the `rebuild-and-start.bat`, if `poetry` and `docker-compose` is installed. +This will build a current `*.whl` and build the Docker container locally. +The configuration that start like this is the `local-docker-compose.yaml`. +Please note that some configuration with a `.env` is necessary. + +## Application Settings + +### Docker configuration / Environmental-Variables +The current design of this application suggests that it is started inside a `docker-compose` configuration. +For `docker-compose` this is commonly done by providing a `.env` file in the root folder. + +To use the environmental configuration start an application part with the `ENV` argument (`webserver ENV`). + +```dotenv +# Defines the container registry used. Default: "ghcr.io/fhswf/aki_prj23_transparenzregister" +CR=ghcr.io/fhswf/aki_prj23_transparenzregister + +# main is the tag the main branch is taged with and is currently in use +TAG=latest + +# Configures the access port for the webserver. +# Default: "80" (local only) +HTTP_PORT: 8888 + +# configures where the application root is based. Default: "/" +DASH_URL_BASE_PATHNAME=/transparenzregister/ + +# Enables basic auth for the application. +# Diabled when one is empty. Default: Disabled +PYTHON_DASH_LOGIN_USERNAME=some-login-to-webgui +PYTHON_DASH_LOGIN_PW=some-pw-to-login-to-webgui + +# How often data should be ingested in houres, Default: "4" +PYTHON_INGEST_SCHEDULE=12 + +# Acces to the mongo db +PYTHON_MONGO_USERNAME=username +PYTHON_MONGO_HOST=mongodb +PYTHON_MONGO_PASSWORD=password +PYTHON_MONGO_PORT=27017 +PYTHON_MONGO_DATABASE=transparenzregister + +# Acces to the postress sql db +PYTHON_POSTGRES_USERNAME=username +PYTHON_POSTGRES_PASSWORD=password +PYTHON_POSTGRES_HOST=postgres-host +PYTHON_POSTGRES_DATABASE=db-name +PYTHON_POSTGRES_PORT=5432 + +# An overwrite path to an sqlite db, overwrites the POSTGRES section +PYTHON_SQLITE_PATH=PathToSQLite3.db +``` + +### Local execution / config file + +Create a `*.json` in the root of this repo with the following structure + (values to be replaces by desired config): +Please note that an `sqlite` entry overwrites the `postgres` entry. +To use the `*.json` use the path to it as an argument when using an entrypoint (`webserver secrets.json`). -The sqlite db is alternative to the postgres section. ```json { "sqlite": "path-to-sqlite.db", @@ -42,35 +109,8 @@ The sqlite db is alternative to the postgres section. } ``` -Alternatively, the secrets can be provided as environment variables. One option to do so is to add a `.env` file with -the following layout: +### sqlite vs. postgres -```dotenv -PYTHON_POSTGRES_USERNAME=postgres -PYTHON_POSTGRES_PASSWORD=postgres -PYTHON_POSTGRES_HOST=postgres -PYTHON_POSTGRES_DATABASE=postgres -PYTHON_POSTGRES_PORT=5432 - -PYTHON_MONGO_USERNAME=username -PYTHON_MONGO_HOST=mongodb -PYTHON_MONGO_PASSWORD=password -PYTHON_MONGO_PORT=27017 -PYTHON_MONGO_DATABASE=transparenzregister - -# An overwrite path to an sqlite db -PYTHON_SQLITE_PATH=PathToSQLite3.db - -PYTHON_DASH_LOGIN_USERNAME=some-login-to-webgui -PYTHON_DASH_LOGIN_PW=some-pw-to-login-to-webgui - -# Every x hours -PYTHON_INGEST_SCHEDULE=12 - -CR=ghcr.io/fhswf/aki_prj23_transparenzregister -TAG=latest - -HTTP_PORT=80 -``` - -The prefix `PYTHON_` can be customized by setting a different `prefix` when constructing the ConfigProvider. +We support both sqlite and postgres because a local db is filled in about 10% of the time the remote db needs to be completed. +Even tough we use the `sqlite` for testing the connection can't manage multithreading or multiprocessing. +This clashes with the webserver. For production mode use the `postgres`-db. diff --git a/docker-compose.yml b/docker-compose.yml index c21b761..526079b 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -67,7 +67,7 @@ services: depends_on: - postgres ports: - - ${HTTP_PORT}:${HTTP_PORT} + - ${HTTP_PORT:-error}:${HTTP_PORT:-error} environment: PYTHON_POSTGRES_USERNAME: ${PYTHON_POSTGRES_USERNAME:-postgres} PYTHON_POSTGRES_PASSWORD: ${PYTHON_POSTGRES_PASSWORD:?error} @@ -76,6 +76,6 @@ services: PYTHON_POSTGRES_PORT: ${PYTHON_POSTGRES_PORT:-5432} PYTHON_DASH_LOGIN_PW: ${PYTHON_DASH_LOGIN_PW} PYTHON_DASH_LOGIN_USERNAME: ${PYTHON_DASH_LOGIN_USERNAME} - DASH_URL_BASE_PATHNAME: /transparenzregister/ - PORT: ${HTTP_PORT} + DASH_URL_BASE_PATHNAME: ${DASH_URL_BASE_PATHNAME:-/transparenzregister/} + PORT: ${HTTP_PORT:-error} HOST: 0.0.0.0 diff --git a/documentations/index.rst b/documentations/index.rst index e29ea16..6046456 100644 --- a/documentations/index.rst +++ b/documentations/index.rst @@ -7,26 +7,6 @@ Das Projekt startete im Sommersemester 2023 und dauerte ein Jahr. Die Dokumentation ist zwar auf Deutsch, wir steigen aber ein mit der `README.md` und der `CONRIBUTING.md`. Diese sind, um Industriestandards zu entsprechen, auf Englisch gehalten. -.. On the 9th September 2023 the following current development state has been determined: - -.. .. drawio-figure:: project_management/Arch.-Planning.drawio - :format: png - :page-index: 1 - -.. The target deployment architecture should look as follows: - -.. .. drawio-figure:: project_management/Arch.-Planning.drawio - :format: png - :page-index: 2 - -.. The timeline to reach this goal is: - -.. .. drawio-figure:: project_management/Arch.-Planning.drawio - :format: png - :page-index: 3 - -.. An example diagram - .. include:: ../README.md :parser: myst_parser.sphinx_ .. include:: ../CONTRIBUTING.md diff --git a/local-docker-compose.yml b/local-docker-compose.yml index 9b1aa50..12a9c98 100644 --- a/local-docker-compose.yml +++ b/local-docker-compose.yml @@ -94,7 +94,7 @@ services: depends_on: - postgres ports: - - 8888:8888 + - ${HTTP_PORT:-8888}:${HTTP_PORT:-8888} environment: PYTHON_POSTGRES_USERNAME: ${PYTHON_POSTGRES_USERNAME:-postgres} PYTHON_POSTGRES_PASSWORD: ${PYTHON_POSTGRES_PASSWORD:?error} @@ -103,7 +103,8 @@ services: PYTHON_POSTGRES_PORT: ${PYTHON_POSTGRES_PORT:-5432} PYTHON_DASH_LOGIN_PW: ${PYTHON_DASH_LOGIN_PW} PYTHON_DASH_LOGIN_USERNAME: ${PYTHON_DASH_LOGIN_USERNAME} - PORT: 8888 + PORT: ${HTTP_PORT:-8888} + HOST: 0.0.0.0 volumes: