Maintenance and Development

Python Image Maintenance - Fixing CVEs

Each version of python image is scheduled once a week for a trivy scan in Gitlab. It automatically checks, if libraries used contain know vulnerabilities, and provides list of affected libraries together with fixed versions. To fix the python version vX, take following steps:

1. Create a branch based on vX to update vX, usually "fix-trivy-scan-vX"

2. In "trivy-scan-vX", update the libraries, in the order: pyfx, datascience, neural

3. Make MR for branch into vX

Updating libraries:

• For Linux packages, those should be fixed automatically once image is rebuilt (It installs latest versions of Linux packages during build phase, so triggering pipeline should be enough)

• For Python packages, there are two kinds of packages:

  • python packages with version fixed in pyproject.toml file, and they need to be updated there, to version containing fix, while fixing major and minor release version - “version = X.Y.*”

    poetry lock --no-update

    will update the poetry.lock, updating just necessary packages (ones modified in pyproject.toml) to match the requirements in pyproject.toml, other packages remain unchanged if possible. Both modified pyproject.toml and poetry.lock should be committed.

  • python packages not specified in in pyproject.toml file, those can be updated by running command:

  • "poetry update packageX packageY"

    This requires poetry install successfully run previously. Without install available on your system, workaround might be used: Add that package into pyproject.toml and set its minimal version to one containing fix.
    run "poetry lock --no-update", this will update lock file based on pyproject.toml, but not update other packages, effectively updating just ones mentioned, and other if necessary due to dependency. Finally, add modified poetry.lock into fix-trivy-scan branch, but not pyproject.toml, to avoid specifying unnecessary packages there.

Image Development

Running python locally

Running python locally allows to test changes without re-running gitlab pipeline in each iteration.
To do so, in a folder with installed you can run poetry run python [script.py]
Alternatively you can add python installed by a poetry to your IDE as an interpreter, and use it from IDE.
To locate where poetry installed python, run: poetry run which python.
This will give you path to python, which you have to add into IDE as interpreter. Also, if you are using Python Console, and made changes to pyfx, you will need to restart the console for changes to take effect.

Using job context