From 7d2ccfa37797cc30c1e7d33ad57753006530f1fd Mon Sep 17 00:00:00 2001 From: avouacr Date: Sat, 19 Sep 2026 17:09:30 +0000 Subject: [PATCH 01/14] chore(deps): upgrade python & cuda --- versions.env | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/versions.env b/versions.env index aeb4912a..f7090b7c 100644 --- a/versions.env +++ b/versions.env @@ -2,10 +2,10 @@ # (utils/generate_matrix.py) and by local builds (utils/build_chains.sh, utils/build_chain.py). # Updated by Renovate, which also keeps the ARG defaults of the Dockerfiles in sync. -CUDA_BASE_IMAGE="nvidia/cuda:12.6.3-cudnn-devel-ubuntu24.04" +CUDA_BASE_IMAGE="nvidia/cuda:12.8.1-cudnn-devel-ubuntu24.04" -PYTHON_VERSION_1="3.13.15" -PYTHON_VERSION_2="3.12.14" +PYTHON_VERSION_1="3.14.7" +PYTHON_VERSION_2="3.13.15" R_VERSION_1="4.6.1" R_VERSION_2="4.5.3" From 9423890dcf0d35ec223696e6d80bb171589edbc9 Mon Sep 17 00:00:00 2001 From: avouacr Date: Sat, 19 Sep 2026 18:42:26 +0000 Subject: [PATCH 02/14] use uv to install python as binary --- AGENTS.md | 6 +-- base/scripts/install-python.sh | 80 +++++++++++++++------------------ docs/improvement-plan.md | 6 +++ python-minimal/requirements.txt | 1 - 4 files changed, 44 insertions(+), 49 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 5a879819..0a4dcddf 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -59,7 +59,7 @@ The valid layer stacks are declared in the `chains` dict in `utils/build_chain.p - Downloads in build scripts must fail loudly: use `wget -nv` (never `-q`, which also hides error messages) and `curl -fsSL` (`-f` fails on HTTP errors, `-S` prints them despite `-s`). Put version lookups (`VERSION=$(curl ...)`) on their own line, and use `set -o pipefail` when they are piped, so that a failed lookup stops the build instead of producing a broken URL. - Third-party apt repositories (PostgreSQL in `base/scripts/install-system-libs.sh`, ubuntugis in `python-datascience/scripts/install-geospatial-python.sh`) are added only for the packages that need them, then removed right after, so that later installs only come from Ubuntu. Install from the repository every package whose Ubuntu counterpart would require an exact version of it (e.g. `libpq-dev` with `postgresql-client`). - Runtime user is `onyxia` (UID 1000, group `users` GID 100) with passwordless sudo; workdir is `/home/onyxia/work`. -- Python is built from source into `/opt/python` (not the system Python); packages are installed with `uv pip install --system`. R comes from rocker scripts (`/rocker_scripts/...`), with `R_HOME=/usr/local/lib/R`. +- Python lives in `/opt/python` (not the system Python). `base/scripts/install-python.sh` installs a pinned uv (`UV_VERSION`, binary in `/usr/local/bin`), then a python-build-standalone build with `uv python install` in `/opt/uv-python`, and makes `/opt/python` a symlink to it. It is deliberately not in uv's default location (`~/.local/share/uv/python`), because `uv pip install --system` ignores Pythons installed there. The script removes the `EXTERNALLY-MANAGED` marker uv adds, which would otherwise block `pip install` and `uv pip install --system` (the reason an earlier uv attempt was dropped). There is no build from source: a Python version only installs once the pinned uv knows it, so if a Renovate Python bump fails the build, merge the uv bump from the "Build tools" PR first. Packages are installed with `uv pip install --system`. - `base/scripts/onyxia-init.sh` is the container init script used by Onyxia at startup (region init script, Vault secrets injection, CA bundles via `PATH_TO_CA_BUNDLE`, `PIP_REPOSITORY`/`R_REPOSITORY` mirrors — see README). ### CI (`.github/workflows/`) @@ -74,7 +74,7 @@ The valid layer stacks are declared in the `chains` dict in `utils/build_chain.p `versions.env` (repo root) is the single source of truth for the Python, R and Spark versions and for the CUDA base image of GPU variants. It is read by the CI matrix (`utils/generate_matrix.py`, through `utils/versions.py`) and by local builds (`utils/build_chains.sh` sources it, `utils/build_chain.py` reads the CUDA image). The only copies are the `ARG` defaults of the `python-minimal`, `r-python-julia`, `r-minimal` and `spark` Dockerfiles, which CI always overrides. `renovate.json` has one regex manager per version slot (Python 1/2, R 1/2, Spark) covering `versions.env` and those Dockerfiles, and groups each slot into a single PR; version 2 only gets patch bumps, so moving it to a new minor release (e.g. when version 1 moves on) is a manual change. A separate manager tracks the CUDA image. When changing a version by hand, edit `versions.env` and the matching Dockerfile `ARG` default. -Tools downloaded at build time (kubectl, helm, AWS CLI, DuckDB CLI, quarto, opencode, Julia, code-server) are pinned in their install script, **not** in Dockerfiles (Dockerfiles only pin the versions the project manages: Python, R, Spark). Each script follows the same model, see `base/scripts/install-kubectl.sh`: +Tools downloaded at build time (kubectl, helm, AWS CLI, DuckDB CLI, quarto, opencode, Julia, code-server, uv) are pinned in their install script, **not** in Dockerfiles (Dockerfiles only pin the versions the project manages: Python, R, Spark). Each script follows the same model, see `base/scripts/install-kubectl.sh`: - a `# renovate: datasource= depName=` comment right above a `_VERSION="x.y.z"` line. One generic regex manager in `renovate.json` picks it up (in `scripts/*.sh` and in `.github/actions/*/action.yml`, where container-structure-test is pinned the same way), and all tool bumps land in a single weekly "Build tools" PR. -- the official release file is downloaded to a `mktemp -d` directory and verified before installing: against the upstream checksum file when there is one (kubectl, helm, quarto, Julia), against the SHA-256 GitHub records for the release asset (read from the GitHub API in the script, with `set -o pipefail`) when there is none (DuckDB, opencode, code-server), and with the PGP signature for the AWS CLI (the public key is embedded in `install-awscli.sh` and expires on 2027-07-01). +- the official release file is downloaded to a `mktemp -d` directory and verified before installing: against the upstream checksum file when there is one (kubectl, helm, quarto, Julia, uv), against the SHA-256 GitHub records for the release asset (read from the GitHub API in the script, with `set -o pipefail`) when there is none (DuckDB, opencode, code-server), and with the PGP signature for the AWS CLI (the public key is embedded in `install-awscli.sh` and expires on 2027-07-01). - amd64 only. diff --git a/base/scripts/install-python.sh b/base/scripts/install-python.sh index 188490fd..f5a0bd74 100755 --- a/base/scripts/install-python.sh +++ b/base/scripts/install-python.sh @@ -1,54 +1,44 @@ #!/bin/bash set -e -# Install system libraries required to build Python from source -savedAptMark="$(apt-mark showmanual)" -/opt/apt-install.sh \ - dpkg-dev \ - libbz2-dev \ - libc6-dev \ - libdb-dev \ - libffi-dev \ - libgdbm-dev \ - liblzma-dev \ - libncursesw5-dev \ - libreadline-dev \ - libsqlite3-dev \ - libssl-dev \ - tk-dev \ - uuid-dev \ - xz-utils \ - zlib1g-dev - -# Build Python from sources -wget -nv https://www.python.org/ftp/python/${PYTHON_VERSION}/Python-${PYTHON_VERSION}.tgz -tar xzvf Python-${PYTHON_VERSION}.tgz -cd Python-${PYTHON_VERSION} -./configure \ - --prefix="${PYTHON_DIR}" \ - --enable-loadable-sqlite-extensions \ - --enable-optimizations \ - --enable-shared \ - --with-lto \ - LDFLAGS="-Wl,-rpath ${PYTHON_DIR}/lib" -make -j"$(nproc)" -make install - -# Clean install files -cd .. -rm -rf Python-${PYTHON_VERSION}.tgz Python-${PYTHON_VERSION} -apt-mark auto '.*' > /dev/null -apt-mark manual $savedAptMark -apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false - -# Useful symlinks -ln -sf "${PYTHON_DIR}/bin/python3" "${PYTHON_DIR}/bin/python" -ln -sf "${PYTHON_DIR}/bin/pip3" "${PYTHON_DIR}/bin/pip" +### Install uv + +# renovate: datasource=github-releases depName=astral-sh/uv +UV_VERSION="0.12.17" + +# Download the official archive and verify its checksum +UV_ARCHIVE="uv-x86_64-unknown-linux-gnu.tar.gz" +UV_URL="https://github.com/astral-sh/uv/releases/download/${UV_VERSION}" +TMP_DIR=$(mktemp -d) +curl -fsSL "${UV_URL}/${UV_ARCHIVE}" -o "${TMP_DIR}/${UV_ARCHIVE}" +curl -fsSL "${UV_URL}/${UV_ARCHIVE}.sha256" -o "${TMP_DIR}/${UV_ARCHIVE}.sha256" +(cd "${TMP_DIR}" && sha256sum --check --strict "${UV_ARCHIVE}.sha256") + +# Install uv +tar -xzf "${TMP_DIR}/${UV_ARCHIVE}" -C "${TMP_DIR}" +install -m 0755 "${TMP_DIR}/uv-x86_64-unknown-linux-gnu/uv" "${TMP_DIR}/uv-x86_64-unknown-linux-gnu/uvx" /usr/local/bin/ +rm -rf "${TMP_DIR}" + + +### Install Python with uv prebuilt python binaries + +# Install Python outside of uv's default location (~/.local/share/uv/python): `uv pip install --system` +# ignores the Pythons installed there. No cache, and no launchers in ~/.local/bin. +export UV_PYTHON_INSTALL_DIR="/opt/uv-python" +export UV_NO_CACHE=1 +uv python install "${PYTHON_VERSION}" --no-bin + +# Expose it at a stable path, ${PYTHON_DIR} (set in the Dockerfile), whose bin/ is on the PATH: +# python, python3, python3.X, pip and the commands installed by packages (jupyter, marimo...) +UV_PYTHON_EXECUTABLE=$(uv python find --managed-python "${PYTHON_VERSION}") +ln -s "$(dirname "$(dirname "$(realpath "${UV_PYTHON_EXECUTABLE}")")")" "${PYTHON_DIR}" + +# uv marks its Pythons as externally managed, which would block `pip install` and `uv pip install --system` +rm "${PYTHON_DIR}/lib/python${PYTHON_VERSION%.*}/EXTERNALLY-MANAGED" # Checks python --version which python -# Upgrade pip & install uv +# Upgrade pip pip install --no-cache-dir --upgrade pip -pip install --no-cache-dir uv diff --git a/docs/improvement-plan.md b/docs/improvement-plan.md index 9d3df45f..c7a98681 100644 --- a/docs/improvement-plan.md +++ b/docs/improvement-plan.md @@ -43,6 +43,7 @@ Published sizes on 2026-09-19 (compressed, amd64): - Add a CI step that reports layer sizes, so regressions show up. - **GPU testing:** GPU images are not tested in CI because they are too big for GitHub-hosted runners to load and test. Once sizes are reduced, check whether they fit, and test them like the CPU ones if so. +<<<<<<< Updated upstream ### 13. Python compiled from source — M `base/scripts/install-python.sh` compiles CPython with `--enable-optimizations --with-lto`. That sits on the critical path of every Python chain (python-minimal and r-python-julia), × 2 versions × CPU/GPU. @@ -78,6 +79,8 @@ Fix: a single `images.yaml` (layers, parents, languages, GPU flag) that drives b - **`RDebugger.r-debugger` VS Code extension:** still installed by `vscode/scripts/install-vscode-extensions.sh`, but its R backend `vscDebugger` was removed, so R debugging doesn't work (the extension only offers to install the package at first use). Remove it from `r_extensions`, and from the vscode tests if they list it. +======= +>>>>>>> Stashed changes ## Optional ideas - CI: Trivy vulnerability scan, and `sbom: true` / `provenance: true` in `docker/build-push-action`. @@ -96,3 +99,6 @@ Fix: a single `images.yaml` (layers, parents, languages, GPU flag) that drives b - **The Git token stays in the clone URL:** containers are isolated and short-lived, and moving it to a credential helper would still leave it readable in plain text. - **DuckDB secret values are not SQL-escaped:** AWS/MinIO credentials can't contain quotes. - **`import tkinter` is broken in the images:** irrelevant for headless images. +- **The `chown -R` over `~/work` at startup stays:** the folder only holds what was just cloned or added by an init script, and the user must own all of it. +- **The image graph stays declared twice** (`chains` in `utils/build_chain.py`, jobs in `main-workflow.yml`): no generated CI. Adding an image means updating both. +- **Python is installed with uv only,** no build from source: the `EXTERNALLY-MANAGED` marker uv adds is removed, and Python is installed outside uv's default location so that `uv pip install --system` uses it. diff --git a/python-minimal/requirements.txt b/python-minimal/requirements.txt index 04a7b6e1..40e9f260 100644 --- a/python-minimal/requirements.txt +++ b/python-minimal/requirements.txt @@ -7,4 +7,3 @@ pandas pyarrow requests s3fs -uv From b985c7759458d66923f5644a0ad31cdd660e6e2b Mon Sep 17 00:00:00 2001 From: avouacr Date: Sat, 19 Sep 2026 19:28:14 +0000 Subject: [PATCH 03/14] fix: versions function --- utils/build_chain.py | 15 ++++++++++++++- utils/build_chains.sh | 17 ++++++++--------- 2 files changed, 22 insertions(+), 10 deletions(-) diff --git a/utils/build_chain.py b/utils/build_chain.py index 932a7166..8835f815 100644 --- a/utils/build_chain.py +++ b/utils/build_chain.py @@ -1,8 +1,8 @@ import argparse import logging import subprocess +from pathlib import Path -from versions import read_versions logging.basicConfig( level=logging.INFO, @@ -10,6 +10,8 @@ ) logger = logging.getLogger(__name__) +VERSIONS_FILE = Path("versions.env") + chains = { "rstudio": ["base", "r-minimal", "r-datascience", "rstudio"], "rstudio-sparkr": ["base", "r-minimal", "spark", "rstudio"], @@ -108,6 +110,17 @@ def build_cli_parser(): return parser +def read_versions(path=VERSIONS_FILE): + """Parse the KEY="value" lines of versions.env into a dict, ignoring comments and blank lines.""" + versions = {} + for line in VERSIONS_FILE.read_text().splitlines(): + line = line.strip() + if line and not line.startswith("#"): + key, value = line.split("=", 1) + versions[key] = value.strip('"') + return versions + + if __name__ == "__main__": # Parse CLI parameters parser = build_cli_parser() diff --git a/utils/build_chains.sh b/utils/build_chains.sh index 1b180e3d..7da1d8c3 100755 --- a/utils/build_chains.sh +++ b/utils/build_chains.sh @@ -11,24 +11,23 @@ DOCKER_BUILD_ARGS="" # Build process -# Maintained versions (PYTHON_VERSION_1/2, R_VERSION_1/2, SPARK_VERSION) -# shellcheck source=versions.env +# Load maintained versions source "$(dirname "$0")/../versions.env" PYTHON_VERSIONS=("$PYTHON_VERSION_1" "$PYTHON_VERSION_2") R_VERSIONS=("$R_VERSION_1" "$R_VERSION_2") for py_ver in "${PYTHON_VERSIONS[@]}"; do - python3 utils/build_chain.py --chain vscode-python --py_version $py_ver $DOCKER_BUILD_ARGS - python3 utils/build_chain.py --chain vscode-pytorch --py_version $py_ver --gpu $DOCKER_BUILD_ARGS - python3 utils/build_chain.py --chain jupyter-pyspark --py_version $py_ver --spark_version $SPARK_VERSION $DOCKER_BUILD_ARGS + python3 build_chain.py --chain vscode-python --py_version $py_ver $DOCKER_BUILD_ARGS + python3 build_chain.py --chain vscode-pytorch --py_version $py_ver --gpu $DOCKER_BUILD_ARGS + python3 build_chain.py --chain jupyter-pyspark --py_version $py_ver --spark_version $SPARK_VERSION $DOCKER_BUILD_ARGS done for r_ver in "${R_VERSIONS[@]}"; do - python3 utils/build_chain.py --chain rstudio --r_version $r_ver $DOCKER_BUILD_ARGS - python3 utils/build_chain.py --chain sparkr --r_version $r_ver --spark_version $SPARK_VERSION $DOCKER_BUILD_ARGS + python3 build_chain.py --chain rstudio --r_version $r_ver $DOCKER_BUILD_ARGS + python3 build_chain.py --chain sparkr --r_version $r_ver --spark_version $SPARK_VERSION $DOCKER_BUILD_ARGS done # r-python-julia images are built with only latest versions of R & Python -python3 utils/build_chain.py --chain rstudio-r-python-julia --r_version ${R_VERSIONS[0]} --py_version ${PYTHON_VERSIONS[0]} $DOCKER_BUILD_ARGS -python3 utils/build_chain.py --chain jupyter-r-python-julia --r_version ${R_VERSIONS[0]} --py_version ${PYTHON_VERSIONS[0]} $DOCKER_BUILD_ARGS +python3 build_chain.py --chain rstudio-r-python-julia --r_version ${R_VERSIONS[0]} --py_version ${PYTHON_VERSIONS[0]} $DOCKER_BUILD_ARGS +python3 build_chain.py --chain jupyter-r-python-julia --r_version ${R_VERSIONS[0]} --py_version ${PYTHON_VERSIONS[0]} $DOCKER_BUILD_ARGS From ddd0d75a13730bb6b14fcd35e826d59fc9cea1f9 Mon Sep 17 00:00:00 2001 From: avouacr Date: Sat, 19 Sep 2026 19:38:54 +0000 Subject: [PATCH 04/14] dev --- .github/workflows/main-workflow-template.yml | 2 -- utils/build_chain.py | 27 +++++++++----------- utils/build_chains.sh | 16 ++++++------ versions.env | 3 ++- 4 files changed, 22 insertions(+), 26 deletions(-) diff --git a/.github/workflows/main-workflow-template.yml b/.github/workflows/main-workflow-template.yml index a5fe4243..c7f1167a 100644 --- a/.github/workflows/main-workflow-template.yml +++ b/.github/workflows/main-workflow-template.yml @@ -39,12 +39,10 @@ jobs: steps: - name: Check out code uses: actions/checkout@v7 - - name: Set up Python uses: actions/setup-python@v7 with: python-version: '3.14' - - id: set-matrix run: | python3 utils/generate_matrix.py --input_image ${{ inputs.base_image }} --output_image ${{ inputs.image }} \ diff --git a/utils/build_chain.py b/utils/build_chain.py index 8835f815..ce972b91 100644 --- a/utils/build_chain.py +++ b/utils/build_chain.py @@ -3,15 +3,12 @@ import subprocess from pathlib import Path - logging.basicConfig( level=logging.INFO, format="%(asctime)s - %(levelname)s - %(message)s", ) logger = logging.getLogger(__name__) -VERSIONS_FILE = Path("versions.env") - chains = { "rstudio": ["base", "r-minimal", "r-datascience", "rstudio"], "rstudio-sparkr": ["base", "r-minimal", "spark", "rstudio"], @@ -36,6 +33,17 @@ } +def read_versions(): + versions_file_path = "versions.env" + versions = {} + for line in Path(versions_file_path).read_text().splitlines(): + line = line.strip() + if line and not line.startswith("#"): + key, value = line.split("=", 1) + versions[key] = value.strip('"') + return versions + + def build_chain(chain_name, r_version, py_version, spark_version, gpu, no_test, push): logger.info(f"Building chain : {chain_name}") @@ -48,7 +56,7 @@ def build_chain(chain_name, r_version, py_version, spark_version, gpu, no_test, # Specify base image for each build step if i == 0: # First step : define external base images - previous_image = read_versions()["CUDA_BASE_IMAGE"] if gpu else "ubuntu:24.04" + previous_image = read_versions()["BASE_IMAGE_GPU"] if gpu else read_versions()["BASE_IMAGE"] else: # Intermediary and final steps : use previous built tag as base image previous_image = tag @@ -110,17 +118,6 @@ def build_cli_parser(): return parser -def read_versions(path=VERSIONS_FILE): - """Parse the KEY="value" lines of versions.env into a dict, ignoring comments and blank lines.""" - versions = {} - for line in VERSIONS_FILE.read_text().splitlines(): - line = line.strip() - if line and not line.startswith("#"): - key, value = line.split("=", 1) - versions[key] = value.strip('"') - return versions - - if __name__ == "__main__": # Parse CLI parameters parser = build_cli_parser() diff --git a/utils/build_chains.sh b/utils/build_chains.sh index 7da1d8c3..536e5da8 100755 --- a/utils/build_chains.sh +++ b/utils/build_chains.sh @@ -12,22 +12,22 @@ DOCKER_BUILD_ARGS="" # Build process # Load maintained versions -source "$(dirname "$0")/../versions.env" +source "versions.env" PYTHON_VERSIONS=("$PYTHON_VERSION_1" "$PYTHON_VERSION_2") R_VERSIONS=("$R_VERSION_1" "$R_VERSION_2") for py_ver in "${PYTHON_VERSIONS[@]}"; do - python3 build_chain.py --chain vscode-python --py_version $py_ver $DOCKER_BUILD_ARGS - python3 build_chain.py --chain vscode-pytorch --py_version $py_ver --gpu $DOCKER_BUILD_ARGS - python3 build_chain.py --chain jupyter-pyspark --py_version $py_ver --spark_version $SPARK_VERSION $DOCKER_BUILD_ARGS + python3 utils/build_chain.py --chain vscode-python --py_version $py_ver $DOCKER_BUILD_ARGS + python3 utils/build_chain.py --chain vscode-pytorch --py_version $py_ver --gpu $DOCKER_BUILD_ARGS + python3 utils/build_chain.py --chain jupyter-pyspark --py_version $py_ver --spark_version $SPARK_VERSION $DOCKER_BUILD_ARGS done for r_ver in "${R_VERSIONS[@]}"; do - python3 build_chain.py --chain rstudio --r_version $r_ver $DOCKER_BUILD_ARGS - python3 build_chain.py --chain sparkr --r_version $r_ver --spark_version $SPARK_VERSION $DOCKER_BUILD_ARGS + python3 utils/build_chain.py --chain rstudio --r_version $r_ver $DOCKER_BUILD_ARGS + python3 utils/build_chain.py --chain sparkr --r_version $r_ver --spark_version $SPARK_VERSION $DOCKER_BUILD_ARGS done # r-python-julia images are built with only latest versions of R & Python -python3 build_chain.py --chain rstudio-r-python-julia --r_version ${R_VERSIONS[0]} --py_version ${PYTHON_VERSIONS[0]} $DOCKER_BUILD_ARGS -python3 build_chain.py --chain jupyter-r-python-julia --r_version ${R_VERSIONS[0]} --py_version ${PYTHON_VERSIONS[0]} $DOCKER_BUILD_ARGS +python3 utils/build_chain.py --chain rstudio-r-python-julia --r_version ${R_VERSIONS[0]} --py_version ${PYTHON_VERSIONS[0]} $DOCKER_BUILD_ARGS +python3 utils/build_chain.py --chain jupyter-r-python-julia --r_version ${R_VERSIONS[0]} --py_version ${PYTHON_VERSIONS[0]} $DOCKER_BUILD_ARGS diff --git a/versions.env b/versions.env index f7090b7c..fd2f7053 100644 --- a/versions.env +++ b/versions.env @@ -2,7 +2,8 @@ # (utils/generate_matrix.py) and by local builds (utils/build_chains.sh, utils/build_chain.py). # Updated by Renovate, which also keeps the ARG defaults of the Dockerfiles in sync. -CUDA_BASE_IMAGE="nvidia/cuda:12.8.1-cudnn-devel-ubuntu24.04" +BASE_IMAGE="ubuntu:24.04" +BASE_IMAGE_GPU="nvidia/cuda:12.8.1-cudnn-devel-ubuntu24.04" PYTHON_VERSION_1="3.14.7" PYTHON_VERSION_2="3.13.15" From c0c1f019c79e1ad595ae34b86c79ecfce2345a27 Mon Sep 17 00:00:00 2001 From: avouacr Date: Sat, 19 Sep 2026 19:41:27 +0000 Subject: [PATCH 05/14] dev --- utils/build_chain.py | 2 +- versions.env | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/utils/build_chain.py b/utils/build_chain.py index ce972b91..44591170 100644 --- a/utils/build_chain.py +++ b/utils/build_chain.py @@ -56,7 +56,7 @@ def build_chain(chain_name, r_version, py_version, spark_version, gpu, no_test, # Specify base image for each build step if i == 0: # First step : define external base images - previous_image = read_versions()["BASE_IMAGE_GPU"] if gpu else read_versions()["BASE_IMAGE"] + previous_image = read_versions()["BASE_IMAGE_GPU"] if gpu else read_versions()["BASE_IMAGE_CPU"] else: # Intermediary and final steps : use previous built tag as base image previous_image = tag diff --git a/versions.env b/versions.env index fd2f7053..8af8440a 100644 --- a/versions.env +++ b/versions.env @@ -2,7 +2,7 @@ # (utils/generate_matrix.py) and by local builds (utils/build_chains.sh, utils/build_chain.py). # Updated by Renovate, which also keeps the ARG defaults of the Dockerfiles in sync. -BASE_IMAGE="ubuntu:24.04" +BASE_IMAGE_CPU="ubuntu:24.04" BASE_IMAGE_GPU="nvidia/cuda:12.8.1-cudnn-devel-ubuntu24.04" PYTHON_VERSION_1="3.14.7" From 1e9f54194551c29e2c9f741e90f0f4eeae4ef2de Mon Sep 17 00:00:00 2001 From: avouacr Date: Sat, 19 Sep 2026 20:32:54 +0000 Subject: [PATCH 06/14] dev --- AGENTS.md | 2 +- base/scripts/install-python.sh | 14 ++--- jupyter/tests.yaml | 108 +++++++++++++++---------------- marimo/tests.yaml | 2 +- python-datascience/tests.yaml | 92 +++++++++++++-------------- python-minimal/Dockerfile | 4 +- python-minimal/tests.yaml | 96 ++++++++++++++-------------- python-pytorch/tests.yaml | 90 +++++++++++++------------- r-python-julia/Dockerfile | 4 +- r-python-julia/tests.yaml | 112 ++++++++++++++++----------------- versions.env | 2 +- 11 files changed, 262 insertions(+), 264 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 0a4dcddf..02074102 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -59,7 +59,7 @@ The valid layer stacks are declared in the `chains` dict in `utils/build_chain.p - Downloads in build scripts must fail loudly: use `wget -nv` (never `-q`, which also hides error messages) and `curl -fsSL` (`-f` fails on HTTP errors, `-S` prints them despite `-s`). Put version lookups (`VERSION=$(curl ...)`) on their own line, and use `set -o pipefail` when they are piped, so that a failed lookup stops the build instead of producing a broken URL. - Third-party apt repositories (PostgreSQL in `base/scripts/install-system-libs.sh`, ubuntugis in `python-datascience/scripts/install-geospatial-python.sh`) are added only for the packages that need them, then removed right after, so that later installs only come from Ubuntu. Install from the repository every package whose Ubuntu counterpart would require an exact version of it (e.g. `libpq-dev` with `postgresql-client`). - Runtime user is `onyxia` (UID 1000, group `users` GID 100) with passwordless sudo; workdir is `/home/onyxia/work`. -- Python lives in `/opt/python` (not the system Python). `base/scripts/install-python.sh` installs a pinned uv (`UV_VERSION`, binary in `/usr/local/bin`), then a python-build-standalone build with `uv python install` in `/opt/uv-python`, and makes `/opt/python` a symlink to it. It is deliberately not in uv's default location (`~/.local/share/uv/python`), because `uv pip install --system` ignores Pythons installed there. The script removes the `EXTERNALLY-MANAGED` marker uv adds, which would otherwise block `pip install` and `uv pip install --system` (the reason an earlier uv attempt was dropped). There is no build from source: a Python version only installs once the pinned uv knows it, so if a Renovate Python bump fails the build, merge the uv bump from the "Build tools" PR first. Packages are installed with `uv pip install --system`. +- Python is not the system Python: `base/scripts/install-python.sh` installs a pinned uv (`UV_VERSION`, binary in `/usr/local/bin`), then a python-build-standalone build with `uv python install` in uv's own layout, `/opt/python/cpython--linux-x86_64-gnu/`. The Dockerfiles define that folder as `PYTHON_DIR` and put its `bin` on the `PATH` (python, python3, python3.X, pip and the commands packages install). Constraints behind this layout: it must not be uv's default location (`uv pip install --system` ignores Pythons there), nor be reached through a symlink (uv then computes broken paths when installing commands of platform wheels, e.g. `charset_normalizer` on Python 3.13). The script removes the `EXTERNALLY-MANAGED` marker uv adds, which would otherwise block `pip install` and `uv pip install --system`. There is no build from source: a Python version only installs once the pinned uv knows it, so if a Renovate Python bump fails the build, merge the uv bump from the "Build tools" PR first. Packages are installed with `uv pip install --system`. - `base/scripts/onyxia-init.sh` is the container init script used by Onyxia at startup (region init script, Vault secrets injection, CA bundles via `PATH_TO_CA_BUNDLE`, `PIP_REPOSITORY`/`R_REPOSITORY` mirrors — see README). ### CI (`.github/workflows/`) diff --git a/base/scripts/install-python.sh b/base/scripts/install-python.sh index f5a0bd74..fbbd8773 100755 --- a/base/scripts/install-python.sh +++ b/base/scripts/install-python.sh @@ -22,17 +22,15 @@ rm -rf "${TMP_DIR}" ### Install Python with uv prebuilt python binaries -# Install Python outside of uv's default location (~/.local/share/uv/python): `uv pip install --system` -# ignores the Pythons installed there. No cache, and no launchers in ~/.local/bin. -export UV_PYTHON_INSTALL_DIR="/opt/uv-python" +# Install Python in the parent folder of ${PYTHON_DIR} (set in the Dockerfile, and whose bin/ is on the PATH). +# It must not be uv's default location (~/.local/share/uv/python), which `uv pip install --system` ignores, +# nor be reached through a symlink, which breaks the paths uv computes when installing package commands. +# UV_PYTHON_INSTALL_DIR is only set here, so that users' own `uv python` commands never manage this Python. +UV_PYTHON_INSTALL_DIR=$(dirname "${PYTHON_DIR}") +export UV_PYTHON_INSTALL_DIR export UV_NO_CACHE=1 uv python install "${PYTHON_VERSION}" --no-bin -# Expose it at a stable path, ${PYTHON_DIR} (set in the Dockerfile), whose bin/ is on the PATH: -# python, python3, python3.X, pip and the commands installed by packages (jupyter, marimo...) -UV_PYTHON_EXECUTABLE=$(uv python find --managed-python "${PYTHON_VERSION}") -ln -s "$(dirname "$(dirname "$(realpath "${UV_PYTHON_EXECUTABLE}")")")" "${PYTHON_DIR}" - # uv marks its Pythons as externally managed, which would block `pip install` and `uv pip install --system` rm "${PYTHON_DIR}/lib/python${PYTHON_VERSION%.*}/EXTERNALLY-MANAGED" diff --git a/jupyter/tests.yaml b/jupyter/tests.yaml index 86bbe970..a5f58134 100644 --- a/jupyter/tests.yaml +++ b/jupyter/tests.yaml @@ -1,54 +1,54 @@ -schemaVersion: "2.0.0" -metadataTest: - user: 1000 - workdir: "/home/onyxia/work" - exposedPorts: ["8888"] -fileExistenceTests: - - name: "user home" - path: "/home/onyxia" - shouldExist: true - permissions: "drwxr-x---" - uid: 1000 - gid: 100 - - name: "onyxia init script" - path: "/opt/onyxia-init.sh" - shouldExist: true - permissions: "-rwxr-xr-x" - uid: 1000 - gid: 100 -commandTests: - - name: "Does the binary exists?" - command: "which" - args: ["helm"] - expectedOutput: ["/usr/local/bin/helm"] - - name: "Does the binary exists?" - command: "which" - args: ["kubectl"] - expectedOutput: ["/usr/local/bin/kubectl"] - - name: "Does the binary exists?" - command: "which" - args: ["python"] - expectedOutput: ['/opt/python/bin/python'] - - name: "Does the binary exists?" - command: "which" - args: ["pip"] - expectedOutput: ['/opt/python/bin/pip'] - - name: "Does the binary exists?" - command: "which" - args: ["jupyter-lab"] - expectedOutput: ["/opt/python/bin/jupyter-lab"] - - name: "Does the binary exists?" - command: "which" - args: ["duckdb"] - expectedOutput: ["/usr/local/bin/duckdb"] - - name: "Does the binary exists?" - command: "which" - args: ["opencode"] - expectedOutput: ["/usr/local/bin/opencode"] - - name: "JupyterLab runs" - command: "bash" - args: ["-c", "jupyter lab --version"] - - name: "Workspace is empty (no leftover install files)" - command: "ls" - args: ["-A", "/home/onyxia/work"] - excludedOutput: ["."] +schemaVersion: "2.0.0" +metadataTest: + user: 1000 + workdir: "/home/onyxia/work" + exposedPorts: ["8888"] +fileExistenceTests: + - name: "user home" + path: "/home/onyxia" + shouldExist: true + permissions: "drwxr-x---" + uid: 1000 + gid: 100 + - name: "onyxia init script" + path: "/opt/onyxia-init.sh" + shouldExist: true + permissions: "-rwxr-xr-x" + uid: 1000 + gid: 100 +commandTests: + - name: "Does the binary exists?" + command: "which" + args: ["helm"] + expectedOutput: ["/usr/local/bin/helm"] + - name: "Does the binary exists?" + command: "which" + args: ["kubectl"] + expectedOutput: ["/usr/local/bin/kubectl"] + - name: "Does the binary exists?" + command: "which" + args: ["python"] + expectedOutput: ['^/opt/python/cpython-[0-9.]+-linux-x86_64-gnu/bin/python$'] + - name: "Does the binary exists?" + command: "which" + args: ["pip"] + expectedOutput: ['^/opt/python/cpython-[0-9.]+-linux-x86_64-gnu/bin/pip$'] + - name: "Does the binary exists?" + command: "which" + args: ["jupyter-lab"] + expectedOutput: ['^/opt/python/cpython-[0-9.]+-linux-x86_64-gnu/bin/jupyter-lab$'] + - name: "Does the binary exists?" + command: "which" + args: ["duckdb"] + expectedOutput: ["/usr/local/bin/duckdb"] + - name: "Does the binary exists?" + command: "which" + args: ["opencode"] + expectedOutput: ["/usr/local/bin/opencode"] + - name: "JupyterLab runs" + command: "bash" + args: ["-c", "jupyter lab --version"] + - name: "Workspace is empty (no leftover install files)" + command: "ls" + args: ["-A", "/home/onyxia/work"] + excludedOutput: ["."] diff --git a/marimo/tests.yaml b/marimo/tests.yaml index f6f50979..ba0a9ec1 100644 --- a/marimo/tests.yaml +++ b/marimo/tests.yaml @@ -28,7 +28,7 @@ commandTests: - name: "Does the binary exists?" command: "which" args: ["marimo"] - expectedOutput: ["/opt/python/bin/marimo"] + expectedOutput: ['^/opt/python/cpython-[0-9.]+-linux-x86_64-gnu/bin/marimo$'] - name: "Does the binary exists?" command: "which" args: ["duckdb"] diff --git a/python-datascience/tests.yaml b/python-datascience/tests.yaml index ef963403..87ec9b9a 100644 --- a/python-datascience/tests.yaml +++ b/python-datascience/tests.yaml @@ -1,46 +1,46 @@ -schemaVersion: "2.0.0" -metadataTest: - user: 1000 - workdir: "/home/onyxia/work" -fileExistenceTests: - - name: "user home" - path: "/home/onyxia" - shouldExist: true - permissions: "drwxr-x---" - uid: 1000 - gid: 100 - - name: "onyxia init script" - path: "/opt/onyxia-init.sh" - shouldExist: true - permissions: "-rwxr-xr-x" - uid: 1000 - gid: 100 -commandTests: - - name: "Does the binary exists?" - command: "which" - args: ["helm"] - expectedOutput: ["/usr/local/bin/helm"] - - name: "Does the binary exists?" - command: "which" - args: ["kubectl"] - expectedOutput: ["/usr/local/bin/kubectl"] - - name: "Does the binary exists?" - command: "which" - args: ["python"] - expectedOutput: ['/opt/python/bin/python'] - - name: "Does the binary exists?" - command: "which" - args: ["pip"] - expectedOutput: ['/opt/python/bin/pip'] - - name: "Does the binary exists?" - command: "which" - args: ["quarto"] - expectedOutput: ["/usr/local/bin/quarto"] - - name: "Does the binary exists?" - command: "which" - args: ["duckdb"] - expectedOutput: ["/usr/local/bin/duckdb"] - - name: "Workspace is empty (no leftover install files)" - command: "ls" - args: ["-A", "/home/onyxia/work"] - excludedOutput: ["."] +schemaVersion: "2.0.0" +metadataTest: + user: 1000 + workdir: "/home/onyxia/work" +fileExistenceTests: + - name: "user home" + path: "/home/onyxia" + shouldExist: true + permissions: "drwxr-x---" + uid: 1000 + gid: 100 + - name: "onyxia init script" + path: "/opt/onyxia-init.sh" + shouldExist: true + permissions: "-rwxr-xr-x" + uid: 1000 + gid: 100 +commandTests: + - name: "Does the binary exists?" + command: "which" + args: ["helm"] + expectedOutput: ["/usr/local/bin/helm"] + - name: "Does the binary exists?" + command: "which" + args: ["kubectl"] + expectedOutput: ["/usr/local/bin/kubectl"] + - name: "Does the binary exists?" + command: "which" + args: ["python"] + expectedOutput: ['^/opt/python/cpython-[0-9.]+-linux-x86_64-gnu/bin/python$'] + - name: "Does the binary exists?" + command: "which" + args: ["pip"] + expectedOutput: ['^/opt/python/cpython-[0-9.]+-linux-x86_64-gnu/bin/pip$'] + - name: "Does the binary exists?" + command: "which" + args: ["quarto"] + expectedOutput: ["/usr/local/bin/quarto"] + - name: "Does the binary exists?" + command: "which" + args: ["duckdb"] + expectedOutput: ["/usr/local/bin/duckdb"] + - name: "Workspace is empty (no leftover install files)" + command: "ls" + args: ["-A", "/home/onyxia/work"] + excludedOutput: ["."] diff --git a/python-minimal/Dockerfile b/python-minimal/Dockerfile index 7a8a39bf..68766cfe 100644 --- a/python-minimal/Dockerfile +++ b/python-minimal/Dockerfile @@ -6,8 +6,8 @@ LABEL maintainer="InseeFrLab " ARG PYTHON_VERSION="3.13.15" ENV PYTHON_VERSION=${PYTHON_VERSION} -# Install Python separately from system installation to avoid conflicts -ARG PYTHON_DIR="/opt/python" +# Python is installed by uv (see /opt/install-python.sh) in /opt/python, in the folder uv names after the version +ARG PYTHON_DIR="/opt/python/cpython-${PYTHON_VERSION}-linux-x86_64-gnu" ENV PATH="${PYTHON_DIR}/bin:${PATH}" USER root diff --git a/python-minimal/tests.yaml b/python-minimal/tests.yaml index af444174..1f32ebd3 100644 --- a/python-minimal/tests.yaml +++ b/python-minimal/tests.yaml @@ -1,48 +1,48 @@ -schemaVersion: "2.0.0" -metadataTest: - user: 1000 - workdir: "/home/onyxia/work" -fileExistenceTests: - - name: "user home" - path: "/home/onyxia" - shouldExist: true - permissions: "drwxr-x---" - uid: 1000 - gid: 100 - - name: "onyxia init script" - path: "/opt/onyxia-init.sh" - shouldExist: true - permissions: "-rwxr-xr-x" - uid: 1000 - gid: 100 -commandTests: - - name: "Does the binary exists?" - command: "which" - args: ["helm"] - expectedOutput: ["/usr/local/bin/helm"] - - name: "Does the binary exists?" - command: "which" - args: ["kubectl"] - expectedOutput: ["/usr/local/bin/kubectl"] - - name: "Does the binary exists?" - command: "which" - args: ["python"] - expectedOutput: ['/opt/python/bin/python'] - - name: "Does the binary exists?" - command: "which" - args: ["pip"] - expectedOutput: ['/opt/python/bin/pip'] - - name: "Does the binary exists?" - command: "which" - args: ["duckdb"] - expectedOutput: ["/usr/local/bin/duckdb"] - - name: "Python is the requested version" - command: "bash" - args: ["-c", "python --version | grep -F \"Python ${PYTHON_VERSION}\""] - - name: "uv runs" - command: "uv" - args: ["--version"] - - name: "Workspace is empty (no leftover install files)" - command: "ls" - args: ["-A", "/home/onyxia/work"] - excludedOutput: ["."] +schemaVersion: "2.0.0" +metadataTest: + user: 1000 + workdir: "/home/onyxia/work" +fileExistenceTests: + - name: "user home" + path: "/home/onyxia" + shouldExist: true + permissions: "drwxr-x---" + uid: 1000 + gid: 100 + - name: "onyxia init script" + path: "/opt/onyxia-init.sh" + shouldExist: true + permissions: "-rwxr-xr-x" + uid: 1000 + gid: 100 +commandTests: + - name: "Does the binary exists?" + command: "which" + args: ["helm"] + expectedOutput: ["/usr/local/bin/helm"] + - name: "Does the binary exists?" + command: "which" + args: ["kubectl"] + expectedOutput: ["/usr/local/bin/kubectl"] + - name: "Does the binary exists?" + command: "which" + args: ["python"] + expectedOutput: ['^/opt/python/cpython-[0-9.]+-linux-x86_64-gnu/bin/python$'] + - name: "Does the binary exists?" + command: "which" + args: ["pip"] + expectedOutput: ['^/opt/python/cpython-[0-9.]+-linux-x86_64-gnu/bin/pip$'] + - name: "Does the binary exists?" + command: "which" + args: ["duckdb"] + expectedOutput: ["/usr/local/bin/duckdb"] + - name: "Python is the requested version" + command: "bash" + args: ["-c", "python --version | grep -F \"Python ${PYTHON_VERSION}\""] + - name: "uv runs" + command: "uv" + args: ["--version"] + - name: "Workspace is empty (no leftover install files)" + command: "ls" + args: ["-A", "/home/onyxia/work"] + excludedOutput: ["."] diff --git a/python-pytorch/tests.yaml b/python-pytorch/tests.yaml index a614673b..bfc060a7 100644 --- a/python-pytorch/tests.yaml +++ b/python-pytorch/tests.yaml @@ -1,45 +1,45 @@ -schemaVersion: "2.0.0" -metadataTest: - user: 1000 - workdir: "/home/onyxia/work" -fileExistenceTests: - - name: "user home" - path: "/home/onyxia" - shouldExist: true - permissions: "drwxr-x---" - uid: 1000 - gid: 100 - - name: "onyxia init script" - path: "/opt/onyxia-init.sh" - shouldExist: true - permissions: "-rwxr-xr-x" - uid: 1000 - gid: 100 -commandTests: - - name: "Does the binary exists?" - command: "which" - args: ["helm"] - expectedOutput: ["/usr/local/bin/helm"] - - name: "Does the binary exists?" - command: "which" - args: ["kubectl"] - expectedOutput: ["/usr/local/bin/kubectl"] - - name: "Does the binary exists?" - command: "which" - args: ["python"] - expectedOutput: ['/opt/python/bin/python'] - - name: "Does the binary exists?" - command: "which" - args: ["pip"] - expectedOutput: ['/opt/python/bin/pip'] - - name: "Does the binary exists?" - command: "which" - args: ["duckdb"] - expectedOutput: ["/usr/local/bin/duckdb"] - - name: "PyTorch runs" - command: "python" - args: ["-c", "import torch"] - - name: "Workspace is empty (no leftover install files)" - command: "ls" - args: ["-A", "/home/onyxia/work"] - excludedOutput: ["."] +schemaVersion: "2.0.0" +metadataTest: + user: 1000 + workdir: "/home/onyxia/work" +fileExistenceTests: + - name: "user home" + path: "/home/onyxia" + shouldExist: true + permissions: "drwxr-x---" + uid: 1000 + gid: 100 + - name: "onyxia init script" + path: "/opt/onyxia-init.sh" + shouldExist: true + permissions: "-rwxr-xr-x" + uid: 1000 + gid: 100 +commandTests: + - name: "Does the binary exists?" + command: "which" + args: ["helm"] + expectedOutput: ["/usr/local/bin/helm"] + - name: "Does the binary exists?" + command: "which" + args: ["kubectl"] + expectedOutput: ["/usr/local/bin/kubectl"] + - name: "Does the binary exists?" + command: "which" + args: ["python"] + expectedOutput: ['^/opt/python/cpython-[0-9.]+-linux-x86_64-gnu/bin/python$'] + - name: "Does the binary exists?" + command: "which" + args: ["pip"] + expectedOutput: ['^/opt/python/cpython-[0-9.]+-linux-x86_64-gnu/bin/pip$'] + - name: "Does the binary exists?" + command: "which" + args: ["duckdb"] + expectedOutput: ["/usr/local/bin/duckdb"] + - name: "PyTorch runs" + command: "python" + args: ["-c", "import torch"] + - name: "Workspace is empty (no leftover install files)" + command: "ls" + args: ["-A", "/home/onyxia/work"] + excludedOutput: ["."] diff --git a/r-python-julia/Dockerfile b/r-python-julia/Dockerfile index 3f799885..083da682 100644 --- a/r-python-julia/Dockerfile +++ b/r-python-julia/Dockerfile @@ -6,8 +6,8 @@ LABEL maintainer="InseeFrLab " ARG PYTHON_VERSION="3.13.15" ENV PYTHON_VERSION=${PYTHON_VERSION} -# Install Python separately from system installation to avoid conflicts -ARG PYTHON_DIR="/opt/python" +# Python is installed by uv (see /opt/install-python.sh) in /opt/python, in the folder uv names after the version +ARG PYTHON_DIR="/opt/python/cpython-${PYTHON_VERSION}-linux-x86_64-gnu" ENV PATH="${PYTHON_DIR}/bin:${PATH}" USER root diff --git a/r-python-julia/tests.yaml b/r-python-julia/tests.yaml index 2c2f80f1..7db29cc4 100644 --- a/r-python-julia/tests.yaml +++ b/r-python-julia/tests.yaml @@ -1,56 +1,56 @@ -schemaVersion: "2.0.0" -metadataTest: - user: 1000 - workdir: "/home/onyxia/work" -fileExistenceTests: - - name: "user home" - path: "/home/onyxia" - shouldExist: true - permissions: "drwxr-x---" - uid: 1000 - gid: 100 - - name: "onyxia init script" - path: "/opt/onyxia-init.sh" - shouldExist: true - permissions: "-rwxr-xr-x" - uid: 1000 - gid: 100 -commandTests: - - name: "Does the binary exists?" - command: "which" - args: ["helm"] - expectedOutput: ["/usr/local/bin/helm"] - - name: "Does the binary exists?" - command: "which" - args: ["kubectl"] - expectedOutput: ["/usr/local/bin/kubectl"] - - name: "Does the binary exists?" - command: "which" - args: ["R"] - expectedOutput: ["/usr/local/bin/R"] - - name: "Does the binary exists?" - command: "which" - args: ["python"] - expectedOutput: ['/opt/python/bin/python'] - - name: "Does the binary exists?" - command: "which" - args: ["pip"] - expectedOutput: ['/opt/python/bin/pip'] - - name: "Does the binary exists?" - command: "which" - args: ["julia"] - expectedOutput: ["/usr/local/bin/julia"] - - name: "Does the binary exists?" - command: "which" - args: ["duckdb"] - expectedOutput: ["/usr/local/bin/duckdb"] - - name: "Python is the requested version" - command: "bash" - args: ["-c", "python --version | grep -F \"Python ${PYTHON_VERSION}\""] - - name: "Julia runs and main package load" - command: "julia" - args: ["-e", "using HDF5"] - - name: "Workspace is empty (no leftover install files)" - command: "ls" - args: ["-A", "/home/onyxia/work"] - excludedOutput: ["."] +schemaVersion: "2.0.0" +metadataTest: + user: 1000 + workdir: "/home/onyxia/work" +fileExistenceTests: + - name: "user home" + path: "/home/onyxia" + shouldExist: true + permissions: "drwxr-x---" + uid: 1000 + gid: 100 + - name: "onyxia init script" + path: "/opt/onyxia-init.sh" + shouldExist: true + permissions: "-rwxr-xr-x" + uid: 1000 + gid: 100 +commandTests: + - name: "Does the binary exists?" + command: "which" + args: ["helm"] + expectedOutput: ["/usr/local/bin/helm"] + - name: "Does the binary exists?" + command: "which" + args: ["kubectl"] + expectedOutput: ["/usr/local/bin/kubectl"] + - name: "Does the binary exists?" + command: "which" + args: ["R"] + expectedOutput: ["/usr/local/bin/R"] + - name: "Does the binary exists?" + command: "which" + args: ["python"] + expectedOutput: ['^/opt/python/cpython-[0-9.]+-linux-x86_64-gnu/bin/python$'] + - name: "Does the binary exists?" + command: "which" + args: ["pip"] + expectedOutput: ['^/opt/python/cpython-[0-9.]+-linux-x86_64-gnu/bin/pip$'] + - name: "Does the binary exists?" + command: "which" + args: ["julia"] + expectedOutput: ["/usr/local/bin/julia"] + - name: "Does the binary exists?" + command: "which" + args: ["duckdb"] + expectedOutput: ["/usr/local/bin/duckdb"] + - name: "Python is the requested version" + command: "bash" + args: ["-c", "python --version | grep -F \"Python ${PYTHON_VERSION}\""] + - name: "Julia runs and main package load" + command: "julia" + args: ["-e", "using HDF5"] + - name: "Workspace is empty (no leftover install files)" + command: "ls" + args: ["-A", "/home/onyxia/work"] + excludedOutput: ["."] diff --git a/versions.env b/versions.env index 8af8440a..ba4c2ac3 100644 --- a/versions.env +++ b/versions.env @@ -1,5 +1,5 @@ # Single source of truth for the versions actively maintained in the images, read by the CI build matrix -# (utils/generate_matrix.py) and by local builds (utils/build_chains.sh, utils/build_chain.py). +# (utils/generate_matrix.py) and by local build scripts (utils/build_chains.sh, utils/build_chain.py). # Updated by Renovate, which also keeps the ARG defaults of the Dockerfiles in sync. BASE_IMAGE_CPU="ubuntu:24.04" From d374c4d0678fe1e7436e9fc52ea7d46a01199b27 Mon Sep 17 00:00:00 2001 From: avouacr Date: Sat, 19 Sep 2026 20:37:00 +0000 Subject: [PATCH 07/14] dev --- AGENTS.md | 6 +- base/scripts/install-python.sh | 78 ++++++++++++---------- docs/improvement-plan.md | 10 +-- jupyter/tests.yaml | 108 +++++++++++++++--------------- marimo/tests.yaml | 2 +- python-datascience/tests.yaml | 92 +++++++++++++------------- python-minimal/Dockerfile | 4 +- python-minimal/requirements.txt | 1 + python-minimal/tests.yaml | 96 +++++++++++++-------------- python-pytorch/tests.yaml | 90 ++++++++++++------------- r-python-julia/Dockerfile | 4 +- r-python-julia/tests.yaml | 112 ++++++++++++++++---------------- 12 files changed, 304 insertions(+), 299 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 02074102..5a879819 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -59,7 +59,7 @@ The valid layer stacks are declared in the `chains` dict in `utils/build_chain.p - Downloads in build scripts must fail loudly: use `wget -nv` (never `-q`, which also hides error messages) and `curl -fsSL` (`-f` fails on HTTP errors, `-S` prints them despite `-s`). Put version lookups (`VERSION=$(curl ...)`) on their own line, and use `set -o pipefail` when they are piped, so that a failed lookup stops the build instead of producing a broken URL. - Third-party apt repositories (PostgreSQL in `base/scripts/install-system-libs.sh`, ubuntugis in `python-datascience/scripts/install-geospatial-python.sh`) are added only for the packages that need them, then removed right after, so that later installs only come from Ubuntu. Install from the repository every package whose Ubuntu counterpart would require an exact version of it (e.g. `libpq-dev` with `postgresql-client`). - Runtime user is `onyxia` (UID 1000, group `users` GID 100) with passwordless sudo; workdir is `/home/onyxia/work`. -- Python is not the system Python: `base/scripts/install-python.sh` installs a pinned uv (`UV_VERSION`, binary in `/usr/local/bin`), then a python-build-standalone build with `uv python install` in uv's own layout, `/opt/python/cpython--linux-x86_64-gnu/`. The Dockerfiles define that folder as `PYTHON_DIR` and put its `bin` on the `PATH` (python, python3, python3.X, pip and the commands packages install). Constraints behind this layout: it must not be uv's default location (`uv pip install --system` ignores Pythons there), nor be reached through a symlink (uv then computes broken paths when installing commands of platform wheels, e.g. `charset_normalizer` on Python 3.13). The script removes the `EXTERNALLY-MANAGED` marker uv adds, which would otherwise block `pip install` and `uv pip install --system`. There is no build from source: a Python version only installs once the pinned uv knows it, so if a Renovate Python bump fails the build, merge the uv bump from the "Build tools" PR first. Packages are installed with `uv pip install --system`. +- Python is built from source into `/opt/python` (not the system Python); packages are installed with `uv pip install --system`. R comes from rocker scripts (`/rocker_scripts/...`), with `R_HOME=/usr/local/lib/R`. - `base/scripts/onyxia-init.sh` is the container init script used by Onyxia at startup (region init script, Vault secrets injection, CA bundles via `PATH_TO_CA_BUNDLE`, `PIP_REPOSITORY`/`R_REPOSITORY` mirrors — see README). ### CI (`.github/workflows/`) @@ -74,7 +74,7 @@ The valid layer stacks are declared in the `chains` dict in `utils/build_chain.p `versions.env` (repo root) is the single source of truth for the Python, R and Spark versions and for the CUDA base image of GPU variants. It is read by the CI matrix (`utils/generate_matrix.py`, through `utils/versions.py`) and by local builds (`utils/build_chains.sh` sources it, `utils/build_chain.py` reads the CUDA image). The only copies are the `ARG` defaults of the `python-minimal`, `r-python-julia`, `r-minimal` and `spark` Dockerfiles, which CI always overrides. `renovate.json` has one regex manager per version slot (Python 1/2, R 1/2, Spark) covering `versions.env` and those Dockerfiles, and groups each slot into a single PR; version 2 only gets patch bumps, so moving it to a new minor release (e.g. when version 1 moves on) is a manual change. A separate manager tracks the CUDA image. When changing a version by hand, edit `versions.env` and the matching Dockerfile `ARG` default. -Tools downloaded at build time (kubectl, helm, AWS CLI, DuckDB CLI, quarto, opencode, Julia, code-server, uv) are pinned in their install script, **not** in Dockerfiles (Dockerfiles only pin the versions the project manages: Python, R, Spark). Each script follows the same model, see `base/scripts/install-kubectl.sh`: +Tools downloaded at build time (kubectl, helm, AWS CLI, DuckDB CLI, quarto, opencode, Julia, code-server) are pinned in their install script, **not** in Dockerfiles (Dockerfiles only pin the versions the project manages: Python, R, Spark). Each script follows the same model, see `base/scripts/install-kubectl.sh`: - a `# renovate: datasource= depName=` comment right above a `_VERSION="x.y.z"` line. One generic regex manager in `renovate.json` picks it up (in `scripts/*.sh` and in `.github/actions/*/action.yml`, where container-structure-test is pinned the same way), and all tool bumps land in a single weekly "Build tools" PR. -- the official release file is downloaded to a `mktemp -d` directory and verified before installing: against the upstream checksum file when there is one (kubectl, helm, quarto, Julia, uv), against the SHA-256 GitHub records for the release asset (read from the GitHub API in the script, with `set -o pipefail`) when there is none (DuckDB, opencode, code-server), and with the PGP signature for the AWS CLI (the public key is embedded in `install-awscli.sh` and expires on 2027-07-01). +- the official release file is downloaded to a `mktemp -d` directory and verified before installing: against the upstream checksum file when there is one (kubectl, helm, quarto, Julia), against the SHA-256 GitHub records for the release asset (read from the GitHub API in the script, with `set -o pipefail`) when there is none (DuckDB, opencode, code-server), and with the PGP signature for the AWS CLI (the public key is embedded in `install-awscli.sh` and expires on 2027-07-01). - amd64 only. diff --git a/base/scripts/install-python.sh b/base/scripts/install-python.sh index fbbd8773..188490fd 100755 --- a/base/scripts/install-python.sh +++ b/base/scripts/install-python.sh @@ -1,42 +1,54 @@ #!/bin/bash set -e -### Install uv - -# renovate: datasource=github-releases depName=astral-sh/uv -UV_VERSION="0.12.17" - -# Download the official archive and verify its checksum -UV_ARCHIVE="uv-x86_64-unknown-linux-gnu.tar.gz" -UV_URL="https://github.com/astral-sh/uv/releases/download/${UV_VERSION}" -TMP_DIR=$(mktemp -d) -curl -fsSL "${UV_URL}/${UV_ARCHIVE}" -o "${TMP_DIR}/${UV_ARCHIVE}" -curl -fsSL "${UV_URL}/${UV_ARCHIVE}.sha256" -o "${TMP_DIR}/${UV_ARCHIVE}.sha256" -(cd "${TMP_DIR}" && sha256sum --check --strict "${UV_ARCHIVE}.sha256") - -# Install uv -tar -xzf "${TMP_DIR}/${UV_ARCHIVE}" -C "${TMP_DIR}" -install -m 0755 "${TMP_DIR}/uv-x86_64-unknown-linux-gnu/uv" "${TMP_DIR}/uv-x86_64-unknown-linux-gnu/uvx" /usr/local/bin/ -rm -rf "${TMP_DIR}" - - -### Install Python with uv prebuilt python binaries - -# Install Python in the parent folder of ${PYTHON_DIR} (set in the Dockerfile, and whose bin/ is on the PATH). -# It must not be uv's default location (~/.local/share/uv/python), which `uv pip install --system` ignores, -# nor be reached through a symlink, which breaks the paths uv computes when installing package commands. -# UV_PYTHON_INSTALL_DIR is only set here, so that users' own `uv python` commands never manage this Python. -UV_PYTHON_INSTALL_DIR=$(dirname "${PYTHON_DIR}") -export UV_PYTHON_INSTALL_DIR -export UV_NO_CACHE=1 -uv python install "${PYTHON_VERSION}" --no-bin - -# uv marks its Pythons as externally managed, which would block `pip install` and `uv pip install --system` -rm "${PYTHON_DIR}/lib/python${PYTHON_VERSION%.*}/EXTERNALLY-MANAGED" +# Install system libraries required to build Python from source +savedAptMark="$(apt-mark showmanual)" +/opt/apt-install.sh \ + dpkg-dev \ + libbz2-dev \ + libc6-dev \ + libdb-dev \ + libffi-dev \ + libgdbm-dev \ + liblzma-dev \ + libncursesw5-dev \ + libreadline-dev \ + libsqlite3-dev \ + libssl-dev \ + tk-dev \ + uuid-dev \ + xz-utils \ + zlib1g-dev + +# Build Python from sources +wget -nv https://www.python.org/ftp/python/${PYTHON_VERSION}/Python-${PYTHON_VERSION}.tgz +tar xzvf Python-${PYTHON_VERSION}.tgz +cd Python-${PYTHON_VERSION} +./configure \ + --prefix="${PYTHON_DIR}" \ + --enable-loadable-sqlite-extensions \ + --enable-optimizations \ + --enable-shared \ + --with-lto \ + LDFLAGS="-Wl,-rpath ${PYTHON_DIR}/lib" +make -j"$(nproc)" +make install + +# Clean install files +cd .. +rm -rf Python-${PYTHON_VERSION}.tgz Python-${PYTHON_VERSION} +apt-mark auto '.*' > /dev/null +apt-mark manual $savedAptMark +apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false + +# Useful symlinks +ln -sf "${PYTHON_DIR}/bin/python3" "${PYTHON_DIR}/bin/python" +ln -sf "${PYTHON_DIR}/bin/pip3" "${PYTHON_DIR}/bin/pip" # Checks python --version which python -# Upgrade pip +# Upgrade pip & install uv pip install --no-cache-dir --upgrade pip +pip install --no-cache-dir uv diff --git a/docs/improvement-plan.md b/docs/improvement-plan.md index c7a98681..f12a29b3 100644 --- a/docs/improvement-plan.md +++ b/docs/improvement-plan.md @@ -44,14 +44,6 @@ Published sizes on 2026-09-19 (compressed, amd64): - **GPU testing:** GPU images are not tested in CI because they are too big for GitHub-hosted runners to load and test. Once sizes are reduced, check whether they fit, and test them like the CPU ones if so. <<<<<<< Updated upstream -### 13. Python compiled from source — M - -`base/scripts/install-python.sh` compiles CPython with `--enable-optimizations --with-lto`. That sits on the critical path of every Python chain (python-minimal and r-python-julia), × 2 versions × CPU/GPU. - -- Fix: install prebuilt python-build-standalone binaries (also PGO+LTO-optimized) with `uv python install` into `/opt/python`. -- The tests expect `/opt/python/bin/python` and `/opt/python/bin/pip`, so `python` and `pip` symlinks may be needed. -- Check that C-extension builds still work, e.g. GDAL's Python bindings in python-datascience. - ### 14. Container startup work — S At every start, the end of `base/scripts/onyxia-init.sh` runs `chown -R` over every folder in `$ROOT_PROJECT_DIRECTORY`. That's slow on volumes with virtualenvs or data, and useless when the script runs as `onyxia` (Jupyter, VS Code), since a normal user can't change file owners. @@ -101,4 +93,4 @@ Fix: a single `images.yaml` (layers, parents, languages, GPU flag) that drives b - **`import tkinter` is broken in the images:** irrelevant for headless images. - **The `chown -R` over `~/work` at startup stays:** the folder only holds what was just cloned or added by an init script, and the user must own all of it. - **The image graph stays declared twice** (`chains` in `utils/build_chain.py`, jobs in `main-workflow.yml`): no generated CI. Adding an image means updating both. -- **Python is installed with uv only,** no build from source: the `EXTERNALLY-MANAGED` marker uv adds is removed, and Python is installed outside uv's default location so that `uv pip install --system` uses it. +- **Python stays built from source** (PGO + LTO): a uv-installed Python (python-build-standalone) was tried twice. Its `EXTERNALLY-MANAGED` marker blocks `pip install` / `uv pip install --system`, `uv pip install --system` ignores Pythons in uv's default location, and a symlinked install breaks uv's paths for package commands (e.g. `charset_normalizer` on 3.13). The source build works as is. diff --git a/jupyter/tests.yaml b/jupyter/tests.yaml index a5f58134..86bbe970 100644 --- a/jupyter/tests.yaml +++ b/jupyter/tests.yaml @@ -1,54 +1,54 @@ -schemaVersion: "2.0.0" -metadataTest: - user: 1000 - workdir: "/home/onyxia/work" - exposedPorts: ["8888"] -fileExistenceTests: - - name: "user home" - path: "/home/onyxia" - shouldExist: true - permissions: "drwxr-x---" - uid: 1000 - gid: 100 - - name: "onyxia init script" - path: "/opt/onyxia-init.sh" - shouldExist: true - permissions: "-rwxr-xr-x" - uid: 1000 - gid: 100 -commandTests: - - name: "Does the binary exists?" - command: "which" - args: ["helm"] - expectedOutput: ["/usr/local/bin/helm"] - - name: "Does the binary exists?" - command: "which" - args: ["kubectl"] - expectedOutput: ["/usr/local/bin/kubectl"] - - name: "Does the binary exists?" - command: "which" - args: ["python"] - expectedOutput: ['^/opt/python/cpython-[0-9.]+-linux-x86_64-gnu/bin/python$'] - - name: "Does the binary exists?" - command: "which" - args: ["pip"] - expectedOutput: ['^/opt/python/cpython-[0-9.]+-linux-x86_64-gnu/bin/pip$'] - - name: "Does the binary exists?" - command: "which" - args: ["jupyter-lab"] - expectedOutput: ['^/opt/python/cpython-[0-9.]+-linux-x86_64-gnu/bin/jupyter-lab$'] - - name: "Does the binary exists?" - command: "which" - args: ["duckdb"] - expectedOutput: ["/usr/local/bin/duckdb"] - - name: "Does the binary exists?" - command: "which" - args: ["opencode"] - expectedOutput: ["/usr/local/bin/opencode"] - - name: "JupyterLab runs" - command: "bash" - args: ["-c", "jupyter lab --version"] - - name: "Workspace is empty (no leftover install files)" - command: "ls" - args: ["-A", "/home/onyxia/work"] - excludedOutput: ["."] +schemaVersion: "2.0.0" +metadataTest: + user: 1000 + workdir: "/home/onyxia/work" + exposedPorts: ["8888"] +fileExistenceTests: + - name: "user home" + path: "/home/onyxia" + shouldExist: true + permissions: "drwxr-x---" + uid: 1000 + gid: 100 + - name: "onyxia init script" + path: "/opt/onyxia-init.sh" + shouldExist: true + permissions: "-rwxr-xr-x" + uid: 1000 + gid: 100 +commandTests: + - name: "Does the binary exists?" + command: "which" + args: ["helm"] + expectedOutput: ["/usr/local/bin/helm"] + - name: "Does the binary exists?" + command: "which" + args: ["kubectl"] + expectedOutput: ["/usr/local/bin/kubectl"] + - name: "Does the binary exists?" + command: "which" + args: ["python"] + expectedOutput: ['/opt/python/bin/python'] + - name: "Does the binary exists?" + command: "which" + args: ["pip"] + expectedOutput: ['/opt/python/bin/pip'] + - name: "Does the binary exists?" + command: "which" + args: ["jupyter-lab"] + expectedOutput: ["/opt/python/bin/jupyter-lab"] + - name: "Does the binary exists?" + command: "which" + args: ["duckdb"] + expectedOutput: ["/usr/local/bin/duckdb"] + - name: "Does the binary exists?" + command: "which" + args: ["opencode"] + expectedOutput: ["/usr/local/bin/opencode"] + - name: "JupyterLab runs" + command: "bash" + args: ["-c", "jupyter lab --version"] + - name: "Workspace is empty (no leftover install files)" + command: "ls" + args: ["-A", "/home/onyxia/work"] + excludedOutput: ["."] diff --git a/marimo/tests.yaml b/marimo/tests.yaml index ba0a9ec1..f6f50979 100644 --- a/marimo/tests.yaml +++ b/marimo/tests.yaml @@ -28,7 +28,7 @@ commandTests: - name: "Does the binary exists?" command: "which" args: ["marimo"] - expectedOutput: ['^/opt/python/cpython-[0-9.]+-linux-x86_64-gnu/bin/marimo$'] + expectedOutput: ["/opt/python/bin/marimo"] - name: "Does the binary exists?" command: "which" args: ["duckdb"] diff --git a/python-datascience/tests.yaml b/python-datascience/tests.yaml index 87ec9b9a..ef963403 100644 --- a/python-datascience/tests.yaml +++ b/python-datascience/tests.yaml @@ -1,46 +1,46 @@ -schemaVersion: "2.0.0" -metadataTest: - user: 1000 - workdir: "/home/onyxia/work" -fileExistenceTests: - - name: "user home" - path: "/home/onyxia" - shouldExist: true - permissions: "drwxr-x---" - uid: 1000 - gid: 100 - - name: "onyxia init script" - path: "/opt/onyxia-init.sh" - shouldExist: true - permissions: "-rwxr-xr-x" - uid: 1000 - gid: 100 -commandTests: - - name: "Does the binary exists?" - command: "which" - args: ["helm"] - expectedOutput: ["/usr/local/bin/helm"] - - name: "Does the binary exists?" - command: "which" - args: ["kubectl"] - expectedOutput: ["/usr/local/bin/kubectl"] - - name: "Does the binary exists?" - command: "which" - args: ["python"] - expectedOutput: ['^/opt/python/cpython-[0-9.]+-linux-x86_64-gnu/bin/python$'] - - name: "Does the binary exists?" - command: "which" - args: ["pip"] - expectedOutput: ['^/opt/python/cpython-[0-9.]+-linux-x86_64-gnu/bin/pip$'] - - name: "Does the binary exists?" - command: "which" - args: ["quarto"] - expectedOutput: ["/usr/local/bin/quarto"] - - name: "Does the binary exists?" - command: "which" - args: ["duckdb"] - expectedOutput: ["/usr/local/bin/duckdb"] - - name: "Workspace is empty (no leftover install files)" - command: "ls" - args: ["-A", "/home/onyxia/work"] - excludedOutput: ["."] +schemaVersion: "2.0.0" +metadataTest: + user: 1000 + workdir: "/home/onyxia/work" +fileExistenceTests: + - name: "user home" + path: "/home/onyxia" + shouldExist: true + permissions: "drwxr-x---" + uid: 1000 + gid: 100 + - name: "onyxia init script" + path: "/opt/onyxia-init.sh" + shouldExist: true + permissions: "-rwxr-xr-x" + uid: 1000 + gid: 100 +commandTests: + - name: "Does the binary exists?" + command: "which" + args: ["helm"] + expectedOutput: ["/usr/local/bin/helm"] + - name: "Does the binary exists?" + command: "which" + args: ["kubectl"] + expectedOutput: ["/usr/local/bin/kubectl"] + - name: "Does the binary exists?" + command: "which" + args: ["python"] + expectedOutput: ['/opt/python/bin/python'] + - name: "Does the binary exists?" + command: "which" + args: ["pip"] + expectedOutput: ['/opt/python/bin/pip'] + - name: "Does the binary exists?" + command: "which" + args: ["quarto"] + expectedOutput: ["/usr/local/bin/quarto"] + - name: "Does the binary exists?" + command: "which" + args: ["duckdb"] + expectedOutput: ["/usr/local/bin/duckdb"] + - name: "Workspace is empty (no leftover install files)" + command: "ls" + args: ["-A", "/home/onyxia/work"] + excludedOutput: ["."] diff --git a/python-minimal/Dockerfile b/python-minimal/Dockerfile index 68766cfe..7a8a39bf 100644 --- a/python-minimal/Dockerfile +++ b/python-minimal/Dockerfile @@ -6,8 +6,8 @@ LABEL maintainer="InseeFrLab " ARG PYTHON_VERSION="3.13.15" ENV PYTHON_VERSION=${PYTHON_VERSION} -# Python is installed by uv (see /opt/install-python.sh) in /opt/python, in the folder uv names after the version -ARG PYTHON_DIR="/opt/python/cpython-${PYTHON_VERSION}-linux-x86_64-gnu" +# Install Python separately from system installation to avoid conflicts +ARG PYTHON_DIR="/opt/python" ENV PATH="${PYTHON_DIR}/bin:${PATH}" USER root diff --git a/python-minimal/requirements.txt b/python-minimal/requirements.txt index 40e9f260..04a7b6e1 100644 --- a/python-minimal/requirements.txt +++ b/python-minimal/requirements.txt @@ -7,3 +7,4 @@ pandas pyarrow requests s3fs +uv diff --git a/python-minimal/tests.yaml b/python-minimal/tests.yaml index 1f32ebd3..af444174 100644 --- a/python-minimal/tests.yaml +++ b/python-minimal/tests.yaml @@ -1,48 +1,48 @@ -schemaVersion: "2.0.0" -metadataTest: - user: 1000 - workdir: "/home/onyxia/work" -fileExistenceTests: - - name: "user home" - path: "/home/onyxia" - shouldExist: true - permissions: "drwxr-x---" - uid: 1000 - gid: 100 - - name: "onyxia init script" - path: "/opt/onyxia-init.sh" - shouldExist: true - permissions: "-rwxr-xr-x" - uid: 1000 - gid: 100 -commandTests: - - name: "Does the binary exists?" - command: "which" - args: ["helm"] - expectedOutput: ["/usr/local/bin/helm"] - - name: "Does the binary exists?" - command: "which" - args: ["kubectl"] - expectedOutput: ["/usr/local/bin/kubectl"] - - name: "Does the binary exists?" - command: "which" - args: ["python"] - expectedOutput: ['^/opt/python/cpython-[0-9.]+-linux-x86_64-gnu/bin/python$'] - - name: "Does the binary exists?" - command: "which" - args: ["pip"] - expectedOutput: ['^/opt/python/cpython-[0-9.]+-linux-x86_64-gnu/bin/pip$'] - - name: "Does the binary exists?" - command: "which" - args: ["duckdb"] - expectedOutput: ["/usr/local/bin/duckdb"] - - name: "Python is the requested version" - command: "bash" - args: ["-c", "python --version | grep -F \"Python ${PYTHON_VERSION}\""] - - name: "uv runs" - command: "uv" - args: ["--version"] - - name: "Workspace is empty (no leftover install files)" - command: "ls" - args: ["-A", "/home/onyxia/work"] - excludedOutput: ["."] +schemaVersion: "2.0.0" +metadataTest: + user: 1000 + workdir: "/home/onyxia/work" +fileExistenceTests: + - name: "user home" + path: "/home/onyxia" + shouldExist: true + permissions: "drwxr-x---" + uid: 1000 + gid: 100 + - name: "onyxia init script" + path: "/opt/onyxia-init.sh" + shouldExist: true + permissions: "-rwxr-xr-x" + uid: 1000 + gid: 100 +commandTests: + - name: "Does the binary exists?" + command: "which" + args: ["helm"] + expectedOutput: ["/usr/local/bin/helm"] + - name: "Does the binary exists?" + command: "which" + args: ["kubectl"] + expectedOutput: ["/usr/local/bin/kubectl"] + - name: "Does the binary exists?" + command: "which" + args: ["python"] + expectedOutput: ['/opt/python/bin/python'] + - name: "Does the binary exists?" + command: "which" + args: ["pip"] + expectedOutput: ['/opt/python/bin/pip'] + - name: "Does the binary exists?" + command: "which" + args: ["duckdb"] + expectedOutput: ["/usr/local/bin/duckdb"] + - name: "Python is the requested version" + command: "bash" + args: ["-c", "python --version | grep -F \"Python ${PYTHON_VERSION}\""] + - name: "uv runs" + command: "uv" + args: ["--version"] + - name: "Workspace is empty (no leftover install files)" + command: "ls" + args: ["-A", "/home/onyxia/work"] + excludedOutput: ["."] diff --git a/python-pytorch/tests.yaml b/python-pytorch/tests.yaml index bfc060a7..a614673b 100644 --- a/python-pytorch/tests.yaml +++ b/python-pytorch/tests.yaml @@ -1,45 +1,45 @@ -schemaVersion: "2.0.0" -metadataTest: - user: 1000 - workdir: "/home/onyxia/work" -fileExistenceTests: - - name: "user home" - path: "/home/onyxia" - shouldExist: true - permissions: "drwxr-x---" - uid: 1000 - gid: 100 - - name: "onyxia init script" - path: "/opt/onyxia-init.sh" - shouldExist: true - permissions: "-rwxr-xr-x" - uid: 1000 - gid: 100 -commandTests: - - name: "Does the binary exists?" - command: "which" - args: ["helm"] - expectedOutput: ["/usr/local/bin/helm"] - - name: "Does the binary exists?" - command: "which" - args: ["kubectl"] - expectedOutput: ["/usr/local/bin/kubectl"] - - name: "Does the binary exists?" - command: "which" - args: ["python"] - expectedOutput: ['^/opt/python/cpython-[0-9.]+-linux-x86_64-gnu/bin/python$'] - - name: "Does the binary exists?" - command: "which" - args: ["pip"] - expectedOutput: ['^/opt/python/cpython-[0-9.]+-linux-x86_64-gnu/bin/pip$'] - - name: "Does the binary exists?" - command: "which" - args: ["duckdb"] - expectedOutput: ["/usr/local/bin/duckdb"] - - name: "PyTorch runs" - command: "python" - args: ["-c", "import torch"] - - name: "Workspace is empty (no leftover install files)" - command: "ls" - args: ["-A", "/home/onyxia/work"] - excludedOutput: ["."] +schemaVersion: "2.0.0" +metadataTest: + user: 1000 + workdir: "/home/onyxia/work" +fileExistenceTests: + - name: "user home" + path: "/home/onyxia" + shouldExist: true + permissions: "drwxr-x---" + uid: 1000 + gid: 100 + - name: "onyxia init script" + path: "/opt/onyxia-init.sh" + shouldExist: true + permissions: "-rwxr-xr-x" + uid: 1000 + gid: 100 +commandTests: + - name: "Does the binary exists?" + command: "which" + args: ["helm"] + expectedOutput: ["/usr/local/bin/helm"] + - name: "Does the binary exists?" + command: "which" + args: ["kubectl"] + expectedOutput: ["/usr/local/bin/kubectl"] + - name: "Does the binary exists?" + command: "which" + args: ["python"] + expectedOutput: ['/opt/python/bin/python'] + - name: "Does the binary exists?" + command: "which" + args: ["pip"] + expectedOutput: ['/opt/python/bin/pip'] + - name: "Does the binary exists?" + command: "which" + args: ["duckdb"] + expectedOutput: ["/usr/local/bin/duckdb"] + - name: "PyTorch runs" + command: "python" + args: ["-c", "import torch"] + - name: "Workspace is empty (no leftover install files)" + command: "ls" + args: ["-A", "/home/onyxia/work"] + excludedOutput: ["."] diff --git a/r-python-julia/Dockerfile b/r-python-julia/Dockerfile index 083da682..3f799885 100644 --- a/r-python-julia/Dockerfile +++ b/r-python-julia/Dockerfile @@ -6,8 +6,8 @@ LABEL maintainer="InseeFrLab " ARG PYTHON_VERSION="3.13.15" ENV PYTHON_VERSION=${PYTHON_VERSION} -# Python is installed by uv (see /opt/install-python.sh) in /opt/python, in the folder uv names after the version -ARG PYTHON_DIR="/opt/python/cpython-${PYTHON_VERSION}-linux-x86_64-gnu" +# Install Python separately from system installation to avoid conflicts +ARG PYTHON_DIR="/opt/python" ENV PATH="${PYTHON_DIR}/bin:${PATH}" USER root diff --git a/r-python-julia/tests.yaml b/r-python-julia/tests.yaml index 7db29cc4..2c2f80f1 100644 --- a/r-python-julia/tests.yaml +++ b/r-python-julia/tests.yaml @@ -1,56 +1,56 @@ -schemaVersion: "2.0.0" -metadataTest: - user: 1000 - workdir: "/home/onyxia/work" -fileExistenceTests: - - name: "user home" - path: "/home/onyxia" - shouldExist: true - permissions: "drwxr-x---" - uid: 1000 - gid: 100 - - name: "onyxia init script" - path: "/opt/onyxia-init.sh" - shouldExist: true - permissions: "-rwxr-xr-x" - uid: 1000 - gid: 100 -commandTests: - - name: "Does the binary exists?" - command: "which" - args: ["helm"] - expectedOutput: ["/usr/local/bin/helm"] - - name: "Does the binary exists?" - command: "which" - args: ["kubectl"] - expectedOutput: ["/usr/local/bin/kubectl"] - - name: "Does the binary exists?" - command: "which" - args: ["R"] - expectedOutput: ["/usr/local/bin/R"] - - name: "Does the binary exists?" - command: "which" - args: ["python"] - expectedOutput: ['^/opt/python/cpython-[0-9.]+-linux-x86_64-gnu/bin/python$'] - - name: "Does the binary exists?" - command: "which" - args: ["pip"] - expectedOutput: ['^/opt/python/cpython-[0-9.]+-linux-x86_64-gnu/bin/pip$'] - - name: "Does the binary exists?" - command: "which" - args: ["julia"] - expectedOutput: ["/usr/local/bin/julia"] - - name: "Does the binary exists?" - command: "which" - args: ["duckdb"] - expectedOutput: ["/usr/local/bin/duckdb"] - - name: "Python is the requested version" - command: "bash" - args: ["-c", "python --version | grep -F \"Python ${PYTHON_VERSION}\""] - - name: "Julia runs and main package load" - command: "julia" - args: ["-e", "using HDF5"] - - name: "Workspace is empty (no leftover install files)" - command: "ls" - args: ["-A", "/home/onyxia/work"] - excludedOutput: ["."] +schemaVersion: "2.0.0" +metadataTest: + user: 1000 + workdir: "/home/onyxia/work" +fileExistenceTests: + - name: "user home" + path: "/home/onyxia" + shouldExist: true + permissions: "drwxr-x---" + uid: 1000 + gid: 100 + - name: "onyxia init script" + path: "/opt/onyxia-init.sh" + shouldExist: true + permissions: "-rwxr-xr-x" + uid: 1000 + gid: 100 +commandTests: + - name: "Does the binary exists?" + command: "which" + args: ["helm"] + expectedOutput: ["/usr/local/bin/helm"] + - name: "Does the binary exists?" + command: "which" + args: ["kubectl"] + expectedOutput: ["/usr/local/bin/kubectl"] + - name: "Does the binary exists?" + command: "which" + args: ["R"] + expectedOutput: ["/usr/local/bin/R"] + - name: "Does the binary exists?" + command: "which" + args: ["python"] + expectedOutput: ['/opt/python/bin/python'] + - name: "Does the binary exists?" + command: "which" + args: ["pip"] + expectedOutput: ['/opt/python/bin/pip'] + - name: "Does the binary exists?" + command: "which" + args: ["julia"] + expectedOutput: ["/usr/local/bin/julia"] + - name: "Does the binary exists?" + command: "which" + args: ["duckdb"] + expectedOutput: ["/usr/local/bin/duckdb"] + - name: "Python is the requested version" + command: "bash" + args: ["-c", "python --version | grep -F \"Python ${PYTHON_VERSION}\""] + - name: "Julia runs and main package load" + command: "julia" + args: ["-e", "using HDF5"] + - name: "Workspace is empty (no leftover install files)" + command: "ls" + args: ["-A", "/home/onyxia/work"] + excludedOutput: ["."] From 7780990746514f21d4b1a1f1a0fc2bb2cea851c3 Mon Sep 17 00:00:00 2001 From: avouacr Date: Sat, 19 Sep 2026 21:35:40 +0000 Subject: [PATCH 08/14] dev --- docs/improvement-plan.md | 96 ---------------------------------------- 1 file changed, 96 deletions(-) delete mode 100644 docs/improvement-plan.md diff --git a/docs/improvement-plan.md b/docs/improvement-plan.md deleted file mode 100644 index f12a29b3..00000000 --- a/docs/improvement-plan.md +++ /dev/null @@ -1,96 +0,0 @@ -# Improvement plan - -Remaining work from the repository audit of 2026-09-19; fixed items have been removed. Item numbers are kept from the original audit. Effort: **S** < 1h, **M** ≈ half a day, **L** multi-day. - -## Open items - -### 5. Spark/Hadoop/Hive install — L - -Audit and simplify the whole Spark layer (`spark/`). None of the questions below have been checked yet. - -- **py4j filename:** `spark/Dockerfile` sets `PYTHONPATH` to `py4j-0.10.9.9-src.zip`. Renovate bumps `SPARK_VERSION` automatically, so when Spark ships a new py4j, `import pyspark` breaks. Quick fix: symlink `${SPARK_HOME}/python/lib/py4j-*-src.zip` to a fixed name in `install-spark-hadoop-hive.sh` and reference that name. -- **Spark tests:** `spark/tests.yaml` still only checks binary paths. Add functional tests like the other layers: `import pyspark` and a local `SparkSession` when Python is present, `library(sparklyr)` and `library(SparkR)` when R is present. -- **Custom Spark build:** Spark comes from `spark-${SPARK_VERSION}-bin-hadoop-${HADOOP_VERSION}-hive-${HIVE_VERSION}-java-${JAVA_VERSION}.tgz`, built by InseeFrLab/Spark-hive and hosted on `minio.lab.sspcloud.fr`. Is it still needed, or would the official Apache distribution or `pip install pyspark` do? -- **Hadoop shipped twice?** A full Hadoop distribution is installed, and `SPARK_DIST_CLASSPATH=$(hadoop classpath)` is set in `spark-env.sh` and the entrypoint. -- **Full Hive 2.3.10 distribution:** Spark built with `-Phive` already bundles the Hive 2.3 client. What is the full distribution used for, beyond the `hive-authentication` and `hive-listener` jars and the postgres JDBC driver? Hive 2.3 is also an end-of-life line, and the postgres JDBC driver is 42.7.3. -- **Hand fixes tied to exact jar names:** remove guava 14, copy guava 27, swap jline, and remove `bundle-2.29.52.jar` "to fix multiple bindings". That last jar is the AWS SDK bundle `hadoop-aws` needs: where does S3A get the SDK from, and will these fixes survive version bumps? -- **Downloads:** - - `HADOOP_VERSION`, `HIVE_VERSION` and the jline/guava/JDBC versions are hardcoded and not tracked by Renovate. - - Nothing is checksum-verified, unlike the other install scripts. - - Hadoop comes from `downloads.apache.org`, which only hosts current releases, so 3.4.2 will return 404 once superseded. Use `archive.apache.org`. -- **SparkR:** installed with `remotes::install_github('apache/spark@v${SPARK_VERSION}', subdir='R/pkg')`, and the R variant also runs rocker's `install_tidyverse.sh`. -- **`spark-entrypoint.sh`:** a copy of the upstream Spark-on-Kubernetes entrypoint, including dead `PYSPARK_MAJOR_PYTHON_VERSION == 2` handling. Compare it with the current upstream version. - -### 12. Image size — M - -Published sizes on 2026-09-19 (compressed, amd64): - -| Image | Size | -|---|---| -| base | 0.95 GB (one 924 MB `RUN` layer) | -| python-minimal | 1.24 GB | -| python-datascience | 1.94 GB (702 MB geospatial + requirements layer) | -| jupyter-python | 2.06 GB | -| vscode-python | 2.44 GB | -| rstudio | 2.5 GB | -| jupyter-pyspark | 3.46 GB | -| jupyter-python GPU | 7.18 GB | -| jupyter-pytorch GPU | 9.5 GB | - -- Re-measure after the next build: the AWS CLI installer leftovers (about 70 MB zip + unzipped folder) are now removed from base. -- Run `dive` on base and python-datascience. Suspected big items: TinyTeX (`install-quarto.sh`) and `build-essential` in base, and the GDAL stack from the ubuntugis PPA in python-datascience. -- Consider moving TinyTeX and quarto to the IDE layers. -- Add a CI step that reports layer sizes, so regressions show up. -- **GPU testing:** GPU images are not tested in CI because they are too big for GitHub-hosted runners to load and test. Once sizes are reduced, check whether they fit, and test them like the CPU ones if so. - -<<<<<<< Updated upstream -### 14. Container startup work — S - -At every start, the end of `base/scripts/onyxia-init.sh` runs `chown -R` over every folder in `$ROOT_PROJECT_DIRECTORY`. That's slow on volumes with virtualenvs or data, and useless when the script runs as `onyxia` (Jupyter, VS Code), since a normal user can't change file owners. - -- Suggested fix: only when running as root (RStudio), and only for files that need it: - ```bash - if [[ $(id -u) = 0 ]]; then - find "$ROOT_PROJECT_DIRECTORY" -mindepth 1 -path "$ROOT_PROJECT_DIRECTORY/lost+found" -prune \ - -o \! -user "$USERNAME" -exec chown --no-dereference "$USERNAME:$GROUPNAME" {} + - fi - ``` -- Complementary, in the helm charts: `fsGroup` with `fsGroupChangePolicy: OnRootMismatch`. - -### 15. Unused CI step — S - -`.github/actions/cache-common-images` pulls `golang` and `dockereng/export-build` in every build job and warns "Failed to restore". It also has no `description`, which actionlint reports as an error. Remove the action and its step in `main-workflow-template.yml`. - -### 17. Two definitions of the image graph — M - -The image stacks are still declared twice: `chains` in `utils/build_chain.py` (local builds) and the jobs in `.github/workflows/main-workflow.yml` (CI). They currently match, and versions and the CUDA image already come from a single `versions.env`, but adding an image means editing both. - -Fix: a single `images.yaml` (layers, parents, languages, GPU flag) that drives both the CI matrix and local builds. Alternatively, generate `main-workflow.yml` from it and check in CI that it's up to date. - -### 21. Small items — S - -- **`RDebugger.r-debugger` VS Code extension:** still installed by `vscode/scripts/install-vscode-extensions.sh`, but its R backend `vscDebugger` was removed, so R debugging doesn't work (the extension only offers to install the package at first use). Remove it from `r_extensions`, and from the vscode tests if they list it. - -======= ->>>>>>> Stashed changes -## Optional ideas - -- CI: Trivy vulnerability scan, and `sbom: true` / `provenance: true` in `docker/build-push-action`. -- Shell: the remaining ~148 shellcheck info/style findings, mostly unquoted variables that never hold spaces (72 of them in `onyxia-init.sh`). Once fixed, raise the CI threshold to `info`. -- Python: unit tests for the tag scheme in `utils/generate_matrix.py`, run in `check-code-quality.yml`. -- Security: `curl --insecure $REGION_INIT_SCRIPT | bash` in `onyxia-init.sh` is tracked in a separate GitHub issue. - -## Decisions (don't reopen) - -- **Images are not built on PRs:** it was tried and costs too much compute. PRs only get linting, which is skipped for Renovate PRs. -- **No notification on build failure:** the pipeline result is checked every Monday morning. -- **A failed layer fails all its children:** the CI job structure is kept, with no GPU/CPU split. -- **GitHub actions stay pinned to version tags,** not digests. -- **Tool versions are pinned in install scripts,** not in Dockerfiles, which only pin Python, R and Spark. -- **amd64 only.** -- **The Git token stays in the clone URL:** containers are isolated and short-lived, and moving it to a credential helper would still leave it readable in plain text. -- **DuckDB secret values are not SQL-escaped:** AWS/MinIO credentials can't contain quotes. -- **`import tkinter` is broken in the images:** irrelevant for headless images. -- **The `chown -R` over `~/work` at startup stays:** the folder only holds what was just cloned or added by an init script, and the user must own all of it. -- **The image graph stays declared twice** (`chains` in `utils/build_chain.py`, jobs in `main-workflow.yml`): no generated CI. Adding an image means updating both. -- **Python stays built from source** (PGO + LTO): a uv-installed Python (python-build-standalone) was tried twice. Its `EXTERNALLY-MANAGED` marker blocks `pip install` / `uv pip install --system`, `uv pip install --system` ignores Pythons in uv's default location, and a symlinked install breaks uv's paths for package commands (e.g. `charset_normalizer` on 3.13). The source build works as is. From bceda4801aa2e3a8ecd6b127772fa360c3d4ee7f Mon Sep 17 00:00:00 2001 From: avouacr Date: Sun, 20 Sep 2026 06:56:06 +0000 Subject: [PATCH 09/14] up code-server --- utils/build_chain.py | 14 ++------------ utils/generate_matrix.py | 6 +++--- vscode/scripts/install-vscode.sh | 2 +- 3 files changed, 6 insertions(+), 16 deletions(-) diff --git a/utils/build_chain.py b/utils/build_chain.py index 44591170..4fddcd8d 100644 --- a/utils/build_chain.py +++ b/utils/build_chain.py @@ -1,7 +1,8 @@ import argparse import logging import subprocess -from pathlib import Path + +from .generate_matrix import read_versions logging.basicConfig( level=logging.INFO, @@ -33,17 +34,6 @@ } -def read_versions(): - versions_file_path = "versions.env" - versions = {} - for line in Path(versions_file_path).read_text().splitlines(): - line = line.strip() - if line and not line.startswith("#"): - key, value = line.split("=", 1) - versions[key] = value.strip('"') - return versions - - def build_chain(chain_name, r_version, py_version, spark_version, gpu, no_test, push): logger.info(f"Building chain : {chain_name}") diff --git a/utils/generate_matrix.py b/utils/generate_matrix.py index c4bc2fa1..03b38e1c 100644 --- a/utils/generate_matrix.py +++ b/utils/generate_matrix.py @@ -9,7 +9,7 @@ DH_ORGA = "inseefrlab" IMAGES_PREFIX = "onyxia" TODAY_DATE = datetime.now(UTC).strftime("%Y.%m.%d") -VERSIONS_FILE = Path(__file__).resolve().parent.parent / "versions.env" +VERSIONS_FILE = "versions.env" def generate_matrix(versions, input_image, output_image, spark_version, gpu_options, version_prefix): @@ -77,10 +77,10 @@ def generate_r_python_julia_matrix(r_version, py_version, input_image, output_im return matrix -def read_versions(path=VERSIONS_FILE): +def read_versions(): """Parse the KEY="value" lines of versions.env into a dict, ignoring comments and blank lines.""" versions = {} - for line in path.read_text().splitlines(): + for line in Path(VERSIONS_FILE).read_text().splitlines(): line = line.strip() if line and not line.startswith("#"): key, value = line.split("=", 1) diff --git a/vscode/scripts/install-vscode.sh b/vscode/scripts/install-vscode.sh index 827b310d..74490de9 100755 --- a/vscode/scripts/install-vscode.sh +++ b/vscode/scripts/install-vscode.sh @@ -2,7 +2,7 @@ set -eo pipefail # renovate: datasource=github-releases depName=coder/code-server -CODE_SERVER_VERSION="4.137.0" +CODE_SERVER_VERSION="4.138.0" # Download the official package and verify its checksum CODE_SERVER_DEB="code-server_${CODE_SERVER_VERSION}_amd64.deb" From ffcebc0225530854f295ef2e872430522d480f4b Mon Sep 17 00:00:00 2001 From: avouacr Date: Sun, 20 Sep 2026 07:00:06 +0000 Subject: [PATCH 10/14] up cuda --- versions.env | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/versions.env b/versions.env index ba4c2ac3..f9183c77 100644 --- a/versions.env +++ b/versions.env @@ -3,7 +3,7 @@ # Updated by Renovate, which also keeps the ARG defaults of the Dockerfiles in sync. BASE_IMAGE_CPU="ubuntu:24.04" -BASE_IMAGE_GPU="nvidia/cuda:12.8.1-cudnn-devel-ubuntu24.04" +BASE_IMAGE_GPU="nvidia/cuda:13.2.1-cudnn-devel-ubuntu24.04" PYTHON_VERSION_1="3.14.7" PYTHON_VERSION_2="3.13.15" From 2c9b90b2a6210dee7572d33227ef015f6e2856ba Mon Sep 17 00:00:00 2001 From: avouacr Date: Sun, 20 Sep 2026 07:01:53 +0000 Subject: [PATCH 11/14] de --- utils/build_chain.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/utils/build_chain.py b/utils/build_chain.py index 4fddcd8d..f73a30d3 100644 --- a/utils/build_chain.py +++ b/utils/build_chain.py @@ -2,7 +2,7 @@ import logging import subprocess -from .generate_matrix import read_versions +from utils.generate_matrix import read_versions logging.basicConfig( level=logging.INFO, From 8b4ea1e5071586276286e78432e2838eab609548 Mon Sep 17 00:00:00 2001 From: avouacr Date: Sun, 20 Sep 2026 07:09:24 +0000 Subject: [PATCH 12/14] dev --- .../actions/cache-common-images/action.yml | 24 ------------------- .github/workflows/main-workflow-template.yml | 10 ++++---- .github/workflows/main-workflow.yml | 1 - python-minimal/Dockerfile | 2 +- r-python-julia/Dockerfile | 2 +- renovate.json | 16 +++++++++++-- utils/build_chains.sh | 14 +++++------ utils/generate_matrix.py | 4 ++-- 8 files changed, 29 insertions(+), 44 deletions(-) delete mode 100644 .github/actions/cache-common-images/action.yml diff --git a/.github/actions/cache-common-images/action.yml b/.github/actions/cache-common-images/action.yml deleted file mode 100644 index b6526a8a..00000000 --- a/.github/actions/cache-common-images/action.yml +++ /dev/null @@ -1,24 +0,0 @@ -name: Cache common Docker images - -runs: - using: composite - steps: - - name: Set up GHA cache - id: cache - uses: actions/cache@v6 - with: - path: cached_images.tar - key: ${{ runner.os }}-cache - - - name: Cache images - if: steps.cache.outputs.cache-hit != 'true' - shell: bash - run: | - docker pull dockereng/export-build - docker pull golang - docker save dockereng/export-build golang -o cached_images.tar - - - name: Restore cached images - if: steps.cache.outputs.cache-hit == 'true' - shell: bash - run: docker load -i cached_images.tar diff --git a/.github/workflows/main-workflow-template.yml b/.github/workflows/main-workflow-template.yml index c7f1167a..bfd44507 100644 --- a/.github/workflows/main-workflow-template.yml +++ b/.github/workflows/main-workflow-template.yml @@ -12,9 +12,10 @@ on: required: true type: string base_image: - description: Name of the base image - required: true + description: Name of the base image (empty for base, whose external base images come from versions.env) + required: false type: string + default: "" languages: description: Languages of the image, which select the versions read from versions.env (python, r or r-python; empty for base) required: false @@ -45,7 +46,7 @@ jobs: python-version: '3.14' - id: set-matrix run: | - python3 utils/generate_matrix.py --input_image ${{ inputs.base_image }} --output_image ${{ inputs.image }} \ + python3 utils/generate_matrix.py --input_image "${{ inputs.base_image }}" --output_image ${{ inputs.image }} \ --languages "${{ inputs.languages }}" --spark ${{ inputs.spark }} --build_gpu ${{ inputs.build_gpu }} docker: runs-on: ubuntu-latest @@ -93,9 +94,6 @@ jobs: with: username: ${{ secrets.DOCKERHUB_USERNAME }} password: ${{ secrets.DOCKERHUB_TOKEN }} - - - name: Restore common Docker images from cache - uses: ./.github/actions/cache-common-images - name: Push to DockerHub if: ${{ github.ref == 'refs/heads/main' }} diff --git a/.github/workflows/main-workflow.yml b/.github/workflows/main-workflow.yml index 9fb6db96..1e5af43f 100644 --- a/.github/workflows/main-workflow.yml +++ b/.github/workflows/main-workflow.yml @@ -15,7 +15,6 @@ jobs: with: image: base context: base - base_image: ubuntu:24.04 secrets: inherit python-minimal: needs: [base] diff --git a/python-minimal/Dockerfile b/python-minimal/Dockerfile index 7a8a39bf..01567a3d 100644 --- a/python-minimal/Dockerfile +++ b/python-minimal/Dockerfile @@ -3,7 +3,7 @@ FROM $BASE_IMAGE LABEL maintainer="InseeFrLab " -ARG PYTHON_VERSION="3.13.15" +ARG PYTHON_VERSION="3.14.7" ENV PYTHON_VERSION=${PYTHON_VERSION} # Install Python separately from system installation to avoid conflicts diff --git a/r-python-julia/Dockerfile b/r-python-julia/Dockerfile index 3f799885..ea8bff00 100644 --- a/r-python-julia/Dockerfile +++ b/r-python-julia/Dockerfile @@ -3,7 +3,7 @@ FROM $BASE_IMAGE LABEL maintainer="InseeFrLab " -ARG PYTHON_VERSION="3.13.15" +ARG PYTHON_VERSION="3.14.7" ENV PYTHON_VERSION=${PYTHON_VERSION} # Install Python separately from system installation to avoid conflicts diff --git a/renovate.json b/renovate.json index 99b10a13..79c03e12 100644 --- a/renovate.json +++ b/renovate.json @@ -79,17 +79,29 @@ "depTypeTemplate": "spark" }, { - "description": "CUDA base image of the GPU variants (versions.env)", + "description": "Base image of the GPU variants (versions.env)", "customType": "regex", "managerFilePatterns": [ "/^versions\\.env$/" ], "matchStrings": [ - "CUDA_BASE_IMAGE=\"(?[^:\"]+):(?[^\"]+)\"" + "BASE_IMAGE_GPU=\"(?[^:\"]+):(?[^\"]+)\"" ], "datasourceTemplate": "docker", "depTypeTemplate": "cuda" }, + { + "description": "Base image of the CPU variants (versions.env)", + "customType": "regex", + "managerFilePatterns": [ + "/^versions\\.env$/" + ], + "matchStrings": [ + "BASE_IMAGE_CPU=\"(?[^:\"]+):(?[^\"]+)\"" + ], + "datasourceTemplate": "docker", + "depTypeTemplate": "base-image" + }, { "description": "Tools downloaded by install scripts and CI actions, pinned with a '# renovate: datasource=... depName=...' comment above a _VERSION=\"x.y.z\" line", "customType": "regex", diff --git a/utils/build_chains.sh b/utils/build_chains.sh index 536e5da8..e2bb7010 100755 --- a/utils/build_chains.sh +++ b/utils/build_chains.sh @@ -18,16 +18,16 @@ PYTHON_VERSIONS=("$PYTHON_VERSION_1" "$PYTHON_VERSION_2") R_VERSIONS=("$R_VERSION_1" "$R_VERSION_2") for py_ver in "${PYTHON_VERSIONS[@]}"; do - python3 utils/build_chain.py --chain vscode-python --py_version $py_ver $DOCKER_BUILD_ARGS - python3 utils/build_chain.py --chain vscode-pytorch --py_version $py_ver --gpu $DOCKER_BUILD_ARGS - python3 utils/build_chain.py --chain jupyter-pyspark --py_version $py_ver --spark_version $SPARK_VERSION $DOCKER_BUILD_ARGS + python3 -m utils.build_chain --chain vscode-python --py_version $py_ver $DOCKER_BUILD_ARGS + python3 -m utils.build_chain --chain vscode-pytorch --py_version $py_ver --gpu $DOCKER_BUILD_ARGS + python3 -m utils.build_chain --chain jupyter-pyspark --py_version $py_ver --spark_version $SPARK_VERSION $DOCKER_BUILD_ARGS done for r_ver in "${R_VERSIONS[@]}"; do - python3 utils/build_chain.py --chain rstudio --r_version $r_ver $DOCKER_BUILD_ARGS - python3 utils/build_chain.py --chain sparkr --r_version $r_ver --spark_version $SPARK_VERSION $DOCKER_BUILD_ARGS + python3 -m utils.build_chain --chain rstudio --r_version $r_ver $DOCKER_BUILD_ARGS + python3 -m utils.build_chain --chain sparkr --r_version $r_ver --spark_version $SPARK_VERSION $DOCKER_BUILD_ARGS done # r-python-julia images are built with only latest versions of R & Python -python3 utils/build_chain.py --chain rstudio-r-python-julia --r_version ${R_VERSIONS[0]} --py_version ${PYTHON_VERSIONS[0]} $DOCKER_BUILD_ARGS -python3 utils/build_chain.py --chain jupyter-r-python-julia --r_version ${R_VERSIONS[0]} --py_version ${PYTHON_VERSIONS[0]} $DOCKER_BUILD_ARGS +python3 -m utils.build_chain --chain rstudio-r-python-julia --r_version ${R_VERSIONS[0]} --py_version ${PYTHON_VERSIONS[0]} $DOCKER_BUILD_ARGS +python3 -m utils.build_chain --chain jupyter-r-python-julia --r_version ${R_VERSIONS[0]} --py_version ${PYTHON_VERSIONS[0]} $DOCKER_BUILD_ARGS diff --git a/utils/generate_matrix.py b/utils/generate_matrix.py index 03b38e1c..db932a02 100644 --- a/utils/generate_matrix.py +++ b/utils/generate_matrix.py @@ -108,12 +108,12 @@ def read_versions(): onyxia_base_tag = f"{IMAGES_PREFIX}-base:latest" matrix = [ { - "base_image_tag": args.input_image, + "base_image_tag": versions["BASE_IMAGE_CPU"], "output_image_main_tag": f"{DH_ORGA}/{onyxia_base_tag}", "output_image_tags": f"{DH_ORGA}/{onyxia_base_tag},{DH_ORGA}/{onyxia_base_tag}-{TODAY_DATE}", }, { - "base_image_tag": versions["CUDA_BASE_IMAGE"], + "base_image_tag": versions["BASE_IMAGE_GPU"], "output_image_main_tag": f"{DH_ORGA}/{onyxia_base_tag}-gpu", "output_image_tags": f"{DH_ORGA}/{onyxia_base_tag}-gpu,{DH_ORGA}/{onyxia_base_tag}-gpu-{TODAY_DATE}", }, From 10d208a7de2d354c0025e3e987e7413901a18cdd Mon Sep 17 00:00:00 2001 From: avouacr Date: Sun, 20 Sep 2026 07:16:37 +0000 Subject: [PATCH 13/14] dev --- utils/build_chains.sh | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/utils/build_chains.sh b/utils/build_chains.sh index e2bb7010..eac6a47c 100755 --- a/utils/build_chains.sh +++ b/utils/build_chains.sh @@ -18,16 +18,16 @@ PYTHON_VERSIONS=("$PYTHON_VERSION_1" "$PYTHON_VERSION_2") R_VERSIONS=("$R_VERSION_1" "$R_VERSION_2") for py_ver in "${PYTHON_VERSIONS[@]}"; do - python3 -m utils.build_chain --chain vscode-python --py_version $py_ver $DOCKER_BUILD_ARGS - python3 -m utils.build_chain --chain vscode-pytorch --py_version $py_ver --gpu $DOCKER_BUILD_ARGS - python3 -m utils.build_chain --chain jupyter-pyspark --py_version $py_ver --spark_version $SPARK_VERSION $DOCKER_BUILD_ARGS + uv run python3 -m utils.build_chain --chain vscode-python --py_version $py_ver $DOCKER_BUILD_ARGS + uv run python3 -m utils.build_chain --chain vscode-pytorch --py_version $py_ver --gpu $DOCKER_BUILD_ARGS + uv run python3 -m utils.build_chain --chain jupyter-pyspark --py_version $py_ver --spark_version $SPARK_VERSION $DOCKER_BUILD_ARGS done for r_ver in "${R_VERSIONS[@]}"; do - python3 -m utils.build_chain --chain rstudio --r_version $r_ver $DOCKER_BUILD_ARGS - python3 -m utils.build_chain --chain sparkr --r_version $r_ver --spark_version $SPARK_VERSION $DOCKER_BUILD_ARGS + uv run python3 -m utils.build_chain --chain rstudio --r_version $r_ver $DOCKER_BUILD_ARGS + uv run python3 -m utils.build_chain --chain sparkr --r_version $r_ver --spark_version $SPARK_VERSION $DOCKER_BUILD_ARGS done # r-python-julia images are built with only latest versions of R & Python -python3 -m utils.build_chain --chain rstudio-r-python-julia --r_version ${R_VERSIONS[0]} --py_version ${PYTHON_VERSIONS[0]} $DOCKER_BUILD_ARGS -python3 -m utils.build_chain --chain jupyter-r-python-julia --r_version ${R_VERSIONS[0]} --py_version ${PYTHON_VERSIONS[0]} $DOCKER_BUILD_ARGS +uv run python3 -m utils.build_chain --chain rstudio-r-python-julia --r_version ${R_VERSIONS[0]} --py_version ${PYTHON_VERSIONS[0]} $DOCKER_BUILD_ARGS +uv run python3 -m utils.build_chain --chain jupyter-r-python-julia --r_version ${R_VERSIONS[0]} --py_version ${PYTHON_VERSIONS[0]} $DOCKER_BUILD_ARGS From b97c34053c7558b8d32fb36f5e9d57ca8125cb26 Mon Sep 17 00:00:00 2001 From: avouacr Date: Sun, 20 Sep 2026 08:34:55 +0000 Subject: [PATCH 14/14] don't upgrade cuda --- versions.env | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/versions.env b/versions.env index f9183c77..02b0b58d 100644 --- a/versions.env +++ b/versions.env @@ -3,7 +3,7 @@ # Updated by Renovate, which also keeps the ARG defaults of the Dockerfiles in sync. BASE_IMAGE_CPU="ubuntu:24.04" -BASE_IMAGE_GPU="nvidia/cuda:13.2.1-cudnn-devel-ubuntu24.04" +BASE_IMAGE_GPU="nvidia/cuda:12.6.3-cudnn-devel-ubuntu24.04" PYTHON_VERSION_1="3.14.7" PYTHON_VERSION_2="3.13.15"