Skip to content

[SPARK-59703][INFRA] Update the dependency list for nogil dockerfile #58955

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
gaogaotiantian wants to merge 2 commits into from
Closed

[SPARK-59703][INFRA] Update the dependency list for nogil dockerfile #58955

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
8f4d1fc
Update the dependency list for nogil dockerfile
gaogaotiantian Sep 21, 2026
72b643c
Add build contexts
gaogaotiantian Sep 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/build_infra_images_cache.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -231,6 +231,8 @@ jobs:
uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a
with:
context: ./dev/spark-test-image/python-314-nogil/
build-contexts: |
root=./
push: true
tags: ghcr.io/apache/spark/apache-spark-github-action-image-pyspark-python-314-nogil-cache:${{ github.ref_name }}-static
cache-from: type=registry,ref=ghcr.io/apache/spark/apache-spark-github-action-image-pyspark-python-314-nogil-cache:${{ github.ref_name }}
Expand Down
12 changes: 8 additions & 4 deletions dev/spark-test-image/python-314-nogil/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,11 @@ ENV VIRTUAL_ENV=/opt/spark-venv
RUN python3.14t -m venv $VIRTUAL_ENV
ENV PATH="$VIRTUAL_ENV/bin:$PATH"

# TODO: Add BASIC_PIP_PKGS and CONNECT_PIP_PKGS when it supports Python 3.14 free threaded
# TODO: Add lxml, grpcio, grpcio-status back when they support Python 3.14 free threaded
RUN python3.14t -m pip install 'numpy>=2.1' 'pyarrow>=23.0.0' 'six==1.16.0' 'pandas==2.3.3' 'pystack>=1.6.0' scipy coverage matplotlib openpyxl jinja2 psutil && \
python3.14t -m pip cache purge
COPY --from=root pyproject.toml ./pyproject.toml

RUN python3.14 -m pip install -U pip

# TODO: Add connect packages when grpcio supports Python 3.14 free threaded
RUN python3.14 -m pip install --group ml_torch --index-url https://download.pytorch.org/whl/cpu && \
python3.14 -m pip install --group ci_classic_standard && \
python3.14 -m pip cache purge