[Question]: al2023 so slow while installing pip requirements

[kostiantyn-lab]

Product

Amazon Linux 2023

What is your question?

building archive for EMR serverless takes a lot of time when using al2023

Dockerfile

FROM amazonlinux:2023 AS base

RUN yum install -y gcc python3 python3-devel

ENV VIRTUAL_ENV=/opt/venv
RUN python3 -m venv $VIRTUAL_ENV
ENV PATH="$VIRTUAL_ENV/bin:$PATH"

RUN python3 -m pip install --upgrade pip && \
    python3 -m pip install great_expectations venv-pack

RUN mkdir /output && venv-pack -o /output/pyspark_ge.tar.gz

FROM scratch AS export
COPY --from=base /output/pyspark_ge.tar.gz /

command

docker buildx build --no-cache --platform linux/arm64 --output . .

building time

Building 2317.3s (11/11) FINISHED

When I use amazonlinux:2 instead of amazonlinux:2023 the build runs much faster

building time

Building 493.2s (11/11) FINISHED

[onikroo]

+1 on this, have experienced the same on AL23 with RUN pip install cfn-lint always taking ~ 2635.0s.

#20 [linux/arm64 6/9] RUN pip install cfn-lint
#20 72.15 Collecting cfn-lint
#20 76.31   Downloading cfn_lint-1.18.4-py3-none-any.whl (4.7 MB)
#20 108.3 Collecting pyyaml>5.4
#20 108.5   Downloading PyYAML-6.0.2-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (720 kB)
#20 344.9 Collecting regex
#20 345.1   Downloading regex-2024.11.6-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (782 kB)
#20 357.0 Collecting aws-sam-translator>=1.91.0
#20 357.2   Downloading aws_sam_translator-1.92.0-py3-none-any.whl (383 kB)
#20 363.6 Collecting sympy>=1.0.0
#20 363.8   Downloading sympy-1.13.3-py3-none-any.whl (6.2 MB)
#20 372.3 Collecting jsonpatch
#20 372.6   Downloading jsonpatch-1.33-py2.py3-none-any.whl (12 kB)
#20 380.6 Collecting networkx<4,>=2.4
#20 380.8   Downloading networkx-3.2.1-py3-none-any.whl (1.6 MB)
#20 387.9 Collecting typing-extensions
#20 388.1   Downloading typing_extensions-4.12.2-py3-none-any.whl (37 kB)
#20 401.4 Collecting jsonschema<5,>=3.2
#20 401.5   Downloading jsonschema-4.23.0-py3-none-any.whl (88 kB)
#20 478.7 Collecting pydantic!=1.10.15,!=1.10.17,<3,>=1.8
#20 478.9   Downloading pydantic-2.9.2-py3-none-any.whl (434 kB)
#20 641.6 Collecting boto3==1.*,>=1.19.5
#20 641.7   Downloading boto3-1.35.57-py3-none-any.whl (139 kB)
#20 646.8 Collecting s3transfer<0.11.0,>=0.10.0
#20 647.0   Downloading s3transfer-0.10.3-py3-none-any.whl (82 kB)
#20 647.1 Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/lib/python3.9/site-packages (from boto3==1.*,>=1.19.5->aws-sam-translator>=1.91.0->cfn-lint) (0.10.0)
#20 809.6 Collecting botocore<1.36.0,>=1.35.57
#20 809.8   Downloading botocore-1.35.57-py3-none-any.whl (12.7 MB)
#20 823.5 Collecting mpmath<1.4,>=1.1.0
#20 823.7   Downloading mpmath-1.3.0-py3-none-any.whl (536 kB)
#20 827.5 Collecting jsonpointer>=1.9
#20 827.7   Downloading jsonpointer-3.0.0-py2.py3-none-any.whl (7.6 kB)
#20 995.1 Collecting rpds-py>=0.7.1
#20 995.3   Downloading rpds_py-0.21.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (361 kB)
#20 999.6 Collecting attrs>=22.2.0
#20 999.7   Downloading attrs-24.2.0-py3-none-any.whl (63 kB)
#20 1009.9 Collecting referencing>=0.28.4
#20 1010.1   Downloading referencing-0.35.1-py3-none-any.whl (26 kB)
#20 1013.5 Collecting jsonschema-specifications>=2023.03.6
#20 1013.6   Downloading jsonschema_specifications-2024.10.1-py3-none-any.whl (18 kB)
#20 1017.7 Collecting annotated-types>=0.6.0
#20 1017.8   Downloading annotated_types-0.7.0-py3-none-any.whl (13 kB)
#20 1443.1 Collecting pydantic-core==2.23.4
#20 1443.2   WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))': /packages/d0/b4/95b5eb47c6dc8692508c3ca04a1f8d6f0884c9dacb34cf3357595cbe73be/pydantic_core-2.23.4-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
#20 1447.8   Downloading pydantic_core-2.23.4-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.8 MB)
#20 1461.9 Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.57->boto3==1.*,>=1.19.5->aws-sam-translator>=1.91.0->cfn-lint) (2.8.1)
#20 1462.0 Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.57->boto3==1.*,>=1.19.5->aws-sam-translator>=1.91.0->cfn-lint) (1.25.10)
#20 1499.7 Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.36.0,>=1.35.57->boto3==1.*,>=1.19.5->aws-sam-translator>=1.91.0->cfn-lint) (1.15.0)
#20 1533.3 Installing collected packages: rpds-py, attrs, typing-extensions, referencing, botocore, s3transfer, pydantic-core, jsonschema-specifications, annotated-types, pydantic, mpmath, jsonschema, jsonpointer, boto3, sympy, regex, pyyaml, networkx, jsonpatch, aws-sam-translator, cfn-lint
#20 2610.4 Successfully installed annotated-types-0.7.0 attrs-24.2.0 aws-sam-translator-1.92.0 boto3-1.35.57 botocore-1.35.57 cfn-lint-1.18.4 jsonpatch-1.33 jsonpointer-3.0.0 jsonschema-4.23.0 jsonschema-specifications-2024.10.1 mpmath-1.3.0 networkx-3.2.1 pydantic-2.9.2 pydantic-core-2.23.4 pyyaml-6.0.2 referencing-0.35.1 regex-2024.11.6 rpds-py-0.21.0 s3transfer-0.10.3 sympy-1.13.3 typing-extensions-4.12.2
#20 DONE 2635.0s