Fixes 30263: Add pkg-config to ingestion-base images so mysqlclient builds

[mohittilala]

Describe your changes:

Fixes #30263

#30024 swapped the ingestion operator image base from python:3.10-bookworm to python:3.10-slim-bookworm. The full image ships /usr/bin/pkg-config via buildpack-deps, the slim image does not, and the apt list never named pkg-config explicitly because it had never needed to.

.[all] pulls the doris plugin, pydoris-custom>=1.0.2,<1.5, whose metadata requires mysqlclient<3,>=2.1.0. mysqlclient publishes no Linux wheels for any version, so pip always builds it from source on Linux, and 2.2.0 switched header discovery from mysql_config to pkg-config. Without the binary the build aborts:

/bin/sh: 1: pkg-config: not found
Trying pkg-config --exists mysqlclient
Command 'pkg-config --exists mysqlclient' returned non-zero exit status 127.
Exception: Can not find valid pkg-config name.
ERROR: Failed to build 'mysqlclient' when getting requirements to build wheel

Adding pkg-config to the apt list in both operator Dockerfiles.

Both files are changed on purpose. docker-openmetadata-ingestion-base.yml builds the non-CI ingestion/operators/docker/Dockerfile, so patching only the .ci variant would leave the released image broken.

ingestion/Dockerfile and ingestion/Dockerfile.ci need no change. They build on apache/airflow:3.1.5-python3.10 and 3.2.1-python3.10, which ship mysqlclient 2.2.7 and 2.2.8 preinstalled, so the requirement is already satisfied and it is never source-built there.

This needs cherry-picking to 1.13 and 1.12.14, both of which carry the same defect. 1.12.13 and earlier are clean, they still used the full python:3.10-bookworm base.

Type of change:

• Bug fix

High-level design:

N/A — small change.

Tests:

Use cases covered

• openmetadata/ingestion-base builds on linux/amd64 and linux/arm64 with INGESTION_DEPENDENCY=all
• mysqlclient resolves and compiles from source inside the slim base

Unit tests

Not applicable, Dockerfile-only change.

Backend integration tests

• Not applicable (no backend API changes).

Ingestion integration tests

• Not applicable (no Python source changes).

Playwright (UI) tests

• Not applicable (no UI changes).

Manual testing performed

Repro script committed at z-local.log/.../test-scripts/repro.sh, RED/GREEN against the real slim base:

apt extras	result
none	fails, traceback identical to the release build
pkg-config	builds, mysqlclient (2, 2, 8, 'final', 0)

Ran the exact 57-line apt block from ingestion/operators/docker/Dockerfile.ci on both arches to confirm nothing is being removed by apt, and that pkg-config is the only thing missing:

Check	Result
default-libmysqlclient-dev	ii 1.1.0, installed
libmariadb-dev	ii 1:10.11.18-0+deb12u1, installed at the upgraded version
.pc files	libmariadb.pc, mariadb.pc, mysqlclient.pc all present
mysql_config	/usr/bin/mysql_config present
pkg-config	absent

Base image check:

$ docker run --rm python:3.10-bookworm      sh -c 'command -v pkg-config'
/usr/bin/pkg-config
$ docker run --rm python:3.10-slim-bookworm sh -c 'command -v pkg-config'
(nothing)

Full ingestion/operators/docker/Dockerfile.ci build with [all] on linux/arm64 is running and will be reported here when it completes.

UI screen recording / screenshots:

Not applicable.

Checklist:

• I have read the CONTRIBUTING document.
• My PR title is Fixes <issue-number>: <short explanation>

---

Summary by Gitar

• Refactored SQL column handler:
  • Updated pk_columns extraction logic in SqlColumnHandlerMixin by adding # pyright: ignore[reportOptionalIterable] to the list comprehension.
• Docker configuration:
  • Added pkg-config to ingestion/operators/docker/Dockerfile and ingestion/operators/docker/Dockerfile.ci to support mysqlclient source builds.

_{This will update automatically on new commits.}

Greptile Summary

This PR restores support for building mysqlclient in the slim ingestion operator images. The main changes are:

• Adds pkg-config to the production operator image.
• Adds the same package to the CI operator image.
• Scopes a Pyright suppression in the SQL column handler.

Confidence Score: 5/5

This looks safe to merge.

• No blocking issues found in the changed code.

Important Files Changed

Filename	Overview
ingestion/operators/docker/Dockerfile	Adds pkg-config to the production ingestion operator image.
ingestion/operators/docker/Dockerfile.ci	Adds pkg-config to the CI ingestion operator image.
ingestion/src/metadata/ingestion/source/database/sql_column_handler.py	Reformats the primary-key cleanup comprehension and scopes its Pyright suppression.

_{Reviews (4): Last reviewed commit: "Merge branch 'main' into fix/ingestion-b..." | Re-trigger Greptile}

Context used (3)

• Context used - CLAUDE.md (source)
• Context used - openmetadata-ui-core-components/CLAUDE.md (source)
• Context used - AGENTS.md (source)

[github-actions]

🟡 Playwright Results — all passed (22 flaky)

✅ 4530 passed · ❌ 0 failed · 🟡 22 flaky · ⏭️ 95 skipped

Shard	Passed	Failed	Flaky	Skipped
🟡 Shard 1	435	0	5	16
✅ Shard 2	11	0	0	0
🟡 Shard 3	822	0	6	8
🟡 Shard 4	815	0	3	18
✅ Shard 5	831	0	0	5
🟡 Shard 6	832	0	3	46
🟡 Shard 7	784	0	5	2

🟡 22 flaky test(s) (passed on retry)

• Flow/TestConnectionModal.spec.ts › modal opens with gate card and capability checks sections (shard 1, 1 retry)
• Flow/TestConnectionModal.spec.ts › success state shows Done button and hides Edit Connection and Retry Test (shard 1, 1 retry)
• Pages/Lineage/LineageRightPanel.spec.ts › Verify custom properties tab IS visible for supported type: chart (shard 1, 1 retry)
• Pages/SearchSettings.spec.ts › Latest preview config wins when a superseded request resolves late (shard 1, 1 retry)
• Flow/SearchRBAC.spec.ts › a fully denied user sees neither asset type when browsing (shard 1, 1 retry)
• Features/ContextCenterArticles.spec.ts › Article listing search filters, clears, and shows empty state (shard 3, 1 retry)
• Features/ContextCenterArticles.spec.ts › Article list cards, recently viewed widget, and pagination work (shard 3, 1 retry)
• Features/ContextCenterArticles.spec.ts › description: switching articles does not bleed unsaved content into next article (shard 3, 1 retry)
• Features/ContextCenterMemories.spec.ts › Shared memory shows the shared-with-specific-people description (shard 3, 1 retry)
• Features/ContextCenterMemories.spec.ts › typing the linked table name in the asset search returns it as a result (shard 3, 1 retry)
• Features/Glossary/GlossaryAdvancedOperations.spec.ts › should change domain on glossary (shard 3, 1 retry)
• Features/Glossary/GlossaryP3Tests.spec.ts › should create glossary with unicode characters in name (shard 4, 1 retry)
• Features/SearchExport.spec.ts › Search mode visible export downloads CSV with tab-specific row count (shard 4, 1 retry)
• Features/Table.spec.ts › should persist page size (shard 4, 1 retry)
• Pages/DomainUIInteractions.spec.ts › Add owner to data product via UI (shard 6, 1 retry)
• Pages/Entity.spec.ts › Domain Propagation (shard 6, 1 retry)
• Pages/ExploreBrowse.spec.ts › service type drill-down disables unrelated roots and query-panel Clear resets it (shard 6, 1 retry)
• Features/AutoPilot.spec.ts › Create Service and check the AutoPilot status (shard 7, 1 retry)
• Pages/Glossary.spec.ts › Approve and reject glossary term from Glossary Listing (shard 7, 1 retry)
• Pages/Lineage/LineageFilters.spec.ts › Verify Impact Analysis service filter selection (shard 7, 1 retry)
• Pages/Lineage/LineageRightPanel.spec.ts › Verify custom properties tab is NOT visible for pipelineService in platform lineage (shard 7, 1 retry)
• Pages/Lineage/LineageRightPanel.spec.ts › Verify custom properties tab is NOT visible for apiService in platform lineage (shard 7, 1 retry)

📦 Download artifacts

How to debug locally

# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip    # view trace

[Copilot]

Pull request overview

Adds the missing pkg-config system dependency to the ingestion operator base images (both CI and non-CI Dockerfiles) so mysqlclient can compile from source when building on python:3.10-slim-bookworm, unblocking openmetadata/ingestion-base builds for INGESTION_DEPENDENCY=all.

Changes:

• Install pkg-config in ingestion/operators/docker/Dockerfile and ingestion/operators/docker/Dockerfile.ci.
• Document why pkg-config is required in slim-based images (mysqlclient 2.2+ build behavior).

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
ingestion/operators/docker/Dockerfile.ci	Adds pkg-config to the apt dependency list for the operator CI image.
ingestion/operators/docker/Dockerfile	Adds pkg-config to the apt dependency list for the operator release/non-CI image.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

ingestion/src/metadata/ingestion/source/database/sql_column_handler.py:223

• Instead of suppressing reportOptionalIterable in the comprehension, coerce pk_columns to a non-optional iterable in the loop. This removes the type-checker ignore and also keeps the code safe if pk_columns is ever None at runtime (e.g., from SQLAlchemy inspector implementations).

        pk_columns = [
            clean_up_starting_ending_double_quotes_in_string(pk_column)
            for pk_column in pk_columns  # pyright: ignore[reportOptionalIterable]
        ]

[sonarqubecloud]

Quality Gate passed for 'open-metadata-ingestion'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[github-actions]

Failed to cherry-pick changes to the 1.13 branch.
Please cherry-pick the changes manually.
You can find more details here.

[gitar-bot]

Code Review 👍 Approved with suggestions 0 resolved / 1 findings

Adds pkg-config to ingestion operator Dockerfiles to resolve mysqlclient build failures and updates SqlColumnHandlerMixin with a type-ignore for primary key extraction. Note that the fallback for pk_columns should return an empty list [] rather than an empty dictionary {} to maintain type consistency.

💡 Quality: Empty-dict fallback for list-typed pk_columns is misleading

📄 ingestion/src/metadata/ingestion/source/database/sql_column_handler.py:187-191

pk_columns semantically represents a list of primary-key column names, but the fallback branch now returns {} instead of []. It happens to work because the value is only iterated at line 220 (iterating an empty dict yields nothing), but using an empty dict where a list is expected is confusing and could break if a future caller relies on list behavior. Use [] as the fallback to match the intended type: pk_constraints.get("constrained_columns") or [].

Fall back to an empty list, matching the list type of pk_columns.

pk_columns = pk_constraints.get("constrained_columns") or []

🤖 Prompt for agents

Code Review: Adds `pkg-config` to ingestion operator Dockerfiles to resolve `mysqlclient` build failures and updates `SqlColumnHandlerMixin` with a type-ignore for primary key extraction. Note that the fallback for `pk_columns` should return an empty list `[]` rather than an empty dictionary `{}` to maintain type consistency.

1. 💡 Quality: Empty-dict fallback for list-typed pk_columns is misleading
   Files: ingestion/src/metadata/ingestion/source/database/sql_column_handler.py:187-191

   pk_columns semantically represents a list of primary-key column names, but the fallback branch now returns `{}` instead of `[]`. It happens to work because the value is only iterated at line 220 (iterating an empty dict yields nothing), but using an empty dict where a list is expected is confusing and could break if a future caller relies on list behavior. Use `[]` as the fallback to match the intended type: `pk_constraints.get("constrained_columns") or []`.

   Fix (Fall back to an empty list, matching the list type of pk_columns.):
   pk_columns = pk_constraints.get("constrained_columns") or []

Options

Display: compact → Showing less information.

Comment with these commands to change the behavior for this request:

Compact

gitar display:verbose

_{Was this helpful? React with 👍 / 👎 | Gitar}