Add Kubeflow Spark Operator compatibility scraper - #4131
Conversation
Soffi AI SummaryThis PR adds Kubeflow Spark Operator to Console's compatibility tracking system. The motivation is to extend the platform's add-on compatibility matrix so that users can see which Spark Operator versions are compatible with which Kubernetes versions, driven by the upstream Version Matrix published in the Spark Operator README. The scraper parses the upstream Version Matrix table, matches documented operator families (2.0–2.3) against real Helm chart Regression tests validate correct filtering of Spark-version tags vs. operator tags, legacy/prerelease exclusion, malformed/missing matrix detection, and out-of-bounds Kubernetes version handling. ChangesKubeflow Spark Operator compatibility scraper
Updated: 2026-09-08 06:07 UTC |
Greptile SummaryAdds Kubeflow Spark Operator compatibility generation and registers its generated compatibility data.
Confidence Score: 4/5The compatibility implementation appears safe to merge, with non-blocking gaps in automated execution and entry-point coverage for the new tests. No functional or security failure was established; the actionable concerns are limited to the tests not running in CI and not covering the scraper’s shared-utility integration. Files Needing Attention: utils/compatibility/scrapers/spark-operator.py, utils/compatibility/tests/README.md
|
| Filename | Overview |
|---|---|
| utils/compatibility/scrapers/spark-operator.py | Adds the matrix parser, version-row builder, and scraper integration; parsing behavior is well tested, but the entry-point integration is not. |
| utils/compatibility/tests/test_spark_operator.py | Covers malformed matrices, version-family filtering, bounds, and operator/chart separation, but does not exercise scrape(). |
| utils/compatibility/tests/README.md | Documents focused and live verification, although the focused suite is not wired into CI. |
| static/compatibilities/spark-operator.yaml | Adds schema-compatible generated metadata and compatibility rows for four documented operator families. |
| static/compatibilities/manifest.yaml | Registers the Spark Operator compatibility file using the expected addon naming convention. |
Reviews (1): Last reviewed commit: "Add Kubeflow Spark Operator compatibilit..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| Run from the repository root: | ||
|
|
||
| ```sh | ||
| python -m unittest discover -s utils/compatibility/tests -v | ||
| ``` |
There was a problem hiding this comment.
The new regression suite is only documented as a manual command. No CI workflow runs this unittest discovery path, so parser regressions can pass automated checks unnoticed. Please add the suite to an existing required workflow or create a focused compatibility-scraper check.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
There was a problem hiding this comment.
Addressed in ff09b05: the existing compatibility schema workflow now runs the focused unittest suite and includes utils/compatibility/** in its PR and master-push path filters. The test job needs only packaging 24.1; no credentials, network calls or Helm are required by the tests.
| def scrape(): | ||
| from utils import fetch_page, get_chart_versions, current_kube_version, update_compatibility_info | ||
|
|
||
| page = fetch_page(README_URL) | ||
| if not page: | ||
| raise ValueError("Could not fetch Spark Operator version matrix") | ||
| rows = build_rows( | ||
| page.decode("utf-8"), get_chart_versions("spark-operator"), current_kube_version() | ||
| ) | ||
| update_compatibility_info("../../static/compatibilities/spark-operator.yaml", rows) |
There was a problem hiding this comment.
The tests cover only parse_matrix and build_rows, leaving the new scrape() entry point untested. A regression in byte decoding, chart lookup, Kubernetes-version retrieval, output-path selection, or updater invocation could therefore pass the focused suite. Please add a mocked entry-point test that verifies these integrations and the rows passed to update_compatibility_info.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
There was a problem hiding this comment.
Addressed in ff09b05: added a mocked scrape() test verifying the source URL, byte decoding, chart and Kubernetes lookup, output path and exact updater rows. A second test verifies that missing, empty, malformed and invalid UTF-8 source data never invokes the updater. All six tests pass locally on Python 3.13 with packaging 24.1.
|
The review fixes are pushed in ff09b05, with all six focused tests passing locally. The Validate Compatibility Schemas run for this commit currently reports action_required and has zero jobs: https://github.com/pluralsh/console/actions/runs/34193507546 . Could a maintainer check whether this contributor workflow needs approval to run? I have not treated that state as a CI pass. |
|
Hi @michaeljguarino, I saw that this PR and #4145 were closed. Are Spark Operator and Antrea compatibility still wanted, or is there something in these implementations that needs changing? A quick pointer would help me understand whether to revise them. Thanks. |
2 participants
Adds Kubeflow Spark Operator to the compatibility manifest with a scraper, application metadata, generated YAML, and focused regression tests. Resolves #4130.
The scraper reads the operator/Kubernetes columns from the upstream Version Matrix, matches concrete operator
appVersionentries from the official Helm index, and uses the shared updater to reduce versions and render chart images.The documented
1.16+minimum is expanded through the repository'sKUBE_VERSION(1.36). These are upstream-declared compatibility ranges; this does not claim independent cluster testing on each Kubernetes version. The checked-in output covers the documented 2.0–2.3 families. The upstream matrix does not yet document 2.4/2.5, so they are deliberately skipped. Legacy compoundv1beta2-...tags are excluded because they do not fit the shared semantic-version sorting/release URL. Base Spark versions are never treated as operator versions.Validation:
python -m unittest discover -s utils/compatibility/tests -v: 6 tests passed with Python 3.13 and packaging 24.1, including malformed/missing matrix, invalid bounds, operator-vs-Spark/chart separation, prerelease and undocumented-family exclusion, source-to-updater entry-point wiring, and no update on unusable source.scrape()entry point against upstream README and Helm index, using Helm 3.18.6: generated four version rows and resolved all four container image references successfully.git diff --cached --check: passed before commit.No full Console application tests or live Kubernetes deployment tests were run; changes are confined to compatibility data and its scraper. See
utils/compatibility/tests/README.mdfor reproduction and scope.Submitted for consideration under the README's $300 new-scraper contributor reward. Payout eligibility and PayPal support are not yet confirmed; see #4130.