From 4edc902bccc83cbff98ede12dc934657018329bd Mon Sep 17 00:00:00 2001 From: Nick Larsen Date: Thu, 16 Jul 2026 21:46:01 +0200 Subject: [PATCH 01/10] Release notes for 26.7.0 New platform features. Discovered via Claude, using a generated script. This will later inform a skill. Note: It misses release note comments inside PR descriptions. Working on a fix. --- modules/ROOT/pages/release-notes.adoc | 2 + .../partials/release-notes/release-26.7.adoc | 152 ++++++++++++++ scripts/gather-release-notes.sh | 186 ++++++++++++++++++ 3 files changed, 340 insertions(+) create mode 100644 modules/ROOT/partials/release-notes/release-26.7.adoc create mode 100755 scripts/gather-release-notes.sh diff --git a/modules/ROOT/pages/release-notes.adoc b/modules/ROOT/pages/release-notes.adoc index e320758f5..f7d3d2b47 100644 --- a/modules/ROOT/pages/release-notes.adoc +++ b/modules/ROOT/pages/release-notes.adoc @@ -7,6 +7,8 @@ The Stackable Data Platform consists of multiple operators that work together. Periodically a platform release is made, including all components of the platform at a specific version. // WARNING: Please keep the empty newlines, otherwise headings are broken. +include::partial$release-notes/release-26.7.adoc[] + include::partial$release-notes/release-26.3.adoc[] include::partial$release-notes/release-25.11.adoc[] diff --git a/modules/ROOT/partials/release-notes/release-26.7.adoc b/modules/ROOT/partials/release-notes/release-26.7.adoc new file mode 100644 index 000000000..388f4a8ff --- /dev/null +++ b/modules/ROOT/partials/release-notes/release-26.7.adoc @@ -0,0 +1,152 @@ +// DRAFT release notes for SDP 26.7.0. +// Sourced from stackabletech issues/PRs labelled (release/26.7.0 OR scheduled-for/26.7.0) +// AND (release-note OR release-note/action-required). 13 issues + 29 PRs. +// All `// TODO(verify)` markers must be resolved before this is merged/published. + +== Release 26.7 + +=== 26.7.0 + +// TODO(verify): exact release date. Tracking issue #847 target was 2026-07-17, shifted ~10 days -> ~2026-07-27. +Released on 2026-07-27. + +// TODO(highlights): written last, once all sections below are agreed. +// [TIP,caption=Release highlights] +// ==== +// * Highlight 1 +// ==== + +// TODO(breaking): overview of breaking changes, written last. Candidates (action-required / chore!): +// * Generic database connection (Druid field rename) - link:#druid-features-26_7_0 +// * Apache Airflow EXPERIMENTAL_FILE_* rename - link:#airflow-removals-26_7_0 +// * Apache Superset EXPERIMENTAL_FILE_* rename - link:#superset-removals-26_7_0 +// * Apache NiFi 1.x CRD fields removed - link:#nifi-removals-26_7_0 +// * Apache Spark application templates now namespaced - link:#spark-fixes-26_7_0 +// [WARNING,caption=Overview of breaking changes] +// ==== +// ==== + +==== New platform features + +===== General + +====== Miscellaneous + +* The SDP now provides a generic database connection mechanism, giving a consistent way to configure connections to external SQL databases across operators. + It has been rolled out to the Apache Airflow (https://github.com/stackabletech/airflow-operator/pull/754[airflow-operator#754]), Apache Hive (https://github.com/stackabletech/hive-operator/pull/674[hive-operator#674]) and Apache Superset (https://github.com/stackabletech/superset-operator/pull/722[superset-operator#722]) operators, and underpins new database connector support in Apache Druid and Trino (see the respective sections below). ++ +-- +[WARNING] +==== +*Breaking:* In the Stackable operator for Apache Druid the field `metadata_storage_database` has been renamed to `metadata_database`. +==== + +// TODO(verify): xref to the generic database connection concept/usage docs once the page exists. +Tracked in https://github.com/stackabletech/issues/issues/238[issues#238]. +-- +* Operators that support OIDC authentication can now configure the `clientAuthenticationMethod`. + This resolves the Apache Druid OIDC issue where a provider advertising `private_key_jwt` could cause authentication to fail (a known issue in the 26.3 release). + // TODO(verify): confirm which operators expose this field and add the relevant xref. + Tracked in https://github.com/stackabletech/issues/issues/838[issues#838]. +* Operators can now resolve product images from registries other than `oci.stackable.tech`, making it possible to install the SDP from a mirror or private registry. + // TODO(verify): confirm this landed for 26.7.0 (issue #716 is a tracking epic) and add the xref to the image-registry docs. + Tracked in https://github.com/stackabletech/issues/issues/716[issues#716]. + +===== Apache Airflow + +Apache Airflow can now authenticate git-sync against a repository using a custom CA, adding first-class support for CAs alongside the existing SSH and basic-auth options. +// TODO(verify): xref to the DAG mounting / git-sync usage guide. +Implemented in https://github.com/stackabletech/airflow-operator/pull/750[airflow-operator#750]. + +===== Apache NiFi + +Apache NiFi can now authenticate git-sync against a repository using a custom CA, adding first-class support for CAs. +// TODO(verify): xref to the NiFi git-sync usage guide. +Implemented in https://github.com/stackabletech/nifi-operator/pull/903[nifi-operator#903]. + +===== Apache Superset + +Two new roles, `worker` and `beat` (limited to a single replica), have been added so that Celery workers and the Celery beat scheduler can be enabled. +// TODO(verify): the results backend currently supports Redis only (no S3); confirm whether to call this out. xref to Superset usage guide. +Implemented in https://github.com/stackabletech/superset-operator/pull/724[superset-operator#724]. + +===== Open Policy Agent + +The OPA operator now supports typed `configOverrides` for OPA's `config.json`, via JSON merge patch, JSON patches or a full file replacement. +// TODO(verify): xref to the OPA configuration overrides docs. +Implemented in https://github.com/stackabletech/opa-operator/pull/818[opa-operator#818]. + +===== OpenSearch + +* Security configuration files are now hot-reloaded when changed, so that settings managed by the operator are applied without Pod restarts. + // TODO(verify): xref to the OpenSearch security docs. + Implemented in https://github.com/stackabletech/opensearch-operator/pull/130[opensearch-operator#130]. +* The operator now writes the actual node FQDNs into `plugins.security.nodes_dn` instead of a generic subject DN. + // TODO(verify): confirm feature vs fix categorisation for opensearch-operator#144. + Implemented in https://github.com/stackabletech/opensearch-operator/pull/144[opensearch-operator#144]. + +===== Trino + +* Trino now supports the PostgreSQL connector, allowing PostgreSQL databases to be queried as a Trino catalog. + // TODO(verify): xref to the Trino PostgreSQL connector docs. + Implemented in https://github.com/stackabletech/trino-operator/pull/883[trino-operator#883]. +* A new `.spec.name.inferred.replaceHyphensWithUnderscores` field on `TrinoCatalog` allows tweaking the catalog name in Trino by replacing `-` with `_`. + This lets you use valid Kubernetes resource names while keeping the convenience of `_` in catalog names. + Implemented in https://github.com/stackabletech/trino-operator/pull/903[trino-operator#903]. + +===== Stackable commons-operator + +The commons-operator restart-controller can now be told to ignore specific ConfigMaps and Secrets. +Set the `restarter.stackable.tech/ignore` label on a ConfigMap or Secret, or the `restarter.stackable.tech/ignore-configmap.x` / `restarter.stackable.tech/ignore-secret.x` annotations on a StatefulSet, to exclude them from triggering restarts. +// TODO(verify): xref to the restarter docs. +Implemented in https://github.com/stackabletech/commons-operator/pull/410[commons-operator#410]. + +===== Stackable secret-operator + +* The name of the CA certificate entry written by a `TrustStore` is now configurable, so it can be aligned with what consumers such as OpenShift expect (`tls.crt` vs `ca.crt`). + // TODO(verify): xref to the TrustStore docs. + Implemented in https://github.com/stackabletech/secret-operator/pull/679[secret-operator#679]. +* Domain components (DCs) can now be added to the subject DN of TLS certificates via the `secrets.stackable.tech/backend.autotls.cert.domain-components-in-subject-dn` volume annotation. + // TODO(verify): xref to the secret-operator volume docs. + Implemented in https://github.com/stackabletech/secret-operator/pull/708[secret-operator#708]. + +// TODO(verify): stackablectl#187 (uninstall Stacks and Demos) - stackablectl is versioned separately; confirm whether it belongs in these platform release notes. + +// ============================================================================ +// TODO: remaining sections (next review rounds) - stubbed to keep the include valid. +// ============================================================================ + +==== Platform improvements + +// TODO: NiFi startup times (nifi#953), OPA DaemonSet maxSurge (opa#819), +// typed OpenSearch configOverrides (opensearch#137), General > Vulnerabilities (CVE counts TODO). + +==== Platform fixes + +// TODO: internal-secret mutability (issues#843; airflow#770, trino#876), +// SecretClasses requiring client cert regression (issues#547), +// Spark connect user args (spark#696), Spark graceful SIGTERM shutdown (docker-images#1564), +// Spark application templates namespaced (spark#694/#720 - BREAKING). + +==== Platform deprecations + +// TODO: product version deprecations (from docker-images version issues). + +==== Platform removals + +// TODO: NiFi 1.x CRD fields (nifi#960 - BREAKING), Airflow/Superset EXPERIMENTAL_FILE_* rename +// (airflow#777, superset#721 - BREAKING), removed product versions. + +==== Supported versions + +// TODO(UNVERIFIED - reconcile against portal.stackable.build screenshot before publishing): +// product versions (new/LTS/deprecated/removed), Kubernetes versions, OpenShift versions. + +==== Upgrade from 26.3 + +// TODO: stackablectl + Helm uninstall/install for 26.3 -> 26.7, plus product-specific upgrade notes. + +[#known-issues-26_7_0] +==== Known issues + +// TODO(verify): none known yet. diff --git a/scripts/gather-release-notes.sh b/scripts/gather-release-notes.sh new file mode 100755 index 000000000..8368d49fe --- /dev/null +++ b/scripts/gather-release-notes.sh @@ -0,0 +1,186 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Gather the GitHub issues and pull requests that feed the release notes for a +# given SDP release, straight from the stackabletech org. +# +# It searches for items carrying BOTH: +# - a release label: release/ OR scheduled-for/ +# - a release-note label: release-note OR release-note/action-required +# +# and prints a report grouped by repository, including the "Release notes" +# snippet from each item body (the section the authors write in the PR template). +# +# It also flags two things that need human attention before a release: +# - items still labelled `scheduled-for/` (merged PRs should be +# re-labelled `release/`; open ones need to land or be dropped). +# - open / unmerged PRs in the set. +# +# Counts are the ground truth from pagination, NOT the GitHub search +# `total_count` (the web UI and that field can be misleading). +# +# Requires: gh (authenticated), jq. +# +# Usage: +# scripts/gather-release-notes.sh -v [--json] +# +# -v, --version Required. e.g. 26.7.0 +# --json Dump the raw combined JSON (all items) instead of the report. +# +# Examples: +# scripts/gather-release-notes.sh -v 26.7.0 +# scripts/gather-release-notes.sh -v 26.7.0 --json > items.json + +# ------------------------------ +# Args parsing +# ------------------------------ + +VERSION="" +OUTPUT="report" + +while [[ "$#" -gt 0 ]]; do + case $1 in + -v|--version) VERSION="$2"; shift ;; + --json) OUTPUT="json" ;; + -h|--help) sed -n '3,30p' "$0"; exit 0 ;; + *) echo "Unknown parameter passed: $1" >&2; exit 1 ;; + esac + shift +done + +if [ -z "$VERSION" ]; then + echo "Usage: gather-release-notes.sh -v [--json]" >&2 + exit 1 +fi + +if [[ ! "$VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]; then + echo "Invalid version format. Please use the major.minor.patch format (e.g. 26.7.0)." >&2 + exit 1 +fi + +for tool in gh jq; do + if ! command -v "$tool" >/dev/null 2>&1; then + echo "Required tool '$tool' not found on PATH." >&2 + exit 1 + fi +done + +# Minor version, e.g. 26.7 from 26.7.0 +MINOR=$(echo "$VERSION" | cut -d. -f1,2) + +RELEASE_LABEL="release/$VERSION" +SCHEDULED_LABEL="scheduled-for/$VERSION" +QUERY="org:stackabletech label:$SCHEDULED_LABEL,$RELEASE_LABEL label:release-note,release-note/action-required" + +# ------------------------------ +# Fetch (single paginated search; the search API already returns bodies) +# ------------------------------ + +ITEMS_FILE=$(mktemp) +trap 'rm -f "$ITEMS_FILE"' EXIT + +# --paginate over search returns one JSON object per page; slurp the .items +# arrays together and de-duplicate by html_url (defensive against overlap). +gh api -X GET search/issues --paginate -f q="$QUERY" --jq '.items[]' \ + | jq -s 'unique_by(.html_url)' > "$ITEMS_FILE" + +if [ "$OUTPUT" = "json" ]; then + cat "$ITEMS_FILE" + exit 0 +fi + +# ------------------------------ +# Helper: is this item an action-required one? +# ------------------------------ +is_action_required() { echo "$1" | jq -e 'any(.labels[].name; . == "release-note/action-required")' >/dev/null; } + +# ------------------------------ +# Counts (ground truth = number of items actually returned) +# ------------------------------ +TOTAL=$(jq 'length' "$ITEMS_FILE") +N_ISSUES=$(jq '[.[] | select(.pull_request == null)] | length' "$ITEMS_FILE") +N_PRS=$(jq '[.[] | select(.pull_request != null)] | length' "$ITEMS_FILE") + +echo "# Release-notes source data for $VERSION (minor: $MINOR)" +echo "#" +echo "# Query: (label:$SCHEDULED_LABEL OR label:$RELEASE_LABEL)" +echo "# AND (label:release-note OR label:release-note/action-required)" +echo +echo "## Counts (ground truth via pagination, not search total_count)" +echo " Issues: $N_ISSUES" +echo " PRs: $N_PRS" +echo " Total: $TOTAL" +echo + +# ------------------------------ +# Hygiene flags +# ------------------------------ +echo "## Hygiene flags" +echo +echo "### Items still labelled '$SCHEDULED_LABEL'" +echo "# (merged PRs should be re-labelled '$RELEASE_LABEL'; open ones must land or be dropped)" +SCHEDULED=$(jq -r --arg L "$SCHEDULED_LABEL" ' + [.[] | select(any(.labels[].name; . == $L))] + | if length == 0 then " (none)" else + .[] | " \(.repository_url | sub(".*/repos/stackabletech/";"")) #\(.number) [" + + (if .pull_request == null then "issue" + elif .pull_request.merged_at != null then "PR-MERGED" + else "PR-OPEN" end) + "] \(.title)" + end' "$ITEMS_FILE") +echo "$SCHEDULED" +echo +echo "### Open / unmerged PRs in the set (resolve before release)" +OPEN_PRS=$(jq -r ' + [.[] | select(.pull_request != null and .state == "open")] + | if length == 0 then " (none)" else + .[] | " \(.repository_url | sub(".*/repos/stackabletech/";"")) #\(.number) \(.title)" + end' "$ITEMS_FILE") +echo "$OPEN_PRS" +echo + +# ------------------------------ +# Items grouped by repository, with release-note snippet +# ------------------------------ +echo "## Items grouped by repository" + +REPOS=$(jq -r '[.[] | .repository_url | sub(".*/repos/stackabletech/";"")] | unique | .[]' "$ITEMS_FILE") + +while IFS= read -r repo; do + [ -z "$repo" ] && continue + echo + echo "### $repo" + # Iterate items for this repo (compact JSON per line) + jq -c --arg R "$repo" '.[] | select((.repository_url | sub(".*/repos/stackabletech/";"")) == $R)' "$ITEMS_FILE" \ + | while IFS= read -r item; do + number=$(echo "$item" | jq -r '.number') + title=$(echo "$item" | jq -r '.title') + url=$(echo "$item" | jq -r '.html_url') + labels=$(echo "$item" | jq -r '[.labels[].name] | join(", ")') + if echo "$item" | jq -e '.pull_request == null' >/dev/null; then + kind="issue" + elif echo "$item" | jq -e '.pull_request.merged_at != null' >/dev/null; then + kind="PR-merged" + else + kind="PR-$(echo "$item" | jq -r '.state')" + fi + marker="" + if is_action_required "$item"; then marker=" **ACTION-REQUIRED**"; fi + + echo "- #$number [$kind]$marker $title" + echo " $url" + echo " labels: $labels" + # Extract the "Release note(s)" section from the body, if present. + snippet=$(echo "$item" | jq -r '.body // ""' | awk ' + BEGIN { grab=0 } + /^#+[[:space:]]*[Rr]elease[[:space:]]*[Nn]ote/ { grab=1; next } + grab && /^#+[[:space:]]/ { exit } + grab { print } + ' | sed 's/^/ /' | sed '/^[[:space:]]*$/d') + if [ -n "$snippet" ]; then + echo " release-note snippet:" + echo "$snippet" + else + echo " release-note snippet: (none found in body)" + fi + done +done <<< "$REPOS" From c48ed9259a01de66a8410198d61f3c46074bb708 Mon Sep 17 00:00:00 2001 From: Nick Larsen Date: Fri, 17 Jul 2026 10:45:34 +0200 Subject: [PATCH 02/10] Improve generated script It no longer tries to find the "release notes" since they are quite free form. Instead, an LLM can do that but after the script pulls the content. --- scripts/gather-release-notes.sh | 83 ++++++++++++++++----------------- 1 file changed, 41 insertions(+), 42 deletions(-) diff --git a/scripts/gather-release-notes.sh b/scripts/gather-release-notes.sh index 8368d49fe..e93408cb2 100755 --- a/scripts/gather-release-notes.sh +++ b/scripts/gather-release-notes.sh @@ -85,15 +85,28 @@ gh api -X GET search/issues --paginate -f q="$QUERY" --jq '.items[]' \ | jq -s 'unique_by(.html_url)' > "$ITEMS_FILE" if [ "$OUTPUT" = "json" ]; then - cat "$ITEMS_FILE" + # Enrich each item with its comments and a trimmed set of fields, so the + # release-note text (which authors write free-form, in the body OR a + # comment) is available in one place for a human/agent to read and + # interpret. Deliberately NO regex extraction here - see the note below. + jq -c '.[]' "$ITEMS_FILE" | while IFS= read -r item; do + full=$(echo "$item" | jq -r '.repository_url | sub(".*/repos/";"")') + number=$(echo "$item" | jq -r '.number') + comments=$(gh api "repos/$full/issues/$number/comments" --jq '[.[] | {author: .user.login, body}]' 2>/dev/null || echo '[]') + echo "$item" | jq -c --argjson comments "$comments" '{ + repo: (.repository_url | sub(".*/repos/";"")), + number, title, html_url, state, + is_pr: (.pull_request != null), + merged: (.pull_request.merged_at != null), + labels: [.labels[].name], + action_required: (any(.labels[].name; . == "release-note/action-required")), + body: (.body // ""), + comments: $comments + }' + done | jq -s '.' exit 0 fi -# ------------------------------ -# Helper: is this item an action-required one? -# ------------------------------ -is_action_required() { echo "$1" | jq -e 'any(.labels[].name; . == "release-note/action-required")' >/dev/null; } - # ------------------------------ # Counts (ground truth = number of items actually returned) # ------------------------------ @@ -139,9 +152,19 @@ echo "$OPEN_PRS" echo # ------------------------------ -# Items grouped by repository, with release-note snippet +# Item inventory grouped by repository (metadata only). +# +# NOTE: we deliberately do NOT try to extract the release-note text here. +# Authors write it free-form - under headings like `### Release notes`, an +# inline `Release-Note:` label, a bare `# Release notes` in a *linked* comment, +# or interleaved with reviewer instructions. A regex can't reliably tell a real +# note from a template checklist line ("- [ ] Release note snippet added") or +# reviewer chatter. Reading and interpreting that text is the skill's job: +# run this script with `--json` to get each item's full body + comments, then +# read them. This inventory is just the deterministic "what's in the release". # ------------------------------ -echo "## Items grouped by repository" +echo "## Item inventory grouped by repository" +echo "# For the release-note text of each item, run this script with --json and read the body + comments." REPOS=$(jq -r '[.[] | .repository_url | sub(".*/repos/stackabletech/";"")] | unique | .[]' "$ITEMS_FILE") @@ -149,38 +172,14 @@ while IFS= read -r repo; do [ -z "$repo" ] && continue echo echo "### $repo" - # Iterate items for this repo (compact JSON per line) - jq -c --arg R "$repo" '.[] | select((.repository_url | sub(".*/repos/stackabletech/";"")) == $R)' "$ITEMS_FILE" \ - | while IFS= read -r item; do - number=$(echo "$item" | jq -r '.number') - title=$(echo "$item" | jq -r '.title') - url=$(echo "$item" | jq -r '.html_url') - labels=$(echo "$item" | jq -r '[.labels[].name] | join(", ")') - if echo "$item" | jq -e '.pull_request == null' >/dev/null; then - kind="issue" - elif echo "$item" | jq -e '.pull_request.merged_at != null' >/dev/null; then - kind="PR-merged" - else - kind="PR-$(echo "$item" | jq -r '.state')" - fi - marker="" - if is_action_required "$item"; then marker=" **ACTION-REQUIRED**"; fi - - echo "- #$number [$kind]$marker $title" - echo " $url" - echo " labels: $labels" - # Extract the "Release note(s)" section from the body, if present. - snippet=$(echo "$item" | jq -r '.body // ""' | awk ' - BEGIN { grab=0 } - /^#+[[:space:]]*[Rr]elease[[:space:]]*[Nn]ote/ { grab=1; next } - grab && /^#+[[:space:]]/ { exit } - grab { print } - ' | sed 's/^/ /' | sed '/^[[:space:]]*$/d') - if [ -n "$snippet" ]; then - echo " release-note snippet:" - echo "$snippet" - else - echo " release-note snippet: (none found in body)" - fi - done + jq -r --arg R "$repo" ' + def kind: if .pull_request == null then "issue" + elif .pull_request.merged_at != null then "PR-merged" + else "PR-" + .state end; + def ar: if any(.labels[].name; . == "release-note/action-required") + then " **ACTION-REQUIRED**" else "" end; + [.[] | select((.repository_url | sub(".*/repos/stackabletech/";"")) == $R)] + | sort_by(.number)[] + | "- #\(.number) [\(kind)]\(ar) \(.title)\n \(.html_url)\n labels: \([.labels[].name] | join(", "))" + ' "$ITEMS_FILE" done <<< "$REPOS" From c6b4621c256994b4243c861a2193a4bc39a1d497 Mon Sep 17 00:00:00 2001 From: Nick Larsen Date: Fri, 17 Jul 2026 13:27:55 +0200 Subject: [PATCH 03/10] Rearranged release notes after catching the missing release notes from issues. --- .../partials/release-notes/release-26.7.adoc | 355 ++++++++++++++---- 1 file changed, 290 insertions(+), 65 deletions(-) diff --git a/modules/ROOT/partials/release-notes/release-26.7.adoc b/modules/ROOT/partials/release-notes/release-26.7.adoc index 388f4a8ff..e6cd98d4d 100644 --- a/modules/ROOT/partials/release-notes/release-26.7.adoc +++ b/modules/ROOT/partials/release-notes/release-26.7.adoc @@ -1,30 +1,37 @@ // DRAFT release notes for SDP 26.7.0. // Sourced from stackabletech issues/PRs labelled (release/26.7.0 OR scheduled-for/26.7.0) -// AND (release-note OR release-note/action-required). 13 issues + 29 PRs. -// All `// TODO(verify)` markers must be resolved before this is merged/published. +// AND (release-note OR release-note/action-required): 13 issues + 29 PRs. +// Wording is taken from the authors' own release-note snippets (body or comment) where they wrote one. +// Items with NO author snippet are drafted from the PR body and marked `// TODO(author snippet)`. +// All `// TODO(...)` markers must be resolved before this is merged/published. == Release 26.7 === 26.7.0 -// TODO(verify): exact release date. Tracking issue #847 target was 2026-07-17, shifted ~10 days -> ~2026-07-27. +// TODO(verify): exact release date. Tracking issue #847 target was 2026-07-17, shifted ~10 days. Released on 2026-07-27. -// TODO(highlights): written last, once all sections below are agreed. +// TODO(highlights): write last, once all sections below are agreed. Likely candidates: +// * Generic database connection mechanism rolled out across operators. +// * Trino gains a first-class PostgreSQL connector. +// * Custom OIDC clientAuthenticationMethod (fixes the 26.3 Druid OIDC issue). // [TIP,caption=Release highlights] // ==== -// * Highlight 1 // ==== -// TODO(breaking): overview of breaking changes, written last. Candidates (action-required / chore!): -// * Generic database connection (Druid field rename) - link:#druid-features-26_7_0 -// * Apache Airflow EXPERIMENTAL_FILE_* rename - link:#airflow-removals-26_7_0 -// * Apache Superset EXPERIMENTAL_FILE_* rename - link:#superset-removals-26_7_0 -// * Apache NiFi 1.x CRD fields removed - link:#nifi-removals-26_7_0 -// * Apache Spark application templates now namespaced - link:#spark-fixes-26_7_0 -// [WARNING,caption=Overview of breaking changes] -// ==== -// ==== +[WARNING,caption=Overview of breaking changes] +==== +The following components of the SDP contain breaking changes for this release: + +* link:#generic-db-26_7_0[Generic database connection] (Apache Airflow, Apache Druid, Apache Hive, Apache Superset) +* link:#airflow-removals-26_7_0[Apache Airflow] (renamed `EXPERIMENTAL_FILE_*` settings) +* link:#nifi-removals-26_7_0[Apache NiFi] (removed NiFi 1.x CRD fields) +* link:#spark-fixes-26_7_0[Apache Spark] (application templates are now namespaced) +* link:#superset-removals-26_7_0[Apache Superset] (renamed `EXPERIMENTAL_FILE_*` settings) +// TODO(verify): confirm the image-registry change (issues#716) shipped in 26.7.0 and is breaking for non-Helm installs; if so keep the line below, otherwise remove it. +// * link:#registry-26_7_0[Image registry / repository resolution] +==== ==== New platform features @@ -32,121 +39,339 @@ Released on 2026-07-27. ====== Miscellaneous -* The SDP now provides a generic database connection mechanism, giving a consistent way to configure connections to external SQL databases across operators. - It has been rolled out to the Apache Airflow (https://github.com/stackabletech/airflow-operator/pull/754[airflow-operator#754]), Apache Hive (https://github.com/stackabletech/hive-operator/pull/674[hive-operator#674]) and Apache Superset (https://github.com/stackabletech/superset-operator/pull/722[superset-operator#722]) operators, and underpins new database connector support in Apache Druid and Trino (see the respective sections below). +* [[generic-db-26_7_0]]The SDP now provides a generic database connection mechanism, giving a consistent, typed way to configure connections to external SQL databases across operators. + The same typed structure is used in every CRD, so it can be copied between stacklets. + -- [WARNING] ==== -*Breaking:* In the Stackable operator for Apache Druid the field `metadata_storage_database` has been renamed to `metadata_database`. +*Breaking:* This replaces the previous free-form connection strings with a typed struct, and requires changes to your custom resources (and, in some cases, your Secrets): + +* Apache Airflow: the `.clusterConfig.credentialsSecret` field is renamed to `.clusterConfig.credentialsSecretName`, and the database connection becomes a typed struct. See https://github.com/stackabletech/airflow-operator/pull/754[airflow-operator#754]. +* Apache Druid: the `metadataStorageDatabase` field is renamed to `metadataDatabase` and restructured. See https://github.com/stackabletech/druid-operator/pull/814[druid-operator#814]. +* Apache Hive: the connection becomes a typed struct; because the `database` field is removed, you must record the `username` and add it to the Secret named in `credentialsSecretName` under the `username` key before upgrading. See https://github.com/stackabletech/hive-operator/pull/674[hive-operator#674]. +* Apache Superset: the connection becomes a typed struct, and the operator now uses an internal Secret for the Superset `SECRET_KEY` (created automatically if missing). See https://github.com/stackabletech/superset-operator/pull/722[superset-operator#722]. ==== -// TODO(verify): xref to the generic database connection concept/usage docs once the page exists. +// TODO(verify): xref to the per-operator database-connection usage guides (docs checking). Tracked in https://github.com/stackabletech/issues/issues/238[issues#238]. -- -* Operators that support OIDC authentication can now configure the `clientAuthenticationMethod`. - This resolves the Apache Druid OIDC issue where a provider advertising `private_key_jwt` could cause authentication to fail (a known issue in the 26.3 release). - // TODO(verify): confirm which operators expose this field and add the relevant xref. +* Setting a custom client authentication method is now supported for Apache Airflow, Apache Superset and Apache Druid. + Since Apache Druid 35 and 36, OIDC authentication can fail if the provider (for example Keycloak) advertises `private_key_jwt`, because Druid may attempt to use it. + This feature lets you override the method so that OIDC authentication works again (this was a known issue in the 26.3 release). Tracked in https://github.com/stackabletech/issues/issues/838[issues#838]. -* Operators can now resolve product images from registries other than `oci.stackable.tech`, making it possible to install the SDP from a mirror or private registry. - // TODO(verify): confirm this landed for 26.7.0 (issue #716 is a tracking epic) and add the xref to the image-registry docs. - Tracked in https://github.com/stackabletech/issues/issues/716[issues#716]. ++ +// TODO(hygiene): issues#838 is still labelled scheduled-for/26.7.0 - should be release/26.7.0. + +// TODO(verify - image registry, issues#716): the author's draft note says the SDP now also publishes Helm Charts to quay.io directly from CI, and that operators installed from there pull product images from quay.io. However the quay.io part may not have shipped for 26.7.0, lfrancke's review questions were unresolved, and it is only breaking for non-Helm installs (a new env var / CLI arg is required). Confirm scope before writing. See https://github.com/stackabletech/issues/issues/716[issues#716]. ===== Apache Airflow -Apache Airflow can now authenticate git-sync against a repository using a custom CA, adding first-class support for CAs alongside the existing SSH and basic-auth options. +The Apache Airflow operator now allows specifying CA certificates when using git-sync. // TODO(verify): xref to the DAG mounting / git-sync usage guide. Implemented in https://github.com/stackabletech/airflow-operator/pull/750[airflow-operator#750]. ===== Apache NiFi -Apache NiFi can now authenticate git-sync against a repository using a custom CA, adding first-class support for CAs. +The Apache NiFi operator now allows specifying CA certificates when using git-sync. // TODO(verify): xref to the NiFi git-sync usage guide. Implemented in https://github.com/stackabletech/nifi-operator/pull/903[nifi-operator#903]. ===== Apache Superset -Two new roles, `worker` and `beat` (limited to a single replica), have been added so that Celery workers and the Celery beat scheduler can be enabled. -// TODO(verify): the results backend currently supports Redis only (no S3); confirm whether to call this out. xref to Superset usage guide. +Apache Superset now supports running asynchronous queries via Celery. +Two new roles, `worker` and `beat` (limited to a single replica), have been added. +Redis is supported as the broker queue and results backend. +// TODO(verify): xref to the Superset usage guide. Implemented in https://github.com/stackabletech/superset-operator/pull/724[superset-operator#724]. ===== Open Policy Agent -The OPA operator now supports typed `configOverrides` for OPA's `config.json`, via JSON merge patch, JSON patches or a full file replacement. -// TODO(verify): xref to the OPA configuration overrides docs. +The Open Policy Agent operator now supports `configOverrides` for its `config.json`. +JSON Patches and JSON Merge Patches let you override the nested structure of the file. +// TODO(verify): xref to the OPA configuration/environment overrides docs. Implemented in https://github.com/stackabletech/opa-operator/pull/818[opa-operator#818]. ===== OpenSearch -* Security configuration files are now hot-reloaded when changed, so that settings managed by the operator are applied without Pod restarts. +* Settings managed by the operator are now hot-reloaded when changed, without Pod restarts. // TODO(verify): xref to the OpenSearch security docs. Implemented in https://github.com/stackabletech/opensearch-operator/pull/130[opensearch-operator#130]. -* The operator now writes the actual node FQDNs into `plugins.security.nodes_dn` instead of a generic subject DN. - // TODO(verify): confirm feature vs fix categorisation for opensearch-operator#144. - Implemented in https://github.com/stackabletech/opensearch-operator/pull/144[opensearch-operator#144]. +* The OpenSearch operator now supports JSON Merge Patches, JSON Patches and user-provided JSON files in `configOverrides`, making it easier to define nested types and non-string values. + // TODO(verify): xref to the OpenSearch configuration/environment overrides docs. Confirm feature vs improvement placement (author wrote "OpenSearch" without a section). + Implemented in https://github.com/stackabletech/opensearch-operator/pull/137[opensearch-operator#137]. ===== Trino -* Trino now supports the PostgreSQL connector, allowing PostgreSQL databases to be queried as a Trino catalog. +* Trino now has first-class support for the PostgreSQL connector, built on the new generic database connection mechanism. + Previously, users had to fall back to the `generic` connector. // TODO(verify): xref to the Trino PostgreSQL connector docs. Implemented in https://github.com/stackabletech/trino-operator/pull/883[trino-operator#883]. * A new `.spec.name.inferred.replaceHyphensWithUnderscores` field on `TrinoCatalog` allows tweaking the catalog name in Trino by replacing `-` with `_`. This lets you use valid Kubernetes resource names while keeping the convenience of `_` in catalog names. Implemented in https://github.com/stackabletech/trino-operator/pull/903[trino-operator#903]. -===== Stackable commons-operator +===== Stackable secret-operator -The commons-operator restart-controller can now be told to ignore specific ConfigMaps and Secrets. -Set the `restarter.stackable.tech/ignore` label on a ConfigMap or Secret, or the `restarter.stackable.tech/ignore-configmap.x` / `restarter.stackable.tech/ignore-secret.x` annotations on a StatefulSet, to exclude them from triggering restarts. -// TODO(verify): xref to the restarter docs. -Implemented in https://github.com/stackabletech/commons-operator/pull/410[commons-operator#410]. +The Stackable secret-operator now supports adding domain components (DCs) to the subject DN of TLS certificates, via the `secrets.stackable.tech/backend.autotls.cert.domain-components-in-subject-dn` volume annotation. +// TODO(verify): xref to the secret-operator volume docs. +Implemented in https://github.com/stackabletech/secret-operator/pull/708[secret-operator#708]. -===== Stackable secret-operator +// TODO(decide): stackablectl#187 (uninstall Stacks and Demos) - stackablectl/stackable-cockpit is versioned separately from the SDP. If it belongs here, the author's note is: +// - All demos and stacks can now be installed in non-default namespaces. +// - `stackablectl demo uninstall` / `stackablectl stack uninstall` remove the demo/stack namespace, the Stackable operators and CRDs, and the operator namespace (use `--skip-operators-and-crds` to keep operators/CRDs). +// See https://github.com/stackabletech/stackablectl/issues/187[stackablectl#187]. + +==== Platform improvements -* The name of the CA certificate entry written by a `TrustStore` is now configurable, so it can be aligned with what consumers such as OpenShift expect (`tls.crt` vs `ca.crt`). - // TODO(verify): xref to the TrustStore docs. - Implemented in https://github.com/stackabletech/secret-operator/pull/679[secret-operator#679]. -* Domain components (DCs) can now be added to the subject DN of TLS certificates via the `secrets.stackable.tech/backend.autotls.cert.domain-components-in-subject-dn` volume annotation. - // TODO(verify): xref to the secret-operator volume docs. - Implemented in https://github.com/stackabletech/secret-operator/pull/708[secret-operator#708]. +===== General -// TODO(verify): stackablectl#187 (uninstall Stacks and Demos) - stackablectl is versioned separately; confirm whether it belongs in these platform release notes. +====== Vulnerabilities -// ============================================================================ -// TODO: remaining sections (next review rounds) - stubbed to keep the include valid. -// ============================================================================ +// TODO(verify): CVE counts. e.g. "NNN CVEs were fixed in the Stackable product images. This includes N critical and NN high-severity CVEs." -==== Platform improvements +====== Miscellaneous + +* ConfigMaps and Secrets can now be excluded from the restart-controller. + Set the `restarter.stackable.tech/ignore` label on a ConfigMap or Secret, or the `restarter.stackable.tech/ignore-configmap.x` / `restarter.stackable.tech/ignore-secret.x` annotations on a StatefulSet. + // TODO(verify): xref to the restarter docs. + Implemented in https://github.com/stackabletech/commons-operator/pull/410[commons-operator#410]. + +===== Apache NiFi -// TODO: NiFi startup times (nifi#953), OPA DaemonSet maxSurge (opa#819), -// typed OpenSearch configOverrides (opensearch#137), General > Vulnerabilities (CVE counts TODO). +// TODO(author snippet): nifi-operator#953 has no author-written release note (only CI links); wording below is drafted from the PR body - please confirm. +When the number of NiFi nodes is fixed (all role-group replicas are set, i.e. no autoscaling), the operator now sets `nifi.cluster.flow.election.max.candidates` to that node count. +The cluster then finishes flow election as soon as all expected nodes have connected, instead of waiting out `nifi.cluster.flow.election.max.wait.time` (5 minutes), which greatly reduces cold-start times. +Implemented in https://github.com/stackabletech/nifi-operator/pull/953[nifi-operator#953]. + +===== Open Policy Agent + +The OPA DaemonSet now uses `maxSurge=1` and `maxUnavailable=0` for its rolling update strategy. +During rolling updates (for example when an OPA config change triggers a restart), a new OPA Pod is created and must become ready before the old one is terminated. +This removes the brief availability gap that previously caused errors in products such as Trino. +Implemented in https://github.com/stackabletech/opa-operator/pull/819[opa-operator#819]. + +===== OpenSearch + +The verification of the OpenSearch nodes' certificates has been extended. +The nodes' certificates now also include the FQDNs of the nodes, which are verified by the other nodes using the `plugins.security.nodes_dn` configuration setting. +Previously, it was only verified that the certificates were issued by the secret-operator. +Implemented in https://github.com/stackabletech/opensearch-operator/pull/144[opensearch-operator#144]. + +===== Stackable secret-operator + +The name of the CA certificate entry written by a `TrustStore` is now configurable. +This makes it possible to align with what consumers such as OpenShift expect, which reads the CA from a Secret under the `tls.crt` key rather than `ca.crt`. +// TODO(author snippet): secret-operator#679 has no author-written release note (only a CRD-naming discussion); wording drafted from the PR body - please confirm. +// TODO(verify): xref to the TrustStore docs. +Implemented in https://github.com/stackabletech/secret-operator/pull/679[secret-operator#679]. ==== Platform fixes -// TODO: internal-secret mutability (issues#843; airflow#770, trino#876), -// SecretClasses requiring client cert regression (issues#547), -// Spark connect user args (spark#696), Spark graceful SIGTERM shutdown (docker-images#1564), -// Spark application templates namespaced (spark#694/#720 - BREAKING). +===== General + +* All operators now request only the parts of the secret data they actually need for operator-managed secret volumes. + This uses the xref:secret-operator:volume.adoc#_secrets_stackable_techprovision_parts[`secrets.stackable.tech/provision-parts`] annotation introduced in SDP 26.3.0. + This resolves the regression where SecretClasses used only to validate server certificates also required a client certificate (`tls.crt` / `tls.key`) to be provided. + Fixed in https://github.com/stackabletech/issues/issues/547[issues#547]. +* Internal secrets are now stored in mutable Kubernetes Secret objects. + Previously these objects were immutable and, if deleted, were recreated with new random values; because immutable Secrets are cached, this could disrupt clusters (for example workers failing to join). + This release migrates existing internal Secrets to mutable versions for Trino (https://github.com/stackabletech/trino-operator/pull/876[trino-operator#876]) and Apache Airflow (https://github.com/stackabletech/airflow-operator/pull/770[airflow-operator#770]). + Tracked in https://github.com/stackabletech/issues/issues/843[issues#843]. + +===== Apache Airflow + +OPA authorization now works for Apache Airflow 3 with any user role. +Previously, OPA rulesets were only applied for users with the `Admin` role. +// TODO(hygiene): airflow-operator#800 is still labelled scheduled-for/26.7.0 - should be release/26.7.0. Underlying fix shipped in docker-images#1512. +Fixed in https://github.com/stackabletech/airflow-operator/pull/800[airflow-operator#800]. + +===== Apache NiFi + +// TODO(author snippet): no release note was written on nifi-opa-plugin#30 (issue) or #31 (fix PR); wording below is a suggestion drafted from the issue/PR - please confirm. +// TODO(hygiene): the issue nifi-opa-plugin#30 is still OPEN and labelled scheduled-for/26.7.0. The fix (nifi-opa-plugin#31) is merged and labelled release/26.7.0. Close the issue and/or move its label to release/26.7.0. +The NiFi OPA authorizer plugin no longer exhausts system threads under load. +Previously, the plugin created a new HTTP client for every authorization request, which could spawn thousands of native threads under NiFi load and lead to errors such as `EAGAIN` or out-of-memory conditions. +With this release the plugin uses a single shared, pooled HTTP client for OPA authorization requests. +No configuration changes are required, as defaults are provided; the pool can be tuned via the `OPA_HTTP_POOL_SIZE`, `OPA_HTTP_CONNECT_TIMEOUT_MS` and `OPA_HTTP_REQUEST_TIMEOUT_MS` environment variables. +Fixed in https://github.com/stackabletech/nifi-opa-plugin/issues/30[nifi-opa-plugin#30]. + +[#spark-fixes-26_7_0] +===== Apache Spark + +* *Breaking:* Spark application templates are now namespaced objects instead of cluster-wide objects. + They were unintentionally released as cluster-wide objects in SDP 26.3. + You must recreate any templates in the namespaces where the applications run. + Fixed in https://github.com/stackabletech/spark-k8s-operator/pull/694[spark-k8s-operator#694] (reapplied in https://github.com/stackabletech/spark-k8s-operator/pull/720[spark-k8s-operator#720]). + // TODO(hygiene): spark-k8s-operator#720 is still labelled scheduled-for/26.7.0 - should be release/26.7.0. +* User arguments are now correctly propagated to the Spark Connect server. + Previously, the newline concatenation in the Spark Connect server entrypoint was wrong, so user arguments were dropped. + Fixed in https://github.com/stackabletech/spark-k8s-operator/pull/696[spark-k8s-operator#696]. +* Spark executors now shut down gracefully on `SIGTERM`. + Previously the executor did not receive `SIGTERM` when the driver did, so it was `SIGKILL`ed after the grace period. + Fixed in https://github.com/stackabletech/docker-images/pull/1564[docker-images#1564]. + +===== Open Policy Agent + +// TODO(BLOCKER - unmerged): the fix PR opa-operator#859 is still OPEN (not merged) and the issue opa-operator#858 is still OPEN and labelled scheduled-for/26.7.0. This note must not ship until #859 is merged; otherwise remove this entry. +The Microsoft Entra backend of the OPA user-info-fetcher now retrieves complete group memberships for policy checks. +Previously, it read only the first page of a user's group memberships, so users belonging to many groups received an incomplete group list. +With this release, the Entra backend follows the result pages until every page has been retrieved (bounded to 100 pages as a safeguard). +Fixed in https://github.com/stackabletech/opa-operator/issues/858[opa-operator#858]. ++ +// TODO(orphan note - needs its own labelled source): lfrancke's release note on issue #858 also describes a SECOND change not traceable to #858/#859 - "the user-info-fetcher's outbound HTTP requests (to Microsoft Entra, Keycloak, and the XFSC AAS) now have a 60-second overall timeout and a 15-second connection timeout (previously none, so a slow backend could hang lookups indefinitely)". Find the PR that implemented this, confirm it's in 26.7.0, and label it - then add it here. + +===== Trino + +Hot-reloading of password file Secrets works again. +In the previous release a change to the Secret triggered a Pod restart; the password file Secrets are now excluded from the restart-controller, so hot-reloading is supported again. +Fixed in https://github.com/stackabletech/trino-operator/pull/868[trino-operator#868]. ==== Platform deprecations -// TODO: product version deprecations (from docker-images version issues). +// TODO(verify): no feature/API deprecations found in the labelled items (only product-version deprecations, which are listed under Supported versions below). Confirm there are none. ==== Platform removals -// TODO: NiFi 1.x CRD fields (nifi#960 - BREAKING), Airflow/Superset EXPERIMENTAL_FILE_* rename -// (airflow#777, superset#721 - BREAKING), removed product versions. +[#airflow-removals-26_7_0] +===== Apache Airflow + +*Breaking:* `EXPERIMENTAL_FILE_HEADER` and `EXPERIMENTAL_FILE_FOOTER` in `webserver_config.py` (for arbitrary Python code) have been renamed to `FILE_HEADER` and `FILE_FOOTER`. +Removed in https://github.com/stackabletech/airflow-operator/pull/777[airflow-operator#777]. + +[#nifi-removals-26_7_0] +===== Apache NiFi + +// TODO(author snippet): nifi-operator#960 has no author-written release note (only a CI link); wording drafted from the PR body - please confirm. +*Breaking:* The NiFi 1.x configuration CRD fields and code have been removed, following the removal of NiFi 1.28.1. +This removes the Prometheus reporting-task Job and its `spec.clusterConfig.createReportingTaskJob` field, the pre-2.x non-rolling upgrade handling, the dedicated metrics port, and the sensitive-properties algorithms that were only supported on NiFi 1.x. +Removed in https://github.com/stackabletech/nifi-operator/pull/960[nifi-operator#960]. + +[#superset-removals-26_7_0] +===== Apache Superset + +*Breaking:* `EXPERIMENTAL_FILE_HEADER` and `EXPERIMENTAL_FILE_FOOTER` in `superset_config.py` (for arbitrary Python code) have been renamed to `FILE_HEADER` and `FILE_FOOTER`. +Removed in https://github.com/stackabletech/superset-operator/pull/721[superset-operator#721]. ==== Supported versions -// TODO(UNVERIFIED - reconcile against portal.stackable.build screenshot before publishing): -// product versions (new/LTS/deprecated/removed), Kubernetes versions, OpenShift versions. +===== Product versions + +// TODO(UNVERIFIED - stop-gap from GitHub): the list below is derived only from the docker-images "Update major/minor versions" issues that carry a release-note label (Airflow, Druid, NiFi, OpenSearch, Superset, ZooKeeper). +// The other products (HBase, Hadoop, Hive, Kafka, Spark, OPA, Trino, Vector) are NOT yet captured here because their version issues are not labelled. +// RECONCILE the complete set against a portal.stackable.build screenshot (source of truth; hub.stackable.tech later) before publishing. + +As with previous SDP releases, many product images have been updated to newer versions. +Refer to the xref:operators:supported_versions.adoc[supported versions] documentation for a complete overview including LTS versions or deprecations. + +====== New versions + +The following new product versions are now supported: + +* Apache Airflow: https://github.com/stackabletech/docker-images/issues/1489[3.2.1] +** Note: the Airflow 3.2.1 webserver no longer serves metrics, and this version uses Python 3.14. // TODO(verify) whether the metrics change is intended. +* Apache Druid: https://github.com/stackabletech/docker-images/issues/1490[37.0.0 (LTS)] +** Note: Druid 37.0.0 switched to AWS SDK v2. TLS is now enforced for S3 connections, and the S3 `region` is now required (the operator passes `aws.region` from `S3Connection.region`). +* Apache NiFi: https://github.com/stackabletech/docker-images/issues/1498[2.9.0] +* OpenSearch: https://github.com/stackabletech/docker-images/issues/1500[3.6.0] +* Apache Superset: https://github.com/stackabletech/docker-images/issues/1504[6.1.0] +* Apache ZooKeeper: https://github.com/stackabletech/docker-images/issues/1506[3.9.5 (LTS)] + +====== Deprecated versions + +The following product versions are deprecated and will be removed in a later release: + +* Apache Airflow: https://github.com/stackabletech/docker-images/issues/1489[3.1.6] +* Apache Druid: https://github.com/stackabletech/docker-images/issues/1490[35.0.1], https://github.com/stackabletech/docker-images/issues/1490[30.0.1] +* Apache NiFi: https://github.com/stackabletech/docker-images/issues/1498[2.7.2] +* OpenSearch: https://github.com/stackabletech/docker-images/issues/1500[3.4.0] +* Apache Superset: https://github.com/stackabletech/docker-images/issues/1504[6.0.0] +* Apache ZooKeeper: https://github.com/stackabletech/docker-images/issues/1506[3.9.4] + +====== Removed versions + +The following product versions are no longer supported. +These images for released product versions remain available in the https://oci.stackable.tech/[Stackable OCI Registry,window=_blank]. +Information on how to browse the registry can be found xref:contributor:project-overview.adoc#docker-images[in the project overview documentation,window=_blank]. + +* Apache Druid: https://github.com/stackabletech/docker-images/issues/1490[34.0.0] +* Apache NiFi: https://github.com/stackabletech/docker-images/issues/1498[1.28.1] +// TODO(verify): complete this list once the full product-version set is reconciled against the portal. + +===== Kubernetes versions + +// TODO(verify): supported Kubernetes versions for 26.7.0. In 26.3 these were 1.35-1.31. Confirm against the release definition. + +===== OpenShift versions + +// TODO(verify): supported OpenShift versions for 26.7.0. In 26.3 these were 4.21-4.18. Confirm against the release definition. ==== Upgrade from 26.3 -// TODO: stackablectl + Helm uninstall/install for 26.3 -> 26.7, plus product-specific upgrade notes. +// TODO(verify): consult/collect product-specific upgrade notes (e.g. the generic database connection migrations for Airflow/Druid/Hive/Superset, and the Druid 37 S3 TLS/region change). + +===== Using stackablectl + +====== Upgrade with a single command + +[source,console] +---- +$ stackablectl release upgrade 26.7 +---- + +====== Upgrade with multiple consecutive commands + +Uninstall the `26.3` release + +[source,console] +---- +$ stackablectl release uninstall 26.3 + +Uninstalled release '26.3' + +Use "stackablectl release list" to list available releases. +# ... +---- + +Install the `26.7` release + +[source,console] +---- +$ stackablectl release install 26.7 + +Installed release '26.7' + +Use "stackablectl operator installed" to list installed operators. +---- + +===== Using Helm + +Use `helm list` to list the currently installed operators. + +You can use the following command to uninstall all operators that are part of the `26.3` release: + +[source,console] +---- +$ helm uninstall airflow-operator -operator ... +release "airflow-operator" uninstalled +release "-operator" uninstalled +... +---- + +Install the `26.7` release + +[NOTE] +==== +`helm repo` subcommands are not supported for OCI registries. +The operators are installed directly, without adding the Helm Chart repository first. +==== + +[source,console] +---- +helm install --wait -operator oci://oci.stackable.tech/sdp-charts/-operator --version 26.7.0 +---- [#known-issues-26_7_0] ==== Known issues -// TODO(verify): none known yet. +// TODO(verify): none identified yet from the labelled items. From de1a970595c61032d203110794e01912b6f4b0a8 Mon Sep 17 00:00:00 2001 From: Nick Larsen Date: Fri, 17 Jul 2026 15:49:18 +0200 Subject: [PATCH 04/10] remove a todo --- modules/ROOT/partials/release-notes/release-26.7.adoc | 1 - 1 file changed, 1 deletion(-) diff --git a/modules/ROOT/partials/release-notes/release-26.7.adoc b/modules/ROOT/partials/release-notes/release-26.7.adoc index e6cd98d4d..7ed0b3955 100644 --- a/modules/ROOT/partials/release-notes/release-26.7.adoc +++ b/modules/ROOT/partials/release-notes/release-26.7.adoc @@ -213,7 +213,6 @@ Fixed in https://github.com/stackabletech/nifi-opa-plugin/issues/30[nifi-opa-plu ===== Open Policy Agent -// TODO(BLOCKER - unmerged): the fix PR opa-operator#859 is still OPEN (not merged) and the issue opa-operator#858 is still OPEN and labelled scheduled-for/26.7.0. This note must not ship until #859 is merged; otherwise remove this entry. The Microsoft Entra backend of the OPA user-info-fetcher now retrieves complete group memberships for policy checks. Previously, it read only the first page of a user's group memberships, so users belonging to many groups received an incomplete group list. With this release, the Entra backend follows the result pages until every page has been retrieved (bounded to 100 pages as a safeguard). From c79a9611f26036e7649676101dc9b97622b3065d Mon Sep 17 00:00:00 2001 From: Nick Larsen Date: Fri, 17 Jul 2026 15:56:12 +0200 Subject: [PATCH 05/10] nifi cold start --- modules/ROOT/partials/release-notes/release-26.7.adoc | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/modules/ROOT/partials/release-notes/release-26.7.adoc b/modules/ROOT/partials/release-notes/release-26.7.adoc index 7ed0b3955..94c98a652 100644 --- a/modules/ROOT/partials/release-notes/release-26.7.adoc +++ b/modules/ROOT/partials/release-notes/release-26.7.adoc @@ -139,9 +139,9 @@ Implemented in https://github.com/stackabletech/secret-operator/pull/708[secret- ===== Apache NiFi -// TODO(author snippet): nifi-operator#953 has no author-written release note (only CI links); wording below is drafted from the PR body - please confirm. -When the number of NiFi nodes is fixed (all role-group replicas are set, i.e. no autoscaling), the operator now sets `nifi.cluster.flow.election.max.candidates` to that node count. -The cluster then finishes flow election as soon as all expected nodes have connected, instead of waiting out `nifi.cluster.flow.election.max.wait.time` (5 minutes), which greatly reduces cold-start times. +For clusters with a fixed number of nodes, the operator now sets `nifi.cluster.flow.election.max.candidates` to the total node count. +Cold-start flow election then completes as soon as all expected nodes report in, typically seconds instead of the up-to-5-minute `nifi.cluster.flow.election.max.wait.time` timeout. +The timeout stays at NiFi's upstream default of 5 minutes as a fallback, so there is no correctness change if fewer nodes than expected show up. Implemented in https://github.com/stackabletech/nifi-operator/pull/953[nifi-operator#953]. ===== Open Policy Agent From e286a0322b7e6f49187dd2d1ce581265591c4499 Mon Sep 17 00:00:00 2001 From: Nick Larsen Date: Fri, 17 Jul 2026 15:57:36 +0200 Subject: [PATCH 06/10] kafka pod FQDNs in cert DNs --- modules/ROOT/partials/release-notes/release-26.7.adoc | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/modules/ROOT/partials/release-notes/release-26.7.adoc b/modules/ROOT/partials/release-notes/release-26.7.adoc index 94c98a652..8eb3e9a37 100644 --- a/modules/ROOT/partials/release-notes/release-26.7.adoc +++ b/modules/ROOT/partials/release-notes/release-26.7.adoc @@ -137,6 +137,14 @@ Implemented in https://github.com/stackabletech/secret-operator/pull/708[secret- // TODO(verify): xref to the restarter docs. Implemented in https://github.com/stackabletech/commons-operator/pull/410[commons-operator#410]. +===== Apache Kafka + +The FQDNs of the Kafka Pods are now included in the subject DNs of their TLS certificates. +OPA Rego rules can now check these FQDNs. +Existing OPA rules may need to be adjusted. +// TODO(verify): xref to the Kafka security usage guide (author linked .../kafka/usage-guide/security/#_authorization). +Implemented in https://github.com/stackabletech/kafka-operator/pull/972[kafka-operator#972]. + ===== Apache NiFi For clusters with a fixed number of nodes, the operator now sets `nifi.cluster.flow.election.max.candidates` to the total node count. From 6993019d1b7ea6d9980d65470ae8819eba5e1263 Mon Sep 17 00:00:00 2001 From: Nick Larsen Date: Fri, 17 Jul 2026 15:58:55 +0200 Subject: [PATCH 07/10] TrustStore .spec.tlsPemCaName no configurable --- modules/ROOT/partials/release-notes/release-26.7.adoc | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/modules/ROOT/partials/release-notes/release-26.7.adoc b/modules/ROOT/partials/release-notes/release-26.7.adoc index 8eb3e9a37..304d89d0b 100644 --- a/modules/ROOT/partials/release-notes/release-26.7.adoc +++ b/modules/ROOT/partials/release-notes/release-26.7.adoc @@ -168,9 +168,8 @@ Implemented in https://github.com/stackabletech/opensearch-operator/pull/144[ope ===== Stackable secret-operator -The name of the CA certificate entry written by a `TrustStore` is now configurable. -This makes it possible to align with what consumers such as OpenShift expect, which reads the CA from a Secret under the `tls.crt` key rather than `ca.crt`. -// TODO(author snippet): secret-operator#679 has no author-written release note (only a CRD-naming discussion); wording drafted from the PR body - please confirm. +The name of the key in the ConfigMap/Secret that holds the PEM-encoded CA certificate of a `TrustStore` is now configurable, via `.spec.tlsPemCaName`. +This is needed, for example, to use the generated Secret within an OpenShift Ingress. // TODO(verify): xref to the TrustStore docs. Implemented in https://github.com/stackabletech/secret-operator/pull/679[secret-operator#679]. From bc2bfc926aa1d581a71a4f8eef2b150d171c382a Mon Sep 17 00:00:00 2001 From: Nick Larsen Date: Fri, 17 Jul 2026 16:00:09 +0200 Subject: [PATCH 08/10] NiFi 1 CRD fields gone, and sensitiveProperties removedl --- modules/ROOT/partials/release-notes/release-26.7.adoc | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/modules/ROOT/partials/release-notes/release-26.7.adoc b/modules/ROOT/partials/release-notes/release-26.7.adoc index 304d89d0b..30ba20633 100644 --- a/modules/ROOT/partials/release-notes/release-26.7.adoc +++ b/modules/ROOT/partials/release-notes/release-26.7.adoc @@ -248,9 +248,13 @@ Removed in https://github.com/stackabletech/airflow-operator/pull/777[airflow-op [#nifi-removals-26_7_0] ===== Apache NiFi -// TODO(author snippet): nifi-operator#960 has no author-written release note (only a CI link); wording drafted from the PR body - please confirm. -*Breaking:* The NiFi 1.x configuration CRD fields and code have been removed, following the removal of NiFi 1.28.1. -This removes the Prometheus reporting-task Job and its `spec.clusterConfig.createReportingTaskJob` field, the pre-2.x non-rolling upgrade handling, the dedicated metrics port, and the sensitive-properties algorithms that were only supported on NiFi 1.x. +*Breaking:* Support for NiFi 1.x has been dropped, along with a few CRD fields that only applied to 1.x and had no effect on NiFi 2.x: + +* `spec.clusterConfig.createReportingTaskJob` (including `enabled` and `podOverrides`) is removed. + NiFi 2.x exposes Prometheus metrics directly at `/nifi-api/flow/metrics/prometheus`, so the reporting-task Job is obsolete. +* The deprecated `spec.clusterConfig.sensitiveProperties.algorithm` values are removed. + Use a supported algorithm instead: `nifiArgon2AesGcm256` (default) or `nifiPbkdf2AesGcm256`. + Removed in https://github.com/stackabletech/nifi-operator/pull/960[nifi-operator#960]. [#superset-removals-26_7_0] From b991f64147122ac2f15da76a32975d610156dcf3 Mon Sep 17 00:00:00 2001 From: Nick Larsen Date: Fri, 17 Jul 2026 19:00:23 +0200 Subject: [PATCH 09/10] Nifi OPA Authorizer thread exhaustion --- modules/ROOT/partials/release-notes/release-26.7.adoc | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/modules/ROOT/partials/release-notes/release-26.7.adoc b/modules/ROOT/partials/release-notes/release-26.7.adoc index 30ba20633..1b33bc00f 100644 --- a/modules/ROOT/partials/release-notes/release-26.7.adoc +++ b/modules/ROOT/partials/release-notes/release-26.7.adoc @@ -195,12 +195,10 @@ Fixed in https://github.com/stackabletech/airflow-operator/pull/800[airflow-oper ===== Apache NiFi -// TODO(author snippet): no release note was written on nifi-opa-plugin#30 (issue) or #31 (fix PR); wording below is a suggestion drafted from the issue/PR - please confirm. -// TODO(hygiene): the issue nifi-opa-plugin#30 is still OPEN and labelled scheduled-for/26.7.0. The fix (nifi-opa-plugin#31) is merged and labelled release/26.7.0. Close the issue and/or move its label to release/26.7.0. -The NiFi OPA authorizer plugin no longer exhausts system threads under load. -Previously, the plugin created a new HTTP client for every authorization request, which could spawn thousands of native threads under NiFi load and lead to errors such as `EAGAIN` or out-of-memory conditions. -With this release the plugin uses a single shared, pooled HTTP client for OPA authorization requests. -No configuration changes are required, as defaults are provided; the pool can be tuned via the `OPA_HTTP_POOL_SIZE`, `OPA_HTTP_CONNECT_TIMEOUT_MS` and `OPA_HTTP_REQUEST_TIMEOUT_MS` environment variables. +The NiFi OPA authorizer plugin no longer exhausts native threads under load. +Previously, the plugin could create a new HTTP client for each authorization request, spawning large numbers of selector and executor threads under NiFi load and eventually exhausting native threads. +With this release, all OPA authorization requests share a single pooled HTTP client with a fixed-size daemon-thread executor. +Existing deployments need no configuration changes, as defaults are provided; the client is configurable via `OPA_HTTP_POOL_SIZE` (default 32), `OPA_HTTP_CONNECT_TIMEOUT_MS` (default 2000) and `OPA_HTTP_REQUEST_TIMEOUT_MS` (default 5000). Fixed in https://github.com/stackabletech/nifi-opa-plugin/issues/30[nifi-opa-plugin#30]. [#spark-fixes-26_7_0] From 109a11bab5709a9e69f434a231ff32c7ab4ce982 Mon Sep 17 00:00:00 2001 From: Nick Larsen Date: Fri, 17 Jul 2026 22:17:02 +0200 Subject: [PATCH 10/10] Product changes Based on data in portal, but with a correction for Spark --- .../partials/release-notes/release-26.7.adoc | 68 ++++++++++++++++--- 1 file changed, 58 insertions(+), 10 deletions(-) diff --git a/modules/ROOT/partials/release-notes/release-26.7.adoc b/modules/ROOT/partials/release-notes/release-26.7.adoc index 1b33bc00f..d7f3352a3 100644 --- a/modules/ROOT/partials/release-notes/release-26.7.adoc +++ b/modules/ROOT/partials/release-notes/release-26.7.adoc @@ -265,25 +265,42 @@ Removed in https://github.com/stackabletech/superset-operator/pull/721[superset- ===== Product versions -// TODO(UNVERIFIED - stop-gap from GitHub): the list below is derived only from the docker-images "Update major/minor versions" issues that carry a release-note label (Airflow, Druid, NiFi, OpenSearch, Superset, ZooKeeper). -// The other products (HBase, Hadoop, Hive, Kafka, Spark, OPA, Trino, Vector) are NOT yet captured here because their version issues are not labelled. -// RECONCILE the complete set against a portal.stackable.build screenshot (source of truth; hub.stackable.tech later) before publishing. +// Reconciled against portal.stackable.build (source of truth) by diffing the 26.3 and 26.7 exports. +// Added = in 26.7 not in 26.3; Removed = in 26.3 not in 26.7; statuses (LTS/experimental/deprecated) from the 26.7 export. +// Overrides where the portal was wrong: +// - Spark 4.1.2 appears in the 26.3 portal export by mistake (it was not in 26.3.0; docker-images#1419 added only 3.5.8). It is treated as ADDED in 26.7. Portal maintainer to be notified. +// - Vector is absent from the portal export; taken from docker-images#1486 (add 0.55.0, remove 0.52.0). Portal maintainer notified. +// Deprecated list = versions newly deprecated this release (status changed to deprecated vs 26.3), not the full standing deprecated set. As with previous SDP releases, many product images have been updated to newer versions. Refer to the xref:operators:supported_versions.adoc[supported versions] documentation for a complete overview including LTS versions or deprecations. +====== New LTS versions + +The following product versions were already available before but are now marked as the xref:compliance:policies.adoc#_product_lifecycle_policy[LTS] version: + +* Apache Hive: https://github.com/stackabletech/docker-images/issues/1495[4.2.0 (LTS)] + ====== New versions The following new product versions are now supported: -* Apache Airflow: https://github.com/stackabletech/docker-images/issues/1489[3.2.1] -** Note: the Airflow 3.2.1 webserver no longer serves metrics, and this version uses Python 3.14. // TODO(verify) whether the metrics change is intended. +* Apache Airflow: https://github.com/stackabletech/docker-images/issues/1489[3.2.2] +** Note: this Airflow version uses Python 3.14. // TODO(verify): the docker-images#1489 note about the webserver no longer serving metrics referred to 3.2.1; confirm it still applies to 3.2.2. * Apache Druid: https://github.com/stackabletech/docker-images/issues/1490[37.0.0 (LTS)] ** Note: Druid 37.0.0 switched to AWS SDK v2. TLS is now enforced for S3 connections, and the S3 `region` is now required (the operator passes `aws.region` from `S3Connection.region`). +* Apache HBase: https://github.com/stackabletech/docker-images/issues/1491[2.6.6 (LTS)] +* Apache Hadoop HDFS: https://github.com/stackabletech/docker-images/issues/1494[3.4.3 (LTS)], https://github.com/stackabletech/docker-images/issues/1494[3.5.0] +* Apache Kafka: https://github.com/stackabletech/docker-images/issues/1497[3.9.2 (LTS)], https://github.com/stackabletech/docker-images/issues/1497[4.2.1 (experimental)] * Apache NiFi: https://github.com/stackabletech/docker-images/issues/1498[2.9.0] -* OpenSearch: https://github.com/stackabletech/docker-images/issues/1500[3.6.0] +* Apache Phoenix: https://github.com/stackabletech/docker-images/issues/1491[5.3.1 (LTS)] +* Apache Spark: https://github.com/stackabletech/docker-images/issues/1502[4.1.2 (LTS)] * Apache Superset: https://github.com/stackabletech/docker-images/issues/1504[6.1.0] * Apache ZooKeeper: https://github.com/stackabletech/docker-images/issues/1506[3.9.5 (LTS)] +* Open Policy Agent: https://github.com/stackabletech/docker-images/issues/1499[1.16.2] +* OpenSearch: https://github.com/stackabletech/docker-images/issues/1500[3.6.0] +* Trino: https://github.com/stackabletech/docker-images/issues/1505[481] +* Vector: https://github.com/stackabletech/docker-images/issues/1486[0.55.0] ====== Deprecated versions @@ -291,10 +308,17 @@ The following product versions are deprecated and will be removed in a later rel * Apache Airflow: https://github.com/stackabletech/docker-images/issues/1489[3.1.6] * Apache Druid: https://github.com/stackabletech/docker-images/issues/1490[35.0.1], https://github.com/stackabletech/docker-images/issues/1490[30.0.1] +* Apache Hadoop HDFS: https://github.com/stackabletech/docker-images/issues/1494[3.4.2] +* Apache Hive: https://github.com/stackabletech/docker-images/issues/1495[4.0.1] +* Apache Kafka: https://github.com/stackabletech/docker-images/issues/1497[3.9.1] * Apache NiFi: https://github.com/stackabletech/docker-images/issues/1498[2.7.2] -* OpenSearch: https://github.com/stackabletech/docker-images/issues/1500[3.4.0] +* Apache Phoenix: https://github.com/stackabletech/docker-images/issues/1491[5.3.0] +* Apache Spark: https://github.com/stackabletech/docker-images/issues/1502[3.5.8], https://github.com/stackabletech/docker-images/issues/1502[4.1.1] * Apache Superset: https://github.com/stackabletech/docker-images/issues/1504[6.0.0] * Apache ZooKeeper: https://github.com/stackabletech/docker-images/issues/1506[3.9.4] +* Open Policy Agent: https://github.com/stackabletech/docker-images/issues/1499[1.12.3] +* OpenSearch: https://github.com/stackabletech/docker-images/issues/1500[3.4.0] +* Trino: https://github.com/stackabletech/docker-images/issues/1505[479] ====== Removed versions @@ -303,16 +327,40 @@ These images for released product versions remain available in the https://oci.s Information on how to browse the registry can be found xref:contributor:project-overview.adoc#docker-images[in the project overview documentation,window=_blank]. * Apache Druid: https://github.com/stackabletech/docker-images/issues/1490[34.0.0] +* Apache HBase: https://github.com/stackabletech/docker-images/issues/1491[2.6.3] +* Apache Hive: https://github.com/stackabletech/docker-images/issues/1495[4.1.0] * Apache NiFi: https://github.com/stackabletech/docker-images/issues/1498[1.28.1] -// TODO(verify): complete this list once the full product-version set is reconciled against the portal. +* Apache Phoenix: https://github.com/stackabletech/docker-images/issues/1491[5.2.1] +* Apache Spark: https://github.com/stackabletech/docker-images/issues/1502[3.5.7], https://github.com/stackabletech/docker-images/issues/1502[4.0.1] +* Open Policy Agent: https://github.com/stackabletech/docker-images/issues/1499[1.8.0] +* Vector: https://github.com/stackabletech/docker-images/issues/1486[0.52.0] ===== Kubernetes versions -// TODO(verify): supported Kubernetes versions for 26.7.0. In 26.3 these were 1.35-1.31. Confirm against the release definition. +This release supports the following Kubernetes versions: + +* `1.36` +* `1.35` +* `1.34` +* `1.33` +* `1.32` +* `1.31` + +// Portal diff shows no Kubernetes versions were dropped between 26.3 and 26.7 (1.36 added). Confirm before publishing. +No Kubernetes versions were dropped in this release. ===== OpenShift versions -// TODO(verify): supported OpenShift versions for 26.7.0. In 26.3 these were 4.21-4.18. Confirm against the release definition. +This release is available in the RedHat Certified Operator Catalog for the following OpenShift versions: + +* `4.22` +* `4.21` +* `4.20` +* `4.19` +* `4.18` + +// Portal diff shows no OpenShift versions were dropped between 26.3 and 26.7 (4.22 added). Confirm before publishing. +No OpenShift versions were dropped in this release. ==== Upgrade from 26.3