Skip to content

Detect TLS cert secret changes in NodeSet reconciler#1990

Merged
openshift-merge-bot[bot] merged 1 commit into
openstack-k8s-operators:mainfrom
oliashish:fix/cert_change_detection_nodeset
Jul 23, 2026
Merged

Detect TLS cert secret changes in NodeSet reconciler#1990
openshift-merge-bot[bot] merged 1 commit into
openstack-k8s-operators:mainfrom
oliashish:fix/cert_change_detection_nodeset

Conversation

@oliashish

@oliashish oliashish commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Cert-manager renewals were invisible because cert secrets are owned by the Certificate CR, not the NodeSet. Add label-based detection in secretWatcherFn and hash comparison in checkDeployment to surface redeployment required signal after cert rotation.

Jira: OSPRH-32671

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com

@openshift-ci
openshift-ci Bot requested review from rabi and rebtoor July 20, 2026 09:38
@github-actions

Copy link
Copy Markdown

OpenStackControlPlane CRD Size Report

Metric Value
CRD JSON size 350002 bytes (342KB)
Base branch size 364822 bytes
Change -4.06%
Status yellow — growing
Threshold reference
Color Range Meaning
🟢 green < 300KB Comfortable
🟡 yellow 300–400KB Growing
🟠 orange 400–750KB Concerning
🔴 red > 750KB Approaching 1.5MB etcd limit (cut in half to allow space for update)

@oliashish
oliashish requested review from slagle and removed request for rebtoor July 20, 2026 09:39
@oliashish

Copy link
Copy Markdown
Contributor Author

/test openstack-operator-build-deploy-kuttl-4-20

}

for name, currentHash := range currentCertHashes {
if deployedHash, exists := deployedSecretHashes[name]; exists && deployedHash != currentHash {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a review with claude has pointed out:

The new checkCertSecretsChanged uses exists && deployedHash != currentHash — a new cert secret that wasn't present at deployment time is not treated as a change. This
means if a new TLS service is added to the NodeSet after the last deployment, cert secrets for it won't trigger redeployment.

This may be intentional (new certs are picked up on the next deployment anyway), but the inconsistency is worth a comment or explicit design choice.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding a new TLS service to the NodeSet means changing instance.Spec.Services. That changes instance.Spec, which changes ConfigHash which hashes the entire instance.Spec. Since ConfigHash != DeployedConfigHash, the deployment is already skipped before we reach here at

if deployment.Status.NodeSetHashes[instance.Name] != instance.Status.ConfigHash {
      continue
 }

So the existing spec-hash check already covers this case.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is also the case of changing a service (that is already in the NodeSet spec) to add a TLS cert, but that seems pretty unlikely.

// listed in AnsibleVarsFrom, so the field-index lookup below would miss them.
labels := obj.GetLabels()
if nodeSetName, ok := labels[deployment.NodeSetLabel]; ok {
if _, hasSvcLabel := labels[deployment.ServiceLabel]; hasSvcLabel {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

claude is recommending to also check the existence of the ServiceKeyLabel here in order to be more precise.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ack.
Added ServiceKeyLabel check for completeness. Technically redundant since EnsureTLSCerts always sets all three labels together, but makes the match more precise.

Cert-manager renewals were invisible because cert secrets are owned by the Certificate CR, not the NodeSet. Add label-based detection in secretWatcherFn and hash comparison in checkDeployment to surface redeployment required signal after cert rotation.

Jira: OSPRH-32671

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@oliashish
oliashish force-pushed the fix/cert_change_detection_nodeset branch from e604e71 to 16c0fac Compare July 23, 2026 09:43
@oliashish

Copy link
Copy Markdown
Contributor Author

/test openstack-operator-build-deploy-kuttl-4-20

@centosinfra-prod-github-app

Copy link
Copy Markdown

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://gateway-cloud-softwarefactory.apps.ocp.cloud.ci.centos.org/zuul/t/rdoproject.org/buildset/7ba1b9012e4d46f986f8f540f768f2c4

✔️ openstack-k8s-operators-content-provider SUCCESS in 3h 05m 11s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 38m 14s
cifmw-crc-podified-edpm-baremetal NODE_FAILURE Node(set) request 099-0000150978 failed in 0s
✔️ openstack-operator-tempest-multinode SUCCESS in 1h 49m 55s
✔️ openstack-operator-edpm-baremetal-minor-update SUCCESS in 2h 21m 38s

@slagle

slagle commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

recheck

@openshift-ci

openshift-ci Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: oliashish, slagle

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-merge-bot
openshift-merge-bot Bot merged commit 788b8c9 into openstack-k8s-operators:main Jul 23, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants