Investigate CI failures

[Jakob-Naucke]

GHA & openshift/release#79393 saw many random failures, most related to finalizer. Most recent run on the latter showed a resource apparently taking 4 minutes to patch. (Also) testing here for a faster feedback loop.

Summary by Sourcery

Integrate the trustee operator more tightly with an external KBS, switching from in-pod secret mounting to API-based secret and policy synchronization, updating trustee configuration and image versions, and enhancing logging and tests to diagnose and stabilize CI behavior.

New Features:

• Introduce JWT-based trustee admin authentication using Ed25519 keys stored in a Kubernetes Secret and wired into the KBS configuration.
• Add logic to push and delete machine disk-encryption secrets and attestation keys to the KBS via its API, including a controller to resynchronize them when the trustee deployment becomes available.
• Create a dedicated ConfigMap for reference values, plus sync routines to propagate them from Kubernetes to the KBS and to handle recomputation and resync on trustee restart.
• Add controllers and helpers to automatically synchronize resource and attestation policies with the KBS and to resync all machine LUKS keys when trustee comes up.

Bug Fixes:

• Avoid reliance on in-memory projected volumes for secret delivery by switching to KBS-backed storage, reducing chances of timing-related mount issues during rollout or deletion.],
• enhancements:[
• Enhance approved image reconciliation and machine/secret finalizers with additional logging and debug information around finalizer execution and cleanup decisions to aid CI failure investigation.
• Refine the reference value controller to watch related Jobs, log more details during reconcile, and simplify adoption logic by removing unused image adoption helpers.
• Update trustee integration tests to validate end-to-end synchronization of machine secrets and attestation keys with KBS across restarts, and add structured timestamped logging utilities for more traceable test output.
• Adjust trustee deployment labels, volumes, and config paths to align with the new KBS storage model and TLS behavior, including support for insecure HTTP mode when no certificate is present.

Build:

• Upgrade the default trustee image/tag to v0.20.0 and align Makefile, operator constants, and KBS configuration with the newer release.
• Switch the kube-rs dependency to a debug branch to gain better visibility into finalizer behavior during reconciliation.
• Enable the env_logger humantime feature to improve log readability in local and CI runs.

CI:

• Change the integration test GitHub Actions workflow to run a targeted attestation test with explicit image overrides and to always gather and upload must-gather diagnostics after the job.

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Jakob-Naucke

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[sourcery-ai]

Reviewer's Guide

Refactors the trustee operator to integrate directly with the KBS API (using JWT auth and kbs-client), replaces in‑pod volume-based secret and policy management with explicit sync controllers and config maps/secrets, updates Trustee/KBS configuration and versions, and extends tests and CI to validate LUKS key and attestation key resync behavior and aid debugging intermittent CI failures.

Sequence diagram for trustee deployment sync controller interactions with KBS

sequenceDiagram
    participant Controller as TrusteeSyncController
    participant Kube as KubernetesAPI
    participant KBS

    Controller->>Kube: watch Deployment label app=TRUSTEE_APP_LABEL
    Kube-->>Controller: Deployment with status Available=True
    Controller->>Controller: trustee_deployment_reconcile()

    alt Deployment Available
        Controller->>KBS: sync_resource_policy() / kbs_client::set_resource_policy
        Controller->>KBS: sync_attestation_policy() / kbs_client::set_attestation_policy

        Controller->>Kube: sync_reference_values_from_configmap()
        Controller->>KBS: sync_reference_values() / kbs_client::set_sample_rv

        Controller->>Kube: sync_all_machine_luks_key() / list Machine, get Secret
        Controller->>KBS: send_secret() / kbs_client::set_resource

        Controller->>Kube: update_attestation_keys() / list Secret
        Controller->>KBS: register_ak() / kbs_client::set_sample_rv
    else Deployment not Available
        Controller-->Controller: Action::await_change()
    end

Loading

File-Level Changes

Change	Details	Files
Refactor trustee reference values handling to use a dedicated ConfigMap and sync them directly to KBS.	Introduce TRUSTEE_RV_MAP ConfigMap and move reference-values.json storage there instead of trustee-data Make ReferenceValue serializable and deserializable and add helpers to recompute and sync reference values to KBS via kbs_client Add sync_reference_values_from_configmap used by trustee deployment reconcile loop Update image/job reconcile tests to expect TRUSTEE_RV_MAP and extra API calls for KBS sync	operator/src/trustee.rs operator/src/reference_values.rs operator/src/kbs-config.toml operator/src/main.rs operator/src/test_utils.rs tests/trusted_execution_cluster.rs
Introduce JWT-based admin auth for KBS and wire up kbs-client to manage secrets, resource policy, attestation policy, and attestation keys remotely instead of via volumes.	Add Ed25519 keypair generation and TRUSTEE_AUTH_SECRET Secret to hold public/private keys; JWTs issued with jsonwebtoken/jsonwebtoken-openssl Change KBS config to use AuthenticatedAuthorization and LocalJson storage backend under TRUSTEE_STORAGE_DIR Replace mount_secret/unmount_secret and projected volume AK wiring with send_secret/delete_secret/register_ak using kbs_client Add sync_resource_policy and sync_attestation_policy sending rego files over KBS admin API Update attestation_key_register flows to call update_attestation_keys(client) which pushes AKs to KBS rather than patching the trustee Deployment	operator/src/trustee.rs operator/src/kbs-config.toml operator/src/main.rs operator/src/register_server.rs operator/src/attestation_key_register.rs operator/Cargo.toml Cargo.toml operator/src/tpm.rego operator/src/test_utils.rs
Introduce a trustee sync controller to react to trustee Deployment availability and resynchronize state with KBS after restarts.	Add launch_trustee_sync_controller that watches Deployments labeled with TRUSTEE_APP_LABEL and runs trustee_deployment_reconcile In trustee_deployment_reconcile, on Available=True, sequentially sync resource policy, attestation policy, reference values from ConfigMap, all machine LUKS keys, and attestation keys to KBS; requeue on failures Label the trustee Deployment with TRUSTEE_APP_LABEL in generate_kbs_deployment and adjust selector accordingly	operator/src/trustee.rs operator/src/main.rs
Change machine keygen flow from mounting secrets into the trustee pod to pushing/removing secrets in KBS, with clearer finalizer logging and deletion behavior.	Keygen finalizer now calls generate_secret then send_secret on apply, and delete_secret on cleanup instead of mounting/unmounting volumes into the trustee Deployment Add logging to keygen_reconcile before and after finalizer() for debugging stuck finalizers and cleanups Ensure machine cleanup skips KBS deletion if TrustedExecutionCluster is terminating or missing	operator/src/register_server.rs operator/src/trustee.rs
Enhance attestation key handling to register AKs in KBS and ensure they are resynced after trustee restarts, and update the OPA policy to check trusted AKs.	update_attestation_keys now lists Secret objects owned by AttestationKey, extracts public_key values, and calls register_ak (kbs_client) instead of mutating the trustee Deployment tpm.rego now requires input.tpm.ak_public to be in trusted_aks reference value Add unit tests for update_attestation_keys success/empty/deleting/error paths and for Ed25519 key generation; extend helper functions to build dummy machines, AK secrets, and trustee auth secrets	operator/src/trustee.rs operator/src/attestation_key_register.rs operator/src/tpm.rego operator/src/test_utils.rs
Add integration tests for LUKS key sync and attestation key sync across trustee restarts, and make logging more structured and timestamped.	New tests in trusted_execution_cluster.rs exercise LUKS key KBS sync on machine creation, trustee restart resync, and deletion, and similar flows for attestation key registration and resync Extend TestContext with wait_for_deployment_ready helper and prepend timestamps to test_info/test_warn output Adjust existing integration tests to use TRUSTEE_RV_MAP instead of trustee-data for reference values and ensure teardown waits for RV ConfigMap deletion	tests/trusted_execution_cluster.rs test_utils/src/lib.rs
Update versions and CI pipeline to use newer Trustee image/config and to focus integration CI on the attestation test with better debug artifacts.	Bump TRUSTEE_VERSION and default TRUSTEE_IMAGE to v0.20.0 and align KBS configuration with new API (jwt admin, kvstorage, LocalJson storage) Switch kube dependency to a debug branch finalizer-dbg to gain additional finalizer diagnostics Modify integration-tests GitHub workflow to generate CRDs, run a focused attestation test with verbose logging, and always collect and upload must-gather artifacts	operator/src/main.rs Makefile operator/src/kbs-config.toml .github/workflows/integration-tests.yml Cargo.toml Cargo.lock

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[Jakob-Naucke]

Now we had a failure, but no log from guest-components. Built a new image with journal console forwarding so we can see.

[Jakob-Naucke]

I'm adding some logging to confirm this, but to me, this looks like 4.5 minutes for a GET (?!) (grepped for ::register_server)

[2026-07-16T19:01:19Z INFO  operator::register_server] Cleanup for machine Machine { metadata: ObjectMeta { annotations: None, creation_timestamp: Some(Time(2026-07-16T18:58:22Z)), deletion_grace_period_seconds: Some(0), deletion_timestamp: Some(Time(2026-07-16T19:01:19Z)), finalizers: Some(["finalizer.machine.trusted-execution-clusters.io"]), generate_name: None, generation: Some(2), labels: None, managed_fields: Some([ManagedFieldsEntry { api_version: Some("trusted-execution-clusters.io/v1alpha1"), fields_type: Some("FieldsV1"), fields_v1: Some(FieldsV1(Object {"f:metadata": Object {"f:finalizers": Object {".": Object {}, "v:\"finalizer.machine.trusted-execution-clusters.io\"": Object {}}, "f:ownerReferences": Object {".": Object {}, "k:{\"uid\":\"531bb1d4-d201-40d7-8816-4c05ec5dcb59\"}": Object {}}}, "f:spec": Object {".": Object {}, "f:id": Object {}}})), manager: Some("unknown"), operation: Some("Update"), subresource: None, time: Some(Time(2026-07-16T18:58:22Z)) }]), name: Some("machine-41d7c026-d8b2-4938-a2a0-a41c111ea38c"), namespace: Some("test-5ef66b38"), owner_references: Some([OwnerReference { api_version: "trusted-execution-clusters.io/v1alpha1", block_owner_deletion: Some(true), controller: Some(true), kind: "TrustedExecutionCluster", name: "trusted-execution-cluster", uid: "531bb1d4-d201-40d7-8816-4c05ec5dcb59" }]), resource_version: Some("9053"), self_link: None, uid: Some("c6f520d4-2fbf-462c-b424-2d1966d3e6a7") }, spec: MachineSpec { id: "41d7c026-d8b2-4938-a2a0-a41c111ea38c" }, status: None }
[2026-07-16T19:05:57Z INFO  operator::register_server] TrustedExecutionCluster trusted-execution-cluster not found, skipping unmount_secret for Machine machine-41d7c026-d8b2-4938-a2a0-a41c111ea38c

https://github.com/Jakob-Naucke/trusted-cluster-operator/blob/d4be8215969db30a187101c8307ffadd8c20328e/operator/src/register_server.rs#L152-L172