Validate native admission end to end

README.md

@@ -33,12 +33,15 @@ The source and GitHub release downloads are public; no repository credentials
 are needed to clone the project or install the CLI.
 
 Live [Ratify/Gatekeeper enforcement (#95)](https://github.com/microsoft/brewlet/issues/95)
-validation remains pending. Real CPU HPA scale-up/down has passed twice on fresh
-disposable clusters with a **fixed-shim candidate** over the 0.5.0 components.
+passed its required matrix twice on fresh local disposable clusters with the
+released verifier/publisher, a corrected Verifier manifest and the fixed shim.
+Real CPU HPA scale-up/down has also passed twice on fresh disposable clusters
+with a **fixed-shim candidate** over the 0.5.0 components.
 The unmodified release exposed a packed-layer GC failure during scale-out;
 [metrics-driven scaling (#94)](https://github.com/microsoft/brewlet/issues/94)
 and the [live runbook](docs/live-validation.md) distinguish that baseline from
-the candidate and its remaining cold-start limits.
+the candidate, fixture-only admission settings and remaining cold-start limits.
+Neither result is an unmodified 0.5.0 pass or production certification.
 See [preview status and validation](https://brewlet.sh/docs/#preview-status-and-validation)
 for the distinction between release smoke, component, and live cluster coverage.
 

admission/README.md

@@ -5,13 +5,14 @@ signatures and identities** before a workload runs. It admits a pod on the
 Brewlet runtime only when the Pod image resolves to a digest with a valid,
 trusted final-image managed-dependency attestation.
 
-> **Preview: live admission validation pending.** Evaluate only in a disposable
-> cluster. The component tests below use real verification and policy logic but
-> substitute registry access and plugin transport. Live registry discovery,
-> external plugin execution, Ratify/Gatekeeper wiring, and Kubernetes admission
-> enforcement remain unproven by this suite; see
-> [#95](https://github.com/microsoft/brewlet/issues/95). This is not a
-> production-readiness claim.
+> **Preview: live candidate validation, not production certification.** Two
+> consecutive fresh local arm64 clusters passed the real registry, external
+> verifier, Ratify/Gatekeeper and Kubernetes admission matrix, including serving
+> JavaApplication-generated Pods. These runs use the released 0.5.0 verifier and
+> publisher, a fixed-shim candidate and the corrected Verifier manifest, not
+> unmodified 0.5.0. See [#95](https://github.com/microsoft/brewlet/issues/95) and
+> the [live runbook](../docs/live-validation.md) for evidence and fixture-only
+> cache, registry and TLS configuration. Evaluate only in a disposable cluster.
 
 It provides the cluster-side enforcement that the managed-dependency-bundles
 design (specification §4.5) leaves to admission policy: requiring a valid,
@@ -99,7 +100,10 @@ Deliver the binary to Ratify one of two ways:
    startup.
 2. **Baked image**: place the binary at
    `/home/nonroot/.ratify/plugins/brewlet-managed-dependencies` in a custom
-   Ratify image and drop `spec.source`.
+   Ratify image, set `RATIFY_CONFIG=/home/nonroot/.ratify` so Ratify discovers
+   that plugin directory, and drop `spec.source`. Pin the resulting image by
+   digest from its first deployment. This is the delivery route exercised by
+   the live scenario.
 
 The plugin binary links Ratify's oras store, and therefore its full dependency
 tree (oras-go plus cloud registry auth SDKs). Build it with the same toolchain
@@ -111,6 +115,17 @@ Prerequisites: Ratify v1.4.x installed as a Gatekeeper external-data provider
 (`ratify-provider`), Gatekeeper installed, and a registry that exposes the
 OCI 1.1 Referrers API.
 
+The pinned Ratify v1.4.5 chart CRD selects the plugin through `spec.name`;
+it rejects `Verifier.spec.type`, even though the SDK exposes a Type field.
+The shipped manifest omits that unsupported field.
+
+Enable Gatekeeper external data and configure its validation webhook with
+`failurePolicy: Fail`. Its rules must cover Pod CREATE/UPDATE and the
+`pods/ephemeralcontainers` UPDATE subresource. Provider timeouts must be shorter
+than the webhook timeout; the live fixture uses 20 and 30 seconds respectively.
+`enforcementAction: deny` does not itself make webhook transport errors fail
+closed when the webhook has `failurePolicy: Ignore`.
+
 ```bash
 # 1. Edit deploy/20-ratify-verifier.yaml: set trustedPublicKey and
 #    expectedBuilderIdentity, and pin the plugin source by digest.
@@ -317,10 +332,19 @@ is admitted.
   claims, no cross-candidate trust merging, and unrelated/overlapping verifier
   success in either selection order. These are component tests, not
   subprocess, registry, or Kubernetes deployment tests.
+- The independently runnable live scenario uses pinned Zot native referrers,
+  Ratify v1.4.5, Gatekeeper v3.18.3 and the real external verifier. Its 47 required
+  assertions cover trusted serving, all invalid-evidence classes, signer
+  rotation with fresh candidate reports, competing verifiers, regular/init/
+  ephemeral admission requests, exclusions, registry failures and provider
+  outage. It never treats missing prerequisites as a skip.
 
 Run:
 
 ```bash
 ( cd core && go test ./pkg/attest/... )
 ( cd admission/ratify-verifier && go test ./... )
 ```
+
+For the live invocation, release/candidate distinction, diagnostics and remaining
+limits, use the [disposable live-validation runbook](../docs/live-validation.md).

admission/deploy/20-ratify-verifier.yaml

@@ -36,7 +36,6 @@ metadata:
   name: verifier-brewlet-managed-dependencies
 spec:
   name: brewlet-managed-dependencies
-  type: brewlet-managed-dependencies
   version: 1.0.0
   artifactTypes: application/vnd.brewlet.attestation.v1+json
   # Option 1: download the plugin binary from a registry at startup. Pin by

admission/ratify-verifier/deploy_test.go

@@ -0,0 +1,32 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+package main
+
+import (
+	"os"
+	"testing"
+
+	"sigs.k8s.io/yaml"
+)
+
+func TestDeployedVerifierUsesV145ChartFields(t *testing.T) {
+	raw, err := os.ReadFile("../deploy/20-ratify-verifier.yaml")
+	if err != nil {
+		t.Fatal(err)
+	}
+	var manifest struct {
+		Spec map[string]any `json:"spec"`
+	}
+	if err := yaml.Unmarshal(raw, &manifest); err != nil {
+		t.Fatal(err)
+	}
+	// The pinned v1.4.5 chart CRD omits type, even though its Go SDK has it.
+	// The live API server rejects the entire Verifier when this field is set.
+	if _, exists := manifest.Spec["type"]; exists {
+		t.Fatal("Ratify v1.4.5 chart CRD rejects spec.type; select the plugin with spec.name")
+	}
+	if manifest.Spec["name"] != pluginName {
+		t.Fatalf("spec.name = %v, want %q", manifest.Spec["name"], pluginName)
+	}
+}

docs/README.md

@@ -36,7 +36,7 @@ Implemented functionality and live end-to-end validation are different:
 |---|---|---|
 | Public release access and local use | The Pages release smoke exercises the released CLI, local Java example, Maven plugin, anonymous chart download, component manifest access, and release provenance. | It renders but does not install the chart or provision nodes. |
 | Kubernetes runtime | The source-built E2E tiers exercise provisioning, serving, manual scaling, and runtime telemetry. | These scenarios do not establish the admission or autoscaling loops below, or production readiness. |
-| Managed-dependency admission | Component tests exercise real DSSE verification and Ratify policy logic with substituted registry access and plugin transport. | Live registry discovery, external plugin execution, Ratify/Gatekeeper wiring, and Kubernetes admission outcomes remain pending in [#95](https://github.com/microsoft/brewlet/issues/95). |
+| Managed-dependency admission | Component tests use substituted registry access and plugin transport; two fresh local arm64 clusters additionally passed 47 real registry, external verifier, Ratify/Gatekeeper and API assertions, including serving JavaApplication Pods. | Live candidate validation uses the released verifier/publisher, corrected Verifier manifest, fixed shim and fixture-only uncached settings. It is not an unmodified 0.5.0 pass or ordinary ephemeral-debug execution support; see [#95](https://github.com/microsoft/brewlet/issues/95) and the [runbook](live-validation.md). |
 | CPU autoscaling | Alongside HPA creation and simulated HPA ownership tests, two fresh local arm64 clusters passed real metrics-server-driven 1-to-3-to-1 scaling, Ready Pods, serving endpoints and three ownership reconciliations with a fixed-shim candidate. | Unmodified 0.5.0 exposed a packed-layer GC scale-out failure. The candidate fixes warm reuse, not cold startup after missing-source GC; see [#94](https://github.com/microsoft/brewlet/issues/94) and the [runbook](live-validation.md). |
 
 [The validation tracker (#93)](https://github.com/microsoft/brewlet/issues/93)
@@ -46,8 +46,9 @@ The existing E2E harness permits skips; a successful run alone is not evidence
 that every assertion executed. Broader strict-mode work is tracked in
 [#13](https://github.com/microsoft/brewlet/issues/13).
 
-Admission remains a coverage gap. CPU validation demonstrated both a release
-defect and the scoped candidate correction; it is not an unmodified 0.5.0 pass.
+Admission exposed an unsupported Verifier manifest field; CPU validation exposed
+a packed-layer GC defect. Both passes use explicitly identified candidate
+corrections rather than an unmodified 0.5.0 installation.
 Do not treat component tests or release smoke results as proof of either live loop,
 or these disposable-cluster runs as production certification.
 

docs/admission-enforcement.md

@@ -7,16 +7,17 @@ valid, trusted final-image managed-dependency attestation. It combines a
 verifies Brewlet's native OCI 1.1 referrer in place so the runtime executes that
 same admitted image.
 
-!!! warning "Preview: live admission validation pending"
+!!! warning "Preview: live candidate validation, not production certification"
 
     Brewlet is a pre-1.0 preview; evaluate this integration only in a disposable
-    cluster. Component tests cover real DSSE verification and Ratify policy
-    decisions, but substitute registry access and plugin transport. They do not
-    prove live referrer discovery, external plugin delivery/execution,
-    Ratify/Gatekeeper wiring, or Kubernetes admission enforcement.
-    [Issue #95](https://github.com/microsoft/brewlet/issues/95) tracks that
-    end-to-end evidence. The policy contract below is implemented, not a
-    production-readiness certification.
+    cluster. Two consecutive fresh local arm64 clusters passed live referrer
+    discovery, external plugin execution, Ratify/Gatekeeper enforcement and
+    serving JavaApplication-generated Pods. The candidate uses the released
+    0.5.0 verifier/publisher, the fixed shim and a corrected Verifier manifest;
+    this is not an unmodified 0.5.0 pass.
+    [Issue #95](https://github.com/microsoft/brewlet/issues/95) and the
+    [runbook](live-validation.md) retain the exact evidence and fixture-only
+    cache, registry and TLS settings. This is not production certification.
 
 The plugin verifies Brewlet's native evidence directly, reusing Brewlet's own
 DSSE and predicate verification code instead of requiring evidence to be
@@ -88,7 +89,9 @@ satisfies the complete contract; claims are never combined across candidates.
 
 - Ratify v1.4.x installed as the `ratify-provider` Gatekeeper external-data
   provider.
-- Gatekeeper installed.
+- Gatekeeper installed with external data enabled, validation webhook
+  `failurePolicy: Fail`, and rules covering Pod CREATE/UPDATE and
+  `pods/ephemeralcontainers` UPDATE.
 - A registry that implements the **OCI 1.1 Referrers API**.
 - A digest-pinned Brewlet application image with a signed final-image
   managed-dependency attestation.
@@ -105,6 +108,11 @@ For private registries, configure the oras store's `authProvider`, such as a
 `k8Secrets` provider backed by a Docker-config Secret. Evidence that cannot be
 authenticated or fetched cannot grant admission.
 
+Keep provider timeouts shorter than webhook timeouts; the live fixture uses
+20 seconds for the provider and 30 seconds for the validating webhook.
+The constraint's `enforcementAction: deny` cannot compensate for webhook
+`failurePolicy: Ignore` during transport failures.
+
 ---
 
 ## Build and deliver the verifier plugin
@@ -143,8 +151,10 @@ Choose one delivery model:
     /home/nonroot/.ratify/plugins/brewlet-managed-dependencies
     ```
 
-    Use a digest-pinned custom Ratify image and remove `spec.source` from the
-    Verifier resource.
+    Set `RATIFY_CONFIG=/home/nonroot/.ratify` to select that plugin directory.
+    Use a digest-pinned custom Ratify image from the first deployment and remove
+    `spec.source` from the Verifier resource. The live scenario exercises this
+    delivery route.
 
 The plugin links Ratify's oras store and its registry-auth dependencies. Build
 it with a toolchain compatible with the Ratify installation.
@@ -184,6 +194,10 @@ The resources configure:
 | [`40-gatekeeper-constrainttemplate.yaml`](https://github.com/microsoft/brewlet/blob/main/admission/deploy/40-gatekeeper-constrainttemplate.yaml) | Gatekeeper ConstraintTemplate | Sends regular, init, and ephemeral container images from Brewlet-runtime pods to Ratify. |
 | [`50-gatekeeper-constraint.yaml`](https://github.com/microsoft/brewlet/blob/main/admission/deploy/50-gatekeeper-constraint.yaml) | Gatekeeper Constraint | Applies the check to Pod CREATE and UPDATE requests, with explicit namespace exclusions. |
 
+Ratify v1.4.5's shipped chart CRD rejects `Verifier.spec.type`; use `spec.name`
+to select the plugin, as the corrected resource does. This defect was exposed
+by live API deployment and is covered by a manifest regression test.
+
 Observe warnings and test known-good and known-bad images before changing the
 constraint to `enforcementAction: deny`.
 
@@ -234,6 +248,20 @@ different key or unsigned image to confirm the expected pass and fail paths.
 
 ## Production trust model and limitations
 
+### Live fixture boundaries
+
+The [live scenario](live-validation.md) disables Gatekeeper response, Ratify
+provider and ORAS discovery caches, and directs the ORAS content cache to an
+immutable empty layout. This makes every candidate fetch real and avoids a
+concurrent content-cache index failure observed with the pinned Ratify version.
+It is a fixture configuration, not validation of production cache concurrency
+or revocation latency. Do not infer a key-rotation or outage guarantee for
+previously cached images from tests of fresh verification requests.
+
+Regular and init image requests and valid ephemeral-container subresource
+updates are exercised at the API boundary. This does **not** establish support
+for running ordinary-image ephemeral debug containers through Brewlet.
+
 ### Key distribution and rotation
 
 The trust anchor is a bare ECDSA P-256 public key. Brewlet does not use

docs/concepts.md

@@ -84,7 +84,7 @@ directories. Each implementation maps to a section of the
 | **`brewlet-node-provisioner`** | Privileged DaemonSet. On opted-in nodes it installs the shim, materializes JDK roots + launcher layers, registers the containerd runtime, and labels the node ready. | Source: [`provisioner/`](https://github.com/microsoft/brewlet/tree/main/provisioner); deployment: [`kubernetes/deploy/node-provisioner.yaml`](https://github.com/microsoft/brewlet/blob/main/kubernetes/deploy/node-provisioner.yaml); spec §5 |
 | **`brewlet-operator`** | Node lifecycle controller. Watches opted-in nodes, manages the provisioner DaemonSet + the `brewlet` RuntimeClass, and tracks node readiness. | [`kubernetes/cmd/manager/`](https://github.com/microsoft/brewlet/tree/main/kubernetes/cmd/manager), spec §8.1 |
 | **`brewlet-admission`** | Mutating+validating webhook. Overwrites compatibility hints from the selected Pod image onto brewlet pods and matches/steers requested JDK/launcher onto compatible nodes. | [`kubernetes/cmd/admission/`](https://github.com/microsoft/brewlet/tree/main/kubernetes/cmd/admission), spec §8.3 |
-| **Ratify/Gatekeeper enforcement** | Optional policy requiring a valid, trusted final-image managed-dependency attestation for every image on a Brewlet-runtime pod; live enforcement validation is pending in [#95](https://github.com/microsoft/brewlet/issues/95). | [Admission enforcement](admission-enforcement.md), [`admission/`](https://github.com/microsoft/brewlet/tree/main/admission) |
+| **Ratify/Gatekeeper enforcement** | Optional policy requiring a valid, trusted final-image managed-dependency attestation for every image on a Brewlet-runtime pod; [#95](https://github.com/microsoft/brewlet/issues/95) records live candidate validation with a corrected manifest and fixture-only settings, not production certification. | [Admission enforcement](admission-enforcement.md), [`admission/`](https://github.com/microsoft/brewlet/tree/main/admission) |
 | **`RuntimeClass/brewlet`** | Routes pods to the shim handler; its `nodeSelector` keeps workloads on ready nodes. | [`deploy/runtimeclass.yaml`](https://github.com/microsoft/brewlet/blob/main/kubernetes/deploy/runtimeclass.yaml), spec §7 |
 | **`JavaApplication` CRD** | The higher-level developer-facing deployment descriptor, reconciled by the operator's `JavaApplication` controller (§8.2). | [`deploy/javaapplication-crd.yaml`](https://github.com/microsoft/brewlet/blob/main/kubernetes/deploy/javaapplication-crd.yaml), spec §9 |
 | **Helm chart** | SpinKube-style single-command activation of the operator + provisioner RBAC + webhook. | [`charts/brewlet/`](https://github.com/microsoft/brewlet/tree/main/kubernetes/charts/brewlet/) |
@@ -141,8 +141,8 @@ directories. Each implementation maps to a section of the
    `nodeAffinity`) onto a node with a compatible JDK/launcher. The optional
    [Ratify/Gatekeeper admission integration](admission-enforcement.md) provides
    a policy requiring a trusted final-image managed-dependency attestation.
-   Its live enforcement path remains pending validation; use disposable
-   evaluation clusters only.
+   Its live candidate validation uses a corrected manifest and fixture-only
+   settings; use disposable evaluation clusters only.
 5. The **containerd shim** requires the CRI-recorded requested image to be
    digest-pinned, resolves that exact target from containerd's content store,
    verifies its selected platform manifest against CRI's image-config digest,

docs/deploying-workloads.md

@@ -16,8 +16,10 @@ The optional [managed-dependency admission integration](admission-enforcement.md
 provides a policy for gating either form in a disposable evaluation cluster. The
 Gatekeeper policy applies to every pod with `runtimeClassName: brewlet`,
 including pods generated by `JavaApplication`, and requires each image to be
-digest-pinned with a trusted final-image attestation. Live enforcement validation
-is pending in [#95](https://github.com/microsoft/brewlet/issues/95).
+digest-pinned with a trusted final-image attestation. Live candidate validation
+passed twice on fresh local clusters; [#95](https://github.com/microsoft/brewlet/issues/95)
+and the [runbook](live-validation.md) distinguish its corrected manifest and
+fixture-only settings from unmodified 0.5.0.
 
 ---