feat: ingest audit logs from edge control planes - #229
Draft
scotwells wants to merge 1 commit into
Draft
Conversation
Anything a customer does against an edge control plane is audited by that cluster's API server, not by the core control plane, so none of it reaches their project timeline. Add an "ingest" subcommand that accepts audit batches from edge control planes and republishes them onto the AUDIT_EVENTS stream the core pipeline already consumes, so Vector, the processor, ClickHouse and the query API need no edge-specific handling. Edge control planes are downstream clusters, so their audit records name projected ns-<upstream-namespace-uid> namespaces that the customer who owns the resource has never seen and must never be shown. The command reverses that mapping before emitting anything, attributes each record to the project that owns it, and stamps the location it came from. Key changes: - New "ingest" subcommand on the existing binary, alongside serve, processor and event-exporter; all flags scoped to the subcommand - Cluster identity and location come from the authenticated client certificate and a registry rendered by infrastructure; scope annotations, user extras and locations arriving in a payload are overwritten rather than trusted - Attribution writes both the scope annotations ClickHouse materializes and the user extras the activity processor reads, because writing one leaves a record correctly scoped in one system and misfiled in the other - Namespaces are un-mapped in every field that can name one, requestURI included, and a re-scan of the encoded record drops anything where one survived rather than publishing an internal identifier - The reverse index is built from upstream project namespaces, so the ingest path needs no credential into any edge cluster - Readiness gates on namespace cache sync and cold-cache records are parked and retried, so a deploy does not misfile a window of records - Index entries are never removed and are persisted across restarts, so records about a deleted namespace resolve for their full retention window - Materialize the location annotation as a column with a set index, and expose it as a CEL filter field and an audit log facet The reverse lookup is mirrored from milo-os/milo, which owns it; see the TODO in internal/edgeingest/resolver.go.
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When a customer acts against an edge control plane, that cluster's own API server audits it — not the core control plane. None of it reached the customer's project, so their audit trail stopped at the boundary of where their workloads actually run. Edge requests now show up in the owning project, attributed correctly and filterable by location alongside every other audit record.
Those records can't simply be replayed. Edge control planes are downstream clusters, so a raw edge audit record names internal
ns-<uid>namespaces the customer has never seen and must never be shown — and it carries no usable tenant, so replayed unchanged it lands in the platform bucket, the one place every operator can read it. A newingestsubcommand receives audit batches from edge control planes, reverses that namespace mapping, attributes each record to the owning project, stamps its location, and republishes onto the audit stream the core pipeline already consumes — so storage, translation and query need no edge-specific handling. It's a new subcommand because nothing today receives audit over the wire: the processor consumes an already-populated stream, and the event exporter handles Events, not audit.The shipper asserts nothing about itself. Cluster identity and location come from the authenticated client certificate and a registry rendered by infrastructure; tenancy and location arriving in a payload are overwritten, not merged. The reverse index is built from upstream project namespaces rather than by reading edge clusters, so ingest holds no credential into any edge cluster and a compromised site can't influence what a namespace resolves to. Namespaces are un-mapped everywhere they appear —
requestURIincluded, which is the one that hides — and a final scan drops any record where one survived. Unresolvable records are dropped and counted rather than filed under a guess; a cold cache is retried rather than mistaken for an unknown tenant; and mappings are retained after a namespace is deleted so older records keep resolving for their full retention window.Note
Unresolved, and wanted in review. #228 decides edge Events ship per-cluster over NATS nkeys and explicitly rejects a central collector; this is central and mTLS. Audit differs because its shipper is the edge kube-apiserver's own audit webhook backend, which speaks TLS and not NATS — there's no edge-side process to hand an nkey to without building one. But two edge paths with two credential models is a real cost and that RFC is still
proposed, so I'd rather this were argued now than discovered later.Out of scope: location as an Activity facet (needs
ActivitySpec.Source, owned by #228), deployment manifests, and core control plane audit, which already reaches the stream directly and needs no rewriting.Related to #228. Depends on milo-os/milo#769, which owns the canonical reverse lookup — this branch carries a mirrored copy with a TODO to delete it, since importing it today would pull
k8s.io/kubernetesinto this server's build graph.Test plan
go build ./...,go vet ./...and the fullgo test ./...suite green; no existing package regressedtask e2e:edge-audit) — recorded audit fixtures shipped in, then queried back through the storage layerrequestURIasserted explicitly; platform-scoped request not misfiled into a project; unresolvable record never stored under any scope; record about a deleted namespace still resolves; payload-claimed tenancy overwritten; one project can't see another's records; location filter and facet return the expected rows