Skip to content

feat: ingest audit logs from edge control planes - #229

Draft
scotwells wants to merge 1 commit into
mainfrom
feat/edge-audit-ingest
Draft

feat: ingest audit logs from edge control planes#229
scotwells wants to merge 1 commit into
mainfrom
feat/edge-audit-ingest

Conversation

@scotwells

@scotwells scotwells commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Summary

When a customer acts against an edge control plane, that cluster's own API server audits it — not the core control plane. None of it reached the customer's project, so their audit trail stopped at the boundary of where their workloads actually run. Edge requests now show up in the owning project, attributed correctly and filterable by location alongside every other audit record.

Those records can't simply be replayed. Edge control planes are downstream clusters, so a raw edge audit record names internal ns-<uid> namespaces the customer has never seen and must never be shown — and it carries no usable tenant, so replayed unchanged it lands in the platform bucket, the one place every operator can read it. A new ingest subcommand receives audit batches from edge control planes, reverses that namespace mapping, attributes each record to the owning project, stamps its location, and republishes onto the audit stream the core pipeline already consumes — so storage, translation and query need no edge-specific handling. It's a new subcommand because nothing today receives audit over the wire: the processor consumes an already-populated stream, and the event exporter handles Events, not audit.

The shipper asserts nothing about itself. Cluster identity and location come from the authenticated client certificate and a registry rendered by infrastructure; tenancy and location arriving in a payload are overwritten, not merged. The reverse index is built from upstream project namespaces rather than by reading edge clusters, so ingest holds no credential into any edge cluster and a compromised site can't influence what a namespace resolves to. Namespaces are un-mapped everywhere they appear — requestURI included, which is the one that hides — and a final scan drops any record where one survived. Unresolvable records are dropped and counted rather than filed under a guess; a cold cache is retried rather than mistaken for an unknown tenant; and mappings are retained after a namespace is deleted so older records keep resolving for their full retention window.

Note

Unresolved, and wanted in review. #228 decides edge Events ship per-cluster over NATS nkeys and explicitly rejects a central collector; this is central and mTLS. Audit differs because its shipper is the edge kube-apiserver's own audit webhook backend, which speaks TLS and not NATS — there's no edge-side process to hand an nkey to without building one. But two edge paths with two credential models is a real cost and that RFC is still proposed, so I'd rather this were argued now than discovered later.

Out of scope: location as an Activity facet (needs ActivitySpec.Source, owned by #228), deployment manifests, and core control plane audit, which already reaches the stream directly and needs no rewriting.

Related to #228. Depends on milo-os/milo#769, which owns the canonical reverse lookup — this branch carries a mirrored copy with a TODO to delete it, since importing it today would pull k8s.io/kubernetes into this server's build graph.

Test plan

  • go build ./..., go vet ./... and the full go test ./... suite green; no existing package regressed
  • 34 unit tests covering registry identity, index retention and restart persistence, cold-cache park-and-retry, and each un-mapped field
  • mTLS listener: unregistered certificate rejected, anonymous handshake refused, readiness red until caches sync while liveness stays green, cold cache asks the shipper to retry
  • End to end against real ClickHouse, NATS and mTLS (task e2e:edge-audit) — recorded audit fixtures shipped in, then queried back through the storage layer
  • …covering: project record lands in its project with no internal namespace anywhere in the result, requestURI asserted explicitly; platform-scoped request not misfiled into a project; unresolvable record never stored under any scope; record about a deleted namespace still resolves; payload-claimed tenancy overwritten; one project can't see another's records; location filter and facet return the expected rows
  • Deployed to a cluster — blocked on the manifests noted above

Anything a customer does against an edge control plane is audited by
that cluster's API server, not by the core control plane, so none of it
reaches their project timeline. Add an "ingest" subcommand that accepts
audit batches from edge control planes and republishes them onto the
AUDIT_EVENTS stream the core pipeline already consumes, so Vector, the
processor, ClickHouse and the query API need no edge-specific handling.

Edge control planes are downstream clusters, so their audit records name
projected ns-<upstream-namespace-uid> namespaces that the customer who
owns the resource has never seen and must never be shown. The command
reverses that mapping before emitting anything, attributes each record to
the project that owns it, and stamps the location it came from.

Key changes:
- New "ingest" subcommand on the existing binary, alongside serve,
  processor and event-exporter; all flags scoped to the subcommand
- Cluster identity and location come from the authenticated client
  certificate and a registry rendered by infrastructure; scope
  annotations, user extras and locations arriving in a payload are
  overwritten rather than trusted
- Attribution writes both the scope annotations ClickHouse materializes
  and the user extras the activity processor reads, because writing one
  leaves a record correctly scoped in one system and misfiled in the
  other
- Namespaces are un-mapped in every field that can name one, requestURI
  included, and a re-scan of the encoded record drops anything where one
  survived rather than publishing an internal identifier
- The reverse index is built from upstream project namespaces, so the
  ingest path needs no credential into any edge cluster
- Readiness gates on namespace cache sync and cold-cache records are
  parked and retried, so a deploy does not misfile a window of records
- Index entries are never removed and are persisted across restarts, so
  records about a deleted namespace resolve for their full retention
  window
- Materialize the location annotation as a column with a set index, and
  expose it as a CEL filter field and an audit log facet

The reverse lookup is mirrored from milo-os/milo, which owns it; see the
TODO in internal/edgeingest/resolver.go.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant