feat: add tenant-safe Trino monitoring routes and per-org query metrics

[EDsCODE]

Summary

Customers who query through Trino get no monitoring in PostHog's Data ops scene. The product monitoring API reports only DuckDB workers and pgwire sessions, and Trino queries never pass through either. This PR adds a tenant-safe Trino counterpart for the PostHog backend to read.

The PostHog side that reads these routes is PostHog/posthog#110369.

• GET /api/v1/orgs/:id/monitoring/trino/snapshot returns the org's Trino lifecycle state, its tier's concurrency and queue limits, in-flight totals, and up to 200 in-flight queries, longest-running first.
• GET /api/v1/orgs/:id/monitoring/trino/series returns one of nine allow-listed metrics, over the same windows as the DuckDB series route.
• Both routes are internal-secret only, like the existing monitoring pair. An SSO identity gets 403.
• The snapshot omits cell ids, principals, resource groups, tier names, status messages, driver counts, error codes, and connection details.
• Query text has literals replaced with ? and comments removed. Text that cannot be lexed becomes a fixed placeholder.
• available: false marks a snapshot whose coordinator or org index could not be read. Zero totals are then not mistaken for an idle warehouse.
• The Trino usage collector now exports per-org Prometheus metrics: an in-flight gauge by state, and counters and histograms for finished queries. The series route reads them.
• The collector now holds a mutex across each poll. Two leader terms on the same pod could otherwise overlap on its maps.

Note

The new series are labelled by org: about 45 series per Trino-enabled org. Finished-query counts are best-effort and are not for billing. docs/metrics.md has the details.

Semantics to check

Field or rule	Meaning
in_flight	Every query that is not finished or failed, including planning and starting
running	In flight and past the queue. This is what the tier's concurrency limit counts, so in_flight = running + queued
blocked	Subset of running: Trino state RUNNING with every driver blocked
longest_running_ms	Longest elapsed time since submission among in-flight queries, queued ones included
Gauge ownership	One collector runs per cell and writes the gauge only for that cell's orgs
Gauge lifetime	Cleared when a poll fails and when the loop stops, so an outage shows as a gap
Series PromQL	Exact org selector on every template; the gauge takes the maximum across pods

Known limits

• The snapshot reads the enabled-org list once per request, uncached. That is fine at the current org count. A short cache is a follow-up.
• The pool observer returns partial results when some pool members answer. The gauge then under-counts during a partial pool outage. That behavior predates this PR.
• A same-pod leader re-election can leave the gauge empty for one poll interval.

Validation

• Masker: table test over strings, numbers, typed and prefixed literals, comments, quoted identifiers, non-ASCII, and four unterminated inputs that must return the placeholder.
• Snapshot: other orgs' queries excluded; the raw body is checked for principals, database names, cell ids, resource groups, status messages, and hashes; seven lifecycle and availability cases; the 200-row cap with totals over the full set.
• Metrics: each finished query counted once across polls; an unknown Trino error type maps to unknown; another cell's orgs get no series; gauges clear on poll failure, on stop, and when an org leaves the cell.
• Collector overlap: TestTrinoUsageCollectorToleratesOverlappingRuns fails under -race without the mutex.
• Series: every template carries $ORG; unknown metrics, DuckDB-only metrics, and unknown windows are rejected before Prometheus is called.
• Ran locally: go test -tags kubernetes ./controlplane/admin ./controlplane/provisioner, go test -race -tags kubernetes -run Trino ./controlplane, and just lint.
• Not run: the integration suites. Nothing was exercised against a live Trino cell or a real Prometheus.

Agent context

Human-driven, agent-assisted. Written with Claude Code: Claude Opus 5.5 planned and coordinated, Claude Sonnet subagents wrote the code task by task, and a separate reviewer model reviewed each task and the whole branch before this PR was opened. The collector mutex and the log lines on the two unavailable paths came out of that whole-branch review.

🤖 Generated with Claude Code

[github-actions]

Test Impact Plan

Deterministic summary of how this PR changes tests, CI runners, and coverage-risk signals.

Summary

Area	Added	Changed	Deleted
Test files	4	0	0
E2E/journey files	0	0	0
Workflow files	0	0	0

Signals

• Test cases: +16 / -0
• Assertions: +47 / -0
• Skips or known failures added: 0
• Workflow continue-on-error added: 0
• Workflow path filters added: 0
• Test commands removed from justfile: 0
• E2E/journey retry lines added: 0

Coverage risk: neutral or increased

No coverage-reduction warnings detected.