observability: propagate trace context to Instance and instrument downstream stages

[rickisba]

Motivation

• Implement Issue v0.1.10-4C: propagate internal trace context from Proxy to Instance and instrument Instance-observed vLLM transport stages #185 to extend the internal CacheRoute trace propagation boundary to include Proxy -> Instance while preserving the exact canonical reserved header vocabulary and process-local, request-scoped collection semantics under the observability umbrella (v0.1.10-4: Add Unified v1 LMCache Gateway, Tier, Adapter, and Queue Observability #141).
• Keep the change narrow and compatibility-preserving: forward only the exact reserved headers as extra HTTP headers during Proxy->Instance forwarding and add request-local Instance collection/fallback without changing payloads, routing, retries, or public surface shapes (Refs: v0.1.10 Epic: KDN Distribution Infrastructure Contracts and Observability Baseline #137, v0.1.10-4: Add Unified v1 LMCache Gateway, Tier, Adapter, and Queue Observability #141, v0.1.10-5: Add v1/Legacy Dual-Profile and LMCache Gateway Regression Validation #142, Epic: migrate CacheRoute to the RFC-defined canonical package architecture #157, RFC: redesign CacheRoute package architecture around contracts, domains, integrations, services, and entrypoints #159, v0.1.10-4A: establish canonical observability v1 contracts and a process-local collector #178, observability: establish canonical v1 contracts and process-local collector #179, v0.1.10-4B: propagate internal trace context from Scheduler to Proxy and instrument Proxy request stages #182, observability: propagate Scheduler trace context and instrument Proxy stages #183, Documentation governance: establish the CacheRoute developer and maintenance handbook #180, docs: establish the CacheRoute developer and maintenance handbook #184, v0.1.10-2: Define v1 Runtime, Artifact, LMCache Observation, Operation Task, and Queue State Models #139, v0.1.10-3: Define Versioned KDN Cache Service Facade and LMCache Gateway Contracts #140).

Description

• Proxy forwarding: proxy/queue/manager.py now encodes the task TraceContext with encode_trace_headers() and passes a fresh mapping of the exact reserved headers through the existing forward_request(..., extra_headers=...) seam without mutating task.instance_body or forwarding arbitrary incoming headers.
• Instance startup & handlers: instance/instance_api.py resolves CACHEROUTE_RUNTIME_PROFILE and CACHEROUTE_TRACE_SAMPLE_RATE once at FastAPI lifespan startup (immutable app.state), and wraps downstream calls with Instance-local collection using a new helper.
• Instance-local helpers: added instance/observability.py which implements resolve_instance_context(), start_instance_trace_session(), collect_non_streaming() and collect_streaming() to validate/accept propagated contexts, perform safe local fallback context creation, create a request-local TraceCollector only when sampled, and record Instance-provenance stages.
• Stage semantics: Instance records only Instance-observed stages using TraceComponent.INSTANCE with the existing TraceStageName enum values: COMPLETION (starts before downstream call, finishes on success/failure/cancel), FIRST_TOKEN (starts at downstream invocation, finishes at first non-empty chunk), and DECODE (starts after first non-empty chunk, finishes at stream end); non-streaming requests skip FIRST_TOKEN/DECODE with stable reasons; parent relationships and deterministic ordering are preserved.
• Documentation & governance: updated doc/architecture/observability-v1.md, doc/research/issue-141-unified-observability.md, and handbook chapters doc/developer-handbook/runtime-flows.md and doc/developer-handbook/configuration-and-interfaces.md to record the exact trust/overwrite boundary and Instance startup resolution rules.
• Tests: added CPU-only deterministic tests under test/observability/test_instance_observability.py and updated test/observability/test_scheduler_proxy_production_paths.py and repository-governance tests to accept the canonical encode/forward pattern.

Changed files (complete list):

• instance/observability.py (new)
• instance/instance_api.py (modified)
• proxy/queue/manager.py (modified)
• test/observability/test_instance_observability.py (new)
• test/observability/test_scheduler_proxy_production_paths.py (modified)
• test/test_repository_governance.py (modified)
• doc/architecture/observability-v1.md (modified)
• doc/research/issue-141-unified-observability.md (modified)
• doc/developer-handbook/runtime-flows.md (modified)
• doc/developer-handbook/configuration-and-interfaces.md (modified)

Exact Proxy->Instance reserved header vocabulary forwarded (unchanged):

• scheduler-request-id
• x-cacheroute-trace-version
• x-cacheroute-trace-id
• x-cacheroute-runtime-profile
• x-cacheroute-trace-sampled
• x-cacheroute-trace-created-at

Confirmations and invariants:

• Proxy and Instance produce separate, process-local immutable RequestTrace snapshots that may share one TraceContext but remain independent; traces are not merged.
• No CacheRoute reserved headers are sent to vLLM (Instance->vLLM forwarding is unchanged and carries no x-cacheroute-* headers or W3C Trace Context).
• Public request/response/SSE/metadata shapes are unchanged and the JSON/SSE payloads remain byte-for-byte compatible with prior behavior (mock and real-vLLM paths preserved).
• No canonical trace data is returned to clients, added to SSE/metadata, or persisted/exported by this change.
• No VLLM_PREFILL stage is emitted by Instance instrumentation; no vLLM or LMCache provenance fields are created.
• Routing, queueing, reservation, prediction, injection, fallback, retries, timeouts, and status codes remain unchanged.
• No new dependencies or package-discovery changes were added and no new top-level runtime package was created.

Handbook impact

• Updated handbook runtime/configuration chapters to document that propagation now reaches Instance, the exact trust/overwrite boundary, Instance startup validation/fallback rules, and request-local collector ownership; this is the required same-PR handbook update.

Testing

• Final head SHA: f4e19b40416372599a57af01ea14ec5e29af4867 and base/main SHA used in this checkout: 28e857c52a7a45a9c6c971be96ec32a4059dad7f.
• Automated validation commands executed and summarized below:
  • python3 -m compileall -q src test scheduler proxy instance — PASSED.
  • python3 -m pytest -q test/test_repository_governance.py — PASSED (14 tests).
  • python3 -m pytest -q test/test_documentation_governance.py — PASSED (12 tests).
  • Focused observability unit tests: python3 -m pytest -q test/observability attempted but the environment lacked runtime test dependencies (notably pydantic and numpy), causing collection errors; these tests are ENVIRONMENT-BLOCKED in this execution.
  • Several broader/packaging commands from the required validation list were attempted but are ENVIRONMENT-BLOCKED in this task environment, specifically: pytest modules that import pydantic, numpy, fastapi or require build tooling; python3 -m build --no-isolation and wheel build/installation were not completed due to missing build/setuptools.build_meta.
• Test outcomes (summary):
  • PASSED: compileall (src/test/scheduler/proxy/instance), test/test_repository_governance.py (14 passed), test/test_documentation_governance.py (12 passed), targeted repository-governance assertions adjusted and validated.
  • ENVIRONMENT-BLOCKED: test/observability collection/run (missing pydantic/numpy), test/test_namespace_layout.py and other contract/wheel tests that require pydantic, numpy, fastapi, or packaging tooling, and python3 -m build (missing build module / setuptools backend in this environment).
  • NOT RUN / NOT COMPLETED: wheel build and isolated-install validations (environment lacks build tooling and network for fresh package installs).

Known limitations and next steps

• The CI / reviewer environment must run the full test matrix with required native/python dependencies installed (pydantic, numpy, fastapi, build toolchain) and produce the full validation results and wheel artifacts requested by Issue v0.1.10-4C: propagate internal trace context from Proxy to Instance and instrument Instance-observed vLLM transport stages #185; those steps were blocked here by the execution environment.
• Request a fresh Codex/maintainer review and CI run that executes the full Required Validation list from the merged-head to confirm all tests and wheel build pass in a dependency-complete environment.

Closes #185
Refs #137, #141, #142, #157, #159, #178, #179, #182, #183, #180, #184, #139, #140

---

Codex Task

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3db2c5559d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Accept propagated headers in the cancellation test stub

The new keyword makes both cases of test_real_ready_worker_cancellation_paths fail because its monkeypatched fake_forward_request in test/observability/test_scheduler_proxy_production_paths.py still accepts only url, data, and use_chunked. Once the ready worker reaches this call, Python raises TypeError before the test can cancel the simulated downstream operation, so the request is finalized as failed rather than cancelled and the assertions at lines 304–305 fail; update that stub to accept extra_headers as the other forwarding stubs now do.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Finalize traces when streaming iterators are closed

When a streaming consumer stops early and closes this async generator—for example, after a client disconnect causes the response body iterator to be closed—Python injects GeneratorExit, which is handled by neither CancelledError nor Exception. Calling aclose() after the first yielded chunk therefore bypasses both the cancellation finalization and the success block, leaving completion/decode stages running and session.request_trace unset; handle generator closure as cancellation so these research timing stages are always finalized.

AGENTS.md reference: AGENTS.md:L203-L212

Useful? React with 👍 / 👎.