v0.1.10-4C: propagate internal trace context from Proxy to Instance and instrument Instance-observed vLLM transport stages

[rickisba]

Context

PR #179 completed the Phase 4A observability foundation under:

• cacheroute.observability;
• cacheroute.observability.v1.

PR #183 completed Phase 4B:

• Scheduler creates the canonical internal TraceContext;
• Scheduler overwrites the reserved propagation headers;
• Proxy validates or locally replaces the context;
• Proxy records prepare queue, ready queue, first response, stream/decode, and completion stages;
• Proxy finalizes one immutable process-local RequestTrace.

PR #184 established the developer handbook and same-PR documentation governance.

The current canonical propagation boundary stops at Proxy. Instance receives the existing OpenAI-compatible body and forwards it to the configured real vLLM endpoint or mock backend, but it does not receive the canonical TraceContext and does not produce an Instance-local immutable trace.

This Issue defines the next narrow implementation slice under #141.

It does not complete or close #141.

Refs #137
Refs #141
Refs #142
Refs #157
Refs #159
Refs #178
Refs #179
Refs #182
Refs #183
Refs #180
Refs #184
Refs #139
Refs #140

Objective

Extend the existing internal trace boundary from Proxy to Instance and produce one deterministic immutable Instance-local RequestTrace describing only the transport boundaries that Instance can actually observe.

The implementation must:

1. preserve the accepted or Proxy-local TraceContext when Proxy forwards to the selected Instance;
2. let Instance validate the context or create a safe local fallback;
3. measure Instance-observed downstream request, first-response, stream, and completion boundaries with the canonical monotonic clock;
4. keep all request bodies, response bodies, SSE events, routing, injection, queueing, fallback, mock behavior, and vLLM behavior unchanged.

Scope boundary

This slice covers:

• Proxy-to-Instance internal TraceContext propagation;
• Instance startup resolution of observability metadata;
• Instance validation and local trace fallback;
• Instance-observed downstream completion;
• Instance-observed first non-empty downstream chunk;
• Instance-observed streaming interval;
• non-streaming, failure, cancellation, and empty-stream finalization;
• one immutable process-local Instance RequestTrace;
• CPU-only Proxy-to-Instance integration coverage.

This slice stops before vLLM and LMCache instrumentation.

No canonical trace header is forwarded from Instance to vLLM.

Canonical propagation

Reuse the existing canonical propagation implementation in:

src/cacheroute/observability/propagation.py

Do not define a second header vocabulary.

Generalize comments and documentation that currently describe the helper as Scheduler-to-Proxy-only, but preserve the existing public exports and wire values.

Reuse the existing reserved header set:

• scheduler-request-id;
• x-cacheroute-trace-version;
• x-cacheroute-trace-id;
• x-cacheroute-runtime-profile;
• x-cacheroute-trace-sampled;
• x-cacheroute-trace-created-at.

Do not introduce a JSON/base64 trace header.

Do not propagate:

• prompts;
• messages;
• generated text;
• Authorization values;
• KID contents;
• request or response bodies;
• physical paths;
• Redis keys;
• KV bytes;
• raw exceptions;
• private vLLM or LMCache objects.

Proxy forwarding requirements

Immediately before the existing Proxy-to-Instance request:

1. encode the current ProxyTask TraceContext through the canonical helper;
2. overwrite the complete reserved observability header set;
3. preserve the current Instance URL;
4. preserve the current OpenAI-compatible request body;
5. preserve all existing non-observability headers;
6. preserve streaming and non-streaming behavior.

The Proxy must forward:

• the accepted Scheduler context when it passed Proxy validation;
• the Proxy-local fallback context when Scheduler propagation was absent or invalid.

Client-provided observability values must not survive the normal Client -> Scheduler -> Proxy -> Instance path.

The added internal headers must not change:

• Instance selection;
• queue ordering;
• reservation or prediction;
• text/KV injection;
• fallback;
• retry;
• timeout;
• response status;
• response streaming.

Instance startup configuration

Instance must resolve the existing settings once during lifespan startup:

• CACHEROUTE_RUNTIME_PROFILE;
• CACHEROUTE_TRACE_SAMPLE_RATE.

Reuse:

• resolve_observability_startup;
• the canonical RuntimeProfile;
• the canonical sample-rate parser.

Requirements:

• persisted profile never remains auto;
• the compatibility default remains legacy;
• invalid sample rate fails closed to 0.0;
• at most one bounded startup warning is emitted;
• observability metadata must not select mock/real vLLM mode;
• observability metadata must not alter routing, injection, registration, heartbeat, or backend selection.

Store the resolved values in Instance application state.

Do not add another RuntimeProfile enum or normalization implementation.

Instance context validation and fallback

Instance must decode the complete reserved header set through the canonical helper.

For an otherwise valid request:

• missing headers must not fail the request;
• malformed headers must not fail the request;
• stale headers must not fail the request;
• runtime-profile mismatch must not fail the request;
• unsupported schema must not fail the request.

A valid accepted context retains:

• trace ID;
• Scheduler request ID;
• resolved Runtime Profile;
• sampled decision;
• original creation timestamp.

When the context cannot be accepted, create one Instance-local context using:

• a locally generated canonical trace ID;
• the local resolved Runtime Profile;
• the local sample rate;
• a bounded local request ID using the existing req_<32 lowercase hex> vocabulary when no Scheduler request ID is available;
• the injected UTC clock.

Log only bounded reason codes. Never log rejected raw header values.

This Issue does not introduce authentication between Proxy and Instance. Trace metadata remains correlation-only and must never authorize a request or select runtime behavior.

Request-local collector ownership

Create one request-local TraceCollector only for sampled requests.

Do not create:

• a global collector;
• a singleton trace registry;
• a trace database;
• persistent trace storage;
• an in-memory unbounded history.

The request handler may use a small Instance-local session/helper object or request state, but canonical models and clocks must remain owned by cacheroute.observability.

No new functional repository-root directory is allowed.

Service-specific integration must remain in the existing Instance hierarchy. Do not introduce cacheroute.services.instance as part of this Issue.

Instance-observed stage semantics

Use existing observability schema vocabulary.

Completion

Use TraceStageName.COMPLETION.

Start immediately before Instance invokes the current real-vLLM or mock downstream helper.

Finish when the downstream response has completed, failed, or been cancelled.

Provenance must use:

• TraceComponent.INSTANCE;
• the context Runtime Profile;
• the injected UTC clock;
• a bounded logical Instance endpoint label where useful.

This stage is an Instance-observed downstream transport interval.

It is not authoritative vLLM execution time.

First response

For streaming requests, use TraceStageName.FIRST_TOKEN.

Start at downstream invocation and finish at the first non-empty downstream chunk observed by Instance.

Use TraceComponent.INSTANCE, not TraceComponent.VLLM.

Do not claim that this timestamp is the internal vLLM token-generation time.

For non-streaming requests, mark the stage skipped with a stable reason.

Decode / stream interval

For streaming requests, use TraceStageName.DECODE.

Start after the first non-empty downstream chunk and finish when the downstream stream ends.

This is an Instance-observed stream interval, not authoritative internal vLLM decode execution.

For non-streaming requests, mark the stage skipped with a stable reason.

VLLM prefill

Do not emit TraceStageName.VLLM_PREFILL in this Issue.

That stage remains reserved for an actual vLLM-correlated signal or a separately reviewed integration boundary.

Do not infer prefill duration from HTTP latency.

Mock mode

Mock mode must remain behaviorally unchanged.

Stages created while using the mock backend still use Instance provenance and must not claim to be vLLM or LMCache measurements.

Parent relationships

For streaming requests:

• completion is the overall downstream span;
• first response may reference completion as its parent;
• decode may reference completion as its parent.

Repeated stage names across Proxy and Instance are valid because each process-local trace has independent ordered stages and common TraceContext correlation.

Do not merge Proxy and Instance snapshots inside either service.

Cross-process aggregation remains future work.

Terminal paths

Finalize without running stages for at least:

• successful streaming response;
• successful non-streaming response;
• real-vLLM downstream failure before first response;
• failure after first response;
• cancellation;
• successful empty stream;
• mock streaming;
• mock non-streaming;
• unsampled request.

Use bounded static canonical errors such as:

• instance downstream request failed;
• instance request cancelled;
• instance stream ended before first response.

Do not copy raw exception strings into canonical traces.

Existing client-visible errors and logs remain unchanged.

Observability failure must never hang or replace the response.

Current wire compatibility

Preserve the existing shape and behavior of:

• Proxy-to-Instance OpenAI-compatible request bodies;
• chat/completions requests;
• completions requests;
• streaming chunks;
• non-streaming JSON;
• status codes;
• content types;
• current response headers;
• current mock responses;
• Instance-to-vLLM request bodies;
• Instance-to-vLLM existing headers;
• Proxy cacheroute_meta;
• Proxy _cacheroute_meta;
• Proxy legacy trace dictionary;
• Scheduler request ID behavior;
• Authorization handling.

The only intentional internal wire change is the canonical reserved trace headers added from Proxy to Instance.

Do not expose RequestTrace through:

• public response JSON;
• SSE events;
• cacheroute_meta;
• _cacheroute_meta;
• public headers;
• debug endpoints;
• logs.

Instance-to-vLLM boundary

Do not forward the CacheRoute reserved trace headers to vLLM.

Do not add W3C Trace Context.

Do not change the downstream payload.

Do not parse or rewrite streaming chunks for observability.

Do not introduce vLLM, LMCache, OpenTelemetry, or exporter dependencies.

Documentation

Update:

• doc/architecture/observability-v1.md;
• doc/research/issue-141-unified-observability.md;
• doc/developer-handbook/runtime-flows.md;
• doc/developer-handbook/configuration-and-interfaces.md;
• doc/developer-handbook/public-api-and-data-models.md only if the reviewed public helper surface changes;
• doc/developer-handbook/development-and-validation.md when validation commands change.

Document clearly:

• propagation now reaches Instance;
• Instance validation and local fallback;
• Instance-local collector ownership;
• Instance-observed first response, stream, and completion semantics;
• no authoritative vLLM prefill/decode timing yet;
• no LMCache hit-token or remote-read observation yet;
• no Instance-to-vLLM trace propagation;
• no external export, persistence, registry, or client-visible trace.

Do not modify the root README.

Governance

Add focused governance proving:

• one canonical module owns all reserved header literals;
• Proxy uses canonical header encoding;
• Instance uses canonical header decoding;
• no service defines a second TraceContext or RuntimeProfile;
• normal Scheduler/Proxy forwarding overwrites client trace headers;
• the exact reserved header set reaches Instance;
• no reserved trace header reaches vLLM;
• request bodies and SSE payloads contain no canonical trace object;
• canonical traces contain no prompt, generated text, Authorization, raw exception, physical path, Redis key, KV bytes, or private runtime object;
• observability helpers remain dependency-light and side-effect-free;
• package discovery remains explicit;
• no documentation or tests enter runtime wheel packages;
• the developer handbook remains synchronized.

Update the previous Phase 4B governance assertion that prohibited every Proxy-to-Instance trace header. Replace it with an exact-whitelist assertion and retain the prohibition at the Instance-to-vLLM boundary.

Required CPU-only tests

Add deterministic tests for at least:

1. Proxy forwarding the accepted TraceContext to Instance;
2. Proxy forwarding a Proxy-local fallback context;
3. complete reserved-header overwrite;
4. preservation of the existing Instance request body;
5. preservation of existing non-observability headers;
6. Instance startup RuntimeProfile resolution;
7. invalid Instance sample rate failing closed;
8. Instance accepting valid propagated context;
9. missing-header local fallback;
10. malformed-header local fallback;
11. stale-header local fallback;
12. runtime-profile-mismatch local fallback;
13. local req_<32 hex> request-ID generation when no Scheduler ID exists;
14. unsampled requests collecting no stages;
15. streaming first-response timing with ManualTraceClock;
16. streaming decode interval with ManualTraceClock;
17. streaming completion timing;
18. non-streaming first-response and decode skipped;
19. non-streaming completion;
20. failure before first response;
21. failure after first response;
22. cancellation;
23. successful empty stream;
24. mock streaming behavior unchanged;
25. mock non-streaming behavior unchanged;
26. no running stage at export;
27. deterministic sequence and parent references;
28. Instance provenance on every new stage;
29. no VLLM_PREFILL stage emitted;
30. no VLLM or LMCache provenance invented;
31. safe static canonical error mapping;
32. no trace headers forwarded Instance-to-vLLM;
33. no trace object added to JSON or SSE;
34. no prompt, messages, generated text, Authorization, raw exception, KID contents, Redis key, or KV payload in canonical trace;
35. Proxy canonical RequestTrace behavior unchanged.

Include one CPU-only in-process Proxy-to-Instance integration test with:

• deterministic TraceContext;
• ManualTraceClock;
• mocked Instance downstream forwarding;
• no live Scheduler;
• no live Proxy process;
• no live Instance process;
• no KDN;
• no Redis;
• no LMCache;
• no vLLM;
• no network;
• no GPU.

The integration test must prove:

• one Scheduler request ID survives Proxy-to-Instance propagation;
• one trace ID is shared by Proxy and Instance contexts;
• Proxy and Instance each produce separate immutable local traces;
• both traces retain their own component provenance;
• response body and streaming shape remain unchanged.

Existing regression requirements

Retain:

• the complete observability suite;
• Scheduler-to-Proxy propagation tests;
• Proxy production-path tests;
• Instance capability registration;
• Instance request handlers;
• mock responses;
• Proxy queue behavior;
• KDN contracts;
• repository governance;
• namespace and source-checkout imports;
• wheel installation and dependency isolation;
• documentation governance.

Do not weaken Phase 4B compatibility assertions except for replacing the old absolute prohibition on Proxy-to-Instance headers with the reviewed exact whitelist.

Required validation

Run from a clean final-head checkout:

python3 -m compileall -q src test scheduler proxy instance

python3 -m pytest -q test/observability

python3 -m pytest -q test/test_documentation_governance.py

python3 -m pytest -q test/test_contract_foundation.py

python3 -m pytest -q test/test_contract_service_migration.py

python3 -m pytest -q test/test_repository_governance.py

python3 -m pytest -q test/test_namespace_layout.py

python3 -m pytest -q -s test/test_source_checkout_imports.py

python3 -m pytest -q test/kdn

python3 -m pytest -q \
  test/test_instance_capability.py \
  test/test_instance_capability_registration.py

python3 -m pytest -q -s test/test_wheel_install.py -m "not network"

python3 -m build --no-isolation

git diff --check "$(git merge-base origin/main HEAD)" HEAD

Also run every existing test file directly affected by changes to:

• Proxy Instance forwarding;
• instance/instance_api.py;
• observability propagation;
• observability startup;
• documentation governance.

Build the wheel from a clean exported source tree and install it in an isolated environment outside the repository.

Report separately:

• PASSED;
• FAILED;
• ERROR;
• SKIPPED;
• DESELECTED;
• NOT RUN;
• ENVIRONMENT-BLOCKED;
• warnings.

Do not report an environment-blocked command as passed.

PR evidence

The final PR body must include:

• final head SHA;
• base/main SHA;
• complete changed-file list;
• exact Proxy-to-Instance header vocabulary;
• confirmation that Instance-to-vLLM headers remain unchanged;
• exact final test counts;
• build and wheel result;
• wheel filename, size, and SHA-256;
• source-checkout and isolated-install module origins;
• confirmation that request/response/SSE shapes are unchanged;
• confirmation that no canonical trace is client-visible;
• confirmation that no VLLM_PREFILL, LMCache, or actual hit-token claim was introduced;
• handbook impact statement;
• known limitations.

Non-goals

• Scheduler stage instrumentation;
• client-to-Scheduler external trace propagation;
• W3C Trace Context;
• OpenTelemetry;
• external exporters;
• trace persistence;
• a trace database;
• a global trace registry;
• a debug trace endpoint;
• client-visible RequestTrace;
• merging Proxy and Instance traces in one service;
• KDN trace propagation;
• Gateway instrumentation;
• production LMCache Gateway I/O;
• LMCache metrics/events;
• actual LMCache hit tokens;
• actual remote-read counters;
• authoritative vLLM prefill timing;
• authoritative vLLM decode timing;
• forwarding trace headers to vLLM;
• changing public request or response contracts;
• changing routing, queueing, injection, fallback, timeout, or retry behavior;
• adding dependencies;
• enabling automatic package discovery;
• moving Proxy or Instance packages;
• adding a functional root directory;
• modifying the root README;
• closing v0.1.10-4: Add Unified v1 LMCache Gateway, Tier, Adapter, and Queue Observability #141.

Acceptance criteria

• Proxy forwards one canonical TraceContext to the selected Instance.
• The accepted or Proxy-local trace ID is preserved.
• Existing Scheduler request ID remains correlated through the normal path.
• Proxy overwrites the complete reserved header set.
• Instance resolves RuntimeProfile and sampling once at startup.
• Instance accepts valid context and locally replaces invalid context.
• Invalid trace metadata never fails an otherwise valid request.
• Instance uses a request-local collector only when sampled.
• Streaming first response is explicitly Instance-observed.
• Streaming decode is explicitly an Instance-observed stream interval.
• Completion covers the Instance downstream request boundary.
• Non-streaming first-response and decode stages are explicitly skipped.
• Empty, failed, and cancelled paths finalize without running stages.
• No VLLM_PREFILL stage is fabricated.
• No stage claims vLLM or LMCache provenance without an actual signal.
• No trace header reaches vLLM.
• Public request, response, SSE, metadata, routing, injection, and fallback behavior remain unchanged.
• Canonical traces contain no sensitive or unbounded payload data.
• Documentation and handbook describe the new boundary accurately.
• Focused, regression, governance, namespace, documentation, wheel, and isolated-install validation passes.
• Project dependencies, package discovery, and root README remain unchanged.

This Issue is the Proxy-to-Instance propagation and Instance-observed transport instrumentation slice only.

It does not close #141.