observability: propagate trace context to Instance and instrument downstream stages

doc/architecture/observability-v1.md

@@ -49,18 +49,17 @@ copied. Ambiguous or overwritten values are labeled `legacy_projected`, never
 upgraded to actual observations. The current `cacheroute_meta` has no
 `request_id`, and this foundation does not add one.
 
-## Scheduler-to-Proxy propagation
+## Internal propagation
 
 The Scheduler now creates the internal context and overwrites the complete
 reserved header set: `scheduler-request-id`, `x-cacheroute-trace-version`,
 `x-cacheroute-trace-id`, `x-cacheroute-runtime-profile`,
 `x-cacheroute-trace-sampled`, and `x-cacheroute-trace-created-at`. The request
 ID allocated by the Scheduler remains authoritative. Client values using these
 names are never trusted; Authorization forwarding and the serialized payload
-remain unchanged. Propagation stops at the Proxy and no trace header or model
-is sent to an Instance or returned to a client.
+remain unchanged. Propagation now reaches the selected Instance through the same exact reserved header set. Proxy encodes its accepted Scheduler context, or its Proxy-local fallback context, immediately before forwarding to Instance and overwrites every reserved value for that internal call. No arbitrary client trace header becomes authoritative and no JSON/base64 trace payload is added to the request body. Instance validates the complete reserved set, accepts matching fresh context, or creates a request-local fallback context using its resolved startup profile and sample rate. The reserved headers stop at Instance: Instance-to-vLLM forwarding does not include CacheRoute trace headers, W3C Trace Context, or any canonical trace object, and no trace is returned to a client.
 
-Scheduler and Proxy each resolve `CACHEROUTE_RUNTIME_PROFILE` through an explicit lifespan startup helper, with `legacy` as the compatibility default. Because the current services do not have an implemented production v1 data path, `auto` resolves with `v1_available=False` and is stored as `legacy`; explicit `legacy`, `test/mock`, and `v1` remain valid metadata values, but no stored or propagated context can remain `auto`. A missing, malformed, stale, request-ID-mismatched, or profile-mismatched context causes a Proxy-local context to be created and never rejects an otherwise valid request. Profile metadata does not select runtime behavior. The propagation freshness rule retains the five-minute maximum age and also accepts only a bounded 30-second future clock skew between Scheduler and Proxy clocks.
+Scheduler and Proxy each resolve `CACHEROUTE_RUNTIME_PROFILE` through an explicit lifespan startup helper, with `legacy` as the compatibility default. Because the current services do not have an implemented production v1 data path, `auto` resolves with `v1_available=False` and is stored as `legacy`; explicit `legacy`, `test/mock`, and `v1` remain valid metadata values, but no stored or propagated context can remain `auto`. A missing, malformed, stale, request-ID-mismatched, or profile-mismatched context causes a Proxy-local context to be created and never rejects an otherwise valid request. Profile metadata does not select runtime behavior. The internal propagation freshness rule retains the five-minute maximum age and accepts only a bounded 30-second future clock skew among Scheduler, Proxy, and Instance clocks.
 
 `CACHEROUTE_TRACE_SAMPLE_RATE` defaults to `0.0`. Invalid configuration fails
 closed to that value. Rates of zero and one disable or enable collection for
@@ -83,3 +82,10 @@ vLLM execution, prefill, decode, Gateway, or LMCache timings.
 Collection remains process-local and immutable. The Legacy trace mapping and
 client metadata are unchanged and are not copied into `RequestTrace`. There is
 no client export, external exporter, registry, debug endpoint, or persistence.
+
+
+## Instance-observed stages
+
+Instance resolves `CACHEROUTE_RUNTIME_PROFILE` and `CACHEROUTE_TRACE_SAMPLE_RATE` once during FastAPI lifespan startup and stores the immutable result in application state. The compatibility default remains `legacy`; `auto` is resolved and never persisted; invalid sample rates fail closed to `0.0` with at most one bounded warning reason. These settings are correlation metadata only and do not select mock versus real vLLM mode, registration, heartbeat, topology discovery, routing, injection, fallback, retries, or timeouts.
+
+For sampled valid Instance requests, collection is request-local. Completion spans only the Instance-observed downstream boundary around the existing mock or real-vLLM helper. Streaming first response spans from downstream invocation to the first non-empty downstream chunk observed by Instance. Streaming decode spans from after that chunk until the downstream stream ends. Non-streaming requests skip first response and decode with `non_streaming_request`. Empty streams, failures, and cancellations finalize without running stages and use bounded static canonical errors. All Instance stages use `TraceComponent.INSTANCE`; this slice does not emit `VLLM_PREFILL`, does not claim vLLM or LMCache provenance, and does not provide authoritative internal vLLM prefill/decode timing, LMCache hit-token data, remote-read data, external export, persistence, registry, aggregation, or client-visible traces. Proxy and Instance may contain repeated stage names in separate process-local traces that share one `TraceContext`.

doc/developer-handbook/configuration-and-interfaces.md

@@ -6,8 +6,8 @@ Catalog only developer-facing surfaces. Source definitions remain authoritative;
 
 | Name | Owner | Type | Exact default | Allowed values | Scope | Effect | Invalid/fallback behavior | Source | Validation |
 |---|---|---|---|---|---|---|---|---|---|
-| `CACHEROUTE_RUNTIME_PROFILE` | Scheduler/Proxy observability | string | unset -> service passes default legacy startup resolution | `auto`, `legacy`, `v1`, `test/mock` plus compatibility aliases accepted by normalizer | startup | Resolves trace runtime metadata | Current Scheduler and Proxy use `v1_available=False`; `auto` resolves to `legacy`; invalid values raise through runtime normalization | `scheduler/scheduler.py`, `proxy/proxy.py`, `src/cacheroute/observability/startup.py` | `test/observability` |
-| `CACHEROUTE_TRACE_SAMPLE_RATE` | Scheduler/Proxy observability | float string | unset -> `0.0` | finite `0.0` through `1.0` | startup | Sets deterministic trace sampled flag | malformed/non-finite/out-of-range values fail closed to `0.0` and expose warning reason | `src/cacheroute/observability/propagation.py` | `test/observability` |
+| `CACHEROUTE_RUNTIME_PROFILE` | Scheduler/Proxy/Instance observability | string | unset -> service passes default legacy startup resolution | `auto`, `legacy`, `v1`, `test/mock` plus compatibility aliases accepted by normalizer | startup | Resolves trace runtime metadata | Current Scheduler, Proxy, and Instance use `v1_available=False`; `auto` resolves to `legacy`; invalid values raise through runtime normalization | `scheduler/scheduler.py`, `proxy/proxy.py`, `instance/instance_api.py`, `src/cacheroute/observability/startup.py` | `test/observability` |
+| `CACHEROUTE_TRACE_SAMPLE_RATE` | Scheduler/Proxy/Instance observability | float string | unset -> `0.0` | finite `0.0` through `1.0` | startup | Sets deterministic trace sampled flag | malformed/non-finite/out-of-range values fail closed to `0.0` and expose warning reason | `instance/instance_api.py`, `src/cacheroute/observability/propagation.py` | `test/observability` |
 | `SCHEDULER_LOG_FILE` | Scheduler | path/string | non-portable Transitional Legacy source default (repository-external absolute log path in `core/config.py`; override for portable deployments) | path-like string | startup | Scheduler log path/config value | falls back to source owner default | `core/config.py`, `scheduler/scheduler.py` | demos |
 | `SCHEDULER_VERBOSE_REQUEST_LOG` | Scheduler | int/string | `1` | intended `0` or `1` | request | Enables verbose request scheduling logs | compared to integer `1` in current source; string environment values do not match that equality | `core/config.py`, `scheduler/scheduler.py` | demos |
 | `SCHEDULER_MODEL_PATH` | Scheduler | path/string | non-portable Transitional Legacy source default from `core/config.py` (repository-external model path; override for portable deployments) | path/model identifier | startup | Model path used by scheduler tokenizer/model setup | falls back to source owner default | `scheduler/scheduler.py`, `core/config.py` | source checkout tests |
@@ -117,7 +117,7 @@ Catalog only developer-facing surfaces. Source definitions remain authoritative;
 | Scheduler control plane | `/healthz`, `/v1/proxy/register`, `/v1/proxy/heartbeat`, `/v1/proxy/unregister`, `/v1/proxy/list`, `/debug/proxy_pool_resources`, `/v1/kdn/register`, `/v1/kdn/heartbeat`, `/v1/kdn/unregister`, `/v1/kdn/list` | Proxy/KDN registration, heartbeat, unregister, list params | JSON control-plane status/resources | Current Transitional |
 | Proxy data plane | `POST /v1/chat/completions`, `POST /v1/completions` | Scheduler-forwarded internal request model and reserved trace headers | Chat may stream `text/event-stream`; completions returns JSON; Proxy collects local prepare/ready/first response/decode/completion stages | Current |
 | Proxy control plane | `/healthz`, `/debug/status`, `/v1/instance/register`, `/v1/instance/heartbeat`, `/v1/instance/resource_snapshot`, `/v1/instance/unregister`, `/v1/instance/list`, `/debug/pool_resource`, `/debug/pool_resource_sources`, `/debug/instance_resources`, `/debug/instance_loads`, `/v1/topology/report`, `/v1/topology/kdn_links` | Instance registration/heartbeat/resource/topology payloads; list supports `include_dead` | JSON pool, resource, and topology snapshots | Current Transitional |
-| Instance data plane | `POST /v1/chat/completions`, `POST /v1/completions` | Proxy-forwarded OpenAI-like body | Forwards to vLLM or mock; no canonical trace context propagated from Proxy to Instance | Current Transitional |
+| Instance data plane | `POST /v1/chat/completions`, `POST /v1/completions` | Proxy-forwarded OpenAI-like body plus exact internal reserved trace headers | Forwards to vLLM or mock with unchanged public JSON/SSE; validates or locally replaces context; no trace header reaches vLLM and no canonical trace is client-visible | Current Transitional |
 | Instance control plane | `GET /healthz`, `POST /v1/kv/inject_ready` | health or KV ready payload | JSON health or KV injection signaling result | Current Transitional |
 | KDN | `/v1/topology/hello`, `/v1/topology/ping`, `/knowledge/snapshot`, `/knowledge/register_text`, `/knowledge/build_kv`, `/knowledge/search/text`, `/knowledge/delete`, `/knowledge/purge_all`, `/knowledge/inject_ready_kv`, `/knowledge/pool_status` | topology, knowledge, KV build/search/delete/inject payloads | JSON knowledge/KV/status payloads; network timing fields when simulated network is enabled | Current Transitional |
 | Client validation | not a service endpoint | CLI accepts full URL, `-H/--header`, `-d/--data/--data-raw`; validates OpenAI-like fields | Prints response and CacheRoute metrics; unsupported body keys are validation errors | Current |

doc/developer-handbook/runtime-flows.md

@@ -8,8 +8,8 @@
 | Text versus KVCache injection | Scheduler, Proxy, KDN, Instance | `Injection_type`: `text`, `kvcache`, `hybrid`; knowledge IDs | Proxy strategies and request preparation | Proxy task-level timing/logging and UI state | If KV is unavailable, current compatibility may recompute text depending on path | Proxy demos | Unified policy package migration |
 | Cache artifact and cache-operation flow | `cacheroute.cache`, KDN contract models, legacy KDN paths | `artifact_`, `cacheop_`, endpoint compatibility IDs | KDN contract requests | Contract JSON models | Unsupported or unknown compatibility must remain explicit | contract foundation tests | Gateway/LMCache operation execution adapters |
 | Instance capability registration | Instance, Proxy | capability identity, compatibility profile, endpoint-like data | `instance/control_plane.py`, demo instance | Proxy Instance list and resource snapshots | Capability mismatch prevents unsafe reuse | instance/proxy demos | Canonical topology registry migration |
-| Scheduler-to-Proxy observability propagation | Scheduler, Proxy | reserved trace headers, `trace_`, request ID, runtime profile | Scheduler forwarding path and Proxy request path | Process-local Proxy trace stages | Proxy validates headers or creates local fallback context | `test/observability` | Client-returned canonical traces, Proxy-to-Instance context |
-| Proxy-local request-stage collection | Proxy | prepare queue, ready queue, first response, decode, completion | Proxy request execution | Process-local collector | These are transport/Proxy-observed intervals, not authoritative vLLM prefill/decode timings | `test/observability` | vLLM/Instance timing instrumentation |
+| Scheduler-to-Proxy-to-Instance observability propagation | Scheduler, Proxy, Instance | reserved trace headers, `trace_`, request ID, runtime profile | Scheduler forwarding path, Proxy request path, Proxy-to-Instance forwarding, Instance handlers | Separate process-local Proxy and Instance trace stages | Proxy and Instance validate headers or create local fallback context; exact reserved set stops at Instance | `test/observability` | Client-returned canonical traces, cross-process aggregation |
+| Proxy-local and Instance-local request-stage collection | Proxy, Instance | prepare queue, ready queue, first response, decode, completion | Proxy queue execution; Instance chat/completion handlers | Request-local collectors and immutable process-local traces | These are transport-observed intervals, not authoritative vLLM prefill/decode timings; no trace header reaches vLLM | `test/observability` | authoritative vLLM/LMCache timing instrumentation |
 | Legacy compatibility paths | Root packages and shims | legacy runtime profile, Redis/LMCache key projections | existing CLIs/demos | Existing debug/README outputs | Shims preserve imports and wire compatibility | governance and wheel tests | Removal after approved migration milestones |
 
-Proxy-observed transport intervals are useful for experiments but are not authoritative vLLM prefill/decode execution timings. Treat vLLM prefill/decode as not currently instrumented by canonical trace collection.
+Proxy- and Instance-observed transport intervals are useful for experiments but are not authoritative vLLM prefill/decode execution timings. Treat vLLM prefill/decode as not currently instrumented by canonical trace collection.

doc/research/issue-141-unified-observability.md

@@ -37,3 +37,6 @@ ready-queue, and downstream transport stages become a process-local immutable
 trace. The canonical trace still is not client metadata, Instance input, or an
 authoritative account of vLLM or LMCache execution. This increment therefore
 does not close the broader Issue #141 research program.
+
+
+Issue #185 extends the internal propagation boundary from Proxy to Instance for the exact reserved header vocabulary only. Instance now records sampled request-local, process-local transport stages for completion, first non-empty streaming response, and stream interval, while preserving public request/response/SSE shapes and keeping Instance-to-vLLM requests free of trace headers. Cross-process aggregation, Scheduler stage collection, vLLM/LMCache provenance, authoritative prefill/decode, external export, persistence, and debug trace retrieval remain future work under the umbrella.