feat(tracing): propagate trace context into Temporal via OTel interceptor

[NiteshDhanpal]

Stack (3/3)

1. feat(tracing): correlate business spans with active observability trace ([WIP] feat(tracing): correlate business spans with active observability trace #465)
2. feat(tracing): establish W3C trace context at FastACP ingress
3. feat(tracing): propagate trace context into Temporal via OTel interceptor ← this PR

Based on obs/02-fastacp-w3c-ingress. Review/merge after PR 2.

What this PR does

Attaches temporalio.contrib.opentelemetry.TracingInterceptor to the Temporal client (both get_temporal_client entrypoints — workers/worker.py and clients/temporal/utils.py) when an OTel observability mode is active (SGP_OBS_MODE in {dual, lgtm}).

This threads the caller's W3C trace context (established at ingress in PR 2) across the workflow → activity boundary, so business spans created inside the model-loop activity see an active observability context and get tagged with its trace_id via obs_correlation() (PR 1).

Why it's needed

The agent model loop runs in Temporal activities. Without context propagation there is no active obs context inside the activity, so obs_correlation() would return {} for the very spans we most want to correlate (invoke_agent / plan / execute_tool / chat).

Safety

Gated on SGP_OBS_MODE in {dual, lgtm} and lazy-imported — the default dd_only mode adds no interceptor, so this is a no-op unless opted in.

Verification

py_compile + unit smoke on both entrypoints: empty in dd_only/unset, [TracingInterceptor] in dual.

Not in this stack (follow-ups)

• Configuring an OTel TracerProvider/OTLP exporter in the pod (this stack handles context propagation; span export to Tempo is separate).
• The egp-api-backend side (app-graph collector + CustomJSONFormatter unified trace_id) — tracked separately.

Greptile Summary

This PR attaches TracingInterceptor from temporalio.contrib.opentelemetry to both Temporal client entrypoints (clients/temporal/utils.py and temporal/workers/worker.py) so W3C trace context established at FastACP ingress propagates into scheduled workflow calls. The change is gated on SGP_OBS_MODE in {dual, lgtm} with a lazy import and ImportError fallback, making it a no-op in the default dd_only mode.

• Both files gain an identical _build_otel_interceptors() helper that conditionally returns a [TracingInterceptor()] list and is wired into connect_kwargs[\"interceptors\"] before Client.connect().
• The interceptor attaches at the client level (outbound workflow scheduling), but the Worker constructor in AgentexWorker.run() still receives only self.interceptors — it does not include the OTel interceptors, so trace context will not be extracted during inbound activity execution (see previously raised issue).

Confidence Score: 4/5

Safe to merge for the client-side path; the worker-side activity context extraction remains broken until the OTel interceptors are also merged into the Worker constructor.

The _build_otel_interceptors() result is wired into Client.connect() in both entrypoints, correctly propagating trace context for outbound workflow scheduling. However, AgentexWorker.run() builds Worker(interceptors=self.interceptors) without including the OTel interceptors, so the TracingInterceptor is never registered on the worker side. Trace context from incoming workflow execution headers will not be extracted into activities, meaning obs_correlation() inside the model-loop activity will still return {} — the core goal stated in the PR description is not yet achieved.

src/agentex/lib/core/temporal/workers/worker.py — specifically the Worker construction in AgentexWorker.run() which needs to merge _build_otel_interceptors() output with self.interceptors.

Important Files Changed

Filename	Overview
src/agentex/lib/core/clients/temporal/utils.py	Adds _build_otel_interceptors() and wires it into Client.connect(); correct for the client-only (FastACP scheduler) path where propagating context into outbound workflow starts is the goal.
src/agentex/lib/core/temporal/workers/worker.py	Adds _build_otel_interceptors() and wires it into Client.connect() inside get_temporal_client(), but AgentexWorker.run() still constructs Worker(interceptors=self.interceptors) without merging the OTel interceptors — trace context will not be extracted during activity execution, which is the stated goal of this PR.

Comments Outside Diff (1)

1. src/agentex/lib/core/temporal/workers/worker.py, line 228-255 (link)

   Worker-side TracingInterceptor never attached

   _build_otel_interceptors() runs inside get_temporal_client() and sets connect_kwargs["interceptors"], which lands on Client.connect(). That covers outbound calls from the client (e.g. scheduling a workflow). However, for the W3C context to be extracted and made active during activity execution, TracingInterceptor must also be passed to the Worker constructor. The current Worker(interceptors=self.interceptors) receives only user-provided interceptors — not the OTel ones — so obs_correlation() inside activities will still see an empty context, defeating the stated goal of this PR.

   A fix requires _build_otel_interceptors() to be called in AgentexWorker.run() (in addition to inside get_temporal_client()), and its result merged into the Worker's interceptors argument alongside self.interceptors.

   Prompt To Fix With AI

   This is a comment left during a code review.
   Path: src/agentex/lib/core/temporal/workers/worker.py
   Line: 228-255
   
   Comment:
   **Worker-side TracingInterceptor never attached**
   
   `_build_otel_interceptors()` runs inside `get_temporal_client()` and sets `connect_kwargs["interceptors"]`, which lands on `Client.connect()`. That covers outbound calls *from* the client (e.g. scheduling a workflow). However, for the W3C context to be extracted and made active *during activity execution*, `TracingInterceptor` must also be passed to the `Worker` constructor. The current `Worker(interceptors=self.interceptors)` receives only user-provided interceptors — not the OTel ones — so `obs_correlation()` inside activities will still see an empty context, defeating the stated goal of this PR.
   
   A fix requires `_build_otel_interceptors()` to be called in `AgentexWorker.run()` (in addition to inside `get_temporal_client()`), and its result merged into the `Worker`'s `interceptors` argument alongside `self.interceptors`.
   
   How can I resolve this? If you propose a fix, please make it concise.

   Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

   

_{Reviews (2): Last reviewed commit: "feat(tracing): propagate trace context i..." | Re-trigger Greptile}