feat(tracing): add memory read-path spans

[Kairo-J]

Which issue or RFC does this PR close?

Closes #1241

Rationale for this change

Memory read requests currently expose only top-level application spans, making it difficult to determine whether latency comes from retrieval, embeddings, reranking, Experience recall, or context assembly.

This change adds stage-level tracing while keeping OpenTelemetry owned by the Server layer. The Runtime remains framework-neutral, and tracing failures do not affect search behavior.

What changes are included in this PR?

• Add internal, framework-neutral RuntimeSpan and RuntimeTracing protocols.
• Inject optional Runtime tracing through BuiltinRuntime.
• Implement ServerTracing.stage() using OpenTelemetry INTERNAL spans.
• Record success, failure, and cancelled outcomes while preserving original exceptions.
• Add the following read-path spans:
  • memory.search
  • memory.rerank
  • experience.search
  • context.prepare
• Emit stable zero-result spans when Memory or Experience configuration is absent.
• Wrap actual reranker calls without changing reranking behavior or policy_id.
• Export only bounded booleans, enums, and counts.
• Do not record queries, content, vectors, IDs, or exception messages.
• Add unit and end-to-end coverage for span hierarchy, privacy, cancellation, disabled tracing, and tracing failure isolation.

Validated span trees:

powercontext search_memory
└── memory.search
    ├── embeddings <model>
    └── memory.rerank
        └── invoke_agent memory_rerank

powercontext prepare_context
├── memory.search
├── experience.search
└── context.prepare

Are there any user-facing changes?

Operators with tracing enabled will see the additional internal spans and bounded attributes.

Search results and error behavior remain unchanged. This PR does not change public APIs, OpenAPI, CLI behavior, database schemas, persisted formats, or dependencies.

How was this change tested?

make test
486 passed, 7 skipped

make check
Lock-file validation, pre-commit checks, Ruff formatting/linting, and ty type checking passed.

The tests also cover missing Memory, unconfigured Experience recall, empty results, reranker fallback, embedding and agent span nesting, sensitive-data exclusion, cancellation, and tracing failure isolation.

AI usage statement

OpenAI Codex (GPT-5.6-Sol) was used to help analyze the issue, assist with making changes, write tests.

[CLAassistant]

All committers have signed the CLA.

[Ethan-Xingyue]

This adds a fifth value for powercontext.operation.unit, but the docs that define the vocabulary aren't updated: the unit table in docs/{en,zh}/rfcs/0046_observability_foundations.md
and the span table in docs/{en,zh}/docs/how-to/trace-with-phoenix.md
.

[Kairo-J]

Get, it was an oversight on my part, and I will add it later.

[Ethan-Xingyue]

The policy_id setter has no callers — policy_id is only ever read, at service.py. I removed locally: uv run ty check stays green and tests/e2e/test_builtin_runtime.py + tests/e2e/test_observability.py still pass (8 passed). The getter is needed — _DynamicPolicyReranker covers the dynamic read — just not the setter.

What is the purpose of adding this setter?

[Kairo-J]

The setter was added because MemoryReranker declares policy_id as a read-write property, and the tracing decorator needs to fully proxy this interface.
It exists to forward future assignments made to the wrapper's policy_id to the underlying reranker; it is not required for the current business logic. Scenarios involving dynamic policy_id updates actually rely solely on the getter.
@Ethan-Xingyue @PsiACE Do you think it should be kept? Unless policy_id is explicitly defined as a read-only property, I would prefer to retain the setter functionality.

[PsiACE]

policy_id is plain state, so please keep it as a regular attribute instead of proxying it through a property with a setter.

[Kairo-J]

Okay, I'll modify _TracingMemoryReranker to use standard properties and synchronize the latest value to the delegate after rerank completes.

[Ethan-Xingyue]

Minor naming thought: this sits right next to the existing powercontext prepare_context span (app.py:1299), which covers the whole request, while this one wraps just the synchronous builder.build_result(). Might be worth something like context.assemble or context.build to make the difference obvious in a trace viewer?

[Kairo-J]

Get, I think it could be changed to context.build, as context.assemble might misleadingly imply the inclusion of the preceding Memory/Experience search, making the scope seem a bit too broad.

[Ethan-Xingyue]

Just a thought, not a blocker: _context() (which awaits self._provider.get() and opens a DB session) and the wait on _lock(scope_id) both happen before the first stage span opens, so they don't show up in the trace. Since the goal here is locating latency in the read path, and test_same_scope_read_only_searches_do_not_serialize_reranking suggests lock contention is already on your radar, those might be interesting to cover eventually.

[Ethan-Xingyue]

Those are my thoughts so far; let's wait for @PsiACE's review before settling on the specifics.

[PsiACE]

Please focus the tests on exported tracing behavior and real regressions: drop the dynamic policy_id, fake _RecordingTracing, and synthetic SystemExit cases, and cover the actual vector-search span tree plus the readiness probe's no-root-span behavior.

[Kairo-J]

Just a thought, not a blocker: _context() (which awaits self._provider.get() and opens a DB session) and the wait on _lock(scope_id) both happen before the first stage span opens, so they don't show up in the trace. Since the goal here is locating latency in the read path, and test_same_scope_read_only_searches_do_not_serialize_reranking suggests lock contention is already on your radar, those might be interesting to cover eventually.

@Ethan-Xingyue
Thanks, that’s indeed a great angle. Currently, the lock wait time during context resolution and the prepare_context phase appears in the external application Span merely as "unattributed exclusive time."

I noticed a detail here: the default RelationalContexts.get() (_context() line 1173) simply invokes the Provider to resolve and cache scoped services (RelationalContexts.get() line 434) without initiating a database session; the actual database read operation still occurs within memory.search. However, the skew caused by delayed attribution is indeed an issue.

Given that context retrieval and lock waiting are runtime concerns spanning multiple stages—whereas the scope of this PR is limited to the memory read phase—I intend to keep the PR focused.

We can add context.resolve and scope.lock.wait as sibling Spans in future work without altering the semantics of the Span introduced here. Thanks for pointing this out.