Feature: Track Prompt Provenance and Resolved Version for Every Agent Run

[Asaf-prog]

Feature: Track Prompt Provenance and Resolved Version for Every Agent Run

Problem

Extra is moving toward supporting prompt sources that are not necessarily static local files, including Langfuse Prompt Management.

Once prompts can be loaded from a remote prompt-management system, knowing only the prompt name or configured label is not enough to understand what actually happened during a production run.

For example, an agent may be configured with:

prompts:
  system:
    langfuse:
      name: billing-agent
      label: production

The production label is mutable.

Today it may point to:

billing-agent version 17

and tomorrow it may point to:

billing-agent version 18

Consider a production incident:

User request
    ↓
Agent gives incorrect response
    ↓
Developer opens the trace several days later

If the run only records:

prompt_name=billing-agent
prompt_label=production

we still cannot answer the most important question:

Which exact prompt did this run execute?

By the time the incident is investigated, production may already point to a different version.

This makes debugging, auditing, replay, evaluation, and regression analysis unreliable.

---

Goal

Every model execution should be traceable back to the exact resolved prompt revision that was actually used.

Extra should record prompt provenance as part of the run/execution metadata.

Conceptually:

Prompt configuration
        ↓
Resolve prompt source
        ↓
Resolve exact prompt revision
        ↓
Render prompt
        ↓
Execute model
        ↓
Record provenance

For a Langfuse-backed prompt, the resulting metadata could look conceptually like:

source=langfuse
name=billing-agent
requested_label=production
resolved_version=17

The important distinction is:

configured reference
        ↓
"production"

resolved immutable revision
        ↓
version 17

Both are useful and should not be conflated.

---

Why This Should Be Generic

This capability should not be implemented as Langfuse-specific tracing logic.

Prompt provenance is an Extra-level concept.

Today a prompt may come from:

local file
Langfuse

In the future it could come from:

another prompt registry
database
remote configuration service
Git-backed prompt store

Higher layers should therefore work with a generic prompt provenance model.

Conceptually:

PromptSource
    ↓
ResolvedPrompt
    ├── content
    └── provenance

For example:

ResolvedPrompt(
    content="...",
    provenance=PromptProvenance(
        source="langfuse",
        identifier="billing-agent",
        requested_ref="production",
        resolved_revision="17",
    ),
)

The exact API should follow the existing prompt-loading architecture, but model execution and tracing should not need to know how Langfuse itself works.

---

Suggested Provenance Model

A resolved prompt should expose metadata similar to:

source
identifier
requested_ref
resolved_revision
content_hash

For example:

{
  "source": "langfuse",
  "identifier": "billing-agent",
  "requested_ref": "production",
  "resolved_revision": "17",
  "content_hash": "sha256:..."
}

Not every source needs every field.

Langfuse example

{
  "source": "langfuse",
  "identifier": "billing-agent",
  "requested_ref": "production",
  "resolved_revision": "17",
  "content_hash": "sha256:..."
}

Local-file example

{
  "source": "file",
  "identifier": "prompts/billing/system.md",
  "requested_ref": null,
  "resolved_revision": null,
  "content_hash": "sha256:..."
}

The content hash gives us a stable fingerprint even when the underlying source has no native version concept.

---

Important: Record the Resolved Version, Not Only the Label

For remote prompt management, this is the core requirement.

The following is insufficient:

name=billing-agent
label=production

because labels can move.

The execution metadata must capture the immutable revision returned by the prompt provider:

name=billing-agent
requested_label=production
resolved_version=17

If configuration pins an explicit version:

langfuse:
  name: billing-agent
  version: 12

then metadata should still record the resolved version:

requested_version=12
resolved_version=12

This keeps the execution metadata consistent regardless of whether the prompt was selected through a mutable label or an immutable version.

---

Run and Trace Metadata

Prompt provenance should be attached to the execution context in a way that makes it available to:

• Extra run metadata.
• Observability/tracing providers.
• Debug logs where appropriate.
• Future evaluation/replay tooling.

For example, a Langfuse trace could contain metadata conceptually like:

{
  "agent_id": "billing_agent",
  "prompt": {
    "source": "langfuse",
    "name": "billing-agent",
    "requested_ref": "production",
    "resolved_version": "17",
    "content_hash": "sha256:abc..."
  }
}

For systems with orchestrators and multiple agents, provenance must be associated with the specific model invocation/node, not stored as one ambiguous run-level value.

For example:

run_id=123

orchestrator
  prompt version=8

billing_agent
  prompt version=17

policy_agent
  prompt version=4

A single run may therefore contain several resolved prompts.

---

Prompt Rendering vs. Prompt Source

There are two separate things worth distinguishing:

Source prompt
     ↓
resolver/interpolation
     ↓
Final rendered prompt

For example, Langfuse version 17 may contain:

You are the billing assistant for {{customer_name}}.

and runtime resolution may produce:

You are the billing assistant for Acme Corp.

The provenance should describe the source revision:

Langfuse billing-agent version 17

It may also be useful to record a hash of the final rendered prompt.

Conceptually:

{
  "source_revision": "17",
  "source_content_hash": "sha256:...",
  "rendered_content_hash": "sha256:..."
}

This allows us to distinguish:

same prompt version + different runtime values

without necessarily persisting the full rendered prompt.

The implementation should evaluate whether both hashes are useful, but the source revision must always remain identifiable.

---

Security and Sensitive Data

Prompt provenance must not introduce accidental leakage of sensitive runtime values.

For example, resolver values may contain:

customer information
authorization context
internal data
PII

Therefore, provenance should primarily store identifiers, versions, and hashes rather than blindly persisting complete rendered prompt content.

Existing observability behavior may already capture model input depending on configuration, but provenance tracking itself should not require raw prompt persistence.

In other words:

tracking exact prompt identity

must not imply:

always store the complete rendered prompt

These should remain separate concerns.

---

Failure Semantics

If a configured remote prompt cannot be resolved, Extra should not invent incomplete provenance.

For example:

requested:
  billing-agent / production

Langfuse lookup fails

If the prompt-loading policy is fail-closed, startup/execution should fail according to that policy.

If a fallback mechanism is introduced later, provenance must make the fallback explicit.

For example:

{
  "source": "file",
  "identifier": "prompts/billing/fallback.md",
  "fallback_from": {
    "source": "langfuse",
    "identifier": "billing-agent",
    "requested_ref": "production"
  }
}

We should never make a trace appear as though the requested Langfuse prompt was successfully used when execution actually used something else.

---

Relationship to Langfuse Prompt Management

This complements the existing issue for loading prompts from Langfuse.

Prompt loading answers:

Where does the prompt come from?

Prompt provenance answers:

Which exact prompt revision produced this execution?

These should be separate architectural responsibilities.

Conceptually:

Langfuse Prompt Loader
        ↓
ResolvedPrompt
        ├── content
        └── provenance
                    ↓
              Agent execution
                    ↓
                 Trace

The prompt loader should provide the metadata.

The execution/tracing layers should consume generic provenance without knowing Langfuse-specific details.

---

Future Use Cases

This metadata becomes foundational for several future capabilities.

Production debugging

"This answer was generated with billing-agent v17."

Prompt regression analysis

Compare:

v17 → successful runs
v18 → increased failures

Evaluations

Run the same dataset against:

prompt v17
vs.
prompt v18

Replay

Reproduce a historical execution using the exact prompt revision that originally ran.

Rollback

Identify which prompt revision introduced a regression and restore the previous version.

Self-improvement

When Extra eventually proposes prompt improvements, a generated diff can clearly state:

based on production prompt version 17
proposed version 18

instead of modifying an unknown moving target.

---

Suggested Flow

agents.yml
   ↓
Prompt reference
   ↓
Prompt loader
   ↓
Resolve exact source revision
   ↓
ResolvedPrompt
├── content
└── provenance
   ↓
Render runtime variables
   ↓
Model invocation
   ↓
Run / invocation metadata
   ↓
Observability provider

---

Scope for v1

For the first version:

• Introduce a generic prompt provenance representation.

• Prompt loaders return both content and provenance.

• Local file prompts include their path and content hash.

• Langfuse prompts include:

  • prompt name;
  • requested label/version;
  • exact resolved version;
  • content hash.

• Propagate provenance through prompt rendering into model execution.

• Attach provenance to relevant run / trace metadata.

• Support multiple prompts within the same run.

• Do not require storing full rendered prompt content.

• Add tests ensuring mutable labels resolve to and record an immutable version.

---

Non-Goals for v1

• Full prompt rollback UI.
• Prompt diff visualization.
• Automatic prompt deployment.
• Historical prompt replay.
• Prompt performance dashboards.
• Evaluation framework.
• Automatically reverting a bad prompt version.

Those capabilities can build on the provenance metadata later.

---

Acceptance Criteria

• Every model invocation can identify the prompt source used.

• Langfuse-backed prompts record the exact resolved immutable version.

• Mutable labels such as production are recorded separately from the resolved version.

• A later label change does not alter the metadata of an existing historical run.

• Local prompts include a stable content hash.

• Prompt provenance is provider/source agnostic at the execution layer.

• Orchestrators and agents can each carry different prompt provenance within the same run.

• Prompt provenance is exposed to configured observability/tracing integrations.

• Resolver/interpolation does not destroy the original source identity.

• Tracking provenance does not require storing sensitive rendered prompt content.

• Failures/fallbacks do not falsely report a prompt that was never used.

• Tests cover:

  • local prompt provenance;
  • Langfuse label → resolved version;
  • explicit Langfuse version;
  • multiple nodes/prompts in one run;
  • label changes between runs;
  • content hashing;
  • propagation into tracing metadata.

Why This Matters

Once prompts become remotely managed and mutable, they effectively become versioned production artifacts.

We already expect to know:

which code commit ran?
which model ran?
which tool ran?

We should be able to answer the same question for prompts:

Which exact prompt produced this result?

Without that information, production traces stop being reproducible as soon as a prompt label moves.