[Feature] Preserve minimal provenance on durable task evidence

[kvnloo]

What outcome should OpenMuse handle?

Durable task evidence should retain enough provenance to tell what was actually observed, where it came from, and how it was acquired.

Today Evidence is intentionally small:

{
  id,
  kind: "mail" | "file" | "web" | "user",
  title,
  excerpt,
  url?
}

That works while each kind has one obvious source, but the current repo is already crossing that boundary:

• delegated read_web persists a direct browser observation as kind: "web";
• recurring monitors persist their page observation as kind: "web";
• Add Parallel as the built-in web search provider #22 persists search-provider excerpts as kind: "web";
• fix: keep browser evidence identity separate from session identity #29 separates browser observation identity from browser-session identity, but the evidence record still does not retain that acquisition provenance.

Those records can have the same URL/excerpt shape while meaning different things. A search excerpt is provider-returned evidence about a page; a browser read is an OpenMuse observation of the page itself. Treating both as the same provenance makes later verification, stale-data handling, debugging, and Learning/export harder.

The invariant I think we need is:

evidence content is not its provenance.

Proposed behavior

Add the smallest provenance envelope that is useful across existing evidence producers, without turning Evidence into a generic knowledge graph.

For example, optional fields along the lines of:

• observedAt
• source / acquisition such as browser, monitor, search, mail, user
• a stable source/entity reference when one exists (messageId, browser session, monitor, provider, etc.)

The exact schema is less important than preserving these distinctions:

• observation identity vs source/container identity;
• direct observation vs provider-supplied excerpt;
• source data vs authority.

I would keep provider-specific payloads out of the core record. Search-provider names or query IDs can live in a small optional provenance object if needed, while the main evidence shape stays stable.

How would we verify it works?

• A direct browser read and a search result for the same URL remain distinguishable after task restart.
• Two observations from one persistent browser session have distinct evidence IDs but share the correct source/session provenance.
• Monitor evidence identifies the monitor and observation time without using the monitor ID as the observation ID.
• Search adapters can change providers without changing the core evidence schema.
• Existing mail/file/user evidence remains backwards-compatible.
• UI can continue rendering title/excerpt/url without knowing provider details.

This seems worth deciding while #22/#28 add search and before more connectors or Automatic Learning depend on the current evidence shape.

AI-use note: I used an AI assistant to trace the current evidence producers and draft this issue; I verified the cited browser, monitor, mail, UI, and #22 search paths against current source/PR heads before filing.

[hachemchaabi]

Hi, I would love to work on this. Could you please assign it to me?

[kvnloo]

Thanks! I'd be glad to see this explored.

I'd keep the first increment deliberately small: preserve acquisition provenance on the existing "Evidence" record rather than introducing a broader evidence/knowledge abstraction.

The main invariants I'd try to pin down are:

• two browser observations can have distinct evidence IDs while sharing the same browser session/source;
• a direct browser observation and a search-provider excerpt for the same URL remain distinguishable;
• monitor evidence retains its monitor/source identity and observation time;
• existing mail/file/user evidence stays backwards-compatible.

An optional provenance/source field plus "observedAt" may be enough for the first pass. I wouldn't add confidence scores, inference graphs, or provider-specific payloads yet.

If you open a PR, happy to review the shape against those invariants.

AI-use note: I used an AI assistant to help inspect the current evidence producers and draft this scope suggestion; I verified the referenced behavior against the current issue and repository code.

[hachemchaabi]

@kvnloo

Opened #43 for this. It keeps the first pass minimal: an optional provenance: { acquisition, observedAt, sourceId?, provider? } on the existing Evidence, covering the four invariants you listed. Happy to adjust the shape.