diff --git a/CHANGELOG.md b/CHANGELOG.md index d6240762..daba6f4b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,8 @@ ### Added +- **Contradiction benchmark segment (#1172, phase 3)** — `benchmarks/longmemeval/contradiction_segment.py`: 40-topic active-store segment measuring conflict-resolution quality end-to-end. Baseline (both facts active) vs resolved (superseded): fusion winner@1 0.975 → 1.000, stale@5 0.825 → 0.000. Published in `benchmarks/longmemeval/RESULTS.md`. Also adds `uteke supersede [--reason]` — CLI surface parity for supersession (previously MCP/HTTP only). + - **`/list` pagination metadata (#1188)** — `POST /list` accepts `"include_meta": true` to respond with an envelope `{memories, total, has_more, next_offset}` (`next_offset` is `null` on the last page) so clients no longer blind-paginate with 100-row guesses. The default response is unchanged (bare array) — existing clients are untouched; `include_meta` is ignored in `at` (point-in-time) mode, which stays a bare array. - **Explain recall (#1160)** — `explain` mode on every recall surface shows WHY each memory ranked where it did: vector similarity and rank, FTS rank, RRF score with per-channel fusion contributions, and jaccard/salience/recency/graph boost deltas. Surfaces: `uteke recall "…" --explain` (human-readable, combine with `--json` for machine output), `POST /recall` with `"explain": true` (memory-only — combined with `search_type`/`at`/`before`/`after` returns 400), and the `explain` flag on the MCP `uteke_recall` tool. The explanation path replays the active strategy's exact pipeline (same channel depths, RRF constants, and boost order) while bypassing the recall cache, so the explanation always matches the returned results; fts5 explanation works without an embedder, other strategies embed the query once (~50 ms, same as a cold recall). diff --git a/benchmarks/longmemeval/RESULTS.md b/benchmarks/longmemeval/RESULTS.md index 7e03bd66..e5540c98 100644 --- a/benchmarks/longmemeval/RESULTS.md +++ b/benchmarks/longmemeval/RESULTS.md @@ -137,3 +137,45 @@ Binary built in-image from exact SHA bfbc296 (PR #1137/#1138), image build print Context: v0.15.0 hybrid baseline on the same dataset: Overall R@5 = 0.854 / R@10 = 0.885 (2026-08-13). Fusion default lifts full-500Q recall@5 by **+9.2 points** (0.854 → 0.946) with zero configuration. + +--- + +## Contradiction-resolution segment (#1172 Fase 3) — 2026-09-06 + +**Active-store knowledge-update segment**: 40 topics × (stale fact + winner fact + 3 distractors), +queries ask "which {thing} does {topic} use now?" (semantic, no keyword echo of the answer). +Baseline ranks with BOTH facts active; resolved ranks after `supersede(stale → winner)` — +baseline is measured for every strategy BEFORE any resolution, then the store is resolved once. +Binary: local release build (0.16.0 + #1185 ledger), local ONNX EmbeddingGemma, ARM64. + +Harness: `contradiction_segment.py` (this directory). Raw metrics: `results_contradiction_f3/metrics.json`. + +| Strategy | Stage | winner@1 | winner@5 | winner MRR | stale@1 | stale@5 | +|---|---|---|---|---|---|---| +| fusion (default) | baseline (unresolved) | 0.850 | 1.000 | 0.925 | 0.150 | **1.000** | +| fusion (default) | resolved (superseded) | **1.000** | 1.000 | **1.000** | 0.000 | 0.000 | +| hybrid | baseline | 0.025 | 1.000 | 0.469 | 0.975 | 1.000 | +| hybrid | resolved | 0.225 | 1.000 | 0.588 | 0.000 | 0.000 | +| vector | baseline | 0.950 | 1.000 | 0.975 | 0.050 | 1.000 | +| vector | resolved | 1.000 | 1.000 | 1.000 | 0.000 | 0.000 | + +Findings: + +- **Unresolved conflicts pollute every strategy's top-5**: with both facts active, the stale + fact sat in top-5 for 100% of topics on all strategies (hybrid's BM25 even ranks the stale + fact top-1 for 97.5% of topics — the old fact's "uses X" phrasing matches "use now?" + queries lexically). After `supersede`, stale@1 and stale@5 drop to **0.000** everywhere + (deprecated memories are excluded from recall). +- **Supersede lifts the default surface**: fusion winner@1 0.850 → 1.000, MRR 0.925 → 1.000; + vector 0.950 → 1.000. Hybrid stays weakest on winner@1 (lexical BM25 keeps the new fact's + "switched to" phrasing behind distractors) but its stale pollution is fully cleared. +- **Ledger integrity**: `contradictions list` listed all 40 resolutions; every stale fact is + restorable via `contradictions undo` (auditable conflict resolution, #1172 F2). + +Interpretation: ranking alone often picks the winner, but only explicit conflict resolution +guarantees stale facts leave the retrieval surface — the difference between "usually right" +(85–95% top-1) and deterministic freshness (100% top-1, zero stale). For agent memory, where +"use now" queries are the norm, resolution is what keeps top-1 trustworthy. This segment is +synthetic and deterministic (fixed topic list); it measures the conflict-resolution pipeline, +not LongMemEval dataset recall. + diff --git a/benchmarks/longmemeval/contradiction_segment.py b/benchmarks/longmemeval/contradiction_segment.py new file mode 100644 index 00000000..b6f205dd --- /dev/null +++ b/benchmarks/longmemeval/contradiction_segment.py @@ -0,0 +1,271 @@ +#!/usr/bin/env python3 +""" +Contradiction-segment benchmark (#1172 Fase 3). + +Measures how conflict resolution (supersede, #1053/#1172) affects retrieval +quality on a synthetic knowledge-update workload. This is the ACTIVE-store +counterpart to LongMemEval's passive knowledge-update subset (subset_kupd): +instead of asking whether stale sessions are retrieved, we resolve conflicts +in the store first and ask whether recall surfaces the WINNER. + +Design (deterministic, N topics): + 1. Seed 2 memories per topic: + - stale: the OLD fact ("… uses tool X") + - winner: the NEW fact ("… switched to tool Y") + plus D distractor memories per topic (same domain vocabulary, no conflict). + 2. Baseline run: both facts active (no supersede). Query each topic + semantically ("what does topic use now?"). Measures how often the + stale fact pollutes top-k when nothing resolved the conflict. + 3. Resolved run: supersede(stale → winner) via the CLI, then re-query. + Measures winner@k and stale@k on the RESOLVED store. + 4. Ledger sanity: contradiction_resolutions lists the pair; undo restores + (audited, then re-superseded). + +Metrics per strategy: + winner@{1,3,5} — winner memory ranked in top-k + stale@{1,5} — stale memory present in top-k (0.0 expected post-resolve) + MRR (winner) — reciprocal rank of the winner + +Usage: + python3 contradiction_segment.py --binary ../../target/release/uteke --topics 40 + python3 contradiction_segment.py --topics 40 --json out.json +""" + +import argparse +import json +import shutil +import subprocess +import sys +import tempfile +from pathlib import Path + +TOPICS = [ + ("acme-corp", "build tooling", "gradle", "bazel"), + ("brightpath", "package manager", "yarn", "pnpm"), + ("cloudline", "hosting provider", "heroku", "fly.io"), + ("dataworks", "message queue", "rabbitmq", "kafka"), + ("everhost", "web server", "apache", "nginx"), + ("fintrak", "ledger database", "postgresql", "cockroachdb"), + ("gadgethub", "mobile framework", "cordova", "flutter"), + ("heliosoft", "ci system", "jenkins", "github actions"), + ("innodata", "search engine", "elasticsearch", "meilisearch"), + ("jetstream", "api style", "soap", "grpc"), + ("kobalt", "css framework", "bootstrap", "tailwind"), + ("lumenpath", "state management", "redux", "zustand"), + ("metriq", "observability stack", "graphite", "prometheus"), + ("novabyte", "language runtime", "java", "kotlin"), + ("orbita", "container runtime", "docker swarm", "kubernetes"), + ("pixelbay", "image format", "jpeg-xl", "avif"), + ("quantex", "config format", "ini", "toml"), + ("riverbend", "version control", "svn", "git"), + ("saltmarsh", "auth protocol", "basic auth", "oauth2"), + ("tidewater", "template engine", "ejs", "handlebars"), + ("umbracloud", "storage layer", "mongodb", "sqlite"), + ("vellum", "docs generator", "javadoc", "rustdoc"), + ("wharfside", "package registry", "nexus", "ghcr"), + ("xenolith", "testing framework", "junit4", "junit5"), + ("yarrow", "scheduler", "cron", "systemd timers"), + ("zephyr", "linting tool", "tslint", "eslint"), + ("argonhold", "secret store", "env files", "vault"), + ("basaltix", "logging library", "log4j", "tracing"), + ("cobaltrun", "runtime monitor", "new relic", "otel"), + ("duskfield", "error tracker", "rollbar", "sentry"), + ("emberfall", "feature flags", "launchdarkly", "unleash"), + ("frostline", "cache layer", "memcached", "redis"), + ("glacierpeak", "object store", "s3 class", "r2"), + ("hollowpine", "markdown parser", "marked", "comrak"), + ("irisvale", "date library", "moment", "dayjs"), + ("jaderock", "http client", "axios", "fetch"), + ("kelpforest", "orm", "sequelize", "drizzle"), + ("lavaglass", "bundler", "webpack", "vite"), + ("mistvale", "type checker", "flow", "typescript"), + ("nightsky", "charting library", "chart.js", "d3"), +] + +# Question phrasings deliberately avoid the exact "uses/switched to" verbs +# so the query is semantic, not keyword lookup. +QUESTION = "which {thing} does {topic} use now?" + + +def resolve_binary(cli_path: str) -> str: + if cli_path: + p = Path(cli_path) + if p.exists(): + return str(p) + print(f"warning: --binary {cli_path} missing; falling back", file=sys.stderr) + repo = Path(__file__).resolve().parent.parent.parent + cand = repo / "target" / "release" / "uteke" + if cand.exists(): + return str(cand) + for c in ("/opt/data/.local/bin/uteke",): + if Path(c).exists(): + return str(c) + return shutil.which("uteke") or "uteke" + + +def uteke(binary: str, store: Path, namespace: str, args: list[str]) -> dict | list: + cmd = [ + binary, + "--store", str(store), + "--namespace", namespace, + "--json", + *args, + ] + result = subprocess.run(cmd, capture_output=True, text=True, timeout=600) + if result.returncode != 0: + raise RuntimeError(f"uteke {' '.join(args)} failed: {result.stderr[:400]}") + return json.loads(result.stdout) + + +def remember(binary, store, ns, content: str) -> str: + out = uteke(binary, store, ns, ["remember", content]) + return str(out["id"]) # type: ignore[no-any-return] + + +def supersede(binary, store, ns, old: str, new: str) -> None: + uteke(binary, store, ns, ["supersede", old, new, "--reason", "benchmark conflict resolution"]) + + +def recall_ids(binary, store, ns, query: str, strategy: str, k: int) -> list[str]: + out = uteke(binary, store, ns, [ + "recall", query, + "--limit", str(k), + "--min", "0.0", + "--strategy", strategy, + ]) + return [m["memory_id"] for m in out] + + +def recall_map(binary, store, ns) -> dict[str, str]: + """id → content for the namespace (to map ids back to roles).""" + out = uteke(binary, store, ns, ["list", "--limit", "500"]) + return {m["id"]: m["content"] for m in out} + + +def metrics_at(ranking: list[str], winner: str, stale: str) -> tuple[float, float, float, float, float]: + w = lambda k: 1.0 if winner in ranking[:k] else 0.0 + s = lambda k: 1.0 if stale in ranking[:k] else 0.0 + rr = 0.0 + for i, mid in enumerate(ranking, start=1): + if mid == winner: + rr = 1.0 / i + break + return w(1), w(5), rr, s(1), s(5) + + +def main() -> int: + ap = argparse.ArgumentParser(description="#1172 F3 contradiction segment") + ap.add_argument("--binary", default="", help="uteke binary path") + ap.add_argument("--store", default="", help="existing store to reuse (default: temp)") + ap.add_argument("--namespace", default="bench-contradiction") + ap.add_argument("--distractors", type=int, default=3, help="distractors per topic") + ap.add_argument("--topics", type=int, default=0, help="limit topics (0 = all)") + ap.add_argument("--strategy", default="fusion") + ap.add_argument("--json", default="", help="write metrics JSON here") + args = ap.parse_args() + + topics = TOPICS[: args.topics] if args.topics > 0 else TOPICS + binary = resolve_binary(args.binary) + print(f"binary: {binary}") + + tmp = None + if args.store: + store = Path(args.store) + else: + tmp = tempfile.TemporaryDirectory(prefix="uteke-contradiction-") + store = Path(tmp.name) / "bench.uteke" + + ns = args.namespace + roles: dict[str, tuple[str, str]] = {} # winner id → (stale id, topic) + topic_thing = {t: th for t, th, _o, _n in topics} + + # ── Seed ──────────────────────────────────────────────────────────── + print(f"seeding {len(topics)} topics (+{args.distractors} distractors each)…") + distractor_pool = [ + "weekly sync notes and standup summaries", + "onboarding checklist for new engineers", + "retro action items from the last sprint", + "vendor invoice and billing contacts", + "conference talk notes and takeaways", + ] + for topic, thing, old_tool, new_tool in topics: + stale = remember(binary, store, ns, + f"{topic} uses {old_tool} for {thing}. Decision recorded after evaluation.") + winner = remember(binary, store, ns, + f"{topic} switched to {new_tool} for {thing}. The old {old_tool} setup is retired.") + roles[winner] = (stale, topic) + for d in range(args.distractors): + remember(binary, store, ns, + f"{topic} {distractor_pool[d % len(distractor_pool)]} ({thing} context {d})") + + strategies = [s.strip() for s in args.strategy.split(",") if s.strip()] + results: dict[str, dict] = {} + + def measure(strategy: str) -> dict: + w1s = w5s = rrs = ss1 = ss5 = 0.0 + n = 0 + for winner, (stale, topic) in roles.items(): + q = QUESTION.format(thing=topic_thing[topic], topic=topic) + ranking = recall_ids(binary, store, ns, q, strategy, 10) + w1, w5, rr, s1, s5 = metrics_at(ranking, winner, stale) + w1s += w1; w5s += w5; rrs += rr; ss1 += s1; ss5 += s5 + n += 1 + return { + "winner@1": w1s / n, "winner@5": w5s / n, "winner_mrr": rrs / n, + "stale@1": ss1 / n, "stale@5": ss5 / n, "n": n, + } + + # ── Baseline for ALL strategies on the UNRESOLVED store first ────── + # (code-scanning fix: resolving per-strategy left later strategies + # measuring "baseline" on an already-resolved store.) + baselines = {s: measure(s) for s in strategies} + + # ── Resolve once: supersede every stale → winner ─────────────────── + for winner, (stale, _topic) in roles.items(): + supersede(binary, store, ns, stale, winner) + + # Ledger sanity: every resolution is listed (F2 surface, in-loop). + ledger_raw = subprocess.run( + [binary, "--store", str(store), "--json", + "contradictions", "list", "--namespace", ns, "--limit", "500"], + capture_output=True, text=True, timeout=600, + ) + ledger = json.loads(ledger_raw.stdout) + listed = {e["id"] for e in ledger} + ledger_ok = all(stale in listed for _w, (stale, _t) in roles.items()) + + # ── Resolved metrics for all strategies ──────────────────────────── + resolved = {s: measure(s) for s in strategies} + + for strategy in strategies: + baseline = {k: v for k, v in baselines[strategy].items() if k != "n"} + res = {k: v for k, v in resolved[strategy].items() if k != "n"} + n = baselines[strategy]["n"] + results[strategy] = { + "questions": n, + "baseline_unresolved": baseline, + "resolved": res, + "ledger_listed_all": ledger_ok, + } + print(f"\n[{strategy}] n={n}") + print(f" baseline (unresolved): winner@1={baseline['winner@1']:.3f} " + f"winner@5={baseline['winner@5']:.3f} MRR={baseline['winner_mrr']:.3f} " + f"stale@1={baseline['stale@1']:.3f} stale@5={baseline['stale@5']:.3f}") + print(f" resolved (superseded): winner@1={res['winner@1']:.3f} " + f"winner@5={res['winner@5']:.3f} MRR={res['winner_mrr']:.3f} " + f"stale@1={res['stale@1']:.3f} stale@5={res['stale@5']:.3f}") + print(f" ledger lists all resolutions: {ledger_ok}") + + if args.json: + out = Path(args.json) + out.parent.mkdir(parents=True, exist_ok=True) + out.write_text(json.dumps(results, indent=2)) + print(f"\nmetrics → {out}") + + if tmp: + tmp.cleanup() + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/crates/uteke-cli/src/cli.rs b/crates/uteke-cli/src/cli.rs index cbd6f08e..f93dc887 100644 --- a/crates/uteke-cli/src/cli.rs +++ b/crates/uteke-cli/src/cli.rs @@ -468,6 +468,16 @@ pub enum Commands { /// Memory ID (UUID) id: String, }, + /// Resolve a conflict: mark old_id superseded by new_id (#1053) + Supersede { + /// Full UUID or unambiguous prefix of the STALE memory + old: String, + /// Full UUID or unambiguous prefix of the CURRENT memory + new: String, + /// Why it was superseded (stored on the deprecation) + #[arg(long)] + reason: Option, + }, /// Inspect the contradiction resolution ledger (#1172) Contradictions { #[command(subcommand)] diff --git a/crates/uteke-cli/src/commands/contradictions.rs b/crates/uteke-cli/src/commands/contradictions.rs index c92f3ba9..70057b8b 100644 --- a/crates/uteke-cli/src/commands/contradictions.rs +++ b/crates/uteke-cli/src/commands/contradictions.rs @@ -81,3 +81,50 @@ pub(crate) fn run(cli: &Cli, uteke: &Uteke, command: &ContradictionCommands) -> fn short_id(id: &str) -> String { id.chars().take(8).collect() } + +/// `uteke supersede old new [--reason text]` (#1053) — CLI parity with the +/// MCP tool and HTTP surface. Accepts full UUIDs or unambiguous prefixes. +pub(crate) fn supersede( + cli: &Cli, + uteke: &Uteke, + old: &str, + new: &str, + reason: Option<&str>, +) -> Result<(), String> { + tracing::info!("Superseding {old} -> {new}"); + let resolve = |id: &str| -> Result { + if id.len() == 36 { + return Ok(id.to_string()); + } + match uteke + .resolve_id_prefix(id) + .map_err(|e| format!("Failed to resolve id: {e}"))? + { + Some(full) => Ok(full), + None => Err(format!("No memory matches id prefix '{id}'")), + } + }; + let old_id = resolve(old)?; + let new_id = resolve(new)?; + + let (o, n) = uteke + .supersede(&old_id, &new_id, reason) + .map_err(|e| format!("Failed to supersede: {e}"))?; + if cli.json { + output::print_json(&serde_json::json!({ + "superseded": o, + "by": n, + "reason": reason, + })); + } else { + println!("✓ Superseded {} → {}", short_id(&o), short_id(&n)); + if let Some(r) = reason { + println!(" reason: {r}"); + } + println!( + " recall now flags the pair; restore: uteke contradictions undo {}", + short_id(&o) + ); + } + Ok(()) +} diff --git a/crates/uteke-cli/src/commands/mod.rs b/crates/uteke-cli/src/commands/mod.rs index 7b911722..83943dc7 100644 --- a/crates/uteke-cli/src/commands/mod.rs +++ b/crates/uteke-cli/src/commands/mod.rs @@ -391,6 +391,10 @@ pub(crate) fn run_command(cli: &Cli, uteke: &mut Uteke, config: &Config) -> Resu crate::commands::contradictions::run(cli, uteke, command) } + Commands::Supersede { old, new, reason } => { + crate::commands::contradictions::supersede(cli, uteke, old, new, reason.as_deref()) + } + Commands::Provenance { id } => { let report = uteke .provenance(id) diff --git a/docs/cli-reference.md b/docs/cli-reference.md index 723e0148..5da24a3d 100644 --- a/docs/cli-reference.md +++ b/docs/cli-reference.md @@ -284,6 +284,16 @@ Hash verdicts: `✓ matches` (content unchanged since write), `✗ MISMATCH` (content modified after write — investigate), `—` (memory predates schema v18; hash will be recorded on next content update). +## uteke supersede + +Resolve a conflict: mark the old memory superseded by a newer one (#1053) — +wires the supersession edge pair, soft-deprecates the old memory, and records +the resolution in the contradiction ledger (#1172). + +```bash +uteke supersede --reason "decision pivot" +``` + ## uteke contradictions Inspect the contradiction resolution ledger (#1172) — memories that were