Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 54 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -236,11 +236,63 @@ jobs:
print("smoke ok", d["metric"], "questions", d["total_questions"])
PY

bench-cli-smoke:
name: Benchmark CLI Isolation Smoke
runs-on: ubuntu-24.04
needs: [build]
steps:
- uses: actions/checkout@v4

- name: Install system dependencies
run: |
sudo apt-get update
sudo apt-get install -y protobuf-compiler libclang-dev

- name: Install Rust
uses: dtolnay/rust-toolchain@master
with:
toolchain: "1.97"

- name: Cache cargo registry
uses: Swatinem/rust-cache@v2
with:
shared-key: "bench-cli-smoke"

- name: Build daemon, CLI, and bench
run: cargo build -p memory-daemon -p memory-cli -p memory-bench

- name: Live-backend locomo smoke (spawn-per-conversation)
run: |
export PATH="$PWD/target/debug:$PATH"
cargo run -p memory-bench -- locomo \
--dataset benchmarks/fixtures/locomo-smoke.json \
--backend cli \
--scorer mock \
--isolation daemon-per-conversation \
--output /tmp/locomo-cli-smoke.json
python3 - <<'PY'
import json
d = json.load(open("/tmp/locomo-cli-smoke.json"))
assert d["conversations"] == 1, d
assert d["isolation"] == "per-conversation daemon", d
assert d["metric"] == "context_hit_rate", d
conv = d["per_conversation"][0]
assert "drain_wait_ms" in conv, conv
assert conv["drain_wait_ms"] >= 0
print("cli smoke ok", d["isolation"], "drain_wait_ms", conv["drain_wait_ms"])
PY

- name: Isolation bleed test
env:
MEMORY_BENCH_LIVE: "1"
run: |
export PATH="$PWD/target/debug:$PATH"
cargo test -p memory-bench cli_isolated_daemons_do_not_bleed -- --nocapture

# Summary job that depends on all other jobs
ci-success:
name: CI Success
needs: [fmt, clippy, test, build, doc, e2e, benchmark-smoke, release-guards]
needs: [fmt, clippy, test, build, doc, e2e, benchmark-smoke, bench-cli-smoke, release-guards]
runs-on: ubuntu-24.04
if: always()
steps:
Expand All @@ -253,6 +305,7 @@ jobs:
[[ "${{ needs.doc.result }}" != "success" ]] || \
[[ "${{ needs.e2e.result }}" != "success" ]] || \
[[ "${{ needs.benchmark-smoke.result }}" != "success" ]] || \
[[ "${{ needs.bench-cli-smoke.result }}" != "success" ]] || \
[[ "${{ needs.release-guards.result }}" != "success" ]]; then
echo "One or more jobs failed"
exit 1
Expand Down
9 changes: 5 additions & 4 deletions .planning/MILESTONES.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,12 @@ and a repo whose backlog is public.

**Spec:** `docs/plans/v3.2-prove-it-plan.md`

**Phases:** 59 Guardrails and Inventory (executing), 60 Real Numbers, 61
Operate It, 62 Cross-encoder rerank (conditional on #39).
**Phases:** 59 Guardrails and Inventory (complete #45), 60 Real Numbers
(60-01 executing), 61 Operate It, 62 Cross-encoder rerank (conditional on #39).

**Known Gaps (now issues):** #39 LOCOMO run, #40 vector/topic quality, #41
backfill, #42 install-service, #43 TOC rebuild, #44 cross-encoder.
**Known Gaps (issues labelled `v3.2`):** #39 LOCOMO run, #40 vector quality,
#47 topic quality, #41 backfill, #42 install-service, #43 TOC rebuild,
#48 uninstall/status, #44 cross-encoder.

---

Expand Down
8 changes: 4 additions & 4 deletions .planning/PROJECT.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,9 @@ them provable.
**Target work:**
- Release pipeline guards (tag on main, crate version matches tag, all five platforms) — Phase 59
- Committed LOCOMO LLM-judge result on the real dataset — Phase 60 / #39
- Quality fixtures for vector search and the topic graph — Phase 60 / #40
- Quality fixtures for vector search (#40) and the topic graph (#47) — Phase 60
- Backfill, `install-service`, offline TOC rebuild, panic audit — Phase 61 / #41 #42 #43
- Claude Code plugin registration + installer uninstall/status — Phase 61
- Claude Code plugin registration + installer uninstall/status — Phase 61 / #48
- Cross-encoder rerank only if 60-02 says retrieval is the bottleneck — Phase 62 / #44

**Previous version:** v3.1.0 (Shipped 2026-09-01) — Make It True. No new
Expand All @@ -37,8 +37,8 @@ The system implements a complete 6-layer cognitive stack with control plane, mul
- Layer 6: Ranking Policy (salience, usage, novelty, lifecycle) + StaleFilter (time-decay, supersession)
- Control: Retrieval Policy (intent routing, tier detection, fallbacks) + MemoryOrchestrator (RRF fusion, optional LLM rerank, explainability)
- Dedup: InFlightBuffer + HNSW composite gate, configurable threshold, fail-open
- Installer: memory-installer crate with RuntimeConverter trait, 5 converters (Claude, Gemini, Codex, Copilot, generic skills), tool mapping tables
- Adapters: Claude Code, Gemini CLI, Copilot CLI, Codex CLI (via installer). OpenCode removed in v3.1 Phase 57 — the converter reported success and wrote nothing
- Installer: memory-installer crate with RuntimeConverter trait, converters for Claude, Gemini, Codex, Copilot, generic skills; tool mapping tables
- Adapters: Claude Code, Gemini CLI, Copilot CLI, Codex CLI (via installer). Supported surfaces are those four; registration for Gemini/Codex/Copilot is v3.3+
- Discovery: ListAgents, GetAgentActivity, agent-filtered topics
- Testing: 1,205 workspace + 60 e2e cargo tests; 114 bats CLI tests; Tier 1 (Claude Code, Codex) gates PRs, Tier 2 (Gemini, Copilot) weekly
- CI/CD: Dedicated E2E job + CLI matrix report; rust-toolchain pinned to 1.97
Expand Down
116 changes: 81 additions & 35 deletions .planning/REQUIREMENTS.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Requirements: Agent Memory v3.0
# Requirements: Agent Memory

**Defined:** 2026-03-22
**Defined:** 2026-03-22 (v3.0); extended 2026-09-01 (v3.2)
**Core Value:** Agent can answer "what were we talking about last week?" without scanning everything

## v3.0 Requirements
Expand Down Expand Up @@ -42,59 +42,105 @@ Requirements for the Competitive Parity & Benchmarks milestone. Each maps to roa
- [x] **BENCH-07**: CI runs benchmark suite (non-blocking, skips LOCOMO without `--dataset` flag)
- [x] **BENCH-08**: JSON + markdown report output for all benchmark types

## Future Requirements (v3.1+)
## v3.2 Requirements (Prove It)

- **ORCH-F01**: Cross-encoder reranking (requires new inference path in memory-embeddings)
Requirements for making v3.1's claims provable and operable. Each maps to
exactly one plan in `docs/plans/v3.2-prove-it-plan.md`.

### Release pipeline (REL)

- [x] **REL-01**: Tagged commit must be an ancestor of `origin/main` (59-01)
- [x] **REL-02**: `workspace.package.version` must equal the tag minus `v` (59-01)
- [x] **REL-03**: Any failed platform build → no GitHub release (59-01)
- [x] **REL-04**: Release body is the matching CHANGELOG section (59-01)

### Benchmarks (BENCH)

- [x] **BENCH-10**: Per-conversation isolation on `--backend cli` (60-01)
- [x] **BENCH-11**: Deterministic drain wait (poll checkpoints, no blind sleep) (60-01)
- [ ] **BENCH-12**: Committed `locomo_llm_judge` full-dataset result (60-02 / #39)
- [ ] **BENCH-13**: Layer switch `bm25|vector|hybrid` on the custom harness (60-03)

### Quality evidence (QUAL)

- [ ] **QUAL-01**: Semantic fixture set, ≥15 tests (60-03 / #40)
- [ ] **QUAL-02**: Topic clustering purity + ARI artifact (60-03 / #47)
- [ ] **QUAL-03**: README "Solid" rows cite committed artifacts (60-03)

### Operate it (OPS)

- [ ] **OPS-01**: `admin backfill-index` resumable, idempotent (61-01 / #41)
- [ ] **OPS-02**: `install-service`/`uninstall-service` macOS+Linux (61-02 / #42)
- [ ] **OPS-03**: Zero fallible `unwrap` on request paths (61-03)
- [ ] **OPS-04**: Hostile-input e2e over all RPCs (61-03)
- [ ] **OPS-05**: `admin rebuild-toc` real (61-04 / #43)

### Installer (INST)

- [ ] **INST-01**: Claude Code plugin registration (CREG/META) (61-05)
- [ ] **INST-02**: `memory-installer uninstall` (61-05 / #48)
- [ ] **INST-03**: `memory-installer status` (61-05 / #48)

## Future Requirements (v3.3+)

- **ORCH-F01**: Cross-encoder reranking — Phase 62 *if* 60-02 shows retrieval is the bottleneck (#44)
- **CLI-F01**: REST/HTTP endpoint wrapping CLI commands
- **CLI-F02**: Python SDK wrapping CLI binary
- **BENCH-F01**: Continuous benchmark regression tracking in CI
- **REG-F01**: Gemini/Codex/Copilot plugin registration
- **INST-F01**: `--for all` / `--all` installer flags
- **OPS-F01**: Windows service install; true double-fork daemonization

## Out of Scope

| Feature | Reason |
|---------|--------|
| REST/HTTP endpoint | Future milestone — CLI-first for v3.0 |
| REST/HTTP endpoint | Future milestone — CLI-first |
| Python SDK | Future milestone — wraps CLI |
| Memory views UI | Future milestone |
| Cross-encoder reranking | Requires new inference path in memory-embeddings; extension point only |
| Cross-encoder reranking | Conditional Phase 62; extension point only until then |
| Multi-agent shared memory changes | Shipped in v2.1 |

## Traceability

### v3.0 (complete)

| Requirement | Phase | Status |
|-------------|-------|--------|
| ORCH-01 | Phase 51 | Complete |
| ORCH-02 | Phase 51 | Complete |
| ORCH-03 | Phase 51 | Complete |
| ORCH-04 | Phase 51 | Complete |
| ORCH-05 | Phase 51 | Complete |
| ORCH-06 | Phase 51 | Complete |
| ORCH-07 | Phase 51 | Complete |
| ORCH-08 | Phase 51 | Complete |
| CLI-01 | Phase 52 | Complete |
| CLI-02 | Phase 52 | Complete |
| CLI-03 | Phase 52 | Complete |
| CLI-04 | Phase 52 | Complete |
| CLI-05 | Phase 52 | Complete |
| CLI-06 | Phase 52 | Complete |
| CLI-07 | Phase 52 | Complete |
| CLI-08 | Phase 52 | Complete |
| CLI-09 | Phase 52 | Complete |
| CLI-10 | Phase 52 | Complete |
| BENCH-01 | Phase 53 | Complete |
| BENCH-02 | Phase 53 | Complete |
| BENCH-03 | Phase 53 | Complete |
| BENCH-04 | Phase 53 | Complete |
| BENCH-05 | Phase 53 | Complete |
| BENCH-06 | Phase 53 | Complete |
| BENCH-07 | Phase 53 | Complete |
| BENCH-08 | Phase 53 | Complete |
| ORCH-01..08 | Phase 51 | Complete |
| CLI-01..10 | Phase 52 | Complete |
| BENCH-01..08 | Phase 53 | Complete |

### v3.2

| Requirement | Plan | Status |
|-------------|------|--------|
| REL-01 | 59-01 | Complete (#45) |
| REL-02 | 59-01 | Complete (#45) |
| REL-03 | 59-01 | Complete (#45) |
| REL-04 | 59-01 | Complete (#45) |
| BENCH-10 | 60-01 | In progress |
| BENCH-11 | 60-01 | In progress |
| BENCH-12 | 60-02 | Open (#39) |
| BENCH-13 | 60-03 | Open (#40) |
| QUAL-01 | 60-03 | Open (#40) |
| QUAL-02 | 60-03 | Open (#47) |
| QUAL-03 | 60-03 | Open |
| OPS-01 | 61-01 | Open (#41) |
| OPS-02 | 61-02 | Open (#42) |
| OPS-03 | 61-03 | Open |
| OPS-04 | 61-03 | Open |
| OPS-05 | 61-04 | Open (#43) |
| INST-01 | 61-05 | Open |
| INST-02 | 61-05 | Open (#48) |
| INST-03 | 61-05 | Open (#48) |

**Coverage:**
- v3.0 requirements: 26 total
- Mapped to phases: 26
- v3.0 requirements: 26 total, all complete
- v3.2 requirements: 19 total, 4 complete (REL), 15 open
- Unmapped: 0 ✓

---
*Requirements defined: 2026-03-22*
*Last updated: 2026-03-22 after spec review*
*Last updated: 2026-09-01 — v3.2 IDs added from the adopted Prove It plan*

20 changes: 10 additions & 10 deletions .planning/ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
- ✅ **v2.7 Multi-Runtime Portability** — Phases 45-50 (shipped 2026-03-22)
- **v3.0 Competitive Parity & Benchmarks** — Phases 51-53 + Phase 51.5 (shipped 2026-05-14; Phase 53 merged as #30)
- ✅ **v3.1 Make It True** — Phases 54-58 (shipped 2026-09-01 as v3.1.0)
- **v3.2 Prove It** — Phases 59-62 (in progress; Phase 59 executing 2026-09-01)
- **v3.2 Prove It** — Phases 59-62 (Phase 59 complete 2026-09-01; Phase 60 executing)

## Phases

Expand Down Expand Up @@ -230,7 +230,7 @@ Phases execute in numeric order: 51 -> 51.5 (merged out-of-band) -> 52 -> 53
| v2.7 Multi-Runtime Portability | 45-50 | 11/11 | Complete | 2026-03-22 |
| v3.0 Competitive Parity | 51-53 + 51.5, 53.5 | 10/10 | Complete | 2026-05-14 |
| v3.1 Make It True | 54-58 | 14/14 | Complete | 2026-09-01 |
| v3.2 Prove It | 59-62 | 3/13 | In progress | Phase 59 executing |
| v3.2 Prove It | 59-62 | 3/13 | In progress | Phase 59 complete; 60-01 executing |

---

Expand Down Expand Up @@ -315,29 +315,29 @@ v3.1 made the claims true. v3.2 makes them provable: a real benchmark number,
evidence behind every "Solid", and a daemon someone can run for a week.
Nothing on this list is a new capability except Phase 62, which is conditional.

### Phase 59: Guardrails and Inventory (3/3 plans) — IN EXECUTION 2026-09-01
### Phase 59: Guardrails and Inventory (3/3 plans) — COMPLETE 2026-09-01 (#45)

- [x] 59-01: Release pipeline checks (ancestor of main, crate version, all five platforms, CHANGELOG notes)
- [x] 59-02: Orphan branch triage (`docs/plans/phase-59-orphan-branch-triage.md`)
- [x] 59-03: Planning truth (PROJECT.md, ROADMAP, STATE, GitHub issues #39–#44)
- [x] 59-02: Orphan branch triage (`docs/plans/phase-59-orphan-branch-triage.md`); OpenCode branch deleted
- [x] 59-03: Planning truth (PROJECT.md, ROADMAP, STATE, 8 GitHub issues labelled `v3.2`)

### Phase 60: Real Numbers (0/3)
### Phase 60: Real Numbers (0/3) — 60-01 executing

- [ ] 60-01: Live-backend isolation for `memory-bench locomo --backend cli`
- [ ] 60-01: Live-backend isolation for `memory-bench locomo --backend cli` (this PR)
- [ ] 60-02: The run — maintainer, needs API key + documented machine (#39)
- [ ] 60-03: Vector and topic quality fixtures (#40)
- [ ] 60-03: Vector (#40) and topic (#47) quality fixtures

### Phase 61: Operate It (0/5)

- [ ] 61-01: Backfill (`admin backfill-index`) (#41); cherry-pick export/import from 59-02
- [ ] 61-02: `install-service` launchd/systemd (#42)
- [ ] 61-03: Panic audit (`unwrap()`/`expect()` on request paths)
- [ ] 61-04: Offline TOC rebuild (#43)
- [ ] 61-05: Installer register / uninstall / status (CREG/META from 59-02)
- [ ] 61-05: Installer register / uninstall / status (CREG/META from 59-02; #48)

### Phase 62: Cross-encoder rerank (conditional) (#44)

- [ ] Only if 60-02 shows retrieval, not generation, is the bottleneck

*Updated: 2026-09-01v3.1.0 released; v3.2 Prove It adopted; Phase 59 executing*
*Updated: 2026-09-02Phase 59 complete (#45); expanded v3.2 plan adopted; 60-01 executing*

26 changes: 15 additions & 11 deletions .planning/STATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,11 @@ gsd_state_version: 1.0
milestone_name: Prove It
status: executing
stopped_at: null
last_updated: "2026-09-01T23:30:00.000Z"
last_activity: 2026-09-01v3.1.0 released; v3.2 adopted; Phase 59 Guardrails and Inventory executing
last_updated: "2026-09-02T01:40:00.000Z"
last_activity: 2026-09-02Phase 60-01 live-backend isolation verified (spawn-per-conversation, checkpoint drain, CLI smoke)
progress:
total_phases: 4
completed_phases: 0
completed_phases: 1
total_plans: 13
completed_plans: 3
percent: 23
Expand All @@ -20,39 +20,42 @@ progress:
See: .planning/PROJECT.md (updated 2026-09-01)

**Core value:** Agent can answer "what were we talking about last week?" without scanning everything
**Current focus:** v3.2 Phase 59 — Guardrails and Inventory. Release pipeline cannot repeat the stale-tag incident; March gsd/ line inventoried; planning docs match v3.1.0 shipped.
**Current focus:** v3.2 Phase 60-01 live-backend isolation ready to merge. Phase 59 Guardrails and Inventory is on `main` (#45).

## Current Position

Phase: 59 of 62 (Guardrails and Inventory)
Status: v3.1.0 shipped 2026-09-01 (5 of 5 platforms). v3.2 Prove It adopted. Phase 59 in execution.
Last activity: 2026-09-01issues #39–#44 opened; release guards + orphan triage + PROJECT.md rewrite
Phase: 60 of 62 (Real Numbers) — plan 60-01 verified locally
Status: v3.1.0 shipped 2026-09-01 (5 of 5 platforms). v3.2 Prove It adopted (expanded spec). Phase 59 complete.
Last activity: 2026-09-0260-01 spawn-per-conversation + GetIndexCheckpoints drain + live CLI smoke + isolation bleed test

Progress: [██░░░░░░░] 3/13 plans (Phase 59). Phases 60–62 not started.
Progress: [██░░░░░░░] 3/13 plans (Phase 59 complete). Phase 60-01 verified, awaiting merge.

## Out-of-band Work

### Open PRs

| PR | What | Status |
|---|---|---|
| _(this branch)_ | Phase 59 Guardrails and Inventory | Open |
| _(this branch)_ | Phase 60-01 live-backend isolation + remaining 59 gaps | Open |

### Open issues (the v3.2 backlog)

| Issue | What | Phase |
|---|---|---|
| #39 | Real LOCOMO LLM-judge run | 60-02 |
| #40 | Vector and topic-graph quality fixtures | 60-03 |
| #40 | Vector quality fixtures | 60-03 / QUAL-01 |
| #47 | Topic-graph quality (purity + ARI) | 60-03 / QUAL-02 |
| #41 | Backfill BM25/vector for pre-v3.1 events | 61-01 |
| #42 | `install-service` (launchd/systemd) | 61-02 |
| #43 | Offline TOC rebuild | 61-04 |
| #48 | Installer uninstall + status | 61-05 |
| #44 | Cross-encoder rerank (conditional) | 62 |

### Recently Merged

| PR | What | Merged |
|---|---|---|
| #45 | Phase 59 Guardrails and Inventory | 2026-09-01 |
| #38 | docs: correct the "no tags" claim and record the release blocker | 2026-08-31 |
| #37 | chore(v3.1): release prep — version 3.1.0, changelog, working release archives | 2026-08-31 |
| #36 | Phase 57 Shop Window & Positioning | 2026-08-31 |
Expand All @@ -65,7 +68,8 @@ Progress: [██░░░░░░░░] 3/13 plans (Phase 59). Phases 60–62
## Decisions

- v3.2 scope: Prove It — no new capabilities except conditional Phase 62
- Maintainer decisions 2026-09-01: cherry-pick March export/import + CREG/META by feature (not by branch); skip OpenCode converter; blog now / product posts after #39; daemonization is unit files not double-fork
- Maintainer decisions 2026-09-01 (all four accepted): cherry-pick March export/import + CREG/META by feature; skip OpenCode (branch deleted); blog now / product posts after #39; daemonization is unit files not double-fork; backfill is stopped-daemon CLI only
- Canonical spec: `docs/plans/v3.2-prove-it-plan.md` (expanded GSD form)
- v3.1.0 first tag push shipped `acc7294` (Cargo.toml 2.7.0) for 17 minutes — Phase 59-01 exists because of that
- March `gsd/phase-56-import-bootstrap` is 88 ahead / 20 behind; naïve merge regresses orchestrator and bench. Inventory: `docs/plans/phase-59-orphan-branch-triage.md`
- HOLD comparison marketing until #39 lands a `locomo_llm_judge` artifact
Loading
Loading