diff --git a/README.md b/README.md index 31ec150e..c11813c2 100644 --- a/README.md +++ b/README.md @@ -144,10 +144,8 @@ Adding another host means implementing one `TranscriptSource` (where its session ## Developer integration -Applications that already own their conversation history can use `memu memorize` -to prepare self-evolve jobs from 1–10 completed sessions for one external agent and -commit the resulting memory, skill, and resource changes. See [Developer integration](docs/developer.md) for the -canonical input contract and the complete prepare → agent → commit workflow. +Use `memu memorize` to turn application-owned conversations into memory and skills. +See the [developer guide](docs/developer.md) for the input format and integration workflow. ## CLI diff --git a/docs/adr/0019-developer-run-workspaces.md b/docs/adr/0019-developer-run-workspaces.md new file mode 100644 index 00000000..ed8dc682 --- /dev/null +++ b/docs/adr/0019-developer-run-workspaces.md @@ -0,0 +1,54 @@ +# 0019: Allocate a Private Workspace for Each Developer Memorize Run + +## Status + +Accepted. + +## Context + +Developer applications submit 1–10 canonical sessions through `memu memorize prepare`, +run an external executor, and commit once. A shared `~/.memu/developer` workspace +couples independent invocations: a pending or failed executor blocks the next run, +and callers cannot address each run independently. + +Related: https://github.com/MrXnneHang/xnnehang.top/issues/183 + +## Decision + +The developer CLI allocates one private directory per prepare invocation using +`tempfile.mkdtemp` below `~/.memu/developer/runs/`. Atomic allocation gives concurrent +prepares distinct paths. Its basename is the opaque run id returned with the +workspace, transcript paths, ordered jobs, executor prompt, and next command. +A batch of N sessions remains one run with 2N + 1 serial jobs. + +`commit `, internal `verify-resources `, and `discard ` +resolve only ids below that root. Path syntax and links redirecting the run +outside its allocated location are rejected. Callers cannot select arbitrary +workspace paths through the CLI. + +Each run retains the existing active marker and content snapshot. Backend commit +failure preserves the run for retry. Successful commit removes the entire run +directory. Explicit discard removes a stopped run without a backend call and can +also remove an incomplete run left by process termination. Ordinary prepare +failure cleans up its newly allocated directory; there is no TTL deletion. + +Allocation and complete-directory deletion belong to the CLI, which owns these +paths. The existing explicit-workspace Python lifecycle functions retain their +behavior, including leaving their caller-owned directory in place. Host adapter +prepare–commit workflows are unchanged. + +## Consequences + +- Applications persist the returned run id and address it explicitly on commit, + verification, or discard. Commands never implicitly select the latest run. +- Applications coordinate one executor per run and serialize operations on that + run. The active marker is not a process lock, and discard must follow executor + termination. +- Separate directories prevent filesystem interference but not backend write + conflicts. Runs updating overlapping RecallFiles must serialize the complete + prepare–evolve–commit cycle, or use disjoint ownership. Serializing commits + alone cannot refresh snapshots created by earlier prepares. +- The CLI change is breaking for callers using an unqualified memorize commit or + verifier command. Complete any old fixed-workspace run with the previous CLI + before upgrading. Old working files are not automatically moved or deleted. +- Evolve-type selection and job granularity remain independent follow-up work. diff --git a/docs/adr/README.md b/docs/adr/README.md index ea967d68..90264089 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -18,3 +18,4 @@ - [0016: Client Event Reporting — One Envelope, a Spool by Default, Bounded Payloads](0016-client-event-reporting.md) - [0017: `config.env` Is Written by a Command — `init` for the Entry, `config` for the Detail](0017-config-env-as-a-command.md) - [0018: Mine Claude Cowork Through the Claude Code Bridge](0018-cowork-through-claude-code-bridge.md) +- [0019: Allocate a Private Workspace for Each Developer Memorize Run](0019-developer-run-workspaces.md) diff --git a/docs/developer.md b/docs/developer.md index f46091f6..9a7a905c 100644 --- a/docs/developer.md +++ b/docs/developer.md @@ -10,16 +10,16 @@ This is an application integration contract. It starts with a completed session completed application session → memu memorize prepare → one external executor processes all jobs serially - → memu memorize commit + → memu memorize commit → configured Local or Cloud backend ``` `prepare` and `commit` are the deterministic parts of the lifecycle. The middle step is real agent work: the executor reads the session, compares it with existing memory and skill files, and makes create, patch, or no-op decisions. -memU uses the fixed working directory `~/.memu/developer`. Version 1.0 permits one active developer run at a time: +Each `prepare` invocation allocates a private directory under `~/.memu/developer/runs/` and returns an opaque `run_id`. Its 1–10 sessions belong to that single run. Concurrent prepares receive distinct directories: ```text -~/.memu/developer/ +~/.memu/developer/runs// ├── input/ projected session transcripts ├── jobs/ numbered executor instructions ├── memory/ writable mirror of memory RecallFiles @@ -145,8 +145,8 @@ Use `-` instead of a file path to read one payload from stdin; stdin cannot be c `prepare` performs the following work before returning: -1. validates every canonical payload before opening the run; -2. writes numbered message-only and full JSONL projections; +1. validates every canonical payload before allocating a private run directory; +2. writes numbered message-only and full JSONL projections into that directory; 3. lists the current RecallFiles from the configured backend and writes them into the workspace's `memory/` and `skill/` directories; 4. snapshots the working copies by content hash; 5. creates all memory jobs, then all skill jobs, then one resource job and the active-run marker. One session creates three jobs; ten sessions create 21. @@ -155,32 +155,34 @@ A successful JSON response for the two-session command above has this shape: ```json { - "workspace": "/home/alice/.memu/developer", + "run_id": "run-a1b2c3d4", + "workspace": "/home/alice/.memu/developer/runs/run-a1b2c3d4", "transcripts": [ { - "memory_path": "/home/alice/.memu/developer/input/1.jsonl", - "skill_path": "/home/alice/.memu/developer/input/1_full.jsonl" + "memory_path": "/home/alice/.memu/developer/runs/run-a1b2c3d4/input/1.jsonl", + "skill_path": "/home/alice/.memu/developer/runs/run-a1b2c3d4/input/1_full.jsonl" }, { - "memory_path": "/home/alice/.memu/developer/input/2.jsonl", - "skill_path": "/home/alice/.memu/developer/input/2_full.jsonl" + "memory_path": "/home/alice/.memu/developer/runs/run-a1b2c3d4/input/2.jsonl", + "skill_path": "/home/alice/.memu/developer/runs/run-a1b2c3d4/input/2_full.jsonl" } ], "jobs": [ - "/home/alice/.memu/developer/jobs/1.txt", - "/home/alice/.memu/developer/jobs/2.txt", - "/home/alice/.memu/developer/jobs/3.txt", - "/home/alice/.memu/developer/jobs/4.txt", - "/home/alice/.memu/developer/jobs/5.txt" + "/home/alice/.memu/developer/runs/run-a1b2c3d4/jobs/1.txt", + "/home/alice/.memu/developer/runs/run-a1b2c3d4/jobs/2.txt", + "/home/alice/.memu/developer/runs/run-a1b2c3d4/jobs/3.txt", + "/home/alice/.memu/developer/runs/run-a1b2c3d4/jobs/4.txt", + "/home/alice/.memu/developer/runs/run-a1b2c3d4/jobs/5.txt" ], - "executor_prompt": "Process this prepared memU self-evolve run in one agent session.\nRead and carry out every job file below in the listed order:\n1. /home/alice/.memu/developer/jobs/1.txt\n2. /home/alice/.memu/developer/jobs/2.txt\n3. /home/alice/.memu/developer/jobs/3.txt\n4. /home/alice/.memu/developer/jobs/4.txt\n5. /home/alice/.memu/developer/jobs/5.txt\nRun one job at a time. Do not parallelize, skip, or reorder jobs. If any job fails, stop and report failure. Do not run `memu memorize commit`. Report success only after every job has completed.", - "next_command": "memu memorize commit" + "executor_prompt": "Process this prepared memU self-evolve run in one agent session.\nRead and carry out every job file below in the listed order:\n1. /home/alice/.memu/developer/runs/run-a1b2c3d4/jobs/1.txt\n2. /home/alice/.memu/developer/runs/run-a1b2c3d4/jobs/2.txt\n3. /home/alice/.memu/developer/runs/run-a1b2c3d4/jobs/3.txt\n4. /home/alice/.memu/developer/runs/run-a1b2c3d4/jobs/4.txt\n5. /home/alice/.memu/developer/runs/run-a1b2c3d4/jobs/5.txt\nRun one job at a time. Do not parallelize, skip, or reorder jobs. If any job fails, stop and report failure. Do not run `memu memorize commit`. Report success only after every job has completed.", + "next_command": "memu memorize commit run-a1b2c3d4" } ``` | Response field | Use | |---|---| -| `workspace` | Fixed memU developer workspace. | +| `run_id` | Opaque memU-allocated id used by commit, verify-resources, and discard. | +| `workspace` | Absolute path of this run's private working directory. | | `transcript` | Present for one input (file or stdin): one object containing `memory_path` and `skill_path`. | | `transcripts` | Present for 2–10 inputs: an array of those objects in CLI argument order. | | `jobs` | Authoritative execution order. | @@ -189,7 +191,7 @@ A successful JSON response for the two-session command above has this shape: Exactly one of `transcript` or `transcripts` is returned. These paths identify the materialized inputs referenced by the jobs; applications normally do not edit them. -Only one prepared run may be active. A second `prepare` is rejected until the current run commits. +Persist `run_id` and the handoff before starting the executor. Each prepare creates a new run; it does not resume or replace an existing run. Resource job instructions include `memu memorize verify-resources ` so verification targets that run's log. ## 3. Execute all evolve jobs @@ -233,6 +235,7 @@ committed = run_json([ "memu", "memorize", "commit", + prepared["run_id"], "--json", ]) ``` @@ -244,16 +247,14 @@ The executor API and process isolation are application choices. memU defines the After the executor reports success, run `next_command` once. Add `--json` when a machine-readable result is required: ```bash -memu memorize commit --json +memu memorize commit run-a1b2c3d4 --json ``` `commit` hashes the workspace's `memory/` and `skill/` files against the pre-evolve snapshot, reads successfully described resources, and submits the resulting records through the configured backend. The response contains `recall_files` and `resources`; either list may be empty after a valid no-op run. On success, memU: -- updates the workspace snapshot; -- removes the projected input, numbered jobs, resource files, and active marker; -- leaves the `memory/` and `skill/` mirrors on disk; +- removes the complete run directory, including the working mirrors and any executor-created temporary files; - makes committed RecallFiles available to normal `list-files` and `retrieve` calls. Only newly created or content-modified files are submitted. File deletion is not part of the v1 commit contract. @@ -262,25 +263,37 @@ Only newly created or content-modified files are submitted. File deletion is not | State | Evidence | Application action | |---|---|---| -| Ready | No `.memorize_run.json` | Call `prepare` with 1–10 canonical sessions. | -| Prepared | Active marker and ordered jobs exist | Start exactly one evolve executor. | -| Executing | Executor is processing the jobs | Do not call another `prepare` or `commit`. | +| Ready | Application has a completed batch | Call `prepare` with 1–10 canonical sessions. | +| Prepared | Returned run directory contains an active marker and ordered jobs | Start exactly one evolve executor for this run. | +| Executing | Executor is processing the jobs | Do not commit, discard, or start another executor for this run. | | Evolve succeeded | Executor completed all jobs | Run `next_command` once. | -| Evolve failed | Executor stopped before all jobs completed | Do not commit. The active run remains for inspection; version 1.0 has no discard command. | -| Commit failed | Command returned non-zero and the active marker remains | Preserve the workspace, fix the backend problem, and retry `commit`; do not repeat `prepare` or evolve. | -| Committed | Active marker and ephemeral run files are gone | The fixed workspace is ready for the next run. | +| Evolve failed | Executor stopped before all jobs completed | Preserve the run for inspection, or explicitly discard it after the executor has stopped. | +| Backend commit failed | Active marker, inputs, jobs, and edits remain | Fix the backend problem and retry `commit `; do not repeat prepare or evolve. | +| Committed / discarded | The run directory is gone | Other runs remain available. | -Partial job execution is not resumable in the developer v1 interface, and there is currently no abort command. Backend commit failure intentionally retains the evolved workspace for commit retry. +To abandon a run after stopping its executor: + +```bash +memu memorize discard run-a1b2c3d4 --json +``` + +Discard removes only that run directory and does not contact the backend. It also accepts incomplete runs left by a process termination during prepare. Ordinary prepare errors remove the newly allocated directory before returning; a hard process termination may leave a directory under `runs/` for inspection and explicit discard. Active runs are never deleted on a timer. + +Partial job execution is not resumable in the developer v1 interface. After executor failure, discard the stopped run and prepare the batch again if needed. Backend commit failure retains the evolved workspace for commit retry. If the CLI reports `committed, but cleanup failed`, the backend accepted the submission even if the active marker still exists. Do not retry commit: stop work on that run and use `discard ` for the remaining directory. This applies to snapshot refresh, working-file removal, and final directory removal failures; discard does not undo committed data. ## Consistency and concurrency -The fixed workspace permits only one active developer run, so applications must not start a second executor or `prepare` while its marker exists. It does not provide backend-level conflict resolution. +Independent prepares have isolated working directories. Within a run, the application must serialize executor work, verification, commit, and discard; the active marker is a commit/retry guard, not a process lock. Directory isolation does not provide backend-level conflict resolution. `prepare` reads the backend once and establishes the baseline for the run. The workspace is not refreshed again before `commit`. If another host or developer run changes the same `(track, name, user scope)` RecallFile during that window, the later successful commit may overwrite the earlier content. Version 1.0 has no ETag, base revision, three-way merge, or conflict copy. -Applications that may overlap with host bridging should serialize runs that can edit the same RecallFiles, or assign non-overlapping RecallFile ownership. +Applications that may overlap with other developer runs or host bridging should serialize the entire prepare → evolve → commit cycle for runs that can edit the same RecallFiles, or assign non-overlapping RecallFile ownership. Serializing only commits does not refresh an already-prepared run's baseline. + +Each new CLI run starts with a fresh backend mirror. The explicit-workspace Python functions retain their existing workspace behavior; callers of those functions continue to own directory lifecycle. + +## Upgrading from the fixed workspace -The prepare mirror is additive/overwriting: RecallFiles returned by the backend are written atomically into the workspace, but local files absent from the backend response are not deletion-synchronized. Applications should treat the backend—not a retained workspace directory—as the source of truth between runs. +`memu memorize commit` and `verify-resources` now require a run id. Applications must retain the returned `run_id` or execute the returned `next_command`; there is no implicit selection of the newest run. Finish an active run in `~/.memu/developer` with the previous CLI version before upgrading. Existing fixed-workspace files are not migrated or removed automatically. The per-run lifecycle applies to the developer CLI; host adapter prepare–commit commands keep their existing behavior. ## Responsibility boundary diff --git a/src/memu/app/memorize/lifecycle.py b/src/memu/app/memorize/lifecycle.py index 9e4aeba4..fde1c5aa 100644 --- a/src/memu/app/memorize/lifecycle.py +++ b/src/memu/app/memorize/lifecycle.py @@ -165,12 +165,19 @@ async def commit_memorize(workspace: MemorizeWorkspace, backend: AgenticMemoryBa resources = read_resources(workspace.resources) result = await backend.commit_results(recall_files=recall_files, resource=resources) - snapshot_tracked(workspace.base, workspace.track_dirs, workspace.manifest) - for stale in workspace.jobs.glob("*.txt"): - stale.unlink() - for stale in workspace.input.glob("*.jsonl"): - stale.unlink() - workspace.resource_log.unlink(missing_ok=True) - workspace.resources.unlink(missing_ok=True) - workspace.active_run.unlink() + try: + snapshot_tracked(workspace.base, workspace.track_dirs, workspace.manifest) + for stale in workspace.jobs.glob("*.txt"): + stale.unlink() + for stale in workspace.input.glob("*.jsonl"): + stale.unlink() + workspace.resource_log.unlink(missing_ok=True) + workspace.resources.unlink(missing_ok=True) + workspace.active_run.unlink() + except OSError as exc: + msg = ( + f"memorize run committed, but cleanup failed at {workspace.base}: {exc}; " + "do not resubmit; discard the stopped run" + ) + raise RuntimeError(msg) from exc return result diff --git a/src/memu/cli.py b/src/memu/cli.py index 596f5b99..dff61953 100644 --- a/src/memu/cli.py +++ b/src/memu/cli.py @@ -14,7 +14,8 @@ memu list-files memu commit results.json memu memorize prepare session.json - memu memorize commit + memu memorize commit + memu memorize discard """ from __future__ import annotations @@ -24,7 +25,10 @@ import json import os import pathlib +import re +import shutil import sys +import tempfile import time from collections.abc import Callable, Coroutine from typing import Any @@ -168,12 +172,27 @@ async def _cmd_commit(args: argparse.Namespace) -> int: return 0 -def _memorize_workspace() -> MemorizeWorkspace: - return MemorizeWorkspace(pathlib.Path(MEMORIZE_WORKSPACE).expanduser()) +def _memorize_workspace(run_id: str | None = None) -> MemorizeWorkspace: + """Allocate a private run, or resolve an existing id within the runs root.""" + root = (pathlib.Path(MEMORIZE_WORKSPACE).expanduser() / "runs").resolve() + if run_id is None: + root.mkdir(parents=True, exist_ok=True) + return MemorizeWorkspace(pathlib.Path(tempfile.mkdtemp(prefix="run-", dir=root))) + if not re.fullmatch(r"run-[a-z0-9_]{1,64}", run_id): + msg = "invalid memorize run id" + raise ValueError(msg) + path = root / run_id + if path.is_symlink() or path.resolve() != path: + msg = "memorize run must be a directory directly below the runs root" + raise ValueError(msg) + if not path.is_dir(): + msg = f"no such memorize run: {run_id}" + raise FileNotFoundError(msg) + return MemorizeWorkspace(path) -def _memorize_commit_command() -> str: - return "memu memorize commit" +def _memorize_commit_command(run_id: str) -> str: + return f"memu memorize commit {run_id}" def _memorize_executor_prompt(prepared: PreparedMemorizeRun) -> str: @@ -213,22 +232,25 @@ async def _cmd_memorize_prepare(args: argparse.Namespace) -> int: print(f"error: no such file: {pathlib.Path(payload).expanduser()}", file=sys.stderr) return 2 memorize_inputs = [_read_memorize_input(payload) for payload in payloads] + backend = _build_backend(args) workspace = _memorize_workspace() - verify_command = "memu memorize verify-resources" - prepared = await prepare_memorize( - memorize_inputs, - workspace, - _build_backend(args), - verify_command=verify_command, - ) + run_id = workspace.base.name + verify_command = f"memu memorize verify-resources {run_id}" + try: + prepared = await prepare_memorize(memorize_inputs, workspace, backend, verify_command=verify_command) + except BaseException: + # No executor has received this freshly allocated run yet. + shutil.rmtree(workspace.base) + raise executor_prompt = _memorize_executor_prompt(prepared) if args.json: response: dict[str, Any] = { + "run_id": run_id, "workspace": str(workspace.base), "jobs": [str(path) for path in prepared.jobs], "executor_prompt": executor_prompt, - "next_command": _memorize_commit_command(), + "next_command": _memorize_commit_command(run_id), } transcripts = [ {"memory_path": str(item.memory_path), "skill_path": str(item.skill_path)} for item in prepared.transcripts @@ -241,16 +263,23 @@ async def _cmd_memorize_prepare(args: argparse.Namespace) -> int: print("prepared developer session") print(f" {len(prepared.jobs)} job(s)") + print(f" run_id: {run_id}") print(f" workspace: {workspace.base}") print("run one external agent session with this prompt:") print(executor_prompt) print("after the agent reports success, run:") - print(f" {_memorize_commit_command()}") + print(f" {_memorize_commit_command(run_id)}") return 0 async def _cmd_memorize_commit(args: argparse.Namespace) -> int: - result = await commit_memorize(_memorize_workspace(), _build_backend(args)) + workspace = _memorize_workspace(args.run_id) + result = await commit_memorize(workspace, _build_backend(args)) + try: + shutil.rmtree(workspace.base) + except OSError as exc: + msg = f"run {args.run_id} committed, but cleanup failed; use memu memorize discard {args.run_id}: {exc}" + raise RuntimeError(msg) from exc if args.json: _print_json(result) return 0 @@ -264,12 +293,25 @@ async def _cmd_memorize_commit(args: argparse.Namespace) -> int: async def _cmd_memorize_verify_resources(args: argparse.Namespace) -> int: - workspace = _memorize_workspace() + workspace = _memorize_workspace(args.run_id) + if not workspace.active_run.is_file(): + msg = "memorize workspace has no active run" + raise RuntimeError(msg) kept = verify_resource_log(workspace.resource_log, workspace.resources) print(f"verified {kept} resource(s)") return 0 +async def _cmd_memorize_discard(args: argparse.Namespace) -> int: + workspace = _memorize_workspace(args.run_id) + shutil.rmtree(workspace.base) + if args.json: + _print_json({"run_id": args.run_id, "discarded": True}) + else: + print(f"discarded memorize run {args.run_id}") + return 0 + + def build_parser() -> argparse.ArgumentParser: parser = argparse.ArgumentParser( prog="memu", @@ -312,7 +354,8 @@ def build_parser() -> argparse.ArgumentParser: _add_common_options(p) p.set_defaults(handler=_cmd_memorize_prepare) - p = memorize_actions.add_parser("commit", help="Commit the active self-evolve run") + p = memorize_actions.add_parser("commit", help="Commit a prepared self-evolve run and remove its directory") + p.add_argument("run_id", help="Run id returned by prepare") _add_common_options(p) p.set_defaults(handler=_cmd_memorize_commit) @@ -320,8 +363,14 @@ def build_parser() -> argparse.ArgumentParser: "verify-resources", help="Internal: verify files logged by generated skill jobs", ) + p.add_argument("run_id", help="Run id returned by prepare") p.set_defaults(handler=_cmd_memorize_verify_resources) + p = memorize_actions.add_parser("discard", help="Remove a run after its executor has stopped, without committing") + p.add_argument("run_id", help="Run id returned by prepare") + p.add_argument("--json", action="store_true", help="Print the raw JSON response") + p.set_defaults(handler=_cmd_memorize_discard) + return parser diff --git a/tests/test_cli.py b/tests/test_cli.py index 132ecdcc..6161c1d9 100644 --- a/tests/test_cli.py +++ b/tests/test_cli.py @@ -26,7 +26,7 @@ def test_parser_covers_all_entry_points() -> None: ["list-files"], ["commit", "payload.json"], ["memorize", "prepare", "input.json"], - ["memorize", "commit"], + ["memorize", "commit", "run-test"], ): args = parser.parse_args(argv) assert callable(args.handler) diff --git a/tests/test_memorize_cli.py b/tests/test_memorize_cli.py index ff80b867..b6199284 100644 --- a/tests/test_memorize_cli.py +++ b/tests/test_memorize_cli.py @@ -12,6 +12,11 @@ from memu.app.memorize.materialize import MaterializedConversation +@pytest.fixture(autouse=True) +def isolated_runs(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(cli, "MEMORIZE_WORKSPACE", str(tmp_path / "developer")) + + def _payload() -> dict[str, Any]: return { "schema_version": "1.0", @@ -39,16 +44,21 @@ def test_parser_covers_memorize_actions() -> None: parser = cli.build_parser() for argv in ( ["memorize", "prepare", "input.json"], - ["memorize", "commit"], - ["memorize", "verify-resources"], + ["memorize", "commit", "run-test"], + ["memorize", "verify-resources", "run-test"], + ["memorize", "discard", "run-test"], ): assert callable(parser.parse_args(argv).handler) + for action in ("commit", "verify-resources", "discard"): + with pytest.raises(SystemExit): + parser.parse_args(["memorize", action]) + with pytest.raises(SystemExit): parser.parse_args(["memorize", "commit", "--workspace", "custom"]) -def test_prepare_uses_fixed_workspace_and_prints_agent_handoff( +def test_prepare_allocates_workspace_and_prints_agent_handoff( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str], @@ -75,9 +85,11 @@ async def fake_prepare(memorize_input: Any, prepared_workspace: Any, selected_ba assert cli.main(["memorize", "prepare", str(payload)]) == 0 assert received["input"][0].items[0].content == "Remember this" - assert received["workspace"].base == workspace + assert received["workspace"].base.parent == workspace / "runs" + workspace = received["workspace"].base + assert workspace.is_dir() assert received["backend"] is backend - assert received["verify_command"] == "memu memorize verify-resources" + assert received["verify_command"] == f"memu memorize verify-resources {workspace.name}" output = capsys.readouterr().out assert "prepared developer session" in output assert "3 job(s)" in output @@ -109,7 +121,10 @@ async def fake_prepare(_memorize_input: Any, prepared_workspace: Any, _backend: assert cli.main(["memorize", "prepare", "-", "--json"]) == 0 output = json.loads(capsys.readouterr().out) + assert Path(output["workspace"]).parent == workspace / "runs" + workspace = Path(output["workspace"]) assert output == { + "run_id": workspace.name, "workspace": str(workspace), "transcript": { "memory_path": str(workspace / "input" / "1.jsonl"), @@ -124,7 +139,7 @@ async def fake_prepare(_memorize_input: Any, prepared_workspace: Any, _backend: "If any job fails, stop and report failure. Do not run `memu memorize commit`. " "Report success only after every job has completed." ), - "next_command": "memu memorize commit", + "next_command": f"memu memorize commit {workspace.name}", } @@ -153,11 +168,13 @@ async def fake_prepare(inputs: Any, workspace: Any, _backend: Any, **_kwargs: An assert cli.main(["memorize", "prepare", *(str(path) for path in payloads), "--json"]) == 0 assert [item.items[0].content for item in received["inputs"]] == [str(index) for index in range(num_sessions)] output = json.loads(capsys.readouterr().out) + workspace = Path(output["workspace"]) + assert workspace.parent == tmp_path / "workspace" / "runs" assert "transcript" not in output assert output["transcripts"] == [ { - "memory_path": str(tmp_path / "workspace" / "input" / f"{index}.jsonl"), - "skill_path": str(tmp_path / "workspace" / "input" / f"{index}_full.jsonl"), + "memory_path": str(workspace / "input" / f"{index}.jsonl"), + "skill_path": str(workspace / "input" / f"{index}_full.jsonl"), } for index in range(1, num_sessions + 1) ] @@ -174,7 +191,7 @@ def test_executor_prompt_preserves_returned_job_order(tmp_path: Path) -> None: assert prompt.index(f"2. {workspace.jobs / '3.txt'}") < prompt.index(f"3. {workspace.jobs / '11.txt'}") -def test_prepare_default_workspace_keeps_next_command_short( +def test_prepare_next_command_targets_allocated_run( monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str], ) -> None: @@ -186,7 +203,8 @@ async def fake_prepare(_memorize_input: Any, workspace: Any, _backend: Any, **_k monkeypatch.setattr(cli, "prepare_memorize", fake_prepare) assert cli.main(["memorize", "prepare", "-", "--json"]) == 0 - assert json.loads(capsys.readouterr().out)["next_command"] == "memu memorize commit" + output = json.loads(capsys.readouterr().out) + assert output["next_command"] == f"memu memorize commit {output['run_id']}" def test_prepare_missing_file_reports_error(capsys: pytest.CaptureFixture[str]) -> None: @@ -210,7 +228,7 @@ def test_prepare_invalid_input_reports_validation_error( assert "at least 1 item" in capsys.readouterr().err -def test_commit_uses_selected_backend_and_fixed_workspace( +def test_commit_uses_selected_backend_and_run_workspace( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str], @@ -227,17 +245,20 @@ async def fake_commit(selected_workspace: Any, selected_backend: Any) -> dict[st } monkeypatch.setattr(cli, "MEMORIZE_WORKSPACE", str(workspace)) + workspace = workspace / "runs" / "run-test" + workspace.mkdir(parents=True) monkeypatch.setattr(cli, "_build_backend", lambda _args: backend) monkeypatch.setattr(cli, "commit_memorize", fake_commit) - assert cli.main(["memorize", "commit"]) == 0 + assert cli.main(["memorize", "commit", "run-test"]) == 0 + assert not workspace.exists() assert received == {"workspace": cli.MemorizeWorkspace(workspace), "backend": backend} output = capsys.readouterr().out assert "committed 1 recall file(s) and 1 resource(s)" in output assert "memory/profile" in output -def test_verify_resources_uses_fixed_workspace_without_backend( +def test_verify_resources_uses_run_workspace_without_backend( tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str], @@ -251,9 +272,12 @@ def fake_verify(log: Path, resources: Path) -> int: return 2 monkeypatch.setattr(cli, "MEMORIZE_WORKSPACE", str(workspace)) + workspace = workspace / "runs" / "run-test" + workspace.mkdir(parents=True) + (workspace / ".memorize_run.json").write_text("{}", encoding="utf-8") monkeypatch.setattr(cli, "verify_resource_log", fake_verify) monkeypatch.setattr(cli, "_build_backend", lambda _args: pytest.fail("verifier must not build a backend")) - assert cli.main(["memorize", "verify-resources"]) == 0 + assert cli.main(["memorize", "verify-resources", "run-test"]) == 0 assert received == (workspace / ".resource.tmp", workspace / "resources.md") assert "verified 2 resource(s)" in capsys.readouterr().out diff --git a/tests/test_memorize_runs.py b/tests/test_memorize_runs.py new file mode 100644 index 00000000..3fe59330 --- /dev/null +++ b/tests/test_memorize_runs.py @@ -0,0 +1,194 @@ +from __future__ import annotations + +import asyncio +import json +from pathlib import Path +from types import SimpleNamespace +from unittest.mock import AsyncMock + +import pytest + +from memu import cli + + +@pytest.fixture() +def rig(tmp_path: Path, monkeypatch: pytest.MonkeyPatch): + root = tmp_path / "developer" + payload = tmp_path / "session.json" + payload.write_text( + json.dumps({"items": [{"type": "message", "role": "user", "content": "Remember this"}]}), encoding="utf-8" + ) + backend = SimpleNamespace( + list_all_recall_files=AsyncMock(return_value={"recall_files": [], "next_cursor": None}), + commit_results=AsyncMock(return_value={"recall_files": [], "resources": []}), + ) + output: list[dict] = [] + monkeypatch.setattr(cli, "MEMORIZE_WORKSPACE", str(root)) + monkeypatch.setattr(cli, "_build_backend", lambda _args: backend) + monkeypatch.setattr(cli, "_print_json", output.append) + monkeypatch.setattr("memu.hosts.templates.resolve", lambda _name, embedded: embedded) + return root, payload, backend, output + + +async def test_concurrent_prepares_isolate_batches_and_commit_only_their_run(rig) -> None: + root, payload, backend, output = rig + barrier = asyncio.Barrier(2) + + async def list_files(**_kwargs): + await barrier.wait() + return {"recall_files": [], "next_cursor": None} + + backend.list_all_recall_files.side_effect = list_files + parser = cli.build_parser() + args = parser.parse_args(["memorize", "prepare", str(payload), str(payload), "--json"]) + results = await asyncio.wait_for( + asyncio.gather(cli._cmd_memorize_prepare(args), cli._cmd_memorize_prepare(args)), timeout=5 + ) + assert results == [0, 0] + first, second = output + assert first["run_id"] != second["run_id"] + for prepared in (first, second): + workspace = Path(prepared["workspace"]) + assert workspace.parent == root / "runs" + assert len(prepared["transcripts"]) == 2 + assert len(prepared["jobs"]) == 5 + assert (workspace / ".memorize_run.json").is_file() + assert f"memu memorize verify-resources {prepared['run_id']}" in Path(prepared["jobs"][-1]).read_text( + encoding="utf-8" + ) + (workspace / "executor-note.txt").write_text("temporary", encoding="utf-8") + + second_files = { + p.relative_to(second["workspace"]): p.read_bytes() for p in Path(second["workspace"]).rglob("*") if p.is_file() + } + await cli._cmd_memorize_commit(parser.parse_args(["memorize", "commit", first["run_id"], "--json"])) + assert not Path(first["workspace"]).exists() + assert { + p.relative_to(second["workspace"]): p.read_bytes() for p in Path(second["workspace"]).rglob("*") if p.is_file() + } == second_files + backend.commit_results.assert_awaited_once() + + +def test_failed_commit_preserves_every_file_and_can_retry(rig) -> None: + _root, payload, backend, output = rig + assert cli.main(["memorize", "prepare", str(payload), "--json"]) == 0 + prepared = output[-1] + workspace = Path(prepared["workspace"]) + (workspace / "memory").mkdir() + (workspace / "memory" / "note.md").write_text("---\nname: note\n---\nremember me", encoding="utf-8") + before = {p.relative_to(workspace): p.read_bytes() for p in workspace.rglob("*") if p.is_file()} + backend.commit_results.side_effect = RuntimeError("store unavailable") + + assert cli.main(["memorize", "commit", prepared["run_id"], "--json"]) == 1 + assert {p.relative_to(workspace): p.read_bytes() for p in workspace.rglob("*") if p.is_file()} == before + backend.commit_results.side_effect = None + assert cli.main(["memorize", "commit", prepared["run_id"], "--json"]) == 0 + assert not workspace.exists() + assert backend.commit_results.call_args.kwargs["recall_files"][0]["name"] == "note" + + +def test_discard_removes_only_selected_run_without_backend(rig, monkeypatch: pytest.MonkeyPatch) -> None: + root, payload, _backend, output = rig + for _ in range(2): + assert cli.main(["memorize", "prepare", str(payload), "--json"]) == 0 + first, second = output + # An incomplete run from an interrupted prepare is also explicitly discardable. + (Path(first["workspace"]) / ".memorize_run.json").unlink() + sentinel = root / "keep.txt" + sentinel.write_text("keep", encoding="utf-8") + monkeypatch.setattr(cli, "_build_backend", lambda _args: pytest.fail("discard must not build a backend")) + assert cli.main(["memorize", "discard", first["run_id"], "--json"]) == 0 + assert output[-1] == {"run_id": first["run_id"], "discarded": True} + assert not Path(first["workspace"]).exists() + assert Path(second["workspace"]).is_dir() + assert sentinel.read_text(encoding="utf-8") == "keep" + + +@pytest.mark.parametrize("stage", ["snapshot", "job", "directory"]) +def test_commit_cleanup_failure_reports_durable_success( + rig, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str], stage: str +) -> None: + _root, payload, backend, output = rig + assert cli.main(["memorize", "prepare", str(payload), "--json"]) == 0 + prepared = output[-1] + workspace = Path(prepared["workspace"]) + + def fail_cleanup(*_args): + msg = "filesystem busy" + raise PermissionError(msg) + + original_unlink = Path.unlink + + def fail_job_unlink(path, *args, **kwargs): + if path.parent == workspace / "jobs": + fail_cleanup() + return original_unlink(path, *args, **kwargs) + + with monkeypatch.context() as patch: + if stage == "snapshot": + patch.setattr("memu.app.memorize.lifecycle.snapshot_tracked", fail_cleanup) + elif stage == "job": + patch.setattr(Path, "unlink", fail_job_unlink) + else: + patch.setattr(cli.shutil, "rmtree", fail_cleanup) + assert cli.main(["memorize", "commit", prepared["run_id"]]) == 1 + assert "committed, but cleanup failed" in capsys.readouterr().err + backend.commit_results.assert_awaited_once() + assert (workspace / ".memorize_run.json").exists() is (stage != "directory") + assert cli.main(["memorize", "discard", prepared["run_id"]]) == 0 + assert not workspace.exists() + + +def test_invalid_batch_does_not_allocate_a_run(rig) -> None: + root, payload, backend, _output = rig + invalid = payload.with_name("invalid.json") + invalid.write_text('{"items": []}', encoding="utf-8") + assert cli.main(["memorize", "prepare", str(payload), str(invalid)]) == 1 + assert not root.exists() + backend.list_all_recall_files.assert_not_awaited() + + +def test_failed_prepare_removes_only_its_new_directory(rig) -> None: + root, payload, backend, output = rig + assert cli.main(["memorize", "prepare", str(payload), "--json"]) == 0 + previous = Path(output[-1]["workspace"]) + backend.list_all_recall_files.side_effect = RuntimeError("list failed") + assert cli.main(["memorize", "prepare", str(payload), "--json"]) == 1 + assert list((root / "runs").iterdir()) == [previous] + assert (previous / ".memorize_run.json").is_file() + + +@pytest.mark.parametrize("action", ["commit", "verify-resources", "discard"]) +@pytest.mark.parametrize("run_id", ["../runs", "/outside", "C:\\outside", "run-../other", "run-missing"]) +def test_run_commands_reject_invalid_or_unknown_ids( + rig, monkeypatch: pytest.MonkeyPatch, action: str, run_id: str +) -> None: + root, _payload, _backend, _output = rig + sentinel = root / "runs" / "run-keep" + sentinel.mkdir(parents=True) + monkeypatch.setattr(cli, "_build_backend", lambda _args: pytest.fail("invalid run must not build a backend")) + assert cli.main(["memorize", action, run_id]) == 1 + assert sentinel.is_dir() + + +def test_run_commands_reject_symlink_directory(rig, tmp_path: Path) -> None: + root, _payload, backend, _output = rig + outside = tmp_path / "outside" + outside.mkdir() + (outside / "keep.txt").write_text("keep", encoding="utf-8") + link = root / "runs" / "run-link" + link.parent.mkdir(parents=True) + try: + link.symlink_to(outside, target_is_directory=True) + except OSError: + pytest.skip("directory symlinks unavailable") + for action in ("commit", "verify-resources", "discard"): + assert cli.main(["memorize", action, "run-link"]) == 1 + assert (outside / "keep.txt").is_file() + backend.commit_results.assert_not_awaited() + + +def test_verify_rejects_incomplete_run(rig) -> None: + root, _payload, _backend, _output = rig + (root / "runs" / "run-incomplete").mkdir(parents=True) + assert cli.main(["memorize", "verify-resources", "run-incomplete"]) == 1