Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
25 changes: 24 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,29 @@ jobs:
- name: Install
run: |
python -m pip install --upgrade pip
python -m pip install -e .[dev]
# [vector] keeps the hnswlib path covered now that it is optional: without
# this the accelerator backend would silently stop being tested.
python -m pip install -e .[dev,vector]
- name: Tests
run: python -m pytest -q --no-header -p no:cacheprovider

no-dependencies:
# The zero-dependency claim is a promise to users, so it gets its own job: install
# the package alone and prove the core still imports and passes.
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.12"
- name: Install without extras
run: |
python -m pip install --upgrade pip
python -m pip install -e .[dev]
- name: No third-party runtime dependency may be installed
run: |
python -c "import importlib.util,sys; bad=[n for n in ('hnswlib','fuse','numpy') if importlib.util.find_spec(n)]; sys.exit('installed: '+','.join(bad)) if bad else None"
python -c "import pyvdisk; print('pyvdisk', pyvdisk.__version__, 'imported with the standard library only')"
- name: Tests (accelerator-dependent cases skip)
run: python -m pytest -q --no-header -p no:cacheprovider
14 changes: 13 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,19 @@
记录用户可见的变更。版本号与 `pyproject.toml` 的 `version` 一致;更早的历史见 git 提交。
本文件从 issue #1(核心缺口复盘)的修复开始维护,条目按该清单的编号标注。

## [Unreleased]
## [0.4.0] - 2026-09-30

### 新增 — 零依赖与跨平台(issue #2)
- **Windows 兼容层**:新增 `pyvdisk/compat.py`,把全部平台原语收敛到一处 —— 定位读写(`os.pread/pwrite` ↔ `lseek`+`read/write` + 互斥)、文件锁(`fcntl.flock` ↔ `msvcrt.locking`)、目录 `fsync`、`uid/gid`、符号链接探测。此前 `pyvdisk/infrastructure/checkpoint.py` 顶层 `import fcntl`,在 Windows 上连导入都会失败。现在除 `compat.py` 外,包内不再直接触碰这些原语,`tests/test_compat.py` 用一条守卫测试固定该约束。
- **去外部依赖**:`dependencies` 由 `["hnswlib>=0.8.0"]` 改为 `[]`。`VectorDisk` 新增内置 flat 索引后端(`pyvdisk/vector_index.py`),无 hnswlib 时自动回退,查询由 O(log n) 变 O(n) 但结果一致;索引文件带 `PVFLAT01` 魔数标识写入方,**两种后端互相可读**(hnswlib 写的盘无 hnswlib 也能查,反之亦然)。hnswlib / fusepy 降级为可选 extra:`pyvdisk[vector]`、`pyvdisk[fuse]`。
- **单文件快速上手**:新增 `examples/quickstart.py`(仅标准库,Windows / Linux 通用,无外部依赖),自底向上演示块设备、事务与检查点、向量检索、结构化日志、VScript、AgentSandbox。

### 新增 — AgentSandbox(issue #2)
- **`AgentSandbox`**:把 DataDisk 包成可直接交给 Agent 框架的工作区。`create()` / `open()`、`tools(style="openai"|"anthropic"|"mcp")`、`dispatch(name, arguments)`、`call()`(返回结构化 `ToolResult`)、`stats()`。
- **六个内置工具**:`write_file`、`read_file`、`list_files`、`make_directory`、`delete_file`、`run_script`;`read_only=True` 时按 capability 层拒绝写(不只是隐藏工具),`allow_delete=False` 可单独关掉删除。
- **路径空间是镜像不是宿主**:`..` 直接拒绝(不解析,便于审计留痕),`C:\Windows\x` 映射为镜像内 `/C:/Windows/x`;`/.system`、`/.vectors`、`/.logs` 及其清单文件对读写与列举一律不可见,脚本入口(`_SandboxFS`)与 Python 入口共用同一套路径规则。
- **审计**:每次工具调用(含被拒绝的调用)写入哈希链审计流 `agent-audit`,记录工具名、参数、状态、错误类型与能力摘要;`audit()` / `verify_audit()` / `retain_audit()` 可查、可校验、可裁剪。
- **`run_script`**:盘内 VScript 执行,默认不允许宿主读写(`host_read_roots` / `host_write_roots` 显式开启),脚本超时与输出/写入字节数均受限,避免脚本刷爆模型上下文。

### 修复 — ACID 与事务
- **A1** DataDisk 事务不再只覆盖 metadata:FS/Vector/Log 副作用走写前撤销日志(`kind=undo`),`abort()` 可回滚。
Expand Down
73 changes: 61 additions & 12 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,8 @@

[![GitHub](https://img.shields.io/badge/GitHub-PyVDisk-181717?logo=github)](https://github.com/HeDaas-Code/pyvdisk)
[![Python](https://img.shields.io/badge/Python-3.9%2B-3776AB?logo=python&logoColor=white)](https://www.python.org/)
[![Tests](https://img.shields.io/badge/Tests-391%20passed-2ea44f)](#验证)
[![Tests](https://img.shields.io/badge/Tests-447%20passed-2ea44f)](#验证)
[![No%20dependencies](https://img.shields.io/badge/dependencies-none-2ea44f)](#验证)

PyVDisk 是一个包含 VScript 脚本运行时的单机 Agentic 数据与执行基础设施项目:DataDisk 负责统一数据与执行基础设施,VScript 负责安全工作流语言、CLI、REPL 与运行时。

Expand All @@ -16,13 +17,13 @@ PyVDisk 是一个包含 VScript 脚本运行时的单机 Agentic 数据与执行

## 架构总览

<div align="center"><table><tr><th colspan="3">Agent / 应用</th></tr><tr><td colspan="3">VScript · Python API · CLI · REPL · HostProxy</td></tr><tr><th>Execution</th><th>Capability</th><th>Storage</th></tr><tr><td>ExecutionService<br>DurableQueue<br>worker · lease · retry · DLQ<br>RunState · Audit</td><td>ScopedDataDisk<br>FS / Vector / Log / Checkpoint<br>逐操作权限</td><td>DataDisk<br>单一 .vdisk<br>WAL · transaction · recovery</td></tr><tr><th>FS</th><th>Vector</th><th>Log</th></tr><tr><td>workspace</td><td>HNSW memory<br>generation · checksum</td><td>trace<br>sequence · replay · ack</td></tr><tr><th colspan="3">VirtualDisk · Volume · lock · fsync · mirror degraded fallback</th></tr></table></div>
<div align="center"><table><tr><th colspan="3">Agent / 应用</th></tr><tr><td colspan="3">VScript · Python API · CLI · REPL · HostProxy</td></tr><tr><th>Execution</th><th>Capability</th><th>Storage</th></tr><tr><td>ExecutionService<br>DurableQueue<br>worker · lease · retry · DLQ<br>RunState · Audit</td><td>ScopedDataDisk<br>FS / Vector / Log / Checkpoint<br>逐操作权限</td><td>DataDisk<br>单一 .vdisk<br>WAL · transaction · recovery</td></tr><tr><th>FS</th><th>Vector</th><th>Log</th></tr><tr><td>workspace</td><td>HNSW / flat memory<br>generation · checksum</td><td>trace<br>sequence · replay · ack</td></tr><tr><th colspan="3">VirtualDisk · Volume · lock · fsync · mirror degraded fallback</th></tr></table></div>

## 架构图谱

**▶ [打开可交互架构图谱](https://htmlpreview.github.io/?https://github.com/HeDaas-Code/pyvdisk/blob/master/normify-pyvdisk/normify.html)** — 点击模块逐层下钻、悬停查看中英双语介绍、`?lang=en` 切换英文、`#module=<id>` 深链直达具体模块。
**▶ [打开可交互架构图谱](https://htmlpreview.github.io/?https://github.com/HeDaas-Code/pyvdisk/blob/normify/normify-pyvdisk/normify.html)** — 点击模块逐层下钻、悬停查看中英双语介绍、`?lang=en` 切换英文、`#module=<id>` 深链直达具体模块。

> GitHub 会过滤 README 中的 `<script>` / `<iframe>`,自包含的交互页无法内联渲染,因此上传的是**一键运行**链接:由 htmlpreview 直接执行仓库内 `normify-pyvdisk/normify.html`(单文件、无外部依赖、无网络请求)。下面的 Mermaid 图则在 GitHub 上原生渲染。
> GitHub 会过滤 README 中的 `<script>` / `<iframe>`,自包含的交互页无法内联渲染,因此上传的是**一键运行**链接:由 htmlpreview 直接执行 [`normify` 归档分支](https://github.com/HeDaas-Code/pyvdisk/tree/normify)中的 `normify-pyvdisk/normify.html`(单文件、无外部依赖、无网络请求)。下面的 Mermaid 图则在 GitHub 上原生渲染。

```mermaid
graph TD
Expand Down Expand Up @@ -68,13 +69,15 @@ graph TD
G --> G_scoped["scoped(ScopedDataDisk)"]
```

图谱产物已归档到独立的 [`normify` 分支](https://github.com/HeDaas-Code/pyvdisk/tree/normify),不占用代码仓库的文件树;下表链接直达该分支:

| 产物 | 说明 |
|---|---|
| [`normify-pyvdisk/normify.html`](normify-pyvdisk/normify.html) | 单文件可交互图谱(272 模块 / 794 API / 124 依赖箭头) |
| [`normify-pyvdisk/outline.md`](normify-pyvdisk/outline.md) | 缩进式模块大纲,适合逐层通读 |
| [`normify-pyvdisk/api-index.json`](normify-pyvdisk/api-index.json) | 全量 API 索引 |
| [`normify-pyvdisk/tree.json`](normify-pyvdisk/tree.json) | 编译产物:模块、每层布局、依赖边、内容指纹 |
| [`normify-pyvdisk/modules/`](normify-pyvdisk/modules) | 272 个模块 Markdown,frontmatter 为机器可读契约(含源码路径与行号证据) |
| [`normify-pyvdisk/normify.html`](https://github.com/HeDaas-Code/pyvdisk/blob/normify/normify-pyvdisk/normify.html) | 单文件可交互图谱(272 模块 / 794 API / 124 依赖箭头) |
| [`normify-pyvdisk/outline.md`](https://github.com/HeDaas-Code/pyvdisk/blob/normify/normify-pyvdisk/outline.md) | 缩进式模块大纲,适合逐层通读 |
| [`normify-pyvdisk/api-index.json`](https://github.com/HeDaas-Code/pyvdisk/blob/normify/normify-pyvdisk/api-index.json) | 全量 API 索引 |
| [`normify-pyvdisk/tree.json`](https://github.com/HeDaas-Code/pyvdisk/blob/normify/normify-pyvdisk/tree.json) | 编译产物:模块、每层布局、依赖边、内容指纹 |
| [`normify-pyvdisk/modules/`](https://github.com/HeDaas-Code/pyvdisk/tree/normify/normify-pyvdisk/modules) | 272 个模块 Markdown,frontmatter 为机器可读契约(含源码路径与行号证据) |

图谱由 Normify 从仓库源码生成,每个叶子模块都带**仓库内真实文件路径 + 行号区间**的 `source` 证据与 SHA-256 指纹,冻结于 commit `6d203c0`;`normify_validate` 结果为 0 error。

Expand All @@ -84,9 +87,17 @@ graph TD

## 快速开始

**零依赖**:核心只用标准库,`pip install pyvdisk` 不需要编译任何东西(hnswlib / fusepy 都是可选加速项)。一条命令跑通全栈:

```bash
python examples/quickstart.py # 单文件、无外部依赖,Windows / Linux 通用
```

```bash
python -m pip install -e .
python -m pip install -e ".[dev]"
python -m pip install -e ".[dev]" # 开发依赖
python -m pip install -e ".[vector]" # 可选:hnswlib 近似最近邻(不装则用内置 flat 索引)
python -m pip install -e ".[fuse]" # 可选:FUSE 挂载
```

```python
Expand All @@ -99,6 +110,35 @@ with DataDisk.create("agent.vdisk", 32 * 1024 * 1024) as disk:
disk.log.append("events", level="INFO", logger="demo", message="started")
```

### 对接 Agent 框架(10 行)

`AgentSandbox` 把 DataDisk 包成一个"给模型用的工作区":路径被限制在镜像内,工具调用全部落进哈希链审计流。

```python
import json
from pyvdisk import AgentSandbox

box = AgentSandbox.create("agent.vdisk") # 或 AgentSandbox.open(...) 续用已有镜像
client = OpenAI().chat.completions # 换成任意框架的客户端
messages = [{"role": "user", "content": "把发现写进 /report.md"}]

while True:
reply = client.create(model="gpt-4o", messages=messages, tools=box.tools())
message = reply.choices[0].message
messages.append(message) # 不追加则模型看不到自己请求过什么
if not message.tool_calls:
break # 模型不再调用工具,结束
for call in message.tool_calls: # 一次可能请求多个工具
result = box.dispatch(call.function.name, json.loads(call.function.arguments))
messages.append({"role": "tool", "tool_call_id": call.id, "content": result})
```

`box.tools()` 支持 `openai` / `anthropic` / `mcp` 三种 schema;`box.dispatch()` 永远返回字符串(失败也是字符串),可以直接塞回消息列表,**被拒绝的调用也一样**——模型看到的是"越权被拒"这条结果,而不是整个 agent loop 崩掉。

三个容易踩的点:`json.loads` 的是 `call.function.arguments`(不是 `call.arguments`);`tool_calls` 是列表,要逐个 `dispatch` 并用各自的 `call.id` 一一对应地回填 `tool_call_id`;每轮都要把 assistant 消息本身也追加进 `messages`。

可运行版本见 [examples/agent_tools.py](examples/agent_tools.py)(内置脚本化"模型",无需 API key 与网络,因此也能进 CI)。

## VScript

VScript 与 PyVDisk 共用 DataDisk、namespace、capability、事务、WAL 和审计模型,支持 Lexer、Parser、解释器、变量、表达式、循环、函数、模块、标准库、事务、rollback、task/await/parallel、cron、触发器、REPL 和盘内脚本。详细规范见 [docs/VSCRIPT_SPEC.md](docs/VSCRIPT_SPEC.md)。
Expand All @@ -112,6 +152,7 @@ pyvdisk vscript run-disk tools.vdisk:/.vscript/scripts/job.vds

## 核心能力

- 零外部依赖:核心仅用标准库,`dependencies = []`;Windows / Linux 同一套代码(`pyvdisk/compat.py` 收敛定位读写、文件锁、目录 fsync、uid/gid 等平台差异)。hnswlib(近似最近邻)与 fusepy(FUSE 挂载)是可选加速项,不装也能跑全栈:`VectorDisk` 自动回退到内置 flat 索引,索引文件带写入方标识,两种后端互相可读。
- Storage Plane:FS、Vector、Log、Checkpoint、Metadata、WAL、VirtualDisk、Volume。
- Execution Plane:ExecutionService、DurableQueue、worker、lease、heartbeat、retry、idempotency、dead-letter、RunState、Audit。
- WAL 检查点:日志结算后整段截断(`wal.ckpt.json` 记录明细),重挂载只重放检查点之后的尾巴;DataDisk 与 VScript 共用同一份 WAL 实现,`vscript run --wal` 是它的真实恢复入口。
Expand Down Expand Up @@ -149,5 +190,13 @@ pyvdisk info image.vdisk
.venv/bin/python -m pytest -q
```

当前回归:391 passed(本地 3.12;CI 覆盖 3.9 / 3.10 / 3.11 / 3.12 四个版本的同一套用例)
其中 3.11 本地缺 hnswlib 时有 5 例跳过(该版本无法编译 hnswlib),CI 环境正常。
CI 在 `3.9 / 3.10 / 3.11 / 3.12` 四个版本上跑同一套用例,并额外跑一个不装任何第三方包的
`no-dependencies` job。两个口径的实测结果(均为 CI 输出,非本地推算):

| 环境 | 结果 |
|---|---|
| `.[dev,vector]`(装了 hnswlib) | **447 passed** |
| `.[dev]`(不装 hnswlib) | **443 passed, 4 skipped** |

跳过的是"两种索引后端结果一致"这类对照用例 —— 它们需要 hnswlib 与内置 flat 索引同时在场,
没装加速器时无法对照。核心路径在两种环境下都完整执行。
120 changes: 120 additions & 0 deletions examples/agent_tools.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,120 @@
#!/usr/bin/env python3
"""Wire AgentSandbox into an agent loop -- offline, no API key, no network.

python examples/agent_tools.py [workdir]

The "model" here is a scripted list of tool calls, so this runs anywhere. The
loop is exactly the one you write against a real framework: ask the model, run
every tool call it asks for, feed the results back, repeat until it stops asking.
Swap ``scripted_model`` for an OpenAI-compatible client and nothing else changes --
``box.tools()`` and ``box.dispatch()`` are the whole integration surface.

Note the last scripted call: the model asks to read ``/etc/passwd`` and gets a
refusal, which is fed back like any other tool result. That is the point of
running an agent against a sandbox rather than a directory.
"""
from __future__ import annotations

import importlib.util
import json
import os
import sys
import tempfile

if importlib.util.find_spec("pyvdisk") is None:
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))

from pyvdisk import AgentSandbox # noqa: E402 - the path fix has to run first

#: What a model would have produced. Each entry is one assistant turn; an entry
#: with several calls reproduces a model asking for more than one tool at once.
SCRIPTED_TURNS = [
{"tool": "list_files", "arguments": {"path": "/"}},
{"tool": "write_file", "arguments": {
"path": "/report.md",
"content": "# Findings\n\nThe sandbox works.\n",
}},
{"tool": "run_script", "arguments": {"source": (
'import std.fs as fs;\n'
'let names = fs.listdir(sandbox, "/");\n'
'print("files:", names);\n'
'fs.write(sandbox, "/from-script.txt", "written inside the sandbox");\n'
)}},
{"tool": "read_file", "arguments": {"path": "/report.md"}},
{"tool": "read_file", "arguments": {"path": "/etc/passwd"}}, # refused
# Two tools in one turn: the loop must answer each with its own call id.
{"calls": [
{"tool": "read_file", "arguments": {"path": "/from-script.txt"}},
{"tool": "list_files", "arguments": {"path": "/", "recursive": True}},
]},
]


class _Message:
"""The assistant message shape the OpenAI client returns."""

def __init__(self, calls):
self.tool_calls = [_ToolCall(f"call_{index}", tool, arguments)
for index, (tool, arguments) in enumerate(calls)]


class _ToolCall:
def __init__(self, call_id, name, arguments):
self.id = call_id
self.function = type("Function", (), {"name": name,
"arguments": json.dumps(arguments, ensure_ascii=False)})()


def scripted_model(messages):
"""Stand-in for one chat completion: returns the next assistant message.

Returns an empty-tool_calls message once the script runs out, which is how a
real model signals "I am done" -- the loop below treats it identically.
"""
# Assistant turns are objects, tool results are plain dicts; count the latter.
asked = sum(1 for message in messages
if isinstance(message, dict) and message.get("role") == "tool")
turn = SCRIPTED_TURNS[asked] if asked < len(SCRIPTED_TURNS) else None
if turn is None:
return _Message([])
calls = [(tool["tool"], tool["arguments"]) for tool in turn.get("calls", [turn])]
return _Message(calls)


def main(argv):
work = argv[1] if len(argv) > 1 else tempfile.mkdtemp(prefix="pyvdisk-agent-")
os.makedirs(work, exist_ok=True)

with AgentSandbox.create(os.path.join(work, "agent.vdisk"), size_bytes=32 << 20) as box:
print("tools offered to the model:", ", ".join(tool["name"] for tool in box.tools()))
print("json schema of one tool:", json.dumps(box.tools()[0], ensure_ascii=False)[:120], "...\n")

messages = [{"role": "user", "content": "Summarise the workspace and write /report.md"}]
step = 0
while step < 12:
# Identical to the README demo: ask, append the assistant message, run
# every call it asked for, feed each result back by its own call id.
message = scripted_model(messages)
messages.append(message)
if not message.tool_calls:
break
for call in message.tool_calls:
step += 1
arguments = json.loads(call.function.arguments)
result = box.dispatch(call.function.name, arguments)
messages.append({"role": "tool", "tool_call_id": call.id, "content": result})
print(f"[{step}] {call.function.name}({call.function.arguments[:60]})")
print(" ->", result.replace("\n", "\n "))

print("\nfinal /report.md:", repr(box.read_text("/report.md")))
report = box.verify_audit()
print(f"audit: {report['length']} chained records, chain ok = {report['ok']}")
print("refusals recorded:",
[row["metrics"]["tool"] for row in box.audit(status="failure")])
print("\nEvery call above is durable, hash-chained and replayable: reopen",
os.path.join(work, "agent.vdisk"), "to see it.")
return 0


if __name__ == "__main__":
raise SystemExit(main(sys.argv))
Loading
Loading