Repository navigation
perf(core): 用量账本分块读并在块间让出事件循环,存储清单的条数不再全量读 - #415
Merged
Merged
Conversation
- 之前:打开用量页(/api/usage)和存储清单(/api/storage 里用量那行的条数)各自同步读完整份 usage.jsonl。334 MB、130131 条的账本上,读盘加解析一次占住事件循环 1863 ms。 - 之后:readAll 每读 4 MB(原有的 READ_CHUNK_BYTES)让出一次事件循环。同一份账本读完共 1.4–2.4 s,8 次测量里相邻两次计时器回调的间隔多数在 40–70 ms,其中 6 次各有一个 160–330 ms 的间隔。 - 条数:第一次问时开始一次同样分块的扫描并返回 null,存储清单那行这时只显示文件大小; 扫描完成后条数留在内存里,每写入一条加一,清空归零,写入失败后作废重数。 - clear 改成异步,条数来自同一个分块扫描。 - 没有动 aggregateUsage:解析出的记录交给它之后仍是一段同步计算,同一份账本上 230–580 ms。 - 测试:跨多个读取块的账本在第一块之后让出事件循环;count 在扫描完成前是 null,之后随 append 增加且不再读盘。 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


替代 #406。#406 指出了下面两处同步全量读,用工作线程处理,为此把
tsx加进发布依赖并去掉了存储清单的条数;维护者 11-15 决定由这个 PR 用不加依赖、不起工作线程、保留条数的做法处理同一个问题,合入后关闭 #406。两个 PR 改的是同一批文件,只合这一个。问题
/api/usage:aggregateUsage(core.usageLog.readAll(), …),readAll同步读完并解析整份data/usage.jsonl。/api/storage:用量那行的stat调usageLog.count(),它也是一次readAll。读的这段时间引擎线程不处理事件投递和别的 HTTP 请求。#406 里报的现场是 1.48 GB 的账本。
改动
UsageLog.readAll()改成异步:每读 4 MB(原有的READ_CHUNK_BYTES)让出一次事件循环。写入是同步的appendFileSync,所以每次让出时文件里都是完整的行。UsageLog.count()返回number | null:第一次问时开始一次同样分块的扫描并返回null;扫描完成后条数留在内存里,每写入一条加一,清空归零,写入失败后作废、下次重数。/api/storage才有条数。UsageLog.clear()改成异步,返回的条数来自同一个分块扫描(StoragePart.clear本来就允许返回 Promise)。/api/storage、/api/usage的响应形状不变,StoragePart契约不变。测量
本机,一份 334 MB、130131 条有效记录的真实账本(只读),
scratch/里的脚本用 1 ms 的计时器记相邻两次回调的间隔:那个 160–330 ms 的间隔没有定位到原因;
--trace-gc的一次运行里,读到末尾时有一次 380 MB 堆上的 Mark-Compact。没有做的
aggregateUsage没动。记录读出来之后它仍是一段同步计算,同一份账本上 230–580 ms(30 天窗口与全部记录)。验证
pnpm test、pnpm run typecheck、pnpm run typecheck:web通过。tests/core/usage-log.test.ts:原有用例改成await;跨多个读取块的那条加了「第一块之后让出事件循环」的断言;新增一条count在扫描完成前是null、之后随append增加且不再读盘。🤖 Generated with Claude Code