Skip to content

perf: keep models loaded across audio/video and group the watcher queue by file kind - #107

Merged
iLevyTate merged 2 commits into
mainfrom
perf/model-grouping
Nov 10, 2026
Merged

iLevyTate merged 2 commits into
mainfrom
perf/model-grouping

Conversation

@iLevyTate

@iLevyTate iLevyTate commented Nov 10, 2026 •

Copy link
Copy Markdown
Owner

Follow-up to #106. It closes the two remaining places where analysis swapped models file by file on a 6 GB GPU.

  • Whisper beside the text model. When the text model already holds a dedicated GPU and canFitExternalLoad says Whisper won't fit beside it, Whisper loads with useGpu: false (whisper.cpp runs on the CPU with any build variant) and registers 0 bytes of GPU memory. Before, each audio or video file evicted one model to load the other; the live log showed [Whisper] Model evicted under memory pressure right after an mp3. Unified memory (Metal or Apple Silicon) is unchanged.
  • The watcher groups by file kind. With one file at a time, the watcher took files in arrival order, so alternating documents and audio swapped Whisper and the text model at every switch. It now keeps taking the kind it just ran (document, image, or audio/video) while one is waiting, for up to 25 picks in a row, then returns to queue order so nothing waits indefinitely. The document batch also takes one kind at a time.

Verified

  • New tests: four in WhisperService.test.js (CPU when it won't fit, GPU when it fits, GPU with no text model, never on Metal) and SmartFolderWatcher.laneGrouping.test.js (grouping and the streak cap). The grouping tests fail on the previous watcher.
  • Watcher, Whisper, media and batch suites: 22 suites and 474 tests pass. Lint, format and typecheck:js are clean.

Not covered

  • A live run with several audio files on the 6 GB GPU still needs checking after merge. Look for cpuBesideText: true in the [Whisper] Loading model log line, and no Whisper evictions between files.

Added: skip work the earlier batch already did

  • The problem. After waiting out an earlier batch (Fix duplicate analysis runs, model swapping, stale cloud overrides and OCR-fallback caching found in a live run #106), the rerun relied on the analysis cache, which keeps results for 30 minutes and at most 500 documents or 300 images. A long batch could outlive the cache, and the rerun would then redo the work.
  • The fix. While waiting, the run reads the earlier batch's results back from the durable ready queue, so progress shows them. Once the batch is free, it sends only files that have no current ready result, meaning the same modified time (filesNotYetReady). If the earlier batch finished everything, the run ends without another batch.
  • Tests. discoverBatchBusy now has 19 tests. The Discover-related suites, 8 suites and 123 tests, all pass.

…ue by file kind

- Whisper runs on the CPU beside the resident text model when the text
  model holds a dedicated GPU and Whisper would not fit next to it; it then
  registers no GPU memory, so the next text load no longer evicts it. Each
  audio or video file used to evict one model for the other. Unified memory
  (Metal/Apple Silicon) is unchanged.
- The smart-folder watcher keeps taking files of the kind it just ran
  (document, image, audio/video) while any are waiting, up to 25 picks in a
  row, then returns to queue order. With one file at a time it used to follow
  arrival order, swapping models whenever kinds alternated.
…it did not finish

The rerun after a busy wait relied on the analysis cache, which keeps
results for 30 minutes and at most 500 documents / 300 images; a long batch
outlived it and the rerun redid the work. While waiting, the run now hydrates
the earlier batch's results from the durable ready queue (so progress shows
them) and, once free, sends only the files without a current ready result
(same modified time). When the earlier batch finished everything, the run
ends without another batch.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant