bug: direct optimize can leave a pending Mutation recovery barrier that a long-lived server never clears #554

Summary

A scheduled omnigraph optimize --cluster … --graph … (direct storage, S3) run against a graph that a long-running omnigraph-server has open can leave a pending Mutation recovery operation on main that the server cannot clear. Every subsequent write through that server fails, for as long as the process stays up. Restarting the server clears it.

We hit this four times in eleven days on a production deployment; the last one froze cross-repo data for ~15 hours.

Error

recovery required for operation 01M0VH9C6D8A4XQCEK5YJKSMPX: pending Mutation
recovery operation on branch 'main' blocks the synchronous write/control
recovery barrier (datasets dataset for node type 'RepoSymbol'); reopen the
graph read-write before retrying

Clients retry and give up — ours budgets ~6s of backoff, deliberately sized to ride out a barrier raised by a concurrent writer and to report a genuinely stuck branch rather than sleep through it. This is the second case: the same operation id comes back on every retry, from multiple independent client processes, for hours.

Why we believe optimize is the trigger

The operation ids are ULIDs, so they date themselves. All four wedges decode to within five seconds of the same wall-clock time, on four different days:

01M01SA4K02SX45CBB16CE60PQ   2026-08-15 04:01:36 UTC
01M0H7PDHBAB7YHQ65E34MH4BJ   2026-08-21 04:01:35 UTC
01M0RYWR2BHBNKPT1QEQPFC1TE   2026-08-24 04:01:40 UTC
01M0VH9C6D8A4XQCEK5YJKSMPX   2026-08-25 04:01:37 UTC

Our scheduled optimize runs daily at 03:20 UTC. Our indexer runs every 4h on the hour, so 04:00 is the first writer to arrive after compaction, and its first write lands at ~04:01.

The dataset named in the barrier is one optimize had just rewritten:

omnigraph optimize → s3://…/graphs/code-bridge.omni (direct, remote)
  node type 'InterfaceBinding'   frags 11 → 1 ✓
  node type 'RepoSymbol'         frags 14 → 1 ✓

repair reports nothing

Preview against the same graph while wedged — no drift anywhere, including on the dataset the barrier names:

repair s3://…/graphs/code-bridge.omni — preview mode, 3 datasets
  node type 'InterfaceBinding'   no_op   no_drift    842
  node type 'PackageMap'         no_op   no_drift   1037
  node type 'RepoSymbol'         no_op   no_drift   2023

So this is not manifest/Lance-HEAD drift, and repair is not the remedy.

What clears it

kubectl rollout restart on the server. Immediately afterwards the same client, same config, same command wrote successfully (symbols=5303 edges=16883 bindings=108). That matches the error's own hint — "reopen the graph read-write" — if the recovery runs at read-write open and the server never reopens while it is up.

What we could not establish

optimize compacts this graph every day, and the wedge appeared on four days out of fourteen, so compaction is necessary and not sufficient. Two candidate discriminators checked and rejected:

• optimize compacted RepoSymbol 14 → 1 on clean days too (verified on a clean day: byte-identical output);
• server uptime spanning the optimize run was the same on clean days (one pod spanned both windows on two consecutive clean days).

Something about the specific write or the compaction result decides it, and we do not know what.

Questions

1. Is direct-storage optimize concurrent with a serving omnigraph-server on the same graph a supported configuration? Our deployment docs assumed it was, on the basis that the publisher CAS lets optimize rebase-and-retry against concurrent writers — which appears to be about write conflicts and not about the recovery barrier.
2. Should the server detect that a store was rewritten underneath it and reopen, rather than requiring a restart?
3. Is a pending Mutation recovery operation expected to be clearable without reopening the graph? If so, what should clear it — and if not, is there a supported way to signal a serving process to reopen?

Environment

• Storage: S3, cluster-managed, storage format 6
• Client CLI where the errors were reproduced by hand: omnigraph 0.10.0, internal-schema 6
• Server: 0.10.0-era build from the moving edge tag. I have not pinned the exact server commit; happy to if it would help.
• Graph: ~2k RepoSymbol rows, ~840 InterfaceBinding, ~1k PackageMap

Happy to supply fuller logs, the ULID decoding, or to test a patch against this deployment.