Skip to content

Continuity: three stacked defects prevent any capsule, and therefore any backup #12

Description

@m10ust

Summary

Continuity has never once succeeded on this instance. Every hourly run is accepted and then fails with:

Continuity operation failed without reporting repository credentials.

and the panel reports FAILED / No recovery copy has been recorded, with sia continuity freeze refusing:

SIA unavailable: source corpus receipt does not bind the live corpus root

Three separate defects stack up. Fixing each one only reveals the next, and none of the three error messages contains the value that caused it, which is why this stayed silent for days on a box that is watched constantly.

Context: 1,818-page corpus, ~3,600 events/day, repository on another machine over Tailscale, restic backing it. That transport is fine. I verified the repo opens, the key is correct, and the repository id matches the config exactly.

1. Shard numbering hole (siamind.py, _validate_event_replay_key)

r"(?:-part-[2-9][0-9]*)?"   # only the FIRST digit is inspected

That accepts part 2-9, and 20-99, and 200-999, but rejects part-10 through part-19 and 100-199. Verified by calling the function directly:

part-1  ACCEPTED   part-9   ACCEPTED   part-19  REJECTED
part-2  ACCEPTED   part-10  REJECTED   part-20  ACCEPTED
part-8  ACCEPTED   part-11  REJECTED   part-100 REJECTED

Consequence on a busy day: the journal fills nine shards of 400, the tenth shard cannot be written, so corpus_write raises on every pulse and, by design, replays the complete marker forever. My part-9 page is 4 distinct ids repeated 100 times each. The ledger never settles, so continuity can never verify it.

Suggest -part-(?:[2-9]|[1-9][0-9]+), or better, validate against MAX_EVENT_SHARDS rather than a digit class.

2. The corpus receipt binds st_dev (siacapsule.py, ~918-927)

The receipt is built from the corpus directory's dev:ino:mode:uid. On btrfs, a subvolume change alters st_dev, so the stored receipt can never match again no matter how long you wait. Mine:

expected  55:2146790:16832:1000
actual    29:2146790:16832:1000

Only the device differs. Inode, mode and uid are identical. This is not a hostile change, it is the filesystem moving under a correct installation, and there is no path to recovery short of rewriting the receipt by hand.

Suggest binding ino+mode+uid plus a content hash, or at minimum including the expected and actual values in the error message.

3. Capsule entry bound vs corpus growth (sialib.py)

MAX_SOURCE_TAIL_RECORDS = 1024
MAX_SOURCE_SCAN_ENTRIES = MAX_SOURCE_TAIL_RECORDS   # both capsule limits alias this

corpus/packages holds 1,553 entries and grows daily. The freeze then reports:

SIA unavailable: capsule source directory exceeds its entry bound

The net effect is that SIA cannot build a capsule of itself once the corpus passes roughly a thousand pages, which means continuity disables itself precisely when the memory becomes worth protecting. On this box that happened quietly, at around 1,024 pages, some time ago.

Suggest sharding packages/ the way events/ is already sharded, or raising the bound with a documented ceiling so the limit is visible before it is hit.

After fixing all three locally

Freeze succeeds, capsule classifies ready, upload completes and verifies:

capsule_id       7d52d88b98ea4bc38e89850f8090c645
snapshot_id      c4f15954ab38bc61e6b929b56fe322f13ec77c22cdbe8f951b6249f0690646cd
state            verified
"Verified recovery copy is available."

The pipeline is sound. It is these three gates. All three fixes are local patches with timestamped backups, and I would rather upstream them properly than carry them.

One general suggestion: every one of these messages would have been diagnosable in minutes if it had printed the offending value, the replay key, the two device numbers, the directory and its entry count. The refusal logic itself is good, it fails closed and leaves nothing half-written, but the reporting is what made this invisible.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions