Add snapshots

[peterjan]

This PR introduces snapshots. A snapshot is a backup of the sqlite database that gets gzipped and uploaded to the Sia network as a tagged, pinned object, so the full database can be retrieved and restored to its state at the time the snapshot was taken.

References #202

[Copilot]

Pull request overview

This PR adds first-class “snapshots” metadata to the SQLite persistence layer so database backups can be recorded and used to prevent the orphan/unpin loop from unpinning objects that are still referenced by a backup (Issue #202).

Changes:

• Replace the store-level Backup hook with CreateSnapshot, which both creates a SQLite backup file and records referenced sia_object_ids as a snapshot.
• Add snapshots / snapshot_objects tables (migration + init schema) and update orphan selection to exclude objects referenced by any snapshot.
• Add unit/integration tests validating snapshot creation, listing, deletion, and orphan-withholding behavior; add a changeset entry.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
sia/sia.go	Store interface updated to CreateSnapshot; backup entrypoint now records snapshots.
sia/persist/sqlite/snapshots.go	Implements snapshot creation/listing/deletion and ties snapshot creation to DB backup.
sia/persist/sqlite/snapshots_test.go	New tests for snapshot lifecycle and orphan withholding.
sia/persist/sqlite/objects.go	Orphan selection now excludes objects referenced by snapshots.
sia/persist/sqlite/objects_test.go	Extends orphan tests to cover snapshot-based withholding.
sia/persist/sqlite/migrations.go	Adds migration creating snapshots and snapshot_objects tables + index.
sia/persist/sqlite/init.sql	Adds the same tables/index to fresh DB initialization.
sia/objects/objects.go	Adds objects.Snapshot model used by listing.
s3/s3_test.go	Backup endpoint test now asserts the backup is recorded as a snapshot.
.changeset/snapshot_orphan_guard.md	Documents the behavior change in backups/orphan handling.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[peterjan]

(still working on exploring the trade-offs between backup API and vacuum into)

[peterjan]

@ChrisSchinnerl

I explored several strategies and ended up reverting it to the simplest approach. The main reason why I think it's not trivial, is because of SetMaxOpenConns. We set that to one to avoid "database is locked" errors, but this also means that concurrent writers will be blocked for the duration of the backup. I tested and benchmarked this and it does effectively block writers. It's effectively a semaphore right so that makes sense. I think for our other apps that's probably fine but I feel this would be a big downside for s3d. In practice it's likely not going to be an issue unless you have a (really) really big node.

An alternative approach I considered is to use VACUUM INTO. The upside with vacuum is that it compacts the database, so you get the smallest possible backup, and that it doesn't block concurrent writing processes what so ever. The big downside however is that it writes all changes to a WAL during the backup. Considering that we have disk size limits already I figured we probably don't want that. In my testing it generated a 50GB WAL backing up a 40GB database, but that was with 10 write processes hammering it for the whole duration of the backup, still.. not ideal.

The most correct approach would be to gate our single connection and use a transaction queue. All transactions flow through that as well as the backup and essentially what it does is it interleaves "regular operations" with backup steps properly. In a way that you are also guaranteed that the backup process makes progress at fixed intervals and doesn't have to fight for the connection. I was implementing that but it felt over-engineered and figured I'd keep it super simple and propose the approach first. I would say it's 200 lines or something, you'd have a txQueue and a txJob and if there's an active backup you step every couple of jobs or after a fixed amount of time... It comes with some footguns and needless overhead because 99.99% of the time you are not backing up.

We could also forget about FIFO entirely and just rely that the backup will eventually finish because after releasing the connection, it'll fight for the connection again with other concurrent write processes and eventually get it and make progress. In practice that might work really well. I wrote that out and tested it and in my benchmark the longest wait was 30s, which I found unacceptable.. At the same time though you eventually finish and you don't block writers during the backup process.

In short:

• VACUUM is not ideal because the WAL it produces but it allows concurrent writes, which is really nice
• single conn backup blocks other writers for the duration of the backup, takes ~4m for a 40GB database, but realistically it's very fast because s3d databases don't grow that big
• multi conn backup potentially never finishes because writes make it restart
• single conn backup with interleaved writes is the cleanest approach

edit:

I have a bunch of benchmark output but wanted to type it out instead.

SQLite backup strategy comparison

Strategy	Completed under load	Duration	Restarts	Steps	Source WAL growth	Backup size	Copy consistent
VACUUM INTO (dedicated connection)	Yes	2m 26s	N/A	1 statement	55.3 GB	37.0 GB	Yes, 150000/150000
Single-connection backup API	Yes	3m 35s	0	90,301	~0	37.2 GB	Yes, 150000/150000
Multi-connection backup API	No, timed out	5m 00s	128,884	2.48M	~40 MB	Not produced	No, livelock
Interleaved backup API, same connection per step	Yes	4m 42s	0	90,339	~0	37.2 GB	Yes, 150000/150000

Strategy	Writes completed	Throughput	Mean latency	Max latency	% < 1 ms	% < 10 ms
VACUUM INTO	13.2M	~90K/s	114 µs	5.60 s	99.87%	99.99%
Single-connection backup API	0, fully blocked	0	N/A	N/A	N/A	N/A
Multi-connection backup API	16.0M	~53K/s	188 µs	2.46 s	98.83%	99.90%
Interleaved backup API, same connection per step	0.9M	~3.2K/s	3.11 ms	30.03 s	53.2%	92.9%

edit 2:

One thing to note is that whatever strategy we choose, we should probably do the following items because it's little cost and real benefit:

• integrity_check + fsync + atomic rename publish for crash-safety
• DELETE-mode destination, we currently don't do that
• write to a temp path, validate, then rename into place
• VACUUM INTO the tmp file into the desintation path to get the compacting

[chris124567]

Nothing to add to Chris's comments

[peterjan]

going to push one more commit that adds the synced concept that gates orphan processing to ensure we don't mess up while snapshots are being actively restored

[chris124567]

@peterjan vibed this up. Seems like the fix is somewhat difficult...

diff --git a/sia/persist/sqlite/snapshots_test.go b/sia/persist/sqlite/snapshots_test.go
index ad35449..e4e37ac 100644
--- a/sia/persist/sqlite/snapshots_test.go
+++ b/sia/persist/sqlite/snapshots_test.go
@@ -227,3 +227,52 @@ func TestSnapshots(t *testing.T) {
 		t.Fatal("unexpected", orphans)
 	}
 }
+
+// TestSnapshotDoesNotRetainFutureObject demonstrates that a snapshot must not
+// retain an object that was created after the snapshot was taken. The current
+// generation-only orphan guard fails this test because both the old snapshot
+// and the later object share the same generation.
+func TestSnapshotDoesNotRetainFutureObject(t *testing.T) {
+	const bucket = "test-bucket"
+
+	store := initTestDB(t, zaptest.NewLogger(t))
+	if err := store.CreateBucket(testAccessKeyID, bucket); err != nil {
+		t.Fatal(err)
+	}
+
+	// Take and complete an empty snapshot.
+	snap, _, err := store.CreateSnapshot()
+	if err != nil {
+		t.Fatal(err)
+	} else if snap.ObjectCount != 0 {
+		t.Fatal("expected an empty snapshot, got", snap.ObjectCount, "objects")
+	} else if err := store.MarkSnapshotPinned(snap.ID, frand.Entropy256()); err != nil {
+		t.Fatal(err)
+	}
+
+	// Upload and pin an object after the snapshot was taken.
+	obj := sdk.Object{}
+	sealed := obj.Seal(types.GeneratePrivateKey())
+	objectID := sealed.ID()
+	md5 := frand.Entropy128()
+	filename := "future-object"
+	if _, _, err := store.PutObject(testAccessKeyID, bucket, "future", md5, nil, 1, &filename); err != nil {
+		t.Fatal(err)
+	} else if err := store.MarkObjectUploaded(bucket, "future", "", md5, sealed, time.Now().Add(time.Hour)); err != nil {
+		t.Fatal(err)
+	} else if _, err := store.MarkObjectPinned(objectID); err != nil {
+		t.Fatal(err)
+	}
+
+	// Since the snapshot predates the object, deleting the object should make
+	// it immediately eligible for unpinning.
+	if _, _, _, err := store.DeleteObject(testAccessKeyID, bucket, s3.ObjectID{Key: "future"}); err != nil {
+		t.Fatal(err)
+	}
+	orphans, err := store.OrphanedObjects(100)
+	if err != nil {
+		t.Fatal(err)
+	} else if len(orphans) != 1 || orphans[0] != objectID {
+		t.Fatalf("snapshot predates object: expected orphan %v to be eligible, got %v", objectID, orphans)
+	}
+}

$ go test ./sia/persist/sqlite -v -count=1 -run "TestSnapshotDoesNotRetainFutureObject"
=== RUN   TestSnapshotDoesNotRetainFutureObject
    logger.go:146: 2026-07-16T17:07:52.865-0400 DEBUG   sqlite  database initialized    {"sqliteVersion": "3.53.2", "schemaVersion": 9, "path": "/tmp/TestSnapshotDoesNotRetainFutureObject2403215939/001/s3d.sqlite"}
    snapshots_test.go:276: snapshot predates object: expected orphan 0e5751c026e543b2e8ab2eb06099daa1d1e5df47778f7787faab45cdf12fe3a8 to be eligible, got []
--- FAIL: TestSnapshotDoesNotRetainFutureObject (0.01s)
FAIL
FAIL    github.com/SiaFoundation/s3d/sia/persist/sqlite 0.020s

[peterjan]

@peterjan vibed this up. Seems like the fix is somewhat difficult...

diff --git a/sia/persist/sqlite/snapshots_test.go b/sia/persist/sqlite/snapshots_test.go
index ad35449..e4e37ac 100644
...
0e5751c026e543b2e8ab2eb06099daa1d1e5df47778f7787faab45cdf12fe3a8 to be eligible, got []
--- FAIL: TestSnapshotDoesNotRetainFutureObject (0.01s)
FAIL
FAIL    github.com/SiaFoundation/s3d/sia/persist/sqlite 0.020s

Thank you, I'll look into it. There's many many edges with snapshot recovery, at least this one is retaining the object which is definitely preferable over unpinning an object that we want to keep. But I'll see what the issue is and try and come up with a good fix for it.

[Copilot]

🔵 Needs a closer look

Backup cleanup can delete a pre-existing temporary file, and the API omits a critical recovery limitation.

Review details

Suppressed comments (2)

sia/persist/sqlite/backup.go:38

• This cleanup can delete a file that existed before the backup call. VACUUM INTO rejects an existing destPath + ".tmp"; the named error then triggers this unconditional removal, so an unrelated adjacent file is lost. Allocate a unique temporary path owned by this invocation (preferably in the destination directory) and only clean up that owned file.

			_ = os.Remove(tmpPath)

openapi.yml:86

• The snapshot can be created while objects are still buffered only on local disk, but those bytes are not included in the uploaded database backup. After restoring elsewhere, those database rows reference files that do not exist and cannot be uploaded or read. Preserve the prior API warning and tell callers to flush pending objects before taking a recoverable snapshot.

      description: >
        Backs up the SQLite metadata database and uploads it to the Sia network
        as a tagged, pinned object so it can be recovered later. Objects that a
        snapshot references are not unpinned until the snapshot is deleted.
        The backup does not block database reads or writes.

• Files reviewed: 25/25 changed files
• Comments generated: 0 new
• Review effort level: Balanced

[chris124567]

@peterjan Courtesy of Claude:

diff --git a/sia/sia.go b/sia/sia.go
index 51335fa..1117f33 100644
--- a/sia/sia.go
+++ b/sia/sia.go
@@ -471,7 +471,9 @@ func (s *Sia) CreateSnapshot(ctx context.Context) (_ s3.Snapshot, err error) {
    if err := s.sdk.PinObject(ctx, obj); err != nil {
        return s3.Snapshot{}, fmt.Errorf("failed to pin snapshot: %w", err)
    }
-   if err := s.store.MarkSnapshotPinned(obj.ID()); err != nil {
+   // the sync loop completes the record itself if it consumes the pin's
+   // event first, in which case the row is no longer awaiting its pin
+   if err := s.store.MarkSnapshotPinned(obj.ID()); err != nil && !errors.Is(err, objects.ErrSnapshotNotFound) {
        return s3.Snapshot{}, fmt.Errorf("failed to mark snapshot pinned: %w", err)
    }
    snap.SiaObjectID = obj.ID()
diff --git a/sia/snapshots_test.go b/sia/snapshots_test.go
index 0cbcfb9..2aeff2b 100644
--- a/sia/snapshots_test.go
+++ b/sia/snapshots_test.go
@@ -3,6 +3,7 @@ package sia_test
 import (
    "bytes"
    "compress/gzip"
+   "context"
    "encoding/json"
    "errors"
    "io"
@@ -177,6 +178,68 @@ func TestCreateSnapshot(t *testing.T) {
    }
 }
 
+// pinHookSDK runs afterPin once the wrapped SDK has pinned an object, before
+// PinObject returns to the caller.
+type pinHookSDK struct {
+   *testutil.MemorySDK
+   afterPin func(obj sdk.Object)
+}
+
+func (s *pinHookSDK) PinObject(ctx context.Context, obj sdk.Object) error {
+   if err := s.MemorySDK.PinObject(ctx, obj); err != nil {
+       return err
+   }
+   s.afterPin(obj)
+   return nil
+}
+
+// TestCreateSnapshotCompletedBySync verifies that a snapshot whose pin event
+// is consumed by a metadata sync before CreateSnapshot marks it pinned is
+// still reported as created and is not rolled back. The sync loop runs
+// independently of the request and completes the record itself when it sees
+// the event first.
+func TestCreateSnapshotCompletedBySync(t *testing.T) {
+   memSDK := testutil.NewMemorySDK()
+   hooked := &pinHookSDK{MemorySDK: memSDK}
+   backend, store := testutil.NewBackend(t, testutil.WithSDK(hooked))
+
+   // the sync loop consumes the pin's event while the request is still
+   // inside PinObject, completing the record before the request can
+   var synced bool
+   hooked.afterPin = func(obj sdk.Object) {
+       memSDK.SetEvents([]sdk.ObjectEvent{snapshotEvent(t, memSDK, obj.ID(), time.Now().Truncate(time.Second))})
+       backend.SyncMetadata(t.Context())
+       synced = true
+   }
+
+   snap, err := backend.CreateSnapshot(t.Context())
+   if err != nil {
+       t.Fatal(err)
+   } else if !synced {
+       t.Fatal("expected the sync to run during the pin")
+   }
+
+   // the snapshot is listed as pinned, not marked for deletion, and its
+   // object is still on the network
+   if snapshots, err := store.ListSnapshots(); err != nil {
+       t.Fatal(err)
+   } else if len(snapshots) != 1 {
+       t.Fatal("unexpected", len(snapshots))
+   } else if snapshots[0].ID != snap.ID {
+       t.Fatal("mismatch", snapshots[0].ID)
+   } else if snapshots[0].SiaObjectID != snap.SiaObjectID {
+       t.Fatal("mismatch", snapshots[0].SiaObjectID)
+   }
+   if ids, err := store.SnapshotsForDeletion(time.Now().Add(sia.SnapshotConfirmDelay)); err != nil {
+       t.Fatal(err)
+   } else if len(ids) != 0 {
+       t.Fatal("snapshot completed by the sync was rolled back", ids)
+   }
+   if !memSDK.Pinned(snap.SiaObjectID) {
+       t.Fatal("expected snapshot object to stay pinned")
+   }
+}
+
 // TestStuckPinningSnapshot verifies that a snapshot left awaiting its pin by a
 // dead process keeps withholding its orphans until the deletion pass confirms
 // the indexer does not hold its object, since that pin may still have been

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[peterjan]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Yeah I'm aware of this one, I have one more commit locally but I'm still soak testing it. It includes the fix you mentioned.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[ChrisSchinnerl]

got a conflict

[Copilot]

🟡 Changes recommended

Stale synchronization state can allow orphan or snapshot deletion before newly published snapshot events are processed.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (1)

sia/sia.go:592

• synced remains true between the one-minute metadata polls, so this check does not ensure that the event stream is drained when orphan deletion starts. For example, a snapshot event published after the last successful poll but before the hourly orphan pass is still unknown locally, and this pass can unpin an object referenced by that snapshot—the corruption this gate is intended to prevent. Serialize orphan processing with a fresh successful metadata drain (or use an indexer watermark/other protocol that proves no snapshot can be published before deletion) instead of treating one past sync as permanently current.

	if !s.synced.Load() {
		s.logger.Debug("deferring orphan processing until object metadata is synced")
		return
	}

• Files reviewed: 25/25 changed files
• Comments generated: 1
• Review effort level: Balanced

[ChrisSchinnerl]

got no more conflict

[peterjan]

got no more conflict

I'm sure an astra run finds 5 more race conditions that 'll never play out.

[chris124567]

edit: nvm