Document RocksDB backup & restore operations - #590
Conversation
Covers the new backup/restore Operations API operations (create_backup, list_backups, verify_backup, delete_backup, purge_backups, restore_backup) and RocksDB support for get_backup (gzipped-by-default tar; gzip=false for plain tar; remote target support). Documents when a database can be restored online vs offline (system db and component-held dbs require the server stopped), the backupPath storage option, and the 5.2 release note. Companion to HarperFast/harper#1831. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…perations # Conflicts: # release-notes/v5-lincoln/5.2.md
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
🚀 Preview DeploymentYour preview deployment is ready! 🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-590 This preview will update automatically when you push new commits. |
Companion to HarperFast/harper#1831. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
🚀 Preview DeploymentYour preview deployment is ready! 🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-590 This preview will update automatically when you push new commits. |
…_count) Matches HarperFast/harper#1831, which now returns snake_case. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
🚀 Preview DeploymentYour preview deployment is ready! 🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-590 This preview will update automatically when you push new commits. |
Backups now has its own sidebar section with an Overview (how the system works, limitations, worked examples for incremental backup/restore and get_backup snapshot download + manual restore) and an Operations page (all seven operations with Operations API and CLI examples). - Add EngineBadge component (globally registered, like VersionBadge) to tag storage-engine support: managed ops are RocksDB-only, get_backup supports RocksDB and LMDB - Remove the Backup Commands section from the CLI commands page; leave a pointer next to the volume-snapshot guidance - Trim operations-api Backup & Restore to the brief index pattern used by Logs/Certificate Management, linking to the new section - Add the backup operations to the CLI operations table - Drop the hyphenated-alias mention entirely Cross-model review (Codex) fixes: scope the whole-database claim and super_user requirement, document the single-root-store restriction, and point LMDB manual restore at storage.path rather than the default path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
🚀 Preview DeploymentYour preview deployment is ready! 🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-590 This preview will update automatically when you push new commits. |
1 similar comment
🚀 Preview DeploymentYour preview deployment is ready! 🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-590 This preview will update automatically when you push new commits. |
🚀 Preview DeploymentYour preview deployment is ready! 🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-590 This preview will update automatically when you push new commits. |
🚀 Preview DeploymentYour preview deployment is ready! 🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-590 This preview will update automatically when you push new commits. |
- overview: split manual snapshot restore into RocksDB (tar) and LMDB (.mdb) examples; mark the RocksDB walkthrough as engine-specific - cli/commands: frame "How Backups Work" as a per-engine choice (volume snapshots for LMDB, RocksDB backup engine for RocksDB) and scope the atomic-snapshot requirement to LMDB - operations-api: move Backup & Restore section down into the operational cluster (after System, before Jobs) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
🚀 Preview DeploymentYour preview deployment is ready! 🔗 Preview URL: https://preview.harper-documentation.harperfabric.com/pr-590 This preview will update automatically when you push new commits. |
| - **One storage root per database.** A database whose tables use per-table `path` storage configs spans multiple root stores and cannot be backed up with these operations. | ||
| - **Backups live on the node that created them.** The backup repository is a local directory. For disaster recovery, copy backup directories off-host, or use `get_backup` to pull snapshots from a running server. | ||
| - **`get_backup` always streams the current state.** It cannot download a historical managed backup; to move a retained backup off-host, copy its backup directory. | ||
| - **A restore is a point-in-time rollback.** In a replicated cluster, coordinate a restore with replication before bringing the node back. |
There was a problem hiding this comment.
Medium: no documented recovery step for a crashed/interrupted restore
restore_backup is destructive (purge-then-copy). If it's interrupted partway — process crash, power loss, disk error — the implementation leaves a .restoring marker next to the database directory and refuses to load that database on the next harper start (logged as an error: "Incomplete restore ... not loading it — rerun the restore to recover"). None of this is mentioned here or in operations.md: a reader who hits this after an interrupted restore has no way to know from the docs that the fix is simply to rerun restore_backup (same backup_id), not that data is lost or the install is broken.
Suggested fix: add a bullet to Limitations (or a short note under "When can a database be restored?"), e.g. "If restore_backup is interrupted before completing (crash, power loss), the affected database is left unloadable — Harper logs an incomplete-restore error and skips it on startup. Rerun restore_backup for the same database (and backup_id, if you passed one) to recover; do not attempt to load or manually repair the directory."
—
Generated by Barber AI
|
|
||
| Through a running server this runs as a background [job](../operations-api/operations.md#jobs): Harper closes the database across all worker threads, restores it, and reloads it. This works only when no loaded component is holding the database open — restoring the `system` database, or a database a component keeps open, requires running the command from the CLI with the server stopped. See [when can a database be restored?](./overview.md#when-can-a-database-be-restored) | ||
|
|
||
| From the CLI with the server stopped, `target_database=<name>` restores into a new database instead of overwriting the source. The target must not already exist; Harper picks the new database up on the next start. |
There was a problem hiding this comment.
Low: target_database constraint is stated more strictly than the implementation
"The target must not already exist" isn't quite what restoreBackupOffline enforces — it accepts an existing empty directory too (isMissingOrEmptyDir), rejecting only a non-empty one. Not harmful (following the doc's advice always works), but a reader with an empty leftover directory under that name would incorrectly conclude they need to pick a different name.
Suggested fix: "The target must not already exist (or must be empty); Harper picks the new database up on the next start."
—
Generated by Barber AI
| - **Whole-database granularity.** There is no per-table backup or restore. (The one exception: `get_backup` on an LMDB database can stream individual tables, and includes the audit store only with `include_audit`.) | ||
| - **One storage root per database.** A database whose tables use per-table `path` storage configs spans multiple root stores and cannot be backed up with these operations. | ||
| - **Backups live on the node that created them.** The backup repository is a local directory. For disaster recovery, copy backup directories off-host, or use `get_backup` to pull snapshots from a running server. | ||
| - **`get_backup` always streams the current state.** It cannot download a historical managed backup; to move a retained backup off-host, copy its backup directory. |
There was a problem hiding this comment.
RocksDB's BackupEngine does not create one self-contained directory per backup ID. The files for retained IDs are split across shared metadata/data areas in <backupPath>/<database> (and Harper keeps logs in transaction_logs/<id>). Copying what looks like one backup directory can therefore omit shared files and produce an unrestorable disaster-recovery copy; a normal recursive copy can also race create_backup, delete_backup, or purge_backups. Please say to copy the entire per-database backup repository while backup writers are quiesced (or from an atomic filesystem snapshot), and update the same wording in operations.md.
|
|
||
| <VersionBadge version="v5.2.0" /> <EngineBadge engines="RocksDB" /> | ||
|
|
||
| Verifies a managed backup's file sizes, and optionally their checksums when `verify_checksum` is `true` (slower). Through a running server this runs as a background [job](../operations-api/operations.md#jobs). `backup_id` is required. |
There was a problem hiding this comment.
The implementation delegates to backups.verify, which verifies only BackupEngine-managed RocksDB files. Harper's separately copied <repository>/transaction_logs/<backup_id> subtree is not checked, even with verify_checksum: true. As written, "managed backup integrity" and "checksum-verified backups" imply the audit log is covered. Please either extend core verification to validate that snapshot or explicitly scope this operation to RocksDB database files and qualify the broad checksum-verification claims in the overview and release notes.
| harper create_backup database=data | ||
| ``` | ||
|
|
||
| The first backup copies the entire database; each one after that only copies files that changed, so frequent backups are cheap. Schedule `create_backup` as often as your recovery point requires, and manage retention with `purge_backups`: |
There was a problem hiding this comment.
Only the RocksDB engine files are incremental/shared. Harper calls db.backup(..., { transactionLogs: true }), and rocksdb-js copies a full transaction-log snapshot into transaction_logs/<backup_id> for every backup; that portion is explicitly non-incremental. With a large audit-retention window, frequent backups can repeatedly copy substantial data and consume much more space than this claims. Please qualify every "only copies what changed" / "frequent backups are cheap" statement and explain the transaction-log cost.
|
|
||
| <VersionBadge version="v5.2.0" /> <EngineBadge engines="RocksDB" /> | ||
|
|
||
| Returns the managed backups for a database, each with its `backup_id`, `timestamp`, `size`, and `file_count`. Returns an empty array if no backups have been created yet. |
There was a problem hiding this comment.
Could we define these fields precisely? timestamp is epoch seconds, size is bytes from RocksDB BackupInfo, and file_count is the RocksDB file count. The separately stored transaction-log snapshot is excluded from both size and file_count, so callers cannot treat them as the backup's total on-disk footprint.
| | Database | Online `restore_backup` (server running) | Offline `harper restore_backup` (server stopped) | | ||
| | -------------------------------------------------- | ---------------------------------------- | ------------------------------------------------ | | ||
| | A user database not opened by any loaded component | Yes — restored in place | Yes | | ||
| | A user database that a loaded component keeps open | No — the job fails with a `409` | Yes | |
There was a problem hiding this comment.
Because restore_backup is created as a background job, the request has already returned a job_id before the job checks whether a component still holds the database open. That internal 409 becomes job status ERROR plus a message from get_job; the caller does not receive an HTTP 409 from the original restore request. Please describe this as the job entering ERROR with instructions to restore offline.
Ethan-Arrowood
left a comment
Ethan-Arrowood
left a comment
There was a problem hiding this comment.
I re-verified @kriszyp's five inline comments against harper#1831's current head — all five still stand, so this needs another round regardless. On top of those, one finding of my own that I'd treat as the blocker: blobs are undocumented end to end, and the snapshot-restore example silently loses blob data for any database with file-backed blobs.
get_backup's tar now carries blobs/<rootIndex>/<relpath> entries. The documented tar -xzf … -C ~/hdb/database/data extracts those inside the RocksDB directory, restoring no blobs to their real roots — a user follows the docs, gets a clean-looking restore, and has lost blob data. exclude_blobs, blob snapshots in managed backups, and blob purge-and-rewrite on restore are all missing, and the "all tables plus the audit/transaction log" scope line plus the 5.2 release-notes entry need the blob carve-in too.
One process note: your CHANGES_REQUESTED summary on this PR says "This looks good" — I think that landed on the wrong PR, since the inline comments are substantive. Either way a re-request will be needed to clear the state.
And the ordering question: #1831's blob work is itself under CHANGES_REQUESTED from kriszyp. These docs should re-sync once that settles — is a final drift pass planned before merge?
sent with Claude Opus 5
|
|
||
| ```bash | ||
| harper stop | ||
| mv ~/hdb/database/data ~/hdb/database/data.old # optional: keep a copy of the previous database |
There was a problem hiding this comment.
Blocker. This snapshot-restore recipe is incomplete against #1831's head and silently loses data. get_backup's tar now includes blobs/<rootIndex>/<relpath> entries; extracting the whole archive with -C ~/hdb/database/data puts those inside the RocksDB directory instead of restoring them to their real blob roots. Any database with file-backed blobs restores "successfully" and is missing its blobs.
The blob surface is undocumented more broadly: exclude_blobs, blob snapshots in managed backups, and blob purge-and-rewrite on restore all need coverage, and the scope line at the top of this page ("all tables plus the audit/transaction log") plus the 5.2 release-notes entry need the blob carve-in.
| - **Managed backups require the RocksDB storage engine.** For LMDB databases, use [`get_backup`](./operations.md#get_backup) or [volume snapshots](../cli/commands.md#backing-up-with-volume-snapshots). | ||
| - **Whole-database granularity.** There is no per-table backup or restore. (The one exception: `get_backup` on an LMDB database can stream individual tables, and includes the audit store only with `include_audit`.) | ||
| - **One storage root per database.** A database whose tables use per-table `path` storage configs spans multiple root stores and cannot be backed up with these operations. | ||
| - **Backups live on the node that created them.** The backup repository is a local directory. For disaster recovery, copy backup directories off-host, or use `get_backup` to pull snapshots from a running server. |
There was a problem hiding this comment.
Re-verified @kriszyp's point at #1831's current head — it stands. RocksDB BackupEngine backups aren't self-contained per id: there are shared data/metadata areas plus transaction_logs/<id> (and now blobs/<id>), so a per-id copy omits shared files, and a live copy races create_backup/purge_backups. This should say to copy the entire per-database repository while writers are quiesced.
| | Database | Online `restore_backup` (server running) | Offline `harper restore_backup` (server stopped) | | ||
| | -------------------------------------------------- | ---------------------------------------- | ------------------------------------------------ | | ||
| | A user database not opened by any loaded component | Yes — restored in place | Yes | | ||
| | A user database that a loaded component keeps open | No — the job fails with a `409` | Yes | |
There was a problem hiding this comment.
Still accurate at #1831's head (@kriszyp raised this): restore_backup returns a job_id first, and the verifyDatabaseClosed 409 is thrown inside the job — so callers never see an HTTP 409, they see it via get_job. This matrix row should describe job status ERROR with the restore-offline message rather than "the job fails with a 409".
| harper create_backup database=data | ||
| ``` | ||
|
|
||
| The first backup copies the entire database; each one after that only copies files that changed, so frequent backups are cheap. Schedule `create_backup` as often as your recovery point requires, and manage retention with `purge_backups`: |
There was a problem hiding this comment.
The incremental-cost framing (@kriszyp's point, still standing) ignores that each backup copies the full transaction log — and now the blob snapshot — so "cheap incremental backups" overstates it. Worth stating what is and isn't incremental.
|
|
||
| <VersionBadge version="v5.2.0" /> <EngineBadge engines="RocksDB" /> | ||
|
|
||
| Verifies a managed backup's file sizes, and optionally their checksums when `verify_checksum` is `true` (slower). Through a running server this runs as a background [job](../operations-api/operations.md#jobs). `backup_id` is required. |
There was a problem hiding this comment.
"Checksum-verified" overstates verify_backup's scope (@kriszyp's point, verified still true): the implementation delegates to backups.verify, which only checks BackupEngine-managed files. The transaction_logs/<id> snapshot — and now the blob snapshot — is never verified, even with verify_checksum: true. Needs qualifying here, in the overview's "checksum-verified" bullet, and in the release notes.
|
|
||
| Through a running server this runs as a background [job](../operations-api/operations.md#jobs): Harper closes the database across all worker threads, restores it, and reloads it. This works only when no loaded component is holding the database open — restoring the `system` database, or a database a component keeps open, requires running the command from the CLI with the server stopped. See [when can a database be restored?](./overview.md#when-can-a-database-be-restored) | ||
|
|
||
| From the CLI with the server stopped, `target_database=<name>` restores into a new database instead of overwriting the source. The target must not already exist; Harper picks the new database up on the next start. |
There was a problem hiding this comment.
"Must not already exist" is stricter than the code (@cb1kenobi's own note, verified at #1831 head): restoreBackupOffline accepts an existing empty directory via isMissingOrEmptyDir, rejecting only non-empty ones.
|
|
||
| <VersionBadge version="v5.2.0" /> <EngineBadge engines="RocksDB" /> | ||
|
|
||
| Returns the managed backups for a database, each with its `backup_id`, `timestamp`, `size`, and `file_count`. Returns an empty array if no backups have been created yet. |
There was a problem hiding this comment.
Worth defining these fields precisely (@kriszyp's point): timestamp is epoch seconds, and size/file_count come from RocksDB BackupInfo, which excludes the transaction-log and blob snapshots — so they aren't the backup's total footprint.
| - **One storage root per database.** A database whose tables use per-table `path` storage configs spans multiple root stores and cannot be backed up with these operations. | ||
| - **Backups live on the node that created them.** The backup repository is a local directory. For disaster recovery, copy backup directories off-host, or use `get_backup` to pull snapshots from a running server. | ||
| - **`get_backup` always streams the current state.** It cannot download a historical managed backup; to move a retained backup off-host, copy its backup directory. | ||
| - **A restore is a point-in-time rollback.** In a replicated cluster, coordinate a restore with replication before bringing the node back. |
There was a problem hiding this comment.
Also from @cb1kenobi's self-review and still open: a crashed restore leaves a .restoring marker, Harper then refuses to load the database, and the fix is simply rerunning restore_backup. Nothing in the docs says so — that's a bad state to hit undocumented.
3 participants
Summary
Documents the RocksDB backup & restore feature shipping in 5.2 (companion to HarperFast/harper#1831), in a dedicated Backups reference section:
reference/backups/overview.md) — how the backup system works (incremental managed backups, snapshot download), its limitations (engine support, granularity, single storage root, online-vs-offline restore matrix), and two worked examples: incremental backup/restore withharper create_backup/harper restore_backup, andharper get_backupstreaming atar.gzoff-host plus the manual restore procedure (stop Harper, extract into the database directory understorage.path).reference/backups/operations.md) — all seven operations (create_backup,list_backups,verify_backup,delete_backup,purge_backups,restore_backup,get_backup) with Operations API and CLI examples.src/components/EngineBadge.tsx, globally registered likeVersionBadge) tags storage-engine support per operation: the managed operations are RocksDB-only;get_backupsupports both RocksDB and LMDB.reference/operations-api/operations.md) — Backup & Restore trimmed to the brief index pattern used by Logs/Certificate Management, linking to the new section.reference/cli/commands.md) — the Backup Commands section is removed (content lives in Backups now); the backup operations are listed in the CLI operations table (reference/cli/operations-api-commands.md).reference/configuration/options.md) — thebackupPathstorage option (default<rootPath>/backup).release-notes/v5-lincoln/5.2.md) — Backup & Restore entry, linking to the new section.Where to look
<storage.path>/<database>(default~/hdb/database/data) — path verified againstDATABASES_DIR_NAMEin core and rocksdb-js's tar layout (entries unpack directly into a directory that opens as a RocksDB database, includingtransaction_logs/). Worth a second pair of eyes since a wrong path here damages user data.get_backuppartial exports, the single-root-store restriction is documented as a limitation, thesuper_userrequirement is scoped to server-invoked operations (offline CLI is filesystem-governed), and LMDB manual restore points atstorage.pathrather than the default-only path. The Gemini review leg failed to run (headless permission denial), so outside-model coverage is Codex-only.engines="RocksDB, LMDB") rendered in the same style as VersionBadge — open to a different design.Reviewers not assigned — leaving that to you.
Generated by an LLM (Claude Fable 5).
🤖 Generated with Claude Code