[WIP][RAPTOR-18075] feat(artifact): add dr artifact code doctor - #866
Draft
ajalon1 wants to merge 14 commits into
Draft
[WIP][RAPTOR-18075] feat(artifact): add dr artifact code doctor#866ajalon1 wants to merge 14 commits into
ajalon1 wants to merge 14 commits into
Conversation
|
🎫 Jira: |
ajalon1
commented
Aug 28, 2026
Comment on lines
+20
to
+21
| // live in their own packages and plug into the Runner. This keeps the layer | ||
| // reusable for a future top-level "dr doctor". |
Contributor
Author
There was a problem hiding this comment.
example package checks I was thinking of:
dr plugin doctor
dr template doctor
dr dotenv doctor
dr auth doctor
So many doctors.
Add internal/doctor: a state-agnostic check-and-report framework that a
future top-level `dr doctor` can reuse. No wapi/sync/workload imports.
- Status enum (OK/WARN/FAIL/SKIP), Result{CheckID, Status, Summary,
Remedy, Details, Fixable}, Check interface (ID, Name, Run(ctx)).
- Runner executes checks in caller order, stamps CheckIDs, and Report
derives counts, the lowercase ok|warn|fail verdict, and the exit code
(1 iff any FAIL).
- Text reporter: header (project dir + artifact or "not linked"),
CHECK/STATUS/DETAIL lipgloss table with tui.TableBorderStyle, remedies
for non-OK rows, summary line with counts + verdict.
- JSON reporter: single pure-JSON object with the pinned schema —
absolute projectDir, artifactId null when unlinked (empty string
normalized), uppercase per-check status, checks in runner order,
summary counts matching the tally, and an optional actions[] array
(omitted entirely for read-only runs, present for repair runs).
TDD with testify; 19 tests pass under -race; task lint clean on
linux/darwin/windows.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
…state Add internal/workload/doctor: the six read-only local checks for `dr artifact code doctor`, built on the generic internal/doctor framework and the existing wapi/sync primitives. Checks perform zero local writes and zero network calls; repairs remain behind --fix/--relink (later milestone). Checks (pinned fixed order): - wapi.presence: state dir exists at current or legacy location - wapi.config: LoadConfig; missing/corrupt/semantic FAIL carries the absolute path in details.path - wapi.manifest: LoadManifest; same FAIL semantics, independent of config - wapi.config-manifest-divergence: config lastSyncedVersionId vs manifest syncedVersionId incl. nil-ness and empty->nil normalization - wapi.rollback: stale .rollback/ tree at current or legacy location, empty dir included - wapi.lock: NON-CREATING probe (open without O_CREATE + non-blocking exclusive flock): absent -> OK, acquirable -> OK (released within Run), held -> FAIL, permission/IO open error -> WARN "cannot inspect" (never misreported as held), Windows -> SKIP via an injected goos seam SKIP cascades: presence FAIL skips everything; config FAIL skips divergence (remote checks will skip too once they exist) while manifest/rollback/lock still run; manifest FAIL skips divergence only. Canonical remedy strings live in remedies.go and are shared by both reporters. Two small exported helpers added to existing packages: wapi.ManifestPath and sync.LockFileName. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Register the doctor command under `artifact code` (inheriting the artifact tree's DATAROBOT_CLI_FEATURE_WORKLOAD gate) and run the six local sync-state checks through the generic framework Runner. - Flags: --dir (default ".", resolved via filepath.Abs with final-component symlink resolution, never prompts), --output-format (text|json), and --yes/-y read from cobra with the DATAROBOT_CLI_NON_INTERACTIVE env var bound via viperx.BindEnv only. - Soft auth probe inside RunE: remote credentials are resolved from the env pair or the stored drconfig profile without prompting, without calling auth.EnsureAuthenticatedE (no login wizard), and without ever writing drconfig.yaml. Local checks run regardless. - Exit 1 iff any check FAILs, via cli.ErrSilent with runtime-set SilenceErrors so the rendered report is not followed by a cobra error echo; usage errors keep their explanatory message. - Text and JSON reporters per the pinned output contract; read-only runs write nothing (verified: state byte-identical, sync.lock never created). Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
…ive artifact Add the four remote checks to internal/workload/doctor and wire them into `dr artifact code doctor`, keeping the pinned 6-local-then-4-remote order: - remote.artifact-exists: 404 -> FAIL "linked artifact not found (deleted?)" with the `doctor --relink` remedy; any other fetch failure -> SKIP. - remote.artifact-locked: locked -> WARN (sync execute refused, preview allowed), fixable=false, never FAIL. - remote.catalog-mismatch: config.CatalogID vs artifact codeRef.CatalogID, FAIL on mismatch (either side absent counts as divergent), both-absent OK. - remote.drift: codeRef.CatalogVersionID vs config.LastSyncedVersionID, WARN on drift with a `sync --dry-run` remedy, no baseline -> OK. All four share ONE artifact snapshot per run through a lazy remoteSnapshot backed by an injected ArtifactGetter seam (production: workload.GetArtifact; tests: fake store with call-count assertion), so a run performs exactly one GetArtifact and a vanished artifact collapses the dependent checks to SKIP. Remote error mapping is pinned: 404 -> FAIL-as-deleted on artifact-exists only; ANY other failure (401/403, 5xx, timeout, unreachable, unauthenticated) -> SKIP with a `dr auth login`/connectivity remedy that never mentions --relink. Empty-string codeRef fields normalize to nil on both sides. The checks stay pure diagnostics: zero local writes, zero server writes. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Implement `dr artifact code doctor --fix`: safe local-only auto-repairs
with a global safety gate.
Global safety gate: the sync lock is probed first (non-creating probe,
same logic as the wapi.lock check). When a live process holds the lock
(or it cannot be inspected), ALL repairs are skipped with reason "sync
in progress" — a sync writes manifest.json in its final phase, so no
repair is safe underneath it.
Three repairs, each reported as an actions[] entry
(performed|skipped-with-reason|not-needed) in both text and JSON:
1. Rebuild manifest from config — Manifest{Version:1, SyncedAt
nil-iff-version-nil, SyncedVersionID: cfg.LastSyncedVersionID,
Files:{}}. Requires a valid config; corrupt config skips with a
re-init remedy.
2. Clear interrupted rollback via sync.RestoreStaleIfPresent — restores
backed-up files to the working tree, removes .rollback/.
3. Clear lock only if acquirable — AcquireSyncLock then release; absent
lock file reports not-needed (never created).
A repair failing mid-write is reported skipped with the error as
reason; remaining repairs still attempt. --fix re-runs the full check
suite and reports post-fix state; exit code reflects POST-fix state.
No-op on a healthy project with explicit "nothing to fix" output.
--fix never touches the server. --fix and --relink are mutually
exclusive (usage error, exit 1, no checks run).
New files:
- internal/workload/doctor/fix.go: RunFix repair suite + three repair ops
- internal/workload/doctor/fix_test.go: 20 unit tests covering all
VAL-FIX scenarios (healthy no-op, missing/corrupt/divergent manifest,
corrupt config, rollback restore, lock safety matrix, held lock gate,
partial failure, working-tree preservation, windows gate, multiple
problems in one run)
- cmd/artifact/code/doctor/fix_cmd_test.go: command-level tests for
fix flows (healthy no-op, missing manifest, corrupt config, held
lock, rollback restore, idempotent second run, mutual exclusion,
deleted artifact + missing manifest composition)
- cmd/artifact/code/doctor/heldlock_{unix,windows}_test.go: platform-
specific held-lock helpers for command-level tests
Modified files:
- cmd/artifact/code/doctor/cmd.go: --fix flag, mutual-exclusion check,
fix-then-rerun wiring, actions in report
- internal/doctor/text.go: writeActions section in text reporter
- internal/workload/doctor/lock.go: extracted newLockCheckWithGoos for
fix's gate probe reuse
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Implement `dr artifact code doctor --relink <new-artifact-id>`: repoints
the project at a new artifact with a fresh sync baseline reset, without
deleting the state directory.
Safety gates (every abort leaves state byte-identical):
- Lock held by live process → abort "sync in progress"
- Not-linked project → error pointing to init (short-circuits before fetch)
- API unreachable/unauthenticated → error abort (relink hard-requires API)
- Target 404 → abort
- Target locked → abort (cannot sync to a locked artifact)
- Target Artifact.Type != "service" → abort (cross-type lineage refused)
- Same-id relink → allowed, warned, BASE reset
Confirm prompt defaults to No (bespoke [y/N] where empty Enter declines;
NOT reader.AskYesNo). Non-interactive (--yes or non-TTY) prints the warning
to stderr and proceeds. Decline/Ctrl-C/EOF aborts with state untouched.
On confirm: config rewritten (artifactId=new, catalogId=new codeRef.CatalogID
normalized empty→nil, lastSyncedVersionId=null), manifest reset to empty
BASE, history.log appended {op:relink, from, to, ts}, working tree untouched,
zero server writes. Post-relink checks re-run and report; actions[] included.
--fix and --relink are mutually exclusive (cobra MarkFlagsMutuallyExclusive
plus belt-and-suspenders guard).
17 unit tests in internal/workload/doctor/relink_test.go cover all gates and
the happy path. 16 command-level tests in cmd/artifact/code/doctor/relink_cmd_test.go
cover the CLI surface (JSON purity, actions array, exit codes, flag shapes).
Manually verified end-to-end on staging: create A → init → delete A → doctor
FAILs with relink remedy → --relink B → doctor healthy (all 10 OK) → sync
targets B (acceptance criterion #1) → second sync no-op → clean up both
fixtures (sweep confirms 0 doctor-test-* remaining).
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
…-authoring guide Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Remove the "Delete %s to re-init." advice from all init branches. The
already-linked check now fetches the linked artifact and branches:
- Corrupt config (unreadable linked state): report unreadable, remedy
names `doctor --fix`, never deletion. No fetch attempted.
- Gone (404) or catalog mismatch: interactive → offer to relink in place
(prompt for new artifact id, then run the doctor --relink path incl.
warn/confirm and safety gates); non-interactive → print guidance naming
`doctor --relink <new-id>`.
- Healthy (or non-404 error): keep abort behavior, point to `doctor` for
diagnosis. No delete advice anywhere.
JSON mode: the already-linked abort emits a single JSON object on stdout
{status:error, error:already-linked, artifactId:<id|null>, remedy:<guidance>}
with human text on stderr, exit 1. HTML escaping is disabled so remedy
strings with <new-artifact-id> survive verbatim (matching the doctor's
JSON reporter).
The interactive offer and confirm prompts are testable via package-level
seams (offerRelinkFn, makeRelinkConfirmFn, isInteractiveFn). The relink
reuses internal/workload/doctor.RunRelink with the same safety gates
(lock probe, 404/locked/type checks, warn+confirm default-No).
Fresh-init path is byte-identical (unchanged output, state files, history
entry). Legacy .wapi/-only projects follow the same branches (init already
calls EnsureMigrated first).
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Three non-blocking issues from the M1 scrutiny validator: 1. internal/doctor/text.go: make error handling consistent across all fmt.Fprint* calls in WriteText and its helpers. The header Fprintf, writeRemedies, writeActions, and writeSummary now all check and propagate errors, matching the existing writeChecksTable posture. 2. internal/doctor/reporters_test.go: add a raw-bytes assertion (before json.Unmarshal) that <, >, and & survive verbatim in the marshaled output, pinning the SetEscapeHTML(false) behavior documented in json.go's doc comment. The previous post-Unmarshal assertion was escaping-invariant and proved nothing. 3. cmd/artifact/code/doctor/cmd.go resolveProjectDir: replace the basename-only symlink heuristic (filepath.Base(resolved) == filepath.Base(abs)) with os.Lstat-based detection on the final component, so a symlink whose target directory shares the link's basename is still detected. Intermediate symlinks (e.g. macOS /tmp → /private/tmp) continue to stay as written. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
…ndings
Bundle 12 non-blocking code-quality findings from the M3 scrutiny and
user-testing rounds onto the ticket branch before the M4 stacked branch is
created.
Scrutiny findings (10):
1. doctor-fix fixLock benign TOCTOU: documented why the stat-then-acquire
ordering is safe (flock is per-open-file-description; post-fix check
suite reports honestly regardless).
2. fixLock bare 'not-needed' reason: now reports "verified acquirable
(acquired and released); no holder detected" for the acquire+release probe.
3. doctor-relink RunRelink mid-write non-atomicity: documented the invariant
in the relinkWrite doc comment (state untouched until first write; on
mid-write failure run doctor --fix; each individual write is atomic via
write-temp-then-rename, but the sequence is not transactional).
4. --relink '' (explicit empty value) is now a usage error (exit 1) instead
of silently behaving as read-only. The repair phase gates on
Flags().Changed so an empty value is rejected before any work begins.
5. RunRelink dereferences opts.Confirm without nil guard: added nil guard
(nil Confirm = decline). Fixed gate-numbering comment drift in the doc
comment (same-id relink is not a numbered gate; it's a post-gate note).
6. init runRelinkFromInit passes context.Background(): now threads
cmd.Context() through to RunRelink so cobra cancellation propagates.
7. init corrupt-config branch: now wraps the underlying LoadConfig error
with %w and includes the config path in both text and JSON-mode stderr
(previously JSON-mode stderr omitted the path that text mode included).
8. init empty entry at new-artifact-ID prompt: documented that empty entry
= decline is the intended default-No UX (consistent with dirprompt.Ask
contract and the bespoke [y/N] confirm prompt).
9. isNotFound/isCatalogMismatch predicate duplication: exported
IsNotFound and IsCatalogMismatch from internal/workload/doctor and
updated cmd/artifact/code/init to use the shared implementations,
removing the duplicated local copies.
10. reader.go:80 bare newline to stdout on read error: added a code comment
noting the cosmetic JSON-purity edge (Ctrl-C at an interactive prompt
in JSON mode can emit a stray newline on stdout; abort paths emit no
JSON anyway). Behavior intentionally unchanged.
User-testing findings (2):
11. Corrupt-config wapi.manifest remedy: WONTFIX — the remedy string is
contract-pinned as canonical per check ID (one exact string owned by
internal/workload/doctor, reused in text and JSON). The wapi.manifest
check always shows RemedyManifest regardless of config state; the
--fix action's skip reason (which points to re-init) is a separate
output in the actions array, not the check remedy.
12. Fresh-init TEXT mode 'Error: Command not found' on stderr: WONTFIX —
pre-existing, not introduced by the init relink-offer change (the
relink-offer commit only touched the already-linked path, not the
fresh-init path). Out of mission scope.
Tests added/updated for behavior changes (items 4, 6, 7, 9):
- TestRunE_RelinkEmptyValue_UsageError: --relink '' exits 1 with usage error
- TestRunE_RelinkUnchanged_ReadOnlyRun: plain read-only run unaffected
- TestRunE_AlreadyLinked_CorruptConfig: error wraps LoadConfig + includes path
- TestRunE_AlreadyLinked_CorruptConfig_JSON: JSON stderr includes config path
- TestRunE_AlreadyLinked_RelinkAccept_PropagatesContext: context propagation
- TestIsNotFound: shared 404 predicate (nil, 404, 500, wrapped 404, plain)
- TestIsCatalogMismatch: shared mismatch predicate (anchor-on-local rule)
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
…y nits Four minimal, scoped edits from the misc-cleanup scrutiny synthesis: 1. TestRunE_AlreadyLinked_RelinkAccept_PropagatesContext (init/cmd_test.go) was vacuous — its assertion held regardless of cmd.Context() vs context.Background(). Added a runRelinkFn package seam (matching the existing offerRelinkFn/makeRelinkConfirmFn pattern) and rewrote the test to capture the context via the seam and assert it equals cmd.Context() using a sentinel value — genuinely falsifiable now. 2. Pinned the fixLock probe-path reason string: added assert.Contains(..., "verified acquirable") on the lock action's Reason in TestRunFix_LockAcquirable_VerifiedNotNeeded. 3. RunRelink's nil-Confirm decline now reports reason 'no confirm function provided; relink declined as a safety default' instead of the misleading 'declined by user' (no prompt happened). 4. Fixed curly-quote typo in TestRunE_RelinkEmptyValue_UsageError doc comment (right double-quote → straight quote). Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Add `dr artifact code doctor` to the user-facing command references that enumerate `artifact code` subcommands (docs/commands/artifact.md and docs/commands/README.md). The repo-root README stays high-level and does not enumerate subcommands, so the update belongs in docs/commands/. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
- Strip mission validation criteria IDs (VAL-*) from doc comments across internal/doctor, internal/workload/doctor, and cmd/artifact/code; the IDs are meaningless outside the mission that produced them. Replaced with short plain-language behavior notes where the code doesn't speak for itself, and deleted outright where it does. - Rename runDoctor to pageDoctor (docs/development/doctor.md diagram updated to match). - Remove the hand-rolled --fix/--relink mutual-exclusion reimplementation from validateRepairFlags; cobra's MarkFlagsMutuallyExclusive error now stands alone. validateRepairFlags retains only the empty --relink value check, and its test asserts cobra's generic message. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
ajalon1
force-pushed
the
aj/RAPTOR-18075-artifact-doctor
branch
from
August 31, 2026 19:52
e180006 to
211b745
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
RATIONALE
RAPTOR-18075: when a project's
.datarobot/workload/(legacy.wapi/) sync state breaks — deleted artifact, interrupted sync/rollback, stale lock, catalog mismatch — users get cryptic failures frominit/syncand no recovery path short of deleting state by hand.dr artifact code doctordiagnoses the state read-only and offers safe, gated repairs.ARCHITECTURE
The feature is layered in three packages: the command layer wires Cobra flags and the soft auth probe, the workload layer owns the wapi-specific checks and repairs, and the generic framework layer owns the ordered runner, the reporters, and exit-code aggregation. The four remote checks share exactly one
GetArtifactper run through the injectedArtifactGetterseam; SKIP cascades (presence FAIL skips all; config FAIL skips divergence + remote) are honest per-run observations. The--fix/--relinkrepair phase is a side branch gated by a global held-lock safety check.flowchart TD subgraph CMD["cmd/artifact/code/doctor (cobra wiring)"] F["flags: --dir, --output-format, --yes, --fix, --relink"] P["soft auth probe (non-fatal, no login wizard)"] RUN["runDoctor"] end subgraph WL["internal/workload/doctor (wapi checks + repairs)"] L["6 local checks: presence, config, manifest, divergence, rollback, lock"] R["4 remote checks: artifact-exists, artifact-locked, catalog-mismatch, drift"] S["one GetArtifact snapshot (ArtifactGetter seam)"] end subgraph CORE["internal/doctor (generic framework)"] RN["Runner (ordered execution)"] REP["Report + exit-code: 1 if any FAIL"] T["text reporter (lipgloss table)"] J["JSON reporter (pure stdout)"] end API["DataRobot API (read-only GET)"] F --> P P --> RUN RUN --> L RUN --> R R --> S S --> API L --> RN R --> RN RN --> REP REP --> T REP --> J P -.->|"offline: remote checks SKIP"| R L -.->|"presence FAIL: skip all"| SK1["remaining checks SKIP"] L -.->|"config FAIL: skip divergence + remote"| SK2["divergence + remote SKIP"] subgraph REPAIR["repair phase (side branch)"] G["global held-lock safety gate"] FIX["--fix: manifest, rollback, lock"] REL["--relink: repoint + fresh BASE"] end RUN -.-> G G -.->|"live holder: skip all repairs"| SK3["all repairs skipped"] G -.-> FIX G -.-> REL REL -.->|"hard-requires API"| APIAdding a new check is five small steps; both reporters pick it up automatically because they render whatever the ordered
Runnerreturns. Check order is pinned and user-visible, and remedies are canonical strings owned byinternal/workload/doctor.flowchart TD A["1. implement doctor.Check: ID, Name, Run(ctx) -> Result"] B["2. add a canonical remedy constant in remedies.go"] C["3. register the constructor in the check-list composition (LocalChecks/RemoteChecks)"] D["4. place it at a deliberate position — order is pinned and user-visible"] E["5. write tests with the fake seams: initProject temp state, ArtifactGetterFunc"] F["both reporters render it automatically"] A --> B B --> C C --> D D --> E E --> FFull doc with the check-authoring guide and details on the SKIP cascades, repair semantics, and output contract:
docs/development/doctor.mdon this branch.CHANGES
internal/doctor(new): generic, state-agnostic doctor framework — OK/WARN/FAIL/SKIP statuses, ordered runner, text (lipgloss table) + JSON reporters. Reusable for a future top-leveldr doctor.internal/workload/doctor(new): 6 local checks (wapi.presence,wapi.config,wapi.manifest,wapi.config-manifest-divergence,wapi.rollback,wapi.lockwith a non-creating probe) + 4 remote checks (remote.artifact-exists,remote.artifact-locked,remote.catalog-mismatch,remote.drift) sharing exactly oneGetArtifactper run. Sensible SKIP cascades; non-404 remote failures report SKIP with a connectivity remedy, never "deleted".cmd/artifact/code/doctor(new): command wiring under the existing workload feature gate; soft auth probe (never launches the login wizard, never writesdrconfig.yaml);--output-format jsonwith stdout kept pure JSON; exit 1 iff any check FAILs.--fix: safe auto-repairs — rebuild manifest from config, restore interrupted rollback (.rollback/→ working tree), clear a stale lock only if acquirable — all behind a global gate that skips every repair while a live process holds the sync lock. Reports per-repairactions[]and re-runs the full check suite so output/exit code reflect the post-fix state.--relink <new-artifact-id>: in-place repoint with a fresh sync BASE — aborts byte-identical on held lock / unreachable API / 404 / locked / non-servicetarget / not-linked; confirm prompt defaults to No; rewritesconfig.json(newartifactId+catalogId,lastSyncedVersionId=null), resetsmanifest.json, appends a{op:relink, from, to, ts}history entry; working tree and server untouched.--fixand--relinkare mutually exclusive.TESTING
task lintgreen on linux/darwin/windows.doctor-test-*fixtures: healthy (never-synced), deleted-artifact 404 → FAIL + relink remedy, catalog mismatch, drift, offline (unreachable endpoint → remote SKIP, exit 0), full relink flow ending withsynctargeting the new artifact, and interactive confirm accept/decline/Ctrl-C via a virtual PTY. All fixtures deleted.TODO
initchange: when a project is already linked to a gone/mismatched artifact, offerdoctor --relinkin place of the current "Delete … to re-init." advice (in progress on this branch)NOTES
initchange lands. A separate stacked PR will carry five additional informational checks (legacy.wapi/migration hint,.drignorepresence, history.log parse health, no-codeRef hint, orphaned.checkouts/) so this diff stays scoped to RAPTOR-18075.doctorruns perform zero local writes and zero server writes (verified by checksum/updatedAtevidence).RELATED