Repository navigation
Parallelize deep verification and report dry-run verification errors as unknown - #280
dihmandrake wants to merge 3 commits into
Conversation
healObject handles a metadata read-quorum failure before its dry-run return, so an admin heal with DryRun set still called deleteIfDangling. When the drives without the version's xl.meta all answered "not found" and outnumbered its parity blocks, the dry run deleted the version, data directory included, from every drive that still held it. A site-wide dry run reaches .minio.sys/config the same way. When DryRun is set, return the quorum error with a set-level result built from the observed read errors, before the purge. Real heals are unchanged. The production change is the first cmd/erasure-healing.go hunk of PR #280, unchanged. The tests are the PR's dry-run tests that do not depend on its parallel deep verification or its dry-run reporting change; both stay out of this commit. The tests now assert the delete counter first and the converted error for each quorum case, and drop the CheckParts stub that only the excluded tests use. Co-authored-by: Jasper Bernhardt <jasper.bernhardt@gmail.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Feng Ruohang <rh@vonng.com>
|
Post-20261026 resolution record. Implemented: c8dd5678b, dd0a2da55. The two dry-run correctness fixes were incorporated independently: object healing returns before dangling-data deletion when DryRun is set, and the peer bucket-heal request carries DryRun to the remote handler. This records only those adopted changes; the proposed parallel deep-verification and other reporting work are not claimed as merged. Verification: Result: The tested dry-run paths do not mutate object or bucket storage; ordinary healing remains active. Older peers ignore the new map key, so cluster-wide dry-run semantics require every peer to be upgraded. These changes are in public main. They are not part of the latest published server release, |
Verify each drive concurrently during deep checks while preserving result positions and normal scan behavior. Dangling-object dry-run preservation landed upstream in c8dd567; keep upstream dry-run fault tests intact alongside overlap, serial control, and inline bitrot coverage. Signed-off-by: Jasper Bernhardt <jasper.bernhardt@gmail.com>
Exercise queued bucket traversal, exact filesystem preservation during dry runs, native repair and content readback, version inventory, and cancellation barriers. Run the broader healing suite three times with and without the race detector. Signed-off-by: Jasper Bernhardt <jasper.bernhardt@gmail.com>
A verification transport or I/O error after valid metadata could otherwise be reported as healthy. Preserve all normal repair decisions and change only dry-run drive reporting. Cover disk, volume, timeout, cancellation and permission failures, plus normal/non-dry controls and skipped or empty object boundaries. Signed-off-by: Jasper Bernhardt <jasper.bernhardt@gmail.com>
0f6c6f6 to
2b6e621
Compare
Contribution Licensing (no CLA, inbound=outbound, DCO required)
This pull request contributes to PGSTY SILO (
pgsty/silo). Code contributionsare accepted under AGPL-3.0-or-later, the same license as the server.
This project does not use a CLA or require a separate Apache-2.0 license grant.
By submitting this pull request I represent that I have the right to contribute
the code changes under this repository's
GNU Affero General Public License v3.0 or later
and retain copyright in my original work. Existing copyright and license
notices remain intact; separately licensed material keeps its applicable terms.
Every commit must carry a DCO
Signed-off-bytrailer(
git commit -s) certifying theDeveloper Certificate of Origin — see
CONTRIBUTING.md.
Description
TL;DR: Verify drives concurrently during deep healing, and report unsuccessful dry-run verification as unknown instead of healthy.
Deep verification currently checks drives sequentially. This patch checks them concurrently for each object, waits for all results, then aggregates them. Normal checks remain serial, and existing congestion backoff remains between objects.
Rebased onto current main after c8dd567 and dd0a2da landed the dry-run dangling-data and peer bucket-heal safeguards. While testing this patch, I also found that unsuccessful deep verification after valid metadata can be reported as healthy during dry runs; this patch reports unsuccessful dry-run verification as unknown without changing normal repair decisions.
AI disclosure: The implementation and tests were written by AI. I do not understand Silo/MinIO internals well enough to independently establish correctness. It worked in my deployment and made deep heals significantly faster, which is why I wanted to contribute it upstream.
Motivation and Context
I wanted deep heals and periodic integrity checks to finish faster on my three-node home-lab cluster with six slower HDDs. Native deep heals were taking much longer than expected even though CPU use was low and most disks were largely idle. Investigation showed that
checkObjectWithAllPartswaits for one drive's verification before starting the next. Scanner-speed and replacement-drive-worker settings do not remove that per-object serialization.Experiments with concurrent native per-drive verification on disposable data showed the benefit of using the disks together. I explored external scanners and RPC coordinators, but they added code and coverage decisions around inline data, versions and metadata. Moving the parallelism into Silo lets the native deep-heal path retain those decisions. The goal is to make the verification phase significantly faster on slower drives while keeping external scheduling and notifications simple.
How to test this PR?
The changes include focused regressions and extended tests for cancellation, multipart and versioned objects, corruption detection, dry-run preservation, and repair/readback. Healing tests passed three times with the race detector after rebasing onto current upstream.
The same patch on an earlier base worked in my three-node home-lab deployment: a bounded three-hour trial checked approximately 1.6 TB of logical data without peer restarts. It reported ten objects with missing copies; this was not a complete clean audit.
Compatibility impact
No API schemas, configuration variables, metrics, headers, routes, on-disk metadata formats, or module/import paths change. Existing MinIO-compatible identifiers are preserved. Deep checks become concurrent per object; dry-run error reporting changes as described above. Reverting the patch requires no storage migration.
Types of changes
Checklist:
git commit -s) per the DCOcommit-idorPR #here)make verifierspassesmake buildpasspgsty/silo.pgsty.com, if needed