Skip to content

Parallelize deep verification and report dry-run verification errors as unknown - #280

Open
dihmandrake wants to merge 3 commits into
pgsty:mainfrom
dihmandrake:upstream-parallel-deep-scan
Open

dihmandrake wants to merge 3 commits into
pgsty:mainfrom
dihmandrake:upstream-parallel-deep-scan

Conversation

@dihmandrake

@dihmandrake dihmandrake commented Nov 2, 2026 •

Copy link
Copy Markdown

Contribution Licensing (no CLA, inbound=outbound, DCO required)

This pull request contributes to PGSTY SILO (pgsty/silo). Code contributions
are accepted under AGPL-3.0-or-later, the same license as the server.
This project does not use a CLA or require a separate Apache-2.0 license grant.
By submitting this pull request I represent that I have the right to contribute
the code changes under this repository's
GNU Affero General Public License v3.0 or later
and retain copyright in my original work. Existing copyright and license
notices remain intact; separately licensed material keeps its applicable terms.
Every commit must carry a DCO Signed-off-by trailer
(git commit -s) certifying the
Developer Certificate of Origin — see
CONTRIBUTING.md.

Description

TL;DR: Verify drives concurrently during deep healing, and report unsuccessful dry-run verification as unknown instead of healthy.

Deep verification currently checks drives sequentially. This patch checks them concurrently for each object, waits for all results, then aggregates them. Normal checks remain serial, and existing congestion backoff remains between objects.

Rebased onto current main after c8dd567 and dd0a2da landed the dry-run dangling-data and peer bucket-heal safeguards. While testing this patch, I also found that unsuccessful deep verification after valid metadata can be reported as healthy during dry runs; this patch reports unsuccessful dry-run verification as unknown without changing normal repair decisions.

AI disclosure: The implementation and tests were written by AI. I do not understand Silo/MinIO internals well enough to independently establish correctness. It worked in my deployment and made deep heals significantly faster, which is why I wanted to contribute it upstream.

Motivation and Context

I wanted deep heals and periodic integrity checks to finish faster on my three-node home-lab cluster with six slower HDDs. Native deep heals were taking much longer than expected even though CPU use was low and most disks were largely idle. Investigation showed that checkObjectWithAllParts waits for one drive's verification before starting the next. Scanner-speed and replacement-drive-worker settings do not remove that per-object serialization.

Experiments with concurrent native per-drive verification on disposable data showed the benefit of using the disks together. I explored external scanners and RPC coordinators, but they added code and coverage decisions around inline data, versions and metadata. Moving the parallelism into Silo lets the native deep-heal path retain those decisions. The goal is to make the verification phase significantly faster on slower drives while keeping external scheduling and notifications simple.

How to test this PR?

The changes include focused regressions and extended tests for cancellation, multipart and versioned objects, corruption detection, dry-run preservation, and repair/readback. Healing tests passed three times with the race detector after rebasing onto current upstream.

MINIO_API_REQUESTS_MAX=10000 CGO_ENABLED=1 go test -race -tags kqueue,dev ./cmd -run '^(TestNativeAudit.*|TestHealing.*|TestHeal.*)$' -count=3 -timeout=5m

The same patch on an earlier base worked in my three-node home-lab deployment: a bounded three-hour trial checked approximately 1.6 TB of logical data without peer restarts. It reported ten objects with missing copies; this was not a complete clean audit.

Compatibility impact

No API schemas, configuration variables, metrics, headers, routes, on-disk metadata formats, or module/import paths change. Existing MinIO-compatible identifiers are preserved. Deep checks become concurrent per object; dry-run error reporting changes as described above. Reverting the patch requires no storage migration.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Optimization (provides speedup with no functional changes)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • All commits are signed off (git commit -s) per the DCO
  • Fixes a regression (If yes, please add commit-id or PR # here)
  • Unit tests added/updated
  • make verifiers passes
  • Relevant package tests and make build pass
  • Compatibility and rollback impact documented
  • Internal documentation updated
  • Public documentation update opened in pgsty/silo.pgsty.com, if needed

Vonng added a commit that referenced this pull request Nov 7, 2026
healObject handles a metadata read-quorum failure before its dry-run
return, so an admin heal with DryRun set still called deleteIfDangling.
When the drives without the version's xl.meta all answered "not found"
and outnumbered its parity blocks, the dry run deleted the version,
data directory included, from every drive that still held it. A
site-wide dry run reaches .minio.sys/config the same way.

When DryRun is set, return the quorum error with a set-level result
built from the observed read errors, before the purge. Real heals are
unchanged.

The production change is the first cmd/erasure-healing.go hunk of
PR #280, unchanged. The tests are the PR's dry-run tests that do not
depend on its parallel deep verification or its dry-run reporting
change; both stay out of this commit. The tests now assert the delete
counter first and the converted error for each quorum case, and drop
the CheckParts stub that only the excluded tests use.

Co-authored-by: Jasper Bernhardt <jasper.bernhardt@gmail.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Feng Ruohang <rh@vonng.com>
Vonng added a commit that referenced this pull request Nov 7, 2026
Takes the dry-run part of PR #280 and credits its author. The parallel deep-verification part stays in #280 for separate review.

Vonng commented Nov 9, 2026 •

Copy link
Copy Markdown
Member

Post-20261026 resolution record.

Implemented: c8dd5678b, dd0a2da55.

The two dry-run correctness fixes were incorporated independently: object healing returns before dangling-data deletion when DryRun is set, and the peer bucket-heal request carries DryRun to the remote handler. This records only those adopted changes; the proposed parallel deep-verification and other reporting work are not claimed as merged.

Verification: TestNativeAuditDryRunPreservesDanglingFiles, TestNativeAuditNonDryRunStillDeletesDangling, and TestPeerHealBucketDryRun passed again in fresh focused verification on 2026-11-09, using a clean checkout of local integration commit bb31c2618. The two implementation commits were also verified as ancestors of public main 16738c230. This supplements the preserved 2026-11-08 results; no full release acceptance is claimed.

Result: The tested dry-run paths do not mutate object or bucket storage; ordinary healing remains active. Older peers ignore the new map key, so cluster-wide dry-run semantics require every peer to be upgraded.

These changes are in public main. They are not part of the latest published server release, RELEASE.2026-10-26T00-00-00Z.

Verify each drive concurrently during deep checks while preserving result positions and normal scan behavior. Dangling-object dry-run preservation landed upstream in c8dd567; keep upstream dry-run fault tests intact alongside overlap, serial control, and inline bitrot coverage.

Signed-off-by: Jasper Bernhardt <jasper.bernhardt@gmail.com>
Exercise queued bucket traversal, exact filesystem preservation during dry runs, native repair and content readback, version inventory, and cancellation barriers. Run the broader healing suite three times with and without the race detector.

Signed-off-by: Jasper Bernhardt <jasper.bernhardt@gmail.com>
A verification transport or I/O error after valid metadata could otherwise be reported as healthy. Preserve all normal repair decisions and change only dry-run drive reporting.

Cover disk, volume, timeout, cancellation and permission failures, plus normal/non-dry controls and skipped or empty object boundaries.

Signed-off-by: Jasper Bernhardt <jasper.bernhardt@gmail.com>
@dihmandrake
dihmandrake force-pushed the upstream-parallel-deep-scan branch from 0f6c6f6 to 2b6e621 Compare November 15, 2026 14:15
@dihmandrake dihmandrake changed the title Parallelize deep verification and fix dry-run safeguards Parallelize deep verification and report dry-run verification errors as unknown Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants