Split out of #165 (review thread).
The problem
ArchiveCommandHandler Stage 3 adds each routed file's size to incrementalSize, then publishes the exact figure once routing drains:
await _mediator.Publish(new RoutingCompleteEvent(Interlocked.Read(ref incrementalSize)), cancellationToken);
JobSink.SetNewByteTotal stores that as _newByteTotal and flips _newByteTotalFinal, after which the ETA's upload denominator is treated as exact.
But a file can still be skipped after it was routed — the large-upload stage skips a file whose local read fails between hashing and upload (ArchiveCommandHandler.cs, the fs.OpenRead try/catch that publishes FileSkippedEvent). Those bytes stay in the denominator and are never credited to uploaded, so:
uploadEta = (newByteTotal − uploaded) / transferRate keeps a nonzero residual for a job that has actually finished uploading;
- the final
pct sits below 100 %.
Note on the ordering
RoutingCompleteEvent fires when routing drains, which is before uploads finish — so this cannot be reconciled at publish time. The sink has to learn about the skip afterwards.
Sketch
- Add
long Size = 0 to FileSkippedEvent (defaulted, so the ~existing call sites are untouched).
- Pass
upload.FileSize at the post-routing skip sites only. The hash-stage skip must keep Size = 0 — those files never reached incrementalSize.
- Add a forwarder that subtracts from the sink's
_newByteTotal.
- Check whether the small-file/TAR upload path has an equivalent skip.
Severity
Low: needs a file to become unreadable in the seconds between hashing and upload. The visible effect is a small residual ETA and a sub-100 % pct at the end of a run — arguably honest, since that file genuinely was not archived.
Split out of #165 (review thread).
The problem
ArchiveCommandHandlerStage 3 adds each routed file's size toincrementalSize, then publishes the exact figure once routing drains:JobSink.SetNewByteTotalstores that as_newByteTotaland flips_newByteTotalFinal, after which the ETA's upload denominator is treated as exact.But a file can still be skipped after it was routed — the large-upload stage skips a file whose local read fails between hashing and upload (
ArchiveCommandHandler.cs, thefs.OpenReadtry/catch that publishesFileSkippedEvent). Those bytes stay in the denominator and are never credited touploaded, so:uploadEta = (newByteTotal − uploaded) / transferRatekeeps a nonzero residual for a job that has actually finished uploading;pctsits below 100 %.Note on the ordering
RoutingCompleteEventfires when routing drains, which is before uploads finish — so this cannot be reconciled at publish time. The sink has to learn about the skip afterwards.Sketch
long Size = 0toFileSkippedEvent(defaulted, so the ~existing call sites are untouched).upload.FileSizeat the post-routing skip sites only. The hash-stage skip must keepSize = 0— those files never reachedincrementalSize._newByteTotal.Severity
Low: needs a file to become unreadable in the seconds between hashing and upload. The visible effect is a small residual ETA and a sub-100 % pct at the end of a run — arguably honest, since that file genuinely was not archived.