Skip to content

Archive: files skipped after routing are not removed from the final new-byte total #175

Description

@woutervanranst

Split out of #165 (review thread).

The problem

ArchiveCommandHandler Stage 3 adds each routed file's size to incrementalSize, then publishes the exact figure once routing drains:

await _mediator.Publish(new RoutingCompleteEvent(Interlocked.Read(ref incrementalSize)), cancellationToken);

JobSink.SetNewByteTotal stores that as _newByteTotal and flips _newByteTotalFinal, after which the ETA's upload denominator is treated as exact.

But a file can still be skipped after it was routed — the large-upload stage skips a file whose local read fails between hashing and upload (ArchiveCommandHandler.cs, the fs.OpenRead try/catch that publishes FileSkippedEvent). Those bytes stay in the denominator and are never credited to uploaded, so:

  • uploadEta = (newByteTotal − uploaded) / transferRate keeps a nonzero residual for a job that has actually finished uploading;
  • the final pct sits below 100 %.

Note on the ordering

RoutingCompleteEvent fires when routing drains, which is before uploads finish — so this cannot be reconciled at publish time. The sink has to learn about the skip afterwards.

Sketch

  • Add long Size = 0 to FileSkippedEvent (defaulted, so the ~existing call sites are untouched).
  • Pass upload.FileSize at the post-routing skip sites only. The hash-stage skip must keep Size = 0 — those files never reached incrementalSize.
  • Add a forwarder that subtracts from the sink's _newByteTotal.
  • Check whether the small-file/TAR upload path has an equivalent skip.

Severity

Low: needs a file to become unreadable in the seconds between hashing and upload. The visible effect is a small residual ETA and a sub-100 % pct at the end of a run — arguably honest, since that file genuinely was not archived.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions