Optionally expose per-peer bitfields in PeerStats - #644
adamgreenwell wants to merge 3 commits into
Conversation
| /// this can surface as. Nothing equivalent happens on Unix, where an open | ||
| /// handle does not restrict other opens, so the check is compiled out. | ||
| #[cfg(windows)] | ||
| fn is_sharing_violation(e: &std::io::Error) -> bool { |
There was a problem hiding this comment.
| /// check. `min` guards the slice: the byte length is validated on ingest, so | ||
| /// the two should agree, but a panic here would take down a stats call. | ||
| fn count_have_pieces(bitfield: &crate::type_aliases::BF, total_pieces: u32) -> u32 { | ||
| let end = (total_pieces as usize).min(bitfield.len()); |
There was a problem hiding this comment.
can we just fix the original bitfield length "at ingest" as it's called in description instead of here?
There was a problem hiding this comment.
Done — on_bitfield clears the padding now, count_have_pieces is gone, and count_ones() is correct everywhere. from_peer lost its total_pieces parameter as a result, so the diff got smaller.
One thing it turned up, worth a look: on_have is a second way in. It bounds the index with get_mut, which checks against the bitfield's length — byte-padded, so up to 7 over the piece count — so a peer could set a padding bit one Have after ingest and put the count back where it was. It's bounded by the piece count now, in the same commit. That also stops a peer claiming a piece that does not exist.
Neither was harmful before this: every other reader either slices to total_pieces first, as on_bitfield's own "peer has full torrent" check does, or indexes by an already-validated piece id. Which is presumably why it went unnoticed — the padding is only visible to something that counts the whole bitfield.
It's a separate commit (Clear a peer bitfield's padding on ingest) so you can take it on its own, independently of the stats work.
| _ => None, | ||
| }, | ||
| have_bitfield: match state { | ||
| PeerState::Live(l) if include_bitfield => Some(l.bitfield.as_raw_slice().to_vec()), |
There was a problem hiding this comment.
why convert to vec and not just clone?
There was a problem hiding this comment.
PeerStats derives Serialize and bitvec is built without its serde feature, so BitBox has no Serialize impl — l.bitfield.clone() does not compile.
Enabling the feature would also put bitvec in the public API: a dependent would need a matching bitvec version just to name the field's type. The allocation is the same either way, so Vec<u8> looked like the cheaper commitment. I have put that reasoning in a comment on the field, since it is not obvious from the call site.
Both are part of the signature of the public TorrentStateLive::per_peer_stats_snapshot, but live in a private module and are not re-exported. A dependent crate can therefore obtain a filter only via Default::default() and cannot name the type to set a field on it, nor name the returned PeerStats in its own signatures.
A bitfield is byte-padded, so the bits past total_pieces are spare. The spec says a peer zeroes them, but on_bitfield validates only the byte length and then stores the peer's bytes verbatim, so a peer can set them. on_have is a second way in: it bounds the index with get_mut, which checks against the bitfield's length rather than the piece count, and that length overshoots by up to 7. Neither is harmful today, because every reader either slices to total_pieces first -- as on_bitfield's own "peer has full torrent" check does -- or indexes by an already-validated piece id. That is also why it has gone unnoticed: the padding is only visible to something that counts the whole bitfield. Clear it once at ingest instead, and bound on_have by the piece count. count_ones() then means "pieces this peer has" everywhere, without each caller having to know about the padding, and a peer can no longer claim a piece that does not exist.
4d605c2 to
fdb7ee1
Compare
Adds have_bitfield and have_pieces to PeerStats, both behind a new PeerStatsFilter::include_bitfield, off by default. Nothing is computed or copied unless it is set, so existing callers are unaffected. This makes piece availability across a swarm computable by an embedder. A per-peer count alone does not: it gives the mean copies per piece, while the useful figure is the minimum. Two peers holding 500 pieces each may overlap entirely or not at all, and only one of those torrents can finish. have_pieces rides along because it is nearly free once the bitfield is in hand. have_bitfield is the raw bytes rather than the internal BF: PeerStats is Serialize, bitvec is built without its serde feature, and naming BitBox in a public field would put bitvec in the public API, where a dependent would need a matching version to read it. From<&Peer> for PeerStats becomes PeerStats::from_peer, since the conversion now takes the flag. Peer is pub(crate), so that impl was not reachable externally.
fdb7ee1 to
b4da595
Compare
Closes #643.
Three commits. The first two stand alone; the third is the actual ask.
1. Re-export
PeerStatsandPeerStatsFilter. Both appear in the signatureof the public
per_peer_stats_snapshotbut live in a private module, so adependent crate can only get a filter from
Default::default()and cannot nameeither type.
2. Clear a peer bitfield's padding on ingest. Your call on the review, and
it is the better one — see below. Independent of the stats work, and takeable
without it.
3. Add
have_bitfieldandhave_piecestoPeerStats, behind a newPeerStatsFilter::include_bitfield, off by default.Against your three concerns
Performance for callers that do not need it. Nothing is computed or copied
unless the flag is set — both fields are
Nonethrough a guard on the samematch, so the default path does exactly what it did before.Defaulthas theflag off, which
bitfields_are_off_by_defaultpins. The HTTP API and everyexisting caller are untouched.
API complexity. One new field on the filter and two on
PeerStats, allopt-in. I put the count behind the same flag rather than adding a second one:
it is nearly free once the bitfield is in hand, and it saves each caller from
re-deriving it.
Code complexity. +155/-18 across three files, and taking your ingest
suggestion made it smaller rather than larger — the counting helper is gone and
from_peerno longer needs atotal_piecesargument. The only structuralchange is
From<&Peer> for PeerStatsbecomingPeerStats::from_peer(peer, include_bitfield), since the conversion now takes the flag.Peerispub(crate), so that impl was not reachable externally.Why the bitfield rather than just a count
The issue originally asked for a count; that was my mistake and I corrected it
there. A count gives the mean copies per piece, but the useful figure is the
minimum. Two peers holding 500 pieces each may overlap entirely or not at
all — identical counts, and only one of those torrents can finish.
The padding, and the second way in
An earlier revision of this PR sliced to
total_pieceswhen counting and leftthe stored bitfield alone. You asked for it to be fixed at ingest instead, so
that is what it does now:
on_bitfieldclears the spare trailing bits, andcount_ones()means "pieces this peer has" everywhere without each callerhaving to know about the padding.
Doing it there surfaced something.
on_haveis a second way in, and it boundsthe index with
get_mut— which checks against the bitfield's length. Thatlength is byte-padded, so it overshoots the piece count by up to 7, and a peer
could set a padding bit one
Havemessage after ingest and undo the masking.It is bounded by the piece count now, in the same commit. That also means a
peer can no longer claim a piece that does not exist.
Neither was harmful before this, which is presumably why it went unnoticed:
every other reader either slices to
total_piecesfirst — ason_bitfield'sown "peer has full torrent" check does — or indexes by an already-validated
piece id. The padding was only ever visible to something that counted the whole
bitfield.
Why
have_bitfieldisVec<u8>and notBFPeerStatsderivesSerializeandbitvecis built without itsserdefeature, so
BitBoxhas noSerializeimpl and cloning the field does notcompile. Enabling the feature would also put
bitvecin the public API, wherea dependent would need a matching version just to name the type. The allocation
is the same either way.
Checks
cargo test -p librqbit(38 passed, 5 ignored),cargo fmt --check, andcargo clippy --all-targets -- -D warningsall clean — the last is stricterthan CI's, and worth running here because
lib.rsopts intoclippy::cast_possible_truncationand the count is ausize.Also running as a
[patch.crates-io]in a desktop client, on a branch carryingthese three commits plus #645; its own suite passes.
Rebased onto current main. Happy to reshape or drop any of it.