fix(protocol): push a key package again when it produced no session - #497
Merged
Merged
Conversation
12 tasks done
A key package was pushed to a peer once, and the peer was then recorded as exchanged with. Every later discovery returned early on that record, so a push the carrier lost was never repeated. Two peers that dial each other at once lose both pushes when the second peer stream supersedes the first and its late frames are dropped (no loss is reported, as the stream-framing spec requires), and the pair then never forms a session. The sent-to set now records when each push was made. Past a 30 second floor, a peer with no MLS session is pushed to again on the next discovery, or on the reconciliation tick while a message waits for that peer's session. The pool hands a peer its own live package until a Welcome consumes it, so the repeat push mints no key material (ADR 0012). The floor is stamped before the send so discovery on every inbound body, or a failing send, cannot turn into a stream of key packages, and a session, confirmed or not, stops it.
…d peers Discovery runs on every inbound body, and once the resend interval had lapsed every established peer reached the MLS storage check on each one, because a peer with a session never had its stamp refreshed. A confirmed session now returns before the floor, and the floor is re-stamped ahead of the storage check, so an unconfirmed session costs one read per interval. The changelog and the session lifecycle now say what the re-arm does not cover: a lost reset push, since the repeat carries no reset.
A neighbour that never answers a key package (encryption opted out, an older SDK) keeps refreshing itself through its own traffic, and with a fixed 30 second floor it was sent a signed key package every 30 seconds for as long as it stayed in range. Each repeat now doubles the wait up to 600 seconds, the ceiling the Welcome retries use. The count rides in the sent-to map, so every path that forgets the peer starts it over.
…backoff Two comments described `key_package_sent_to` as stopping a group member from ever being probed again; it now lets `rearm_key_package_for_peer` repeat the push on its doubling backoff. A const assert pins that five doublings of the resend interval reach the cap, since the shift bound is what also keeps the wait from wrapping to zero.
…nstant The tests aged a push by the 601 second cap through `Instant::checked_sub`, which has no answer on a host booted less than ten minutes ago, so they would panic on a fresh CI runner. The wait is now a function the cap test checks directly, and the aging helper starts the backoff over so it only needs the 31 second base. A reset push also starts the backoff over, since it opens a new exchange, and the cap is pinned to the Welcome retry ceiling it claims to match.
mizanisoffline
force-pushed
the
fix/resend-lost-key-package
branch
from
October 1, 2026 12:18
681d0a0 to
14fa8cb
Compare
The reconciliation tick re-arms the key package push for every peer with a message queued behind a session. It turns out a block leaves that queue alone: blocking drops the inbound decryption queue, but the outbound one is only discarded once a session becomes ready, which for a blocked peer is never. The push record survives too. So after a block, the tick happily advertised our presence and our key package to the blocked peer once per backoff window, for as long as the queued message lived. The peer stored it, replied (dropped), and started a Welcome ladder against us that could never confirm. Blocking is supposed to be bidirectional. This is not great. Discovery never had the problem because it checks the block before it gets anywhere near the push. The tick reused the push path without that caller's gate, and the push path itself has none. Put the check in the re-arm itself, so every trigger that reaches it inherits it. Please don't hang a send path's invariants off each of its callers; the next caller forgets.
The re-arm stamps the window before it sends, which is right: a send that keeps failing must cost one attempt per window, not one per inbound frame. But it also bumped the repeat count before the send, so a frame that never left the device counted as a peer that was asked and ignored us. The backoff exists for the second case only. Counting the first walks an unreachable peer up the ladder to the ten minute cap while it is away. On BLE that is harmless, because losing the neighbour forgets the record. Over a carrier with no neighbour-lost event, the internet for one, nothing resets the count, so when the peer comes back the next re-push can be ten minutes out. The Welcome ladder resets on a reachability edge; this one had no such escape. Keep the stamp, and count a repeat only when the push actually went out, or when the storage check found a session (that is still a check we want to back off). A failing send now retries once per base window, which is the cost the stamp was meant to bound anyway.
A reset push opens a brand new exchange, so it zeroes the re-arm's repeat count, and every other push keeps the count it found. The lifecycle doc and the changelog both state that rule. Nothing in the suite checked it: delete the reset branch and every test still passes. Pin both halves directly on the send path. A plain push must keep the count, a reset push must start it over.
bahdotsh
approved these changes
Oct 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A lost key package could leave two peers without an MLS session forever.
The engine pushed its key package to a peer once, recorded the peer in
key_package_sent_to, and returned early on that record at every later discovery. A carrier that lost that one frame therefore lost the exchange for good. Two peers that dial each other at once hit this routinely: each announces the peer on the first peer stream and pushes its key package there, the second stream supersedes the first, and the superseded stream's late frames are dropped without a loss report (asdocs/spec/stream-framing.md, "What a receiver owes", requires). If both pushes were on it, the stream stays up, no session ever forms, and queued messages end inMax retries exceeded.Found on a device: an iPhone running the Network-framework peer-stream manager against the Python host. About 1 in 5 fresh first contacts reproduced it. Logs from the failing round show both streams superseded at connect, then no key-package, Welcome or session event on either side, and a message queued indefinitely. The same race exists between two Python hosts, which both dial too.
Fix (
crates/offline-protocol/src/protocol/):key_package_sent_tois now a map from peer to(instant of the last push, repeat count). It keeps its existing size limit.rearm_key_package_for_peer(insession.rs, next torearm_welcome_for_peer) pushes again when no MLS session exists with that peer and the last push is older than the backoff window: 30 seconds (KEY_PACKAGE_RESEND_INTERVAL_SECS), doubling with each repeat up to 10 minutes (KEY_PACKAGE_RESEND_CAP_SECS, pinned to the Welcome retry ceiling).on_neighbor_discovered_via, and from the reconciliation tick for every peer with messages queued behind a session. The second covers a quiet pair that is never rediscovered.session_reset = false, so a receiver only refreshes its stored copy and never tears down or replaces a session.Not covered:
INVALID_CIPHERTEXTon every message). The re-arm stops as soon as a session exists, and ADR 0006 deliberately triggers a re-key only on an epoch mismatch. That needs its own investigation.Docs:
docs/state-machines/session-lifecycle.md: a new transition, and a section stating the invariant, the failure it prevents, the bounds and backoff, and the lost-reset limit.CHANGELOG.md: an entry under [Unreleased] / Fixed.data_sync.rs,send.rs) now say a probed member is repeated only on the re-arm backoff.Type of change
fix— bug fixtest— adding or correcting testsTesting
New tests in
crates/offline-protocol/src/protocol/tests/mod.rs, using the existing two-engine harness:a_lost_key_package_is_pushed_again_on_rediscoverya_lost_key_package_is_pushed_again_from_the_tick_while_a_message_waitsa_key_package_is_not_pushed_again_inside_the_interval_or_once_a_session_exists: also checks that an unconfirmed session found in storage re-stamps the window without pushinga_key_package_repeat_backs_off: the doubling, and the cap checked onkey_package_resend_wait_secsdirectlyEach drops both engines' first key packages, then checks recovery or the bound. No test ages a stamp by more than about a minute:
Instantcounts from boot, so a ten-minute offset does not exist on a freshly booted CI runner.Applied to
mainwithout the fix, the recovery tests fail: the queued message never arrives, and no session forms. With the fix they pass.Checklist
cargo fmt --all -- --checkpassescargo clippy --workspace -- -D warningspassescargo test --workspacepasses (32 suites, 3,206 tests, doctests included)RUSTDOCFLAGS="-D warnings" cargo doc -p offline-protocol --no-depspassescargo-denyis satisfied (no dependency changes)CHANGELOG.mdupdatedunsafe