Skip to content

fix(protocol): push a key package again when it produced no session - #497

Merged
bahdotsh merged 8 commits into
mainfrom
fix/resend-lost-key-package
Oct 1, 2026
Merged

bahdotsh merged 8 commits into
mainfrom
fix/resend-lost-key-package

Conversation

@mizanisoffline

@mizanisoffline mizanisoffline commented Oct 1, 2026 •

Copy link
Copy Markdown
Member

Summary

A lost key package could leave two peers without an MLS session forever.

The engine pushed its key package to a peer once, recorded the peer in key_package_sent_to, and returned early on that record at every later discovery. A carrier that lost that one frame therefore lost the exchange for good. Two peers that dial each other at once hit this routinely: each announces the peer on the first peer stream and pushes its key package there, the second stream supersedes the first, and the superseded stream's late frames are dropped without a loss report (as docs/spec/stream-framing.md, "What a receiver owes", requires). If both pushes were on it, the stream stays up, no session ever forms, and queued messages end in Max retries exceeded.

Found on a device: an iPhone running the Network-framework peer-stream manager against the Python host. About 1 in 5 fresh first contacts reproduced it. Logs from the failing round show both streams superseded at connect, then no key-package, Welcome or session event on either side, and a message queued indefinitely. The same race exists between two Python hosts, which both dial too.

Fix (crates/offline-protocol/src/protocol/):

  • When each push happened, and how often it was repeated. key_package_sent_to is now a map from peer to (instant of the last push, repeat count). It keeps its existing size limit.
  • Re-arm. rearm_key_package_for_peer (in session.rs, next to rearm_welcome_for_peer) pushes again when no MLS session exists with that peer and the last push is older than the backoff window: 30 seconds (KEY_PACKAGE_RESEND_INTERVAL_SECS), doubling with each repeat up to 10 minutes (KEY_PACKAGE_RESEND_CAP_SECS, pinned to the Welcome retry ceiling).
  • Two triggers. It runs at the early return in on_neighbor_discovered_via, and from the reconciliation tick for every peer with messages queued behind a session. The second covers a quiet pair that is never rediscovered.
  • Bounds.
    • A confirmed session returns before anything else, from memory. Discovery runs on every inbound body, so an established peer never reaches MLS storage on this path.
    • The window is stamped before the storage check and before the send. An unconfirmed session, a failing storage read or a send that keeps failing costs at most one check per window, not one per frame.
    • Any session, confirmed or not, stops the push. An unconfirmed session stays with the Welcome lifecycle and the confirmation probes.
    • The backoff means a peer that never answers (an old SDK, encryption opted out, a group member with no 1:1 session) settles at one push per 10 minutes. A reset push, or forgetting the peer (session reset, neighbour lost), starts it over.
    • The first push is unchanged.
  • ADR 0012 still holds. The pool hands a peer its own live package until a Welcome consumes it, so the repeat push mints no key material.
  • Receivers are unaffected. The repeat push carries session_reset = false, so a receiver only refreshes its stored copy and never tears down or replaces a session.

Not covered:

  • A lost reset push (re-key, unblock). The repeat carries no reset, so a peer that kept its session keeps it. Repeating the reset flag would tear down a replacement session whose Welcome is in flight, so this belongs with the split-session work below.
  • A related variant seen once, where both sides hold a session for the pair but disagree on it (INVALID_CIPHERTEXT on every message). The re-arm stops as soon as a session exists, and ADR 0006 deliberately triggers a re-key only on an epoch mismatch. That needs its own investigation.

Docs:

  • docs/state-machines/session-lifecycle.md: a new transition, and a section stating the invariant, the failure it prevents, the bounds and backoff, and the lost-reset limit.
  • CHANGELOG.md: an entry under [Unreleased] / Fixed.
  • The group capability probe comments (data_sync.rs, send.rs) now say a probed member is repeated only on the re-arm backoff.

Type of change

  • fix — bug fix
  • test — adding or correcting tests

Testing

New tests in crates/offline-protocol/src/protocol/tests/mod.rs, using the existing two-engine harness:

  • a_lost_key_package_is_pushed_again_on_rediscovery
  • a_lost_key_package_is_pushed_again_from_the_tick_while_a_message_waits
  • a_key_package_is_not_pushed_again_inside_the_interval_or_once_a_session_exists: also checks that an unconfirmed session found in storage re-stamps the window without pushing
  • a_key_package_repeat_backs_off: the doubling, and the cap checked on key_package_resend_wait_secs directly

Each drops both engines' first key packages, then checks recovery or the bound. No test ages a stamp by more than about a minute: Instant counts from boot, so a ten-minute offset does not exist on a freshly booted CI runner.

Applied to main without the fix, the recovery tests fail: the queued message never arrives, and no session forms. With the fix they pass.

Checklist

  • cargo fmt --all -- --check passes
  • cargo clippy --workspace -- -D warnings passes
  • cargo test --workspace passes (32 suites, 3,206 tests, doctests included)
  • RUSTDOCFLAGS="-D warnings" cargo doc -p offline-protocol --no-deps passes
  • cargo-deny is satisfied (no dependency changes)
  • Commits follow Conventional Commits
  • Docs / CHANGELOG.md updated
  • No new unsafe
  • UDL unchanged; no binding regeneration needed

A key package was pushed to a peer once, and the peer was then recorded as
exchanged with. Every later discovery returned early on that record, so a
push the carrier lost was never repeated. Two peers that dial each other at
once lose both pushes when the second peer stream supersedes the first and
its late frames are dropped (no loss is reported, as the stream-framing spec
requires), and the pair then never forms a session.

The sent-to set now records when each push was made. Past a 30 second floor,
a peer with no MLS session is pushed to again on the next discovery, or on
the reconciliation tick while a message waits for that peer's session. The
pool hands a peer its own live package until a Welcome consumes it, so the
repeat push mints no key material (ADR 0012). The floor is stamped before the
send so discovery on every inbound body, or a failing send, cannot turn into
a stream of key packages, and a session, confirmed or not, stops it.
…d peers

Discovery runs on every inbound body, and once the resend interval had
lapsed every established peer reached the MLS storage check on each one,
because a peer with a session never had its stamp refreshed. A confirmed
session now returns before the floor, and the floor is re-stamped ahead of
the storage check, so an unconfirmed session costs one read per interval.

The changelog and the session lifecycle now say what the re-arm does not
cover: a lost reset push, since the repeat carries no reset.
A neighbour that never answers a key package (encryption opted out, an
older SDK) keeps refreshing itself through its own traffic, and with a fixed
30 second floor it was sent a signed key package every 30 seconds for as
long as it stayed in range. Each repeat now doubles the wait up to 600
seconds, the ceiling the Welcome retries use. The count rides in the
sent-to map, so every path that forgets the peer starts it over.
…backoff

Two comments described `key_package_sent_to` as stopping a group member from
ever being probed again; it now lets `rearm_key_package_for_peer` repeat the
push on its doubling backoff. A const assert pins that five doublings of the
resend interval reach the cap, since the shift bound is what also keeps the
wait from wrapping to zero.
…nstant

The tests aged a push by the 601 second cap through `Instant::checked_sub`,
which has no answer on a host booted less than ten minutes ago, so they
would panic on a fresh CI runner. The wait is now a function the cap test
checks directly, and the aging helper starts the backoff over so it only
needs the 31 second base.

A reset push also starts the backoff over, since it opens a new exchange,
and the cap is pinned to the Welcome retry ceiling it claims to match.
@mizanisoffline
mizanisoffline force-pushed the fix/resend-lost-key-package branch from 681d0a0 to 14fa8cb Compare October 1, 2026 12:18
The reconciliation tick re-arms the key package push for every peer
with a message queued behind a session. It turns out a block leaves
that queue alone: blocking drops the inbound decryption queue, but
the outbound one is only discarded once a session becomes ready,
which for a blocked peer is never. The push record survives too.

So after a block, the tick happily advertised our presence and our
key package to the blocked peer once per backoff window, for as long
as the queued message lived. The peer stored it, replied (dropped),
and started a Welcome ladder against us that could never confirm.
Blocking is supposed to be bidirectional. This is not great.

Discovery never had the problem because it checks the block before
it gets anywhere near the push. The tick reused the push path
without that caller's gate, and the push path itself has none.

Put the check in the re-arm itself, so every trigger that reaches it
inherits it. Please don't hang a send path's invariants off each of
its callers; the next caller forgets.
The re-arm stamps the window before it sends, which is right: a send
that keeps failing must cost one attempt per window, not one per
inbound frame. But it also bumped the repeat count before the send,
so a frame that never left the device counted as a peer that was
asked and ignored us.

The backoff exists for the second case only. Counting the first walks
an unreachable peer up the ladder to the ten minute cap while it is
away. On BLE that is harmless, because losing the neighbour forgets
the record. Over a carrier with no neighbour-lost event, the internet
for one, nothing resets the count, so when the peer comes back the
next re-push can be ten minutes out. The Welcome ladder resets on a
reachability edge; this one had no such escape.

Keep the stamp, and count a repeat only when the push actually went
out, or when the storage check found a session (that is still a
check we want to back off). A failing send now retries once per base
window, which is the cost the stamp was meant to bound anyway.
A reset push opens a brand new exchange, so it zeroes the re-arm's
repeat count, and every other push keeps the count it found. The
lifecycle doc and the changelog both state that rule. Nothing in the
suite checked it: delete the reset branch and every test still
passes.

Pin both halves directly on the send path. A plain push must keep
the count, a reset push must start it over.
@bahdotsh
bahdotsh merged commit 4278b9a into main Oct 1, 2026
24 checks passed
@github-actions github-actions Bot locked and limited conversation to collaborators Oct 1, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants