Repository navigation
Description
syn master (5191a98) and 3.4.2 (34749b6), OTP 27.3.4.11.
After a partial net split heals, one node can be left without the entry of a live registered process while every other node has it, and syn does not bring the entry back on its own.
Take three nodes in one Scope: NodeA, NodeB and NodeCommon. A process on NodeA, PidOnNodeA, is registered under some name and all three nodes hold the entry. The connection between NodeA and NodeB drops while both stay connected to NodeCommon. NodeB purges NodeA's entries, no longer sees the name, starts its own PidOnNodeB and registers it under the same name. The sync_register goes to NodeB's nodes_map, which is NodeCommon alone; NodeCommon owns neither process, so handle_registry_sync replaces PidOnNodeA with PidOnNodeB, the newer registration. NodeA hears nothing. Then PidOnNodeB dies before the connection comes back: NodeB's DOWN handler removes it and broadcasts sync_unregister, again to NodeCommon alone, and the sync_unregister handler there finds the same pid and removes the entry. NodeCommon now has nothing for the name.
NodeA and NodeB reconnect and exchange snapshots. NodeB's has nothing for the name, NodeA's gives NodeB PidOnNodeA back as a plain new entry. Neither owner sees a conflict, so resolve_conflict never runs, and the keep-local branch, the one place in the protocol that re-broadcasts a kept registration with a fresh time, sends nothing. NodeCommon never lost its connection to NodeA, so no snapshot from NodeA reaches it either.
End state: all three nodes connected and in the scope's subcluster, syn:lookup returns {PidOnNodeA, Meta} on NodeA and NodeB and undefined on NodeCommon. It lasts until NodeCommon resyncs with NodeA (after a disconnect, or a restart of either scope process) or NodeA broadcasts the entry again (a registration that changes its metadata, or a conflict NodeA wins). No resolver is called on this route, so the default resolution and a custom one end the same. If PidOnNodeB is still alive when the owners reconnect, it heals: both owners see the conflict in the other's snapshot, and the one that keeps its process re-broadcasts it to NodeCommon too.
Reproduction
The TLA+ model in test/tla finds it as soon as it gets a third node. With one change to test/tla/syn.cfg,
Nodes = {n1, n2, n3}
the command from test/tla/README.md (TLC 2.18, the jar in test/tla) reports Invariant AllRegistered is violated at state 12 after about a second. n1 is NodeA, n2 is NodeB and n3 is NodeCommon; the only difference from the story above is that n1 registers after the cut, so n2 never holds n1's entry.
The trace, condensed to the steps that matter
- n1 queues a registration of
a. - The n1-n2 link is cut and a
DOWNis queued on both sides; n3 keeps both links. - n1 registers
aat time 0 and sendssync_registerto n3 alone, since n2 is not visible. - n2 queues a registration of
a; n1 and n2 handle theirDOWNand forget each other. - n2 registers
aat time 1 and sendssync_registerto n3 alone. - n2's process dies: n2 removes its entry and sends
sync_unregisterto n3. - n3 handles n1's
sync_registerand holdsaunder n1, then n2's with the newer time and replaces it, then n2'ssync_unregisterand drops it. - Every queue is empty, n3 sees both other nodes,
registered[a]is 1 because n1's process lives, and n3 holds nothing.AllRegisteredfails for n3.
TLC stops there because AllRegistered is checked per node, and n3 sees both other nodes while n1 and n2 are still apart. To check the state after the split heals, we added an invariant that applies only once every link is back and every queue is empty, in a module next to syn.tla:
---- MODULE syn_connected ----
EXTENDS syn
Quiescent == \A n \in Nodes: Len(inbox[n]) = 0
FullyConnected == \A n \in Nodes: visible_nodes[n] = AllOtherNodes(n)
LiveNames == {r \in DOMAIN registered: registered[r] > 0}
AllRegisteredWhenConnected ==
(Quiescent /\ FullyConnected) =>
\A n \in Nodes: AllRegisteredForNode(locally_registered[n]) = LiveNames
====syn_connected.cfg is syn.cfg with Nodes = {n1, n2, n3} and AllRegisteredWhenConnected in place of AllRegistered. Run on these two files, the README command reports Invariant AllRegisteredWhenConnected is violated at state 17 after about ten seconds. In that state every link is back, every queue is empty, n1 and n2 hold a under n1, and n3 holds nothing. The two-node configuration cannot reach either state: with two nodes no node owns neither process.
We also have a deterministic three-node test with peer (the cut made with erlang:disconnect_node/1, PidOnNodeB killed with exit/2, bounded waits only). It fails on master 5191a98 in 10 of 10 runs and on 3.4.2 in every run, with the default resolution and with a custom resolver that keeps the older start time from Meta; the variant that leaves PidOnNodeB alive until the reconnect passes every time. The peers run with prevent_overlapping_partitions set to false, only so that NodeCommon keeps both connections; with the OTP default, global on NodeCommon drops both. We can turn it into a case for syn_registry_SUITE.
Where we hit it
A 15-node cluster with prevent_overlapping_partitions at its default and a resolver that keeps the process with the older start time. global's partition prevention produced waves of disconnects, and one node was cut from NodeA five times in five seconds. Three nodes ended up with no entry for a live process while every node was connected to every other and in the scope's subcluster, and still had none the next day; two more lost it the same way and got it back only when a later wave cut them from NodeA and the reconnect snapshot restored it. The cut node's logs have a gap in the second that matters, so the route on the cluster is our reconstruction: PidOnNodeB seems to have lost a conflict to NodeA's snapshot on NodeB (our resolver keeps the older process; the default one would have kept PidOnNodeB), NodeB then told the third nodes to drop it, and NodeA never saw PidOnNodeB. The end state matches the test.
Not the late unregister from test/tla/README.md
In the case listed under "Calls Made From Another Node", a node writes an entry for a process on another node, and a sync_unregister that node sent earlier removes it. Here every node registers only its own processes and every message arrives in order; the entry is lost because the one message that would restore it on the third node, the winner's re-broadcast after a conflict, is never sent, since the winner's node never sees the challenger.
Once the direction is agreed I can send a PR, or the test case for the suite on its own.
Activity
ostinelli commentedon Oct 19, 2026
This should already be fixed on master in 37710b9. NodeCommon used to forget NodeA's registration as soon as NodeB's newer one arrived, so it had nothing to fall back on when NodeB's process died. Now it keeps NodeA's registration aside and brings it back in that case. I'm also considering updating the TLA+ model to match, since it doesn't keep anything aside yet and your three-node run still fails there.
I'm also thinking of covering the case where NodeB's process loses a conflict instead of dying, which takes a different path. Could you run your
peertest against master to confirm?vheathen commentedon Oct 19, 2026
Thanks, that was quick. Confirmed: on master (a950280) our
peertest passes in 10 of 10 runs, with the default resolution and with a resolver that keeps the older start time, and it still fails on 5191a98. NodeCommon has PidOnNodeA back as soon as PidOnNodeB dies, while NodeA and NodeB are still apart. I had to relax one step of the test: it expected NodeCommon to be empty right after the kill, which is exactly what changed.The conflict case is the one we hit on the cluster, so we'd be glad to see it covered as well, and I'm happy to run our setup against it when it lands.
vheathen commentedon Oct 19, 2026
I also wrote a
peertest for the conflict case, since that is the route we hit on the cluster. It still fails on master (a950280), the same way as on 5191a98. NodeB merges NodeA's snapshot after it has registered PidOnNodeB, the resolver keeps the older PidOnNodeA, NodeB stops PidOnNodeB and sendssync_unregisterwith reasonsyn_conflict_resolution, and NodeA never sees PidOnNodeB. NodeCommon has PidOnNodeA in its shadow table but does not bring it back:handle_registry_unregister_syncskipsshow_shadowedfor that reason and leaves it to the node that kept its process, and NodeA never had a conflict, so it sends nothing. 10 of 10 runs fail on both commits. With the default resolver PidOnNodeB wins and the same sequence converges.To get that order every time, the test suspends the scope processes with
sys:suspendand cuts the link a second time, withdist_auto_connectset toonceandconnect_allset to false so the cut stays a cut. Happy to share it or turn it into a case for the suite if that helps.vheathen commentedon Oct 29, 2026
One more data point, this time from a 7-node cluster rather than the
peertest. The nodes run master 5191a98 with #93 on top (no changes to the registry or scope code),prevent_overlapping_partitionsat its default, and a resolver that keeps the older start time from Meta.To emulate a split, we froze one node with SIGSTOP until the others had taken over its names, then let it run again. When it resumed, global's overlapping-partition prevention cut the connections between several pairs of nodes within about 400 ms. In that window one node, NodeCommon below, was left with no entry for a live registered process. It stayed that way for about 14 minutes, while all 7 nodes were connected and the other six held the process, and healed only when a new process on NodeCommon registered the name, lost the conflict, and NodeA re-broadcast its entry.
syn logs nothing on a third node when it replaces or purges an entry, and three nodes have gaps in their logs in that second, so the sequence below is our reconstruction. NodeCommon had just lost its own conflict for the name and held NodeA's process. NodeB, cut from NodeA at the time, registered its own process, and NodeCommon replaced NodeA's entry with it as the newer registration. NodeB's process then lost a conflict on NodeB (against NodeCommon's earlier process, which NodeCommon had already stopped) and was stopped, and 8 ms later global disconnected NodeB from NodeCommon. NodeCommon purged NodeB's entries and had nothing left for the name. NodeB and NodeCommon reconnected and resynced about 130 ms later, but NodeB's snapshot had nothing for the name either, and from then on the cluster stayed fully connected. Nothing else brought the entry back: NodeCommon's link to NodeA did not drop again after their resync earlier in that second, and NodeA never saw NodeB's process, so it never re-broadcast.
So this is the same shape with a third way for the challenger to leave NodeCommon: the purge on scope down. Reading 37710b9, master restores the shadowed entry on that path. Had NodeB's
sync_unregisterreached NodeCommon before the disconnect, it would be the conflict case from my previous comment, which master does not restore yet.For scale, in the same test runs each rolling restart of the 7 nodes produced 390 to 469 overlapping-partition disconnects and 77 to 431 name conflicts for one group of singleton names, so the conditions for this come up on every rollout.
ostinelli commentedon Nov 12, 2026
Hey @vheathen, thanks for your patience and for the numbers from your cluster.
All three cases you found have the same cause. NodeCommon replaces PidOnNodeA with PidOnNodeB. Then PidOnNodeB goes away (it dies, it loses a conflict on NodeB, or NodeB gets disconnected), and NodeCommon is left with nothing: it has thrown PidOnNodeA away, and nobody sends it again.
Instead of fixing each case on its own, I changed how nodes keep registrations. It's on #97, which I'd like to release as 3.5.0:
So in all three of your cases, NodeCommon goes back to PidOnNodeA on its own.
I replayed your Oct 29 sequence with real nodes:
v3.5, NodeCommon ends up with PidOnNodeA either way.Your cases are now tests in
test/syn_protocol_SUITE.erl. The TLA+ model intest/tlais rewritten too: it checks that once all nodes are connected, they all show the same process for every name, like the check you added in your report. The late unregister I mentioned in #94 is fixed as well.Some things to know before trying it:
v3.5for your test. In production, all nodes need to be upgraded together, not one at a time.Before I release 3.5.0, could you run on
v3.5:peertests, the one where PidOnNodeB dies and the conflict one;If anything fails, please send me the test or the logs.
vheathen commentedon Nov 13, 2026
Thanks for the redesign. We ran both peer tests and the 7-node setup on
v3.5(67c44fe), and everything converged.The test where PidOnNodeB dies passes 10 of 10. We had to keep the cut from healing on its own: in 1 of 10 runs NodeA and NodeB reconnected about 2 ms after the cut, before the test could register PidOnNodeB. The peers now start with
connect_all falseanddist_auto_connect once, as in the conflict test. The same test still fails on 5191a98.The conflict test can't follow its original path on
v3.5. It relies on NodeB handling NodeA's snapshot after the cut, andv3.5drops that snapshot because NodeA is no longer connected. If we let the test continue past the reconnect, all nodes agree. We also wrote a version where NodeB handles the snapshot before the cut and its messages to NodeA are lost in the cut, similar to yourconflict_second_split. It passes 10 of 10 onv3.5and fails on a950280.Our nodes need per-scope handlers, so the 7-node setup ran
v3.5with #93 merged in. The merge doesn't touch the registry code:syn_registry,syn_pgandsyn_gen_scopeare the same as in 67c44fe.We froze a node with SIGSTOP six times, including nodes that held 161, 81, 18 and 15 of the 161 names, two of them for 60 s, and did three rolling restarts. After each step, once the nodes had reconnected, every node showed the same live process for every name. In one check during a freeze, the two nodes that hadn't dropped the frozen node yet kept its 81 registrations aside and showed the newer ones. After every thaw, each name went back to its original process. The first two rolling restarts logged 497 and 499 overlapping-partition disconnect requests and 1,853 and 1,886
registry_conflictnotices.One flaky test:
three_nodes_updateinsyn_registry_SUITEfailed once on OTP 29. It sets the event handlers without waiting for the first registration to reach the slaves, so a slow slave 2 reports an extraon_process_registered. We can reproduce it on 3.4.2 too, by suspending slave 2's scope process.