Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
21518a4
Test on OTP 29 with rebar3 3.27.
ostinelli Oct 24, 2026
7718969
Model the 3.5 registry in TLA+.
ostinelli Oct 24, 2026
2a3eb6f
Keep the registrations of every node, and show one by a fixed rule.
ostinelli Oct 24, 2026
da3d599
Add tests that drive the registry through specific message orders.
ostinelli Oct 24, 2026
cf3b36e
Document the 3.5 registry.
ostinelli Oct 24, 2026
42591ea
Keep calls from hidden nodes working.
ostinelli Oct 24, 2026
a7dfe56
Wait for the messages in the ordering tests, not for silence.
ostinelli Oct 24, 2026
d854869
Document what changes when upgrading to 3.5.
ostinelli Oct 24, 2026
d1ef636
Reconnect the test nodes after a failed test, and sync the stepped se…
ostinelli Oct 24, 2026
9ca0e58
Fix the callbacks and calls listed in the upgrade note.
ostinelli Oct 24, 2026
e5f7289
Ignore tmp/ for work notes, and point the model to INTERNALS.md.
ostinelli Oct 24, 2026
c7eca6c
Split the restart model check in two, and describe the model as it is.
ostinelli Oct 24, 2026
76f2202
Describe only the current behavior in the docs and comments.
ostinelli Oct 24, 2026
34c8e11
Take a shorter path for names that only one node holds.
ostinelli Oct 24, 2026
d06f2b0
Add guidance for agents working on the code.
ostinelli Oct 24, 2026
dae4b73
Give the resolver the two registrations in the same order on both nodes.
ostinelli Oct 24, 2026
410c94e
Describe how to keep cut nodes apart, and what happens between versions.
ostinelli Oct 24, 2026
548d717
Test that messages from Syn 3.4 nodes are ignored.
ostinelli Oct 24, 2026
295c2c2
Describe what a call returns when the caller's scope process crashes.
ostinelli Oct 24, 2026
afabfb2
Mark the registration time as unused where the resolver gets called.
ostinelli Oct 24, 2026
0e7264c
Qualify the docs on resolver agreement, calls after a crash, and dist…
ostinelli Oct 24, 2026
8e3be02
Cover a result lost in a crashed mailbox, and who decides again after…
ostinelli Oct 24, 2026
2cb42d4
Load the event handler without the catch that OTP 29 deprecates.
ostinelli Oct 24, 2026
a9b4c76
Cover issue #96 with three nodes holding the name.
ostinelli Nov 7, 2026
1a51164
Keep hidden nodes out of syncing.
ostinelli Nov 7, 2026
0a8ac67
Keep nodes that stall or leave from costing the others.
ostinelli Nov 7, 2026
ed6e837
Remove dead code.
ostinelli Nov 7, 2026
4e1740c
Correct the docs where they differ from the code.
ostinelli Nov 7, 2026
7160e60
Make the tests fail only when the code is wrong.
ostinelli Nov 7, 2026
83c159b
Run the tests through rebar3 in the Makefile.
ostinelli Nov 7, 2026
3bbbfe4
Let the benchmark take any counts.
ostinelli Nov 7, 2026
67c44fe
Start parallel model runs a few seconds apart.
ostinelli Nov 7, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,8 @@ jobs:
name: Tests on OTP ${{matrix.otp}} with rebar v${{matrix.rebar3}}
strategy:
matrix:
otp: ['28', '27']
rebar3: ['3.25.0']
otp: ['29', '28', '27']
rebar3: ['3.27.3']

steps:
- uses: actions/checkout@v7
Expand Down
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -29,3 +29,4 @@ erl_crash.dump
test/results/*
!test/results/.keep
doc/
tmp/
8 changes: 8 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
Treat documentation and code comments as timeless current-state guidance.
Never use changelog language such as "used to", "previously", "now", "no
longer", or "changed from", and never narrate an edit or compare the current
implementation with an earlier one. Git history is the place for change
history. Describe the current contract and explain its rationale when useful
(for example, "done this way so it doesn't break Z").

When committing, add yourself as a Co-author.
129 changes: 98 additions & 31 deletions INTERNALS.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,25 +10,65 @@ them to the other nodes.

This serializes per node operations and allows keeping per node consistency.

When a node receives registrations of the same name from different nodes, its table shows the most recent one. It keeps
the others aside until their node drops them: if the one it shows goes away, it shows the most recent of them, unless it
went away because of a conflict resolution, since the node that keeps its process then tells everyone.

### What each node has
Every node keeps a copy of the registrations and group memberships held by each of the other nodes, as that node last
told it. A node only changes its copy of another node's data when that node tells it to: with its full data when the two
nodes find each other, and with each change after that.

### Which registration a node shows
A name can be registered on two nodes at once, for example while they can't see each other. A node that holds the name
for one of its own processes shows its own. Any other node shows the most recent of the registrations it has, the node
name breaking ties, so that every node shows the same one. It keeps the others aside: if the one it shows goes, it shows
the next one. A node never hides a registration because it expects another node to drop it.

### Conflicts
When a node holds a name and learns that another node holds it too, it runs the conflict resolution, see
[`syn_event_handler`](syn_event_handler.html). If its own process loses, it drops the name, kills the process if the
default resolution is used, and tells every other node. If its own process wins, it does nothing: the other node drops
its own. Each of the two nodes decides on its own, so the resolution must keep the same process on both. Both nodes
pass it the two registrations in the same order, by node name, so that a resolution that only looks at what it gets
decides the same on both, given the same versions of the two registrations. While an update of one of them is on its
way, the two nodes can decide on different versions; a node that still holds its own registration decides again when
it gets the update.

### Calls for a process on another node
When the operation is called from another node, the result is sent to the scope process of the calling node
in format `{'3.0', sync_register_reply, ...}` (or `sync_unregister_reply`, `sync_join_reply`, `sync_leave_reply`),
in format `{'3.5', sync_register_reply, ...}` (or `sync_unregister_reply`, `sync_join_reply`, `sync_leave_reply`),
through the same process that sends it all the other data, so that it gets processed in order with it. The calling
scope process applies it like any other sync message and only then replies to the caller, so that its tables are
already updated when the call returns.

If the process' node doesn't know the calling node yet, it first sends it its local data, as when it receives a
discover message, so that the calling node knows it by the time the result gets there.
If the operation makes the process' node lose a conflict, for example an update that changes what the resolution
compares, the result goes out before the node drops the name, so that the calling node doesn't show the name again
after it was dropped.

If the process' node doesn't know the scope process of the calling node, it first sends it its local data and asks for
the calling node's data, so that the two nodes know each other by the time the result gets there. If the calling node
still doesn't know the scope process of the process' node when the result gets there, it doesn't apply it and the call
returns success: the data of that node reaches it with its snapshot.

If the scope process of the calling node crashes before it applies the result, a result already in its mailbox is lost
with it. A result still on its way goes to whichever scope process runs under the same name when it gets there: if the
restarted one is already running, it handles the result as above and replies, and if none is running, the result is
lost. When the result is lost, the call exits after the usual 5 second timeout of a call, although the process' node
did the operation, and the restarted scope process gets the change with that node's snapshot.

If the call is made from a scope process (i.e. from a callback), that process can't wait for one: the process' node
replies to it right away, and the scope process of the calling node applies the result in the same way when it gets to it.
Such a call returns the result of the process' node, and never `race_condition`.
Such a call returns the result of the process' node: `race_condition` then only comes from the process' node itself,
for example for an unregister of a name it holds for another process.

Any other caller waits for the scope process of its node. If that scope process is running a callback that waits for
the caller, directly or through other processes, neither can go on until one of the calls times out after 5 seconds;
the process' node does the operation either way, whether the caller's call then returns or exits. Callbacks therefore
must not wait for processes that call Syn in the same scope and kind (see `syn_event_handler`).

If the process' node runs a version of Syn that does not support this, it replies to the caller directly, and the caller
updates the tables of its node. That node logs the unknown call the first time, and then gets direct calls until it disconnects.
A hidden node takes no part in syncing: it isn't discovered (see Joining below), and scope processes don't sync with
it, since it is usually connected to some of the nodes only and would make them disagree with the others. A call from a
hidden node for a process on another node, or to a hidden node for one of its processes, still works: the process'
node does the operation and the call returns its result, but the calling node doesn't show the change. So an
unregistration or update of that name from the calling node, which looks the name up locally, returns
`{error, undefined}`.

## Scope Subclusters
Syn implements Scopes, which are a way to create namespaced, logical overlay networks running on top of the Erlang
Expand All @@ -49,34 +89,61 @@ the following happens:
* `syn_registry_by_pid_users` (of type `bag`).
* `syn_pg_by_name_users` (of type `ordered_set`).
* `syn_pg_by_pid_users` (of type `ordered_set`).

These tables are owned by the `syn_backbone` process, so that if the related scope processes were to crash the data is not lost.
In such case, upon respawn the scope process purges the data of all remote nodes, rebuilds the monitors
for the local processes that are still alive and removes from the tables the ones that died meanwhile.
The remote data will then get synchronized again.

These tables hold what the node shows, and are owned by the `syn_backbone` process, so that if the related scope
processes were to crash the data is not lost. In such case, upon respawn the scope process purges the data of all
remote nodes, rebuilds the monitors for the local processes that are still alive and removes from the tables the
ones that died meanwhile. The remote data will then get synchronized again.
* The registry scope process also keeps, in a table of its own, the registrations of other nodes that it doesn't show.
* The 2 newly created scope processes each join a subcluster (one for registry, one for process groups)
with the other processes in the Erlang distributed cluster that handle the same Scope (which have the same name).

### Subcluster protocol

#### Messages
* Everything a scope process sends to other nodes goes through its sender process, so that another node gets it in
the order it was sent. A connection too busy to take a message, for example to a node that stopped reading from
it, doesn't hold up the others: the messages for that node wait in order until its connection takes them again,
while the other nodes, and the results of their calls, get theirs.
* Syn's scope processes don't open connections between nodes: a message to a node that isn't connected is dropped,
and that node gets the data again when it connects. There is one narrow exception: if the connection drops between
the moment a scope process checks it and the moment it starts monitoring the other node's scope process, Erlang
connects the two nodes again, as it does whenever any process sends something to a node it isn't connected to. With
the kernel parameter `dist_auto_connect` set to `once`, Erlang doesn't reconnect two nodes on its own once their
connection dropped, and this can't happen.
* A call made through the API for a process on another node connects the nodes, as any call does. With
`dist_auto_connect` set to `once`, a call for a process on a node whose connection dropped exits with `nodedown`
instead, until the nodes get connected explicitly, for example with `net_kernel:connect_node/1`.
* Every message carries the scope process that sent it. A node only applies messages from the scope process it knows
for that node: once it knows the new scope process of a node that restarted, it ignores what the old one sent. A
late message from the old one can still make the node take it for a while; the old one's DOWN then makes the node
ask that node for its data again.
* Messages are tagged with the version of the protocol, `'3.5'`, so that nodes running a version of Syn with another
protocol ignore each other.

#### Joining

* Just after initialization, every scope process tries to send a discovery message in format `{'3.0', discover, self()}`
to every process with the _same name_ running on all the nodes in the Erlang cluster.
* Scope processes monitor node events, so when a new node joins the cluster the same discovery message is sent to the
new node.
* When a scope process receives the discovery message, it:
* Replies with its local data with an ack message in format `{'3.0', ack_sync, self(), LocalData}`.
* Starts monitoring the remote scope node process.
* Adds the remote node to the list of the subcluster nodes.
* The scope process that receives the `ack_sync` message:
* If it's an unknown node, it first:
* Starts monitoring the remote scope node process.
* Adds the remote node to the list of the subcluster nodes.
* Sends it its local data with another ack message, before a conflict resolution can change that data.
* Stores the received data of the remote node.

* Just after initialization, every scope process sends a discovery message in format `{'3.5', discover, self()}`
to every process with the _same name_ running on all the visible nodes in the Erlang cluster.
* Scope processes monitor node events, so when a new visible node joins the cluster the same discovery message is sent
to the new node. A node that never answers it, for example one that doesn't run the scope, stops being waited for
when its connection drops.
* When a scope process receives the discovery message, it replies with its local data with an ack message in format
`{'3.5', ack_sync, self(), LocalData}`. If it didn't know the scope process that sent it, it starts monitoring it,
adds its node to the list of the subcluster nodes, and asks it for its own data unless it already did.
* When a scope process receives an `ack_sync` message from a scope process it didn't know, it first starts monitoring
it, adds its node to the list of the subcluster nodes, and sends it its local data, before a conflict resolution
can change that data.
* The data in an `ack_sync` replaces everything the scope process had from that node.
* A discovery or `ack_sync` message from a node this one isn't connected to was sent before the connection dropped:
it is ignored, and the two nodes find each other again when they reconnect.
* A scope process from a node whose scope process this one already knows means the other one restarted: the data of
the old one goes.

#### Leaving

* When a scope process of a remote node dies, all the other scope processes are notified because they were monitoring it.
* When a scope process of a remote node dies, or the connection to its node drops, all the other scope processes are
notified because they were monitoring it.
* The data related to the remote node that left the subcluster is removed locally on every node.
* If the node is still connected when the notification gets handled, its scope process restarted, or the notification
comes from a connection that dropped and came back: the scope process asks that node for its data again.
31 changes: 13 additions & 18 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
PROJECT_DIR:=$(strip $(shell dirname $(realpath $(lastword $(MAKEFILE_LIST)))))

.PHONY: all compile_test clean dialyzer run test proper killzombies bench

all:
@rebar3 compile

Expand Down Expand Up @@ -27,36 +29,29 @@ else
-eval 'syn:start().'
endif

test: compile_test
test:
ifdef suite
@# 'make test suite=syn_registry_SUITE'
ct_run -noinput -dir $(PROJECT_DIR)/test -logdir $(PROJECT_DIR)/test/results \
-suite $(suite) \
-pa `rebar3 as test path` \
-erl_args -config $(PROJECT_DIR)/test/sys
rebar3 ct --sname ct --sys_config $(PROJECT_DIR)/test/sys.config --suite $(PROJECT_DIR)/test/$(suite)
else
ct_run -noinput -dir $(PROJECT_DIR)/test -logdir $(PROJECT_DIR)/test/results \
-pa `rebar3 as test path` \
-erl_args -config $(PROJECT_DIR)/test/sys
rebar3 ct --sname ct --sys_config $(PROJECT_DIR)/test/sys.config
endif

proper: compile_test
proper:
ifdef suite
ct_run -noinput -dir $(PROJECT_DIR)/test/property -logdir $(PROJECT_DIR)/test/results \
-suite $(suite) \
-pa `rebar3 as test path` \
-erl_args -config $(PROJECT_DIR)/test/sys
rebar3 ct --sname ct --sys_config $(PROJECT_DIR)/test/sys.config --dir $(PROJECT_DIR)/test/property \
--suite $(PROJECT_DIR)/test/property/$(suite)
else
ct_run -noinput -dir $(PROJECT_DIR)/test/property -logdir $(PROJECT_DIR)/test/results \
-pa `rebar3 as test path` \
-erl_args -config $(PROJECT_DIR)/test/sys
rebar3 ct --sname ct --sys_config $(PROJECT_DIR)/test/sys.config --dir $(PROJECT_DIR)/test/property
endif

killzombies:
@pkill -9 -f beam 2>/dev/null; true
@# only the nodes started by the tests, and the test runs themselves
@pkill -9 -f "sname (syn_slave_|syn_protocol_|syn_hidden_|syn_prop_|syn_pg_prop_)" 2>/dev/null; true
@pkill -9 -f "[r]ebar3 ct --sname ct" 2>/dev/null; true
@sleep 1
@epmd -daemon
@echo "Zombie beam processes killed, epmd restarted."
@echo "Test nodes killed, epmd running."

bench: compile_test
@erl -pa `rebar3 as test path` \
Expand Down
4 changes: 2 additions & 2 deletions OPTIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,9 +73,9 @@ With `strict_mode` turned on:
```elixir
iex> pid = spawn(fn -> receive do _ -> :ok end end)
#PID<0.108.0>
iex> :syn.register(:users, "hedy", pid).
iex> :syn.register(:users, "hedy", pid)
{:error, :not_self}
iex> :syn.register(:users, "hedy", self()).
iex> :syn.register(:users, "hedy", self())
:ok
```

Expand Down
14 changes: 14 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,20 @@ Ensure that `syn` is started with your application, for example by adding it in
]}.
```

### Running different versions together
Nodes sync only with nodes running the same version of Syn's protocol: a node running Syn 3.5 and a node running an
earlier version ignore each other's messages, so a cluster with both splits into one group per version, each with its
own registry and process groups:

* a name can be registered once in each group, so two processes can hold it at the same time, until the nodes of one
of the versions stop and their registrations go with them;
* a call for a process on a node running the other version fails: on a 3.5 node it raises
`{invalid_remote_scope, Scope, Node}`;
* each node logs a warning for every message it gets from the other version.

Upgrade all the nodes of a cluster together: stop them all and start them with the new version, or start the new
version as a separate cluster (for example with another cookie) and move the traffic to it.

## Quickstart

### Registry
Expand Down
2 changes: 0 additions & 2 deletions src/syn.app.src
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,6 @@
{vsn, "3.4.2"},
{registered, [
syn_backbone,
syn_pg,
syn_registry,
syn_sup
]},
{applications, [
Expand Down
Loading
Loading