diff --git a/reference/replication/clustering.md b/reference/replication/clustering.md index 05bc4c1a5..566eee1dd 100644 --- a/reference/replication/clustering.md +++ b/reference/replication/clustering.md @@ -32,6 +32,9 @@ Adds a new Harper instance to the cluster. If `subscriptions` are provided, it c - `table` — table name - `subscribe` — if `true`, transactions on the remote table are replicated locally - `publish` — if `true`, transactions on the local table are replicated to the remote node +- `sendsTo` / `receivesFrom` _(optional)_ — database-scoped controlled-flow entries for this node, mirroring the config-route `replicates.sendsTo` / `replicates.receivesFrom` syntax (see [Controlling Replication Flow](./overview.md#controlling-replication-flow)). Each entry is a database name or an object: `{ database, excludeTables? }`. When provided, replication with this node is restricted to the listed databases; to replicate all databases except specific tables, use a wildcard entry (`excludeTables` with no `database`). Mutually exclusive with `subscriptions`. + + > **Note**: `sendsTo` / `receivesFrom` here scope only the connection to _this_ node — they don't change how the local node advertises itself to the rest of the cluster. A node's own directional (non-mesh) `hdb_nodes` self-record is derived solely from its `harper-config.yaml` routes, so replicating `system` without collapsing to a full mesh requires directional routes in config, not just `add_node`/`set_node` scoping (see [Replicating the `system` database with controlled flow](./overview.md#replicating-the-system-database-with-controlled-flow)). **Request**: diff --git a/reference/replication/overview.md b/reference/replication/overview.md index a9526c82d..550cc2648 100644 --- a/reference/replication/overview.md +++ b/reference/replication/overview.md @@ -198,7 +198,38 @@ replication: In this example, the local node only receives from `node-two` (one-way inbound) and only sends to `node-three` (one-way outbound). -> **Note**: When using controlled flow replication, avoid replicating the `system` database. The `system` database contains node configurations, so replicating it would cause all nodes to have identical (and incorrect) route configurations. +You can also scope flow per database, so different databases flow in different directions between the same two nodes. Use `sendsTo` / `receivesFrom` entries with a `database`: + +```yaml +replication: + databases: + - cardata + - config + - system + routes: + - hostname: node-two + replicates: + sendsTo: + - database: config # push central config downstream + - database: system # push central config (users, roles, schemas) downstream + receivesFrom: + - database: cardata # aggregate telemetry upstream +``` + +### Replicating the `system` database with controlled flow + + + +Before v5.2, replicating the `system` database under controlled flow was discouraged: because `hdb_nodes` (the node registry) lives in `system` and each node advertised itself as a full-mesh participant, replicating `system` caused every node to discover and directly connect to every other node — collapsing a constrained topology into a full mesh. + +As of v5.2 you can replicate `system` while keeping a constrained topology. When a node has directional routes, it advertises a **directional** registry record derived from those routes (which neighbors it sends to / receives from) instead of a blanket "connect to everyone." A discovered non-neighbor node therefore is not subscribed to and does not receive a replication connection. This lets central configuration — users, roles, and schemas — propagate transitively across the whole cluster while user-database connections stay on the routes you configured. For example, in a `roadside → middle → core` aggregation tree, a role created on a roadside node reaches the core through the middle tier, yet the core never opens a direct replication subscription to a roadside node. + +Notes and current limitations: + +- This applies only when a node has **directional** routes (`replicates` with `sends`/`receives` or `sendsTo`/`receivesFrom`). A node with no directional routes keeps the legacy full-mesh advertisement. +- This constrains replication subscriptions only. On-demand residency/retrieval connections (for example, sharded or invalidated-cache reads) use a separate mechanism governed by data residency, not by this registry record, and can still open a direct socket to a non-neighbor node. +- Central visibility of every node is not guaranteed: an aggregation node may not list every distant leaf in its `hdb_nodes` registry (the registry relay differs from data relay). This does not open a connection either way. +- Route changes to a node's own directionality take effect on restart. ### Explicit Subscriptions diff --git a/release-notes/v5-lincoln/5.2.md b/release-notes/v5-lincoln/5.2.md index 992f6fd9c..a9df5b8e4 100644 --- a/release-notes/v5-lincoln/5.2.md +++ b/release-notes/v5-lincoln/5.2.md @@ -31,3 +31,24 @@ The `set_configuration` operation now accepts `"replicated": true` to apply a co ### Middleware routing and ordering Components can now declare `host` and `urlPath` in `config.yaml`, or pass them to `server.http()`, `server.ws()`, and `server.upgrade()`, to create middleware chains routed by virtual hostname, URL prefix, or both. The new `name`, `before`, and `after` options provide explicit middleware ordering. See [HTTP middleware routing](/reference/v5/http/overview#middleware-routing) and [`HttpOptions`](/reference/v5/http/api#httpoptions). + +## Replication + +### Replicating the System Database with a Constrained Topology + +Controlled-flow replication can now include the `system` database while keeping a constrained (non-mesh) topology. + +Previously, replicating `system` was discouraged when using [controlled replication flow](/reference/v5/replication/overview#controlling-replication-flow): because the node registry (`hdb_nodes`) lives in `system` and each node advertised itself as a full-mesh participant, replicating `system` caused every node to discover and directly connect to every other node — defeating the point of a constrained topology. + +As of 5.2, a node with directional routes advertises a **directional** registry record derived from those routes (the neighbors it sends to / receives from) instead of a blanket "connect to everyone." A discovered non-neighbor node therefore is not subscribed to and does not receive a replication connection. This lets central configuration — users, roles, and schemas — propagate transitively across the entire cluster while user-database connections stay on the routes you configured. + +For example, in a `roadside → middle → core` aggregation tree, replicating `system` now lets a role created on a roadside node reach the core (through the middle tier) without the core ever opening a direct replication subscription to a roadside node. + +Behavior notes: + +- This applies only to nodes that have **directional** routes. A node with no directional routes keeps the legacy full-mesh advertisement, so existing full-mesh clusters are unaffected. +- This constrains replication subscriptions only; on-demand residency/retrieval connections (e.g. sharded or invalidated-cache reads) are a separate mechanism, governed by data residency, and are unaffected by this record. +- `add_node` / `set_node` with database-scoped `sendsTo` / `receivesFrom` entries now restrict replication to those databases (consistent with the config-route behavior; see [Add Node](/reference/v5/replication/clustering#add-node)). To replicate all databases while excluding specific tables, use a wildcard entry (`excludeTables` with no `database`). This scopes the connection to that one peer only — it does not make the local node advertise a directional (non-mesh) `hdb_nodes` self-record, which is derived solely from the node's own config routes. Constraining `system` replication cluster-wide still requires directional routes in `harper-config.yaml`. +- Central visibility of every node is not guaranteed: an aggregation node may not list every distant leaf in its `hdb_nodes` registry. This does not open a connection either way. + +See [Controlling Replication Flow](/reference/v5/replication/overview#controlling-replication-flow) for configuration details.