NetCluster Server

A clustering server. Ingest position reports for things that move; serve clustered vector tiles.

Geospatial databases index points for proximity — what is near here, what is inside this polygon — and leave clustering to the client. So a map of moving things ends up running two systems: one for the queries, and a separate supercluster instance rebuilt on a timer for the markers. This closes that seam: the primary index is a net hierarchy, so clustering is a first-class query and the index is never rebuilt.

POST /v1/collections/fleet/positions     ->  917,000 reports/s
GET  /v1/collections/fleet/tiles/10/379/580.mvt  ->  0.18 ms, 115 markers

Two branches:

branch	what
master	the server, the Node client, deployment manifests
netcluster	the index on its own — a direct Rust port of netcluster-js, usable as a plain crate

crates/netcluster/          the index          (no dependencies)
crates/netcluster-server/   the HTTP server    (axum, tokio)
clients/node/               the Node client    (no dependencies)
deploy/k8s/                 Kubernetes manifests
docs/DEPLOY.md              Docker, Kubernetes, GCP, AWS

netcluster is the base; master merges it and builds on top. Both share crates/netcluster, so library work lands on netcluster and merges forward cleanly.

Measured

500,000 points, Apple M-series, one process. The JavaScript column is netcluster-js measured on the same machine; memory is peak RSS of the whole process, runtime included.

	supercluster	netcluster-js	this (Rust)
insert one point	872,000 µs (full reload)	2.10 µs	0.86 µs
move one point	872,000 µs (full reload)	2.13 µs	0.65 µs
remove one point	872,000 µs (full reload)	7.94 µs	1.76 µs
peak RSS	551 MB	244 MB	172 MB

Through HTTP, with JSON parsing, at 100,000 devices:

ingest (cold)     100,000 devices  0.11 s   917,000 reports/s
ingest (moves)    100,000 moves    0.11 s   952,000 reports/s
tile z=8   (35 markers)    0.12 ms    1.2 KB
tile z=10 (115 markers)    0.18 ms    3.0 KB

And the number that matters most — 8 clients reading while a writer saturates the index:

32,091 tile reads in 3 s   p50 0.19 ms   p99 4.58 ms
  ... while 2,780,000 positions were written concurrently

That is the whole reason this is a Rust service rather than Lua inside Redis. In the Redis version a wide query blocked every other client for its full duration (18.6 ms p50 on the primary). Here queries take &self and mutations take &mut self, so an RwLock gives real concurrent readers.

Run

docker run -p 8080:8080 renatex314/netcluster-server

or from source:

docker compose up                   # or: cargo run --release -p netcluster-server

Then open http://localhost:8080/ for a live demo: a simulated fleet of up to 200,000 vehicles moving continuously, rendered by MapLibre straight from the .mvt endpoint.

The image is 9.6 MB to pull (37 MB on disk) — a distroless base with no shell and no package manager, and a 1.3 MB binary. It is published for linux/amd64 and linux/arm64, runs as non-root under a read-only root filesystem, and its health check is a flag on the binary (--health), which is why it needs no curl.

NETCLUSTER_ADDR             0.0.0.0:8080   listen address
NETCLUSTER_SWEEP_SECONDS    10             how often to drop expired devices
NETCLUSTER_AUTO_CREATE      1              create a collection on first write
NETCLUSTER_DATA_DIR         (unset)        snapshot directory; unset = no persistence
NETCLUSTER_SNAPSHOT_SECONDS 60             snapshot interval

Persistence is opt-in. Without NETCLUSTER_DATA_DIR the server keeps nothing, which is the right default when devices report on a timer — the index refills itself in about a second per million devices. Set it when reporting is event-driven, where a parked vehicle would otherwise never reappear after a restart. See docs/DEPLOY.md.

Turn NETCLUSTER_AUTO_CREATE off in production: with it on, a typo in a collection name silently creates an empty collection instead of returning 404, and you debug an empty map instead of reading an error.

CLI

npm install -g netcluster-client     # or: npx netcluster-client <command>

netcluster create fleet --categories idle,enroute,delivering --ttl 300
netcluster seed fleet --count 50000
netcluster load fleet points.geojson         # bulk-load GeoJSON
netcluster clusters fleet --zoom 6
netcluster watch

Manages collections, loads and inspects devices, runs queries, forces snapshots and verifies invariants. --json on any command for scripting; exit codes mean 0 fine, 1 failed, 2 wrong usage. Full list in clients/node/.

Node client

npm install netcluster-client

import { NetClusterClient } from 'netcluster-client';

const fleet = new NetClusterClient({ url: 'http://localhost:8080' }).collection('fleet');
await fleet.create({ ttlSeconds: 300, categories: ['idle', 'enroute', 'delivering'] });

// batches on a timer AND coalesces by device id, so a vehicle reporting ten
// times between flushes sends one entry with its latest position
const reporter = fleet.reporter({ flushMs: 500 });
onGpsFix((f) => reporter.report({ id: f.deviceId, lng: f.lng, lat: f.lat }));

// already have GeoJSON? send it as-is
await fleet.reportGeoJSON(await (await fetch('/fleet.geojson')).json());

const { features } = await fleet.getClusters({ bbox: [-47, -24, -46, -23], zoom: 12 });
const tile = await fleet.getTile(12, 1517, 2323);   // Uint8Array of MVT

Zero dependencies, TypeScript declarations bundled, and it knows the replication rules below — writes fan out to every replica, reads go to one, and client.forViewer(sessionId) pins a viewer so markers do not flicker.

cd clients/node
npm run example         # a guided tour of every function, then exits
npm run example:fleet   # 20,000 vehicles reporting continuously

example.mjs ends by asserting it called every public method on the client, and npm test runs it — so a method added and not demonstrated fails the build. Full documentation in clients/node/.

API

# a collection, with named categories so filters read as words
curl -X PUT localhost:8080/v1/collections/fleet -H 'content-type: application/json' \
  -d '{"max_zoom":16,"ttl_seconds":300,"categories":["idle","enroute","delivering"]}'

# report positions (batch)
curl -X POST localhost:8080/v1/collections/fleet/positions -H 'content-type: application/json' \
  -d '[{"id":"truck-1","lng":-46.6333,"lat":-23.5505,"cat":"delivering"},
       {"id":"truck-2","lng":-46.6340,"lat":-23.5510,"cat":"delivering"}]'

# or post GeoJSON straight through -- same endpoint, same upsert
curl -X POST localhost:8080/v1/collections/fleet/positions -H 'content-type: application/json' \
  --data-binary @fleet.geojson

# vector tiles -- MapLibre and Leaflet consume these natively
curl localhost:8080/v1/collections/fleet/tiles/10/379/580.mvt

# GeoJSON, in the exact shape supercluster emits
curl 'localhost:8080/v1/collections/fleet/clusters?bbox=-47,-24,-46,-23&zoom=12'

# only the delivering ones -- precomputed, not scanned
curl 'localhost:8080/v1/collections/fleet/clusters?bbox=-47,-24,-46,-23&zoom=12&cat=delivering'

# which marker is my vehicle inside right now?
curl 'localhost:8080/v1/collections/fleet/devices/truck-1/cluster?zoom=12'

PUT /v1/collections/{name}	create; idempotent, 409 on a different geometry
GET /v1/collections	list, with stats
DELETE /v1/collections/{name}	drop
POST /v1/collections/{name}/positions	batch ingest — compact or GeoJSON, see GeoJSON
DELETE /v1/collections/{name}/devices/{id}	remove one device
GET .../devices/{id}	is it registered? 200 with position, category and staleness, or 404 (HEAD for a bare check)
GET .../clusters?bbox=&zoom=&cat=	GeoJSON; ?f.<name>= for declared dimensions, ?where= to search text
GET .../tiles/{z}/{x}/{y}.mvt	vector tile (.json for tile-space GeoJSON)
GET .../devices/{id}/cluster?zoom=	which marker contains this device
GET .../clusters/{id}/children	one expansion step, plus expansion_zoom
GET .../clusters/{id}/leaves?limit=&offset=	the individual devices inside
POST .../snapshot	write a snapshot now (persistence must be on)
GET .../verify	full invariant check — admin only, O(N²)
GET /healthz, GET /metrics	liveness, Prometheus

clusters is a clustering query at every zoom, so it is not a way to list devices. Zoom is clamped to max_zoom, and points closer than the cluster radius at that zoom come back as one cluster carrying point_count — no device id, no props. Filtering that response in your own code silently drops every device inside such a cluster, and a depot full of parked vehicles is exactly that case. To reach members, use /clusters/{id}/leaves.

GeoJSON

GeoJSON goes both ways. /clusters has always emitted it; it goes in too, on the same endpoint, with the same upsert semantics:

curl -X POST localhost:8080/v1/collections/fleet/positions \
  -H 'content-type: application/json' --data-binary @fleet.geojson

netcluster load fleet fleet.geojson          # or through the CLI

await nc.collection('fleet').reportGeoJSON(featureCollection);

A bare array is always the compact form; GeoJSON must arrive as {"type": "FeatureCollection", "features": [...]}, which is what every GeoJSON producer emits anyway. Sniffing each element to decide would put a branch on the hottest parse in the server and would still guess wrong on a mixed array, so the format is settled once by the shape of the container.

Reading rules

id	feature.id, where GeoJSON says it goes, then properties.id. ?id_property=plate names another one — and naming it is strict: a feature missing it is rejected rather than falling back, because a silent fallback keys half a fleet one way and half the other. A numeric id becomes its decimal form, so 7 and "7" are one device.
position	geometry.coordinates. Point only. A third coordinate is altitude and is ignored.
properties	stored verbatim, byte for byte — never reparsed, so a read hands the original text straight to the serialiser. null means "leave what is stored alone", the same as omitting props; {} clears.
category	properties.cat, then properties.category. ?cat_property=status names another. Index or declared name.
anything else	ignored. RFC 7946 §6.1 allows foreign members on a Feature and real files carry them, so bbox, title and friends pass through harmlessly.

Rejections name the feature. A Polygon is refused rather than quietly reduced to a centroid, a null geometry is refused, and coordinates written the wrong way round are caught whenever the swap puts a longitude past ±90 into the latitude slot. Everything comes back as this API's {"error", "code"} body — 400 with bad_geojson for content this handler judges, 422 with unprocessable_body for a shape serde rejects, both naming the offending index:

{"code":"bad_geojson","error":"features[8123] has a null geometry, so it has no position to cluster"}

What it costs. GeoJSON is roughly twice the bytes per point, and it shows — this is the whole ingest path, through HTTP, with real JSON parsing, at 100,000 devices:

body	reports/s	posted
[{id, lng, lat}]	1,071,000	6.7 MB
[{id, lng, lat, props}]	943,000	9.7 MB
FeatureCollection	876,000	13.4 MB
FeatureCollection + properties	828,000	15.1 MB
FeatureCollection + properties + category	722,000	15.9 MB

The last row pays for one extra skip-scan over each properties object to find the category — everything not asked for is stepped over rather than materialised, which is what lets properties stay stored as untouched bytes. The compact path is untouched by any of this, and was measured before and after to confirm it.

Reproduce with node scripts/bench-ingest.mjs. All five rows come from one run, so they are comparable with each other; the absolute figures move ±10% with machine state, which is why they differ from the ones under Measured taken on a separate run.

One thing to know: a device's category is fixed when it is first seen. A later report moves it and replaces its properties, but does not re-file it into another category. That is not a GeoJSON quirk — the compact path behaves the same way. If a vehicle's status changes, DELETE it and report it again.

Attaching data to a device

Any JSON object, returned verbatim on single-point features and on getDevice:

curl -X POST localhost:8080/v1/collections/fleet/positions -H 'content-type: application/json' \
  -d '[{"id":"truck-1","lng":-46.6333,"lat":-23.5505,
        "props":{"plate":"ABC-1234","driver":"Ana","battery":87}}]'

• Omit props and the device keeps what it had. Positions arrive many times a second and attributes change rarely, so a position report does not have to resend the number plate to avoid erasing it. Send {} to clear.

  props	effect
  omitted	unchanged — the ordinary position update
  {...}	replaces the whole object
  {}	clears

• There is no partial update. props replaces wholesale, so changing one field means resending the object. That is deliberate: merge semantics on nested values are ambiguous — given a stored {"nested":{"a":1}}, a patch of {"nested":{"b":2}} could reasonably replace nested or merge into it — and "replaces wholesale" is not. Keep props small and it is a non-issue; anything you change often and independently is usually a category.

• Clusters carry none — forty vehicles do not share a battery level. Use /clusters/{id}/leaves to reach the members.

• In vector tiles, top-level scalars become MVT tags so you can style by them. Nested objects and arrays are skipped rather than stringified: a tile value is a scalar, and quietly turning {"a":1} into the text {"a":1} would produce a filter that silently never matches.

• A device with props has them as its GeoJSON properties, matching the JavaScript library. The device id is on the feature itself (feature.id), which is where GeoJSON puts it. A device without props still gets {"id": ...}.

• max_props_bytes caps each blob, default 1024, 0 refuses properties entirely. Memory is bounded by devices times this number, so it is a real limit: at a million devices every kilobyte allowed is a gigabyte promised. props_bytes in /stats reports what is actually held.

Anything you want to filter or group by belongs in categories instead — that is indexed and costs nothing per update, whereas props are opaque payload.

Filters are declared up front and matched exactly. Declare dimensions when you need more than one, and they combine:

curl -X PUT localhost:8080/v1/collections/fleet -H 'content-type: application/json' -d '{
  "dimensions": [
    {"name": "client", "values": ["1", "7", "22"], "multi": true},
    {"name": "status", "values": ["idle", "enroute"]}
  ],
  "filters": [["client"], ["status"], ["client", "status"]]
}'

curl 'localhost:8080/v1/collections/fleet/clusters?bbox=-47,-24,-46,-23&zoom=12&f.client=7&f.status=enroute'

A dimension takes either values (the labels) or capacity (how many distinct ones may exist), so you do not have to know every client id up front — with a capacity they are interned as they arrive, and the ceiling is how many can coexist rather than how large an id can get. On such a dimension a value nothing has reported yet is an empty result rather than a 400, since the server cannot tell it from a client that has not started reporting.

multi lets one device hold several values for a dimension — a vehicle owned by three clients — which a single category cannot express. Values ride in dims on the compact form, or in properties under the dimension's own name in GeoJSON, and re-reporting a device with different values re-files it even if it has not moved: a status change never moves the vehicle.

Each declared shape is a separate aggregate, which is what filtering costs.

A substring cannot be precomputed, so it gets its own query, which scans. Declare the fields it may search and they are extracted from props at ingest:

curl -X PUT localhost:8080/v1/collections/fleet -H 'content-type: application/json' \
  -d '{"text": ["plate", "driver"]}'

curl 'localhost:8080/v1/collections/fleet/clusters?bbox=…&zoom=12&where=plate~abc'
curl 'localhost:8080/v1/collections/fleet/clusters?bbox=…&zoom=12&where=plate~abc,driver=ana&f.client=7'

~ is a substring and = the whole value, both ignoring case; terms are ANDed and combine with ?f.. Results cluster exactly as an unfiltered query would, restricted to the matches. It costs O(devices), not O(markers) — about 1.5 ms over a 180,000-device fleet, against a lookup that stays flat however large the fleet grows. Use a dimension whenever the values can be declared and keep where for the search box. Not available on tiles, which refuse it rather than serving an unfiltered one.

Still out of reach: ranges, OR across values, and anything in props that is not a declared text field.

A field whose distinct values never stop growing is out too, at any capacity — a per-trip or per-order id. Every shape holds a running total per combination per device per tree level, so values that never repeat give each device its own bucket: the aggregates become a second copy of the fleet and any ceiling fills. That is a lookup, not a map filter — resolve it in your own database and ask the index about the ids it returns. Sizing and the trade-offs are in the JavaScript library's docs/FILTERING.md, which describes the same mechanism.

Unknown top-level fields are rejected with 422 rather than ignored: a stray "plate" outside props is a mistake, and silently discarding it means finding out weeks later that nothing was ever stored.

Tuning the clustering

Set per collection, at creation:

curl -X PUT localhost:8080/v1/collections/fleet -H 'content-type: application/json' \
  -d '{"radius":40,"extent":512,"max_zoom":16,"hysteresis":0.25,
       "categories":["idle","enroute","delivering"],"ttl_seconds":300}'

	default
radius	40	cluster radius in screen pixels
extent	512	tile extent those pixels are measured against
max_zoom	16	finest zoom the index resolves; queries are clamped to it, so points closer than the radius at this zoom (~44 m at the defaults) always return as a cluster
hysteresis	0.25	how far an assignment stretches before a point is re-homed
categories	[]	filter labels; a label's position in the list is its index. Shorthand for one dimension named cat
dimensions	[]	properties you can filter on: {"name", "values" | "capacity", "multi"}. Set this or categories, never both
text	[]	property fields ?where= may search; each costs one string per device
filters	one per dimension	combinations a query may name, e.g. [["client"],["status"],["client","status"]]
max_props_bytes	1024	largest per-device props blob; 0 refuses properties
ttl_seconds	300	drop a device that has not reported for this long

radius and extent are one knob in two parts: what matters is the ratio. At the defaults a cluster is 40px across on a 512px tile, so radius: 80, extent: 1024 clusters identically.

Too many markers, too cluttered — raise radius. 60–80 gives noticeably fewer, larger clusters. This is almost always the right dial, and the only one most people need.

Clusters break apart too early as you zoom in — raise max_zoom. It is the zoom at which clustering stops entirely. Hard-capped at 20: beyond that the fixed-point cell resolution runs out.

Markers reshuffle distractingly while vehicles move — raise hysteresis. This is the one people do not know they want. At 0 a point is re-homed the instant it strictly violates its covering constraint, so a vehicle idling on a boundary flickers between two clusters. At 0.25 the existing assignment survives 25% past that, trading a slightly looser worst-case radius — 2(1+h)·r_z instead of 2·r_z — for far fewer visible changes. Try 0.5 if churn is still visible; it also costs less CPU, because fewer moves take the repair path.

Filtering costs nothing extra at query time and nothing extra per update: a point belongs to exactly one category, so it touches exactly one aggregate slice per level regardless of how many categories exist.

Geometry is fixed once a collection exists. Re-PUTting the same values is idempotent; different values return 409. That is deliberate — silently keeping the old geometry would leave two deployments disagreeing about what a cluster means while both believe they configured it. To change it, drop and recreate, or use a new name. ttl_seconds is not geometry and can be changed the same way, but it too requires a recreate today.

Architecture

This is not a database. It holds no truth — the authority for where your devices are lives wherever the reports come from (Kafka, MQTT, your existing Redis, Postgres), and this is a materialised view of that stream. That single fact deletes the entire durability chapter: no write-ahead log, no snapshot format, no compaction, no replication protocol, no failover, no split-brain.

Cold start does the work persistence would have. At ~1 µs per insert, a 500,000 device fleet is rebuilt in about a second, so a process that dies is a process you restart.

It also means the scaling model is replication, not sharding:

position stream ──┬──> replica A (full index) ──> queries
                  ├──> replica B (full index) ──> queries
                  └──> replica C (full index) ──> queries

Read capacity scales by adding processes. No leader, no consensus, no rebalancing, because there is nothing to protect.

Deployment specifics — Docker, Kubernetes manifests, GKE, Cloud Run, ECS, EKS, sizing and operational limits — are in docs/DEPLOY.md.

Two consequences worth knowing before you deploy it:

• Do not shard geographically. An ordinary spatial index can be split by region, because an R-tree or grid query is spatially local. This hierarchy is globally coupled at coarse zooms — a cluster at z=0 spans continents, so a vehicle in Brazil and one in Angola can share a parent. Shard by collection (fleet A, fleet B), never by region.
• Route a client stickily. Replicas consuming updates in different interleavings build slightly different trees, so cluster ids and groupings can differ between them. Visually that is markers flickering as a client's polls bounce across replicas. Hashing the client or the viewport to a replica fixes it, and is far cheaper than forcing deterministic global ordering.

Set a TTL. A vehicle that stops reporting does not stop existing in the index, and clusters quietly fill with ghosts until every count on the map reads high. The sweep runs off the async runtime and drops devices in small batches, so it never holds the write lock long enough to stall queries.

How the index works

For every zoom level z it maintains a net of the live point set at scale r_z = radius / (extent · 2^z), with three invariants repaired locally on every update:

• Nesting — C_0 ⊆ C_1 ⊆ … ⊆ C_maxZoom ⊆ P
• Separation — distinct centers of C_z are more than r_z apart
• Covering — every p ∈ C_{z+1} \ C_z has a parent within r_z

Those are the invariants of a compressed net-tree over the Web-Mercator plane. Two guarantees follow and hold permanently: cluster radius ≤ 2·r_z, and cluster count ≤ |OPT(r_z/2)|. Full detail in crates/netcluster/README.md.

Tests

cargo test --release

• The index re-derives every invariant from scratch and is checked against long randomised operation streams, plus a differential test that replays 15,680 operations recorded from the JavaScript implementation and compares the complete device-to-representative map at every zoom.
• The MVT encoder is checked by decoding its own wire format.
• The server layer is checked for id interning, category resolution, expiry, tile coverage, and concurrent readers running against a live writer.
• The Node client has 17 integration tests that spawn the real server binary and drive it over HTTP, so they exercise the wire format rather than a mock (cd clients/node && npm test).

What this deliberately does not do

Geofencing, polygon WITHIN/INTERSECTS, webhooks, persistence, replication. Those belong to a full geospatial database — PostGIS and its peers — and you can run this alongside one for the map, which is the part they leave to you.

License

MIT