feat(pathways): configurable state prefix for pathway state - #106
Conversation
Two deployables that share one PostgreSQL connection string contended for a
single cluster leader lease. The lease key `pathway-cluster-leader` was a global
constant, not namespaced by pathwayName or anything else, and the state tables
used fixed names. The loser logged `Could not acquire lease, becoming worker`,
never started its pump, and never recovered while the other holder kept
renewing. Writes still succeeded, so the failure was silent at the API layer.
Every state library this package owns now accepts an optional `statePrefix`:
- `PostgresPathwayCoordinator` -> `${p}pathway_leases`, `${p}pathway_instances`
and a `leaseKey` of `${p}pathway-cluster-leader`
- `createPostgresPumpStateManagerFactory` -> `${p}pathway_pump_state`
- `PostgresPathwayState` -> `${p}pathway_state`
- `PathwayClusterOptions` -> `statePrefix` and an explicit `leaseKey` escape hatch
`ClusterManager` resolves its lease key from `leaseKey`, then `statePrefix`, then
`coordinator.leaseKey`, then the default, so the prefix is set in one place only.
It also logs the resolved key on start and on every role change, which makes this
class of contention visible instead of silent.
The default is no prefix. Existing deployments keep their exact table names and
lease key value with no migration.
A shared `pathway_instances` table also made the leader dial another deployable's
pods, which produced the repeating `Worker WS error` log. Prefixing fixes that too.
Prefixes reach SQL as part of an identifier, so they are validated against
`^[A-Za-z_][A-Za-z0-9_]*$` with a 40 character cap.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
4146c73 to
58e6c03
Compare
CI notePush events on this repository are not creating GitHub Actions runs. Two pushes to Actions itself is healthy: To get this PR verified I added Manual run: https://github.com/flowcore-io/flowcore-pathways/actions/runs/32989521917 The |
|
CI is green on the PR head commit
Run: https://github.com/flowcore-io/flowcore-pathways/actions/runs/32989521917 |
58e6c03 to
7ba51e2
Compare
Correction to the CI note above, and cleanupMy earlier comment blamed push-event delivery on this repository. That was wrong. The real cause was the GitHub Actions incident that started at 2026-08-26 15:11 UTC — The queued runs arrived once GitHub drained the backlog. Every one of them passed:
Since the normal push path works, the |
1 participant
Description
Two deployables that share ONE PostgreSQL connection string contend for a single cluster leader lease. The lease key
pathway-cluster-leaderis a global constant — not namespaced bypathwayName, data core, or anything else — and the state tables use fixed names.The loser logs
Could not acquire lease, becoming workerandNot starting pump — this instance is not the cluster leader, then never starts its pump. It never recovers while the other holder keeps renewing. Writes still succeed, so the failure is silent at the API layer and only the projection stalls.Observed in production on 2026-08-26 in
flowcore-io/flowcore-container-service, namespaceflowcore-public:compute-apiandcompute-reconcilershare a connection string by design. The reconciler took the lease and both compute-api pods stayed pumpless for more than 6 minutes.Task:
ed7ce624-3ce8-459f-adf8-b1b8a686ac47What changed
Every state library this package owns accepts an optional
statePrefix.statePrefix: "compute_api"PostgresPathwayStatepathway_statecompute_api_pathway_statePostgresPathwayCoordinatorpathway_leasescompute_api_pathway_leasesPostgresPathwayCoordinatorpathway_instancescompute_api_pathway_instancespathway_pump_statecompute_api_pathway_pump_statepathway-cluster-leadercompute_api_pathway-cluster-leaderClusterManagerresolves its lease key fromleaseKey, thenstatePrefix, thencoordinator.leaseKey, then the default. Setting the prefix on the coordinator alone is enough — one config point.ClusterManagernow also logs the resolvedleaseKeyon start and on every role change. Two deployables that report the same key are contending. That makes this class of failure visible instead of silent.Second defect
The task noted the reconciler leader logging
Worker WS erroragainstws://127.0.0.1:9091every 10 seconds. Same root cause: a sharedpathway_instancestable made the leader dial the other deployable's pods. Prefixing fixes it. Covered by a test.Safety
A prefix reaches SQL as part of an identifier, so it is validated against
^[A-Za-z_][A-Za-z0-9_]*$with a 40 character cap. Injection-shaped input is rejected at construction, not at query time.Backwards compatibility
The default is no prefix. Existing deployments keep their exact table names and their lease key value. There is no migration. An explicit
tableName,leasesTable,instancesTableorleaseKeystill overrides everything.Acceptance
src/pathways/state-prefix.tstests/cluster-state-prefix.test.ts,tests/postgres-state-prefix.test.tstests/state-prefix.test.ts,tests/postgres-state-prefix.test.tstests/postgres-state-prefix.test.tsType of Change
Testing
deno test -A, 38 passed, 186 steps, 0 failedtests/postgres-state-prefix.test.tsruns against a live database and drops what it createsChecklist
deno fmt,deno lint,deno check src/mod.ts,deno run -A bin/build-npm.tsall cleanFollow-up
Two Usable fragments describe the constant lease key and should be updated after release: Cluster Mode Complete Guide
f538652e-fdf5-4a3d-9a77-ddb32201093aand API Referencee2ff5c89-0f4f-4f17-aff6-870e9ac1327e.🤖 Generated with Claude Code