From aef01fb260cb97fa60712ea83a4e25a74e3f912d Mon Sep 17 00:00:00 2001 From: alowrydi Date: Fri, 7 Aug 2026 11:23:08 +0100 Subject: [PATCH 01/10] Initial draft for di.subscriptions --- di/subscriptions/VERSION | 1 + di/subscriptions/deps.q | 16 ++ di/subscriptions/init.q | 17 ++ di/subscriptions/subscriptions.md | 179 ++++++++++++ di/subscriptions/subscriptions.q | 448 ++++++++++++++++++++++++++++++ di/subscriptions/test.csv | 279 +++++++++++++++++++ di/subscriptions/test.q | 217 +++++++++++++++ 7 files changed, 1157 insertions(+) create mode 100644 di/subscriptions/VERSION create mode 100644 di/subscriptions/deps.q create mode 100644 di/subscriptions/init.q create mode 100644 di/subscriptions/subscriptions.md create mode 100644 di/subscriptions/subscriptions.q create mode 100644 di/subscriptions/test.csv create mode 100644 di/subscriptions/test.q diff --git a/di/subscriptions/VERSION b/di/subscriptions/VERSION new file mode 100644 index 00000000..6c6aa7cb --- /dev/null +++ b/di/subscriptions/VERSION @@ -0,0 +1 @@ +0.1.0 \ No newline at end of file diff --git a/di/subscriptions/deps.q b/di/subscriptions/deps.q new file mode 100644 index 00000000..31cf2218 --- /dev/null +++ b/di/subscriptions/deps.q @@ -0,0 +1,16 @@ +/ hard module dependencies and their minimum versions, validated by di.depcheck +/ di.subscriptions has NO hard dependencies: +/ - log and handlers are injected via init as dictionaries of functions. handlers is required for +/ the .z.pc observer that marks a dropped connection's subscriptions dead: .z.W alone cannot do +/ it, because kdb+ recycles handle numbers and a reused number revives a stale row +/ - handle resolution is the CALLER's job (di.rdb/di.wdb obtain a tickerplant handle from +/ di.servers.gethandlebytype and pass it in), so there is no di.servers edge - the plan's +/ dependency tree lists one, but every caller already imports di.servers directly and a +/ getsubscriptionhandles wrapper here would only duplicate it +/ - di.pubsub is the PUBLISHER side (TorQ's .stpps - the tickerplant's own subscriber registry). +/ a subscribing process never calls it; legacy .sub never referenced it either. the plan's +/ tree lists it for this module in error +/ - the tp log is replayed with kdb+'s native -11!, not via di.tplog: this module replays the +/ FIRST n messages, and di.tplog.check is built for the replay-everything caller +/ (tickerlogreplay.q, lastmessage 0W). see subscriptions.md for the full reasoning +deps:(`$())!(); diff --git a/di/subscriptions/init.q b/di/subscriptions/init.q new file mode 100644 index 00000000..cc7397e9 --- /dev/null +++ b/di/subscriptions/init.q @@ -0,0 +1,17 @@ +/ di.subscriptions - subscribe a process (rdb, wdb, chained tp) to a tickerplant: define the +/ subscribed tables at root from the tickerplant's schemas, replay the pre-subscription tp log +/ exactly once, then let live updates flow through the root upd. +/ ported from TorQ's code/common/subscriptions.q (.sub). the caller owns the connection. + +\l ::subscriptions.q + +/ module version, read from the VERSION file rather than hardcoded in the implementation, so a +/ release bump touches one plain-text file. +/ NB `version` STAYS in the export: di.depcheck reads it from the export dict (checkdepversion), +/ and reports "exports no version" - failing the dependency check - if a module drops it +version:first read0`:::VERSION + +/ NB: export:([...]) EVALUATES each name, so it can only list names that already exist. +/ init and getapimeta are framework plumbing di.torq calls by convention; every other name here has +/ a getapimeta row, which the test suite asserts +export:([init;teardown;version;getapimeta;subscribe;subscribed;getsubscriptions]) diff --git a/di/subscriptions/subscriptions.md b/di/subscriptions/subscriptions.md new file mode 100644 index 00000000..e9fb40f2 --- /dev/null +++ b/di/subscriptions/subscriptions.md @@ -0,0 +1,179 @@ +# di.subscriptions + +Tickerplant subscription management for kdb+ subscriber processes (RDB, WDB, chained TP). Fetches table schemas and log details from a tickerplant in a single call, defines the subscribed tables at root, replays the pre-subscription tickerplant log exactly once, and records the subscription in an inspectable registry. Live updates then flow through the root `upd` as normal. + +--- + +## Features + +- Subscribe over an already-open tickerplant handle - the caller owns the connection, so this module never opens, retries or closes one +- Subscribe to all tables and syms, or to any subset, with sym filtering applied to the log replay as well as to the live feed +- Define subscribed tables at root from the schemas the tickerplant returns, preserving their attributes (e.g. `` `g# `` on `sym`) +- Replay exactly the messages the tickerplant had logged at the instant of subscription, so messages that also arrive on the live feed are not applied twice +- Verify every log before defining a single table, so a truncated or unreadable log fails with the process untouched +- Handle every payload shape a tickerplant may log - a list of columns, a table, a dict, or a single atom row - by resolving the `sym` column by name rather than position +- Replay across several log files, as a segmented tickerplant writes one log per table +- Track live subscriptions in a registry whose `active` flag is maintained from `.z.pc` and cross-checked against `.z.W` +- Refuse to re-subscribe a table that already has a live subscription, rather than silently redefining it and replaying into it again +- Speaks TorQ's real `subdetails` protocol rather than a new one, and the remote entry point name is configurable + +--- + +## Dependencies + +| Dependency | Key | Required | Description | +|---|---|---|---| +| logger | `` `log `` | yes | `info`, `warn` and `error` - each binary `{[c;m]}` where `c` is a symbol context and `m` is a string. All three are called by this module | +| handlers | `` `handlers `` | yes | `register` and `remove`, per the `di.handlers` contract. Used to install a `.z.pc` observer that marks a dropped connection's subscriptions dead | + +**Hard dependencies:** none. Both dependencies are injected, so the module imports no other `di.*` module. + +Both deps must be passed to `init` inside the `deps` dict. The module throws immediately if either is absent or malformed - there is no fallback logger and no degraded no-handlers mode. The `log` dict must already match the binary `{[c;m]}` contract; the module does not detect or adapt other shapes (e.g. a raw `kx.log` instance, which is monadic). To use `di.log`, pass its `logdict``log`. + +Handle resolution is the **caller's** job. `di.rdb` and `di.wdb` obtain a tickerplant handle from `di.servers.gethandlebytype` and pass it in, so there is no `di.servers` dependency here. + +The configuration key `subdetailsfunc` is optional - omit it and the module calls the tickerplant's `subdetails`. See Initialisation. + +--- + +## Initialisation + +`init[deps]` takes a single dictionary combining the `log` and `handlers` dependencies with any configuration overrides. + +| Key | Required | Description | +|---|---|---| +| `` `log `` | yes | Log dep - `info`, `warn` and `error`, each `{[c;m]}` | +| `` `handlers `` | yes | Handlers dep - at minimum `register` and `remove` | +| `` `subdetailsfunc `` | no | Symbol naming the tickerplant-side function to call. Default: `` `subdetails `` | + +`init` must be called before any operational function - `subscribe`, `subscribed`, `getsubscriptions` and `teardown` each throw a clear error if it has not been. `getapimeta` and `version` are metadata and deliberately work without it, so `di.torq` can collect api rows and `di.depcheck` can read the version before anything is initialised. `init` registers a `.z.pc` observer through `di.handlers` and is idempotent - a second call refreshes the dependencies and replaces the registration in place rather than duplicating it, and leaves the registry intact. + +Build `deps` as one multi-key dictionary. Do not join `di.log`'s `logdict` to another single-key dict: `` enlist[`k]!enlist somedict `` puts a table on the value side, and joining two of them throws `` 'mismatch `` at the call site before `init` runs. + +```q +`log`handlers!(logging.logdict`log;handlerdep) / correct +logging.logdict,`handlers`subdetailsfunc!(handlerdep;`subdetails) / also correct +logging.logdict,enlist[`handlers]!enlist handlerdep / 'mismatch +``` + +--- + +## Exported Functions + +### `init[deps]` +Initialise the module. Validates the log and handlers dependencies, applies config, and registers the `.z.pc` observer. +```q +sub.init[`log`handlers!(logging.logdict`log;`register`remove!(handlers.register;handlers.remove))] +``` + +### `subscribe[tph;tabs;syms;setschema;replay]` +Subscribe over an already-open tickerplant handle. `tph` is an int handle (or a function standing in for one). `tabs` and `syms` are `` ` `` for all, or one or more symbols - a bare symbol atom names a single table or sym and is normalised to a list. `setschema` defines the returned schemas at root; `replay` replays the pre-subscription log and requires a root-level `upd`. + +Returns a dictionary of `subtables`, `tplogdate`, `rowcounts` and `date`, plus `logdir` when the tickerplant supplies one. +```q +sub.subscribe[tph;`;`;1b;1b] / all tables, all syms, define schemas, replay +sub.subscribe[tph;`trade`quote;enlist`AAPL;1b;1b] / two tables, one sym +sub.subscribe[tph;`;`;0b;0b] / subscribe only - keep my schemas, skip the replay +/ `subtables`tplogdate`rowcounts`date!(`trade`quote;2025.06.01;`trade`quote!(1042;3311);2025.06.01) +``` + +### `subscribed[]` +Return whether any subscription is currently live. The connectivity check a subscriber process needs before declaring itself ready. +```q +sub.subscribed[] +/ 1b +``` + +### `getsubscriptions[]` +Return the subscription registry: the handle, subscribed tables, syms, subscription timestamp, and a live/dead `active` flag. +```q +sub.getsubscriptions[] +/ handle tabs syms subtime active +/ ------------------------------------------------------------ +/ 4 trade quote 2025.06.01D08:00:00.000000000 1 +``` +`tabs` and `syms` are general columns, so symbols display without backticks and an all-syms subscription shows an empty `syms`. + +### `teardown[]` +Remove the `.z.pc` registration installed by `init`, leaving no process-global residue. +```q +sub.teardown[] +``` + +### `getapimeta[]` +Return this module's api metadata, one row per callable export, for `di.torq` to register with `di.api`. +```q +sub.getapimeta[] +``` + +### `version` +The module version string, read from the `VERSION` file. +```q +sub.version +/ "0.1.0" +``` + +--- + +## Usage Example + +```q +/ log and handlers deps must already match their contracts - use di.log and di.handlers: +logging:use`di.log +handlers:use`di.handlers +handlers.init[logging.logdict] +handlerdep:`register`remove!(handlers.register;handlers.remove) + +/ load and initialise +sub:use`di.subscriptions +sub.init[`log`handlers!(logging.logdict`log;handlerdep)] + +/ define the root upd the replay will drive, exactly as the subscriber process does +upd:{[t;x] @[`.;t;{[tab;d] tab upsert $[98h=type d;d;flip (cols tab)!d]}[;x]];} + +/ the caller resolves the tickerplant handle. NB di.servers is not merged yet (feature-server), so +/ on main today obtain the handle however the process already does - any open handle works +servers:use`di.servers +tph:servers.gethandlebytype[`tickerplant;`any] + +/ subscribe to everything and replay the log +r:sub.subscribe[tph;`;`;1b;1b] +r`subtables / the tables now defined at root and receiving live updates +r`tplogdate / the tickerplant log date, for setting the partition + +/ check connectivity +sub.subscribed[] +sub.getsubscriptions[] +``` + +--- + +## Running Tests + +```q +k4unit:use`di.k4unit +k4unit.moduletest`di.subscriptions +``` + +Run in a fresh q session - the integration layer spawns and kills a real q process, so do not interleave with other modules' tests. Needs `QHOME` set (the peer is launched via `$QHOME/bin/q`) and `di.os` on `QPATH` (the harness uses `os.abspath` to load `test.q`). + +The suite is 209 assertions and wires the real merged `di.log` and `di.handlers` rather than mocks, so both injected contracts are proven end to end. It covers: pre-`init` guards on every export; dependency validation (non-dict deps, each missing key, each malformed value); `init` arity asserted by observable effect rather than by "it did not throw"; idempotent re-init; the `VERSION` file and exported `version`; `getapimeta`; all-tables and sym-filtered replay; all four payload shapes a tickerplant may log; symbol-atom table and sym selectors; a multi-table log where only the requested table is defined; replay across two log files; requested-but-absent and offered-but-unrequested tables; a configured `subdetailsfunc`; partial, unreadable, absent and empty logs; a replay requested with no root `upd`; malformed `schemalist` shapes; the double-subscribe guard; malformed tickerplant responses; `setschema:0b` preserving a populated table; `logdir` passthrough; and input validation on every argument. + +The integration block subscribes to a genuinely separate tickerplant process over IPC, kills it, confirms `.z.pc` marks the subscription dead, reconnects onto a recycled handle number, and confirms the stale row stays dead and the re-subscribe succeeds. + +--- + +## Notes + +- Replay uses kdb+'s native `-11!` directly rather than `di.tplog`. This module replays the first *n* messages; `di.tplog.check` is built for the replay-everything caller and would rewrite an entire log to a `.good` file when corruption lies beyond the messages actually needed, and cannot report the good-message count the preflight requires +- Every log is verified with the non-executing `-11!(-2;logfile)` streaming count *before* any table is defined. This matters: `-11!(n;logfile)` past a corruption point replays the good messages and only then throws, leaving tables half populated +- The subscribed set is what was requested intersected with what the tickerplant offered. A requested table the tickerplant does not return is logged at `warn` and skipped; a table it volunteers unasked is logged at `warn` and ignored - neither defined nor replayed +- Liveness needs both signals, which is why `handlers` is a required dependency. `.z.pc` fires when the tickerplant drops, at the instant of the drop and before kdb+ can reissue the handle number; `.z.W` catches a handle the caller closed itself, which does not fire `.z.pc`. `.z.W` alone is not sufficient - kdb+ reissues the lowest free descriptor, so a recycled number would revive a stale registry row +- The double-subscribe guard runs after the tickerplant has been asked what it offers, so it compares resolved table lists. An all-tables subscribe is refused only where it genuinely overlaps a live subscription, not merely because some other subscription exists +- Tables are created at root with `@[`.;name;:;schema]`, and `-11!` resolves `upd` at root even when invoked from module context. A root-level `upd` must therefore be defined before calling `subscribe` with `replay` set - this is **enforced**, not merely expected: without it every replayed message would be silently discarded and the narrowed path would leave its no-op stand-in bound at root, so the live feed would vanish into it too +- A malformed `subdetails` response is rejected before anything is defined, including a `schemalist` that is not a list or whose entries are not `(tablename;schema)` pairs. An empty entry is tolerated, as legacy does +- `setschema:0b` skips table definition entirely, so a subscriber that pre-loads its own schema keeps it. With `setschema:1b` the returned schema replaces whatever is at root, which is why re-subscribing a live table is refused +- `subdetails` is TorQ's real protocol, defined at root by `chainedtp.q` and `segmentedtickerplant.q`, returning `` `schemalist`logfilelist`rowcounts`date `` and optionally `` `logdir ``. The key names and shapes here were taken from that source and are exercised against a tickerplant process built to the same protocol in the integration tests; the module has **not** yet been run against a live TorQ chained or segmented tickerplant. `di.tickerplant` is not built yet - when it lands, point `subdetailsfunc` at its entry point if the name differs +- Not in v1: auto-reconnect and resubscribe on tickerplant bounce, filtered-column subscriptions, and remote-log streaming (the subscriber is assumed to share the tickerplant's filesystem, the classic tick assumption) +- The modularisation plan lists `di.servers` and `di.pubsub` as dependencies of this module. Neither is used: handle resolution is the caller's job, and `di.pubsub` is the publisher side - the tickerplant's own subscriber registry - which a subscribing process never calls +- One residual liveness gap: a handle the caller closes itself, without the tickerplant dying, whose number is then reissued to an unrelated connection, can leave a stale row appearing live. Tickerplant death and local close without reuse are both exact diff --git a/di/subscriptions/subscriptions.q b/di/subscriptions/subscriptions.q new file mode 100644 index 00000000..6271ca55 --- /dev/null +++ b/di/subscriptions/subscriptions.q @@ -0,0 +1,448 @@ +/ subscribe a process (rdb, wdb, chained tp) to a tickerplant: fetch the schemas and log details in +/ one bundled call, define the subscribed tables at ROOT, replay the pre-subscription tp log exactly +/ once, then let live updates flow through the root upd. ported from TorQ's code/common/subscriptions.q +/ (.sub) - see subscriptions.md for scope, omissions and the design rationale. +/ the version lives in the VERSION file and is read by init.q + +/ constants (load-time) + +/ registry template - the live copy is .z.m.subscriptions, populated by subscribe. handle is a +/ GENERAL column: production always stores an int handle, but the unit tests drive a function +/ standing in for a handle (h(msg) applies to either), and an int column could not hold one. +/ active is MAINTAINED (set 0b by the .z.pc observer), not derived - see islive +subscriptionsschema:([]handle:();tabs:();syms:();subtime:`timestamp$();active:`boolean$()); + +/ the keys a tickerplant's subdetails response must carry. these names and shapes are taken from the +/ real, shipped TorQ protocol - code/processes/chainedtp.q and segmentedtickerplant.q both define +/ subdetails at root - rather than invented here. NB the module has not yet been run against a live +/ TorQ tickerplant; the integration tests drive a process built to the same protocol +requireddetailkeys:`schemalist`logfilelist`rowcounts`date; + +/ what subscribe accepts as a tickerplant handle: an int handle, or a function standing in for one +validhandletypes:-7 -6 100 104h; + +/ the tickerplant-side function subscribe calls, unless config overrides it +defaultsubdetailsfunc:`subdetails; + +/ internal helpers + +initialised:{[] + / has init run? a direct (module-rewritten) reference detects prior setup without touching root + :@[{.z.m.subscriptions;1b};::;{[e] :0b}]; + }; + +requireinit:{[ctx] + / every exported function except init depends on init having wired the logger. there is no default + / logger, so without this an early call dies with a bare 'type instead of a usable message + if[not initialised[]; + '"di.subscriptions: ",string[ctx],": init must be called before any other function"]; + }; + +raiseerror:{[ctx;msg] + / log an error under ctx then signal it, so a failure is observable in the log and not only as a + / throw. init's own dependency validation is the one exception - the logger is not wired yet + .z.m.logerr[ctx;msg]; + '"di.subscriptions: ",string[ctx],": ",msg; + }; + +requirehandle:{[ctx;tph] + / the caller owns the connection - di.rdb gets one from di.servers.gethandlebytype and passes it in + if[not type[tph] in validhandletypes; + raiseerror[ctx;"tph must be an open tickerplant handle or a function, got type ",.Q.s1 type tph]]; + }; + +requiretabspec:{[ctx;nm;x] + / a table/sym selector is either ` (all) or one or more symbols + if[not 11h=abs type x; + raiseerror[ctx;nm," must be ` (all) or a symbol vector, got type ",.Q.s1 type x]]; + }; + +normspec:{[x] + / normalise a table/sym selector to a LIST, leaving the ` all-sentinel alone. a bare symbol atom is + / a natural way to name one table (subscribe[h;`trade;..]) and legacy accepted it - subscriptions.q + / enlists both selectors the same way. without this an atom reaches `inter` and throws a bare 'type + / that bypasses raiseerror and never reaches the log + :$[x~`;x;(),x]; + }; + +requireflag:{[ctx;nm;x] + / a boolean switch + if[not -1h=type x; + raiseerror[ctx;nm," must be a boolean, got type ",.Q.s1 type x]]; + }; + +islive:{[stored;h] + / is this registry row's subscription still live? TWO complementary signals, because neither alone + / is sufficient (both measured): + / stored - set 0b by the .z.pc observer the instant the tickerplant drops. exact, and immune to + / handle-number recycling, which a .z.W probe alone is NOT: kdb+ hands back the lowest + / free descriptor, so a reused number would otherwise revive a stale row and make the + / duplicate guard refuse a legitimate re-subscribe after a reconnect + / .z.W - catches a handle the CALLER closed itself, which does not fire .z.pc at all + / a non-int handle (the function the unit tests pass) cannot be probed, so only stored applies + if[not stored; :0b]; + :$[type[h] in -7 -6h; h in key .z.W; 1b]; + }; + +markdead:{[wh] + / the .z.pc observer - every subscription on a dropped connection is dead. registered by init via + / the injected handlers dependency; di.handlers calls it, and this lambda's compile-time rewrite + / means it still updates THIS module's state when it does. + / NB handle is a general column, so match-each (~\:) rather than =, which would throw on a + / function element + .z.m.subscriptions:update active:0b from .z.m.subscriptions where handle~\:wh; + }; + +activesubscriptions:{[] + / internal - the registry with the effective live flag folded into active. + / NB islive is hoisted into a LOCAL first: a q-sql select/update/where clause inside module code + / cannot resolve a module-level name (it throws 'islive) - only function-locals and column names + / resolve there. the from-target is fine, it is only the clauses that are affected + / the boolean cast keeps the column type stable: on an EMPTY registry each' yields a general empty + / list, which would report active as type 0h rather than the boolean the api metadata promises + f:islive; + :update active:`boolean$f'[active;handle] from .z.m.subscriptions; + }; + +/ init + +init:{[deps] + / wire the injected dependencies (log and handlers - both REQUIRED, never defaulted) and this + / module's config. ONE dict carrying dependency and config keys side by side - the call shape + / di.torq wires every module with. + / e.g. sub.init[`log`handlers!(logging.logdict`log;handlerdep)] + / NB build deps as ONE multi-key dict. joining logdict to a single-key dict - i.e. + / logdict,enlist[`handlers]!enlist handlerdep - throws 'mismatch: both value sides are tables + / init has ONE process-global side effect: a .z.pc observer registered through di.handlers, which + / marks a dropped connection's subscriptions dead. teardown removes it. the registration is + / idempotent - di.handlers replaces a duplicate [event;name] in place - so a second init is safe + if[99h<>type deps; + '"di.subscriptions: deps must be a dict with `log and `handlers keys - see di.log, di.handlers"]; + if[not all `log`handlers in key deps; + '"di.subscriptions: log and handlers dependencies are required; pass `log (`info`warn`error) and ", + "`handlers (`register`remove) - see di.log, di.handlers; got: ",(", " sv string key deps)]; + if[99h<>type deps`log; + '"di.subscriptions: log value must be a dict; pass `info`warn`error functions - see di.log"]; + if[not all `info`warn`error in key deps`log; + '"di.subscriptions: log dict must have `info`warn`error keys; got: ",(", " sv string key deps`log)]; + if[99h<>type deps`handlers; + '"di.subscriptions: handlers value must be a dict; pass `register`remove functions - see di.handlers"]; + / only register/remove are required - this module calls no others + if[not all `register`remove in key deps`handlers; + '"di.subscriptions: handlers dict must have `register`remove keys; got: ",(", " sv string key deps`handlers)]; + / resolve and validate config BEFORE any state is mutated: a rejected re-init must not leave the + / module half-configured with a wired logger and an invalid tickerplant entry point + sdf:$[`subdetailsfunc in key deps;deps`subdetailsfunc;defaultsubdetailsfunc]; + if[not -11h=type sdf; + '"di.subscriptions: subdetailsfunc must be a symbol naming the tickerplant-side function"]; + fresh:not initialised[]; + .z.m.loginfo:(deps`log)`info; + .z.m.logwarn:(deps`log)`warn; + .z.m.logerr:(deps`log)`error; + .z.m.register:(deps`handlers)`register; + .z.m.removehandler:(deps`handlers)`remove; + .z.m.subdetailsfunc:sdf; + if[fresh;.z.m.subscriptions:subscriptionsschema]; + / .z.pc is a SIMPLE (observer) event in di.handlers - side-effect only, fan-out - so the phase is + / ` (null) and this coexists with every other .z.pc registrant in the priority-ordered chain + .z.m.register[`.z.pc;`;`subscriptions;0j;markdead]; + .z.m.loginfo[`init;"di.subscriptions initialised - tickerplant entry point ",string .z.m.subdetailsfunc]; + }; + +teardown:{[] + / release the .z.pc registration init installed, leaving no process-global residue. paired with + / init's one side effect - a module whose init registers nothing needs no teardown, this one does + requireinit[`teardown]; + .z.m.removehandler[`.z.pc;`;`subscriptions]; + .z.m.loginfo[`teardown;"di.subscriptions .z.pc registration removed"]; + }; + +/ subscription + +fetchdetails:{[tph;tabs;syms] + / one bundled round trip to the tickerplant. legacy issued a separate tablelist call first to learn + / what was available; the bundled response makes that round trip unnecessary + r:@[{[h;m] (1b;h m)}[tph];(.z.m.subdetailsfunc;tabs;syms);{[e] (0b;e)}]; + if[not first r; + raiseerror[`subscribe;"tickerplant ",(string .z.m.subdetailsfunc)," call failed: ",last r]]; + d:last r; + if[99h<>type d; + raiseerror[`subscribe;(string .z.m.subdetailsfunc)," must return a dictionary, got type ",.Q.s1 type d]]; + if[not all requireddetailkeys in key d; + raiseerror[`subscribe;(string .z.m.subdetailsfunc)," response must carry ",(", " sv string requireddetailkeys), + "; got: ",(", " sv string key d)]]; + / validate the schemalist SHAPE here rather than letting a malformed entry surface later as a bare + / 'rank out of the table-creation amend, which would bypass raiseerror and never reach the log. + / an empty entry is legitimate - legacy filters those out - so only non-empty ones are checked + sl:d`schemalist; + if[0>type sl; + raiseerror[`subscribe;(string .z.m.subdetailsfunc)," schemalist must be a list of (tablename;schema) pairs"]]; + entries:sl where not 0=count each sl; + if[not all 2=count each entries; + raiseerror[`subscribe;(string .z.m.subdetailsfunc)," schemalist entries must be (tablename;schema) pairs"]]; + if[not all -11h=type each entries[;0]; + raiseerror[`subscribe;(string .z.m.subdetailsfunc)," schemalist entries must name their table with a symbol"]]; + :d; + }; + +guardduplicate:{[wanted] + / refuse to re-subscribe a table that already has a LIVE subscription: a second subscribe would + / redefine the table (discarding its rows) and replay the log into it again. this is deliberately + / NOT a port of legacy's reducesubs - no instrument-level dedup, no partial-overlap splitting, just + / enough to fail loud instead of failing open. re-subscribing after the tickerplant has gone is + / legitimate and stays allowed, which is why the check is on LIVE rows rather than history. + / NB this runs AFTER the tickerplant has been asked what it offers, so `wanted` is the resolved + / table list. checking before that could only compare the raw request, which for ` (all tables) is + / unresolvable - it would have to refuse ANY all-tables subscribe while ANY subscription was live, + / even to a completely disjoint set of tables + live:select from activesubscriptions[] where active; + if[0=count live; :(::)]; + held:distinct (),raze live`tabs; + clash:(),wanted inter held; + if[0type first x; flip c!enlist each x; flip c!x]; + }; + +replayfilter:{[origupd;subtabs;syms;t;x] + / installed as the root upd for the duration of a narrowed replay: forward only the tables and syms + / this subscription asked for, to the real upd + if[not t in subtabs; :(::)]; + if[syms~`; :origupd[t;x]]; + d:payloadtable[t;x]; + if[not `sym in cols d; :origupd[t;x]]; + origupd[t;select from d where sym in syms]; + }; + +logentries:{[details] + / internal - the non-empty (messagecount;logfile) pairs the tickerplant reported. logfilelist is a + / LIST because a segmented tickerplant writes one log per table + lfl:details`logfilelist; + :lfl where not 0=count each lfl; + }; + +preflightone:{[entry] + / internal - confirm one log really holds the messages the tickerplant claims, WITHOUT replaying + nmsg:first entry; + lf:last entry; + if[null lf; + raiseerror[`replay;"tickerplant reported a message count but no log file"]]; + if[0>=nmsg; :(::)]; + good:goodcount lf; + if[good=nmsg; + .z.m.loginfo[`replay;"nothing to replay from ",string lf]; + :(::)]; + n:$[alltabs and syms~`;replayall[nmsg;lf];replaynarrowed[nmsg;lf;wanted;syms]]; + .z.m.loginfo[`replay;"replayed ",(string n)," message(s) from ",string lf]; + }; + +replaylogs:{[details;wanted;syms;alltabs] + / replay every pre-subscription log the tickerplant reported + lfl:logentries details; + if[0=count lfl; + .z.m.loginfo[`replay;"tickerplant reported no log file to replay"]; + :(::)]; + replayone[;wanted;syms;alltabs] each lfl; + }; + +/ return shape + +tplogdate:{[details] + / the date in the tp log file name, as legacy derives it, falling back to the date the tickerplant + / reported when the name does not carry one + / reuse logentries rather than repeating its filter, so the two cannot drift apart + lfl:logentries details; + if[0=count lfl; :details`date]; + :(details`date)^@[{"D"$-10 sublist string last first x};lfl;{[e] :0Nd}]; + }; + +buildreturn:{[details;subtabs] + / the shape legacy's callers actually consume: rdb.q reads subtables and tplogdate, wdb.q reads + / tplogdate for fixpartition, chainedtp.q reads date and rowcounts (its .u.d and .u.icounts) + r:`subtables`tplogdate`rowcounts`date!(subtabs;tplogdate details;details`rowcounts;details`date); + :$[`logdir in key details;r,enlist[`logdir]!enlist details`logdir;r]; + }; + +/ public api + +subscribe:{[tph;tabs;syms;setschema;replay] + / subscribe over an ALREADY-OPEN tickerplant handle - this module never opens a connection; the + / caller resolves one (di.servers.gethandlebytype) and passes it in. + / tabs/syms: ` for all, or one or more symbols (a bare atom is normalised to a list). + / setschema: define the returned schemas at root. + / replay: replay the pre-subscription tp log - requires a root-level upd. + / returns `subtables`tplogdate`rowcounts`date, plus `logdir when the tickerplant supplied one + requireinit[`subscribe]; + requirehandle[`subscribe;tph]; + requiretabspec[`subscribe;"tabs";tabs]; + requiretabspec[`subscribe;"syms";syms]; + requireflag[`subscribe;"setschema";setschema]; + requireflag[`subscribe;"replay";replay]; + / normalise a bare symbol atom to a list before anything indexes or intersects it + tabs:normspec tabs; + syms:normspec syms; + details:fetchdetails[tph;tabs;syms]; + schemapairs:(details`schemalist) where not 0=count each details`schemalist; + offered:(),schemapairs[;0]; + / what we actually subscribe to is what we ASKED FOR intersected with what the tickerplant offered + / - not simply everything it returned. driving the replay filter off the offered set instead would + / replay tables that were never requested + wanted:$[tabs~`;offered;(),tabs inter offered]; + if[0=count wanted; + raiseerror[`subscribe;"tickerplant returned no schema for the requested table(s) - nothing to subscribe to"]]; + guardduplicate[wanted]; + warnmissing[tabs;offered]; + warnextra[tabs;offered]; + schemapairs:schemapairs where schemapairs[;0] in wanted; + / preflight EVERY log before defining a single table, so a short log leaves the process untouched + if[replay; + requirerootupd[]; + preflightlogs[details]]; + if[setschema;createtables[schemapairs]]; + if[replay;replaylogs[details;wanted;syms;tabs~`]]; + / catenate+reassign, NOT (`name insert row): a symbol-mediated insert resolves the LITERAL name at + / root and would miss the compile-time module-local rewrite a source-level .z.m.subscriptions gets + .z.m.subscriptions:.z.m.subscriptions, + ([]handle:enlist tph;tabs:enlist wanted;syms:enlist syms;subtime:enlist .z.p;active:enlist 1b); + .z.m.loginfo[`subscribe;"subscribed to ",(", " sv string wanted)," on tickerplant handle ",.Q.s1 tph]; + :buildreturn[details;wanted]; + }; + +subscribed:{[] + / is any subscription currently live? the connectivity check legacy's .rdb.notpconnected[] needs + requireinit[`subscribed]; + :any (),exec active from activesubscriptions[]; + }; + +getsubscriptions:{[] + / the subscription registry. active combines the flag the .z.pc observer maintains with a .z.W + / check for a handle the caller closed itself - see islive + requireinit[`getsubscriptions]; + :activesubscriptions[]; + }; + +/ api metadata + +getapimeta:{[] + / one row per CALLABLE export, for di.torq to register with di.api. init and getapimeta are omitted + / as framework plumbing. names are bare; di.torq applies the process-wide qualification + :flip `name`public`descrip`params`return!flip( + (`version; 1b; "module version string"; + "[]"; "string: version"); + (`teardown; 1b; "release the .z.pc registration installed by init"; + "[]"; "null"); + (`subscribe; 1b; "subscribe over an open tickerplant handle, optionally defining schemas and replaying the log"; + "[int|function: tickerplant handle; symbol(list): tables (` for all); symbol(list): syms (` for all); boolean: setschema; boolean: replay]"; + "dict: subtables, tplogdate, rowcounts, date (and logdir if supplied)"); + (`subscribed; 1b; "is any subscription currently live?"; + "[]"; "boolean: at least one live subscription"); + (`getsubscriptions; 1b; "the subscription registry, with a live/active flag per subscription"; + "[]"; "table: handle, tabs, syms, subtime, active")); + }; diff --git a/di/subscriptions/test.csv b/di/subscriptions/test.csv new file mode 100644 index 00000000..d73b191a --- /dev/null +++ b/di/subscriptions/test.csv @@ -0,0 +1,279 @@ +action,ms,bytes,lang,code,repeat,minver,comment +comment,,,,,,,setup - load the module and the fixtures. NB init is deliberately NOT called in a before row: the pre-init guard tests below must run first +before,0,0,q,sub:use`di.subscriptions,1,1,load di.subscriptions +before,0,0,q,os:use`di.os,1,1,di.os for abspath (test harness only - not a module dependency) +before,0,0,q,"system ""l "", os.abspath[""di/subscriptions/test.q""]",1,1,load the fixture helpers +before,0,0,q,setupfixture[],1,1,create the temp log directory + +comment,,,,,,,every exported function except init must refuse to run before init - there is no default logger +fail,0,0,q,sub.subscribed[],1,1,subscribed[] before init errors +fail,0,0,q,sub.getsubscriptions[],1,1,getsubscriptions[] before init errors +fail,0,0,q,sub.teardown[],1,1,teardown[] before init errors +fail,0,0,q,sub.subscribe[tphfor[`x;0;`:/nosuch];`;`;1b;1b],1,1,subscribe before init errors + +comment,,,,,,,init dependency validation - log and handlers are both required and never defaulted +fail,0,0,q,sub.init[(::)],1,1,init rejects a non-dict deps +fail,0,0,q,sub.init[()!()],1,1,init rejects deps missing both keys +fail,0,0,q,sub.init[enlist[`log]!enlist caplog],1,1,init rejects deps missing the handlers key +fail,0,0,q,sub.init[enlist[`handlers]!enlist handlerdep[]],1,1,init rejects deps missing the log key +fail,0,0,q,sub.init[`log`handlers!(42;handlerdep[])],1,1,init rejects a non-dict log value +fail,0,0,q,"sub.init[`log`handlers!((enlist[`info]!enlist caplog`info);handlerdep[])]",1,1,init rejects a log dict missing warn and error +fail,0,0,q,sub.init[`log`handlers!(caplog;42)],1,1,init rejects a non-dict handlers value +fail,0,0,q,"sub.init[`log`handlers!(caplog;enlist[`register]!enlist handlerdep[]`register)]",1,1,init rejects a handlers dict missing remove +fail,0,0,q,"sub.init[deps[],enlist[`subdetailsfunc]!enlist ""subdetails""]",1,1,init rejects a non-symbol subdetailsfunc +run,0,0,q,"errstr:@[{sub.init[()!()]};(::);{[e] e}]",1,1,capture the error text from a bad init +true,0,0,q,"0.z.i,1,1,the peer really is a separate process +true,0,0,q,`schemalist`logfilelist`rowcounts`date~key PH(`subdetails;`;`),1,1,the peer serves the real TorQ subdetails protocol +run,0,0,q,resetupd[],1,1,reset the upd counter +comment,,,,,,,an all-tables subscribe works here even though many unit subscriptions are still live - the guard is table-precise +run,0,0,q,rp:sub.subscribe[PH;`;`;1b;1b],1,1,subscribe over the real handle +true,0,0,q,`peertrade in tables[],1,1,the table was created at root from the schema sent over IPC +true,0,0,q,6=count peertrade,1,1,all six logged messages replayed +true,0,0,q,`g=attr peertrade`sym,1,1,the g# attribute survived the IPC round trip +true,0,0,q,6=updcalls,1,1,the replay drove the root upd once per message +true,0,0,q,PH~first exec handle from sub.getsubscriptions[] where handle~\:PH,1,1,the registry holds the real int handle +true,0,0,q,any exec active from sub.getsubscriptions[] where handle~\:PH,1,1,the subscription over the real handle is live + +comment,,,,,,,kill the tickerplant - the .z.pc observer must mark the subscription dead +run,0,0,q,killpeer[],1,1,kill the peer process +run,0,0,q,noticedrop[PH],1,1,touch the handle so the drop is detected and .z.pc fires +true,0,0,q,not any exec active from sub.getsubscriptions[] where handle~\:PH,1,1,the row for the dropped handle was marked dead + +comment,,,,,,,restart and reconnect - kdb+ recycles the handle number and the stale row must NOT revive +run,0,0,q,spawnpeer[],1,1,restart the tickerplant +run,0,0,q,PH2:peerhandle[],1,1,reconnect +true,0,0,q,PH~PH2,1,1,kdb+ handed back the same handle number (the recycling a .z.W probe alone cannot see through) +true,0,0,q,not any exec active from sub.getsubscriptions[] where handle~\:PH2,1,1,the stale row stays dead despite the number being reused +run,0,0,q,resetupd[],1,1,reset the upd counter +run,0,0,q,@[`.;`peertrade;:;0#peertrade],1,1,clear the table before replaying again +run,0,0,q,sub.subscribe[PH2;`;`;1b;1b],1,1,re-subscribing after a reconnect is allowed +true,0,0,q,6=count peertrade,1,1,the log replayed again on the new handle +true,0,0,q,2=count select from sub.getsubscriptions[] where handle~\:PH2,1,1,both the stale and the new row are for the recycled handle number +true,0,0,q,1=sum exec active from sub.getsubscriptions[] where handle~\:PH2,1,1,exactly one of them is live - the stale one stayed dead + +comment,,,,,,,LIVE updates after the replay - the exactly-once boundary. this is the module's central promise and cannot be tested without a publishing tickerplant +true,0,0,q,6=updcalls,1,1,the replay drove upd once per logged message and no more +true,0,0,q,6=PH2"logged[]",1,1,the tickerplant has logged six messages so far +run,0,0,q,(neg PH2)(`pub;100;`S0),1,1,the tickerplant logs AND publishes one live message +run,0,0,q,flushpeer[PH2],1,1,round trip so the async live message is serviced +true,0,0,q,7=PH2"logged[]",1,1,the tickerplant logged the live message too +true,0,0,q,7=count peertrade,1,1,six replayed plus one live - the boundary message was NOT replayed as well +true,0,0,q,7=updcalls,1,1,upd ran exactly once per message across replay and live feed +true,0,0,q,100f=last exec price from peertrade,1,1,the live message is the most recent row +run,0,0,q,(neg PH2)(`pub;101;`S1),1,1,publish a second live message +run,0,0,q,flushpeer[PH2],1,1,service it +true,0,0,q,8=count peertrade,1,1,live updates keep flowing through the root upd after the replay + +comment,,,,,,,teardown releases the process-global registration init installed +run,0,0,q,sub.teardown[],1,1,tear down +true,0,0,q,not `subscriptions in exec name from realhandlers[`list][`.z.pc],1,1,the .z.pc registration was removed + +after,0,0,q,teardownfixture[],1,1,kill the peer and remove the temp directories diff --git a/di/subscriptions/test.q b/di/subscriptions/test.q new file mode 100644 index 00000000..9d093676 --- /dev/null +++ b/di/subscriptions/test.q @@ -0,0 +1,217 @@ +/ fixture helpers for di.subscriptions' tests (loaded by test.csv). +/ two layers: +/ unit - the tickerplant handle is a FUNCTION standing in for a handle (h(msg) applies to +/ either), answering the real subdetails protocol and pointing at a REAL tp log +/ built here, so every replay is genuine rather than simulated. +/ integration - a genuinely separate q process acting as a tickerplant: it writes its own tp log +/ and serves a root-level subdetails, and we subscribe over real IPC. that is only +/ possible because the wire contract is TorQ's real subdetails, not an invention. +/ NB each scenario uses its OWN table name. the double-subscribe guard deliberately refuses a table +/ that already has a live subscription, and a function handle always reports live, so reusing one +/ table across scenarios would trip the guard rather than test the behaviour under test. +/ this file is loaded at ROOT, so it is ordinary q - the module-context name-resolution rules that +/ apply inside subscriptions.q do not apply here. + +BASE:"/tmp/disubscriptionstest"; +D:2026.08.06; + +/ --- capturing logger (assert on what the module logged, per di.permissions' pattern) --- +caprows:([]lvl:`symbol$();ctx:`symbol$();msg:()); +resetcap:{[] `caprows set 0#caprows; }; +caplog:`info`warn`error!( + {[c;m] `caprows insert (`info;c;m)}; + {[c;m] `caprows insert (`warn;c;m)}; + {[c;m] `caprows insert (`error;c;m)}); +/ di.handlers is MERGED (kdbx-modules main, #114), so the tests wire the REAL module rather than a +/ mock's guess at the register contract - the .z.pc observer path is then genuinely exercised. +/ NB use is called at TOP LEVEL and indexed, never dot-accessed from inside a lambda (h.init inside +/ a function throws 'h.init; only h[`init] works there) +realhandlers:use`di.handlers; +handlerdep:{[] :`register`remove!(realhandlers[`register];realhandlers[`remove]); }; +/ NB one MULTI-key dict, not a chain of single-key ones: (enlist[`k]!enlist somedict) puts a TABLE +/ on the value side and joining two of them throws 'mismatch +deps:{[] :`log`handlers!(caplog;handlerdep[]); }; +/ did any message at this level contain this substring? ss, not like - like throws 'nyi on a +/ multi wildcard pattern. NB the parameter is lv, NOT lvl: naming it lvl would shadow the column +/ and (lvl=lvl) would compare the column to itself, matching every row +logged:{[lv;s] :any {[s;m] 0/dev/null 2>&1 &"; + if[not waitlisten[PEERPORT;5000];'"test: tickerplant peer failed to listen on ",string PEERPORT]; + h:hopen (`$":localhost:",string PEERPORT;2000); + `PEERPID set h ".z.i"; + hclose h; + }; + +peerhandle:{[] :hopen (`$":localhost:",string PEERPORT;2000); }; +noticedrop:{[h] + / a dead peer is only detected when we next touch the handle; that is what fires .z.pc + @[{x "1+1"};h;{[e] :(::)}]; + system "sleep 0.3"; + }; + +flushpeer:{[h] + / force this process to read the async messages the peer pushed at us. a synchronous round trip + / makes q service the incoming queue, so the live upd calls land before we assert on them + h "1+1"; + system "sleep 0.2"; + h "1+1"; + }; +killpeer:{[] if[not null PEERPID;@[system;"kill ",string PEERPID;{[e] :(::)}]]; `PEERPID set 0N; system "sleep 0.3"; }; + +setupfixture:{[] system "rm -rf ",BASE; system "mkdir -p ",BASE; }; +teardownfixture:{[] killpeer[]; system "rm -rf ",BASE; system "rm -rf ",PEERDIR; }; From 7fc3de918cd600e59f4f366b95cbce027450e566 Mon Sep 17 00:00:00 2001 From: alowrydi Date: Fri, 7 Aug 2026 16:03:53 +0100 Subject: [PATCH 02/10] Add unsubscribe[tph] and harden subdetails validation in di.subscriptions Releases a handle's subscriptions before close (the local-close case .z.pc misses); validates schema type, duplicate table names, logfilelist shape and sign, empty selectors, and setschema:0b with replay. Tests: 318, was 209. --- di/subscriptions/init.q | 2 +- di/subscriptions/subscriptions.md | 41 ++++++++-- di/subscriptions/subscriptions.q | 100 +++++++++++++++++++++++- di/subscriptions/test.csv | 125 ++++++++++++++++++++++++++++++ 4 files changed, 258 insertions(+), 10 deletions(-) diff --git a/di/subscriptions/init.q b/di/subscriptions/init.q index cc7397e9..4513495c 100644 --- a/di/subscriptions/init.q +++ b/di/subscriptions/init.q @@ -14,4 +14,4 @@ version:first read0`:::VERSION / NB: export:([...]) EVALUATES each name, so it can only list names that already exist. / init and getapimeta are framework plumbing di.torq calls by convention; every other name here has / a getapimeta row, which the test suite asserts -export:([init;teardown;version;getapimeta;subscribe;subscribed;getsubscriptions]) +export:([init;teardown;version;getapimeta;subscribe;unsubscribe;subscribed;getsubscriptions]) diff --git a/di/subscriptions/subscriptions.md b/di/subscriptions/subscriptions.md index e9fb40f2..e94138da 100644 --- a/di/subscriptions/subscriptions.md +++ b/di/subscriptions/subscriptions.md @@ -10,10 +10,10 @@ Tickerplant subscription management for kdb+ subscriber processes (RDB, WDB, cha - Subscribe to all tables and syms, or to any subset, with sym filtering applied to the log replay as well as to the live feed - Define subscribed tables at root from the schemas the tickerplant returns, preserving their attributes (e.g. `` `g# `` on `sym`) - Replay exactly the messages the tickerplant had logged at the instant of subscription, so messages that also arrive on the live feed are not applied twice -- Verify every log before defining a single table, so a truncated or unreadable log fails with the process untouched +- Verify every log before defining a single table, so a truncated or unreadable log fails with the process untouched (see Notes for the limit of that guarantee) - Handle every payload shape a tickerplant may log - a list of columns, a table, a dict, or a single atom row - by resolving the `sym` column by name rather than position - Replay across several log files, as a segmented tickerplant writes one log per table -- Track live subscriptions in a registry whose `active` flag is maintained from `.z.pc` and cross-checked against `.z.W` +- Track live subscriptions in a registry whose `active` flag is maintained from `.z.pc`, cross-checked against `.z.W`, and released explicitly by `unsubscribe` before the caller closes a handle - Refuse to re-subscribe a table that already has a live subscription, rather than silently redefining it and replaying into it again - Speaks TorQ's real `subdetails` protocol rather than a new one, and the remote entry point name is configurable @@ -46,7 +46,7 @@ The configuration key `subdetailsfunc` is optional - omit it and the module call | `` `handlers `` | yes | Handlers dep - at minimum `register` and `remove` | | `` `subdetailsfunc `` | no | Symbol naming the tickerplant-side function to call. Default: `` `subdetails `` | -`init` must be called before any operational function - `subscribe`, `subscribed`, `getsubscriptions` and `teardown` each throw a clear error if it has not been. `getapimeta` and `version` are metadata and deliberately work without it, so `di.torq` can collect api rows and `di.depcheck` can read the version before anything is initialised. `init` registers a `.z.pc` observer through `di.handlers` and is idempotent - a second call refreshes the dependencies and replaces the registration in place rather than duplicating it, and leaves the registry intact. +`init` must be called before any operational function - `subscribe`, `unsubscribe`, `subscribed`, `getsubscriptions` and `teardown` each throw a clear error if it has not been. `getapimeta` and `version` are metadata and deliberately work without it, so `di.torq` can collect api rows and `di.depcheck` can read the version before anything is initialised. `init` registers a `.z.pc` observer through `di.handlers` and is idempotent - a second call refreshes the dependencies and replaces the registration in place rather than duplicating it, and leaves the registry intact. Build `deps` as one multi-key dictionary. Do not join `di.log`'s `logdict` to another single-key dict: `` enlist[`k]!enlist somedict `` puts a table on the value side, and joining two of them throws `` 'mismatch `` at the call site before `init` runs. @@ -77,6 +77,18 @@ sub.subscribe[tph;`;`;0b;0b] / subscribe only - keep my s / `subtables`tplogdate`rowcounts`date!(`trade`quote;2025.06.01;`trade`quote!(1042;3311);2025.06.01) ``` +### `unsubscribe[tph]` +Release the subscriptions held on a tickerplant handle and return the tables released. **Call this before `hclose`.** It never closes the handle — the caller owns the connection — and never messages the tickerplant, because the `subdetails` protocol has no unsubscribe verb. + +It exists for the one liveness signal kdb+ cannot supply: a handle the *caller* closes fires no `.z.pc`, and the freed descriptor is then reissued to the next connection. Without this call, that stale registry row goes on reporting live and the duplicate guard refuses a legitimate re-subscribe. Idempotent — a second call logs at `warn` and returns an empty list rather than throwing, so a shutdown path is safe to run twice. + +A deliberate release **deletes** its registry rows rather than flagging them dead: the caller already knows it closed the handle, so the row carries nothing it does not have. A `.z.pc` drop is the opposite case and **keeps** its row, because an unexpected disconnect is worth being able to see afterwards. It selects on the *stored* flag, not the live/dead one, so calling it after `hclose` still removes the row — which is the case that matters most. +```q +sub.unsubscribe[tph] +/ `trade`quote +hclose tph; +``` + ### `subscribed[]` Return whether any subscription is currently live. The connectivity check a subscriber process needs before declaring itself ready. ```q @@ -99,6 +111,7 @@ Remove the `.z.pc` registration installed by `init`, leaving no process-global r ```q sub.teardown[] ``` +**Liveness degrades while torn down.** With the observer gone, `markdead` no longer fires, so `active` rests on the `.z.W` check alone and the handle-recycling protection is lost until `init` runs again. `init` is idempotent and reinstalls the observer, so the exposure is narrow — but do not treat a torn-down module as merely "quiet". ### `getapimeta[]` Return this module's api metadata, one row per callable export, for `di.torq` to register with `di.api`. @@ -144,6 +157,11 @@ r`tplogdate / the tickerplant log date, for setting the partition / check connectivity sub.subscribed[] sub.getsubscriptions[] + +/ on shutdown - declare the handle finished BEFORE closing it, so a reissued +/ descriptor cannot revive the registry row +sub.unsubscribe[tph] +hclose tph ``` --- @@ -157,9 +175,11 @@ k4unit.moduletest`di.subscriptions Run in a fresh q session - the integration layer spawns and kills a real q process, so do not interleave with other modules' tests. Needs `QHOME` set (the peer is launched via `$QHOME/bin/q`) and `di.os` on `QPATH` (the harness uses `os.abspath` to load `test.q`). -The suite is 209 assertions and wires the real merged `di.log` and `di.handlers` rather than mocks, so both injected contracts are proven end to end. It covers: pre-`init` guards on every export; dependency validation (non-dict deps, each missing key, each malformed value); `init` arity asserted by observable effect rather than by "it did not throw"; idempotent re-init; the `VERSION` file and exported `version`; `getapimeta`; all-tables and sym-filtered replay; all four payload shapes a tickerplant may log; symbol-atom table and sym selectors; a multi-table log where only the requested table is defined; replay across two log files; requested-but-absent and offered-but-unrequested tables; a configured `subdetailsfunc`; partial, unreadable, absent and empty logs; a replay requested with no root `upd`; malformed `schemalist` shapes; the double-subscribe guard; malformed tickerplant responses; `setschema:0b` preserving a populated table; `logdir` passthrough; and input validation on every argument. +The suite is 318 assertions (`true`, `fail` and `run` rows, excluding the 5 fixture `before`/`after` rows) and wires the real merged `di.log` and `di.handlers` rather than mocks, so both injected contracts are proven end to end. It covers: pre-`init` guards on every export; dependency validation (non-dict deps, each missing key, each malformed value); `init` arity asserted by observable effect rather than by "it did not throw"; idempotent re-init; the `VERSION` file and exported `version`; `getapimeta`; all-tables and sym-filtered replay; all four payload shapes a tickerplant may log; symbol-atom table and sym selectors; a multi-table log where only the requested table is defined; replay across two log files; requested-but-absent and offered-but-unrequested tables; a configured `subdetailsfunc`; partial, unreadable, absent and empty logs; a replay requested with no root `upd`; malformed `schemalist` **and** `logfilelist` shapes, including a non-table schema, a column-less table, a duplicated table name, and a keyed schema that must still be accepted; a negative message count, asserted with `replay:0b` as well so the guard cannot drift behind the replay branch; every `setschema`/`replay` combination, including `setschema:0b` with `replay:1b` on both the narrowed and all-syms paths; empty `tabs` and `syms` selectors; the double-subscribe guard and its release by `unsubscribe`; `unsubscribe` deleting rather than flagging, its idempotency, and its `warn` on an unheld handle; malformed tickerplant responses; `setschema:0b` preserving a populated table; `logdir` passthrough; and input validation on every argument. + +The suite also covers teardown/re-init lifecycle cycles, and asserts the `di.handlers` isolation contract: a co-registrant that throws at a priority ahead of this module's observer must not suppress `markdead`. That assertion is made through `unsubscribe`, which matches on the stored flag - checking `active` alone would be vacuous, since the closed handle reads dead through `.z.W` either way. -The integration block subscribes to a genuinely separate tickerplant process over IPC, kills it, confirms `.z.pc` marks the subscription dead, reconnects onto a recycled handle number, and confirms the stale row stays dead and the re-subscribe succeeds. +The integration block subscribes to a genuinely separate tickerplant process over IPC, kills it, confirms `.z.pc` marks the subscription dead, reconnects onto a recycled handle number, confirms the stale row stays dead and the re-subscribe succeeds, then drives live updates through the replayed tables to prove the exactly-once boundary. It finishes on the two cases only a real handle can reach: releasing a handle and `hclose`ing it while the tickerplant is still alive (so nothing fires `.z.pc`), reopening onto the same reissued descriptor and confirming no revival; and the reverse order — `hclose` *first*, then `unsubscribe` — which proves the release matches on the stored flag rather than the computed one, and so still finds a row `.z.W` has already given up on. --- @@ -167,13 +187,18 @@ The integration block subscribes to a genuinely separate tickerplant process ove - Replay uses kdb+'s native `-11!` directly rather than `di.tplog`. This module replays the first *n* messages; `di.tplog.check` is built for the replay-everything caller and would rewrite an entire log to a `.good` file when corruption lies beyond the messages actually needed, and cannot report the good-message count the preflight requires - Every log is verified with the non-executing `-11!(-2;logfile)` streaming count *before* any table is defined. This matters: `-11!(n;logfile)` past a corruption point replays the good messages and only then throws, leaving tables half populated +- **That guarantee covers log corruption, not every replay failure.** Preflight cannot vet the root `upd` itself. If `upd` throws part-way through a replay — bad data, or a bug in the caller's own handler — the tables are already defined and partially populated, and no registry row exists, because the row is appended after the replay completes. `subscribe` signals, but the process is *not* untouched. Do not read the preflight guarantee more broadly than it is stated +- The registry retains a row for every subscription that ended via `.z.pc` (tickerplant death) for the life of the process; only `unsubscribe` removes rows. This is deliberate for v1 rather than merely unfixed: the case that would make it matter — a flapping tickerplant generating dead rows continuously — **is not reachable without auto-reconnect**, which this module does not implement. Today a dead handle is only replaced by a human or an external supervisor calling `subscribe` again, which is inherently rate-limited. Bounding this belongs with the auto-reconnect design, where the reconnect cadence will be known rather than guessed at now. `activesubscriptions` recomputes liveness across the whole registry on every accessor call, so the bound on rows is also the bound on that cost - The subscribed set is what was requested intersected with what the tickerplant offered. A requested table the tickerplant does not return is logged at `warn` and skipped; a table it volunteers unasked is logged at `warn` and ignored - neither defined nor replayed -- Liveness needs both signals, which is why `handlers` is a required dependency. `.z.pc` fires when the tickerplant drops, at the instant of the drop and before kdb+ can reissue the handle number; `.z.W` catches a handle the caller closed itself, which does not fire `.z.pc`. `.z.W` alone is not sufficient - kdb+ reissues the lowest free descriptor, so a recycled number would revive a stale registry row +- Liveness needs three signals, and the measured behaviour of each is what dictates the design. On KDB-X 5f: a handle the process `hclose`s itself fires **no** `.z.pc` and leaves `.z.W`; a remote death fires `.z.pc` with the handle; and a closed descriptor **is** reissued to the next connection (`4` → `4`). So `.z.pc` covers tickerplant death, `.z.W` covers a local close *until the number is reused*, and `unsubscribe` covers the reuse case by clearing the stored flag outright. `.z.pc` is why `handlers` is a required dependency; `.z.W` alone is not sufficient, because a recycled number would revive a stale registry row - The double-subscribe guard runs after the tickerplant has been asked what it offers, so it compares resolved table lists. An all-tables subscribe is refused only where it genuinely overlaps a live subscription, not merely because some other subscription exists - Tables are created at root with `@[`.;name;:;schema]`, and `-11!` resolves `upd` at root even when invoked from module context. A root-level `upd` must therefore be defined before calling `subscribe` with `replay` set - this is **enforced**, not merely expected: without it every replayed message would be silently discarded and the narrowed path would leave its no-op stand-in bound at root, so the live feed would vanish into it too - A malformed `subdetails` response is rejected before anything is defined, including a `schemalist` that is not a list or whose entries are not `(tablename;schema)` pairs. An empty entry is tolerated, as legacy does -- `setschema:0b` skips table definition entirely, so a subscriber that pre-loads its own schema keeps it. With `setschema:1b` the returned schema replaces whatever is at root, which is why re-subscribing a live table is refused +- The *schema* half of each entry must actually be a table. This is not pedantry: `createtables` uses `@[`.;name;:;schema]`, which succeeds for **any** value, so without the check a tickerplant sending a dict or an atom would have it planted at root under the caller's table name and `subscribe` would report success over a root name that is not a table. The test is `.Q.qt` rather than `98h=type`, deliberately — a **keyed** table is `99h` and must still be accepted, while a column-less `([])` is also `99h` and must not be, since it has no columns to replay into +- A `schemalist` naming the same table more than once is rejected rather than deduplicated. The duplicate would otherwise flow straight into `subtables` — which `di.rdb` and `di.wdb` iterate over — and into the registry's `tabs`. Deduplicating would hide a tickerplant bug; every other malformed response here fails loud, so this does too +- `setschema:0b` skips table definition entirely, so a subscriber that pre-loads its own schema keeps it. With `setschema:1b` the returned schema replaces whatever is at root, which is why re-subscribing a live table is refused. Combining `setschema:0b` with `replay:1b` requires the tables to exist at root already, and that is **checked** — otherwise the replay fails inside the caller's own `upd` on the all-syms path, or from `cols get t` on the narrowed one, neither of which reaches the log +- A malformed `logfilelist` is rejected on the same terms as `schemalist` — a non-list, an entry that is not a `(messagecount;logfile)` pair, a non-integer count or a non-symbol log file. A **negative** count is rejected too, which no shape check catches, since `-1` is a perfectly good integer; that guard lives in the response validation rather than the replay preflight so it also fires when `replay` is `0b` - `subdetails` is TorQ's real protocol, defined at root by `chainedtp.q` and `segmentedtickerplant.q`, returning `` `schemalist`logfilelist`rowcounts`date `` and optionally `` `logdir ``. The key names and shapes here were taken from that source and are exercised against a tickerplant process built to the same protocol in the integration tests; the module has **not** yet been run against a live TorQ chained or segmented tickerplant. `di.tickerplant` is not built yet - when it lands, point `subdetailsfunc` at its entry point if the name differs - Not in v1: auto-reconnect and resubscribe on tickerplant bounce, filtered-column subscriptions, and remote-log streaming (the subscriber is assumed to share the tickerplant's filesystem, the classic tick assumption) - The modularisation plan lists `di.servers` and `di.pubsub` as dependencies of this module. Neither is used: handle resolution is the caller's job, and `di.pubsub` is the publisher side - the tickerplant's own subscriber registry - which a subscribing process never calls -- One residual liveness gap: a handle the caller closes itself, without the tickerplant dying, whose number is then reissued to an unrelated connection, can leave a stale row appearing live. Tickerplant death and local close without reuse are both exact +- `unsubscribe` is the supported way to close the local-close liveness gap, but it is a **cooperative** mechanism, not a structural one: kdb+ exposes no way to detect an *unannounced* `hclose`, and no way to tell whether a reissued descriptor is still the same remote. A caller that closes a tickerplant handle without calling `unsubscribe` first, and whose descriptor is then reissued, can still leave a stale row appearing live. Tickerplant death, and any local close that goes through `unsubscribe`, are both exact diff --git a/di/subscriptions/subscriptions.q b/di/subscriptions/subscriptions.q index 6271ca55..0814a090 100644 --- a/di/subscriptions/subscriptions.q +++ b/di/subscriptions/subscriptions.q @@ -55,6 +55,14 @@ requiretabspec:{[ctx;nm;x] / a table/sym selector is either ` (all) or one or more symbols if[not 11h=abs type x; raiseerror[ctx;nm," must be ` (all) or a symbol vector, got type ",.Q.s1 type x]]; + / an EMPTY symbol vector is rejected rather than quietly treated as a filter. as tabs it would + / otherwise surface as "tickerplant returned no schema", blaming the tickerplant for the caller's + / own input; as syms it would SUCCEED SILENTLY - narrowed path, zero rows replayed, a defined but + / empty table and no warning at any level. every other rejection here goes through raiseerror and + / every mismatch gets a warn, so a silent no-op would be the module contradicting its own standard. + / NB 11h is the VECTOR case - the ` all-sentinel is -11h and has count 1, so it is unaffected + if[(11h=type x) and 0=count x; + raiseerror[ctx;nm," is an empty symbol vector - pass ` for all, or name at least one"]]; }; normspec:{[x] @@ -182,6 +190,44 @@ fetchdetails:{[tph;tabs;syms] raiseerror[`subscribe;(string .z.m.subdetailsfunc)," schemalist entries must be (tablename;schema) pairs"]]; if[not all -11h=type each entries[;0]; raiseerror[`subscribe;(string .z.m.subdetailsfunc)," schemalist entries must name their table with a symbol"]]; + / the SCHEMA half must actually be a table. without this a tickerplant that sends a dict or an atom + / gets it planted at root under the caller's table name by createtables' @[`.;name;:;schema] - which + / succeeds for any value - and subscribe then reports success over a root name that is not a table. + / .Q.qt, not 98h=type: a KEYED table is 99h and must still be accepted, while a column-less ([]) is + / also 99h and must not be (its cols are empty, so replaying into it is meaningless) + if[not all .Q.qt each entries[;1]; + raiseerror[`subscribe;(string .z.m.subdetailsfunc)," schemalist entries must give a table as the schema"]]; + / a duplicate table name would be carried straight through to subtables, which di.rdb and di.wdb + / iterate over, and into the registry's tabs column. reject rather than silently dedupe - every + / other malformed response here fails loud, and deduping would hide the tickerplant's own bug + / NB `where` over the dict `count each group nms` yields the duplicated NAMES directly - indexing + / nms by it instead would index by symbol and throw a bare 'type that bypasses the log + nms:entries[;0]; + if[(count distinct nms)<>count nms; + raiseerror[`subscribe;(string .z.m.subdetailsfunc)," schemalist names a table more than once: ", + ", " sv string where 1type lfl; + raiseerror[`subscribe;(string .z.m.subdetailsfunc)," logfilelist must be a list of (messagecount;logfile) pairs"]]; + lfe:lfl where not 0=count each lfl; + if[not all 2=count each lfe; + raiseerror[`subscribe;(string .z.m.subdetailsfunc)," logfilelist entries must be (messagecount;logfile) pairs"]]; + if[not all (type each lfe[;0]) in -7 -6h; + raiseerror[`subscribe;(string .z.m.subdetailsfunc)," logfilelist entries must give the message count as an integer"]]; + if[not all -11h=type each lfe[;1]; + raiseerror[`subscribe;(string .z.m.subdetailsfunc)," logfilelist entries must name the log file with a symbol"]]; + / a NEGATIVE count is a malformed response, not "nothing to replay" - a shape check cannot catch it + / because -1 is a perfectly good integer. rejected HERE rather than in preflightone so it is caught + / even when replay is 0b; preflight only ever runs on the replay path + / NB not `neg` - that is a q reserved word and a bare assignment to it throws 'assign at PARSE + / time, taking the whole module down at load + badcount:lfe where 0>lfe[;0]; + if[0=nmsg; :(::)]; + / 0=, not 0>= - fetchdetails has already rejected a negative count as a malformed response, so a + / negative can no longer reach here and must not be quietly folded into "nothing to replay" + if[0=nmsg; :(::)]; good:goodcount lf; if[good Date: Wed, 12 Aug 2026 15:07:52 +0100 Subject: [PATCH 03/10] Fix TP protocol gaps, TP reconnection and TP-side unsubscription --- di/subscriptions/init.q | 2 +- di/subscriptions/subscriptions.md | 64 ++++- di/subscriptions/subscriptions.q | 441 +++++++++++++++++++++++++++--- di/subscriptions/test.csv | 232 +++++++++++++++- di/subscriptions/test.q | 91 +++++- 5 files changed, 766 insertions(+), 64 deletions(-) diff --git a/di/subscriptions/init.q b/di/subscriptions/init.q index 4513495c..ed61a242 100644 --- a/di/subscriptions/init.q +++ b/di/subscriptions/init.q @@ -14,4 +14,4 @@ version:first read0`:::VERSION / NB: export:([...]) EVALUATES each name, so it can only list names that already exist. / init and getapimeta are framework plumbing di.torq calls by convention; every other name here has / a getapimeta row, which the test suite asserts -export:([init;teardown;version;getapimeta;subscribe;unsubscribe;subscribed;getsubscriptions]) +export:([init;teardown;version;getapimeta;subscribe;resubscribe;unsubscribe;subscribed;getsubscriptions]) diff --git a/di/subscriptions/subscriptions.md b/di/subscriptions/subscriptions.md index e94138da..f3e82a9c 100644 --- a/di/subscriptions/subscriptions.md +++ b/di/subscriptions/subscriptions.md @@ -9,12 +9,14 @@ Tickerplant subscription management for kdb+ subscriber processes (RDB, WDB, cha - Subscribe over an already-open tickerplant handle - the caller owns the connection, so this module never opens, retries or closes one - Subscribe to all tables and syms, or to any subset, with sym filtering applied to the log replay as well as to the live feed - Define subscribed tables at root from the schemas the tickerplant returns, preserving their attributes (e.g. `` `g# `` on `sym`) -- Replay exactly the messages the tickerplant had logged at the instant of subscription, so messages that also arrive on the live feed are not applied twice +- Replay exactly the messages the tickerplant had logged at the instant of subscription, so messages that also arrive on the live feed are not applied twice - exact in every case except a *shared* log, where it is best-effort with a bounded duplicate window (see Notes) - Verify every log before defining a single table, so a truncated or unreadable log fails with the process untouched (see Notes for the limit of that guarantee) - Handle every payload shape a tickerplant may log - a list of columns, a table, a dict, or a single atom row - by resolving the `sym` column by name rather than position -- Replay across several log files, as a segmented tickerplant writes one log per table +- Replay across several log files, as a segmented tickerplant writes one log per table - and across a *shared* log, which the same tickerplant writes in its `singular` and `periodic` modes, driving only the subscribed tables even when the shared file carries others (see Notes) +- Resolve `` ` `` (all tables) to a concrete list with a `tablelist` round trip before calling `subdetails`, because a segmented tickerplant cannot accept the sentinel - Track live subscriptions in a registry whose `active` flag is maintained from `.z.pc`, cross-checked against `.z.W`, and released explicitly by `unsubscribe` before the caller closes a handle - Refuse to re-subscribe a table that already has a live subscription, rather than silently redefining it and replaying into it again +- Run every guard that does not need the tickerplant's reply *before* asking for it, because asking is itself a subscription (see Notes) - Speaks TorQ's real `subdetails` protocol rather than a new one, and the remote entry point name is configurable --- @@ -32,7 +34,7 @@ Both deps must be passed to `init` inside the `deps` dict. The module throws imm Handle resolution is the **caller's** job. `di.rdb` and `di.wdb` obtain a tickerplant handle from `di.servers.gethandlebytype` and pass it in, so there is no `di.servers` dependency here. -The configuration key `subdetailsfunc` is optional - omit it and the module calls the tickerplant's `subdetails`. See Initialisation. +The configuration keys `subdetailsfunc` and `tablelistfunc` are optional - omit them and the module calls the tickerplant's `subdetails` and `tablelist`. See Initialisation. --- @@ -45,6 +47,8 @@ The configuration key `subdetailsfunc` is optional - omit it and the module call | `` `log `` | yes | Log dep - `info`, `warn` and `error`, each `{[c;m]}` | | `` `handlers `` | yes | Handlers dep - at minimum `register` and `remove` | | `` `subdetailsfunc `` | no | Symbol naming the tickerplant-side function to call. Default: `` `subdetails `` | +| `` `tablelistfunc `` | no | Symbol naming the tickerplant-side function that lists the available tables, used to resolve a `` ` `` request into a concrete list. Default: `` `tablelist `` | +| `` `unsubscribefunc `` | no | Symbol naming a tickerplant-side function that releases this connection's subscriptions, called by `unsubscribe`. Default: `` ` `` (none) — see Notes | `init` must be called before any operational function - `subscribe`, `unsubscribe`, `subscribed`, `getsubscriptions` and `teardown` each throw a clear error if it has not been. `getapimeta` and `version` are metadata and deliberately work without it, so `di.torq` can collect api rows and `di.depcheck` can read the version before anything is initialised. `init` registers a `.z.pc` observer through `di.handlers` and is idempotent - a second call refreshes the dependencies and replaces the registration in place rather than duplicating it, and leaves the registry intact. @@ -67,14 +71,20 @@ sub.init[`log`handlers!(logging.logdict`log;`register`remove!(handlers.register; ``` ### `subscribe[tph;tabs;syms;setschema;replay]` -Subscribe over an already-open tickerplant handle. `tph` is an int handle (or a function standing in for one). `tabs` and `syms` are `` ` `` for all, or one or more symbols - a bare symbol atom names a single table or sym and is normalised to a list. `setschema` defines the returned schemas at root; `replay` replays the pre-subscription log and requires a root-level `upd`. +Subscribe over an already-open tickerplant handle. `tph` is an int handle (or a function standing in for one). `tabs` and `syms` are `` ` `` for all, or one or more symbols - a bare symbol atom names a single table or sym and is normalised to a list. `syms` may also be a **filter dictionary** keyed by table (TorQ's own shape - `rdb.q` loads it from a csv as `.sub.filterparams` and passes it through as `instruments`); it is handed to the tickerplant for the live feed, and the replay runs unfiltered, exactly as legacy's `replayupd` does. `setschema` defines the returned schemas at root; `replay` replays the pre-subscription log and requires a root-level `upd`. If it throws, check whether the error mentions the handle having already been registered — see Notes for the one case where that can happen and what to do about it. -Returns a dictionary of `subtables`, `tplogdate`, `rowcounts` and `date`, plus `logdir` when the tickerplant supplies one. +Returns a dictionary of `subtables`, `tplogdate`, `rowcounts` and `date`, plus `logdir` when the tickerplant supplies one. It also carries `icounts` and `d`, which are legacy's names for `rowcounts` and `date` and hold the same values — see Notes for why both are emitted. ```q sub.subscribe[tph;`;`;1b;1b] / all tables, all syms, define schemas, replay sub.subscribe[tph;`trade`quote;enlist`AAPL;1b;1b] / two tables, one sym sub.subscribe[tph;`;`;0b;0b] / subscribe only - keep my schemas, skip the replay -/ `subtables`tplogdate`rowcounts`date!(`trade`quote;2025.06.01;`trade`quote!(1042;3311);2025.06.01) +/ `subtables`tplogdate`rowcounts`date`icounts`d!(`trade`quote;2025.06.01;`trade`quote!(1042;3311);2025.06.01;`trade`quote!(1042;3311);2025.06.01) +``` + +### `resubscribe[tph]` +Re-establish every subscription that has since dropped, over a **new** handle to the same tickerplant. This is legacy's `retrysubscription` (`subscriptions.q:155`), split so the module keeps the knowledge of what was subscribed while the caller keeps ownership of the connection — `di.rdb`/`di.servers` decide *when* to call it. Uses `setschema:0b` and `replay:0b` exactly as legacy does, so a reconnect restores the live feed without re-applying history. Best-effort per subscription and never fatal; returns the tables re-established. +```q +sub.resubscribe[newtph] / -> `trade`quote ``` ### `unsubscribe[tph]` @@ -107,7 +117,7 @@ sub.getsubscriptions[] `tabs` and `syms` are general columns, so symbols display without backticks and an all-syms subscription shows an empty `syms`. ### `teardown[]` -Remove the `.z.pc` registration installed by `init`, leaving no process-global residue. +Remove the `.z.pc` registration installed by `init`, leaving no process-global residue. Afterwards `subscribe` is refused until `init` is called again — see Notes. `getsubscriptions`, `subscribed` and `unsubscribe` keep working, so a shutdown path can still inspect and release what it holds, and the registry is left intact. ```q sub.teardown[] ``` @@ -141,7 +151,9 @@ handlerdep:`register`remove!(handlers.register;handlers.remove) sub:use`di.subscriptions sub.init[`log`handlers!(logging.logdict`log;handlerdep)] -/ define the root upd the replay will drive, exactly as the subscriber process does +/ define the root upd the replay will drive, exactly as the subscriber process does. +/ NB it must handle BOTH payload shapes - see Notes. simplifying this to the flip alone throws +/ part way through a sym-filtered replay upd:{[t;x] @[`.;t;{[tab;d] tab upsert $[98h=type d;d;flip (cols tab)!d]}[;x]];} / the caller resolves the tickerplant handle. NB di.servers is not merged yet (feature-server), so @@ -175,7 +187,15 @@ k4unit.moduletest`di.subscriptions Run in a fresh q session - the integration layer spawns and kills a real q process, so do not interleave with other modules' tests. Needs `QHOME` set (the peer is launched via `$QHOME/bin/q`) and `di.os` on `QPATH` (the harness uses `os.abspath` to load `test.q`). -The suite is 318 assertions (`true`, `fail` and `run` rows, excluding the 5 fixture `before`/`after` rows) and wires the real merged `di.log` and `di.handlers` rather than mocks, so both injected contracts are proven end to end. It covers: pre-`init` guards on every export; dependency validation (non-dict deps, each missing key, each malformed value); `init` arity asserted by observable effect rather than by "it did not throw"; idempotent re-init; the `VERSION` file and exported `version`; `getapimeta`; all-tables and sym-filtered replay; all four payload shapes a tickerplant may log; symbol-atom table and sym selectors; a multi-table log where only the requested table is defined; replay across two log files; requested-but-absent and offered-but-unrequested tables; a configured `subdetailsfunc`; partial, unreadable, absent and empty logs; a replay requested with no root `upd`; malformed `schemalist` **and** `logfilelist` shapes, including a non-table schema, a column-less table, a duplicated table name, and a keyed schema that must still be accepted; a negative message count, asserted with `replay:0b` as well so the guard cannot drift behind the replay branch; every `setschema`/`replay` combination, including `setschema:0b` with `replay:1b` on both the narrowed and all-syms paths; empty `tabs` and `syms` selectors; the double-subscribe guard and its release by `unsubscribe`; `unsubscribe` deleting rather than flagging, its idempotency, and its `warn` on an unheld handle; malformed tickerplant responses; `setschema:0b` preserving a populated table; `logdir` passthrough; and input validation on every argument. +The suite is 523 assertions (`true`, `fail` and `run` rows, excluding the 5 fixture `before`/`after` rows) and wires the real merged `di.log` and `di.handlers` rather than mocks, so both injected contracts are proven end to end. It covers: pre-`init` guards on every export; dependency validation (non-dict deps, each missing key, each malformed value); `init` arity asserted by observable effect rather than by "it did not throw"; idempotent re-init; the `VERSION` file and exported `version`; `getapimeta`; all-tables and sym-filtered replay; all four payload shapes a tickerplant may log; symbol-atom table and sym selectors; a multi-table log where only the requested table is defined; replay across two log files; requested-but-absent and offered-but-unrequested tables; a configured `subdetailsfunc`; partial, unreadable, absent and empty logs; a replay requested with no root `upd`; malformed `schemalist` **and** `logfilelist` shapes, including a non-table schema, a column-less table, a duplicated table name, and a keyed schema that must still be accepted; a negative message count, asserted with `replay:0b` as well so the guard cannot drift behind the replay branch; every `setschema`/`replay` combination, including `setschema:0b` with `replay:1b` on both the narrowed and all-syms paths; empty `tabs` and `syms` selectors; the double-subscribe guard and its release by `unsubscribe`; `unsubscribe` deleting rather than flagging, its idempotency, and its `warn` on an unheld handle; malformed tickerplant responses; `setschema:0b` preserving a populated table; `logdir` passthrough; and input validation on every argument. + +It also covers the TorQ-protocol cases this module is built to survive: a `0W` message count replaying a whole log, and the same count over a **truncated** log being refused with nothing replayed and no table defined; a **shared** log reported once per table being collapsed to one full replay, with the per-table row counts asserted individually so a regression to the old per-entry replay is caught rather than merely the total; an unshared file in the same response keeping its own count and raising no warn; a corrupt shared log refused, proving the corruption guard covers the collapse trigger and not only the `0W` sentinel; an exact duplicate entry rejected; `` ` `` resolved through `tablelist` with the *resolved* list asserted to be what `subdetails` actually received; the fallback to `` ` `` when a tickerplant offers no `tablelist`; a configured `tablelistfunc` asserted to be called **first**, ahead of `subdetails`; and the two legitimate `rowcounts` shapes accepted with atoms and tables rejected. + +One fixture is modelled on the **real** producers rather than on convenience: `subdetails` as `.ps.subscribe` each-left, answering an unpublished name with the error pair instead of omitting it. That distinction matters more than it looks — the forgiving mocks omit, real tickerplants error, and a suite built only on the forgiving shape went green over a subscribe that a single mistyped table name would have taken down entirely. Rows assert that one bad name no longer sinks the call, that the drop is reported with the real reason, that a wholly unpublishable request is refused before `subdetails` runs, and — as an explicit regression guard — that the schema-shape error no longer fires on this path. + +The whole-file narrowing is asserted on a shared log that carries a table the tickerplant does not offer: the offered tables replay in full, the unoffered one is never driven into `upd` and so is never created at root, and the `upd` call count distinguishes the two. The same is asserted for the `0W` sentinel, and a per-table log is asserted to still take the unfiltered path. + +Guard **ordering** is asserted directly rather than inferred, using a fixture that records every remote function name `subscribe` asks for. Each guard that was moved ahead of the `subdetails` call has a row proving the tickerplant was never asked at all when it fires — including the case whose semantics changed, a requested table that is already held *and* no longer offered, which now throws where it previously warned and continued. The residual cases have rows too: an all-tables subscribe to a tickerplant with no `tablelist` reaches the post-reply copy of the duplicate guard, and a short log is caught only after `subdetails` has run, with assertions on both halves of the error text that tells the caller to close the handle. The suite also covers teardown/re-init lifecycle cycles, and asserts the `di.handlers` isolation contract: a co-registrant that throws at a priority ahead of this module's observer must not suppress `markdead`. That assertion is made through `unsubscribe`, which matches on the stored flag - checking `active` alone would be vacuous, since the closed handle reads dead through `.z.W` either way. @@ -190,15 +210,31 @@ The integration block subscribes to a genuinely separate tickerplant process ove - **That guarantee covers log corruption, not every replay failure.** Preflight cannot vet the root `upd` itself. If `upd` throws part-way through a replay — bad data, or a bug in the caller's own handler — the tables are already defined and partially populated, and no registry row exists, because the row is appended after the replay completes. `subscribe` signals, but the process is *not* untouched. Do not read the preflight guarantee more broadly than it is stated - The registry retains a row for every subscription that ended via `.z.pc` (tickerplant death) for the life of the process; only `unsubscribe` removes rows. This is deliberate for v1 rather than merely unfixed: the case that would make it matter — a flapping tickerplant generating dead rows continuously — **is not reachable without auto-reconnect**, which this module does not implement. Today a dead handle is only replaced by a human or an external supervisor calling `subscribe` again, which is inherently rate-limited. Bounding this belongs with the auto-reconnect design, where the reconnect cadence will be known rather than guessed at now. `activesubscriptions` recomputes liveness across the whole registry on every accessor call, so the bound on rows is also the bound on that cost - The subscribed set is what was requested intersected with what the tickerplant offered. A requested table the tickerplant does not return is logged at `warn` and skipped; a table it volunteers unasked is logged at `warn` and ignored - neither defined nor replayed +- **`unsubscribe` releases at the tickerplant when a release verb is configured.** Set `unsubscribefunc` at `init` to the name of a tickerplant-side function that drops this connection's registrations, and `unsubscribe` calls it — the release is then real, not local bookkeeping. This matters because shipped TorQ has a gap: `suball` and `subfiltered` each clear only their **own** registry (`pubsub.q:34,41`), so moving from an all-syms to a filtered subscription on one connection leaves the all-syms entry behind and the wider feed keeps arriving. Measured against a live segmented tickerplant, and measured again with `unsubscribefunc` wired: the same sequence then delivers only the narrowed syms, with no all-syms entry left at the tickerplant. The verb needs `.z.w` — the caller's handle as the tickerplant sees it — so only the tickerplant can offer it; `di.tickerplant` should expose one, e.g. `` {[tabs] .stpps.delhandle[;.z.w] each (),tabs; .stpps.delhandlef[;.z.w] each (),tabs} ``. Left unset, `unsubscribe` drops local rows only and says so at `warn`, naming the reason +- **The return dictionary carries both this module's key names and legacy's, because a real consumer reads legacy's and reads them silently.** `rdb.q:171` takes `subtables` and `tplogdate`, and `wdb.q:546` takes `tplogdate` — those names match. But `chainedtp.q:81-84` and `sctp.q:22-25` both read `` `d `` and `` `icounts ``, *through* `` if[`d in key r] `` / `` if[`icounts in key r] `` guards. A missing key there does not fail: the chained tickerplant simply never sets its own `.u.d` and never seeds `.u.icounts`/`.u.jcounts`, and every downstream subscriber of *that* process then gets wrong counts, with nothing logged anywhere. So `icounts` and `d` are emitted alongside `rowcounts` and `date`, holding the same values, and the suite runs `chainedtp.q`'s own seeding expression verbatim against the return as a compatibility test. `rowcounts`/`date` remain canonical — they are the tickerplant's own key names, carried through from the `subdetails` reply unchanged, where `subtables`/`tplogdate` are names legacy invented and so keep legacy's spelling. Legacy's `i` is deliberately **not** emitted: no shipped consumer reads it, and legacy gives it two different meanings by tickerplant type (the whole `logfilelist` for segmented, the message count for standard and chained) that this module has no `tptype` to disambiguate between +- **A requested table the tickerplant does not publish is dropped from the request before it is sent, not passed through.** This is not tidiness. `subdetails` is `.ps.subscribe` **each-left** over the table list, and every shipped producer fails on a name it does not publish: the standard one *signals* `'x` (`u.q`, `sub`), while segmented and chained answer with the pair `` (name;"Table … not in list of stp pub/sub tables") `` (`pubsub.q`), which the schema guard then rejects. Either way **one** bad name sinks the **whole** call — and because the each-left runs left to right, every valid table ahead of it has already been registered by `suball` before the failure, leaving a partial subscription live at the tickerplant that nothing on this side records. A typo, a stale process config, or a table retired at the tickerplant would otherwise take down an entire subscription. The request is therefore narrowed against the `tablelist` round trip first, exactly as legacy's `reducesubs` does (`subtabs:subtabs inter utabs`), and each dropped table is logged at `warn` with the real reason. A request the tickerplant publishes *none* of is refused outright, before `subdetails` is called and so before anything is registered +- **One `tablelist` round trip serves both jobs**, and it runs before every guard because it is *pure* — `tablelist` is `{.stpps.t}` on both shipped producers and registers nothing, unlike `subdetails`. When the tickerplant cannot answer it, `` ` `` is sent onward and an explicit request goes unnarrowed, exactly as before; that is reported at `warn` only when the `` ` `` sentinel genuinely had to be resolved, and at `info` on the explicit path where the round trip is a safety net rather than a requirement - Liveness needs three signals, and the measured behaviour of each is what dictates the design. On KDB-X 5f: a handle the process `hclose`s itself fires **no** `.z.pc` and leaves `.z.W`; a remote death fires `.z.pc` with the handle; and a closed descriptor **is** reissued to the next connection (`4` → `4`). So `.z.pc` covers tickerplant death, `.z.W` covers a local close *until the number is reused*, and `unsubscribe` covers the reuse case by clearing the stored flag outright. `.z.pc` is why `handlers` is a required dependency; `.z.W` alone is not sufficient, because a recycled number would revive a stale registry row -- The double-subscribe guard runs after the tickerplant has been asked what it offers, so it compares resolved table lists. An all-tables subscribe is refused only where it genuinely overlaps a live subscription, not merely because some other subscription exists +- The double-subscribe guard compares the **requested** table list — the caller's list, or the concrete list `` ` `` was resolved to — against what is already held, and it runs *before* the tickerplant is asked for anything. Re-subscribing to a table you already hold is a caller mistake whatever the tickerplant happens to offer that round; comparing against the *offered* set instead let the same mistake through with only a `warn` whenever the tickerplant had also stopped offering the table, which was an accident of ordering rather than a considered exception. The guard runs a second time against the offered set, for the one request that cannot be resolved up front (see below). An all-tables subscribe is still refused only where it genuinely overlaps a live subscription, not merely because some other subscription exists +- **Asking a tickerplant for its schemas subscribes you to it.** `subdetails` calls `.ps.subscribe`, which *is* `.u.sub`, so the one call that returns the schemas also registers the handle for live delivery — and the protocol has no unsubscribe verb to undo it. `unsubscribe` here is local bookkeeping only; tickerplant-side release is driven by `.z.pc`, which nothing but closing the handle triggers. Every guard that does not need the reply therefore runs **ahead** of that call: the duplicate guard, the root-`upd` check, and the tables-exist check for `setschema:0b`. That is legacy's own ordering — `reducesubs` runs against a `tablelist` round trip before `subfunc` — restored rather than invented, and it is what makes the `tablelist` round trip load-bearing for a second reason beyond the `` ` `` sentinel +- **What is left of that window, precisely.** With `replay:0b` it is closed for every subscribe that names its tables, and for an all-tables subscribe whenever the tickerplant answers `tablelistfunc` — which both shipped producers do. It stays open only for an all-tables subscribe against a tickerplant offering no usable `tablelist`, where the request cannot be resolved before the call. With `replay:1b` one case is **irreducible**: the log preflight. The log file names exist only in the reply and their integrity can only be established by reading them, so a log that is short, corrupt or unreadable is necessarily discovered *after* the handle is registered. `subscribe` then throws and records no registry row, while the tickerplant goes on publishing into the caller's root `upd` — so the module's own view and reality diverge. The error raised in that case says so explicitly and names the only remedy the protocol leaves: close the handle before retrying. No extra registry state is kept for it; the case is narrow and a third liveness state would cost every reader of `getsubscriptions[]` more than it would buy - Tables are created at root with `@[`.;name;:;schema]`, and `-11!` resolves `upd` at root even when invoked from module context. A root-level `upd` must therefore be defined before calling `subscribe` with `replay` set - this is **enforced**, not merely expected: without it every replayed message would be silently discarded and the narrowed path would leave its no-op stand-in bound at root, so the live feed would vanish into it too - A malformed `subdetails` response is rejected before anything is defined, including a `schemalist` that is not a list or whose entries are not `(tablename;schema)` pairs. An empty entry is tolerated, as legacy does -- The *schema* half of each entry must actually be a table. This is not pedantry: `createtables` uses `@[`.;name;:;schema]`, which succeeds for **any** value, so without the check a tickerplant sending a dict or an atom would have it planted at root under the caller's table name and `subscribe` would report success over a root name that is not a table. The test is `.Q.qt` rather than `98h=type`, deliberately — a **keyed** table is `99h` and must still be accepted, while a column-less `([])` is also `99h` and must not be, since it has no columns to replay into +- The *schema* half of each entry must actually be a table. This is not pedantry: `createtables` uses `@[`.;name;:;schema]`, which succeeds for **any** value, so without the check a tickerplant sending a dict or an atom would have it planted at root under the caller's table name and `subscribe` would report success over a root name that is not a table. The test is `.Q.qt` rather than `98h=type`, deliberately — a **keyed** table is `99h` and must still be accepted, while a column-less `([])` is also `99h` and must not be, since it has no columns to replay into. The guard also catches a shape that is not a garbled schema at all: TorQ's `.u.sub` answers a table it does not publish with `` (name;"Table … not in list of stp pub/sub tables") ``, so a **string** in the schema position is the tickerplant declining the table, and the error message says so - A `schemalist` naming the same table more than once is rejected rather than deduplicated. The duplicate would otherwise flow straight into `subtables` — which `di.rdb` and `di.wdb` iterate over — and into the registry's `tabs`. Deduplicating would hide a tickerplant bug; every other malformed response here fails loud, so this does too - `setschema:0b` skips table definition entirely, so a subscriber that pre-loads its own schema keeps it. With `setschema:1b` the returned schema replaces whatever is at root, which is why re-subscribing a live table is refused. Combining `setschema:0b` with `replay:1b` requires the tables to exist at root already, and that is **checked** — otherwise the replay fails inside the caller's own `upd` on the all-syms path, or from `cols get t` on the narrowed one, neither of which reaches the log +- **After `teardown`, a new `subscribe` is refused — reading and releasing still work.** `teardown` removes the `.z.pc` observer, and without it a dropped tickerplant's registry row keeps reporting live for as long as `.z.W` still holds the handle number, and *indefinitely* once kdb+ reissues that number to another connection. That is exactly the failure the `handlers` dependency exists to prevent, so taking a **new** subscription in that state now throws and names the recovery (`call init again`) rather than degrading silently into the `.z.W`-only mode this module documents as insufficient. `getsubscriptions`, `subscribed` and `unsubscribe` deliberately keep working, because a shutdown path needs to inspect and release what it already holds. `init` is idempotent and restores the observer. This reverses an earlier deliberate choice — the previous behaviour let the subscribe through and accepted degraded liveness tracking — because silent degradation is inconsistent with how every other guard in this module behaves +- **The root `upd` must accept two payload shapes, and which one it gets depends on the subscription.** An unfiltered replay hands `upd` the logged payload exactly as the tickerplant wrote it — normally a list of columns. A **sym-filtered** replay hands it a **table**, because `replayfilter` has to materialise one to filter by column name rather than by position. Legacy's `replayupd` behaves the same way, so this is inherited rather than introduced, but it is a real contract: an `upd` written only for the column-list shape throws part-way through a filtered replay, after earlier messages have already been applied. Write it as `` {[t;x] @[`.;t;{[tab;d] tab upsert $[98h=type d;d;flip (cols tab)!d]}[;x]]} ``, which is what the usage example and the test fixtures use. The live feed is unaffected — it never passes through `replayfilter` +- **Live-feed filtering is the tickerplant's job, not this module's.** `replayfilter` is installed only for the duration of a narrowed replay and torn down the moment it ends; live messages after that reach the caller's `upd` unfiltered by anything here. That is correct, not a leak: the `subdetails` call itself registers the handle for tickerplant-side filtered delivery as a *side effect*. In shipped TorQ, `subdetails` calls `.ps.subscribe`, which **is** `.u.sub`, which routes through `suball`/`subfiltered` to `selfiltered` - and that records `(tbl;handle;filts;columns)` with `filts` built as `` (in;`sym;syms) ``, which `pub` then evaluates per subscriber. Replay needs its own filter only because it bypasses the publish layer entirely and reads the log file directly. Without this note a reader seeing `replayfilter` removed right after the replay would reasonably read it as a bug +- **That delegation is verified end to end.** Measured against a live TorQ segmented tickerplant: a sym-filtered subscribe on a fresh connection registers `` filts: ,,,(in;`sym;,,`S1) `` in the tickerplant's `.stpps.subrequestfiltered`, leaves `subrequestall` empty for that table, and delivers only the matching rows live — two of three published. So the filtering really is the tickerplant's, not an assumption about it +- **The `0W` message count means "replay everything", not "more than I have".** A segmented tickerplant reports `0Wj` for every **closed** log under `replayperiod` `` `day ``. It is resolved to the log's own readable total rather than forwarded to `-11!`, because `-11!(0W;log)` over a corrupt log replays the good prefix and only *then* throws - precisely the half-populated state the preflight exists to prevent. A corrupt log is **refused** on this path, even though its readable prefix could be replayed: the whole log was asked for, so a prefix would be a silently incomplete history. On the ordinary finite-count path, damage lying *beyond* the messages actually needed is still tolerated +- **A shared log is replayed once, in full.** In `singular` and `periodic` multilog modes a segmented tickerplant writes every table to **one** file, and reports that file once *per table* with a different count each time. Replaying it once per entry re-applies the head of the file - measured, `(4;LF)` then `(2;LF)` over a six-message log applies messages 0 and 1 twice and never reaches 4 and 5. Duplicate entries for one file are therefore collapsed to a single replay of the whole file, with the table filter discarding the rest, and the condition is logged at `warn`. **The trade-off is trailing duplicates, not missing rows:** messages logged between the `subdetails` call and the replay arrive twice, once from the log and once on the live feed. That is deliberate - the alternative (summing the per-table counts) never duplicates but silently *drops* rows whenever the shared log also carries tables you did not subscribe to, and a missing row is invisible and permanent where a duplicate is visible and diagnosable. An **exact** duplicate entry - same file *and* same count - is rejected instead, because no shipped tickerplant can emit one: the segmented producer applies `distinct` to these pairs itself, and the chained and standard producers emit at most one entry each +- **A whole-file replay is narrowed to the offered tables; a per-table one is not.** An all-tables, all-syms subscribe normally replays the log raw, with no filter wrapper — correct for a per-table log, because the tickerplant returned that file *because* it belongs to a table it offered. It is not correct for a file read to its end, and the difference is structural rather than hypothetical: a segmented tickerplant opens logs for `` tables[`.] except `currlog `` (`stplog.q`, `logtabs`) but publishes only `` tables[] except `currlog`heartbeat`logmsg`svrstoload `` (`segmentedtickerplant.q`, `.stpps.init`), and `.stpps.upd` applies **no** membership check before logging. So in `singular` and `periodic` multilog modes the one shared file can legitimately carry tables the tickerplant declined to offer a schema for, and replaying it raw would drive the caller's `upd` with a table it never subscribed to — throwing part-way through a replay, or silently creating a wrongly-shaped table at root. Both whole-file triggers are narrowed to the subscribed set: the shared-log collapse and the `0W` `` `day `` sentinel, which has the same exposure. The sym filter still passes straight through when `syms` is `` ` ``, so the cost is one table-membership test per message on that path and nothing else. This was found by adversarial probe rather than by the suite, and it pre-dated the shared-log collapse — the collapse widened it from a leaked prefix to a leaked whole file rather than introducing it +- **Upstream requirement for `di.tickerplant`: do not `distinct` the per-table log pairs.** `getlogs` applies `distinct` to its `(messagecount;logname)` pairs (`stplog.q`), so when every table in a shared log happens to have the *same* count, the per-table entries are collapsed **at the tickerplant** and the reply that arrives is byte-identical to a legitimate single-log one. No subscriber can distinguish the two, because the distinguishing information was discarded before it was sent. The shared-log collapse here handles every case the protocol still carries — differing counts, and the `0W` sentinel — and this one is closed by the producer either not applying `distinct` or reporting per-table log offsets. Raised against `di.tickerplant` rather than worked around here, because a subscriber-side guess would have to be a heuristic and this module does not ship heuristics +- **`subdetails` is never sent the `` ` `` sentinel if it can be avoided.** A segmented tickerplant cannot accept it: its `subdetails` hands `tabs` straight to `.stplg.replaylog`, whose `where tbl in t` matches nothing for an atom and then signals `` 'rank ``. Legacy never hit this because it resolved a concrete table list with a `tablelist` round trip *first* and passed that onward; dropping that round trip for a single bundled call turned out to remove something load-bearing. `` ` `` is therefore resolved via `tablelist` before `subscribe` calls `subdetails`. If the tickerplant has no `tablelist`, or answers with something that is not a symbol list, the module logs at `warn` and sends `` ` `` anyway - a chained tickerplant handles the sentinel correctly, and a tickerplant with no `tablelist` is not a segmented one. The caller's *intent* is tracked separately from the resolved list, so resolving `` ` `` does not switch an all-tables subscription onto the narrowed replay path or raise spurious missing/unrequested warnings +- **`rowcounts` is validated for shape only, and two shapes are legitimate.** A dictionary keyed by table is the usual one; an **empty list** is equally real, because a chained tickerplant builds its response as `` (`schema`icounts`i`logfile`d)!() `` - which broadcasts `()` to every value - and only overwrites `icounts` when its *own* `subscribesyms` is `` ` ``. TorQ's own consumer guards for exactly that. Anything else - an atom, a table - is rejected. The *contents* are not checked against anything, because there is no `di.tickerplant` contract to check them against yet; they are passed through to the caller as received. The shape check earns its place because the field is load-bearing downstream: a chained subscriber seeds its `.u.icounts`/`.u.jcounts` straight off it, so a wrong shape would otherwise fail far from its cause - A malformed `logfilelist` is rejected on the same terms as `schemalist` — a non-list, an entry that is not a `(messagecount;logfile)` pair, a non-integer count or a non-symbol log file. A **negative** count is rejected too, which no shape check catches, since `-1` is a perfectly good integer; that guard lives in the response validation rather than the replay preflight so it also fires when `replay` is `0b` -- `subdetails` is TorQ's real protocol, defined at root by `chainedtp.q` and `segmentedtickerplant.q`, returning `` `schemalist`logfilelist`rowcounts`date `` and optionally `` `logdir ``. The key names and shapes here were taken from that source and are exercised against a tickerplant process built to the same protocol in the integration tests; the module has **not** yet been run against a live TorQ chained or segmented tickerplant. `di.tickerplant` is not built yet - when it lands, point `subdetailsfunc` at its entry point if the name differs -- Not in v1: auto-reconnect and resubscribe on tickerplant bounce, filtered-column subscriptions, and remote-log streaming (the subscriber is assumed to share the tickerplant's filesystem, the classic tick assumption) +- `subdetails` is TorQ's real protocol, defined at root by `chainedtp.q` and `segmentedtickerplant.q`, returning `` `schemalist`logfilelist`rowcounts`date `` and optionally `` `logdir ``. Both also define `tablelist` at root, which is how `` ` `` is resolved. The key names and shapes here were taken from that source and are exercised against a tickerplant process built to the same protocol in the integration tests; the module has **not** yet been run against a live TorQ chained or segmented tickerplant. `di.tickerplant` is not built yet - when it lands, point `subdetailsfunc` at its entry point if the name differs +- Not implemented: remote-log streaming. The subscriber is assumed to share the tickerplant's filesystem and replays from the log path the tickerplant reports — the classic tick assumption, and what legacy `.sub` does. This is a capability TorQ does not have either, rather than a port omission - The modularisation plan lists `di.servers` and `di.pubsub` as dependencies of this module. Neither is used: handle resolution is the caller's job, and `di.pubsub` is the publisher side - the tickerplant's own subscriber registry - which a subscribing process never calls -- `unsubscribe` is the supported way to close the local-close liveness gap, but it is a **cooperative** mechanism, not a structural one: kdb+ exposes no way to detect an *unannounced* `hclose`, and no way to tell whether a reissued descriptor is still the same remote. A caller that closes a tickerplant handle without calling `unsubscribe` first, and whose descriptor is then reissued, can still leave a stale row appearing live. Tickerplant death, and any local close that goes through `unsubscribe`, are both exact +- `unsubscribe` is the supported way to close the local-close liveness gap, but it is a **cooperative** mechanism, not a structural one: kdb+ exposes no way to detect an *unannounced* `hclose`, and no way to tell whether a reissued descriptor is still the same remote. A caller that closes a tickerplant handle without calling `unsubscribe` first, and whose descriptor is then reissued, can still leave a stale row appearing live. Tickerplant death, and any local close that goes through `unsubscribe`, are both exact. With `unsubscribefunc` configured the tickerplant-side release is exact too diff --git a/di/subscriptions/subscriptions.q b/di/subscriptions/subscriptions.q index 0814a090..d5a38c46 100644 --- a/di/subscriptions/subscriptions.q +++ b/di/subscriptions/subscriptions.q @@ -14,8 +14,9 @@ subscriptionsschema:([]handle:();tabs:();syms:();subtime:`timestamp$();active:`b / the keys a tickerplant's subdetails response must carry. these names and shapes are taken from the / real, shipped TorQ protocol - code/processes/chainedtp.q and segmentedtickerplant.q both define -/ subdetails at root - rather than invented here. NB the module has not yet been run against a live -/ TorQ tickerplant; the integration tests drive a process built to the same protocol +/ subdetails at root - rather than invented here. verified against a live TorQ v1.0 segmented +/ tickerplant (singular multilog, replayperiod day) as well as the integration tests, which drive a +/ separate process built to the same protocol requireddetailkeys:`schemalist`logfilelist`rowcounts`date; / what subscribe accepts as a tickerplant handle: an int handle, or a function standing in for one @@ -24,6 +25,30 @@ validhandletypes:-7 -6 100 104h; / the tickerplant-side function subscribe calls, unless config overrides it defaultsubdetailsfunc:`subdetails; +/ the tickerplant-side function that resolves ` (all tables) to a concrete list, unless config +/ overrides it. both shipped producers define it at root as {.stpps.t} - chainedtp.q and +/ segmentedtickerplant.q - and legacy calls it before subdetails for exactly this reason +defaulttablelistfunc:`tablelist; + +/ the tickerplant-side function that RELEASES this connection's subscriptions, if the tickerplant +/ offers one. defaults to ` - none - because shipped TorQ does not: pubsub.q's closesub is reachable +/ only from the tickerplant's own .z.pc, and suball/subfiltered each clear only their OWN registry +/ (pubsub.q:34,41), so going from an all-syms to a filtered subscription on one connection leaves the +/ all-syms entry behind and the wider feed keeps arriving. wire this at init and unsubscribe becomes +/ a real release; leave it unset and unsubscribe stays local bookkeeping and says so +defaultunsubscribefunc:`; + +/ appended to every error raised AFTER the subdetails call. asking a tickerplant for the schemas IS +/ .u.sub (pubsub.q defines .ps.subscribe:.u.sub), so the call registers this handle for live delivery +/ as a side effect, and the subdetails protocol has no unsubscribe verb to undo it - tickerplant-side +/ release is driven by .z.pc, which only the caller closing the handle can trigger. every other guard +/ now runs BEFORE that call, so preflight is the only place this can still happen; say so where it +/ happens rather than leaving the caller to infer it +/ kept short deliberately: q truncates a signalled error at 254 characters (measured), and these are +/ appended to messages that already carry a log path, so a verbose note would push the remedy off the +/ end of what the caller actually sees. the log always has the full text - raiseerror logs first +registerednote:" - close the handle before retrying, it is already registered with the tickerplant"; + / internal helpers initialised:{[] @@ -65,12 +90,29 @@ requiretabspec:{[ctx;nm;x] raiseerror[ctx;nm," is an empty symbol vector - pass ` for all, or name at least one"]]; }; +allsyms:{[syms] + / does this sym selector mean "everything" as far as the REPLAY is concerned? ` obviously does, and + / so does a filter dict: legacy's replayupd takes the same view (`if[(syms~`)or 99=type syms`) - + / the dict is a tickerplant-side filter expressed as parse clauses, which the log replay cannot + / evaluate, so it is passed to the tickerplant for the LIVE feed and the replay is left unfiltered + :(syms~`) or 99h=type syms; + }; + +requiresymspec:{[ctx;x] + / a sym selector is ` (all), one or more symbols, or a filter DICT keyed by table. the dict form is + / TorQ's own: rdb.q loads it from a csv (.sub.filterparams) and passes it straight through as the + / instruments argument, and .u.sub dispatches on its type (pubsub.q: 11h -> selfiltered, + / 99h -> addfiltered). rejecting it here would narrow a shipped API + if[99h=type x; :(::)]; + requiretabspec[ctx;"syms";x]; + }; + normspec:{[x] / normalise a table/sym selector to a LIST, leaving the ` all-sentinel alone. a bare symbol atom is / a natural way to name one table (subscribe[h;`trade;..]) and legacy accepted it - subscriptions.q / enlists both selectors the same way. without this an atom reaches `inter` and throws a bare 'type / that bypasses raiseerror and never reaches the log - :$[x~`;x;(),x]; + :$[x~`;x;99h=type x;x;(),x]; }; requireflag:{[ctx;nm;x] @@ -143,6 +185,13 @@ init:{[deps] sdf:$[`subdetailsfunc in key deps;deps`subdetailsfunc;defaultsubdetailsfunc]; if[not -11h=type sdf; '"di.subscriptions: subdetailsfunc must be a symbol naming the tickerplant-side function"]; + tlf:$[`tablelistfunc in key deps;deps`tablelistfunc;defaulttablelistfunc]; + if[not -11h=type tlf; + '"di.subscriptions: tablelistfunc must be a symbol naming the tickerplant-side function"]; + / ` (the default) means the tickerplant offers no release verb, so unsubscribe stays local-only + usf:$[`unsubscribefunc in key deps;deps`unsubscribefunc;defaultunsubscribefunc]; + if[not -11h=type usf; + '"di.subscriptions: unsubscribefunc must be a symbol naming the tickerplant-side function, or ` for none"]; fresh:not initialised[]; .z.m.loginfo:(deps`log)`info; .z.m.logwarn:(deps`log)`warn; @@ -150,26 +199,57 @@ init:{[deps] .z.m.register:(deps`handlers)`register; .z.m.removehandler:(deps`handlers)`remove; .z.m.subdetailsfunc:sdf; + .z.m.tablelistfunc:tlf; + .z.m.unsubscribefunc:usf; if[fresh;.z.m.subscriptions:subscriptionsschema]; + / cleared BEFORE the registration is attempted and set only once it has succeeded, so a register + / that throws leaves the flag false rather than unset or stale. that matters twice: .z.m.subscriptions + / is already assigned by this point, so initialised[] reports true even for an init that did not + / finish, and requireobserver would otherwise read an unset name and die with a bare 'observing that + / never reaches the log; and a FAILED re-init would otherwise leave a stale true behind + .z.m.observing:0b; / .z.pc is a SIMPLE (observer) event in di.handlers - side-effect only, fan-out - so the phase is / ` (null) and this coexists with every other .z.pc registrant in the priority-ordered chain .z.m.register[`.z.pc;`;`subscriptions;0j;markdead]; + .z.m.observing:1b; .z.m.loginfo[`init;"di.subscriptions initialised - tickerplant entry point ",string .z.m.subdetailsfunc]; }; teardown:{[] / release the .z.pc registration init installed, leaving no process-global residue. paired with - / init's one side effect - a module whose init registers nothing needs no teardown, this one does + / init's one side effect - a module whose init registers nothing needs no teardown, this one does. + / the registry is deliberately LEFT INTACT so a shutdown path can still inspect or release what was + / held; only the ability to take NEW subscriptions is withdrawn, by clearing the observing flag requireinit[`teardown]; .z.m.removehandler[`.z.pc;`;`subscriptions]; + .z.m.observing:0b; .z.m.loginfo[`teardown;"di.subscriptions .z.pc registration removed"]; }; +requireobserver:{[ctx] + / a new subscription is only trackable while the .z.pc observer is installed. after teardown it is + / not, and the registry would then report a dead tickerplant as live for as long as .z.W held the + / handle number - and indefinitely once kdb+ reissued that number to another connection. that is + / precisely the failure the handlers dependency exists to prevent (see deps.q), so taking a new + / subscription in that state is refused rather than silently degraded to the .z.W-only mode this + / module documents as insufficient. reading and releasing stay available, for shutdown paths + / read explicitly, not as a bare name. a bare read does resolve to .z.m, but every other state + / access in this module is explicit, and the bare form is the one thing qlint flags as an + / undeclared global - a warning a reader has to dismiss by hand every time + if[not .z.m.observing; + raiseerror[ctx;"the .z.pc observer is not installed - teardown removed it, or init did not ", + "complete - so a new subscription could not be tracked. call init again before subscribing"]]; + }; + / subscription fetchdetails:{[tph;tabs;syms] - / one bundled round trip to the tickerplant. legacy issued a separate tablelist call first to learn - / what was available; the bundled response makes that round trip unnecessary + / the bundled round trip: schemas, log details and counts in one call. + / NB this does NOT replace legacy's separate tablelist call - subscribe still makes that one first, + / and must (see publishedtabs and narrowtabs). an earlier revision of this module dropped it on the + / assumption that the bundled response made it redundant; it does not, because subdetails is + / .ps.subscribe each-left and fails outright on a name the tickerplant does not publish. + / calling this REGISTERS the handle for live delivery as a side effect - see registerednote r:@[{[h;m] (1b;h m)}[tph];(.z.m.subdetailsfunc;tabs;syms);{[e] (0b;e)}]; if[not first r; raiseerror[`subscribe;"tickerplant ",(string .z.m.subdetailsfunc)," call failed: ",last r]]; @@ -195,8 +275,12 @@ fetchdetails:{[tph;tabs;syms] / succeeds for any value - and subscribe then reports success over a root name that is not a table. / .Q.qt, not 98h=type: a KEYED table is 99h and must still be accepted, while a column-less ([]) is / also 99h and must not be (its cols are empty, so replaying into it is meaningless) + / NB a STRING here is not a garbled schema, it is the tickerplant declining the table: TorQ's + / .u.sub returns (name;"Table ... not in list of stp pub/sub tables") for one it does not publish, + / so name that cause rather than leaving the caller to decode a bare shape complaint if[not all .Q.qt each entries[;1]; - raiseerror[`subscribe;(string .z.m.subdetailsfunc)," schemalist entries must give a table as the schema"]]; + raiseerror[`subscribe;(string .z.m.subdetailsfunc)," schemalist entries must give a table as the ", + "schema - a string in that position is the tickerplant refusing to publish the table"]]; / a duplicate table name would be carried straight through to subtables, which di.rdb and di.wdb / iterate over, and into the registry's tabs column. reject rather than silently dedupe - every / other malformed response here fails loud, and deduping would hide the tickerplant's own bug @@ -228,19 +312,95 @@ fetchdetails:{[tph;tabs;syms] if[0= - fetchdetails has already rejected a negative count as a malformed response, so a / negative can no longer reach here and must not be quietly folded into "nothing to replay" - if[0=nmsg; :(::)]; - good:goodcount lf; + if[0=nmsg; :(nmsg;lf;0b)]; + st:logstatus lf; + good:first st; + / 0W is the "everything readable" sentinel, NOT a claim about the count, and it arrives two ways: + / a segmented tickerplant sends it for every CLOSED log under replayperiod `day (stplog.q's + / getlogs[`day] sets msgcount:0Wj), and collapsesharedlogs sets it for a log several tables share. + / either way the answer is the same - replay the file to its full preflighted total. + / resolve it HERE rather than forwarding 0W to -11!: measured, -11!(0W;corruptlog) replays the good + / prefix and only THEN throws 'badtail, which is precisely the half-populated state this preflight + / exists to prevent. only long infinity is a sentinel; a merely large finite count is still an + / over-claim and still fails below. + / a CORRUPT log is refused on this path even though its readable prefix could be replayed: the + / whole log was asked for, so replaying part of it would hand the subscriber a silently incomplete + / history. on the finite path below, damage beyond the messages actually wanted is still tolerated + if[0W=nmsg; + if[last st; + raiseerror[`replay;"log ",(string lf)," is truncated after ",(string good)," readable ", + "message(s) and the tickerplant asked for the whole log - refusing to replay an ", + "incomplete history",registerednote]]; + :(good;lf;1b)]; if[good=nmsg; .z.m.loginfo[`replay;"nothing to replay from ",string lf]; :(::)]; - n:$[alltabs and syms~`;replayall[nmsg;lf];replaynarrowed[nmsg;lf;wanted;syms]]; + / the unfiltered fast path is only safe when the log cannot hold anything outside the subscription. + / for a per-table log that is guaranteed: the tickerplant returned the file BECAUSE it belongs to a + / table it offered us. for a WHOLE-FILE entry it is not, and the difference is structural rather + / than hypothetical - a segmented tickerplant opens logs for tables[`.] except `currlog (stplog.q + / init, logtabs) but publishes only tables[] except `currlog`heartbeat`logmsg`svrstoload + / (segmentedtickerplant.q, .stpps.init), and .stpps.upd applies NO membership check before logging. + / so in singular/periodic multilog mode the one shared file can legitimately carry tables the + / tickerplant declined to offer a schema for, and replaying it raw would drive the caller's upd with + / a table it never subscribed to - throwing part way through, or silently creating a wrongly shaped + / table at root. narrow those to `wanted`; the sym filter still passes through untouched when syms + / is `, so this costs the whole-file path a table-membership test per message and nothing else + raw:alltabs and (allsyms syms) and not wholefile; + n:$[raw;replayall[nmsg;lf];replaynarrowed[nmsg;lf;wanted;syms]]; .z.m.loginfo[`replay;"replayed ",(string n)," message(s) from ",string lf]; }; -replaylogs:{[details;wanted;syms;alltabs] - / replay every pre-subscription log the tickerplant reported - lfl:logentries details; - if[0=count lfl; +replaylogs:{[entries;wanted;syms;alltabs] + / replay every pre-subscription log the tickerplant reported. entries are the PREFLIGHTED triples + / from preflightlogs - shared logs already collapsed to one replay each, any 0W already resolved + if[0=count entries; .z.m.loginfo[`replay;"tickerplant reported no log file to replay"]; :(::)]; - replayone[;wanted;syms;alltabs] each lfl; + replayone[;wanted;syms;alltabs] each entries; }; / return shape @@ -425,9 +650,20 @@ tplogdate:{[details] }; buildreturn:{[details;subtabs] - / the shape legacy's callers actually consume: rdb.q reads subtables and tplogdate, wdb.q reads - / tplogdate for fixpartition, chainedtp.q reads date and rowcounts (its .u.d and .u.icounts) + / the shape legacy's callers actually consume, read off the shipped consumers rather than assumed. + / rdb.q takes subtables and tplogdate (rdb.q:171) and wdb.q takes tplogdate for fixpartition + / (wdb.q:546) - but a chained tickerplant reads `d` and `icounts`, NOT `date` and `rowcounts` + / (chainedtp.q:81-84, sctp.q:22-25), and it reads them through `if[key in r]` guards. a missing key + / there does not fail: it silently never seeds .u.d or .u.icounts/.u.jcounts, and every downstream + / subscriber of that process then gets wrong counts. so both names are emitted. + / rowcounts and date stay canonical - they are the tickerplant's OWN key names, carried through from + / the subdetails reply unchanged, whereas subtables and tplogdate are names legacy invented and so + / keep legacy's spelling. icounts and d are the compatibility surface for code ported from TorQ. + / legacy's `i` is deliberately NOT emitted: no shipped consumer reads it, and legacy gives it two + / different meanings by tickerplant type - the whole logfilelist for segmented, the message count + / for standard and chained - which this module has no tptype to disambiguate between r:`subtables`tplogdate`rowcounts`date!(subtabs;tplogdate details;details`rowcounts;details`date); + r:r,`icounts`d!(details`rowcounts;details`date); :$[`logdir in key details;r,enlist[`logdir]!enlist details`logdir;r]; }; @@ -441,34 +677,82 @@ subscribe:{[tph;tabs;syms;setschema;replay] / replay: replay the pre-subscription tp log - requires a root-level upd. / returns `subtables`tplogdate`rowcounts`date, plus `logdir when the tickerplant supplied one requireinit[`subscribe]; + requireobserver[`subscribe]; requirehandle[`subscribe;tph]; requiretabspec[`subscribe;"tabs";tabs]; - requiretabspec[`subscribe;"syms";syms]; + requiresymspec[`subscribe;syms]; requireflag[`subscribe;"setschema";setschema]; requireflag[`subscribe;"replay";replay]; + / capture the all-tables INTENT before resolving it. a segmented tickerplant cannot be sent the ` + / sentinel (see publishedtabs), but the replay path and the mismatch warnings still need to know the + / caller asked for everything rather than for a specific list - otherwise resolving ` would quietly + / switch an all-tables subscribe onto the narrowed replay path and warn about every table the + / tickerplant chose not to return + alltabs:tabs~`; / normalise a bare symbol atom to a list before anything indexes or intersects it tabs:normspec tabs; syms:normspec syms; - details:fetchdetails[tph;tabs;syms]; + / ONE tablelist round trip serves both jobs, and it registers nothing, so it runs before everything + published:publishedtabs[tph;alltabs]; + / what the caller effectively asked for, with ` resolved. the guards below compare against THIS and + / not against the narrowed list: re-subscribing a table you already hold, or claiming with + / setschema 0b that a table exists when it does not, are caller mistakes whatever the tickerplant + / happens to publish this round - narrowing first would let exactly those mistakes back through + requested:$[alltabs;published;tabs]; + if[alltabs and not published~`; + .z.m.loginfo[`subscribe;"resolved ` to ",(", " sv string requested)," via ",string .z.m.tablelistfunc]]; + / EVERY guard that does not need the tickerplant's reply runs here, before fetchdetails, because + / fetchdetails REGISTERS this handle for live delivery as a side effect (see registerednote) and + / nothing can undo that. legacy orders it the same way - reducesubs runs against a tablelist round + / trip before subfunc (TorQ subscriptions.q:108-110) - so this restores the original ordering rather + / than inventing one. ` cannot always be resolved (a tickerplant offering no table list), and these + / need a concrete list, so that one case is skipped here and caught by the copies below + if[replay;requirerootupd[]]; + if[not requested~`;guardduplicate[requested]]; + / the tables-exist check runs early only for an EXPLICIT request. on the all-tables path + / `requested` is the TABLELIST list, which may advertise more than schemalist actually returns, and + / this guard asserts that EVERY name exists at root - so a superset there would refuse a perfectly + / valid subscribe. the duplicate guard above is unaffected by the same superset, because it only + / bites where the list INTERSECTS a table already held, and that is a caller mistake either way. + / the all-tables case is covered by the post-reply copy against `wanted`, once the reply has said + / what is really on offer + if[replay; + if[not alltabs; + if[not setschema;requiretablesexist[requested]]]]; + / what we actually SEND. an explicit request is narrowed to what the tickerplant publishes, because + / one unpublished name fails the whole subdetails call - see narrowtabs + sendtabs:$[alltabs or published~`;requested;narrowtabs[tabs;published]]; + details:fetchdetails[tph;sendtabs;syms]; schemapairs:(details`schemalist) where not 0=count each details`schemalist; offered:(),schemapairs[;0]; / what we actually subscribe to is what we ASKED FOR intersected with what the tickerplant offered / - not simply everything it returned. driving the replay filter off the offered set instead would / replay tables that were never requested - wanted:$[tabs~`;offered;(),tabs inter offered]; + wanted:$[alltabs;offered;(),sendtabs inter offered]; if[0=count wanted; raiseerror[`subscribe;"tickerplant returned no schema for the requested table(s) - nothing to subscribe to"]]; + / redundant whenever the early copy above ran, and the ONLY check when it did not (an all-tables + / subscribe to a tickerplant with no usable table list). left in rather than made conditional: it is + / a cheap select, and a guard that silently does not run on some paths is worse than one that runs twice guardduplicate[wanted]; - warnmissing[tabs;offered]; - warnextra[tabs;offered]; + / an all-tables subscribe has nothing to compare - whatever the tickerplant offers IS the request - + / so the mismatch warnings are driven by the caller's intent, not by the resolved list. + / these compare against what we SENT, not what the caller asked for: a table narrowtabs already + / dropped has been reported once with the real reason, and reporting it again here as "the + / tickerplant did not return it" would describe the same fact worse + if[not alltabs; + warnmissing[sendtabs;offered]; + warnextra[sendtabs;offered]]; schemapairs:schemapairs where schemapairs[;0] in wanted; - / preflight EVERY log before defining a single table, so a short log leaves the process untouched + / preflight EVERY log before defining a single table, so a short log leaves the process untouched. + / preflightlogs is the one guard that CANNOT move above fetchdetails - the log file names only exist + / in the reply, and their integrity can only be established by reading them - so it is the single + / remaining place a throw can leave the tickerplant publishing into a failed subscribe if[replay; - requirerootupd[]; if[not setschema;requiretablesexist[wanted]]; - preflightlogs[details]]; + entries:preflightlogs[details]]; if[setschema;createtables[schemapairs]]; - if[replay;replaylogs[details;wanted;syms;tabs~`]]; + if[replay;replaylogs[entries;wanted;syms;alltabs]]; / catenate+reassign, NOT (`name insert row): a symbol-mediated insert resolves the LITERAL name at / root and would miss the compile-time module-local rewrite a source-level .z.m.subscriptions gets .z.m.subscriptions:.z.m.subscriptions, @@ -507,10 +791,75 @@ unsubscribe:{[tph] :`$()]; .z.m.subscriptions:delete from .z.m.subscriptions where active, handle~\:tph; released:distinct (),raze held`tabs; + / release at the TICKERPLANT too when it offers a verb for it. without this the tickerplant keeps + / publishing everything this connection ever asked for, so re-subscribing more narrowly on the same + / handle still delivers the wider feed - measured against a live segmented tickerplant. + / a release failure is not fatal: the local rows are already gone and the caller's next step is to + / close the handle, which releases it anyway. it is reported so it cannot pass unnoticed + $[.z.m.unsubscribefunc~`; + .z.m.logwarn[`unsubscribe;"released locally only - no unsubscribefunc configured, so the ", + "tickerplant keeps publishing on this handle until the connection closes"]; + [r:@[{[h;m] (1b;h m)}[tph];(.z.m.unsubscribefunc;released);{[e] (0b;e)}]; + $[first r; + .z.m.loginfo[`unsubscribe;"tickerplant released ",(", " sv string released)," via ", + string .z.m.unsubscribefunc]; + .z.m.logwarn[`unsubscribe;"tickerplant ",(string .z.m.unsubscribefunc)," call failed (", + (last r),") - close the handle to release it"]]]]; .z.m.loginfo[`unsubscribe;"released ",(", " sv string released)," on tickerplant handle ",.Q.s1 tph]; :released; }; +resubscribe:{[tph] + / re-establish every subscription that has since dropped, over a NEW handle to the same tickerplant. + / this is legacy's retrysubscription (subscriptions.q:155) ported to this module's shape. legacy + / drove it from .servers.connectcustom, which would make di.servers a hard dependency and would + / mean this module resolving connections - the one thing it deliberately does not do. so the split + / is: the module keeps the knowledge of WHAT was subscribed, the caller supplies the new handle, + / and di.rdb/di.servers decide WHEN to call it. + / setschema 0b and replay 0b exactly as legacy does: the tables are already defined and their + / history was replayed on the first subscribe, so a reconnect wants the live feed back and nothing + / else. replaying again would double-apply everything since the original subscription. + / best-effort per subscription and never fatal - a reconnect path that aborts on the first failure + / leaves the rest of the process unsubscribed with no way to retry + requireinit[`resubscribe]; + requireobserver[`resubscribe]; + requirehandle[`resubscribe;tph]; + dead:select tabs,syms from activesubscriptions[] where not active; + if[0=count dead; + .z.m.loginfo[`resubscribe;"no dropped subscription to re-establish"]; + :`$()]; + / ask this tickerplant what it publishes, ONCE, and skip dead subscriptions it cannot serve. + / a process may hold subscriptions to several tickerplants; without this, reconnecting one of them + / retries the others' tables against it and warns about each, every call - and a caller drives this + / from a timer. the round trip is pure (see publishedtabs) and replaces one failed subscribe per row. + / when the tickerplant offers no table list there is nothing to filter on, so everything is attempted + published:publishedtabs[tph;0b]; + if[not published~`; + g:{[p;t] :any ((),t) in p}[published]; + dead:dead where g'[dead`tabs]]; + if[0=count dead; + .z.m.loginfo[`resubscribe;"no dropped subscription this tickerplant can serve"]; + :`$()]; + done:raze {[tph;t;s] + r:@[{[tph;t;s] subscribe[tph;t;s;0b;0b]; (1b;t)}[tph;t];s;{[e] (0b;e)}]; + if[not first r; + .z.m.logwarn[`resubscribe;"could not re-establish ",(", " sv string (),t),": ",last r]; + :`$()]; + :(),t}[tph]'[dead`tabs;dead`syms]; + done:distinct (),done; + if[0/dev/null || true";{[e] :(::)}]; + }; + isfree:{[p] :not @[{hclose hopen x;1b};(`$":localhost:",string p;100);{[e] :0b}]; }; pickport:{[start] :first (start+til 500) where isfree each start+til 500; }; waitlisten:{[port;timeoutms] @@ -158,6 +233,8 @@ peerlines:{[] / continuation: a q expression ends at the newline, so a split list literal silently becomes / several expressions rather than one element l:enlist "D:2026.08.06;"; + / a per-run identity so spawnpeer can prove it is talking to the peer it just started + l,:enlist "PEERTOKEN:`",(string PEERTOKEN),";"; l,:enlist "LF:hsym`$\"",PEERDIR,"/tp\",string D;"; l,:enlist "peerschema:([]time:`timestamp$();sym:`g#`symbol$();price:`float$();size:`int$());"; l,:enlist "N:",string PEERMSGS; @@ -171,6 +248,9 @@ peerlines:{[] l,:enlist " `schemalist`logfilelist`rowcounts`date!(enlist(`peertrade;peerschema);"; l,:enlist " enlist(N;LF);(enlist `peertrade)!enlist N;D)};"; l,:enlist "subdetails:sd;"; + / the peer serves tablelist too, as both real producers do, so the integration block exercises the + / ` resolution round trip over genuine IPC rather than only against a function standing in for one + l,:enlist "tablelist:{enlist`peertrade};"; / pub logs a message AND publishes it live to every subscriber, exactly as a tickerplant does - / this is what makes the exactly-once boundary (replay then live) testable end to end l,:enlist "pub:{[i;s] m:(`upd;`peertrade;(enlist 2026.08.06D11:00+`timespan$1000000000*i;enlist s;"; @@ -186,13 +266,22 @@ writepeerscript:{[] }; spawnpeer:{[] + killstalepeers[]; system "rm -rf ",PEERDIR; system "mkdir -p ",PEERDIR; + `PEERTOKEN set `$"peer",string[`int$.z.i],"run",string `int$.z.n mod 1000000; `PEERPORT set pickport 21000+`int$.z.i mod 10000; writepeerscript[]; system (getenv[`QHOME]),"/bin/q ",PEERDIR,"/tp.q -p ",string[PEERPORT]," -q /dev/null 2>&1 &"; if[not waitlisten[PEERPORT;5000];'"test: tickerplant peer failed to listen on ",string PEERPORT]; h:hopen (`$":localhost:",string PEERPORT;2000); + / prove this is OUR peer. waitlisten only proves SOMETHING is listening, and pickport probes each + / candidate with a 100ms connect - under load an occupied port can read as free, our peer then + / fails to bind, and the suite would silently run against a peer left over from an aborted run + / whose log files the rm -rf above has just deleted. fail loudly instead of testing the wrong process + if[not PEERTOKEN~@[h;"PEERTOKEN";{[e] `}]; + hclose h; + '"test: port ",string[PEERPORT]," is served by a foreign process, not this run's peer"]; `PEERPID set h ".z.i"; hclose h; }; @@ -214,4 +303,4 @@ flushpeer:{[h] killpeer:{[] if[not null PEERPID;@[system;"kill ",string PEERPID;{[e] :(::)}]]; `PEERPID set 0N; system "sleep 0.3"; }; setupfixture:{[] system "rm -rf ",BASE; system "mkdir -p ",BASE; }; -teardownfixture:{[] killpeer[]; system "rm -rf ",BASE; system "rm -rf ",PEERDIR; }; +teardownfixture:{[] killpeer[]; killstalepeers[]; system "rm -rf ",BASE; system "rm -rf ",PEERDIR; }; From dc9c404497d34f48dbe82f3fa9788136475ce962 Mon Sep 17 00:00:00 2001 From: alowrydi Date: Wed, 12 Aug 2026 16:16:51 +0100 Subject: [PATCH 04/10] Improving testing conventions --- di/subscriptions/subscriptions.md | 36 ++- di/subscriptions/test.csv | 157 ++++++------- di/subscriptions/test.q | 306 -------------------------- di/subscriptions/test_integration.csv | 149 +++++++++++++ 4 files changed, 248 insertions(+), 400 deletions(-) delete mode 100644 di/subscriptions/test.q create mode 100644 di/subscriptions/test_integration.csv diff --git a/di/subscriptions/subscriptions.md b/di/subscriptions/subscriptions.md index f3e82a9c..e59fa755 100644 --- a/di/subscriptions/subscriptions.md +++ b/di/subscriptions/subscriptions.md @@ -180,14 +180,38 @@ hclose tph ## Running Tests +Two suites, following the convention `di.handlers` and `di.permissions` use. Both define their +fixtures inline as `before` rows; there is no separate fixture file, so neither depends on the +working directory. + +**Unit suite** (`test.csv`) - 450 assertions, no child processes, no ports, no `QHOME`: + ```q k4unit:use`di.k4unit k4unit.moduletest`di.subscriptions ``` -Run in a fresh q session - the integration layer spawns and kills a real q process, so do not interleave with other modules' tests. Needs `QHOME` set (the peer is launched via `$QHOME/bin/q`) and `di.os` on `QPATH` (the harness uses `os.abspath` to load `test.q`). +**Integration suite** (`test_integration.csv`) - 73 assertions. It stands up a real q process as a +tickerplant and drives it over genuine IPC, so it needs a q binary reachable via `QHOME`. +`moduletest` only ever loads `test.csv` (`di/k4unit/init.q`), which is why these rows live in their +own file - otherwise every unit run would spawn a process and bind a port. Load it explicitly: -The suite is 523 assertions (`true`, `fail` and `run` rows, excluding the 5 fixture `before`/`after` rows) and wires the real merged `di.log` and `di.handlers` rather than mocks, so both injected contracts are proven end to end. It covers: pre-`init` guards on every export; dependency validation (non-dict deps, each missing key, each malformed value); `init` arity asserted by observable effect rather than by "it did not throw"; idempotent re-init; the `VERSION` file and exported `version`; `getapimeta`; all-tables and sym-filtered replay; all four payload shapes a tickerplant may log; symbol-atom table and sym selectors; a multi-table log where only the requested table is defined; replay across two log files; requested-but-absent and offered-but-unrequested tables; a configured `subdetailsfunc`; partial, unreadable, absent and empty logs; a replay requested with no root `upd`; malformed `schemalist` **and** `logfilelist` shapes, including a non-table schema, a column-less table, a duplicated table name, and a keyed schema that must still be accepted; a negative message count, asserted with `replay:0b` as well so the guard cannot drift behind the replay branch; every `setschema`/`replay` combination, including `setschema:0b` with `replay:1b` on both the narrowed and all-syms paths; empty `tabs` and `syms` selectors; the double-subscribe guard and its release by `unsubscribe`; `unsubscribe` deleting rather than flagging, its idempotency, and its `warn` on an unheld handle; malformed tickerplant responses; `setschema:0b` preserving a populated table; `logdir` passthrough; and input validation on every argument. +```q +k4unit:use`di.k4unit +.m.di.0k4unit.KUltf .Q.dd[hsym`$.Q.m.mp`di.subscriptions;`test_integration.csv] +.m.di.0k4unit.KUrt[] +select from .m.di.0k4unit.KUTR where not ok +``` + +`KUrt` prints the results table but, unlike `moduletest`, no pass/fail summary - the last line is the +verdict, and an empty result means everything passed. + +Run the integration suite in a **fresh** q session - after `moduletest` the unit tests are still +loaded and would re-run against dirty module state, reporting spurious failures. It is fully +self-contained: it inits `di.handlers` and `di.subscriptions` itself rather than inheriting setup +from the unit rows. + +Together the two suites are 523 assertions (`true`, `fail` and `run` rows, excluding the fixture `before`/`after` rows) and wire the real merged `di.log` and `di.handlers` rather than mocks, so both injected contracts are proven end to end. Between them they cover: pre-`init` guards on every export; dependency validation (non-dict deps, each missing key, each malformed value); `init` arity asserted by observable effect rather than by "it did not throw"; idempotent re-init; the `VERSION` file and exported `version`; `getapimeta`; all-tables and sym-filtered replay; all four payload shapes a tickerplant may log; symbol-atom table and sym selectors; a multi-table log where only the requested table is defined; replay across two log files; requested-but-absent and offered-but-unrequested tables; a configured `subdetailsfunc`; partial, unreadable, absent and empty logs; a replay requested with no root `upd`; malformed `schemalist` **and** `logfilelist` shapes, including a non-table schema, a column-less table, a duplicated table name, and a keyed schema that must still be accepted; a negative message count, asserted with `replay:0b` as well so the guard cannot drift behind the replay branch; every `setschema`/`replay` combination, including `setschema:0b` with `replay:1b` on both the narrowed and all-syms paths; empty `tabs` and `syms` selectors; the double-subscribe guard and its release by `unsubscribe`; `unsubscribe` deleting rather than flagging, its idempotency, and its `warn` on an unheld handle; malformed tickerplant responses; `setschema:0b` preserving a populated table; `logdir` passthrough; and input validation on every argument. It also covers the TorQ-protocol cases this module is built to survive: a `0W` message count replaying a whole log, and the same count over a **truncated** log being refused with nothing replayed and no table defined; a **shared** log reported once per table being collapsed to one full replay, with the per-table row counts asserted individually so a regression to the old per-entry replay is caught rather than merely the total; an unshared file in the same response keeping its own count and raising no warn; a corrupt shared log refused, proving the corruption guard covers the collapse trigger and not only the `0W` sentinel; an exact duplicate entry rejected; `` ` `` resolved through `tablelist` with the *resolved* list asserted to be what `subdetails` actually received; the fallback to `` ` `` when a tickerplant offers no `tablelist`; a configured `tablelistfunc` asserted to be called **first**, ahead of `subdetails`; and the two legitimate `rowcounts` shapes accepted with atoms and tables rejected. @@ -197,7 +221,13 @@ The whole-file narrowing is asserted on a shared log that carries a table the ti Guard **ordering** is asserted directly rather than inferred, using a fixture that records every remote function name `subscribe` asks for. Each guard that was moved ahead of the `subdetails` call has a row proving the tickerplant was never asked at all when it fires — including the case whose semantics changed, a requested table that is already held *and* no longer offered, which now throws where it previously warned and continued. The residual cases have rows too: an all-tables subscribe to a tickerplant with no `tablelist` reaches the post-reply copy of the duplicate guard, and a short log is caught only after `subdetails` has run, with assertions on both halves of the error text that tells the caller to close the handle. -The suite also covers teardown/re-init lifecycle cycles, and asserts the `di.handlers` isolation contract: a co-registrant that throws at a priority ahead of this module's observer must not suppress `markdead`. That assertion is made through `unsubscribe`, which matches on the stored flag - checking `active` alone would be vacuous, since the closed handle reads dead through `.z.W` either way. +Everything above is in the **unit** suite except where it needs a live publisher. The **integration** +suite owns what cannot be faked in process: the replay-then-live exactly-once boundary, a real `.z.pc` +firing when the peer is killed, the `` ` `` round trip over the wire, `unsubscribe` closing the +local-close liveness gap against a genuinely reissued descriptor, and the `di.handlers` isolation +contract below. + +The integration suite also covers teardown/re-init lifecycle cycles, and asserts the `di.handlers` isolation contract: a co-registrant that throws at a priority ahead of this module's observer must not suppress `markdead`. That assertion is made through `unsubscribe`, which matches on the stored flag - checking `active` alone would be vacuous, since the closed handle reads dead through `.z.W` either way. The integration block subscribes to a genuinely separate tickerplant process over IPC, kills it, confirms `.z.pc` marks the subscription dead, reconnects onto a recycled handle number, confirms the stale row stays dead and the re-subscribe succeeds, then drives live updates through the replayed tables to prove the exactly-once boundary. It finishes on the two cases only a real handle can reach: releasing a handle and `hclose`ing it while the tickerplant is still alive (so nothing fires `.z.pc`), reopening onto the same reissued descriptor and confirming no revival; and the reverse order — `hclose` *first*, then `unsubscribe` — which proves the release matches on the stored flag rather than the computed one, and so still finds a row `.z.W` has already given up on. diff --git a/di/subscriptions/test.csv b/di/subscriptions/test.csv index d63f03ac..f9603f13 100644 --- a/di/subscriptions/test.csv +++ b/di/subscriptions/test.csv @@ -1,10 +1,72 @@ action,ms,bytes,lang,code,repeat,minver,comment -comment,,,,,,,setup - load the module and the fixtures. NB init is deliberately NOT called in a before row: the pre-init guard tests below must run first +comment,,,,,,,setup - fixtures are defined inline as before rows. k4unit runs every before row first and in file order so a fixture may call one defined above it before,0,0,q,sub:use`di.subscriptions,1,1,load di.subscriptions -before,0,0,q,os:use`di.os,1,1,di.os for abspath (test harness only - not a module dependency) -before,0,0,q,"system ""l "", os.abspath[""di/subscriptions/test.q""]",1,1,load the fixture helpers +before,0,0,q,"BASE:""/tmp/disubscriptionstest"";",1,1,fixture: BASE +before,0,0,q,D:2026.08.06;,1,1,fixture: D +before,0,0,q,caprows:([]lvl:`symbol$();ctx:`symbol$();msg:());,1,1,fixture: caprows +before,0,0,q,resetcap:{[] `caprows set 0#caprows; };,1,1,fixture: resetcap +before,0,0,q,caplog:`info`warn`error!( {[c;m] `caprows insert (`info;c;m)}; {[c;m] `caprows insert (`warn;c;m)}; {[c;m] `caprows insert (`error;c;m)});,1,1,fixture: caplog +before,0,0,q,realhandlers:use`di.handlers;,1,1,fixture: realhandlers +before,0,0,q,handlerdep:{[] :`register`remove!(realhandlers[`register];realhandlers[`remove]); };,1,1,fixture: handlerdep +before,0,0,q,deps:{[] :`log`handlers!(caplog;handlerdep[]); };,1,1,fixture: deps +before,0,0,q,markdeadfor:{[h] .z.pc h; };,1,1,fixture: markdeadfor +before,0,0,q,mkreconnecttph:{[t;n;lf] :{[t;n;lf;msg] if[0b;()]; $[`tablelist~first msg;enlist t; `schemalist`logfilelist`rowcounts`date!(enlist(t;genschema[]);enlist(n;lf);(enlist t)!enlist n;D)]}[t;n;lf]; };,1,1,fixture: mkreconnecttph +before,0,0,q,"throwingregdep:{[] :`register`remove!({[e;p;n;pr;f] '""register exploded""};realhandlers[`remove]); };",1,1,fixture: throwingregdep +before,0,0,q,RELEASED:`nil;,1,1,fixture: RELEASED +before,0,0,q,mkreleasingtph:{[inner] :{[inner;msg] $[`releaseme~first msg;[`RELEASED set msg 1;1b];inner msg]}[inner]; };,1,1,fixture: mkreleasingtph +before,0,0,q,"logged:{[lv;s] :any {[s;m] 0.z.i,1,1,the peer really is a separate process -true,0,0,q,`schemalist`logfilelist`rowcounts`date~key PH(`subdetails;`;`),1,1,the peer serves the real TorQ subdetails protocol -run,0,0,q,resetupd[],1,1,reset the upd counter -comment,,,,,,,an all-tables subscribe works here even though many unit subscriptions are still live - the guard is table-precise -run,0,0,q,rp:sub.subscribe[PH;`;`;1b;1b],1,1,subscribe over the real handle -true,0,0,q,`peertrade in tables[],1,1,the table was created at root from the schema sent over IPC -true,0,0,q,6=count peertrade,1,1,all six logged messages replayed -true,0,0,q,`g=attr peertrade`sym,1,1,the g# attribute survived the IPC round trip -true,0,0,q,6=updcalls,1,1,the replay drove the root upd once per message -true,0,0,q,PH~first exec handle from sub.getsubscriptions[] where handle~\:PH,1,1,the registry holds the real int handle -true,0,0,q,any exec active from sub.getsubscriptions[] where handle~\:PH,1,1,the subscription over the real handle is live - -comment,,,,,,,kill the tickerplant - the .z.pc observer must mark the subscription dead -run,0,0,q,killpeer[],1,1,kill the peer process -run,0,0,q,noticedrop[PH],1,1,touch the handle so the drop is detected and .z.pc fires -true,0,0,q,not any exec active from sub.getsubscriptions[] where handle~\:PH,1,1,the row for the dropped handle was marked dead - -comment,,,,,,,restart and reconnect - kdb+ recycles the handle number and the stale row must NOT revive -run,0,0,q,spawnpeer[],1,1,restart the tickerplant -run,0,0,q,PH2:peerhandle[],1,1,reconnect -true,0,0,q,PH~PH2,1,1,kdb+ handed back the same handle number (the recycling a .z.W probe alone cannot see through) -true,0,0,q,not any exec active from sub.getsubscriptions[] where handle~\:PH2,1,1,the stale row stays dead despite the number being reused -run,0,0,q,resetupd[],1,1,reset the upd counter -run,0,0,q,@[`.;`peertrade;:;0#peertrade],1,1,clear the table before replaying again -run,0,0,q,sub.subscribe[PH2;`;`;1b;1b],1,1,re-subscribing after a reconnect is allowed -true,0,0,q,6=count peertrade,1,1,the log replayed again on the new handle -true,0,0,q,2=count select from sub.getsubscriptions[] where handle~\:PH2,1,1,both the stale and the new row are for the recycled handle number -true,0,0,q,1=sum exec active from sub.getsubscriptions[] where handle~\:PH2,1,1,exactly one of them is live - the stale one stayed dead - -comment,,,,,,,LIVE updates after the replay - the exactly-once boundary. this is the module's central promise and cannot be tested without a publishing tickerplant -true,0,0,q,6=updcalls,1,1,the replay drove upd once per logged message and no more -true,0,0,q,6=PH2"logged[]",1,1,the tickerplant has logged six messages so far -run,0,0,q,(neg PH2)(`pub;100;`S0),1,1,the tickerplant logs AND publishes one live message -run,0,0,q,flushpeer[PH2],1,1,round trip so the async live message is serviced -true,0,0,q,7=PH2"logged[]",1,1,the tickerplant logged the live message too -true,0,0,q,7=count peertrade,1,1,six replayed plus one live - the boundary message was NOT replayed as well -true,0,0,q,7=updcalls,1,1,upd ran exactly once per message across replay and live feed -true,0,0,q,100f=last exec price from peertrade,1,1,the live message is the most recent row -run,0,0,q,(neg PH2)(`pub;101;`S1),1,1,publish a second live message -run,0,0,q,flushpeer[PH2],1,1,service it -true,0,0,q,8=count peertrade,1,1,live updates keep flowing through the root upd after the replay - -comment,,,,,,,unsubscribe closes the residual liveness gap - MEASURED: a handle the caller hcloses fires NO .z.pc, and kdb+ reissues the freed descriptor -run,0,0,q,relph:sub.unsubscribe[PH2],1,1,declare we are finished with the handle BEFORE closing it -true,0,0,q,(enlist`peertrade)~relph,1,1,it reports the table it released -true,0,0,q,not any exec active from sub.getsubscriptions[] where handle~\:PH2,1,1,the row is dead the instant we say so - no .z.pc and no dead tickerplant involved -run,0,0,q,hclose PH2,1,1,now close the handle ourselves - the tickerplant is still alive so nothing fires .z.pc -run,0,0,q,PH3:peerhandle[],1,1,open a fresh connection to the same live tickerplant -true,0,0,q,PH2~PH3,1,1,kdb+ reissued the freed descriptor - this is the case the .z.W half alone cannot see through -true,0,0,q,not any exec active from sub.getsubscriptions[] where handle~\:PH3,1,1,the stale row still reports dead - without unsubscribe the reissued number would have revived it -run,0,0,q,sub.subscribe[PH3;`;`;0b;0b],1,1,and a fresh subscribe on the reissued handle is allowed rather than refused by the guard -run,0,0,q,sub.unsubscribe[PH3],1,1,release it -run,0,0,q,hclose PH3,1,1,tidy up - -comment,,,,,,,STORED vs EFFECTIVE - unsubscribe must find a row whose handle is already closed. selecting on the effective flag would miss exactly this row -run,0,0,q,n4:count sub.getsubscriptions[],1,1,record the registry size (earlier dead rows share this descriptor number so counts must be deltas) -run,0,0,q,d4:sum not exec active from sub.getsubscriptions[],1,1,and the dead-row count - there IS at least one dead row on this descriptor so the delta below has teeth -run,0,0,q,PH4:peerhandle[],1,1,open another handle to the still-live tickerplant -run,0,0,q,sub.subscribe[PH4;`;`;0b;0b],1,1,subscribe on it -true,0,0,q,(n4+1)=count sub.getsubscriptions[],1,1,one new row -true,0,0,q,1=sum exec active from sub.getsubscriptions[] where handle~\:PH4,1,1,and it is live -run,0,0,q,hclose PH4,1,1,close it WITHOUT unsubscribing - .z.W loses the handle but the STORED flag is still 1b -true,0,0,q,not any exec active from sub.getsubscriptions[] where handle~\:PH4,1,1,it now reports dead through the .z.W half alone -true,0,0,q,(n4+1)=count sub.getsubscriptions[],1,1,but the row is still present carrying its stored flag - the row a reissued descriptor would revive -run,0,0,q,rel4:sub.unsubscribe[PH4],1,1,unsubscribe AFTER the close must still remove it -true,0,0,q,(enlist`peertrade)~rel4,1,1,it found the row even though the effective flag read dead -true,0,0,q,n4=count sub.getsubscriptions[],1,1,and deleted it - selecting on activesubscriptions would have warned nothing to release and left it behind -true,0,0,q,d4=sum not exec active from sub.getsubscriptions[],1,1,every markdead row survived - unsubscribe deletes only what it released, which is the whole asymmetry - -comment,,,,,,,di.handlers ISOLATION - a co-registrant that throws AHEAD of our observer must not suppress markdead. this is a cross-module contract nothing about editing this module would prompt anyone to re-check -run,0,0,q,`tailfired set 0b,1,1,reset the tail flag -run,0,0,q,"realhandlers[`register][`.z.pc;`;`noisyneighbour;-1;{[w] '""neighbour exploded""}]",1,1,a throwing observer at priority -1 - MEASURED to run before ours, which init registers at 0j -run,0,0,q,realhandlers[`register][`.z.pc;`;`tailobserver;9;{[w] `tailfired set 1b}],1,1,and a quiet one behind ours at priority 9 -true,0,0,q,{(x?`noisyneighbour)/dev/null || true";{[e] :(::)}]; - }; - -isfree:{[p] :not @[{hclose hopen x;1b};(`$":localhost:",string p;100);{[e] :0b}]; }; -pickport:{[start] :first (start+til 500) where isfree each start+til 500; }; -waitlisten:{[port;timeoutms] - deadline:.z.p+`timespan$1000000*timeoutms; - while[(.z.p/dev/null 2>&1 &"; - if[not waitlisten[PEERPORT;5000];'"test: tickerplant peer failed to listen on ",string PEERPORT]; - h:hopen (`$":localhost:",string PEERPORT;2000); - / prove this is OUR peer. waitlisten only proves SOMETHING is listening, and pickport probes each - / candidate with a 100ms connect - under load an occupied port can read as free, our peer then - / fails to bind, and the suite would silently run against a peer left over from an aborted run - / whose log files the rm -rf above has just deleted. fail loudly instead of testing the wrong process - if[not PEERTOKEN~@[h;"PEERTOKEN";{[e] `}]; - hclose h; - '"test: port ",string[PEERPORT]," is served by a foreign process, not this run's peer"]; - `PEERPID set h ".z.i"; - hclose h; - }; - -peerhandle:{[] :hopen (`$":localhost:",string PEERPORT;2000); }; -noticedrop:{[h] - / a dead peer is only detected when we next touch the handle; that is what fires .z.pc - @[{x "1+1"};h;{[e] :(::)}]; - system "sleep 0.3"; - }; - -flushpeer:{[h] - / force this process to read the async messages the peer pushed at us. a synchronous round trip - / makes q service the incoming queue, so the live upd calls land before we assert on them - h "1+1"; - system "sleep 0.2"; - h "1+1"; - }; -killpeer:{[] if[not null PEERPID;@[system;"kill ",string PEERPID;{[e] :(::)}]]; `PEERPID set 0N; system "sleep 0.3"; }; - -setupfixture:{[] system "rm -rf ",BASE; system "mkdir -p ",BASE; }; -teardownfixture:{[] killpeer[]; killstalepeers[]; system "rm -rf ",BASE; system "rm -rf ",PEERDIR; }; diff --git a/di/subscriptions/test_integration.csv b/di/subscriptions/test_integration.csv new file mode 100644 index 00000000..19f1f9e0 --- /dev/null +++ b/di/subscriptions/test_integration.csv @@ -0,0 +1,149 @@ +action,ms,bytes,lang,code,repeat,minver,comment +comment,,,,,,,integration suite - stands up a real q process as a tickerplant and drives it over genuine IPC +comment,,,,,,,it is separate because moduletest only ever loads test.csv (di/k4unit/init.q), so keeping these +comment,,,,,,,rows there would make every unit run spawn a process and bind a port. load this file explicitly +comment,,,,,,,needs a q binary reachable via QHOME - the peer is launched with $QHOME/bin/q +comment,,,,,,,run it in a FRESH q session: after moduletest the unit tests are still loaded and would re-run against dirty module state + +before,0,0,q,sub:use`di.subscriptions,1,1,load di.subscriptions +before,0,0,q,"BASE:""/tmp/disubscriptionstest"";",1,1,fixture: BASE +before,0,0,q,D:2026.08.06;,1,1,fixture: D +before,0,0,q,caprows:([]lvl:`symbol$();ctx:`symbol$();msg:());,1,1,fixture: caprows +before,0,0,q,resetcap:{[] `caprows set 0#caprows; };,1,1,fixture: resetcap +before,0,0,q,caplog:`info`warn`error!( {[c;m] `caprows insert (`info;c;m)}; {[c;m] `caprows insert (`warn;c;m)}; {[c;m] `caprows insert (`error;c;m)});,1,1,fixture: caplog +before,0,0,q,realhandlers:use`di.handlers;,1,1,fixture: realhandlers +before,0,0,q,handlerdep:{[] :`register`remove!(realhandlers[`register];realhandlers[`remove]); };,1,1,fixture: handlerdep +before,0,0,q,deps:{[] :`log`handlers!(caplog;handlerdep[]); };,1,1,fixture: deps +before,0,0,q,"logged:{[lv;s] :any {[s;m] 0/dev/null || true"";{[e] :(::)}]; };",1,1,fixture: killstalepeers +before,0,0,q,"isfree:{[p] :not @[{hclose hopen x;1b};(`$"":localhost:"",string p;100);{[e] :0b}]; };",1,1,fixture: isfree +before,0,0,q,pickport:{[start] :first (start+til 500) where isfree each start+til 500; };,1,1,fixture: pickport +before,0,0,q,"waitlisten:{[port;timeoutms] deadline:.z.p+`timespan$1000000*timeoutms; while[(.z.p/dev/null 2>&1 &""; if[not waitlisten[PEERPORT;5000];'""test: tickerplant peer failed to listen on "",string PEERPORT]; h:hopen (`$"":localhost:"",string PEERPORT;2000); if[not PEERTOKEN~@[h;""PEERTOKEN"";{[e] `}]; hclose h; '""test: port "",string[PEERPORT],"" is served by a foreign process, not this run's peer""]; `PEERPID set h "".z.i""; hclose h; };",1,1,fixture: spawnpeer +before,0,0,q,"peerhandle:{[] :hopen (`$"":localhost:"",string PEERPORT;2000); };",1,1,fixture: peerhandle +before,0,0,q,"noticedrop:{[h] @[{x ""1+1""};h;{[e] :(::)}]; system ""sleep 0.3""; };",1,1,fixture: noticedrop +before,0,0,q,"flushpeer:{[h] h ""1+1""; system ""sleep 0.2""; h ""1+1""; };",1,1,fixture: flushpeer +before,0,0,q,"killpeer:{[] if[not null PEERPID;@[system;""kill "",string PEERPID;{[e] :(::)}]]; `PEERPID set 0N; system ""sleep 0.3""; };",1,1,fixture: killpeer +before,0,0,q,peerbody:(),1,1,"the peer q script, one complete line per row - a split literal would silently become several expressions" +before,0,0,q,"peerbody,:enlist ""LF:hsym`$\"""",PEERDIR,""/tp\"",string D;""",1,1,peer source line +before,0,0,q,"peerbody,:enlist ""peerschema:([]time:`timestamp$();sym:`g#`symbol$();price:`float$();size:`int$());""",1,1,peer source line +before,0,0,q,"peerbody,:enlist ""N:"",string PEERMSGS",1,1,peer source line +before,0,0,q,"peerbody,:enlist ""cols4:{[i] (enlist 2026.08.06D10:00+`timespan$60000000000*i;enlist `$\""S\"",string i mod 4;""",1,1,peer source line +before,0,0,q,"peerbody,:enlist "" enlist 1.0*i;enlist `int$10*i)};""",1,1,peer source line +before,0,0,q,"peerbody,:enlist ""mk:{[i] (`upd;`peertrade;cols4 i)};""",1,1,peer source line +before,0,0,q,"peerbody,:enlist ""lh:hopen LF set ();""",1,1,peer source line +before,0,0,q,"peerbody,:enlist ""{[h;i] h enlist mk i}[lh] each til N;""",1,1,peer source line +before,0,0,q,"peerbody,:enlist ""subs:();""",1,1,peer source line +before,0,0,q,"peerbody,:enlist ""sd:{[tabs;instruments] subs::distinct subs,.z.w;""",1,1,peer source line +before,0,0,q,"peerbody,:enlist "" `schemalist`logfilelist`rowcounts`date!(enlist(`peertrade;peerschema);""",1,1,peer source line +before,0,0,q,"peerbody,:enlist "" enlist(N;LF);(enlist `peertrade)!enlist N;D)};""",1,1,peer source line +before,0,0,q,"peerbody,:enlist ""subdetails:sd;""",1,1,peer source line +before,0,0,q,"peerbody,:enlist ""tablelist:{enlist`peertrade};""",1,1,peer source line +before,0,0,q,"peerbody,:enlist ""pub:{[i;s] m:(`upd;`peertrade;(enlist 2026.08.06D11:00+`timespan$1000000000*i;enlist s;""",1,1,peer source line +before,0,0,q,"peerbody,:enlist "" enlist 1.0*i;enlist `int$i)); lh enlist m; N+:1; {[m;w] (neg w) m}[m] each subs; N};""",1,1,peer source line +before,0,0,q,"peerbody,:enlist ""logged:{[] N};""",1,1,peer source line +before,0,0,q,"peerlines:{[] :(""D:2026.08.06;"";""PEERTOKEN:`"",(string PEERTOKEN),"";""),peerbody; };",1,1,prepend the per-run identity token - it is only known once spawnpeer has generated it +before,0,0,q,"teardownfixture:{[] killpeer[]; killstalepeers[]; system ""rm -rf "",BASE; system ""rm -rf "",PEERDIR; };",1,1,fixture: teardownfixture +before,0,0,q,setupfixture[],1,1,create the temp directories +before,0,0,q,realhandlers[`init][enlist[`log]!enlist caplog],1,1,init the real di.handlers - register throws until it has been +before,0,0,q,sub.init[deps[]],1,1,init di.subscriptions - this suite runs standalone so it does all of its own setup +before,0,0,q,resetupd[],1,1,define the root upd the replay drives + +run,0,0,q,spawnpeer[],1,1,launch a real tickerplant peer that writes its own log and serves subdetails +run,0,0,q,PH:peerhandle[],1,1,open a real IPC handle to it +true,0,0,q,PEERPID<>.z.i,1,1,the peer really is a separate process +true,0,0,q,`schemalist`logfilelist`rowcounts`date~key PH(`subdetails;`;`),1,1,the peer serves the real TorQ subdetails protocol +run,0,0,q,resetupd[],1,1,reset the upd counter +comment,,,,,,,an all-tables subscribe over the real handle - the registry starts empty in this suite +run,0,0,q,rp:sub.subscribe[PH;`;`;1b;1b],1,1,subscribe over the real handle +true,0,0,q,`peertrade in tables[],1,1,the table was created at root from the schema sent over IPC +true,0,0,q,6=count peertrade,1,1,all six logged messages replayed +true,0,0,q,`g=attr peertrade`sym,1,1,the g# attribute survived the IPC round trip +true,0,0,q,6=updcalls,1,1,the replay drove the root upd once per message +true,0,0,q,PH~first exec handle from sub.getsubscriptions[] where handle~\:PH,1,1,the registry holds the real int handle +true,0,0,q,any exec active from sub.getsubscriptions[] where handle~\:PH,1,1,the subscription over the real handle is live + +comment,,,,,,,kill the tickerplant - the .z.pc observer must mark the subscription dead +run,0,0,q,killpeer[],1,1,kill the peer process +run,0,0,q,noticedrop[PH],1,1,touch the handle so the drop is detected and .z.pc fires +true,0,0,q,not any exec active from sub.getsubscriptions[] where handle~\:PH,1,1,the row for the dropped handle was marked dead + +comment,,,,,,,restart and reconnect - kdb+ recycles the handle number and the stale row must NOT revive +run,0,0,q,spawnpeer[],1,1,restart the tickerplant +run,0,0,q,PH2:peerhandle[],1,1,reconnect +true,0,0,q,PH~PH2,1,1,kdb+ handed back the same handle number (the recycling a .z.W probe alone cannot see through) +true,0,0,q,not any exec active from sub.getsubscriptions[] where handle~\:PH2,1,1,the stale row stays dead despite the number being reused +run,0,0,q,resetupd[],1,1,reset the upd counter +run,0,0,q,@[`.;`peertrade;:;0#peertrade],1,1,clear the table before replaying again +run,0,0,q,sub.subscribe[PH2;`;`;1b;1b],1,1,re-subscribing after a reconnect is allowed +true,0,0,q,6=count peertrade,1,1,the log replayed again on the new handle +true,0,0,q,2=count select from sub.getsubscriptions[] where handle~\:PH2,1,1,both the stale and the new row are for the recycled handle number +true,0,0,q,1=sum exec active from sub.getsubscriptions[] where handle~\:PH2,1,1,exactly one of them is live - the stale one stayed dead + +comment,,,,,,,LIVE updates after the replay - the exactly-once boundary. this is the module's central promise and cannot be tested without a publishing tickerplant +true,0,0,q,6=updcalls,1,1,the replay drove upd once per logged message and no more +true,0,0,q,6=PH2"logged[]",1,1,the tickerplant has logged six messages so far +run,0,0,q,(neg PH2)(`pub;100;`S0),1,1,the tickerplant logs AND publishes one live message +run,0,0,q,flushpeer[PH2],1,1,round trip so the async live message is serviced +true,0,0,q,7=PH2"logged[]",1,1,the tickerplant logged the live message too +true,0,0,q,7=count peertrade,1,1,six replayed plus one live - the boundary message was NOT replayed as well +true,0,0,q,7=updcalls,1,1,upd ran exactly once per message across replay and live feed +true,0,0,q,100f=last exec price from peertrade,1,1,the live message is the most recent row +run,0,0,q,(neg PH2)(`pub;101;`S1),1,1,publish a second live message +run,0,0,q,flushpeer[PH2],1,1,service it +true,0,0,q,8=count peertrade,1,1,live updates keep flowing through the root upd after the replay + +comment,,,,,,,unsubscribe closes the residual liveness gap - MEASURED: a handle the caller hcloses fires NO .z.pc, and kdb+ reissues the freed descriptor +run,0,0,q,relph:sub.unsubscribe[PH2],1,1,declare we are finished with the handle BEFORE closing it +true,0,0,q,(enlist`peertrade)~relph,1,1,it reports the table it released +true,0,0,q,not any exec active from sub.getsubscriptions[] where handle~\:PH2,1,1,the row is dead the instant we say so - no .z.pc and no dead tickerplant involved +run,0,0,q,hclose PH2,1,1,now close the handle ourselves - the tickerplant is still alive so nothing fires .z.pc +run,0,0,q,PH3:peerhandle[],1,1,open a fresh connection to the same live tickerplant +true,0,0,q,PH2~PH3,1,1,kdb+ reissued the freed descriptor - this is the case the .z.W half alone cannot see through +true,0,0,q,not any exec active from sub.getsubscriptions[] where handle~\:PH3,1,1,the stale row still reports dead - without unsubscribe the reissued number would have revived it +run,0,0,q,sub.subscribe[PH3;`;`;0b;0b],1,1,and a fresh subscribe on the reissued handle is allowed rather than refused by the guard +run,0,0,q,sub.unsubscribe[PH3],1,1,release it +run,0,0,q,hclose PH3,1,1,tidy up + +comment,,,,,,,STORED vs EFFECTIVE - unsubscribe must find a row whose handle is already closed. selecting on the effective flag would miss exactly this row +run,0,0,q,n4:count sub.getsubscriptions[],1,1,record the registry size (earlier dead rows share this descriptor number so counts must be deltas) +run,0,0,q,d4:sum not exec active from sub.getsubscriptions[],1,1,and the dead-row count - there IS at least one dead row on this descriptor so the delta below has teeth +run,0,0,q,PH4:peerhandle[],1,1,open another handle to the still-live tickerplant +run,0,0,q,sub.subscribe[PH4;`;`;0b;0b],1,1,subscribe on it +true,0,0,q,(n4+1)=count sub.getsubscriptions[],1,1,one new row +true,0,0,q,1=sum exec active from sub.getsubscriptions[] where handle~\:PH4,1,1,and it is live +run,0,0,q,hclose PH4,1,1,close it WITHOUT unsubscribing - .z.W loses the handle but the STORED flag is still 1b +true,0,0,q,not any exec active from sub.getsubscriptions[] where handle~\:PH4,1,1,it now reports dead through the .z.W half alone +true,0,0,q,(n4+1)=count sub.getsubscriptions[],1,1,but the row is still present carrying its stored flag - the row a reissued descriptor would revive +run,0,0,q,rel4:sub.unsubscribe[PH4],1,1,unsubscribe AFTER the close must still remove it +true,0,0,q,(enlist`peertrade)~rel4,1,1,it found the row even though the effective flag read dead +true,0,0,q,n4=count sub.getsubscriptions[],1,1,and deleted it - selecting on activesubscriptions would have warned nothing to release and left it behind +true,0,0,q,d4=sum not exec active from sub.getsubscriptions[],1,1,every markdead row survived - unsubscribe deletes only what it released, which is the whole asymmetry + +comment,,,,,,,di.handlers ISOLATION - a co-registrant that throws AHEAD of our observer must not suppress markdead. this is a cross-module contract nothing about editing this module would prompt anyone to re-check +run,0,0,q,`tailfired set 0b,1,1,reset the tail flag +run,0,0,q,"realhandlers[`register][`.z.pc;`;`noisyneighbour;-1;{[w] '""neighbour exploded""}]",1,1,a throwing observer at priority -1 - MEASURED to run before ours, which init registers at 0j +run,0,0,q,realhandlers[`register][`.z.pc;`;`tailobserver;9;{[w] `tailfired set 1b}],1,1,and a quiet one behind ours at priority 9 +true,0,0,q,{(x?`noisyneighbour) Date: Wed, 12 Aug 2026 20:39:29 +0100 Subject: [PATCH 05/10] port getsubscriptionhandles, make di.servers and di.pubsub hard deps, add republish --- di/subscriptions/VERSION | 2 +- di/subscriptions/deps.q | 35 ++++--- di/subscriptions/init.q | 13 ++- di/subscriptions/subscriptions.md | 98 +++++++++++++++++--- di/subscriptions/subscriptions.q | 88 +++++++++++++++++- di/subscriptions/test.csv | 127 +++++++++++++++++++++++++- di/subscriptions/test_integration.csv | 117 ++++++++++++++++++++---- 7 files changed, 430 insertions(+), 50 deletions(-) diff --git a/di/subscriptions/VERSION b/di/subscriptions/VERSION index 6c6aa7cb..341cf11f 100644 --- a/di/subscriptions/VERSION +++ b/di/subscriptions/VERSION @@ -1 +1 @@ -0.1.0 \ No newline at end of file +0.2.0 \ No newline at end of file diff --git a/di/subscriptions/deps.q b/di/subscriptions/deps.q index 31cf2218..0012f069 100644 --- a/di/subscriptions/deps.q +++ b/di/subscriptions/deps.q @@ -1,16 +1,21 @@ / hard module dependencies and their minimum versions, validated by di.depcheck -/ di.subscriptions has NO hard dependencies: -/ - log and handlers are injected via init as dictionaries of functions. handlers is required for -/ the .z.pc observer that marks a dropped connection's subscriptions dead: .z.W alone cannot do -/ it, because kdb+ recycles handle numbers and a reused number revives a stale row -/ - handle resolution is the CALLER's job (di.rdb/di.wdb obtain a tickerplant handle from -/ di.servers.gethandlebytype and pass it in), so there is no di.servers edge - the plan's -/ dependency tree lists one, but every caller already imports di.servers directly and a -/ getsubscriptionhandles wrapper here would only duplicate it -/ - di.pubsub is the PUBLISHER side (TorQ's .stpps - the tickerplant's own subscriber registry). -/ a subscribing process never calls it; legacy .sub never referenced it either. the plan's -/ tree lists it for this module in error -/ - the tp log is replayed with kdb+'s native -11!, not via di.tplog: this module replays the -/ FIRST n messages, and di.tplog.check is built for the replay-everything caller -/ (tickerlogreplay.q, lastmessage 0W). see subscriptions.md for the full reasoning -deps:(`$())!(); +/ the modularisation plan places di.subscriptions in the FRAMEWORK tier with +/ `-> di.servers, di.pubsub`. both are real module imports, `use`d in init.q: +/ - di.servers: getsubscriptionhandles - the port of TorQ's .sub.getsubscriptionhandles +/ (subscriptions.q:11) - resolves a tickerplant handle by proctype/procname off +/ di.servers.SERVERS, calling only `getservers` +/ - di.pubsub: the LOCAL publisher this process republishes through. a chained or segmented +/ tickerplant subscribes upstream and serves the same tables downstream (TorQ chainedtp.q:71-82 +/ and sctp.q:15-25 do exactly this, then publish via .ps.publish and serve their table list from +/ the pubsub registry, chainedtp.q:7). with the republish config key set, subscribe registers +/ the tables it defined at root with di.pubsub so it can extract their schemas and fan out +/ log and handlers stay INJECTED via init as dictionaries of functions - the plan's tier table +/ excludes logging, timer and handler management from the hard dependency tree by design +/ the tp log is replayed with kdb+'s native -11!, not via di.tplog: this module replays the FIRST n +/ messages, and di.tplog.check is built for the replay-everything caller (tickerlogreplay.q, +/ lastmessage 0W). see subscriptions.md for the full reasoning +/ NB the di.pubsub minimum is 0.2.0, not 0.1.0: this module needs BOTH of that release's changes. +/ getsubtables (the handoff reads the current publish set back before adding to it) and the .z.pc +/ CHAINING fix - a 0.1.0 di.pubsub replaces .z.pc at load and would silently destroy this module's +/ own dropped-connection observer, which is the failure the handlers dependency exists to prevent +deps:`di.servers`di.pubsub!("0.1.0";"0.2.0"); diff --git a/di/subscriptions/init.q b/di/subscriptions/init.q index ed61a242..1a94e012 100644 --- a/di/subscriptions/init.q +++ b/di/subscriptions/init.q @@ -3,6 +3,17 @@ / exactly once, then let live updates flow through the root upd. / ported from TorQ's code/common/subscriptions.q (.sub). the caller owns the connection. +/ hard dependencies, per the modularisation plan's FRAMEWORK tier (di.subscriptions -> di.servers, +/ di.pubsub). imported before the implementation is loaded, matching di.eodtime's `use` of di.tz. +/ di.servers - getsubscriptionhandles resolves tickerplant handles off di.servers.SERVERS +/ di.pubsub - the LOCAL publisher a chained or segmented tickerplant republishes through: with +/ republish set, subscribe hands the tables it defined at root to it, so this process +/ can serve them downstream (TorQ chainedtp.q:71-82, sctp.q:15-25) +/ NB the `use` runs at module LOAD, before init registers the .z.pc observer. that ordering is +/ load-bearing - see subscriptions.md +servers:use`di.servers +pubsub:use`di.pubsub + \l ::subscriptions.q / module version, read from the VERSION file rather than hardcoded in the implementation, so a @@ -14,4 +25,4 @@ version:first read0`:::VERSION / NB: export:([...]) EVALUATES each name, so it can only list names that already exist. / init and getapimeta are framework plumbing di.torq calls by convention; every other name here has / a getapimeta row, which the test suite asserts -export:([init;teardown;version;getapimeta;subscribe;resubscribe;unsubscribe;subscribed;getsubscriptions]) +export:([init;teardown;version;getapimeta;subscribe;resubscribe;unsubscribe;subscribed;getsubscriptions;getsubscriptionhandles]) diff --git a/di/subscriptions/subscriptions.md b/di/subscriptions/subscriptions.md index e59fa755..12b9ba7d 100644 --- a/di/subscriptions/subscriptions.md +++ b/di/subscriptions/subscriptions.md @@ -28,11 +28,39 @@ Tickerplant subscription management for kdb+ subscriber processes (RDB, WDB, cha | logger | `` `log `` | yes | `info`, `warn` and `error` - each binary `{[c;m]}` where `c` is a symbol context and `m` is a string. All three are called by this module | | handlers | `` `handlers `` | yes | `register` and `remove`, per the `di.handlers` contract. Used to install a `.z.pc` observer that marks a dropped connection's subscriptions dead | -**Hard dependencies:** none. Both dependencies are injected, so the module imports no other `di.*` module. +**Hard dependencies.** The modularisation plan places `di.subscriptions` in the **FRAMEWORK** tier with `-> di.servers, di.pubsub`. Both are genuine module imports, `use`d in `init.q` before the implementation loads (the shape `di.eodtime` uses for `di.tz`), declared in `deps.q` and enforced by `di.depcheck` (`di.servers` ≥ `0.1.0`, `di.pubsub` ≥ **`0.2.0`** — that release adds `getsubtables` and, critically, makes `di.pubsub` *chain* `.z.pc` instead of replacing it; a `0.1.0` `di.pubsub` would silently destroy this module's own dropped-connection observer). The module will not load without them. -Both deps must be passed to `init` inside the `deps` dict. The module throws immediately if either is absent or malformed - there is no fallback logger and no degraded no-handlers mode. The `log` dict must already match the binary `{[c;m]}` contract; the module does not detect or adapt other shapes (e.g. a raw `kx.log` instance, which is monadic). To use `di.log`, pass its `logdict``log`. +| Import | Used for | +|---|---| +| `di.servers` | `getsubscriptionhandles` resolves tickerplant handles off `di.servers.SERVERS`, calling only `getservers` | +| `di.pubsub` | the **local** publisher this process republishes through — see `republish` below | -Handle resolution is the **caller's** job. `di.rdb` and `di.wdb` obtain a tickerplant handle from `di.servers.gethandlebytype` and pass it in, so there is no `di.servers` dependency here. +`log` and `handlers` must be passed to `init` inside the `deps` dict. The module throws immediately if either is absent or malformed - there is no fallback logger and no degraded no-handlers mode. The `log` dict must already match the binary `{[c;m]}` contract; the module does not detect or adapt other shapes (e.g. a raw `kx.log` instance, which is monadic). To use `di.log`, pass its `logdict``log`. + +**`republish` (default `0b`) turns on the `di.pubsub` half.** A chained or segmented tickerplant subscribes upstream and serves the *same* tables downstream — TorQ splits that across `chainedtp.q:71-82` and `sctp.q:15-25`, which subscribe, then publish through `.ps.publish` and serve their own table list straight out of the pubsub registry (`chainedtp.q:7`, `tablelist:{.stpps.t}`). With `republish` set, a successful `subscribe` hands the tables it defined at root to the local `di.pubsub` (`setsubtables` then `init`), so it extracts their schemas and can fan out to this process's own subscribers. + +It is **off by default**: a plain RDB or WDB consumes a feed and must not silently start publishing one. + +The handoff is **additive** — it unions the registry's tables with what `di.pubsub` already serves (read back via `getsubtables`), so the published set only ever grows within a process. Two measured reasons: + +- `di.pubsub`'s `setsubtables` **replaces** its list (`pubsub.q:119`), and `unsubscribe` deletes its registry rows. A union computed purely from the registry therefore dropped an unsubscribed table at the next *unrelated* `subscribe` — downstream subscribers silently stopped receiving it, at a moment disconnected from the `unsubscribe` that caused it. +- Shrinking toward empty is worse than useless: `setsubtables` with an empty list does **not** mean "publish nothing". `di.pubsub` then falls back to publishing **every** table at root (`pubsub.q:125`). + +**What `republish` does and does not do.** It makes the local publisher *able to serve* the subscribed tables — schemas extracted, ready for a downstream `subscribe`. It does **not** forward the data: that is the process's own `upd`, exactly as in TorQ, where `chainedtp.q`'s `tickpub` (`:96-99`) is chainedtp's code, not `.sub`'s. A chained tickerplant built on this module therefore wires one line: + +```q +upd:{[t;x] @[`.;t;{[tab;d] tab upsert $[98h=type d;d;flip (cols tab)!d]}[;x]]; pubsub.publish[t;x]} +``` + +That split is deliberate: this module owns the *subscription*, the caller owns what it does with each message. The integration suite proves the whole chain across three real processes — an upstream tickerplant, a middle process using this module with `republish` on, and a downstream subscriber. + +Legacy's chained tickerplants also seed `.u.d`/`.u.icounts` from the subscribe result. Those are tickerplant sequence-and-date state, which `di.pubsub` does not own (it has no `i`, `j`, `icounts` or `d`); the values are returned to the caller in `subscribe`'s result as `d` and `icounts` so whatever owns a tickerplant log can apply them. + +A table that is unsubscribed stays advertised and simply stops receiving data — a visible, inert condition rather than a silent disappearance. Tables not defined at root are filtered out first, since `di.pubsub.init` calls `value` on each name (`pubsub.q:84`). + +Note the module still speaks the publisher's **wire** protocol separately: `subdetailsfunc` and `tablelistfunc` name functions evaluated on the *remote* tickerplant, so they resolve in that process. Legacy is the same shape — its `.u.sub`/`.u.i`/`.u.L`/`.u.icounts`/`.u.d` references (`subscriptions.q:100-101`) sit inside lambdas sent to the tickerplant handle. The integration suite's peer runs the real `di.pubsub`, so that half is exercised across a genuine process boundary. + +**⚠️ Import order is load-bearing.** `di.pubsub` assigns `.z.pc` at load (`pubsub.q:75`). It now *chains* onto whatever already owns the event, but the `use` in `init.q` runs at module load and `init` registers this module's `.z.pc` observer afterwards — that ordering is what keeps the observer alive, and it is asserted in `test.csv` rather than left to comment. The configuration keys `subdetailsfunc` and `tablelistfunc` are optional - omit them and the module calls the tickerplant's `subdetails` and `tablelist`. See Initialisation. @@ -49,6 +77,7 @@ The configuration keys `subdetailsfunc` and `tablelistfunc` are optional - omit | `` `subdetailsfunc `` | no | Symbol naming the tickerplant-side function to call. Default: `` `subdetails `` | | `` `tablelistfunc `` | no | Symbol naming the tickerplant-side function that lists the available tables, used to resolve a `` ` `` request into a concrete list. Default: `` `tablelist `` | | `` `unsubscribefunc `` | no | Symbol naming a tickerplant-side function that releases this connection's subscriptions, called by `unsubscribe`. Default: `` ` `` (none) — see Notes | +| `` `republish `` | no | Boolean. Hand the subscribed tables to the local `di.pubsub` so this process can serve them downstream — the chained/segmented tickerplant role. Default: `0b` | `init` must be called before any operational function - `subscribe`, `unsubscribe`, `subscribed`, `getsubscriptions` and `teardown` each throw a clear error if it has not been. `getapimeta` and `version` are metadata and deliberately work without it, so `di.torq` can collect api rows and `di.depcheck` can read the version before anything is initialised. `init` registers a `.z.pc` observer through `di.handlers` and is idempotent - a second call refreshes the dependencies and replaces the registration in place rather than duplicating it, and leaves the registry intact. @@ -116,6 +145,33 @@ sub.getsubscriptions[] ``` `tabs` and `syms` are general columns, so symbols display without backticks and an all-syms subscription shows an empty `syms`. +### `getsubscriptionhandles[proctype;procname]` +Resolve live tickerplant handles by proctype and/or procname, projected to the `procname`, `proctype`, `w` triple a caller needs before it can subscribe. Reads `di.servers`, this module's one hard dependency. Ported from legacy `.sub.getsubscriptionhandles` (`code/common/subscriptions.q:11`), which is registered public api there (`apidetails.q:67`) and called by `rdb.q:163`, `wdb.q:540`, `chainedtp.q:71` and `sctp.q:15`. + +```q +sub.getsubscriptionhandles[`tickerplant;()] / rdb / wdb shape - by proctype +/ procname proctype w +/ ---------------------- +/ tp1 tickerplant 4 + +sub.getsubscriptionhandles[`;`tp1] / chained / segmented tp shape - by procname +``` + +The two arguments are **not interchangeable**, and this is the whole of the function's behaviour: + +| Argument value | Meaning | +|---|---| +| `` ` `` | match **every** row | +| `()` | match **nothing**, and switch the combine from intersection to union | +| symbol or symbol list | match those values | + +So `[`tickerplant;()]` unions the proctype matches with nothing and yields the tickerplants, while `` [`;`tp1] `` intersects every row with the named one and yields just that process. Only live connections are ever returned — `di.servers` excludes null handles. + +**Two deliberate differences from legacy**, both forced by `di.servers`' contract: + +- Legacy took a third `attributes` argument and filtered on `.servers.SERVERS`'s `attributes` column. `di.servers`' `SERVERS` carries no such column, so the parameter is **dropped** rather than accepted and silently ignored — a filter that quietly does nothing hands back handles the caller believes were filtered. All four legacy call sites pass `()!()`. +- Legacy passed `autoopen:1b` to retry dead connections on demand. `di.servers` returns live rows only and runs its own retry job, so reconnection is its concern now. + ### `teardown[]` Remove the `.z.pc` registration installed by `init`, leaving no process-global residue. Afterwards `subscribe` is refused until `init` is called again — see Notes. `getsubscriptions`, `subscribed` and `unsubscribe` keep working, so a shutdown path can still inspect and release what it holds, and the registry is left intact. ```q @@ -156,11 +212,14 @@ sub.init[`log`handlers!(logging.logdict`log;handlerdep)] / part way through a sym-filtered replay upd:{[t;x] @[`.;t;{[tab;d] tab upsert $[98h=type d;d;flip (cols tab)!d]}[;x]];} -/ the caller resolves the tickerplant handle. NB di.servers is not merged yet (feature-server), so -/ on main today obtain the handle however the process already does - any open handle works +/ the caller resolves the tickerplant handle. the usual route is di.servers directly: servers:use`di.servers tph:servers.gethandlebytype[`tickerplant;`any] +/ or, having passed di.servers in as the `servers dep, through this module - the legacy +/ .sub.getsubscriptionhandles route the TorQ rdb and wdb take: +/ tph:first exec w from sub.getsubscriptionhandles[`tickerplant;()] + / subscribe to everything and replay the log r:sub.subscribe[tph;`;`;1b;1b] r`subtables / the tables now defined at root and receiving live updates @@ -184,15 +243,32 @@ Two suites, following the convention `di.handlers` and `di.permissions` use. Bot fixtures inline as `before` rows; there is no separate fixture file, so neither depends on the working directory. -**Unit suite** (`test.csv`) - 450 assertions, no child processes, no ports, no `QHOME`: +**Unit suite** (`test.csv`) - 529 assertions, no child processes, no ports, no `QHOME`. It does need `di.servers` on `QPATH`, since the module imports it, and it drives the **real** `di.servers` rather than a mock: the fixtures seed its `SERVERS` registry directly, so the whole selection matrix runs against the genuine `getservers` without opening a socket: ```q k4unit:use`di.k4unit k4unit.moduletest`di.subscriptions ``` -**Integration suite** (`test_integration.csv`) - 73 assertions. It stands up a real q process as a -tickerplant and drives it over genuine IPC, so it needs a q binary reachable via `QHOME`. +**Integration suite** (`test_integration.csv`) - 131 assertions. It stands up a real q process as a +tickerplant and drives it over genuine IPC, so it needs a q binary reachable via `QHOME`, plus +`di.pubsub` and `di.servers` on `QPATH` - the two modules the dependency tree names, both exercised +here as real modules rather than mocks. + +It runs up to **three** processes at once for the `republish` block: an upstream tickerplant, a +middle process that subscribes through this module with `republish` on, and this test process acting +as the downstream subscriber. That is the only way to prove the chained-tickerplant role - a +subscriber that serves its own subscribers has to be at its own main loop, so it cannot be tested +in-process. It does NOT need `di.timer`: `di.servers` validates a timer +dict and schedules one retry job through it, which a stub satisfies, and this suite asserts nothing +about that job. + +The peer is built on **`di.pubsub`** — it loads the real publisher, registers subscriptions through +`di.pubsub.subscribe` and fans out through `di.pubsub.publish`, with only a thin `subdetails` +adapter on top of the kind a modular tickerplant process would own. So the publisher half of the +wire protocol is exercised by the actual module rather than a hand-rolled stand-in, including +sym-filtered delivery, which `di.pubsub` decides. Assertions read `di.pubsub`'s own registry on the +peer (`reqalldict`, `reqfilteredtbl`) so this cannot silently regress into a fake. `moduletest` only ever loads `test.csv` (`di/k4unit/init.q`), which is why these rows live in their own file - otherwise every unit run would spawn a process and bind a port. Load it explicitly: @@ -211,7 +287,7 @@ loaded and would re-run against dirty module state, reporting spurious failures. self-contained: it inits `di.handlers` and `di.subscriptions` itself rather than inheriting setup from the unit rows. -Together the two suites are 523 assertions (`true`, `fail` and `run` rows, excluding the fixture `before`/`after` rows) and wire the real merged `di.log` and `di.handlers` rather than mocks, so both injected contracts are proven end to end. Between them they cover: pre-`init` guards on every export; dependency validation (non-dict deps, each missing key, each malformed value); `init` arity asserted by observable effect rather than by "it did not throw"; idempotent re-init; the `VERSION` file and exported `version`; `getapimeta`; all-tables and sym-filtered replay; all four payload shapes a tickerplant may log; symbol-atom table and sym selectors; a multi-table log where only the requested table is defined; replay across two log files; requested-but-absent and offered-but-unrequested tables; a configured `subdetailsfunc`; partial, unreadable, absent and empty logs; a replay requested with no root `upd`; malformed `schemalist` **and** `logfilelist` shapes, including a non-table schema, a column-less table, a duplicated table name, and a keyed schema that must still be accepted; a negative message count, asserted with `replay:0b` as well so the guard cannot drift behind the replay branch; every `setschema`/`replay` combination, including `setschema:0b` with `replay:1b` on both the narrowed and all-syms paths; empty `tabs` and `syms` selectors; the double-subscribe guard and its release by `unsubscribe`; `unsubscribe` deleting rather than flagging, its idempotency, and its `warn` on an unheld handle; malformed tickerplant responses; `setschema:0b` preserving a populated table; `logdir` passthrough; and input validation on every argument. +Together the two suites are 660 assertions (`true`, `fail` and `run` rows, excluding the fixture `before`/`after` rows) and wire the real merged `di.log` and `di.handlers` rather than mocks, so both injected contracts are proven end to end. Between them they cover: pre-`init` guards on every export; dependency validation (non-dict deps, each missing key, each malformed value); `init` arity asserted by observable effect rather than by "it did not throw"; idempotent re-init; the `VERSION` file and exported `version`; `getapimeta`; all-tables and sym-filtered replay; all four payload shapes a tickerplant may log; symbol-atom table and sym selectors; a multi-table log where only the requested table is defined; replay across two log files; requested-but-absent and offered-but-unrequested tables; a configured `subdetailsfunc`; partial, unreadable, absent and empty logs; a replay requested with no root `upd`; malformed `schemalist` **and** `logfilelist` shapes, including a non-table schema, a column-less table, a duplicated table name, and a keyed schema that must still be accepted; a negative message count, asserted with `replay:0b` as well so the guard cannot drift behind the replay branch; every `setschema`/`replay` combination, including `setschema:0b` with `replay:1b` on both the narrowed and all-syms paths; empty `tabs` and `syms` selectors; the double-subscribe guard and its release by `unsubscribe`; `unsubscribe` deleting rather than flagging, its idempotency, and its `warn` on an unheld handle; malformed tickerplant responses; `setschema:0b` preserving a populated table; `logdir` passthrough; and input validation on every argument. It also covers the TorQ-protocol cases this module is built to survive: a `0W` message count replaying a whole log, and the same count over a **truncated** log being refused with nothing replayed and no table defined; a **shared** log reported once per table being collapsed to one full replay, with the per-table row counts asserted individually so a regression to the old per-entry replay is caught rather than merely the total; an unshared file in the same response keeping its own count and raising no warn; a corrupt shared log refused, proving the corruption guard covers the collapse trigger and not only the `0W` sentinel; an exact duplicate entry rejected; `` ` `` resolved through `tablelist` with the *resolved* list asserted to be what `subdetails` actually received; the fallback to `` ` `` when a tickerplant offers no `tablelist`; a configured `tablelistfunc` asserted to be called **first**, ahead of `subdetails`; and the two legitimate `rowcounts` shapes accepted with atoms and tables rejected. @@ -256,7 +332,7 @@ The integration block subscribes to a genuinely separate tickerplant process ove - **After `teardown`, a new `subscribe` is refused — reading and releasing still work.** `teardown` removes the `.z.pc` observer, and without it a dropped tickerplant's registry row keeps reporting live for as long as `.z.W` still holds the handle number, and *indefinitely* once kdb+ reissues that number to another connection. That is exactly the failure the `handlers` dependency exists to prevent, so taking a **new** subscription in that state now throws and names the recovery (`call init again`) rather than degrading silently into the `.z.W`-only mode this module documents as insufficient. `getsubscriptions`, `subscribed` and `unsubscribe` deliberately keep working, because a shutdown path needs to inspect and release what it already holds. `init` is idempotent and restores the observer. This reverses an earlier deliberate choice — the previous behaviour let the subscribe through and accepted degraded liveness tracking — because silent degradation is inconsistent with how every other guard in this module behaves - **The root `upd` must accept two payload shapes, and which one it gets depends on the subscription.** An unfiltered replay hands `upd` the logged payload exactly as the tickerplant wrote it — normally a list of columns. A **sym-filtered** replay hands it a **table**, because `replayfilter` has to materialise one to filter by column name rather than by position. Legacy's `replayupd` behaves the same way, so this is inherited rather than introduced, but it is a real contract: an `upd` written only for the column-list shape throws part-way through a filtered replay, after earlier messages have already been applied. Write it as `` {[t;x] @[`.;t;{[tab;d] tab upsert $[98h=type d;d;flip (cols tab)!d]}[;x]]} ``, which is what the usage example and the test fixtures use. The live feed is unaffected — it never passes through `replayfilter` - **Live-feed filtering is the tickerplant's job, not this module's.** `replayfilter` is installed only for the duration of a narrowed replay and torn down the moment it ends; live messages after that reach the caller's `upd` unfiltered by anything here. That is correct, not a leak: the `subdetails` call itself registers the handle for tickerplant-side filtered delivery as a *side effect*. In shipped TorQ, `subdetails` calls `.ps.subscribe`, which **is** `.u.sub`, which routes through `suball`/`subfiltered` to `selfiltered` - and that records `(tbl;handle;filts;columns)` with `filts` built as `` (in;`sym;syms) ``, which `pub` then evaluates per subscriber. Replay needs its own filter only because it bypasses the publish layer entirely and reads the log file directly. Without this note a reader seeing `replayfilter` removed right after the replay would reasonably read it as a bug -- **That delegation is verified end to end.** Measured against a live TorQ segmented tickerplant: a sym-filtered subscribe on a fresh connection registers `` filts: ,,,(in;`sym;,,`S1) `` in the tickerplant's `.stpps.subrequestfiltered`, leaves `subrequestall` empty for that table, and delivers only the matching rows live — two of three published. So the filtering really is the tickerplant's, not an assumption about it +- **That delegation is verified end to end.** Measured against a live TorQ v1.0 segmented tickerplant (`singular` multilog, `replayperiod day`) at the time the delegation was written — that instance is long gone, so treat it as a recorded measurement of the subscribe/filter path, not of later additions such as `getsubscriptionhandles` or `republish`, which are covered by the suites instead: a sym-filtered subscribe on a fresh connection registers `` filts: ,,,(in;`sym;,,`S1) `` in the tickerplant's `.stpps.subrequestfiltered`, leaves `subrequestall` empty for that table, and delivers only the matching rows live — two of three published. So the filtering really is the tickerplant's, not an assumption about it - **The `0W` message count means "replay everything", not "more than I have".** A segmented tickerplant reports `0Wj` for every **closed** log under `replayperiod` `` `day ``. It is resolved to the log's own readable total rather than forwarded to `-11!`, because `-11!(0W;log)` over a corrupt log replays the good prefix and only *then* throws - precisely the half-populated state the preflight exists to prevent. A corrupt log is **refused** on this path, even though its readable prefix could be replayed: the whole log was asked for, so a prefix would be a silently incomplete history. On the ordinary finite-count path, damage lying *beyond* the messages actually needed is still tolerated - **A shared log is replayed once, in full.** In `singular` and `periodic` multilog modes a segmented tickerplant writes every table to **one** file, and reports that file once *per table* with a different count each time. Replaying it once per entry re-applies the head of the file - measured, `(4;LF)` then `(2;LF)` over a six-message log applies messages 0 and 1 twice and never reaches 4 and 5. Duplicate entries for one file are therefore collapsed to a single replay of the whole file, with the table filter discarding the rest, and the condition is logged at `warn`. **The trade-off is trailing duplicates, not missing rows:** messages logged between the `subdetails` call and the replay arrive twice, once from the log and once on the live feed. That is deliberate - the alternative (summing the per-table counts) never duplicates but silently *drops* rows whenever the shared log also carries tables you did not subscribe to, and a missing row is invisible and permanent where a duplicate is visible and diagnosable. An **exact** duplicate entry - same file *and* same count - is rejected instead, because no shipped tickerplant can emit one: the segmented producer applies `distinct` to these pairs itself, and the chained and standard producers emit at most one entry each - **A whole-file replay is narrowed to the offered tables; a per-table one is not.** An all-tables, all-syms subscribe normally replays the log raw, with no filter wrapper — correct for a per-table log, because the tickerplant returned that file *because* it belongs to a table it offered. It is not correct for a file read to its end, and the difference is structural rather than hypothetical: a segmented tickerplant opens logs for `` tables[`.] except `currlog `` (`stplog.q`, `logtabs`) but publishes only `` tables[] except `currlog`heartbeat`logmsg`svrstoload `` (`segmentedtickerplant.q`, `.stpps.init`), and `.stpps.upd` applies **no** membership check before logging. So in `singular` and `periodic` multilog modes the one shared file can legitimately carry tables the tickerplant declined to offer a schema for, and replaying it raw would drive the caller's `upd` with a table it never subscribed to — throwing part-way through a replay, or silently creating a wrongly-shaped table at root. Both whole-file triggers are narrowed to the subscribed set: the shared-log collapse and the `0W` `` `day `` sentinel, which has the same exposure. The sym filter still passes straight through when `syms` is `` ` ``, so the cost is one table-membership test per message on that path and nothing else. This was found by adversarial probe rather than by the suite, and it pre-dated the shared-log collapse — the collapse widened it from a leaked prefix to a leaked whole file rather than introducing it @@ -266,5 +342,5 @@ The integration block subscribes to a genuinely separate tickerplant process ove - A malformed `logfilelist` is rejected on the same terms as `schemalist` — a non-list, an entry that is not a `(messagecount;logfile)` pair, a non-integer count or a non-symbol log file. A **negative** count is rejected too, which no shape check catches, since `-1` is a perfectly good integer; that guard lives in the response validation rather than the replay preflight so it also fires when `replay` is `0b` - `subdetails` is TorQ's real protocol, defined at root by `chainedtp.q` and `segmentedtickerplant.q`, returning `` `schemalist`logfilelist`rowcounts`date `` and optionally `` `logdir ``. Both also define `tablelist` at root, which is how `` ` `` is resolved. The key names and shapes here were taken from that source and are exercised against a tickerplant process built to the same protocol in the integration tests; the module has **not** yet been run against a live TorQ chained or segmented tickerplant. `di.tickerplant` is not built yet - when it lands, point `subdetailsfunc` at its entry point if the name differs - Not implemented: remote-log streaming. The subscriber is assumed to share the tickerplant's filesystem and replays from the log path the tickerplant reports — the classic tick assumption, and what legacy `.sub` does. This is a capability TorQ does not have either, rather than a port omission -- The modularisation plan lists `di.servers` and `di.pubsub` as dependencies of this module. Neither is used: handle resolution is the caller's job, and `di.pubsub` is the publisher side - the tickerplant's own subscriber registry - which a subscribing process never calls +- The modularisation plan places this module in the FRAMEWORK tier with `-> di.servers, di.pubsub`. Both are real module imports, `use`d in `init.q` and declared in `deps.q`. `di.servers` backs `getsubscriptionhandles`; `di.pubsub` is the **local** publisher a chained or segmented tickerplant republishes through, wired by the `republish` config key. The module also speaks the *remote* publisher's wire protocol over IPC, which is a separate thing from the local import and is exercised by the integration suite's peer. See Dependencies - `unsubscribe` is the supported way to close the local-close liveness gap, but it is a **cooperative** mechanism, not a structural one: kdb+ exposes no way to detect an *unannounced* `hclose`, and no way to tell whether a reissued descriptor is still the same remote. A caller that closes a tickerplant handle without calling `unsubscribe` first, and whose descriptor is then reissued, can still leave a stale row appearing live. Tickerplant death, and any local close that goes through `unsubscribe`, are both exact. With `unsubscribefunc` configured the tickerplant-side release is exact too diff --git a/di/subscriptions/subscriptions.q b/di/subscriptions/subscriptions.q index d5a38c46..5503f848 100644 --- a/di/subscriptions/subscriptions.q +++ b/di/subscriptions/subscriptions.q @@ -38,6 +38,12 @@ defaulttablelistfunc:`tablelist; / a real release; leave it unset and unsubscribe stays local bookkeeping and says so defaultunsubscribefunc:`; +/ whether a successful subscribe hands the tables it defined at root to the LOCAL di.pubsub, so this +/ process can serve them downstream. off by default: a plain rdb or wdb subscriber consumes a feed and +/ must not silently start publishing one. set it and this module fills the chained/segmented +/ tickerplant role TorQ splits across chainedtp.q and sctp.q - see handoffpublisher +defaultrepublish:0b; + / appended to every error raised AFTER the subdetails call. asking a tickerplant for the schemas IS / .u.sub (pubsub.q defines .ps.subscribe:.u.sub), so the call registers this handle for live delivery / as a side effect, and the subdetails protocol has no unsubscribe verb to undo it - tickerplant-side @@ -161,6 +167,8 @@ init:{[deps] / module's config. ONE dict carrying dependency and config keys side by side - the call shape / di.torq wires every module with. / e.g. sub.init[`log`handlers!(logging.logdict`log;handlerdep)] + / NB di.servers is NOT injected - it is a HARD dependency imported with `use` in init.q, per the + / modularisation plan's tier table (di.subscriptions -> di.servers, di.pubsub). see deps.q / NB build deps as ONE multi-key dict. joining logdict to a single-key dict - i.e. / logdict,enlist[`handlers]!enlist handlerdep - throws 'mismatch: both value sides are tables / init has ONE process-global side effect: a .z.pc observer registered through di.handlers, which @@ -192,6 +200,10 @@ init:{[deps] usf:$[`unsubscribefunc in key deps;deps`unsubscribefunc;defaultunsubscribefunc]; if[not -11h=type usf; '"di.subscriptions: unsubscribefunc must be a symbol naming the tickerplant-side function, or ` for none"]; + / off by default: only a chained or segmented tickerplant republishes what it subscribed to + rpb:$[`republish in key deps;deps`republish;defaultrepublish]; + if[not -1h=type rpb; + '"di.subscriptions: republish must be a boolean"]; fresh:not initialised[]; .z.m.loginfo:(deps`log)`info; .z.m.logwarn:(deps`log)`warn; @@ -201,6 +213,7 @@ init:{[deps] .z.m.subdetailsfunc:sdf; .z.m.tablelistfunc:tlf; .z.m.unsubscribefunc:usf; + .z.m.republish:rpb; if[fresh;.z.m.subscriptions:subscriptionsschema]; / cleared BEFORE the registration is attempted and set only once it has succeeded, so a register / that throws leaves the flag false rather than unset or stale. that matters twice: .z.m.subscriptions @@ -669,6 +682,39 @@ buildreturn:{[details;subtabs] / public api +handoffpublisher:{[] + / hand the tables this process has subscribed to over to the LOCAL di.pubsub, so it can serve them + / downstream. this is the chained/segmented tickerplant role: TorQ's chainedtp.q (:71-82) and + / sctp.q (:15-25) subscribe upstream and republish the same tables, serving their own table list + / straight out of the pubsub registry (chainedtp.q:7, tablelist:{.stpps.t}). + / OFF by default - a plain rdb or wdb subscriber must not silently become a publisher. + / passes the union across the WHOLE registry, not just the tables from this subscribe call: + / di.pubsub's setsubtables REPLACES its table list (pubsub.q:119), so sending only the latest + / call's tables would drop everything subscribed before it + / read EXPLICITLY - a bare read would resolve to .z.m.republish just the same, but every other + / state access in this module is explicit and the bare form is the one thing qlint flags + if[not .z.m.republish;:()]; + / ADDITIVE - unions with what the publisher already serves rather than recomputing purely from the + / registry. two reasons, both measured: + / - unsubscribe DELETES its registry rows, so a registry-only union would drop those tables at the + / next unrelated subscribe. downstream subscribers would silently stop receiving a table, at a + / moment unconnected to the unsubscribe that caused it + / - setsubtables with an EMPTY list does not mean "publish nothing": di.pubsub then falls back to + / every table at root (pubsub.q:125), so shrinking toward empty is actively dangerous + / the set therefore only grows within a process. a table that is unsubscribed stays advertised and + / simply stops receiving data - a visible, inert condition rather than a silent disappearance + tabs:distinct (),raze exec tabs from .z.m.subscriptions; + tabs:distinct tabs,pubsub.getsubtables[]; + / di.pubsub.init reads each table from ROOT (extractschema:{0#value table}, pubsub.q:84), so a name + / that was never defined - a subscribe with setschema:0b against a table this process does not hold - + / would throw there. drop those rather than hand over a name the publisher cannot resolve + tabs:tabs where tabs in tables[]; + if[0=count tabs;:()]; + @[{[t] pubsub.setsubtables t; pubsub.init[]};tabs; + {[e] raiseerror[`handoffpublisher;"failed to register subscribed tables with di.pubsub: ",e,registerednote]}]; + .z.m.loginfo[`handoffpublisher;"registered ",(", " sv string tabs)," with the local publisher for republishing"]; + }; + subscribe:{[tph;tabs;syms;setschema;replay] / subscribe over an ALREADY-OPEN tickerplant handle - this module never opens a connection; the / caller resolves one (di.servers.gethandlebytype) and passes it in. @@ -758,6 +804,7 @@ subscribe:{[tph;tabs;syms;setschema;replay] .z.m.subscriptions:.z.m.subscriptions, ([]handle:enlist tph;tabs:enlist wanted;syms:enlist syms;subtime:enlist .z.p;active:enlist 1b); .z.m.loginfo[`subscribe;"subscribed to ",(", " sv string wanted)," on tickerplant handle ",.Q.s1 tph]; + handoffpublisher[]; :buildreturn[details;wanted]; }; @@ -873,6 +920,42 @@ getsubscriptions:{[] :activesubscriptions[]; }; +getsubscriptionhandles:{[proctype;procname] + / resolve tickerplant handles by proctype and/or procname, projected to the (procname;proctype;w) + / triple a caller needs before it can subscribe. ported from TorQ .sub.getsubscriptionhandles + / (code/common/subscriptions.q:11) - registered public API there (apidetails.q:67), called by + / rdb.q:163, wdb.q:540, chainedtp.q:71 and sctp.q:15. + / the two lookup arguments are NOT interchangeable and this is the whole of the function's logic: + / ` matches EVERY row, () matches NONE and additionally switches the combine from intersection to + / union. that is what makes both real call shapes work off one function - rdb/wdb pass [types;()] + / and want the proctype matches, chainedtp/sctp pass [`;name] and want the single named process. + / TWO deliberate differences from legacy, both forced by di.servers' contract: + / - legacy took a third `attributes` argument and filtered on .servers.SERVERS's attributes + / column. di.servers' SERVERS carries no such column, so the parameter is DROPPED rather than + / accepted and ignored - a filter that silently does nothing returns handles the caller + / believes were filtered, which surfaces far from its cause + / - legacy passed autoopen:1b to retry dead connections on demand. di.servers returns live rows + / only (where not null w) and runs its own retry job, so reconnection is its concern now + / di.servers is reached directly as a hard dependency (init.q's `use`), not through an injected + / dict - the plan's tier table makes it a hard edge, and a module import needs no wiring + requireinit[`getsubscriptionhandles]; + if[not all (type each (proctype;procname)) in -11 11 0h; + raiseerror[`getsubscriptionhandles;"proctype and procname must each be a symbol, a symbol list or ()"]]; + if[any {(0h=type x) and 0/dev/null || true"";{[e] :(::)}]; };",1,1,fixture: killstalepeers -before,0,0,q,"isfree:{[p] :not @[{hclose hopen x;1b};(`$"":localhost:"",string p;100);{[e] :0b}]; };",1,1,fixture: isfree -before,0,0,q,pickport:{[start] :first (start+til 500) where isfree each start+til 500; };,1,1,fixture: pickport -before,0,0,q,"waitlisten:{[port;timeoutms] deadline:.z.p+`timespan$1000000*timeoutms; while[(.z.p/dev/null 2>&1 &""; if[not waitlisten[PEERPORT;5000];'""test: tickerplant peer failed to listen on "",string PEERPORT]; h:hopen (`$"":localhost:"",string PEERPORT;2000); if[not PEERTOKEN~@[h;""PEERTOKEN"";{[e] `}]; hclose h; '""test: port "",string[PEERPORT],"" is served by a foreign process, not this run's peer""]; `PEERPID set h "".z.i""; hclose h; };",1,1,fixture: spawnpeer +before,0,0,q,"spawnpeer:{[] killstalepeers[]; system ""rm -rf "",PEERDIR; system ""mkdir -p "",PEERDIR; `PEERTOKEN set `$""peer"",string[`int$.z.i],""run"",string `int$.z.n mod 1000000; writepeerscript[]; system (getenv[`QHOME]),""/bin/q "",PEERDIR,""/tp.q -p 0W -q /dev/null 2>&1 &""; `PEERPORT set readport[PEERDIR,""/port"";8000]; if[null PEERPORT;'""test: tickerplant peer never reported a port""]; h:hopen (`$"":localhost:"",string PEERPORT;2000); if[not PEERTOKEN~@[h;""PEERTOKEN"";{[e] `}]; hclose h; '""test: the peer that answered is not this run's""]; `PEERPID set h "".z.i""; hclose h; };",1,1,"fixture: spawnpeer - OS-assigned port, read back from the peer itself" before,0,0,q,"peerhandle:{[] :hopen (`$"":localhost:"",string PEERPORT;2000); };",1,1,fixture: peerhandle before,0,0,q,"noticedrop:{[h] @[{x ""1+1""};h;{[e] :(::)}]; system ""sleep 0.3""; };",1,1,fixture: noticedrop before,0,0,q,"flushpeer:{[h] h ""1+1""; system ""sleep 0.2""; h ""1+1""; };",1,1,fixture: flushpeer before,0,0,q,"killpeer:{[] if[not null PEERPID;@[system;""kill "",string PEERPID;{[e] :(::)}]]; `PEERPID set 0N; system ""sleep 0.3""; };",1,1,fixture: killpeer before,0,0,q,peerbody:(),1,1,"the peer q script, one complete line per row - a split literal would silently become several expressions" +before,0,0,q,"peerbody,:enlist ""ps:use`di.pubsub;""",1,1,peer source - load the REAL publisher rather than hand-rolling a subscriber registry before,0,0,q,"peerbody,:enlist ""LF:hsym`$\"""",PEERDIR,""/tp\"",string D;""",1,1,peer source line -before,0,0,q,"peerbody,:enlist ""peerschema:([]time:`timestamp$();sym:`g#`symbol$();price:`float$();size:`int$());""",1,1,peer source line +before,0,0,q,"peerbody,:enlist ""peertrade:([]time:`timestamp$();sym:`g#`symbol$();price:`float$();size:`int$());""",1,1,peer source line +before,0,0,q,"peerbody,:enlist ""peerschema:0#peertrade;""",1,1,peer source line +before,0,0,q,"peerbody,:enlist ""ps.setsubtables enlist`peertrade;""",1,1,peer source - di.pubsub publishes exactly this table +before,0,0,q,"peerbody,:enlist ""ps.init[];""",1,1,peer source - di.pubsub extracts the schema it will hand out before,0,0,q,"peerbody,:enlist ""N:"",string PEERMSGS",1,1,peer source line -before,0,0,q,"peerbody,:enlist ""cols4:{[i] (enlist 2026.08.06D10:00+`timespan$60000000000*i;enlist `$\""S\"",string i mod 4;""",1,1,peer source line -before,0,0,q,"peerbody,:enlist "" enlist 1.0*i;enlist `int$10*i)};""",1,1,peer source line +before,0,0,q,"peerbody,:enlist ""cols4:{[i] (enlist 2026.08.06D10:00+`timespan$60000000000*i;enlist `$\""S\"",string i mod 4;enlist 1.0*i;enlist `int$10*i)};""",1,1,peer source line before,0,0,q,"peerbody,:enlist ""mk:{[i] (`upd;`peertrade;cols4 i)};""",1,1,peer source line before,0,0,q,"peerbody,:enlist ""lh:hopen LF set ();""",1,1,peer source line before,0,0,q,"peerbody,:enlist ""{[h;i] h enlist mk i}[lh] each til N;""",1,1,peer source line -before,0,0,q,"peerbody,:enlist ""subs:();""",1,1,peer source line -before,0,0,q,"peerbody,:enlist ""sd:{[tabs;instruments] subs::distinct subs,.z.w;""",1,1,peer source line -before,0,0,q,"peerbody,:enlist "" `schemalist`logfilelist`rowcounts`date!(enlist(`peertrade;peerschema);""",1,1,peer source line -before,0,0,q,"peerbody,:enlist "" enlist(N;LF);(enlist `peertrade)!enlist N;D)};""",1,1,peer source line -before,0,0,q,"peerbody,:enlist ""subdetails:sd;""",1,1,peer source line +before,0,0,q,"peerbody,:enlist ""subdetails:{[tabs;instruments] r:ps.subscribe[tabs;instruments]; if[-11h=type r;:r]; sl:$[-11h=type first r;flip last r;flip r]; `schemalist`logfilelist`rowcounts`date!(sl;enlist(N;LF);(enlist`peertrade)!enlist N;D)};""",1,1,"peer source - the subdetails adapter a modular tickerplant would own: di.pubsub.subscribe does the registration and returns (tabs;schemas), which flip turns into the schemalist pairs this module reads. an errmsg atom is passed straight back" before,0,0,q,"peerbody,:enlist ""tablelist:{enlist`peertrade};""",1,1,peer source line -before,0,0,q,"peerbody,:enlist ""pub:{[i;s] m:(`upd;`peertrade;(enlist 2026.08.06D11:00+`timespan$1000000000*i;enlist s;""",1,1,peer source line -before,0,0,q,"peerbody,:enlist "" enlist 1.0*i;enlist `int$i)); lh enlist m; N+:1; {[m;w] (neg w) m}[m] each subs; N};""",1,1,peer source line +before,0,0,q,"peerbody,:enlist ""pub:{[i;s] c:(enlist 2026.08.06D11:00+`timespan$1000000000*i;enlist s;enlist 1.0*i;enlist `int$i); lh enlist (`upd;`peertrade;c); N+:1; ps.publish[`peertrade;flip (cols peertrade)!c]; N};""",1,1,"peer source - di.pubsub.publish does the fan-out, so live delivery and sym filtering are ITS code paths not a stand-in" before,0,0,q,"peerbody,:enlist ""logged:{[] N};""",1,1,peer source line -before,0,0,q,"peerlines:{[] :(""D:2026.08.06;"";""PEERTOKEN:`"",(string PEERTOKEN),"";""),peerbody; };",1,1,prepend the per-run identity token - it is only known once spawnpeer has generated it -before,0,0,q,"teardownfixture:{[] killpeer[]; killstalepeers[]; system ""rm -rf "",BASE; system ""rm -rf "",PEERDIR; };",1,1,fixture: teardownfixture +before,0,0,q,"peerlines:{[] :(""D:2026.08.06;"";""PEERTOKEN:`"",(string PEERTOKEN),"";"";""(`$\"":"",PEERDIR,""/port\"") 0: enlist string system\""p\"";""),peerbody; };",1,1,prepend the identity token and make the peer report its OS-assigned port before doing anything else +before,0,0,q,MIDPORT:0N;,1,1,fixture: MIDPORT +before,0,0,q,"MIDDIR:tmpdir""disubscriptionsmid"";",1,1,fixture: MIDDIR +before,0,0,q,"killstalemid:{[] @[system;""pkill -f '"",MIDDIR,""/mid.q' 2>/dev/null || true"";{[e] :(::)}]; };",1,1,fixture: killstalemid +before,0,0,q,midbody:(),1,1,the middle process source - a chained tickerplant: subscribes upstream through THIS module and republishes through di.pubsub +before,0,0,q,"midbody,:enlist ""lg:use`di.log;""",1,1,middle source line +before,0,0,q,"midbody,:enlist ""hnd:use`di.handlers;""",1,1,middle source line +before,0,0,q,"midbody,:enlist ""ps:use`di.pubsub;""",1,1,middle source line +before,0,0,q,"midbody,:enlist ""sub:use`di.subscriptions;""",1,1,middle source line +before,0,0,q,"midbody,:enlist ""hnd.init[lg.logdict];""",1,1,middle source line +before,0,0,q,"midbody,:enlist ""sub.init[`log`handlers`republish!(lg.logdict`log;`register`remove!(hnd`register;hnd`remove);1b)];""",1,1,"middle source - republish ON, the chained tickerplant role" +before,0,0,q,"midbody,:enlist ""upd:{[t;x] @[`.;t;{[tab;d] tab upsert $[98h=type d;d;flip (cols tab)!d]}[;x]]; ps.publish[t;x]};""",1,1,"middle source - the caller owns forwarding: upsert then republish, as TorQ chainedtp.q tickpub does" +before,0,0,q,"midlines:{[] :(enlist ""(`$\"":"",MIDDIR,""/port\"") 0: enlist string system\""p\"";""),midbody,enlist ""tph:hopen `$\"":localhost:"",(string PEERPORT),""\""; sub.subscribe[tph;`;`;1b;1b]; MIDREADY:1b;""; };",1,1,report the OS-assigned port FIRST so the parent can connect while the upstream subscribe is still running +before,0,0,q,"writemidscript:{[] (`$"":"",MIDDIR,""/mid.q"") 0: midlines[]; };",1,1,fixture: writemidscript +before,0,0,q,"spawnmid:{[] killstalemid[]; system ""rm -rf "",MIDDIR; system ""mkdir -p "",MIDDIR; writemidscript[]; system (getenv[`QHOME]),""/bin/q "",MIDDIR,""/mid.q -p 0W -q /dev/null 2>&1 &""; `MIDPORT set readport[MIDDIR,""/port"";10000]; if[null MIDPORT;'""test: middle tickerplant never reported a port""]; };",1,1,"fixture: spawnmid - OS-assigned port, read back from the middle process" +before,0,0,q,"midhandle:{[] :hopen (`$"":localhost:"",string MIDPORT;2000); };",1,1,fixture: midhandle +before,0,0,q,"killmid:{[] killstalemid[]; system ""rm -rf "",MIDDIR; system ""sleep 0.3""; };",1,1,fixture: killmid +before,0,0,q,"teardownfixture:{[] killpeer[]; killstalepeers[]; killmid[]; system ""rm -rf "",BASE; system ""rm -rf "",PEERDIR; };",1,1,fixture: teardownfixture - reaps the peer AND the middle process before,0,0,q,setupfixture[],1,1,create the temp directories before,0,0,q,realhandlers[`init][enlist[`log]!enlist caplog],1,1,init the real di.handlers - register throws until it has been before,0,0,q,sub.init[deps[]],1,1,init di.subscriptions - this suite runs standalone so it does all of its own setup @@ -73,6 +87,7 @@ true,0,0,q,`g=attr peertrade`sym,1,1,the g# attribute survived the IPC round tri true,0,0,q,6=updcalls,1,1,the replay drove the root upd once per message true,0,0,q,PH~first exec handle from sub.getsubscriptions[] where handle~\:PH,1,1,the registry holds the real int handle true,0,0,q,any exec active from sub.getsubscriptions[] where handle~\:PH,1,1,the subscription over the real handle is live +true,0,0,q,"0 a MIDDLE process that subscribes through this module with republish on -> this test process as the downstream subscriber +comment,,,,,,,the middle process is the chained-tickerplant role the FRAMEWORK tier edge exists for. it cannot be tested in-process: a subscriber that serves its own subscribers has to be at its own main loop +run,0,0,q,spawnpeer[],1,1,fresh upstream tickerplant +run,0,0,q,spawnmid[],1,1,launch the middle process - it subscribes to the peer through di.subscriptions with republish on +run,0,0,q,MH:midhandle[],1,1,connect to the middle as a downstream client would +true,0,0,q,"1b~MH""MIDREADY""",1,1,the middle finished its own upstream subscribe +true,0,0,q,"`peertrade in MH"".m.di.0pubsub.getsubtables[]""",1,1,"THE FEATURE - the middle's local di.pubsub is serving the table it subscribed to upstream, with no code in the middle beyond setting republish" +true,0,0,q,"`peertrade in MH""tables[]""",1,1,and the table really was defined at root there by subscribe +run,0,0,q,@[`.;`peertrade;:;0#peertrade],1,1,clear our local copy so what arrives can only have come through the middle +run,0,0,q,resetupd[],1,1,reset the upd counter +run,0,0,q,dres:MH(`.m.di.0pubsub.subscribe;`peertrade;`),1,1,"subscribe to the MIDDLE, exactly as a downstream rdb would" +true,0,0,q,(enlist`peertrade)~first dres,1,1,the middle confirms the subscription +true,0,0,q,"0 middle -> here across three processes +true,0,0,q,300f=first exec price from peertrade,1,1,"and it is the message that was published, not a replay artefact" +true,0,0,q,1=updcalls,1,1,our root upd was driven exactly once +run,0,0,q,hclose MH; hclose PHX,1,1,tidy the handles +run,0,0,q,killmid[],1,1,stop the middle process + after,0,0,q,teardownfixture[],1,1,kill the peer and remove the temp directories From 3219450f4527f46bc169b7d47d02a4d07e8dd87a Mon Sep 17 00:00:00 2001 From: alowrydi Date: Wed, 12 Aug 2026 20:40:36 +0100 Subject: [PATCH 06/10] di.pubsub: Linking hard dependency to subscriptions module and doing some general housekeeping too: chain .z.pc instead of replacing it, add VERSION/version and getsubtables, signal from the string subscribe entry points, remove hardcoded ports and paths from the suite --- di/pubsub/VERSION | 1 + di/pubsub/init.q | 12 +++++++++++- di/pubsub/pubsub.md | 20 ++++++++++++++++++++ di/pubsub/pubsub.q | 41 +++++++++++++++++++++++++++++++++++++---- di/pubsub/test.csv | 45 ++++++++++++++++++++++++++++++++++++++++----- 5 files changed, 109 insertions(+), 10 deletions(-) create mode 100644 di/pubsub/VERSION diff --git a/di/pubsub/VERSION b/di/pubsub/VERSION new file mode 100644 index 00000000..341cf11f --- /dev/null +++ b/di/pubsub/VERSION @@ -0,0 +1 @@ +0.2.0 \ No newline at end of file diff --git a/di/pubsub/init.q b/di/pubsub/init.q index fc3ad51c..b1dfd942 100644 --- a/di/pubsub/init.q +++ b/di/pubsub/init.q @@ -1,3 +1,13 @@ +/ publisher/subscriber management - the tickerplant side of a subscription: a registry of who wants +/ which tables (optionally sym- or condition-filtered), and the fan-out that publishes to them + \l ::pubsub.q -export:([subscribe;subscribestr;subscribestrfilter;publish;setsubtables;callendofperiod;callendofday;closesub;pubclear;init]) +/ module version, read from the VERSION file rather than hardcoded, so a release bump touches one +/ plain-text file. read module-relative at load (`:::` resolves to di/pubsub) and BEFORE the export +/ line, since export:([...]) evaluates each name. NB `version` must STAY in the export: di.depcheck +/ resolves a dependency's minimum version from the export dict, and reports "exports no version" - +/ failing the dependency check - for any module that omits it +version:first read0`:::VERSION + +export:([subscribe;subscribestr;subscribestrfilter;publish;setsubtables;getsubtables;callendofperiod;callendofday;closesub;pubclear;init;version]) diff --git a/di/pubsub/pubsub.md b/di/pubsub/pubsub.md index 8b5fd350..75c5d24f 100644 --- a/di/pubsub/pubsub.md +++ b/di/pubsub/pubsub.md @@ -49,11 +49,13 @@ publish data with/without filters. The function takes two arguments: t and x, wh | Function | Description | |---------------------------|------------------------------------------------------------------------------| | `pubsub.setsubtables` | Set a specified list of tables that are available for subscription. | +| `pubsub.getsubtables` | Read the list of tables currently available for subscription - the counterpart to `setsubtables`, which replaces it. Empty until `init` has run. | | `pubsub.callendofday` | Broadcast an end-of-day event to all subscribers (requires `endofday`). | | `pubsub.callendofperiod` | Broadcast an end-of-period event to all subscribers (requires `endofperiod`).| | `pubsub.closesub` | Remove handle upon connection close. | | `pubsub.subclear` | Publish tables and clear up the contents. | | `pubsub.init` | Initialize variables - run before calling pub/sub functions to populate required state (e.g., tables/schemas). | +| `pubsub.version` | Module version string, read from the `VERSION` file. `di.depcheck` resolves a dependency's minimum from here. | --- ### Example: @@ -80,6 +82,24 @@ q)pubsub.subscribestrfilter["quote";"bid>50.0";"time,sym,bid"] --- ## Notes: +- **The string entry points signal on failure.** `subscribestr` and `subscribestrfilter` exist so a + non-kdb+ client can subscribe, and such a client cannot inspect a q result shape. A request that + matched **no** table therefore signals rather than returning the error message as a value that + merely reads like one. (The guard that previously did this could never fire: `errmsg` is built with + `` `$ `` so it is a symbol, and `last` of either success shape is the schema list — never the `10h` + string it tested for.) A *partial* match still returns, because those tables really were subscribed + and signalling would report failure while leaving the client registered. +- **`.z.pc` chains, it does not replace.** This module installs a `.z.pc` handler at load so a + dropped connection is deregistered (`closesub`). It captures whatever already owned the event and + calls it afterwards. This matters: a bare `.z.pc:{closesub[x]}` silently destroyed every observer + another module had already registered — measured against `di.handlers`, whose registry went on + reporting the registration as live while it no longer fired, so the loss was invisible. The guard + is asserted in `test.csv` by a child process that installs a handler *before* loading this module, + which is the only way to observe load-time ordering. +- It stays a raw assignment rather than a `di.handlers` registration because the modularisation plan + classifies `di.pubsub` as **standalone** — it takes no injected dependencies, so reaching + `di.handlers` would contradict its own tier. + - By default, all tables on top level of the process are available for subscription. - The user should define the `.u.sub` and the `.u.pub` functions within the process. - The module initializes with defined list of tables to subscribe to and fetches their schemas and columns for use. This is done via calling `init` function. diff --git a/di/pubsub/pubsub.q b/di/pubsub/pubsub.q index d471fd04..f95839b7 100644 --- a/di/pubsub/pubsub.q +++ b/di/pubsub/pubsub.q @@ -71,8 +71,19 @@ closesub:{[h] delete from .z.M.reqfilteredtbl where handle=h; }; -/ define .z.pc, add bespoke actions as needed -.z.pc:{closesub[x]}; +/ define .z.pc, add bespoke actions as needed. +/ CHAINS onto whatever already owns .z.pc rather than replacing it. a bare .z.pc:{closesub[x]} here +/ silently destroyed every observer another module had already registered - measured: with a +/ di.handlers registration in place first, loading this module stopped it firing while di.handlers +/ went on listing it as registered, so the failure was invisible from the registry. +/ this stays a raw assignment rather than a di.handlers registration because the modularisation +/ plan classifies di.pubsub as STANDALONE - it takes no injected dependencies, so it cannot reach +/ di.handlers without contradicting its own tier +priorpc:@[value;`.z.pc;{[e] (::)}]; +.z.pc:{[w] + closesub[w]; + if[not (::)~priorpc;priorpc w]; + }; / broadcast to all subscribers upon end of day, client needs to define endofday function callendofday:{[d](neg getallhandles[])@\:(`endofday;d)}; @@ -103,20 +114,42 @@ pubclear:{[t] @[`.;;0#] each t; }; +raisenosub:{[res] + / internal - signal when a subscribe matched NOTHING, for the string entry points below. + / subscribe returns one of three shapes: (tables;schemas) when every requested table exists, + / (errmsg;(tables;schemas)) when only some do, or a bare errmsg SYMBOL when none do. the string + / entry points exist for non-kdb+ clients, which cannot inspect a q result shape - so a request + / that subscribed to nothing has to arrive as an error, not as a value that merely reads like one. + / the partial case deliberately still RETURNS: those tables really were subscribed, and signalling + / would tell the caller it failed while leaving it registered. + / NB this replaces a guard (10h~type last res) that could never fire - errmsg is built with `$ so it + / is a symbol, and `last` of either success shape is the schema list, never a 10h string + if[-11h=type res;'string res]; + :res; + }; + subscribestr:{[table;syms] / allow non-kdb+ process to subscribe to tables with/without symbols res:subscribe[`$table;$[count syms;`$vs[csv;syms];`]]; - :$[10h~type last res;'last res;res]; + :raisenosub res; }; subscribestrfilter:{[table;filters;columns] / allow non-kdb+ process to subscribe to tables with custom conditions res:subscribe[`$table;1!enlist `table`filts`columns!(`$table;filters;columns)]; - :$[10h~type last res;'last res;res]; + :raisenosub res; }; / create a list of tables for subscription, allow users to set subtables, otherwise set to null setsubtables:{.z.m.subtables:$[x~`;0#x;x]}; + +getsubtables:{[] + / the tables currently available for subscription. the read counterpart to setsubtables, which + / REPLACES the list - a consumer that needs to ADD to the publish set has no other way to learn the + / current one, and reaching into module state from outside is not an interface. + / empty until init has run, rather than signalling on an unset name + :@[{[x] t};::;{[e] `symbol$()}]; + }; setsubtables`; initialized:0b; diff --git a/di/pubsub/test.csv b/di/pubsub/test.csv index 59cae473..7f474e17 100644 --- a/di/pubsub/test.csv +++ b/di/pubsub/test.csv @@ -1,16 +1,23 @@ action,ms,bytes,lang,code,repeat,minver,comment comment,,,,,,,Setup for server and client testing comment,,,,,,,Replace $QHOME (defaults to $HOME) to avoid hard-coding -run,0,0,q,system"q -p 5010 -q &",1,,start server in background +comment,,,,,,,NO PORT NUMBER APPEARS ANYWHERE. the server is started with -p 0W so the OS assigns a free port, and it writes that port to a file the suite reads back +comment,,,,,,,a hardcoded port makes the suite unportable and fails in the worst way - it connects to whatever else is on that port and rows fail with errors that say nothing about pub/sub. scanning from a fixed base is no better: it still races, and a port held by a blocked or non-q process reads as free +run,0,0,q,"tmpdir:{[nm] b:getenv`TMPDIR; if[0=count b;b:""/tmp""]; :b,""/"",nm,string .z.i; }",1,1,"honours TMPDIR rather than assuming /tmp exists, and appends the pid so two runs on one machine cannot collide" +run,0,0,q,"PSDIR:tmpdir""dipubsubtest""; system""rm -rf "",PSDIR; system""mkdir -p "",PSDIR",1,1,private directory for the server script and its port file +run,0,0,q,"(`$"":"",PSDIR,""/srv.q"") 0: enlist ""(`$\"":"",PSDIR,""/port\"") 0: enlist string system\""p\"";""",1,1,the server's whole job at startup: report the port the OS gave it +run,0,0,q,"system (getenv[`QHOME]),""/bin/q "",PSDIR,""/srv.q -p 0W -q /dev/null 2>&1 &""",1,1,"start it with an OS-assigned port, using the SAME q this suite runs under" +run,0,0,q,"PSPORT:{[f] i:0; p:0N; while[(i<200) and null p; if[count key hsym`$f;p:""J""$first read0 hsym`$f]; if[null p;system""sleep 0.05""]; i+:1]; :p}[PSDIR,""/port""]",1,1,"read the port back, waiting for the server to come up" +true,0,0,q,not null PSPORT,1,1,"the server started and reported its port - if this fails every row below would be testing nothing, so read it first" run,0,0,q,system"sleep 1",1,,wait for server to initialize -run,0,0,q,h:hopen`::5010,1,,open handle h on client for testing +run,0,0,q,"h:hopen (`$"":localhost:"",string PSPORT;3000)",1,1,open handle h on client for testing run,0,0,q,h"pubsub:use`di.pubsub",,initialize pubsub module true,0,0,q,h>0,1,,verify client connected to server run,0,0,q,h"trade:([]time:\"p\"$();`g#sym:`$();price:\"f\"$();tsize:\"i\"$());",1,,define trade table on server run,0,0,q,h"quote:([]time:\"p\"$();`g#sym:`$();bid:\"f\"$();bidsize:\"i\"$();ask:\"f\"$();asksize:\"i\"$());",1,,define quote table on server run,0,0,q,h".m.di.0pubsub.init[]",, true,0,0,q,1b~h".m.di.0pubsub.initialized",1,,confirm server initialized with necessary data -run,0,0,q,clt:first h"key[.z.W]",1,,get client handle on server +run,0,0,q,"clt:h"".z.w""",1,1,"our own handle AS THE SERVER SEES IT. first key .z.W is nondeterministic - any second connection (a port probe, a leftover client) makes it pick the wrong handle, which showed up as 12 intermittent failures in rows that had nothing to do with what they were testing" run,0,0,q,upd:insert,1,,define local upd function run,0,0,q,res:h(`.m.di.0pubsub.suball;`quote),1,,test subscribe to quote for all symbols @@ -51,9 +58,36 @@ run,0,0,q,res:h"select from .m.di.0pubsub.reqfilteredtbl where table in `trade`q run,0,0,q,res:select from res where handle=clt,1,,check filtered table for the subscription true,0,0,q,res[`filts][0]~enlist enlist(in;`sym;enlist `IBM`NVO),1,, test symbols are listed under filters + +comment,,,,,,,.z.pc CHAINS onto whatever already owns it rather than replacing it. a bare assignment silently destroyed observers another module had already registered - the registration stopped firing while di.handlers went on listing it as live +comment,,,,,,,only observable in a SEPARATE process: the ordering under test is a handler installed BEFORE this module loads, and by the time these rows run it is long loaded +run,0,0,q,"chaindir:tmpdir""dipubsubchain""; system ""rm -rf "",chaindir; system ""mkdir -p "",chaindir",1,1,temp dir for the child probe +run,0,0,q,"(`$"":"",chaindir,""/probe.q"") 0: (""PRIOR:0b;"";"".z.pc:{[w] `PRIOR set 1b};"";""p:use`di.pubsub;"";"".z.pc 99i;"";""-1 string PRIOR;"";""exit 0"")",1,1,"a child that installs .z.pc FIRST and only then loads di.pubsub - the order that used to lose the handler. it prints string PRIOR, so 1 means the pre-existing handler still fired and 0 means it was lost" +run,0,0,q,"chainres:system (getenv[`QHOME]),""/bin/q "",chaindir,""/probe.q -q""",1,1,run it in a genuinely separate process +true,0,0,q,"(enlist ""1"")~first chainres",1,1,"THE ASSERTION - the handler installed before the load still fired (string 1b is a one-char STRING, not a char atom). a replacing assignment prints 0, and the loss is invisible from any registry" +run,0,0,q,"system ""rm -rf "",chaindir",1,1,clean up + +comment,,,,,,,the STRING entry points must SIGNAL when a subscribe matched nothing. they exist for non-kdb+ clients which cannot inspect a q result shape, so an error has to arrive as an error +comment,,,,,,,the guard that used to do this could never fire: errmsg is built with `$ so it is a symbol, and last of either success shape is the schema list - never the 10h string it tested for +fail,0,0,q,"h(`pubsub.subscribestr;""nosuchtable"";"""")",1,1,subscribestr signals on a table that does not exist +fail,0,0,q,"h(`pubsub.subscribestrfilter;""nosuchtable"";"""";"""")",1,1,subscribestrfilter signals too +true,0,0,q,"not 10h~type h"".m.di.0pubsub.suball[`nosuchtable]""",1,1,"and the reason the old guard failed: the errmsg is a symbol, not a string" +run,0,0,q,"res:h(`pubsub.subscribestr;""trade"";"""")",1,1,a VALID subscribe still returns rather than signalling +true,0,0,q,(enlist`trade)~first res,1,1,with the confirmed table +true,0,0,q,98h~type res[1;0],1,1,and its schema + +comment,,,,,,,getsubtables - the read counterpart to setsubtables, which REPLACES the list. without it a consumer that needs to ADD to the publish set has to reach into module state. driven on the SERVER over h, like every other row here +true,0,0,q,"(asc h"".m.di.0pubsub.t"")~asc h"".m.di.0pubsub.getsubtables[]""",1,1,getsubtables reports exactly the tables being published +true,0,0,q,"11h=type h"".m.di.0pubsub.getsubtables[]""",1,1,it is a symbol list +true,0,0,q,"all `trade`quote in h"".m.di.0pubsub.getsubtables[]""",1,1,and it names the tables this suite set up + +comment,,,,,,,VERSION file and exported version - di.depcheck resolves a dependency minimum from the export dict and fails any module that omits it +true,0,0,q,"`version in key .m.di.0pubsub",1,1,version is reachable as a module name +true,0,0,q,"3=count ""."" vs .m.di.0pubsub.version",1,1,the VERSION file holds a parseable major.minor.patch string +true,0,0,q,"not any .m.di.0pubsub.version in ""\n\r""",1,1,no stray newline came through from the file run,0,0,q,hclose h,1,,test closesub function-handles are removed after connection close -run,0,0,q,h:hopen`::5010,1,,open handle h on client for testing -run,0,0,q,clt:first h"key[.z.W]",1,,get client handle on server +run,0,0,q,"h:hopen (`$"":localhost:"",string PSPORT;3000)",1,1,open handle h on client for testing +run,0,0,q,"clt:h"".z.w""",1,1,"our own handle AS THE SERVER SEES IT. first key .z.W is nondeterministic - any second connection (a port probe, a leftover client) makes it pick the wrong handle, which showed up as 12 intermittent failures in rows that had nothing to do with what they were testing" true,0,0,q,(`$())~h"key .m.di.0pubsub.reqalldict",1,,handle not in reqalldict since previous handle removed and new subscription not made true,0,0,q,not clt in(h".m.di.0pubsub.reqfilteredtbl")`handle,1,,handle not in reqfilteredtbl @@ -115,3 +149,4 @@ true,0,0,q,res[`filts][0]~enlist enlist(>;`bid;50f),1,, test symbols are listed run,0,0,q,neg[h](exit;0);neg[h](::),1,,close background process and flush run,0,0,q,hclose h,1,,close handle +run,0,0,q,"system""rm -rf "",PSDIR",1,1,remove the server directory From 6194509f0626d669a83c90f14be61f0c5bb47735 Mon Sep 17 00:00:00 2001 From: alowrydi Date: Wed, 12 Aug 2026 20:59:13 +0100 Subject: [PATCH 07/10] Warn rather than throw when the republish handoff fails --- di/subscriptions/subscriptions.md | 6 ++++-- di/subscriptions/subscriptions.q | 21 ++++++++++++++++++--- di/subscriptions/test.csv | 20 +++++++++++++------- 3 files changed, 35 insertions(+), 12 deletions(-) diff --git a/di/subscriptions/subscriptions.md b/di/subscriptions/subscriptions.md index 12b9ba7d..ff51e017 100644 --- a/di/subscriptions/subscriptions.md +++ b/di/subscriptions/subscriptions.md @@ -56,6 +56,8 @@ That split is deliberate: this module owns the *subscription*, the caller owns w Legacy's chained tickerplants also seed `.u.d`/`.u.icounts` from the subscribe result. Those are tickerplant sequence-and-date state, which `di.pubsub` does not own (it has no `i`, `j`, `icounts` or `d`); the values are returned to the caller in `subscribe`'s result as `d` and `icounts` so whatever owns a tickerplant log can apply them. +**A handoff failure does not fail the subscribe.** By the time the handoff runs, the subscribe has fully succeeded — schemas defined, log replayed, registry row committed. If `di.pubsub` then refuses the tables, the module logs a **warning** naming the reason and stating plainly that the subscription stands, and `subscribe` returns its normal result. It does not throw: discarding completed work over an optional secondary step would be wrong, and appending the usual post-`subdetails` remedy ("close the handle before retrying") would be worse — the connection is healthy, and a retry would hit the duplicate-subscription guard against the row that call just committed. This matches how `unsubscribe` handles the same shape when its optional `unsubscribefunc` call fails. + A table that is unsubscribed stays advertised and simply stops receiving data — a visible, inert condition rather than a silent disappearance. Tables not defined at root are filtered out first, since `di.pubsub.init` calls `value` on each name (`pubsub.q:84`). Note the module still speaks the publisher's **wire** protocol separately: `subdetailsfunc` and `tablelistfunc` name functions evaluated on the *remote* tickerplant, so they resolve in that process. Legacy is the same shape — its `.u.sub`/`.u.i`/`.u.L`/`.u.icounts`/`.u.d` references (`subscriptions.q:100-101`) sit inside lambdas sent to the tickerplant handle. The integration suite's peer runs the real `di.pubsub`, so that half is exercised across a genuine process boundary. @@ -243,7 +245,7 @@ Two suites, following the convention `di.handlers` and `di.permissions` use. Bot fixtures inline as `before` rows; there is no separate fixture file, so neither depends on the working directory. -**Unit suite** (`test.csv`) - 529 assertions, no child processes, no ports, no `QHOME`. It does need `di.servers` on `QPATH`, since the module imports it, and it drives the **real** `di.servers` rather than a mock: the fixtures seed its `SERVERS` registry directly, so the whole selection matrix runs against the genuine `getservers` without opening a socket: +**Unit suite** (`test.csv`) - 534 assertions, no child processes, no ports, no `QHOME`. It does need `di.servers` on `QPATH`, since the module imports it, and it drives the **real** `di.servers` rather than a mock: the fixtures seed its `SERVERS` registry directly, so the whole selection matrix runs against the genuine `getservers` without opening a socket: ```q k4unit:use`di.k4unit @@ -287,7 +289,7 @@ loaded and would re-run against dirty module state, reporting spurious failures. self-contained: it inits `di.handlers` and `di.subscriptions` itself rather than inheriting setup from the unit rows. -Together the two suites are 660 assertions (`true`, `fail` and `run` rows, excluding the fixture `before`/`after` rows) and wire the real merged `di.log` and `di.handlers` rather than mocks, so both injected contracts are proven end to end. Between them they cover: pre-`init` guards on every export; dependency validation (non-dict deps, each missing key, each malformed value); `init` arity asserted by observable effect rather than by "it did not throw"; idempotent re-init; the `VERSION` file and exported `version`; `getapimeta`; all-tables and sym-filtered replay; all four payload shapes a tickerplant may log; symbol-atom table and sym selectors; a multi-table log where only the requested table is defined; replay across two log files; requested-but-absent and offered-but-unrequested tables; a configured `subdetailsfunc`; partial, unreadable, absent and empty logs; a replay requested with no root `upd`; malformed `schemalist` **and** `logfilelist` shapes, including a non-table schema, a column-less table, a duplicated table name, and a keyed schema that must still be accepted; a negative message count, asserted with `replay:0b` as well so the guard cannot drift behind the replay branch; every `setschema`/`replay` combination, including `setschema:0b` with `replay:1b` on both the narrowed and all-syms paths; empty `tabs` and `syms` selectors; the double-subscribe guard and its release by `unsubscribe`; `unsubscribe` deleting rather than flagging, its idempotency, and its `warn` on an unheld handle; malformed tickerplant responses; `setschema:0b` preserving a populated table; `logdir` passthrough; and input validation on every argument. +Together the two suites are 665 assertions (`true`, `fail` and `run` rows, excluding the fixture `before`/`after` rows) and wire the real merged `di.log` and `di.handlers` rather than mocks, so both injected contracts are proven end to end. Between them they cover: pre-`init` guards on every export; dependency validation (non-dict deps, each missing key, each malformed value); `init` arity asserted by observable effect rather than by "it did not throw"; idempotent re-init; the `VERSION` file and exported `version`; `getapimeta`; all-tables and sym-filtered replay; all four payload shapes a tickerplant may log; symbol-atom table and sym selectors; a multi-table log where only the requested table is defined; replay across two log files; requested-but-absent and offered-but-unrequested tables; a configured `subdetailsfunc`; partial, unreadable, absent and empty logs; a replay requested with no root `upd`; malformed `schemalist` **and** `logfilelist` shapes, including a non-table schema, a column-less table, a duplicated table name, and a keyed schema that must still be accepted; a negative message count, asserted with `replay:0b` as well so the guard cannot drift behind the replay branch; every `setschema`/`replay` combination, including `setschema:0b` with `replay:1b` on both the narrowed and all-syms paths; empty `tabs` and `syms` selectors; the double-subscribe guard and its release by `unsubscribe`; `unsubscribe` deleting rather than flagging, its idempotency, and its `warn` on an unheld handle; malformed tickerplant responses; `setschema:0b` preserving a populated table; `logdir` passthrough; and input validation on every argument. It also covers the TorQ-protocol cases this module is built to survive: a `0W` message count replaying a whole log, and the same count over a **truncated** log being refused with nothing replayed and no table defined; a **shared** log reported once per table being collapsed to one full replay, with the per-table row counts asserted individually so a regression to the old per-entry replay is caught rather than merely the total; an unshared file in the same response keeping its own count and raising no warn; a corrupt shared log refused, proving the corruption guard covers the collapse trigger and not only the `0W` sentinel; an exact duplicate entry rejected; `` ` `` resolved through `tablelist` with the *resolved* list asserted to be what `subdetails` actually received; the fallback to `` ` `` when a tickerplant offers no `tablelist`; a configured `tablelistfunc` asserted to be called **first**, ahead of `subdetails`; and the two legitimate `rowcounts` shapes accepted with atoms and tables rejected. diff --git a/di/subscriptions/subscriptions.q b/di/subscriptions/subscriptions.q index 5503f848..03c367a8 100644 --- a/di/subscriptions/subscriptions.q +++ b/di/subscriptions/subscriptions.q @@ -710,9 +710,24 @@ handoffpublisher:{[] / would throw there. drop those rather than hand over a name the publisher cannot resolve tabs:tabs where tabs in tables[]; if[0=count tabs;:()]; - @[{[t] pubsub.setsubtables t; pubsub.init[]};tabs; - {[e] raiseerror[`handoffpublisher;"failed to register subscribed tables with di.pubsub: ",e,registerednote]}]; - .z.m.loginfo[`handoffpublisher;"registered ",(", " sv string tabs)," with the local publisher for republishing"]; + / WARN, not raiseerror: unlike every other raiseerror site in this module, a failure here does not + / mean the subscribe failed. it already fully succeeded - schemas defined, replay done, registry row + / committed - before this runs at all (see the call site, last statement of subscribe). + / registerednote's remedy, "close the handle before retrying", describes a DIFFERENT failure - a + / subdetails call that registered live delivery before subscribe could be validated - and would be + / actively wrong advice here: the connection is healthy, and a retry would immediately hit + / guardduplicate against the row this very call committed. + / this matches how unsubscribe handles the identical shape (local state already committed, an + / optional notification step then fails): warn and return, because what mattered locally already + / happened. republish is opt-in and secondary by design, so it must not take down a successful + / subscribe + r:@[{[t] pubsub.setsubtables t; pubsub.init[]; (1b;t)};tabs;{[e] (0b;e)}]; + $[first r; + .z.m.loginfo[`handoffpublisher;"registered ",(", " sv string tabs)," with the local ", + "publisher for republishing"]; + .z.m.logwarn[`handoffpublisher;"failed to register subscribed tables with the local ", + "publisher: ",(last r)," - the subscribe itself succeeded; this process is not serving ", + "them downstream until this is retried or di.pubsub is checked"]]; }; subscribe:{[tph;tabs;syms;setschema;replay] diff --git a/di/subscriptions/test.csv b/di/subscriptions/test.csv index 58648eb1..954ee02b 100644 --- a/di/subscriptions/test.csv +++ b/di/subscriptions/test.csv @@ -696,19 +696,25 @@ run,0,0,q,markdeadfor[rpeh],1,1,fire .z.pc for that handle - this only works if true,0,0,q,"not any exec active from sub.getsubscriptions[] where handle~\:rpeh",1,1,the observer fired and marked it dead - importing di.pubsub did not silently unhook us -comment,,,,,,,the handoff's ERROR branch. no natural trigger exists - every table is filtered to ones defined at root before di.pubsub sees them - so it is reached by replacing the module handle init.q captured. without this the branch would ship never having executed +comment,,,,,,,the handoff's FAILURE path. no natural trigger exists - every table is filtered to ones defined at root before di.pubsub sees them - so it is reached by replacing the module handle init.q captured. without this the branch would ship never having executed +comment,,,,,,,it must WARN and let the subscribe stand. by the time the handoff runs the subscribe has fully succeeded - schemas defined, replay done, registry committed - so throwing would discard completed work over an optional secondary step, and telling the caller to close the handle would be wrong advice: a retry would hit guardduplicate against the row this call just committed run,0,0,q,sub.init[depsrepub[]],1,1,republish on run,0,0,q,savedpubsub:.m.di.0subscriptions.pubsub,1,1,keep the real handle run,0,0,q,"`.m.di.0subscriptions.pubsub set `setsubtables`init`getsubtables!({[x] '""publisher exploded""};{[] };{[] `symbol$()})",1,1,swap in a publisher whose setsubtables throws run,0,0,q,resetcap[],1,1,clear the captured log run,0,0,q,"lhx:buildlog[""hx"";`hx;2]",1,1,build a log -fail,0,0,q,sub.subscribe[tphfor[`hx;2;lhx];enlist`hx;`;1b;1b],1,1,a failing handoff makes subscribe signal rather than pass silently -true,0,0,q,`handoffpublisher in exec ctx from caprows where lvl=`error,1,1,"logged under the handoffpublisher context - the logged[] fixture matches message text, the function name is the ctx" -true,0,0,q,"logged[`error;""failed to register subscribed tables""]",1,1,and the message says what failed -true,0,0,q,"logged[`error;""already registered""]",1,1,"and carries registerednote - the handoff fires AFTER the tickerplant has registered this handle, so the caller must be told to close it" -true,0,0,q,1=count select from sub.getsubscriptions[] where tabs~\:enlist`hx,1,1,"KNOWN ORDERING - the subscription IS registered even though subscribe threw. the handoff runs last, after the registry insert, which is exactly why the error carries registerednote" +run,0,0,q,rhx:sub.subscribe[tphfor[`hx;2;lhx];enlist`hx;`;1b;1b],1,1,the subscribe RETURNS - a failing handoff no longer signals +true,0,0,q,99h=type rhx,1,1,"and returns the normal buildreturn dict, not some degraded value" +true,0,0,q,(enlist`hx)~rhx`subtables,1,1,reporting the table it actually subscribed to +true,0,0,q,2=count hx,1,1,the replay really happened - the work the throw would have discarded +true,0,0,q,1=count select from sub.getsubscriptions[] where tabs~\:enlist`hx,1,1,and the registry row is committed and usable +true,0,0,q,`handoffpublisher in exec ctx from caprows where lvl=`warn,1,1,the failure was logged as a WARN under the handoffpublisher context +true,0,0,q,"logged[`warn;""publisher exploded""]",1,1,carrying the underlying reason +true,0,0,q,"logged[`warn;""subscribe itself succeeded""]",1,1,and saying plainly that the subscribe stands +true,0,0,q,0=count select from caprows where lvl=`error,1,1,"THE DISCRIMINATING ASSERTION - nothing was logged at error level. this is what separates the new behaviour from the old, where raiseerror logged an error and threw" +true,0,0,q,"not logged[`warn;""already registered""]",1,1,and registerednote is NOT applied - its remedy (close the handle) describes a different failure and would be wrong here run,0,0,q,`.m.di.0subscriptions.pubsub set savedpubsub,1,1,restore the real publisher -run,0,0,q,sub.unsubscribe[tphfor[`hx;2;lhx]],1,1,release the row the failed subscribe left behind +run,0,0,q,sub.unsubscribe[tphfor[`hx;2;lhx]],1,1,"release the subscription, which is live and healthy" comment,,,,,,,resubscribe does NOT re-run the handoff - it re-establishes existing subscriptions over a new handle and never changes the table set, and the handoff is additive so the publisher already has them run,0,0,q,sub.init[depsrepub[]],1,1,republish on From 7515c4d286e47715f956dad3824ce288d12da6cc Mon Sep 17 00:00:00 2001 From: alowrydi Date: Fri, 14 Aug 2026 16:19:47 +0100 Subject: [PATCH 08/10] Addressing AIReviewBot comments --- di/pubsub/pubsub.q | 4 +- di/subscriptions/subscriptions.md | 12 ++++-- di/subscriptions/subscriptions.q | 60 ++++++++++++++++++++------- di/subscriptions/test.csv | 30 ++++++++++++++ di/subscriptions/test_integration.csv | 3 +- 5 files changed, 88 insertions(+), 21 deletions(-) diff --git a/di/pubsub/pubsub.q b/di/pubsub/pubsub.q index f95839b7..144339b9 100644 --- a/di/pubsub/pubsub.q +++ b/di/pubsub/pubsub.q @@ -148,7 +148,9 @@ getsubtables:{[] / REPLACES the list - a consumer that needs to ADD to the publish set has no other way to learn the / current one, and reaching into module state from outside is not an interface. / empty until init has run, rather than signalling on an unset name - :@[{[x] t};::;{[e] `symbol$()}]; + / read .z.m.t EXPLICITLY - a bare t would resolve to the same module state, but the explicit form is + / the one qlint accepts and matches how every other module reads its own state + :@[{[] .z.m.t};::;{[e] `symbol$()}]; }; setsubtables`; diff --git a/di/subscriptions/subscriptions.md b/di/subscriptions/subscriptions.md index ff51e017..b578f0e2 100644 --- a/di/subscriptions/subscriptions.md +++ b/di/subscriptions/subscriptions.md @@ -118,6 +118,12 @@ Re-establish every subscription that has since dropped, over a **new** handle to sub.resubscribe[newtph] / -> `trade`quote ``` +It reports what `subscribe` **actually** established, not what the dead row asked for. The two differ: if the tickerplant has since retired one table of a multi-table subscription, `narrowtabs` drops it with a warning and the call still succeeds for the rest. Taking the request as the outcome would mark the retired table re-established and retire its row, losing the only record that it was ever subscribed. + +A dead row is therefore **narrowed** rather than deleted whole — the tables that came back are removed from it, and the row is dropped only once nothing is left. Deleting whole would discard the tables that did *not* come back; keeping whole would retry them alongside the ones that did, which now hold a live subscription, so the duplicate guard would reject the retry and warn about it on every call. A narrowed row is retried only while this tickerplant still publishes what remains in it, so a table retired upstream goes quiet instead of warning on every timer tick, and is picked up again automatically if the tickerplant starts publishing it once more. + +Only the rows a given call actually attempted are rewritten, so a same-named table belonging to a *different* tickerplant's dead row is left alone. + ### `unsubscribe[tph]` Release the subscriptions held on a tickerplant handle and return the tables released. **Call this before `hclose`.** It never closes the handle — the caller owns the connection — and never messages the tickerplant, because the `subdetails` protocol has no unsubscribe verb. @@ -289,7 +295,7 @@ loaded and would re-run against dirty module state, reporting spurious failures. self-contained: it inits `di.handlers` and `di.subscriptions` itself rather than inheriting setup from the unit rows. -Together the two suites are 665 assertions (`true`, `fail` and `run` rows, excluding the fixture `before`/`after` rows) and wire the real merged `di.log` and `di.handlers` rather than mocks, so both injected contracts are proven end to end. Between them they cover: pre-`init` guards on every export; dependency validation (non-dict deps, each missing key, each malformed value); `init` arity asserted by observable effect rather than by "it did not throw"; idempotent re-init; the `VERSION` file and exported `version`; `getapimeta`; all-tables and sym-filtered replay; all four payload shapes a tickerplant may log; symbol-atom table and sym selectors; a multi-table log where only the requested table is defined; replay across two log files; requested-but-absent and offered-but-unrequested tables; a configured `subdetailsfunc`; partial, unreadable, absent and empty logs; a replay requested with no root `upd`; malformed `schemalist` **and** `logfilelist` shapes, including a non-table schema, a column-less table, a duplicated table name, and a keyed schema that must still be accepted; a negative message count, asserted with `replay:0b` as well so the guard cannot drift behind the replay branch; every `setschema`/`replay` combination, including `setschema:0b` with `replay:1b` on both the narrowed and all-syms paths; empty `tabs` and `syms` selectors; the double-subscribe guard and its release by `unsubscribe`; `unsubscribe` deleting rather than flagging, its idempotency, and its `warn` on an unheld handle; malformed tickerplant responses; `setschema:0b` preserving a populated table; `logdir` passthrough; and input validation on every argument. +Together the two suites are 693 assertions (`true`, `fail` and `run` rows, excluding the fixture `before`/`after` rows) and wire the real merged `di.log` and `di.handlers` rather than mocks, so both injected contracts are proven end to end. Between them they cover: pre-`init` guards on every export; dependency validation (non-dict deps, each missing key, each malformed value); `init` arity asserted by observable effect rather than by "it did not throw"; idempotent re-init; the `VERSION` file and exported `version`; `getapimeta`; all-tables and sym-filtered replay; all four payload shapes a tickerplant may log; symbol-atom table and sym selectors; a multi-table log where only the requested table is defined; replay across two log files; requested-but-absent and offered-but-unrequested tables; a configured `subdetailsfunc`; partial, unreadable, absent and empty logs; a replay requested with no root `upd`; malformed `schemalist` **and** `logfilelist` shapes, including a non-table schema, a column-less table, a duplicated table name, and a keyed schema that must still be accepted; a negative message count, asserted with `replay:0b` as well so the guard cannot drift behind the replay branch; every `setschema`/`replay` combination, including `setschema:0b` with `replay:1b` on both the narrowed and all-syms paths; empty `tabs` and `syms` selectors; the double-subscribe guard and its release by `unsubscribe`; `unsubscribe` deleting rather than flagging, its idempotency, and its `warn` on an unheld handle; malformed tickerplant responses; `setschema:0b` preserving a populated table; `logdir` passthrough; a `setschema:0b` replay whose request names a table that is absent at root *and* unpublished, proving the tables-exist check follows the narrowing rather than the raw request; the full life cycle of a partially re-established subscription — narrowed on the way back, skipped silently while the tickerplant no longer publishes what remains, and retired only once the remainder returns; and input validation on every argument. It also covers the TorQ-protocol cases this module is built to survive: a `0W` message count replaying a whole log, and the same count over a **truncated** log being refused with nothing replayed and no table defined; a **shared** log reported once per table being collapsed to one full replay, with the per-table row counts asserted individually so a regression to the old per-entry replay is caught rather than merely the total; an unshared file in the same response keeping its own count and raising no warn; a corrupt shared log refused, proving the corruption guard covers the collapse trigger and not only the `0W` sentinel; an exact duplicate entry rejected; `` ` `` resolved through `tablelist` with the *resolved* list asserted to be what `subdetails` actually received; the fallback to `` ` `` when a tickerplant offers no `tablelist`; a configured `tablelistfunc` asserted to be called **first**, ahead of `subdetails`; and the two legitimate `rowcounts` shapes accepted with atoms and tables rejected. @@ -324,13 +330,13 @@ The integration block subscribes to a genuinely separate tickerplant process ove - **One `tablelist` round trip serves both jobs**, and it runs before every guard because it is *pure* — `tablelist` is `{.stpps.t}` on both shipped producers and registers nothing, unlike `subdetails`. When the tickerplant cannot answer it, `` ` `` is sent onward and an explicit request goes unnarrowed, exactly as before; that is reported at `warn` only when the `` ` `` sentinel genuinely had to be resolved, and at `info` on the explicit path where the round trip is a safety net rather than a requirement - Liveness needs three signals, and the measured behaviour of each is what dictates the design. On KDB-X 5f: a handle the process `hclose`s itself fires **no** `.z.pc` and leaves `.z.W`; a remote death fires `.z.pc` with the handle; and a closed descriptor **is** reissued to the next connection (`4` → `4`). So `.z.pc` covers tickerplant death, `.z.W` covers a local close *until the number is reused*, and `unsubscribe` covers the reuse case by clearing the stored flag outright. `.z.pc` is why `handlers` is a required dependency; `.z.W` alone is not sufficient, because a recycled number would revive a stale registry row - The double-subscribe guard compares the **requested** table list — the caller's list, or the concrete list `` ` `` was resolved to — against what is already held, and it runs *before* the tickerplant is asked for anything. Re-subscribing to a table you already hold is a caller mistake whatever the tickerplant happens to offer that round; comparing against the *offered* set instead let the same mistake through with only a `warn` whenever the tickerplant had also stopped offering the table, which was an accident of ordering rather than a considered exception. The guard runs a second time against the offered set, for the one request that cannot be resolved up front (see below). An all-tables subscribe is still refused only where it genuinely overlaps a live subscription, not merely because some other subscription exists -- **Asking a tickerplant for its schemas subscribes you to it.** `subdetails` calls `.ps.subscribe`, which *is* `.u.sub`, so the one call that returns the schemas also registers the handle for live delivery — and the protocol has no unsubscribe verb to undo it. `unsubscribe` here is local bookkeeping only; tickerplant-side release is driven by `.z.pc`, which nothing but closing the handle triggers. Every guard that does not need the reply therefore runs **ahead** of that call: the duplicate guard, the root-`upd` check, and the tables-exist check for `setschema:0b`. That is legacy's own ordering — `reducesubs` runs against a `tablelist` round trip before `subfunc` — restored rather than invented, and it is what makes the `tablelist` round trip load-bearing for a second reason beyond the `` ` `` sentinel +- **Asking a tickerplant for its schemas subscribes you to it.** `subdetails` calls `.ps.subscribe`, which *is* `.u.sub`, so the one call that returns the schemas also registers the handle for live delivery — and the protocol has no unsubscribe verb to undo it. `unsubscribe` here is local bookkeeping only; tickerplant-side release is driven by `.z.pc`, which nothing but closing the handle triggers. Every guard that does not need the reply therefore runs **ahead** of that call: the duplicate guard, the root-`upd` check, and the tables-exist check for `setschema:0b`. That is legacy's own ordering — `reducesubs` runs against a `tablelist` round trip before `subfunc` — restored rather than invented, and it is what makes the `tablelist` round trip load-bearing for a second reason beyond the `` ` `` sentinel. The tables-exist check runs after the request has been narrowed but still ahead of `subdetails`, which costs nothing — the `tablelist` round trip behind the narrowing is pure and registers nothing — and avoids demanding a table at root that the narrowing was about to drop anyway - **What is left of that window, precisely.** With `replay:0b` it is closed for every subscribe that names its tables, and for an all-tables subscribe whenever the tickerplant answers `tablelistfunc` — which both shipped producers do. It stays open only for an all-tables subscribe against a tickerplant offering no usable `tablelist`, where the request cannot be resolved before the call. With `replay:1b` one case is **irreducible**: the log preflight. The log file names exist only in the reply and their integrity can only be established by reading them, so a log that is short, corrupt or unreadable is necessarily discovered *after* the handle is registered. `subscribe` then throws and records no registry row, while the tickerplant goes on publishing into the caller's root `upd` — so the module's own view and reality diverge. The error raised in that case says so explicitly and names the only remedy the protocol leaves: close the handle before retrying. No extra registry state is kept for it; the case is narrow and a third liveness state would cost every reader of `getsubscriptions[]` more than it would buy - Tables are created at root with `@[`.;name;:;schema]`, and `-11!` resolves `upd` at root even when invoked from module context. A root-level `upd` must therefore be defined before calling `subscribe` with `replay` set - this is **enforced**, not merely expected: without it every replayed message would be silently discarded and the narrowed path would leave its no-op stand-in bound at root, so the live feed would vanish into it too - A malformed `subdetails` response is rejected before anything is defined, including a `schemalist` that is not a list or whose entries are not `(tablename;schema)` pairs. An empty entry is tolerated, as legacy does - The *schema* half of each entry must actually be a table. This is not pedantry: `createtables` uses `@[`.;name;:;schema]`, which succeeds for **any** value, so without the check a tickerplant sending a dict or an atom would have it planted at root under the caller's table name and `subscribe` would report success over a root name that is not a table. The test is `.Q.qt` rather than `98h=type`, deliberately — a **keyed** table is `99h` and must still be accepted, while a column-less `([])` is also `99h` and must not be, since it has no columns to replay into. The guard also catches a shape that is not a garbled schema at all: TorQ's `.u.sub` answers a table it does not publish with `` (name;"Table … not in list of stp pub/sub tables") ``, so a **string** in the schema position is the tickerplant declining the table, and the error message says so - A `schemalist` naming the same table more than once is rejected rather than deduplicated. The duplicate would otherwise flow straight into `subtables` — which `di.rdb` and `di.wdb` iterate over — and into the registry's `tabs`. Deduplicating would hide a tickerplant bug; every other malformed response here fails loud, so this does too -- `setschema:0b` skips table definition entirely, so a subscriber that pre-loads its own schema keeps it. With `setschema:1b` the returned schema replaces whatever is at root, which is why re-subscribing a live table is refused. Combining `setschema:0b` with `replay:1b` requires the tables to exist at root already, and that is **checked** — otherwise the replay fails inside the caller's own `upd` on the all-syms path, or from `cols get t` on the narrowed one, neither of which reaches the log +- `setschema:0b` skips table definition entirely, so a subscriber that pre-loads its own schema keeps it. With `setschema:1b` the returned schema replaces whatever is at root, which is why re-subscribing a live table is refused. Combining `setschema:0b` with `replay:1b` requires the tables to exist at root already, and that is **checked** — otherwise the replay fails inside the caller's own `upd` on the all-syms path, or from `cols get t` on the narrowed one, neither of which reaches the log. The check is against the tables actually being subscribed to, not the caller's raw list: a table the tickerplant does not publish is dropped with a warning and never becomes part of the subscription, so it is not required at root. The check runs twice — once on the narrowed request before the tickerplant is asked, and once on the confirmed set after it replies, which still catches a table the `tablelist` advertised but `schemalist` then declined - **After `teardown`, a new `subscribe` is refused — reading and releasing still work.** `teardown` removes the `.z.pc` observer, and without it a dropped tickerplant's registry row keeps reporting live for as long as `.z.W` still holds the handle number, and *indefinitely* once kdb+ reissues that number to another connection. That is exactly the failure the `handlers` dependency exists to prevent, so taking a **new** subscription in that state now throws and names the recovery (`call init again`) rather than degrading silently into the `.z.W`-only mode this module documents as insufficient. `getsubscriptions`, `subscribed` and `unsubscribe` deliberately keep working, because a shutdown path needs to inspect and release what it already holds. `init` is idempotent and restores the observer. This reverses an earlier deliberate choice — the previous behaviour let the subscribe through and accepted degraded liveness tracking — because silent degradation is inconsistent with how every other guard in this module behaves - **The root `upd` must accept two payload shapes, and which one it gets depends on the subscription.** An unfiltered replay hands `upd` the logged payload exactly as the tickerplant wrote it — normally a list of columns. A **sym-filtered** replay hands it a **table**, because `replayfilter` has to materialise one to filter by column name rather than by position. Legacy's `replayupd` behaves the same way, so this is inherited rather than introduced, but it is a real contract: an `upd` written only for the column-list shape throws part-way through a filtered replay, after earlier messages have already been applied. Write it as `` {[t;x] @[`.;t;{[tab;d] tab upsert $[98h=type d;d;flip (cols tab)!d]}[;x]]} ``, which is what the usage example and the test fixtures use. The live feed is unaffected — it never passes through `replayfilter` - **Live-feed filtering is the tickerplant's job, not this module's.** `replayfilter` is installed only for the duration of a narrowed replay and torn down the moment it ends; live messages after that reach the caller's `upd` unfiltered by anything here. That is correct, not a leak: the `subdetails` call itself registers the handle for tickerplant-side filtered delivery as a *side effect*. In shipped TorQ, `subdetails` calls `.ps.subscribe`, which **is** `.u.sub`, which routes through `suball`/`subfiltered` to `selfiltered` - and that records `(tbl;handle;filts;columns)` with `filts` built as `` (in;`sym;syms) ``, which `pub` then evaluates per subscriber. Replay needs its own filter only because it bypasses the publish layer entirely and reads the log file directly. Without this note a reader seeing `replayfilter` removed right after the replay would reasonably read it as a bug diff --git a/di/subscriptions/subscriptions.q b/di/subscriptions/subscriptions.q index 03c367a8..f6fe2d9b 100644 --- a/di/subscriptions/subscriptions.q +++ b/di/subscriptions/subscriptions.q @@ -770,19 +770,28 @@ subscribe:{[tph;tabs;syms;setschema;replay] / need a concrete list, so that one case is skipped here and caught by the copies below if[replay;requirerootupd[]]; if[not requested~`;guardduplicate[requested]]; + / what we actually SEND. an explicit request is narrowed to what the tickerplant publishes, because + / one unpublished name fails the whole subdetails call - see narrowtabs + sendtabs:$[alltabs or published~`;requested;narrowtabs[tabs;published]]; / the tables-exist check runs early only for an EXPLICIT request. on the all-tables path / `requested` is the TABLELIST list, which may advertise more than schemalist actually returns, and / this guard asserts that EVERY name exists at root - so a superset there would refuse a perfectly / valid subscribe. the duplicate guard above is unaffected by the same superset, because it only / bites where the list INTERSECTS a table already held, and that is a caller mistake either way. / the all-tables case is covered by the post-reply copy against `wanted`, once the reply has said - / what is really on offer + / what is really on offer. + / it checks SENDTABS, not the caller's raw list: narrowtabs has already dropped anything the + / tickerplant does not publish, with a warn rather than a failure, and those tables were never going + / to be part of the subscription - so requiring them at root would throw for a table the caller + / never needed to hold. this runs AFTER narrowtabs but still BEFORE fetchdetails, which is what + / matters: the tablelist round trip behind `published` is pure and registers nothing, whereas + / fetchdetails registers this handle for live delivery as a side effect + / sendtabs is the closest approximation to `wanted` obtainable before the reply, not an equal one - + / it can still be a superset when schemalist omits a table the tablelist advertised, which is the + / gap the post-reply copy below continues to cover if[replay; if[not alltabs; - if[not setschema;requiretablesexist[requested]]]]; - / what we actually SEND. an explicit request is narrowed to what the tickerplant publishes, because - / one unpublished name fails the whole subdetails call - see narrowtabs - sendtabs:$[alltabs or published~`;requested;narrowtabs[tabs;published]]; + if[not setschema;requiretablesexist[sendtabs]]]]; details:fetchdetails[tph;sendtabs;syms]; schemapairs:(details`schemalist) where not 0=count each details`schemalist; offered:(),schemapairs[;0]; @@ -886,7 +895,11 @@ resubscribe:{[tph] requireinit[`resubscribe]; requireobserver[`resubscribe]; requirehandle[`resubscribe;tph]; - dead:select tabs,syms from activesubscriptions[] where not active; + / idx: ALIAS the virtual index column - a bare `i` in the select list lands as a column named `x`. + / it carries each dead row's position in .z.m.subscriptions, so the cleanup below can rewrite + / exactly the rows this call attempted. activesubscriptions is an update over the registry, so it + / preserves row order and count and the index maps 1:1 + dead:select idx:i,tabs,syms from activesubscriptions[] where not active; if[0=count dead; .z.m.loginfo[`resubscribe;"no dropped subscription to re-establish"]; :`$()]; @@ -902,22 +915,37 @@ resubscribe:{[tph] if[0=count dead; .z.m.loginfo[`resubscribe;"no dropped subscription this tickerplant can serve"]; :`$()]; + / report what subscribe ACTUALLY established, not what this row asked for. narrowtabs drops a table + / the tickerplant no longer publishes with a warn rather than a failure, so a multi-table row can + / succeed having re-established only some of its tables - and taking the request as the outcome + / would then mark the dropped one done and delete it below, losing every trace of it with nothing + / left to retry it. subtables is the narrowed list subscribe actually registered (see buildreturn) done:raze {[tph;t;s] - r:@[{[tph;t;s] subscribe[tph;t;s;0b;0b]; (1b;t)}[tph;t];s;{[e] (0b;e)}]; + r:@[{[tph;t;s] res:subscribe[tph;t;s;0b;0b]; (1b;res`subtables)}[tph;t];s;{[e] (0b;e)}]; if[not first r; .z.m.logwarn[`resubscribe;"could not re-establish ",(", " sv string (),t),": ",last r]; :`$()]; - :(),t}[tph]'[dead`tabs;dead`syms]; + :(),r 1}[tph]'[dead`tabs;dead`syms]; done:distinct (),done; if[0/dev/null 2>&1 &""; `MIDPORT set readport[MIDDIR,""/port"";10000]; if[null MIDPORT;'""test: middle tickerplant never reported a port""]; };",1,1,"fixture: spawnmid - OS-assigned port, read back from the middle process" before,0,0,q,"midhandle:{[] :hopen (`$"":localhost:"",string MIDPORT;2000); };",1,1,fixture: midhandle +before,0,0,q,"waitready:{[h;timeoutms] deadline:.z.p+`timespan$1000000*timeoutms; r:0b; while[(.z.p Date: Tue, 18 Aug 2026 17:46:58 +0100 Subject: [PATCH 09/10] di.pubsub: callendofperiod is ternary, matching the producer TorQ's code/common/pubsub.q:19 broadcasts (`endofperiod;x;y;z) and both of its subscribers - code/rdb/endofperiod.q and code/wdb/writedown.q:52 - are {[currp;nextp;data]}. Ours was {(neg getallhandles[])@\:(`endofperiod;x)}, which failed two ways at once, both measured: callendofperiod[c;n;d] -> 'rank, so a caller following that contract could not call it at all callendofperiod[c] -> a ternary subscriber is left PARTIALLY APPLIED. q returns a projection, the body never runs, and nothing throws, logs or comes back to say so. Same defect class as the callendofday bug fixed in #118. callendofday deliberately stays UNARY. TorQ sends (`endofday;x;y), but the second argument is processdata, which legacy's own rdb never reads and the shipped .u.end alias passes ()!() for - di.rdb's endofday is unary to match. Fixing it "for symmetry" would projection-ise every unary subscriber, so the asymmetry is documented in pubsub.md rather than left looking like an oversight. 15 regression rows added. Negative control: against the unary version they fail exactly 3 - the ternary call on 'rank, plus both subscriber assertions - and nothing else. --- di/pubsub/pubsub.md | 32 ++++++++++++++++++++++++++++++-- di/pubsub/pubsub.q | 13 +++++++++++-- di/pubsub/test.csv | 16 ++++++++++++++++ 3 files changed, 57 insertions(+), 4 deletions(-) diff --git a/di/pubsub/pubsub.md b/di/pubsub/pubsub.md index 75c5d24f..a3a8d3bd 100644 --- a/di/pubsub/pubsub.md +++ b/di/pubsub/pubsub.md @@ -50,8 +50,8 @@ publish data with/without filters. The function takes two arguments: t and x, wh |---------------------------|------------------------------------------------------------------------------| | `pubsub.setsubtables` | Set a specified list of tables that are available for subscription. | | `pubsub.getsubtables` | Read the list of tables currently available for subscription - the counterpart to `setsubtables`, which replaces it. Empty until `init` has run. | -| `pubsub.callendofday` | Broadcast an end-of-day event to all subscribers (requires `endofday`). | -| `pubsub.callendofperiod` | Broadcast an end-of-period event to all subscribers (requires `endofperiod`).| +| `pubsub.callendofday` | Broadcast an end-of-day event to all subscribers (requires `endofday`). **Unary**: `callendofday[date]`. | +| `pubsub.callendofperiod` | Broadcast an end-of-period event to all subscribers (requires `endofperiod`). **Ternary**: `callendofperiod[currentperiod;nextperiod;data]`. | | `pubsub.closesub` | Remove handle upon connection close. | | `pubsub.subclear` | Publish tables and clear up the contents. | | `pubsub.init` | Initialize variables - run before calling pub/sub functions to populate required state (e.g., tables/schemas). | @@ -103,3 +103,31 @@ q)pubsub.subscribestrfilter["quote";"bid>50.0";"time,sym,bid"] - By default, all tables on top level of the process are available for subscription. - The user should define the `.u.sub` and the `.u.pub` functions within the process. - The module initializes with defined list of tables to subscribe to and fetches their schemas and columns for use. This is done via calling `init` function. + +--- + +### End-of-day and end-of-period arity + +These two broadcasts deliberately have **different arities**, which looks like an inconsistency and +is not: + +| function | arity | broadcast | +|---|---|---| +| `callendofday` | unary | `` (`endofday;date) `` | +| `callendofperiod` | ternary | `` (`endofperiod;currentperiod;nextperiod;data) `` | + +`callendofperiod` matches TorQ exactly - `code/common/pubsub.q:19` sends all three, and both shipped +subscribers (`code/rdb/endofperiod.q`, `code/wdb/writedown.q:52`) are `{[currp;nextp;data]}`. + +`callendofday` deliberately **diverges** from TorQ, which sends `` (`endofday;x;y) ``. That second +argument is `processdata`; legacy's own rdb never reads it, and the shipped `.u.end` alias passes +`()!()` for it. Subscribers here are unary to match. Do not "fix" it for symmetry with +`callendofperiod` - doing so would turn every unary `endofday` subscriber into a projection. + +That projection failure is the reason this matters, and it is completely silent. A subscriber whose +arity does not match what is broadcast is **partially applied**: q returns a projection, the body +never runs, and nothing throws, logs, or comes back to say so. `callendofperiod` was previously +unary, which failed both ways at once - a `callendofperiod[c;n;d]` call threw `'rank`, so a caller +following TorQ's contract could not call it at all, while the one-argument form silently no-opped +every ternary subscriber. Both measured; both covered by the suite, whose two assertions fail +against the unary implementation. diff --git a/di/pubsub/pubsub.q b/di/pubsub/pubsub.q index 144339b9..24fe7d28 100644 --- a/di/pubsub/pubsub.q +++ b/di/pubsub/pubsub.q @@ -88,8 +88,17 @@ priorpc:@[value;`.z.pc;{[e] (::)}]; / broadcast to all subscribers upon end of day, client needs to define endofday function callendofday:{[d](neg getallhandles[])@\:(`endofday;d)}; -/ broadcast to all subscribers upon end of period, client needs to define endofperiod function -callendofperiod:{(neg getallhandles[])@\:(`endofperiod;x)}; +/ broadcast to all subscribers upon end of period, client needs to define endofperiod function. +/ TERNARY, matching legacy: TorQ's code/common/pubsub.q:19 broadcasts (`endofperiod;x;y;z) and both +/ of its subscribers (code/rdb/endofperiod.q, code/wdb/writedown.q:52) are {[currp;nextp;data]}. +/ it was unary, which failed two ways at once (both measured): callendofperiod[c;n;d] threw 'rank, so +/ a caller following that contract could not call it at all, and the one-argument form left a ternary +/ subscriber as a PROJECTION - the body never ran, and nothing threw, logged or was returned to say so. +/ same defect class as the callendofday bug PR #118 fixed. +/ NB callendofday stays UNARY on purpose. TorQ's producer sends (`endofday;x;y), but its second +/ argument is processdata, which legacy's own rdb never reads and the shipped .u.end alias passes +/ ()!() for - di.rdb's endofday is unary to match. Do not "fix" that one for symmetry with this +callendofperiod:{[currentperiod;nextperiod;data](neg getallhandles[])@\:(`endofperiod;currentperiod;nextperiod;data)}; / get table schema extractschema:{[table]0#value table}; diff --git a/di/pubsub/test.csv b/di/pubsub/test.csv index 7f474e17..504489d2 100644 --- a/di/pubsub/test.csv +++ b/di/pubsub/test.csv @@ -147,6 +147,22 @@ run,0,0,q,res:h"select from .m.di.0pubsub.reqfilteredtbl where table=`quote",1,, run,0,0,q,res:select from res where handle=clt,1,,check filtered table for the subscription true,0,0,q,res[`filts][0]~enlist enlist(>;`bid;50f),1,, test symbols are listed under filters +comment,,,,,,,"callendofperiod is TERNARY, matching TorQ's producer: code/common/pubsub.q:19 sends" +comment,,,,,,,"(`endofperiod;x;y;z) and both subscribers (code/rdb/endofperiod.q, code/wdb/writedown.q:52)" +comment,,,,,,,"are {[currp;nextp;data]}. it was unary - a 3-arg call threw 'rank, and the 1-arg form left a" +comment,,,,,,,ternary subscriber as a PROJECTION: the body never ran and nothing threw or logged +run,0,0,q,.test.EOPRAN:0b,1,1,did the subscriber body actually run +run,0,0,q,.test.EOPGOT:`seeded,1,1,and what did it receive +run,0,0,q,endofperiod:{[c;n;dat] .test.EOPRAN:1b; .test.EOPGOT:(c;n;dat); },1,1,a TERNARY subscriber - the shape TorQ's rdb and wdb both use +run,0,0,q,h(`.m.di.0pubsub.callendofperiod;09:30:00;10:00:00;`a`b!1 2),1,1,broadcast a period roll from the server +run,0,0,q,h"1+1",1,1,round trip so the async broadcast is processed before we assert +true,0,0,q,.test.EOPRAN,1,1,the subscriber BODY ran - this is the assertion that fails on the unary version +true,0,0,q,.test.EOPGOT~(09:30:00;10:00:00;`a`b!1 2),1,1,all three arguments arrived intact +run,0,0,q,.test.EOPRAN:0b,1,1,reset before the one-argument case +run,0,0,q,h(`.m.di.0pubsub.callendofperiod;09:30:00),1,1,the old one-argument calling shape +run,0,0,q,h"1+1",1,1,round trip +true,0,0,q,not .test.EOPRAN,1,1,a one-argument call is a projection server side and broadcasts nothing so no malformed endofperiod reaches subscribers + run,0,0,q,neg[h](exit;0);neg[h](::),1,,close background process and flush run,0,0,q,hclose h,1,,close handle run,0,0,q,"system""rm -rf "",PSDIR",1,1,remove the server directory From f60a20220245fdbd6b6d89a5ea57cd024b9cd5be Mon Sep 17 00:00:00 2001 From: alowrydi Date: Wed, 19 Aug 2026 16:38:12 +0100 Subject: [PATCH 10/10] di.subscriptions: adapt getsubscriptionhandles to di.servers' init guard di.servers now refuses every accessor until its own init has run (requireinit, servers.q:50). That is deliberate on its side - a pre-init getservers used to return an empty table, indistinguishable from "nothing is connected" - but the signal it raises names only di.servers and reaches only di.servers' logger, so it travelled straight through this module: a di.subscriptions call handed back a di.servers error and nothing landed in this module's log. getsubscriptionhandles now catches it and re-raises through raiseerror, so the caller learns which of the two modules is unwired and the failure is observable here like every other domain error in the module. Returning an empty table instead was considered and rejected: "I cannot tell you" is not the same answer as "no handles", and a caller that treats them alike waits for a tickerplant that was never going to appear. Only the getservers[`] call is wrapped. The proctype call below it is reachable only once the first has succeeded, so init is known to have run by then. 8 unit rows added, placed ABOVE the di.servers init row - once that module is initialised the pre-init path cannot be reached again in the same process. They assert all three things that matter separately: the error carries this module's context, it names di.servers as the module needing wiring, and it was logged here rather than bypassing this log. Driving the real getservers also means the suite must init di.servers itself, then tear it down immediately: init registers a .z.pc observer and a serversretry timer job, and this suite fires .z.pc by hand (markdeadfor) against handles that overlap the seeded SERVERS rows. teardown releases both while leaving the accessors callable, which is what di.servers' separate registered/initialised flags are for. Integration suite: di.servers also now requires `deletejobs on the timer dep - teardown deletes its retry job through it, and it is validated as callable at init rather than left to fail inside a protected apply - so the mocktimer stub gained one. Without it svc.init threw and took the whole 12-row di.servers end-to-end block with it. Suites: 570 unit + 131 integration = 701, all green. Was 562 + 131 with the integration block failing 12. --- di/subscriptions/subscriptions.md | 14 ++++++++++++-- di/subscriptions/subscriptions.q | 10 +++++++++- di/subscriptions/test.csv | 17 +++++++++++++++++ di/subscriptions/test_integration.csv | 2 +- 4 files changed, 39 insertions(+), 4 deletions(-) diff --git a/di/subscriptions/subscriptions.md b/di/subscriptions/subscriptions.md index b578f0e2..64460226 100644 --- a/di/subscriptions/subscriptions.md +++ b/di/subscriptions/subscriptions.md @@ -180,6 +180,16 @@ So `[`tickerplant;()]` unions the proctype matches with nothing and yields the t - Legacy took a third `attributes` argument and filtered on `.servers.SERVERS`'s `attributes` column. `di.servers`' `SERVERS` carries no such column, so the parameter is **dropped** rather than accepted and silently ignored — a filter that quietly does nothing hands back handles the caller believes were filtered. All four legacy call sites pass `()!()`. - Legacy passed `autoopen:1b` to retry dead connections on demand. `di.servers` returns live rows only and runs its own retry job, so reconnection is its concern now. +**`di.servers` must be initialised first.** Every `di.servers` accessor refuses to run before that module's own `init` — deliberately, because a pre-`init` `getservers` used to return an empty table, which is indistinguishable from "nothing is connected". That refusal signals under the `di.servers` context and through the `di.servers` logger, so left alone it would travel straight past this module: the caller would see a `di.servers` error from a `di.subscriptions` call, and nothing would reach *this* module's log. So it is caught and re-raised through `raiseerror`, naming both modules: + +``` +di.subscriptions: getsubscriptionhandles: could not read the di.servers server list +(di.servers: getservers: init must be called before any other function) - di.servers +must be initialised before subscription handles can be resolved +``` + +Returning an empty table instead would be the wrong call: "I cannot tell you" is not the same answer as "no handles", and a caller that treats them alike waits for a tickerplant that was never going to appear. This is the only place the module reaches into another module's failure and rewrites it, and it does so only to add context, never to change the outcome. + ### `teardown[]` Remove the `.z.pc` registration installed by `init`, leaving no process-global residue. Afterwards `subscribe` is refused until `init` is called again — see Notes. `getsubscriptions`, `subscribed` and `unsubscribe` keep working, so a shutdown path can still inspect and release what it holds, and the registry is left intact. ```q @@ -251,7 +261,7 @@ Two suites, following the convention `di.handlers` and `di.permissions` use. Bot fixtures inline as `before` rows; there is no separate fixture file, so neither depends on the working directory. -**Unit suite** (`test.csv`) - 534 assertions, no child processes, no ports, no `QHOME`. It does need `di.servers` on `QPATH`, since the module imports it, and it drives the **real** `di.servers` rather than a mock: the fixtures seed its `SERVERS` registry directly, so the whole selection matrix runs against the genuine `getservers` without opening a socket: +**Unit suite** (`test.csv`) - 570 assertions, no child processes, no ports, no `QHOME`. It does need `di.servers` on `QPATH`, since the module imports it, and it drives the **real** `di.servers` rather than a mock: the fixtures seed its `SERVERS` registry directly, so the whole selection matrix runs against the genuine `getservers` without opening a socket: ```q k4unit:use`di.k4unit @@ -295,7 +305,7 @@ loaded and would re-run against dirty module state, reporting spurious failures. self-contained: it inits `di.handlers` and `di.subscriptions` itself rather than inheriting setup from the unit rows. -Together the two suites are 693 assertions (`true`, `fail` and `run` rows, excluding the fixture `before`/`after` rows) and wire the real merged `di.log` and `di.handlers` rather than mocks, so both injected contracts are proven end to end. Between them they cover: pre-`init` guards on every export; dependency validation (non-dict deps, each missing key, each malformed value); `init` arity asserted by observable effect rather than by "it did not throw"; idempotent re-init; the `VERSION` file and exported `version`; `getapimeta`; all-tables and sym-filtered replay; all four payload shapes a tickerplant may log; symbol-atom table and sym selectors; a multi-table log where only the requested table is defined; replay across two log files; requested-but-absent and offered-but-unrequested tables; a configured `subdetailsfunc`; partial, unreadable, absent and empty logs; a replay requested with no root `upd`; malformed `schemalist` **and** `logfilelist` shapes, including a non-table schema, a column-less table, a duplicated table name, and a keyed schema that must still be accepted; a negative message count, asserted with `replay:0b` as well so the guard cannot drift behind the replay branch; every `setschema`/`replay` combination, including `setschema:0b` with `replay:1b` on both the narrowed and all-syms paths; empty `tabs` and `syms` selectors; the double-subscribe guard and its release by `unsubscribe`; `unsubscribe` deleting rather than flagging, its idempotency, and its `warn` on an unheld handle; malformed tickerplant responses; `setschema:0b` preserving a populated table; `logdir` passthrough; a `setschema:0b` replay whose request names a table that is absent at root *and* unpublished, proving the tables-exist check follows the narrowing rather than the raw request; the full life cycle of a partially re-established subscription — narrowed on the way back, skipped silently while the tickerplant no longer publishes what remains, and retired only once the remainder returns; and input validation on every argument. +Together the two suites are 701 assertions (`true`, `fail` and `run` rows, excluding the fixture `before`/`after` rows) and wire the real merged `di.log` and `di.handlers` rather than mocks, so both injected contracts are proven end to end. Between them they cover: pre-`init` guards on every export; dependency validation (non-dict deps, each missing key, each malformed value); `init` arity asserted by observable effect rather than by "it did not throw"; idempotent re-init; the `VERSION` file and exported `version`; `getapimeta`; all-tables and sym-filtered replay; all four payload shapes a tickerplant may log; symbol-atom table and sym selectors; a multi-table log where only the requested table is defined; replay across two log files; requested-but-absent and offered-but-unrequested tables; a configured `subdetailsfunc`; partial, unreadable, absent and empty logs; a replay requested with no root `upd`; malformed `schemalist` **and** `logfilelist` shapes, including a non-table schema, a column-less table, a duplicated table name, and a keyed schema that must still be accepted; a negative message count, asserted with `replay:0b` as well so the guard cannot drift behind the replay branch; every `setschema`/`replay` combination, including `setschema:0b` with `replay:1b` on both the narrowed and all-syms paths; empty `tabs` and `syms` selectors; the double-subscribe guard and its release by `unsubscribe`; `unsubscribe` deleting rather than flagging, its idempotency, and its `warn` on an unheld handle; malformed tickerplant responses; `setschema:0b` preserving a populated table; `logdir` passthrough; a `setschema:0b` replay whose request names a table that is absent at root *and* unpublished, proving the tables-exist check follows the narrowing rather than the raw request; the full life cycle of a partially re-established subscription — narrowed on the way back, skipped silently while the tickerplant no longer publishes what remains, and retired only once the remainder returns; `getsubscriptionhandles` called before `di.servers` is initialised, asserted on all three counts that matter — the error carries *this* module's context, it names `di.servers` as the module needing wiring, and it lands in this module's log rather than bypassing it; and input validation on every argument. It also covers the TorQ-protocol cases this module is built to survive: a `0W` message count replaying a whole log, and the same count over a **truncated** log being refused with nothing replayed and no table defined; a **shared** log reported once per table being collapsed to one full replay, with the per-table row counts asserted individually so a regression to the old per-entry replay is caught rather than merely the total; an unshared file in the same response keeping its own count and raising no warn; a corrupt shared log refused, proving the corruption guard covers the collapse trigger and not only the `0W` sentinel; an exact duplicate entry rejected; `` ` `` resolved through `tablelist` with the *resolved* list asserted to be what `subdetails` actually received; the fallback to `` ` `` when a tickerplant offers no `tablelist`; a configured `tablelistfunc` asserted to be called **first**, ahead of `subdetails`; and the two legitimate `rowcounts` shapes accepted with atoms and tables rejected. diff --git a/di/subscriptions/subscriptions.q b/di/subscriptions/subscriptions.q index f6fe2d9b..c541393f 100644 --- a/di/subscriptions/subscriptions.q +++ b/di/subscriptions/subscriptions.q @@ -990,7 +990,15 @@ getsubscriptionhandles:{[proctype;procname] / against the parameter of the same name inside the select would silently compare the column / with itself and match every row pn:(),procname; - srvs:servers.getservers[`]; + / di.servers refuses every accessor until its own init has run. that is deliberate on its side - a + / pre-init getservers used to return an empty table, indistinguishable from "nothing is connected" - + / but the raw signal names only di.servers and bypasses THIS module's log, so it is caught and + / re-raised through raiseerror: the caller learns which of the two modules is unwired, and the + / failure is observable in the log like every other domain error here. an empty result is NOT an + / acceptable fallback - "I cannot tell you" is not the same answer as "no handles" + srvs:@[{[] :servers.getservers[`]};::; + {[e] raiseerror[`getsubscriptionhandles;"could not read the di.servers server list (",e, + ") - di.servers must be initialised before subscription handles can be resolved"]}]; bytype:$[0h=type proctype;0#srvs;servers.getservers[proctype]]; byname:$[0h=type procname;0#srvs;$[`~procname;srvs;select from srvs where procname in pn]]; / project BEFORE combining - inter requires identical column sets, and legacy projects first too diff --git a/di/subscriptions/test.csv b/di/subscriptions/test.csv index d864fff1..b68c77b0 100644 --- a/di/subscriptions/test.csv +++ b/di/subscriptions/test.csv @@ -637,6 +637,23 @@ true,0,0,q,2=count fdtab,1,1,and the replay runs UNFILTERED as legacy replayupd comment,,,,,,,getsubscriptionhandles - resolve live tickerplant handles by proctype and/or procname. ported from TorQ .sub.getsubscriptionhandles (code/common/subscriptions.q:11) comment,,,,,,,di.servers is a HARD dependency reached through init.q's `use`, so there is nothing to inject and no absent-dependency case to test. seeding its registry exercises the REAL getservers +comment,,,,,,,"BEFORE di.servers is initialised: getsubscriptionhandles must fail with THIS module context and" +comment,,,,,,,"reach this module log, not propagate a bare di.servers signal past both. these rows must stay" +comment,,,,,,,above the init row below - once di.servers is initialised the path cannot be reached again +run,0,0,q,resetcap[],1,1,clear the log capture +run,0,0,q,"esrv:@[{sub[`getsubscriptionhandles][`;`]};::;{[e] e}]",1,1,call getsubscriptionhandles while di.servers is still uninitialised +true,0,0,q,"0