From 031a21b9283a24ba377672051dc0ac2e445a194e Mon Sep 17 00:00:00 2001 From: ascottDI Date: Tue, 28 Jul 2026 16:19:01 +0100 Subject: [PATCH 1/9] initial creation of servers module plus refinements --- di/servers/init.q | 3 + di/servers/servers.md | 124 +++++++++++++++++++++++ di/servers/servers.q | 231 ++++++++++++++++++++++++++++++++++++++++++ di/servers/test.csv | 31 ++++++ 4 files changed, 389 insertions(+) create mode 100644 di/servers/init.q create mode 100644 di/servers/servers.md create mode 100644 di/servers/servers.q create mode 100644 di/servers/test.csv diff --git a/di/servers/init.q b/di/servers/init.q new file mode 100644 index 00000000..97ceb222 --- /dev/null +++ b/di/servers/init.q @@ -0,0 +1,3 @@ +/ connection management and handle-by-type lookup for the modular torq world. +\l ::servers.q +export:([init;startup;getservers;gethandlebytype;waitfortype;getapimeta]) diff --git a/di/servers/servers.md b/di/servers/servers.md new file mode 100644 index 00000000..b1698e43 --- /dev/null +++ b/di/servers/servers.md @@ -0,0 +1,124 @@ +# di.servers + +Connection management and handle-by-type lookup for the modular TorQ world — the `di.*` +analogue of TorQ's `.servers` (`code/handlers/trackservers.q` + `servers.q`), scoped down +for v1: no discovery service, no password/access-list files, no non-TorQ process tracking, +no FinSpace. `process.csv` here is a static **phone book** (who to *dial*), **not** an +identity source — self-identity comes from config, injected by `di.torq`. + +FRAMEWORK-tier module: no hard `di.*` dependencies; `log`, `timer` and `handlers` are all +**injected** (all required, no fallback). + +## init and config + +Standard **one-arg `init[deps]`**: `di.torq` merges this process's resolved config slice into +the same `deps` dict it passes the injectables in, so `deps` carries both the injectable +dependencies **and** the config keys. `init` wires the deps, records self-identity, and +installs two one-time process-global side effects — a `.z.pc` cleanup handler and a 10s retry +timer job. It is **idempotent** (guarded by an internal `registered` flag): `di.torq` calls +it once per process, but a second call refreshes the dep refs without re-registering (a +duplicate `di.timer.addjob` id would throw). `init` does **not** open connections. + +`deps` keys: + +| key | kind | meaning | +|---|---|---| +| `log` | injectable | binary `` `info`warn`error `` `{[c;m]}` logger dict | +| `timer` | injectable | di.timer contract; `addjob` = the 6-arg `custom` form `{[id;func;params;period;mode;opts]}` | +| `handlers` | injectable | di.handlers contract; `register[event;phase;nm;pri;func]` | +| `proctype`/`procname` | config | this process's own identity (required); used to exclude self from `process.csv` | +| `connections` | config | proctypes this process should dial (symbols, or strings from a `.toml` cascade — normalised). Optional; default = none | +| `processcsv` | config | **path** to `process.csv`; supplied by di.torq. Optional; required only once `connections` is non-empty | + +```q +svc:use`di.servers +svc.init[deps] / deps = injectables + config, assembled by di.torq +svc.startup[] / open the configured connections (reads init config) +h:svc.gethandlebytype[`hdb;`any] +h "1+1" +``` + +## Exported functions + +| Function | Signature | Description | +|---|---|---| +| `init` | `init[deps]` | Wire deps + config, record identity, install the `.z.pc` handler + retry job. Idempotent. | +| `startup` | `startup[]` | Read `process.csv` (`processcsv`), drop self, connect to each row whose proctype is in `connections`. A failed connection is logged (not raised) and left as `w:0Ni` for `retry`. No-op if no connections configured. | +| `getservers` | `getservers[proctype]` | Live (`w` non-null) `SERVERS` rows for a proctype. | +| `gethandlebytype` | `gethandlebytype[proctype;selection]` | One live handle via `` `any``/`roundrobin`/`last``; `0Ni` if none. Bumps usage stats. | +| `waitfortype` | `waitfortype[proctype;timeoutms;pollms]` | Block until a live connection exists or timeout; `1b`/`0b`. Caller decides if a timeout is fatal. `startup` must have run first. | +| `getapimeta` | `getapimeta[]` | This module's api metadata, one row per export, for `di.torq` to register with `di.api`. | + +Export is deliberately conservative — only functions `di.torq` or a consumer actually calls +(so `di.api` lists exactly these). The rest are **internal**: `retry` (the scheduled +`serversretry` job — passed to the timer *by value* at init, so it needs no export; it first +runs `cleanup` to sweep ungracefully-vanished handles, then reopens every dead handle), +`cleanup`, `formathp`, `opencon`, `readprocesscsv`, `retryrows`, `selector`, `updatestats`, +`signalfound`, `raiseerror`; plus state (`SERVERS`, `self`, `registered`, `HOPENTIMEOUT`, +`connections`, `processcsv`). + +## The `SERVERS` table + +```q +SERVERS:([]procname:`symbol$();proctype:`symbol$();hpup:`symbol$();w:`int$();hits:`int$();startp:`timestamp$();lastp:`timestamp$();endp:`timestamp$()) +``` + +A direct analogue of legacy TorQ's `.servers.SERVERS`: `w` is the live handle (`0Ni` when +disconnected), `hits`/`lastp` drive handle selection, `startp`/`endp` track lifecycle. + +## `.z.pc` registration via di.handlers + +`.z.pc` (connection closed) is a **simple/observer** event in di.handlers — side-effect only, +fan-out — so di.servers registers its cleanup callback through the injected `handlers` +dependency rather than assigning `.z.pc` directly: + +```q +(handlers[`register])[`.z.pc;`;`servers;0j;pcfunc] +``` + +`register`'s signature is `register[event;phase;nm;pri;func]`; for a simple event the `phase` +must be `` ` `` (null) — di.handlers rejects a non-null phase on an observer event. This lets +di.servers' disconnect hook coexist with every other `.z.pc` registrant in the same +priority-ordered fan-out. + +## Conventions (learnings from di.config) + +- **One-arg `init[deps]`** with config folded into `deps` (the project convention; matches + `di.eodtime`'s optional-config-in-deps pattern), not a two-arg `init[config;deps]`. +- **Three-flat-var logging** — `.z.m.loginfo`/`.z.m.logwarn`/`.z.m.logerr`, matching + `consistency.md`, `di.compression` and `di.config`. (The project hasn't globally frozen this + vs. the single-dict form — flag before changing.) +- **`raiseerror` (log-then-signal)** for all post-init domain errors (`formathp` unknown + ipctype, `selector` unknown selection, missing `process.csv`). `init`'s own dependency + validation is the one exception (plain `'` — no logger yet). +- **`getapimeta`** exported; a test asserts it documents exactly the module's exports. No + `version` export / VERSION file yet — deferred to the di.depcheck rollout, as in di.config. +- **Env-free** — di.servers reads no environment variable; the `process.csv` path arrives via + `config`processcsv` (di.torq resolves it), holding di.config's env-free boundary. + +## Open items / not yet done + +- **Live-peer integration tests are the next step.** `test.csv` currently covers the mockable + surface (init validation, dep-wiring via recording mocks, idempotency, a no-op `startup`, + `getservers`/`gethandlebytype` with no servers, `getapimeta`). The connection behaviour — + `startup` connecting to a real peer and logging a failed dial, `gethandlebytype` returning a + live remote handle, the retry cycle recovering an ungraceful kill (`cleanup`+reopen), + `waitfortype` connected-vs-timeout — needs a genuinely separate spawned `q` peer (a + self-connect returns pseudo-handle `0`, not a real socket). Since `retry`/`cleanup`/`formathp` + are internal, exercise the retry cycle by invoking the callback the **mock timer captured** at + `addjob` (tests the actually-wired path), not a direct export. See the child-q spawn recipe in + the q-gotchas reference. +- **Provider modules not in kdbx-modules yet.** `di.log` (feature-logging branch) and + `di.handlers` aren't here yet, so the injected contracts are mocked in tests. The handlers + mock uses the real `register[event;phase;nm;pri;func]` shape from `handlers.q`. +- **`config`processcsv` and the assembled `connections` list** depend on di.torq's config + wiring — coordinate when di.torq's servers dep is built. +- Scoped-out (v1): discovery service, password/access-list files, non-TorQ tracking, and the + `tcps`/`unix` socket types end-to-end (only `tcp` is wired through `startup`). + +## Tests + +```q +k4unit:use`di.k4unit +k4unit.moduletest`di.servers +``` diff --git a/di/servers/servers.q b/di/servers/servers.q new file mode 100644 index 00000000..450bfb9d --- /dev/null +++ b/di/servers/servers.q @@ -0,0 +1,231 @@ +/ connection management and handle-by-type lookup for the modular torq world - the di.* analogue +/ of TorQ's .servers (code/handlers/trackservers.q + servers.q), scoped down for v1: no discovery +/ service, no password/access-list files, no non-torq process tracking, no FinSpace. process.csv +/ is a static phone book (who to dial), NOT an identity source - self-identity comes from config. +/ FRAMEWORK-tier module: no hard di.* deps; log, timer and handlers are injected (all required). +/ standard one-arg init[deps]: di.torq merges this process's config slice (proctype/procname, +/ connections, processcsv) into the same deps dict it passes the injectables in. conventions match +/ di.config: strict init validation (no fallback), three-flat-var logging, log-then-signal via +/ raiseerror, getapimeta for di.api, and the env-free boundary (the process.csv path arrives via +/ config; di.servers reads no env itself). + +/ --- module-local state (initial values at load; read/written via .z.m at runtime) --- + +SERVERS:([] + procname:`symbol$(); + proctype:`symbol$(); + hpup:`symbol$(); + w:`int$(); + hits:`int$(); + startp:`timestamp$(); + lastp:`timestamp$(); + endp:`timestamp$()); + +HOPENTIMEOUT:2000; + +self:`proctype`procname!``; + +/ guards init's one-time process-global side effects (the .z.pc observer + the retry timer job) so +/ init is IDEMPOTENT - di.torq calls it once per process, but a second call (a test re-run, a +/ future re-init) must not re-register: di.timer.addjob throws on a duplicate id. the dep refs are +/ always refreshed; only the one-time registrations are guarded. +registered:0b; + +raiseerror:{[ctx;msg] + / internal - log an error under ctx via the injected logger, then signal it, so a failure is + / observable in the log as well as thrown. used for all post-init domain errors (init's own + / dependency validation signals with a plain ' - the logger is not wired yet). + .z.m.logerr[ctx;msg]; + '"di.servers: ",string[ctx],": ",msg; + }; + +init:{[deps] + / wire the injected deps (log/timer/handlers - all required, no fallback) and this process's + / config (proctype/procname identity, connections, processcsv), and install the one-time side + / effects (a .z.pc cleanup observer via handlers + a 10s serversretry job via timer). config + / arrives in the SAME deps dict (the one-arg init convention - di.torq merges the config slice + / into it). idempotent (see `registered). does NOT open connections - that is startup's job. + if[99h<>type deps; + '"di.servers: deps must be a dict of injectables + config"]; + if[not all `log`timer`handlers in key deps; + '"di.servers: log, timer and handlers dependencies are required (see di.log, di.timer, di.handlers)"]; + if[99h<>type deps`log; + '"di.servers: log value must be a dict; pass `info`warn`error functions"]; + if[not all (`info`warn`error) in key deps`log; + '"di.servers: log dict must have `info`warn`error keys; got: ",(", " sv string key deps`log)]; + if[99h<>type deps`timer; + '"di.servers: timer value must be a dict (see di.timer)"]; + if[99h<>type deps`handlers; + '"di.servers: handlers value must be a dict (see di.handlers)"]; + if[not all `proctype`procname in key deps; + '"di.servers: proctype and procname (self-identity) are required in deps"]; + .z.m.loginfo:deps[`log]`info; + .z.m.logwarn:deps[`log]`warn; + .z.m.logerr:deps[`log]`error; + .z.m.timer:deps`timer; + .z.m.handlers:deps`handlers; + .z.m.self:`proctype`procname!deps`proctype`procname; + .z.m.connections:$[`connections in key deps;deps`connections;`symbol$()]; + .z.m.processcsv:$[`processcsv in key deps;deps`processcsv;""]; + if[not .z.m.registered; + / .z.pc is a SIMPLE (observer) event in di.handlers - side-effect only, fan-out. registered via + / the injected handlers dep with di.handlers' register[event;phase;nm;pri;func] contract; phase + / is ` (null) for a simple event, pri 0. the callback marks a closed handle's row disconnected. + / (param `wh`, not `w`, so it does not shadow the SERVERS column w.) + pcfunc:{[wh] .z.m.SERVERS:update endp:.z.p,w:0Ni from .z.m.SERVERS where w=wh; }; + (.z.m.handlers[`register])[`.z.pc;`;`servers;0j;pcfunc]; + / di.timer mode-1h period is in SECONDS, so 10 = a 10-second retry (a bare 10000 here would be + / ~2.8h - the latent typo that made dead-handle recovery effectively never fire in early POCs). + (.z.m.timer[`addjob])[`serversretry;retry;();10;1;()!()]; + .z.m.registered:1b; + ]; + .z.m.loginfo[`init;"di.servers initialised"]; + }; + +formathp:{[host;port;ipctype] + / internal - build a connection-handle symbol for `tcp`/`tcps`/`unix. only `tcp is exercised by + / startup in v1; the others exist for a future SOCKETTYPE-style config. + h:string $[host=`localhost;`localhost;host]; + p:string port; + $[ipctype=`tcp; lower `$":",h,":",p; + ipctype=`tcps;lower `$":tcps://",h,":",p; + ipctype=`unix;lower `$":unix://",p; + raiseerror[`formathp;"unknown ipctype ",string ipctype]] + }; + +opencon:{[hpup] + / open a connection, logging (not erroring) on failure - a downed peer isn't necessarily an + / error at connect time; retry keeps trying. NOTE the timeout form is hopen[(handle;timeoutms)] + / (a single 2-item list), not the dyadic hopen[handle;timeoutms], which throws 'rank. + r:@[{(hopen (x;.z.m.HOPENTIMEOUT);"")};hpup;{(0Ni;x)}]; + if[null first r;.z.m.logwarn[`servers;"failed to open connection to ",(string hpup),": ",last r]]; + first r + }; + +readprocesscsv:{[path] + / internal - read the static process.csv phone book (host,port,proctype,procname). the PATH is + / supplied by the caller (from config`processcsv); di.servers reads no env itself, holding + / di.config's env-free boundary - di.torq resolves the path and puts it in config. + fsym:`$":",path; + if[0=count key fsym;raiseerror[`readprocesscsv;"process.csv not found at ",path]]; + ("SISS";enlist",") 0: fsym + }; + +startup:{[] + / open connections to every process.csv row whose proctype is in the configured connections list, + / excluding this process's own row. reads the config stored at init. a failed connection is logged + / (not raised) and left as w:0Ni for retry. a no-op if no connections are configured. + / normalise connections to symbols to match process.csv's `proctype column (always a symbol via + / the "S" spec): a .q settings file gives symbols already (`$ throws 'type on a symbol - it is + / NOT idempotent, hence the type check); a .toml one gives plain strings (TOML has no symbol). + conns:.z.m.connections; + conns:$[11h=abs type conns;conns;`$conns]; + if[0=count conns;.z.m.loginfo[`servers;"no configured connections to make"];:()]; + if[0=count .z.m.processcsv;raiseerror[`startup;"processcsv (path to process.csv) is required in config to open connections"]]; + procs:readprocesscsv[.z.m.processcsv]; + pt:.z.m.self`proctype; + pn:.z.m.self`procname; + procs:update isme:(proctype=pt)&procname=pn from procs; + procs:select from procs where not isme; + procs:select from procs where proctype in conns; + if[0=count procs;.z.m.loginfo[`servers;"no process.csv rows match the configured connections"];:()]; + {[row] + hpup:formathp[row`host;row`port;`tcp]; + w:opencon[hpup]; + if[not null w;.z.m.loginfo[`servers;"connected to ",(string row`proctype),"/",(string row`procname)," at ",string hpup]]; + / catenate+reassign, NOT `tablename insert - a symbol-based insert into `.z.m.SERVERS` misses + / the compile-time module-local rewrite a source-level .z.m.SERVERS gets, silently targeting the + / wrong (literal) table. + newrow:([]procname:enlist row`procname;proctype:enlist row`proctype;hpup:enlist hpup;w:enlist w;hits:enlist 0i;startp:enlist $[null w;0Np;.z.p];lastp:enlist .z.p;endp:enlist 0Np); + .z.m.SERVERS:.z.m.SERVERS,newrow; + } each 0!procs; + }; + +retryrows:{[rows] + / internal - reattempt opencon for the given SERVERS row indices, updating w/lastp (and startp on + / a successful reconnect). + hs:opencon each exec hpup from .z.m.SERVERS where i in rows; + .z.m.SERVERS:update w:hs,lastp:.z.p from .z.m.SERVERS where i in rows; + .z.m.SERVERS:update startp:.z.p from .z.m.SERVERS where i in rows, not null w; + }; + +cleanup:{[] + / internal - sweep any row whose handle has vanished from key .z.W (a peer that died WITHOUT a + / clean .z.pc on this side) and mark it disconnected, so retry will reopen it. the .z.pc observer + / already catches clean closes; this catches the ungraceful ones. + dead:exec w from .z.m.SERVERS where not null w, not w in key .z.W; + if[count dead;.z.m.SERVERS:update endp:.z.p,w:0Ni from .z.m.SERVERS where w in dead]; + }; + +retry:{[] + / internal - the scheduled `serversretry job (driven by the injected timer; passed by value at + / init, so it needs no export). first sweep ungracefully-vanished handles (cleanup), then reopen + / every dead (null) handle - so both clean and unclean drops are recovered on the retry cycle. + cleanup[]; + rows:exec i from .z.m.SERVERS where null w; + if[count rows;retryrows[rows]]; + }; + +getservers:{[pt] + / every live (non-null handle) SERVERS row for a proctype. + select from .z.m.SERVERS where proctype=pt, not null w + }; + +selector:{[tab;selection] + / internal - pick one row from a live-server table by algorithm. + $[selection=`roundrobin;first `lastp xasc tab; + selection=`any; rand tab; + selection=`last; last `lastp xasc tab; + raiseerror[`selector;"unknown selection type ",string selection]] + }; + +updatestats:{[wh] + / internal - bump hits/lastp on the row whose handle was just handed out. + .z.m.SERVERS:update lastp:.z.p,hits:1+hits from .z.m.SERVERS where w=wh + }; + +gethandlebytype:{[pt;selection] + / get a single live handle for a proctype via a selection algorithm (`any`roundrobin`last), or + / 0Ni if none is connected. bumps usage stats on the chosen row. + r:getservers[pt]; + if[0=count r;:0Ni]; + wh:(selector[r;selection])`w; + updatestats[wh]; + wh + }; + +signalfound:{[pt] + / internal - log and return 1b once a connection to pt exists. + .z.m.loginfo[`servers;"connected to ",string pt]; + 1b + }; + +waitfortype:{[pt;timeoutms;pollms] + / block until at least one LIVE connection to pt exists, or timeoutms elapses. the DI-scoped + / analogue of legacy TorQ's startupdepcycles - "fail fast, but wait for a hard dependency to come + / up". startup must have run first (so a pt row exists to reattempt). polls retry between tries, + / sleeping pollms. returns 1b once connected, 0b on timeout - the CALLER decides if that is fatal. + / NOTE the blocking system"sleep" is fine at startup (single-threaded; the injected timer's .z.ts + / just doesn't fire during the sleep). + deadline:.z.p+`timespan$1000000*`long$timeoutms; + .z.m.loginfo[`servers;"waiting up to ",(string timeoutms),"ms for a ",(string pt)," connection"]; + while[(0=count getservers pt) and .z.p Date: Thu, 30 Jul 2026 09:32:27 +0100 Subject: [PATCH 2/9] Adding in temporary intergration tests until actual intergration available --- di/servers/servers.md | 22 +++++++------- di/servers/servers.q | 6 ++++ di/servers/test.csv | 50 ++++++++++++++++++++----------- di/servers/test.q | 70 +++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 121 insertions(+), 27 deletions(-) create mode 100644 di/servers/test.q diff --git a/di/servers/servers.md b/di/servers/servers.md index b1698e43..b85c3248 100644 --- a/di/servers/servers.md +++ b/di/servers/servers.md @@ -98,16 +98,14 @@ priority-ordered fan-out. ## Open items / not yet done -- **Live-peer integration tests are the next step.** `test.csv` currently covers the mockable - surface (init validation, dep-wiring via recording mocks, idempotency, a no-op `startup`, - `getservers`/`gethandlebytype` with no servers, `getapimeta`). The connection behaviour — - `startup` connecting to a real peer and logging a failed dial, `gethandlebytype` returning a - live remote handle, the retry cycle recovering an ungraceful kill (`cleanup`+reopen), - `waitfortype` connected-vs-timeout — needs a genuinely separate spawned `q` peer (a - self-connect returns pseudo-handle `0`, not a real socket). Since `retry`/`cleanup`/`formathp` - are internal, exercise the retry cycle by invoking the callback the **mock timer captured** at - `addjob` (tests the actually-wired path), not a direct export. See the child-q spawn recipe in - the q-gotchas reference. +- **Live-peer integration tests are in place** (`test.q` + `test.csv`, 33 checks). They spawn a + genuinely separate `q` peer (a self-connect returns pseudo-handle `0`, not a real socket) and + cover: `startup` connecting to a live peer and logging a failed dial while excluding self, + `gethandlebytype` returning a live remote handle (`2=h"1+1"`), the retry cycle recovering an + ungraceful kill (`cleanup`+reopen), and `waitfortype` connected-vs-timeout — plus the mockable + surface (init validation, dep-wiring, idempotency, input validation, `getapimeta`). Because + `retry`/`cleanup` are internal, the retry cycle is driven by invoking the callback the **mock + timer captured** at `addjob` (the actually-wired path), not a direct export. - **Provider modules not in kdbx-modules yet.** `di.log` (feature-logging branch) and `di.handlers` aren't here yet, so the injected contracts are mocked in tests. The handlers mock uses the real `register[event;phase;nm;pri;func]` shape from `handlers.q`. @@ -118,6 +116,10 @@ priority-ordered fan-out. ## Tests +Run in a fresh q session (spawns and kills a real peer process; don't interleave with other +modules' tests). Needs `QHOME` set (the peer is launched via `$QHOME/bin/q`) and `di.os` on +`QPATH` (the harness uses `os.abspath` to load `test.q`): + ```q k4unit:use`di.k4unit k4unit.moduletest`di.servers diff --git a/di/servers/servers.q b/di/servers/servers.q index 450bfb9d..898b4fda 100644 --- a/di/servers/servers.q +++ b/di/servers/servers.q @@ -168,6 +168,7 @@ retry:{[] getservers:{[pt] / every live (non-null handle) SERVERS row for a proctype. + if[not -11h=type pt;raiseerror[`getservers;"proctype must be a symbol"]]; select from .z.m.SERVERS where proctype=pt, not null w }; @@ -187,6 +188,8 @@ updatestats:{[wh] gethandlebytype:{[pt;selection] / get a single live handle for a proctype via a selection algorithm (`any`roundrobin`last), or / 0Ni if none is connected. bumps usage stats on the chosen row. + if[not -11h=type pt;raiseerror[`gethandlebytype;"proctype must be a symbol"]]; + if[not -11h=type selection;raiseerror[`gethandlebytype;"selection must be a symbol (`any`roundrobin`last)"]]; r:getservers[pt]; if[0=count r;:0Ni]; wh:(selector[r;selection])`w; @@ -207,6 +210,9 @@ waitfortype:{[pt;timeoutms;pollms] / sleeping pollms. returns 1b once connected, 0b on timeout - the CALLER decides if that is fatal. / NOTE the blocking system"sleep" is fine at startup (single-threaded; the injected timer's .z.ts / just doesn't fire during the sleep). + if[not -11h=type pt;raiseerror[`waitfortype;"proctype must be a symbol"]]; + if[not (abs type timeoutms) within 5 7h;raiseerror[`waitfortype;"timeoutms must be an integer (ms)"]]; + if[not (abs type pollms) within 5 7h;raiseerror[`waitfortype;"pollms must be an integer (ms)"]]; deadline:.z.p+`timespan$1000000*`long$timeoutms; .z.m.loginfo[`servers;"waiting up to ",(string timeoutms),"ms for a ",(string pt)," connection"]; while[(0=count getservers pt) and .z.p +/ cleanup, tested in isolation from the auto .z.pc hook (which di.handlers would install for real). +handlercalls:([]event:`symbol$();name:`symbol$()); +mockhandlers:`register`remove`list!( + {[ev;ph;nm;pri;fn]`handlercalls upsert(ev;nm)}; + {[ev;ph;nm]}; + {[ev]}); + +warnlogged:{[s] any (exec msg from logrows where lvl=`warn) like "*",s,"*"}; +firejob:{[id] timerjobs[id][]}; + +/ --- real peer process fixture --- +FIXDIR:"/tmp/diserverstest"; +isfree:{[p] not @[{hclose hopen x;1b};(`$":localhost:",string p;100);0b]}; +pickport:{[start] first (start+til 500) where isfree each start+til 500}; +PEERPORT:0N; DEADPORT:0N; PEERPID:0N; + +waitlisten:{[port;timeoutms] + deadline:.z.p+`timespan$1000000*timeoutms; + while[(.z.p/dev/null 2>&1 &"; + if[not waitlisten[PEERPORT;3000];'"test: peer failed to listen on ",string PEERPORT]; + h:hopen (`$":localhost:",string PEERPORT;2000); + PEERPID::h ".z.i"; + hclose h;}; + +killpeer:{[] if[not null PEERPID;@[system;"kill ",string PEERPID;{}]]; PEERPID::0N; system "sleep 0.3";}; + +setupfixture:{[] + / pick two free ports (peer + a never-listening dead one), then write a header'd process.csv + / phone book with self, the peer (otherproc), and a dead proctype. + PEERPORT::pickport 20000+`int$.z.i mod 20000; + DEADPORT::pickport PEERPORT+1; + system "mkdir -p ",FIXDIR; + (`$":",FIXDIR,"/process.csv") 0: ( + "host,port,proctype,procname"; + "localhost,",string[PEERPORT-2],",selfproc,selfinst"; + "localhost,",string[PEERPORT],",otherproc,otherinst"; + "localhost,",string[DEADPORT],",deadproc,deadinst"); + }; + +teardownfixture:{[] killpeer[]; system "rm -rf ",FIXDIR;}; + +/ build the deps dict di.torq would assemble: injectables + this process's config slice. +svrdeps:{[conns] `log`timer`handlers`proctype`procname`connections`processcsv!(mocklog;mocktimer;mockhandlers;`selfproc;`selfinst;conns;FIXDIR,"/process.csv")}; From efb35be5c94caf3b4ffbc6db21d2ce80cd0cc222 Mon Sep 17 00:00:00 2001 From: ascottDI Date: Fri, 31 Jul 2026 16:11:56 +0100 Subject: [PATCH 3/9] aligned with di.toml module. Increased robustness of init guards, cut AI fluff code --- di/servers/servers.md | 6 ++++-- di/servers/servers.q | 19 ++++++++++--------- di/servers/test.csv | 4 +++- 3 files changed, 17 insertions(+), 12 deletions(-) diff --git a/di/servers/servers.md b/di/servers/servers.md index b85c3248..c4d893ce 100644 --- a/di/servers/servers.md +++ b/di/servers/servers.md @@ -47,7 +47,7 @@ h "1+1" | `getservers` | `getservers[proctype]` | Live (`w` non-null) `SERVERS` rows for a proctype. | | `gethandlebytype` | `gethandlebytype[proctype;selection]` | One live handle via `` `any``/`roundrobin`/`last``; `0Ni` if none. Bumps usage stats. | | `waitfortype` | `waitfortype[proctype;timeoutms;pollms]` | Block until a live connection exists or timeout; `1b`/`0b`. Caller decides if a timeout is fatal. `startup` must have run first. | -| `getapimeta` | `getapimeta[]` | This module's api metadata, one row per export, for `di.torq` to register with `di.api`. | +| `getapimeta` | `getapimeta[]` | This module's api metadata, one row per **callable** API function (`init`/`getapimeta` plumbing omitted), for `di.torq` to register with `di.api`. | Export is deliberately conservative — only functions `di.torq` or a consumer actually calls (so `di.api` lists exactly these). The rest are **internal**: `retry` (the scheduled @@ -91,7 +91,9 @@ priority-ordered fan-out. - **`raiseerror` (log-then-signal)** for all post-init domain errors (`formathp` unknown ipctype, `selector` unknown selection, missing `process.csv`). `init`'s own dependency validation is the one exception (plain `'` — no logger yet). -- **`getapimeta`** exported; a test asserts it documents exactly the module's exports. No +- **`getapimeta`** exported; a test asserts it documents exactly the module's *callable* + exports — `init`/`getapimeta` are plumbing (di.torq calls them by convention) and are + deliberately omitted from the registry rows, matching di.toml and the skill convention. No `version` export / VERSION file yet — deferred to the di.depcheck rollout, as in di.config. - **Env-free** — di.servers reads no environment variable; the `process.csv` path arrives via `config`processcsv` (di.torq resolves it), holding di.config's env-free boundary. diff --git a/di/servers/servers.q b/di/servers/servers.q index 898b4fda..150f830c 100644 --- a/di/servers/servers.q +++ b/di/servers/servers.q @@ -59,6 +59,8 @@ init:{[deps] '"di.servers: handlers value must be a dict (see di.handlers)"]; if[not all `proctype`procname in key deps; '"di.servers: proctype and procname (self-identity) are required in deps"]; + if[not all -11h=type each deps`proctype`procname; + '"di.servers: proctype and procname must be symbols"]; .z.m.loginfo:deps[`log]`info; .z.m.logwarn:deps[`log]`warn; .z.m.logerr:deps[`log]`error; @@ -85,7 +87,7 @@ init:{[deps] formathp:{[host;port;ipctype] / internal - build a connection-handle symbol for `tcp`/`tcps`/`unix. only `tcp is exercised by / startup in v1; the others exist for a future SOCKETTYPE-style config. - h:string $[host=`localhost;`localhost;host]; + h:string host; p:string port; $[ipctype=`tcp; lower `$":",h,":",p; ipctype=`tcps;lower `$":tcps://",h,":",p; @@ -225,13 +227,12 @@ waitfortype:{[pt;timeoutms;pollms] }; getapimeta:{[] - / this module's api metadata, one row per exported function, for di.torq to register with di.api. - / names are bare (di.torq qualifies them). one (name;public;descrip;params;return) row per line. + / this module's api metadata, one row per CALLABLE API function, for di.torq to register with + / di.api. init/getapimeta are plumbing (di.torq calls them by convention) and are deliberately NOT + / listed - the registry describes the callable api, not plumbing. names are bare (di.torq qualifies). :flip `name`public`descrip`params`return!flip( - (`init; 0b; "wire injected deps + config and install the pc observer + retry job (idempotent)"; "[dict: deps - `log`timer`handlers + `proctype`procname (+`connections`processcsv)]"; "null"); - (`startup; 1b; "open connections to configured proctypes from process.csv (reads init config)"; "[]"; "null"); - (`getservers; 1b; "live SERVERS rows for a proctype"; "[symbol: proctype]"; "table: live server rows"); - (`gethandlebytype; 1b; "one live handle for a proctype via any/roundrobin/last selection"; "[symbol: proctype; symbol: selection]"; "int: handle, 0Ni if none"); - (`waitfortype; 1b; "block until a proctype connects or timeout elapses"; "[symbol: proctype; long: timeoutms; long: pollms]"; "boolean: 1b connected, 0b timed out"); - (`getapimeta; 0b; "this module's api metadata rows"; "[]"; "table: metadata rows")); + (`startup; 1b; "open connections to configured proctypes from process.csv (reads init config)"; "[]"; "null"); + (`getservers; 1b; "live SERVERS rows for a proctype"; "[symbol: proctype]"; "table: live server rows"); + (`gethandlebytype; 1b; "one live handle for a proctype via any/roundrobin/last selection"; "[symbol: proctype; symbol: selection]"; "int: handle, 0Ni if none"); + (`waitfortype; 1b; "block until a proctype connects or timeout elapses"; "[symbol: proctype; long: timeoutms; long: pollms]"; "boolean: 1b connected, 0b timed out")); }; diff --git a/di/servers/test.csv b/di/servers/test.csv index e769e7d9..652476e8 100644 --- a/di/servers/test.csv +++ b/di/servers/test.csv @@ -42,6 +42,8 @@ comment,,,,,,,input validation + getapimeta fail,0,0,q,"svc.getservers[""nosuch""]",1,1,getservers rejects a non-symbol proctype fail,0,0,q,"svc.gethandlebytype[`hdb;""any""]",1,1,gethandlebytype rejects a non-symbol selection fail,0,0,q,"svc.waitfortype[`hdb;""x"";200]",1,1,waitfortype rejects a non-integer timeout -true,0,0,q,(asc key svc)~asc exec name from svc.getapimeta[],1,1,getapimeta documents exactly the exports +fail,0,0,q,"svc.init[`log`timer`handlers`proctype`procname!(mocklog;mocktimer;mockhandlers;""rdb"";`p)]",1,1,init rejects a non-symbol proctype +true,0,0,q,(asc (key svc) except `init`getapimeta)~asc exec name from svc.getapimeta[],1,1,getapimeta documents exactly the callable exports (plumbing omitted) +true,0,0,q,not any `init`getapimeta in exec name from svc.getapimeta[],1,1,getapimeta omits plumbing (init/getapimeta not registered) true,0,0,q,`name`public`descrip`params`return~cols svc.getapimeta[],1,1,getapimeta rows carry the registry columns after,0,0,q,teardownfixture[],1,1,kill the peer and remove the fixture dir From eb7c1897c82ccbca4bbb611220290fa1bbf89fbe Mon Sep 17 00:00:00 2001 From: ascottDI Date: Wed, 5 Aug 2026 15:08:25 +0100 Subject: [PATCH 4/9] pulling in modules that are merged to main --- di/config/config.md | 265 ++++++++++++++++++++++++++++++++++++++++++ di/config/config.q | 177 ++++++++++++++++++++++++++++ di/config/init.q | 3 + di/config/test.csv | 81 +++++++++++++ di/dbwrite/dbwrite.md | 211 +++++++++++++++++++++++++++++++++ di/dbwrite/dbwrite.q | 236 +++++++++++++++++++++++++++++++++++++ di/dbwrite/init.q | 6 + di/dbwrite/test.csv | 246 +++++++++++++++++++++++++++++++++++++++ di/eodtime/eodtime.md | 157 +++++++++++++++++++++++++ di/eodtime/eodtime.q | 98 ++++++++++++++++ di/eodtime/init.q | 7 ++ di/eodtime/test.csv | 92 +++++++++++++++ di/log/init.q | 4 + di/log/log.md | 193 ++++++++++++++++++++++++++++++ di/log/log.q | 193 ++++++++++++++++++++++++++++++ di/log/test.csv | 218 ++++++++++++++++++++++++++++++++++ di/toml/init.q | 4 + di/toml/test.csv | 80 +++++++++++++ di/toml/test.q | 30 +++++ di/toml/toml.md | 104 +++++++++++++++++ di/toml/toml.q | 128 ++++++++++++++++++++ 21 files changed, 2533 insertions(+) create mode 100644 di/config/config.md create mode 100644 di/config/config.q create mode 100644 di/config/init.q create mode 100644 di/config/test.csv create mode 100644 di/dbwrite/dbwrite.md create mode 100644 di/dbwrite/dbwrite.q create mode 100644 di/dbwrite/init.q create mode 100644 di/dbwrite/test.csv create mode 100644 di/eodtime/eodtime.md create mode 100644 di/eodtime/eodtime.q create mode 100644 di/eodtime/init.q create mode 100644 di/eodtime/test.csv create mode 100644 di/log/init.q create mode 100644 di/log/log.md create mode 100644 di/log/log.q create mode 100644 di/log/test.csv create mode 100644 di/toml/init.q create mode 100644 di/toml/test.csv create mode 100644 di/toml/test.q create mode 100644 di/toml/toml.md create mode 100644 di/toml/toml.q diff --git a/di/config/config.md b/di/config/config.md new file mode 100644 index 00000000..f6a10549 --- /dev/null +++ b/di/config/config.md @@ -0,0 +1,265 @@ +# di.config + +Configuration loading and cascade resolution for the modular TorQ world. It replaces +TorQ's in-process config handling (`torq.q`: `loadf`, `loadconfig`, `loadaddconfig`, +`overrideconfig`). `di.config` resolves a settings **cascade** over a builtin root and an +app root — reading **both** flat `name:value` `.q` files and `.toml` files — and returns +the merged configuration as **one flat dict**. `di.torq` calls it once at startup +(`config:cascade[...]`) and hands that dict to each module's `init`. + +## The config model — one flat dict + +`di.config` resolves config into a **returned flat dict**; it does not execute settings +files into namespaces or hold any global store. This is the shape `di.torq` consumes: + +```q +config:use`di.config +cfg:config.cascade[builtinroot;approot;`rdb;`rdb1] / -> `subscribeto`rows`... !(...) +/ di.torq stamps identity and hands the whole dict to each module's init: +cfg:cfg,`proctype`procname!(`rdb;`rdb1) +rdb.init[cfg;`log`timer!(logdep;timerdep)] +``` + +`cascade` and `parsefile` are **pure** — no `init`, no logger, no globals — because +`di.torq` resolves config *before* the logger dependency has been built. Only +`overrideconfig` logs (it reports skipped/rejected overrides), so it — and only it — +requires `init`. + +## Precedence + +The cascade is resolved over **two roots** (`builtinroot`, `approot`) × a **three-tier +name sequence** (`default` → `{proctype}` → `{procname}`). Precedence is **name-major**: + +1. A more specific **name** always wins — `{procname}` beats `{proctype}` beats `default`. +2. **Within a name**, the later (app) root overrides the builtin root. +3. **Within a single tier** (one root + one name), the `.toml` file wins over the `.q` + file on a key clash — useful mid-migration when both formats coexist. + +So a key set in both `builtin/{proctype}` and `app/default` resolves to the **builtin +proctype** value, not the app default (name beats root). Effective load order, low → high +priority: + +``` +builtin/default(.q,.toml) → app/default → builtin/{proctype} → app/{proctype} + → builtin/{procname} → app/{procname} (each tier: .toml overrides .q) +``` + +`overrideconfig` sits on **top** of all of this — the command-line layer (see below). + +> **Note — name-major, not root-major.** The KDBX-POC's own vendored `di.config` used +> *root-major* precedence (all builtin tiers, then all app tiers), where `app/default` +> would beat `builtin/{proctype}`. `di.config` deliberately keeps the **name-major** rule +> TorQ has always used; `.toml` support was layered on without changing it. + +> **Note — avoid `-` in config file paths.** A source-level backtick symbol literal +> containing `-` parses as subtraction (`` `:/a-b.q `` → `` `:/a `` `- b.q`), not one +> symbol. The runtime string paths `cascade` builds internally are unaffected. + +## Settings file formats + +- **`.q`** — plain `name:value` lines. Split on the first `:`; the RHS is run through + `value`, so `` `:hdb ``, `` `trade`quote ``, `` `symbol$() `` all work. Blank lines and + `/`-comment lines are skipped. These are **flat** files (one process's config), **not** + the namespaced, executable settings files legacy TorQ `\l`-loaded. +- **`.toml`** — delegated to **`di.toml`**, loaded lazily (only when a `.toml` file is + actually present). TOML has no symbol type, so every TOML string comes back as a q + string, never a symbol — consumers that need a symbol normalise at the point of use + (`` `$ `` is a no-op on an already-a-symbol value). + +## Import and init + +`cascade` and `parsefile` need no `init` and no logger — call them directly: + +```q +config:use`di.config +cfg:config.cascade["/opt/torqx/di/torq/settings";"/opt/app/settings";`rdb;`rdb1] +``` + +`overrideconfig` logs, so wire a `log` dependency via `init` first — there is no silent +fallback. Pass an already-conforming binary `` `info`warn`error `` dict of `{[c;m]}` +loggers (context symbol, message string); the module performs **no** adaptation. + +```q +/ option 1: di.log — the standard logger (exports binary info/warn/error) +logger:use`di.log +logdep:`info`warn`error!(logger.info;logger.warn;logger.error) +config.init[enlist[`log]!enlist logdep] + +/ option 2: any hand-rolled binary {[c;m]} logger +mylog:`info`warn`error!( + {[c;m] -1 string[c],": INFO ",m;}; + {[c;m] -1 string[c],": WARN ",m;}; + {[c;m] -2 string[c],": ERROR ",m;}); +config.init[enlist[`log]!enlist mylog] +``` + +A logger that is *not* already binary `{[c;m]}` (e.g. a raw monadic +[`kx.log`](https://github.com/KxSystems/logging) instance) must be wrapped by the caller +first — di.config does no wrapping. + +## Exported functions + +| Function | Signature | Description | +|---|---|---| +| `cascade` | `cascade[builtinroot;approot;proctype;procname]` | Resolve the settings cascade over the two roots × the `default`/`{proctype}`/`{procname}` name sequence and **return one flat dict**. Pure — no `init`, no logger, no globals. Precedence is name-major with `.toml > .q` within a tier (see above). A key set nowhere is simply absent; an all-missing cascade returns an empty dict. | +| `parsefile` | `parsefile[path]` | Parse a single settings file into a flat dict. A `.toml` path is delegated to `di.toml` (loaded lazily, behind the guard rail below); a `.q` path is flat `name:value` lines. A missing file (either extension) → empty dict. An existing `.toml` file with no `di.toml` on `QPATH` signals a clear error naming the file. Pure. | +| `overrideconfig` | `overrideconfig[config;params]` | Apply the **command-line override layer** on top of a resolved config dict — the top tier of the cascade. `di.torq` parses the process command line (`.Q.opt .z.x`) and calls this after `cascade`, so launch-time flags (e.g. `-loglevel info`, `-rows 5000`) win over the settings files. `config` is the dict from `cascade`; `params` is a dict keyed by setting name (symbol) with string (or list-of-string) values, each parsed into that setting's **existing** type. Only keys already present with a basic type are overridden; unknown keys, non-basic types, unparseable values, and multi-value overrides of a single-valued (scalar/string) setting are logged and skipped. A vector setting (e.g. a symbol list) accepts multiple values. Returns the **updated config dict**. Requires `init`. | +| `init` | `init[deps]` | Wire the injected logger for `overrideconfig`. `deps` is a dict with a required `log` key — an already-conforming binary `` `info`warn`error `` dict of `{[c;m]}` functions. Errors immediately if the log dependency is missing or malformed. | +| `getapimeta` | `getapimeta[]` | This module's API metadata — one `(name;public;descrip;params;return)` row per exported function, for `di.torq` to register with `di.api`. | + +Internal helpers — `parsetier` (per-tier `.q`+`.toml` merge), `applyoverride`, +`parsefailed`, and `hexchars` — are deliberately not exported. + +## Injectable dependencies + +| Injectable | Required keys | Function signature | +|---|---|---| +| `log` | `` `info`warn`error `` | `{[c;m]}` (context symbol, message string). Caller passes an already-conforming binary dict — built from `di.log` or hand-rolled. No adaptation is done in the module. Needed only by `overrideconfig`. | + +## Hard dependencies + +None mandatory. `cascade`/`parsefile` gain a **lazy, soft dependency on `di.toml`**: it is +`` use``d only when a `.toml` file is actually encountered, so the `.q`-only path needs +nothing. **di.toml does not need to be in the repo (or on `QPATH`) for di.config to load or +run** — it is required only at the moment a `.toml` settings file is actually parsed. + +### The `.toml` guard rail + +`parsefile` guards the di.toml load so a missing module produces a clear, actionable error +rather than a cryptic `notfound`. The order is deliberate: + +1. **Check the file exists first.** A missing file (either extension) → empty dict. So a + missing `.toml` tier *never* triggers the di.toml requirement — the internal + `parsetier`/`cascade` probe `base,".toml"` on every tier, and absent tiers cost nothing. +2. **Then, only for a `.toml` file that actually exists, require di.toml.** The internal + `requiretoml` helper attempts `` use`di.toml `` under protected evaluation; if it is not + found, it signals a clear error **naming the offending file**: + ``` + 'di.config: cannot parse TOML file '/path/to/x.toml' - the di.toml module was not found + on QPATH; a di.toml module is required to parse .toml settings (underlying: notfound: di.toml) + ``` +3. **Parse.** With di.toml resolved, the file is delegated to it. + +Because this signal happens at config-resolution time — *before* the logger dependency is +wired — it surfaces via the raised error (which aborts startup with the reason), not via the +injected logger. + +### Contract di.config expects from di.toml + +A di.toml module must export a **`parsefile`** function taking a settings-file path string +and returning a **flat dict** of `setting → value`: + +```q +(use`di.toml)[`parsefile] "/path/to/settings.toml" / -> `dir`rows!(":appdb";100) +``` + +TOML has no symbol type, so di.toml returns every TOML string as a **q char string** (never +a symbol); di.config applies no coercion, so consumers that need a symbol normalise at the +point of use (`` `$ `` is a no-op on an already-a-symbol value). + +> **⚠️ `di.toml` is not yet in kdbx-modules** — it currently lives only in the KDBX-POC and +> lands here with the PoC merge (a di.toml module is in development against the contract +> above). Until it is on `QPATH`, the `.toml` half of a tier is **dormant** in this repo: a +> `.toml` file that exists but cannot be parsed raises the guard error above; the `.q` path +> works standalone. Full `.toml` resolution is verified in the PoC (where `di.toml` is +> vendored); this repo's suite covers the guard's error path (di.toml absent) and the +> file-exists-before-delegation ordering. + +## Design notes & open gaps + +- **Returned dict, not a global store.** Resolution is a pure function returning a dict. + There is no queryable config store and no per-namespace `getmodule` — `di.torq` passes the + whole resolved dict to every module and each reads the keys it cares about. A missing key + is a caller bug (surfaced as a q null on an inline index), not something the config layer + silently papers over — every setting is expected to be booted with a default in the + `default` tier. +- **Retired: the executable-namespaced `.q` path.** An earlier revision also shipped a + `loadcascade`/`loadconfig`/`getmodule` path that `\l`-loaded namespaced, code-bearing `.q` + settings files (`.rdb.subscribeto:...`) into root namespaces and sliced them per module. + That parallel model was **removed** in favour of the single flat-dict cascade `di.torq` + actually consumes. If runtime config reload or an in-process store is ever needed, that is + a deliberate, separately-designed addition — not a silent revival of the globals path. +- **`overrideconfig` is the command-line layer, applied by `di.torq` — not a manual step.** + It mirrors TorQ's `overrideconfig`/`override[]`, which TorQ runs at startup off + `.Q.opt .z.x` so launch-time flags beat the settings files. di.config just does the + type-aware apply onto the resolved dict (which is why it stays env-free and never reads + `.z.x` itself). It deviates from TorQ deliberately on error handling: per-setting problems + (unknown name, non-basic type, value that won't parse to the target type) are logged and + skipped — a single bad override never aborts the batch and a null is never written. +- **`overrideconfig` parse-failure detection is type-aware.** Most basic types parse to a + null on a bad value, which `applyoverride` rejects via a null check. Boolean (`1h`) and + byte (`4h`) are the only in-scope basic types with **no null** — `"B"$"bad"` yields `0b`, + `"X"$"gg"` yields `0x00`, both non-null — so a bad value would slip past a null check and + silently corrupt config. `parsefailed` handles these two explicitly (boolean must be + `"0"`/`"1"`; byte must be even-length hex); anything else is rejected and logged, never + written. +- **String-typed (`10h`) settings are overridable — as text, no parse.** A char-string + setting's override value *is* already a string, so `applyoverride` takes it as-is rather + than casting it (any string is valid; there is no parse-failure case). This matters because + TOML has no symbol type, so a `.toml`-origin setting like `dir = ":appdb"` or + `loglevel = "info"` comes back as a string — without this, such settings could not be + command-line-overridden at all, whereas their `.q`-origin symbol equivalents (`` dir:`:appdb ``) + could. di.config stays policy-free: it preserves whatever type the setting already had + (string in, string out; symbol in, symbol out), leaving symbol-vs-string normalisation to + the consumer as everywhere else. +- **The sentinel merge key.** `cascade`'s accumulator is seeded with a sentinel + `` (enlist`)!enlist(::) `` key so its value list stays general from the first merge — a + same-typed value list coalesces to a typed vector that `,:` then refuses to widen (e.g. + adding a `` `symbol$() `` key to a dict whose values so far were all plain symbols). The + sentinel is dropped before the dict is returned. +- **Logging uses the three-flat-var convention** — `.z.m.loginfo`/`.z.m.logwarn`/`.z.m.logerr`, + called as `.z.m.loginfo[\`ctx;"msg"]` — matching `consistency.md` and `di.compression`. The + injected `log` value is still the same `` `info`warn`error `` dict; `init` fans it out into + the three module-local vars. + +## Tests + +```q +k4unit:use`di.k4unit +k4unit.moduletest`di.config +``` + +The `.toml` path is exercised in the KDBX-POC (where `di.toml` is vendored), not here — +see the hard-dependencies note above. + +## Follow-ups (deferred until other modules land) + +di.config is complete for v1 in isolation. The items below are intentionally deferred; +each is keyed to the module or milestone that unblocks it. + +### ⚠️ Logging call convention — NOT set in stone (ask before changing) + +di.config uses the **three-flat-var** convention (`.z.m.loginfo`/`.z.m.logwarn`/`.z.m.logerr`), +matching `consistency.md` and `di.compression`, but the project hasn't finalised this versus +the **single-dict** form (`.z.m.log[\`info][…]`). **Before changing di.config's logging — or +wiring logging in another module — stop and ask the user which convention is authoritative.** +Switching is mechanical (storage + call sites only; the injected input contract is identical). + +### When `di.toml` lands in kdbx-modules (with the PoC merge) + +- **Add committed `.toml` tests.** Today `di.toml` is not on this repo's `QPATH`, so a + committed `.toml`/mixed-tier test would fail here (verified in the PoC instead). Once + `di.toml` resolves, add tests covering: a `.toml` tier parsed into the dict, `.toml` + winning over `.q` within a tier, and a mixed `.q`+`.toml` cascade. + +### When `di.log` merges to main + +- **Add a committed `di.log` integration test.** The contract match (binary + `` `info`warn`error `` dict) is verified manually and stood in for by the kx.log-wrapper + emission test. Once `di.log` resolves on `QPATH`, add a test that builds the dict from + `di.log` and drives `overrideconfig` through it. + +### When `di.torq` is built + +- **Prove the end-to-end flow.** Add a multi-module integration test (under + `di.torq`/`di.inttest`, not here) exercising: `di.torq` resolves `KDBCONFIG`/`KDBAPPCONFIG` + → roots and `proctype`/`procname` → identity, calls `cascade`, then `overrideconfig` with + the command-line params, then hands the resolved dict to each module's `init`. +- **Hold the env-free boundary.** di.config reads no env vars or process identity — di.torq + owns that resolution and passes the roots and identity explicitly. + +### When `di.depcheck` is built (versioning rollout) + +- **Add a `version` export and `deps.q`.** When `di.depcheck` lands, add `version:"…"` to the + export and a minimal `deps.q` (di.config has no hard deps) as part of the coordinated + repo-wide rollout, not a di.config-only change. diff --git a/di/config/config.q b/di/config/config.q new file mode 100644 index 00000000..8feb594b --- /dev/null +++ b/di/config/config.q @@ -0,0 +1,177 @@ +/ configuration cascade resolution for the modular torq world - replaces torq.q's +/ loadf/loadconfig/loadaddconfig/overrideconfig. resolves a settings cascade over a builtin root +/ and an app root, reading both flat name:value .q files and .toml files, and RETURNS the merged +/ config as one flat dict (the shape di.torq consumes). reads no env/identity - the caller supplies +/ the roots and process identity. +/ precedence is name-major (more specific NAME wins; within a name the app root wins) and, within a +/ tier, .toml wins over .q. cascade/parsefile are pure (no init/logger) - di.torq resolves config +/ before the logger exists; only overrideconfig logs, so only it needs init. di.toml is a soft dep +/ (not yet in kdbx-modules): parsefile loads it lazily, only when a .toml file is present. + +init:{[deps] + / wire the injected logger (required by overrideconfig; no fallback). deps: a dict with a `log + / key holding a binary `info`warn`error dict of {[c;m]} loggers. cascade/parsefile don't need init. + if[99h<>type deps; + '"di.config: deps must be a dict with `log key"]; + if[not `log in key deps; + '"di.config: log dependency is required; pass `info`warn`error functions keyed on `log"]; + if[99h<>type deps`log; + '"di.config: log value must be a dict; pass `info`warn`error functions"]; + if[not all (`info`warn`error) in key deps`log; + '"di.config: log dict must have `info`warn`error keys; got: ",(", " sv string key deps`log)]; + .z.m.loginfo:deps[`log]`info; + .z.m.logwarn:deps[`log]`warn; + .z.m.logerr:deps[`log]`error; + .z.m.loginfo[`init;"di.config initialised"]; + }; + +/ --- settings cascade resolution (.q + .toml -> flat dict) --- + +requiretoml:{[path] + / internal - resolve di.toml to parse a .toml file. di.toml is a soft dep (needed only once a + / .toml file appears); guard the load so a missing one gives a clear error naming the file, not a + / cryptic `notfound: di.toml`. returns the di.toml module. + :@[use;`di.toml;{[p;e] + msg:"di.config: cannot parse TOML file '",p,"' - the di.toml module was not found on QPATH; "; + msg,:"a di.toml module is required to parse .toml settings (underlying: ",e,")"; + 'msg + }[path;]]; + }; + +parsefile:{[path] + / parse ONE settings file into a flat dict. .toml -> di.toml (via requiretoml); .q -> flat + / `name:value` lines (split on first ":", RHS through value). blank, "/"-comment and non-pair + / (no ":") lines are skipped; a missing file -> empty dict. existence is checked FIRST, so a + / missing .toml tier never triggers the di.toml requirement. + fsym:`$":",path; + if[0=count key fsym; :()!()]; + if[path like "*.toml"; :(requiretoml[path])[`parsefile] path]; + lines:read0 fsym; + lines:lines where 0 empty. + (parsefile base,".q"),parsefile base,".toml" + }; + +cascade:{[builtinroot;approot;proctype;procname] + / resolve the cascade over the two roots x (default;proctype;procname) and RETURN the merged flat + / dict. pure (no init/logger/globals). name-major: a more specific NAME wins over the root layer, + / and within a name the app root wins (parsetier gives .toml>.q within a tier). the accumulator is + / seeded with a sentinel (`)!(::) key so its value list stays general from the first merge (a + / same-typed value list coalesces to a typed vector that ,: then won't widen); dropped before return. + / proctype/procname must be non-null symbol atoms: string`` is "", which would build a bogus + / "/" base and read a file named ".q"/".toml". a null means identity resolution failed, so + / surface it. (a null symbol is still type -11h, so the null check is separate from the type check.) + if[not all -11h=type each (proctype;procname); + '"di.config: cascade requires proctype and procname to be symbol atoms"]; + if[(null proctype)|null procname; + '"di.config: cascade requires non-null proctype and procname; got ",(-3!proctype)," and ",-3!procname]; + dirs:(builtinroot;approot); + / dedup (distinct keeps first-occurrence order, preserving least->most-specific). without it a + / repeated name re-applies its tier at a LATER slot: procname~proctype is redundant, but + / procname~`default would put default LAST and wrongly override proctype. + names:distinct `default,proctype,procname; + bases:raze {[ds;nm] ds,\:"/",string nm}[dirs;] each names; + acc:{[a;b] a,parsetier b}/[(enlist `)!enlist(::);bases]; + acc _ ` + }; + +/ --- command-line override layer (top of the cascade) --- + +hexchars:"0123456789abcdefABCDEF"; + +parsefailed:{[t;raw;vals] + / internal - true if any raw string failed to parse to type t. bool (1h) and byte (4h) have no + / null, so a bad parse ("B"$"bad"->0b, "X"$"gg"->0x00) slips past a null check - check their form + / explicitly. other types null on failure. + :$[1h=abs t;not all raw in (enlist"0";enlist"1"); + 4h=abs t;not all {(0=count[x] mod 2) and all x in hexchars} each raw; + any null vals]; + }; + +applyoverride:{[name;cur;raw] + / internal - parse raw into cur's type; return (applied;newvalue). cur's type drives the parse. + / applied is 0b (newvalue=cur) if cur is not a basic type, a value failed to parse, or a + / single-valued setting got other than one value. + t:type cur; + if[not (abs t) within (1;-1+count .Q.t); + .z.m.logerr[`overrideconfig;"cannot override ",(string name),": not a basic type"]; + :(0b;cur); + ]; + raw:$[10h=type raw;enlist raw;raw]; + / scalar-atom (t<0) and string (10h) settings are single-valued: require exactly one value. reject + / a multi- or zero-value override rather than silently taking the first or writing a null. vector + / settings (t>0, not 10h) take as many as given. + if[(1<>count raw) and ((t<0)|(10h=t)); + .z.m.logerr[`overrideconfig;"cannot override ",(string name),": expected a single value, got ",string count raw]; + :(0b;cur); + ]; + / a string (10h) is already text - take the override as-is (no parse). lets .toml-origin string + / settings be overridden like symbol (.q-origin) ones; type is preserved either way. + if[10h=t; + vals:first raw; + .z.m.loginfo[`overrideconfig;"setting ",(string name)," to ",-3!vals]; + :(1b;vals); + ]; + vals:(upper .Q.t abs t)$'raw; + if[parsefailed[t;raw;vals]; + .z.m.logerr[`overrideconfig;"cannot override ",(string name),": value did not parse"]; + :(0b;cur); + ]; + / reduce to scalar only after parsefailed (which checks the whole list). + if[t<0;vals:first vals]; + .z.m.loginfo[`overrideconfig;"setting ",(string name)," to ",-3!vals]; + :(1b;vals); + }; + +overrideconfig:{[config;params] + / apply the command-line override layer onto a resolved config dict - the TOP cascade tier + / (di.torq calls this after cascade with .Q.opt .z.x, so launch flags win over files). config is + / the cascade dict; params keys settings (symbol) to string / string-list values, parsed into each + / setting's existing type. unknown keys, non-basic types and bad values are logged and skipped + / (never aborts the batch, never writes a null). returns the updated config dict. + if[99h<>type config; + .z.m.logerr[`overrideconfig;err:"di.config: config must be a dict"]; + 'err; + ]; + if[99h<>type params; + .z.m.logerr[`overrideconfig;err:"di.config: params must be a dict keyed by setting name"]; + 'err; + ]; + vars:key params; + if[0 merged flat dict (name-major, .toml>.q)"; "[string: builtinroot; string: approot; symbol: proctype; symbol: procname]"; "dict: merged setting -> value"); + (`overrideconfig; 1b; "apply the command-line override layer onto a resolved config dict"; "[dict: resolved config; dict: setting name (symbol) -> string or list of strings]"; "dict: updated config"); + (`parsefile; 1b; "parse one .q or .toml settings file into a flat dict"; "[string: file path (.q flat name:value, or .toml via di.toml)]"; "dict: setting -> value (empty if the file is missing)"); + (`getapimeta; 0b; "this module's api metadata rows"; "[]"; "table: metadata rows")); + }; diff --git a/di/config/init.q b/di/config/init.q new file mode 100644 index 00000000..2e06d014 --- /dev/null +++ b/di/config/init.q @@ -0,0 +1,3 @@ +/ configuration loading and cascade resolution for the modular torq world. +\l ::config.q +export:([init;cascade;overrideconfig;parsefile;getapimeta]) diff --git a/di/config/test.csv b/di/config/test.csv new file mode 100644 index 00000000..c257e8f7 --- /dev/null +++ b/di/config/test.csv @@ -0,0 +1,81 @@ +action,ms,bytes,lang,code,repeat,minver,comment +before,0,0,q,cfg:use`di.config,1,1,load the module +before,0,0,q,"logtab:([]lvl:`symbol$();ctx:`symbol$();msg:())",1,1,capturing log table +before,0,0,q,"mocklog:`info`warn`error!({[c;m]`logtab upsert(`info;c;m)};{[c;m]`logtab upsert(`warn;c;m)};{[c;m]`logtab upsert(`error;c;m)})",1,1,define a capturing binary logger +before,0,0,q,cfg.init[enlist[`log]!enlist mocklog],1,1,init with the required log dependency +before,0,0,q,"system""mkdir -p /tmp/diconfigbuiltin""",1,1,builtin root for the cascade +before,0,0,q,"system""mkdir -p /tmp/diconfigapp""",1,1,app root for the cascade +before,0,0,q,"(`:/tmp/diconfigbuiltin/default.q) 0: (""diconfiga:1"";""diconfigb:1"")",1,1,builtin/default flat name:value (two settings) +before,0,0,q,"(`:/tmp/diconfigbuiltin/rdb.q) 0: enlist ""diconfigb:10""",1,1,builtin/rdb proctype tier sets diconfigb +before,0,0,q,"(`:/tmp/diconfigapp/default.q) 0: (""diconfiga:100"";""diconfigb:5"")",1,1,app/default flat name:value (two settings) +before,0,0,q,"(`:/tmp/diconfigapp/rdb1.q) 0: enlist ""diconfiga:999""",1,1,app/rdb1 procname tier sets diconfiga +before,0,0,q,"basecfg:`enabled`rows`syms`tab`byte`str!(0b;100;`a`b;([]a:`long$());0x01;"":olddb"")",1,1,base config dict for the overrideconfig tests (str is a char-string setting, as a .toml value would be) +before,0,0,q,"system""mkdir -p /tmp/diconfigskip""",1,1,dir for the malformed-line skip test +before,0,0,q,"(`:/tmp/diconfigskip/default.q) 0: (""a:1"";"""";""/ a comment"";""straynocolon"";""b:2"")",1,1,a .q file mixing valid pairs with blank/comment/no-colon lines +before,0,0,q,"system""mkdir -p /tmp/diconfigtoml""",1,1,dir for the toml-guard test +before,0,0,q,"(`:/tmp/diconfigtoml/present.toml) 0: enlist ""a = 1""",1,1,a .toml file that exists but cannot be parsed without di.toml (absent in kdbx-modules) +comment,,,,,,,init - dependency validation +fail,0,0,q,cfg.init[(::)],1,1,init rejects a non-dictionary deps +fail,0,0,q,cfg.init[enlist[`foo]!enlist 1],1,1,init rejects deps without a log key +fail,0,0,q,cfg.init[enlist[`log]!enlist 42],1,1,init rejects a non-dict log value +fail,0,0,q,cfg.init[enlist[`log]!enlist ((enlist`info)!enlist {[c;m]})],1,1,init rejects a log dict missing warn/error +comment,,,,,,,parsefile - parse one flat .q settings file into a dict (.toml delegated to di.toml) +true,0,0,q,"1~cfg.parsefile[""/tmp/diconfigbuiltin/default.q""]`diconfiga",1,1,parsefile reads a flat name:value .q file into a dict +true,0,0,q,"2=count cfg.parsefile[""/tmp/diconfigbuiltin/default.q""]",1,1,parsefile returns every setting in the file +true,0,0,q,"0=count cfg.parsefile[""/tmp/diconfignope/default.q""]",1,1,parsefile returns an empty dict for a missing .q file +true,0,0,q,"(`a`b!1 2)~cfg.parsefile[""/tmp/diconfigskip/default.q""]",1,1,parsefile skips blank/comment/no-colon lines and returns only valid pairs (no-colon line must not crash the split) +comment,,,,,,,parsefile - .toml guard rail (di.toml is a soft dep; absent in kdbx-modules so the error path is exercised here) +true,0,0,q,"0=count cfg.parsefile[""/tmp/diconfignope/absent.toml""]",1,1,a MISSING .toml contributes nothing and does NOT require di.toml (existence checked before delegation) +fail,0,0,q,"cfg.parsefile[""/tmp/diconfigtoml/present.toml""]",1,1,an EXISTING .toml with no di.toml on QPATH signals rather than parsing +true,0,0,q,"(@[cfg.parsefile;""/tmp/diconfigtoml/present.toml"";{x}]) like ""*di.toml module was not found*""",1,1,the .toml guard error names the missing di.toml module +comment,,,,,,,cascade - merged flat dict over two roots x default/proctype/procname (name-major; .toml>.q within a tier) +true,0,0,q,"100~(cfg.cascade[""/tmp/diconfigbuiltin"";""/tmp/diconfigapp"";`rdb;`rdbnofile])`diconfiga",1,1,within a name the later (app) root overrides the builtin root +true,0,0,q,"10~(cfg.cascade[""/tmp/diconfigbuiltin"";""/tmp/diconfigapp"";`rdb;`rdbnofile])`diconfigb",1,1,name-major: builtin proctype beats app default (would be 5 under root-major) +true,0,0,q,"999~(cfg.cascade[""/tmp/diconfigbuiltin"";""/tmp/diconfigapp"";`rdb;`rdb1])`diconfiga",1,1,name-major: the procname tier (app/rdb1) wins over every less-specific name +true,0,0,q,"10~(cfg.cascade[""/tmp/diconfigbuiltin"";""/tmp/diconfigapp"";`rdb;`rdb1])`diconfigb",1,1,a key absent from the procname tier keeps its most-specific set value (builtin proctype) +true,0,0,q,"not (`) in key cfg.cascade[""/tmp/diconfigbuiltin"";""/tmp/diconfigapp"";`rdb;`rdb1]",1,1,the sentinel merge key is dropped from the returned dict +true,0,0,q,"0=count cfg.cascade[""/tmp/diconfignope1"";""/tmp/diconfignope2"";`rdb;`rdb1]",1,1,an all-missing cascade resolves to an empty dict +true,0,0,q,"10~(cfg.cascade[""/tmp/diconfigbuiltin"";""/tmp/diconfigapp"";`rdb;`default])`diconfigb",1,1,dedup: procname=`default must NOT re-apply the default tier last and override proctype (would be 5 without distinct) +true,0,0,q,"100~(cfg.cascade[""/tmp/diconfigbuiltin"";""/tmp/diconfigapp"";`rdb;`rdb])`diconfiga",1,1,dedup: proctype~procname resolves correctly (each tier once) +fail,0,0,q,"cfg.cascade[""/tmp/diconfigbuiltin"";""/tmp/diconfigapp"";`rdb;(`)]",1,1,cascade rejects a null procname (else it resolves a bogus root/.q tier) +fail,0,0,q,"cfg.cascade[""/tmp/diconfigbuiltin"";""/tmp/diconfigapp"";(`);`rdb1]",1,1,cascade rejects a null proctype +fail,0,0,q,"cfg.cascade[""/tmp/diconfigbuiltin"";""/tmp/diconfigapp"";""rdb"";`rdb1]",1,1,cascade rejects a non-symbol (string) proctype +comment,,,,,,,overrideconfig - command-line override layer applied onto a resolved config dict +true,0,0,q,"1b~(cfg.overrideconfig[basecfg;`enabled`rows!(enlist""1"";enlist""500"")])`enabled",1,1,bool parsed and set from string +true,0,0,q,"500~(cfg.overrideconfig[basecfg;`enabled`rows!(enlist""1"";enlist""500"")])`rows",1,1,long parsed and set from string +true,0,0,q,"basecfg~cfg.overrideconfig[basecfg;()!()]",1,1,empty params returns the config unchanged +true,0,0,q,"0b~(cfg.overrideconfig[basecfg;enlist[`enabled]!enlist ""bad""])`enabled",1,1,unparseable boolean is rejected - the value is left unchanged (not silently set to 0b via a null) +true,0,0,q,"any (exec msg from logtab) like ""*did not parse*""",1,1,rejected boolean override is logged +true,0,0,q,"0xff~(cfg.overrideconfig[basecfg;enlist[`byte]!enlist ""ff""])`byte",1,1,valid hex byte parsed and applied +true,0,0,q,"0x01~(cfg.overrideconfig[basecfg;enlist[`byte]!enlist ""gg""])`byte",1,1,invalid hex byte is rejected (no null - must not slip through as 0x00) +true,0,0,q,"`xx`yy~(cfg.overrideconfig[basecfg;enlist[`syms]!enlist (""xx"";""yy"")])`syms",1,1,symbol list parsed and set from strings +true,0,0,q,""":newdb""~(cfg.overrideconfig[basecfg;enlist[`str]!enlist "":newdb""])`str",1,1,a string-typed setting is overridden as-is (this is what makes .toml string settings overridable) +true,0,0,q,"10h~type (cfg.overrideconfig[basecfg;enlist[`str]!enlist "":newdb""])`str",1,1,string override preserves the string (10h) type +true,0,0,q,"100~(cfg.overrideconfig[basecfg;enlist[`rows]!enlist (""55"";""66"")])`rows",1,1,a multi-value override on a scalar setting is rejected (not silently first) - left unchanged +true,0,0,q,""":olddb""~(cfg.overrideconfig[basecfg;enlist[`str]!enlist (""aa"";""bb"")])`str",1,1,a multi-value override on a string setting is rejected - left unchanged +true,0,0,q,"any (exec msg from logtab) like ""*expected a single value*""",1,1,a multi-value override on a single-valued setting is logged +true,0,0,q,"not `nope in key cfg.overrideconfig[basecfg;enlist[`nope]!enlist enlist""1""]",1,1,an unknown setting is not added to the config dict +true,0,0,q,"(key basecfg)~key cfg.overrideconfig[basecfg;enlist[`nope]!enlist enlist""1""]",1,1,an unknown setting leaves the config keys unchanged +true,0,0,q,"any (exec msg from logtab) like ""*unknown setting*""",1,1,unknown setting logged +true,0,0,q,"(cfg.overrideconfig[basecfg;enlist[`tab]!enlist enlist""1""])[`tab]~basecfg`tab",1,1,a non-basic-type setting (table) is left unchanged +true,0,0,q,"any (exec msg from logtab) like ""*not a basic type*""",1,1,non-basic-type setting logged +fail,0,0,q,cfg.overrideconfig[42;()!()],1,1,overrideconfig rejects a non-dict config +fail,0,0,q,cfg.overrideconfig[basecfg;42],1,1,overrideconfig rejects non-dict params +fail,0,0,q,"cfg.overrideconfig[basecfg;enlist[1]!enlist enlist""x""]",1,1,overrideconfig rejects non-symbol params keys +comment,,,,,,,getapimeta - module api metadata +true,0,0,q,(asc key cfg)~asc exec name from cfg.getapimeta[],1,1,getapimeta documents exactly the module's exports +true,0,0,q,`name`public`descrip`params`return~cols cfg.getapimeta[],1,1,getapimeta rows carry the registry columns +true,0,0,q,1b~(1!cfg.getapimeta[])[`cascade]`public,1,1,a public API function (cascade) is marked public +true,0,0,q,0b~(1!cfg.getapimeta[])[`init]`public,1,1,framework plumbing (init) is not public +comment,,,,,,,logger emission - real logger via a caller-side binary wrapper (must run last; re-inits cfg) +run,0,0,q,kxlogger:use`kx.log,1,1,load the kx.log module +run,0,0,q,kxinst:kxlogger.createLog[],1,1,create a real kx.log instance +run,0,0,q,kxh:hopen `:/tmp/diconfigkxsink.txt,1,1,open a file sink to capture output +run,0,0,q,kxinst.add[kxh;`info],1,1,route info-level output to the file sink +run,0,0,q,"kxbinary:`info`warn`error!({[c;m]kxinst[`info][string[c],"": "",m]};{[c;m]kxinst[`warn][string[c],"": "",m]};{[c;m]kxinst[`error][string[c],"": "",m]})",1,1,wrap the monadic logger into the binary {[c;m]} contract +run,0,0,q,cfg.init[enlist[`log]!enlist kxbinary],1,1,re-init di.config with the conforming binary logger +run,0,0,q,"kxres:cfg.overrideconfig[enlist[`kxrows]!enlist 5;enlist[`kxrows]!enlist enlist""7""]",1,1,drive a real info-level log through the wrapped logger +run,0,0,q,hclose kxh,1,1,flush and close the file sink +true,0,0,q,7~kxres`kxrows,1,1,the override was applied (returned dict carries the new value) +true,0,0,q,"any (read0 `:/tmp/diconfigkxsink.txt) like ""*overrideconfig: setting*""",1,1,the override emitted an info line through the real logger +true,0,0,q,"any (read0 `:/tmp/diconfigkxsink.txt) like ""*kxrows*""",1,1,emitted message names the overridden setting diff --git a/di/dbwrite/dbwrite.md b/di/dbwrite/dbwrite.md new file mode 100644 index 00000000..b5af7fda --- /dev/null +++ b/di/dbwrite/dbwrite.md @@ -0,0 +1,211 @@ +# di.dbwrite + +Write, sort, and attribute utilities for kdb+ processes that persist data to disk (rdb, wdb, tickerlogreplay). + +A **config table** (`tabname`,`att`,`column`,`sort`) drives which columns are sorted and which attributes are applied per table. Load it once with `readcsv` (from a CSV) or `setconfig` (from an in-memory table); `sort` and `savedown` read from module state automatically. A table with no explicit entry falls back to a `default` row; if no config has been loaded, the built-in default (sort every table by `time` ascending) applies. + +--- + +## Features + +- Write an in-memory table to a date-partitioned HDB with `savedown` — enumerates syms, writes, sorts per stored config, then runs `.Q.gc[]` +- Append rows to an existing partition with `appenddown` — enumerates syms and appends; sort separately when the partition is complete +- Sort on-disk table partitions by configured columns using `xasc`, then apply kdb+ attributes (`p`,`s`,`g`,`u`) +- Config loaded once via `readcsv` (from a CSV) or `setconfig` (from a table); inspectable at any time with `getconfig` +- Sort and attribute errors are caught-and-logged (a single partition failure does not halt the run); config and write errors are raised to the caller with a `di.dbwrite:` prefix + +--- + +## Dependencies + +| Dependency | Key | Required | Description | +|---|---|---|---| +| logger | `` `log `` | yes | Functions `info`,`warn`,`error` — each binary `{[c;m] ...}` (context symbol, message string) | + +The `log` dependency must be passed to `init`. The module throws if it is absent, `(::)`, or missing any of the three keys. The `log` value must **already** be a binary `` `info`warn`error!{[c;m]} `` dict — each function takes a context symbol `c` (the calling function) and a message string `m`. `init` performs **no** adaptation and fans the dict out into `.z.m.loginfo`/`.z.m.logwarn`/`.z.m.logerr`. Build it from `di.log` (which exports binary `info`/`warn`/`error`) or hand-roll one; a raw monadic [`kx.log`](https://github.com/KxSystems/logging) instance must be wrapped by the caller first — the module will not do it. + +```q +logger:use`di.log +logdep:`info`warn`error!(logger.info;logger.warn;logger.error) +dbwrite:use`di.dbwrite +dbwrite.init[enlist[`log]!enlist logdep] + +/ a raw kx.log instance is monadic - wrap it to binary {[c;m]} before passing: +/ loginst:(use`kx.log)[`createLog][] +/ `info`warn`error!({[c;m]loginst[`info][string[c],": ",m]};…) +``` + +--- + +## The config table + +| Column | Type | Description | +|---|---|---| +| `tabname` | symbol | Table name, or `` `default `` as a catch-all fallback | +| `att` | symbol | Attribute applied after sort: `p`,`s`,`g`,`u`, or empty (`` ` ``) for none | +| `column` | symbol | Column to sort and/or attribute | +| `sort` | boolean | `1b` — include in the `xasc` sort key; `0b` — attribute only | + +Build it directly and load with `setconfig`, or read it from a CSV with `readcsv`. A CSV must have exactly the four columns `tabname,att,column,sort` in **any** order (the result is normalised to canonical order); a missing, extra, or misnamed column raises a clear `di.dbwrite:` error rather than silently mis-parsing. + +``` +tabname,att,column,sort +trade,p,sym,1 +trade,,price,0 +default,,time,1 +``` + +--- + +## Functions + +| Function | Description | +|---|---| +| `init[deps]` | Wire injected dependencies; must be called first | +| `readcsv[file]` | Read a config CSV and store it in module state | +| `setconfig[t]` | Store a hand-built config table in module state | +| `getconfig[]` | Return the currently stored config (`::` if not yet set) | +| `sort[tabname;dirs]` | Sort on-disk partition(s) for a table and apply attributes per stored config | +| `savedown[dir;part;tabname;data]` | Write an in-memory table to an HDB partition, sort it, then run gc | +| `appenddown[dir;part;tabname;data]` | Append rows to an existing partition (no sort) | +| `applyattr[dloc;colname;att]` | Apply a single kdb+ attribute to an on-disk column | + +--- + +### `init[deps]` + +Wire injected dependencies. Must be called before any other function. Also resets the stored sort config to `(::)`. + +| Arg | Type | Description | +|---|---|---| +| `deps` | dict | Must contain `` `log `` → `` `info`warn`error!(infofn;warnfn;errfn) `` | + +Throws (prefixed `di.dbwrite:`) if `deps` is not a dict, `log` is missing, or the log dict lacks any required key. + +--- + +### `readcsv[file]` + +Read a config CSV and store it in module state (equivalent to calling `setconfig` with the parsed result). The stored config is used by subsequent calls to `sort` and `savedown`. + +| Parameter | Type | Description | +|---|---|---| +| `file` | symbol/hsym/string | Path to the CSV. Symbols are coerced with `hsym`; strings are converted via `hsym `$`. Errors (`di.dbwrite:`) if not a symbol or string. | + +The CSV is loaded with `0:` — the header row names the columns, so their order does not matter — and the resulting table is validated by `checkconfig` before storing. A clear `di.dbwrite:` error is raised if the columns are not exactly `tabname`,`att`,`column`,`sort`, any `tabname`/`column` is null, `sort` is not boolean, or any `att` value is not in `` ` `p`s`g`u ``. Config CSVs are expected to be well-formed: `0:` silently pads/truncates ragged rows and coerces non-`0`/`1` sort values, so those are not separately rejected. + +```q +dbwrite.readcsv `:config/sort.csv +dbwrite.readcsv "config/sort.csv" +``` + +--- + +### `setconfig[t]` + +Store a hand-built config table in module state. Alternative to `readcsv` when the config is constructed in-session rather than read from a file. Validates the table before storing — throws (`di.dbwrite:`) on any schema or content error. + +| Parameter | Type | Description | +|---|---|---| +| `t` | table | Config table with columns `tabname`,`att`,`column`,`sort` | + +```q +dbwrite.setconfig ([] tabname:`trade`default; att:`p`; column:`sym`time; sort:11b) +``` + +--- + +### `getconfig[]` + +Return the currently stored sort config. Returns `(::)` if `init` has been called but neither `readcsv` nor `setconfig` has been called yet. + +```q +dbwrite.getconfig[] +``` + +--- + +### `sort[tabname;dirs]` + +Sort and apply attributes to the on-disk partition(s) for one table, using the config stored by `readcsv` or `setconfig`. Falls back to the built-in default (sort by `time` ascending) if no config has been loaded. + +| Parameter | Type | Description | +|---|---|---| +| `tabname` | symbol | Table name. Errors (`di.dbwrite:`) if not a symbol. | +| `dirs` | hsym, or list of hsyms | Partition directory or directories. | + +Row lookup: the table's own rows → the `default` row → otherwise a warn is logged and `()` returned. Each partition is processed independently; a failure on one is logged and does not halt the rest. + +```q +dbwrite.sort[`trade; (`:hdb/2024.01.02/trade; `:hdb/2024.01.03/trade)] +``` + +--- + +### `savedown[dir;part;tabname;data]` + +Write an in-memory table to a date-partitioned HDB partition, sort it per the stored config, then run `.Q.gc[]`. Enumerates symbol columns against the HDB sym file before writing. Sorting and attributes are applied by `sort` *after* the write — so an attribute like `p#` is only applied once its column is correctly grouped. + +| Parameter | Type | Description | +|---|---|---| +| `dir` | hsym | HDB root directory (e.g. `` `:hdb ``). | +| `part` | date/month/int | Partition value. | +| `tabname` | symbol | Table name — determines the partition subdirectory. | +| `data` | table | In-memory table to write. | + +Throws on write failure. + +```q +dbwrite.savedown[`:hdb; 2024.01.02; `trade; data] +``` + +--- + +### `appenddown[dir;part;tabname;data]` + +Append rows to an existing on-disk partition (enumerates syms); does **not** sort. Keeping sort separate allows multiple intraday appends without re-sorting a growing partition on each call — sort once when the partition is complete. + +| Parameter | Type | Description | +|---|---|---| +| `dir` | hsym | HDB root directory. | +| `part` | date/month/int | Partition value. | +| `tabname` | symbol | Table name. | +| `data` | table | Rows to append. | + +Throws (prefixed `di.dbwrite:`) if the partition does not exist, or on write failure. + +```q +/ intraday: append each batch as it arrives +dbwrite.appenddown[`:hdb; 2024.01.02; `trade; batch] +/ end-of-day: sort once when done +dbwrite.sort[`trade; .Q.par[`:hdb; 2024.01.02; `trade]] +``` + +--- + +### `applyattr[dloc;colname;att]` + +Apply a single kdb+ attribute to one on-disk column (best-effort: logs and swallows errors so a run continues). A non-attribute `att` (the empty sentinel or any value outside `` `p`s`g`u ``) is a silent no-op. + +```q +dbwrite.applyattr[`:hdb/2024.01.02/trade; `sym; `p] +``` + +--- + +## Running tests + +```q +k4unit:use`di.k4unit +k4unit.moduletest`di.dbwrite +``` + +The suite injects binary mock loggers (`{[c;m] ...}`): a no-op logger, and a capturing logger that records `(level;context;msg)` so log behaviour can be asserted. It also wires a real `kx.log` instance (wrapped caller-side into the binary `{[c;m]}` contract) through `init` to confirm the module works end-to-end against the system logger. On-disk behaviour (sort, attributes, `savedown`/`appenddown`) is exercised against real splayed partitions and cleaned up afterwards. It covers: dependency-injection validation; `readcsv` stores-and-verifies / column-order independence / header-validation failures; `setconfig` happy path and validation failures; `sort` edge cases / resolution / on-disk results / multi-dir / non-fatal partition failure; `savedown` write+sort (default and explicit config, and a table without `sym`); `appenddown` append-without-sort then explicit sort, and the non-existent-partition error; `applyattr`; the binary `{[c;m]}` context-tagged logging contract; and the real `kx.log` integration. + +--- + +## Exported symbols + +```q +export:([init;readcsv;setconfig;getconfig;sort;applyattr;savedown;appenddown]) +``` diff --git a/di/dbwrite/dbwrite.q b/di/dbwrite/dbwrite.q new file mode 100644 index 00000000..3eb6306c --- /dev/null +++ b/di/dbwrite/dbwrite.q @@ -0,0 +1,236 @@ +/ dbwrite - write, sort, and attribute utilities for on-disk data +/ used by processes that persist data to disk (rdb, wdb, tickerlogreplay) + +/ attributes that may legitimately appear in a config (empty leaves a column unattributed) +validatts:``p`s`g`u; + +/ built-in fallback config - sort every table by time ascending when no config is supplied +defaultparams:([] tabname:enlist`default; att:enlist`; column:enlist`time; sort:enlist 1b); + +init:{[deps] + / wire the injected logger - required, no silent fallback. deps: a dict with a `log key + / holding a binary `info`warn`error dict of {[c;m]} loggers (context symbol, message string), + / from di.log or hand-rolled. no adaptation here, so a monadic kx.log instance must be wrapped + / first. e.g. di.dbwrite.init[enlist[`log]!enlist logdep] + if[99h<>type deps; + '"di.dbwrite: deps must be a dict with a `log key"]; + if[not `log in key deps; + '"di.dbwrite: log dependency is required; pass `info`warn`error functions keyed on `log"]; + if[99h<>type deps`log; + '"di.dbwrite: log value must be a dict of `info`warn`error functions"]; + if[not all (`info`warn`error) in key deps`log; + '"di.dbwrite: log dict must have `info`warn`error keys; got: ",(", " sv string key deps`log)]; + .z.m.loginfo:deps[`log]`info; + .z.m.logwarn:deps[`log]`warn; + .z.m.logerr:deps[`log]`error; + .z.m.sortconfig:(::); + }; + +readcsv:{[file] + / read a sort-config csv and store it in .z.m.sortconfig; used by sort and savedown + / the csv must have the columns tabname,att,column,sort (in any order) + / file: hsym, bare symbol, or string path + if[10h=type file; file:hsym `$file]; + if[-11h=type file; if[not ":" = first string file; file:hsym file]]; + if[not -11h=type file; + .z.m.logerr[`readcsv;err:"di.dbwrite: readcsv file must be a symbol or string path, got type ",string type file]; + 'err; + ]; + t:parsecsv @[readfile; file; readerr[file]]; + checkconfig t; + .z.m.loginfo[`readcsv;"read ",(string count t)," sort config row(s) from ",string file]; + .z.m.sortconfig:t; + }; + +setconfig:{[t] + / set .z.m.sortconfig from an in-memory table; alternative to readcsv when config is built in-session + / t must be a table with columns tabname,att,column,sort + checkconfig t; + .z.m.sortconfig:t; + }; + +getconfig:{[] + / return the current sort config stored in .z.m.sortconfig; (::) if not yet set + :.z.m.sortconfig; + }; + +/ internal - protected file read; only the i/o so a genuine read failure gets the readerr message +readfile:{[file] + / returns the raw csv lines; header validation and parsing happen in parsecsv + .z.m.loginfo[`readfile;"reading sort config from ",string file]; + :read0 file; + }; + +/ internal - log and rethrow a csv read failure +readerr:{[file;e] + / build the message once, log it under the read context, then rethrow it to the caller + m:"di.dbwrite: failed to read ",string[file],": ",e; + .z.m.logerr[`readerr;m]; + 'm; + }; + +/ internal - parse csv lines into a config table +parsecsv:{[lines] + / load via 0: - the header row (enlist delim) names the columns, so column order does not + / matter; cast sort to boolean if present. structural validation is left to checkconfig. + t:((count "," vs first lines)#"S";enlist",") 0: lines; + :$[`sort in cols t; update sort:"B"$string sort from t; t]; + }; + +/ internal - validate a sort-config table, signalling a clear error if it is malformed +checkconfig:{[t] + / guards every sort call so a hand-built or csv-derived table is rejected early if wrong + if[98h<>type t; + .z.m.logerr[`checkconfig;err:"di.dbwrite: config must be a table with columns `tabname`att`column`sort"]; + 'err; + ]; + c:cols t; + badcols:c where not c in `tabname`att`column`sort; + if[count badcols; + .z.m.logerr[`checkconfig;err:"di.dbwrite: unrecognised config column(s): ",", " sv string badcols]; + 'err; + ]; + missingcols:(`tabname`att`column`sort) where not (`tabname`att`column`sort) in c; + if[count missingcols; + .z.m.logerr[`checkconfig;err:"di.dbwrite: missing required config column(s): ",", " sv string missingcols]; + 'err; + ]; + if[any null t`tabname; + .z.m.logerr[`checkconfig;err:"di.dbwrite: config tabname must not be null"]; + 'err; + ]; + if[any null t`column; + .z.m.logerr[`checkconfig;err:"di.dbwrite: config column must not be null"]; + 'err; + ]; + if[not 1h=type t`sort; + .z.m.logerr[`checkconfig;err:"di.dbwrite: the sort column must be boolean"]; + 'err; + ]; + badatts:at where not (at:distinct t`att) in validatts; + if[count badatts; + .z.m.logerr[`checkconfig;err:"di.dbwrite: unrecognised attribute(s) in att column: ",", " sv string badatts]; + 'err; + ]; + }; + +sort:{[tabname;dirs] + / sort and apply attributes to the on-disk partition dirs for one table using .z.m.sortconfig + / both the sort column order and the att assignments (p/s/g/u) are driven by the config + / falls back to defaultparams if config has not been set via readcsv or setconfig + / tabname: symbol; dirs: hsym or list of hsyms (partition directories e.g. from .Q.par) + config:$[(::)~.z.m.sortconfig; defaultparams; .z.m.sortconfig]; + checkconfig config; + if[not -11h=type tabname; + .z.m.logerr[`sort;err:"di.dbwrite: tabname must be a symbol, got type ",string type tabname]; + 'err; + ]; + st:string tabname; + .z.m.loginfo[`sort;"sorting the ",st," table"]; + sp:getsortparams[config;tabname]; + if[not count sp; :()]; + sortdir[sp] each distinct (),dirs; + .z.m.loginfo[`sort;"finished sorting the ",st," table"]; + }; + +/ internal - resolve which config rows apply to a table +getsortparams:{[config;tab] + / tab is the table-name symbol (NOT a table); named to avoid clashing with the tabname column + / a table uses its own rows; unlisted tables fall back to the default row, else are skipped + if[count tabsp:select from config where tabname=tab; + .z.m.loginfo[`getsortparams;"sort params found for: ",string[tab]]; + :tabsp; + ]; + if[count defsp:select from config where tabname=`default; + .z.m.loginfo[`getsortparams;"no sort params for: ",string[tab],"; using defaults"]; + :defsp; + ]; + .z.m.logwarn[`getsortparams;"no sort params for: ",string[tab],"; skipping sort"]; + :0#config; + }; + +/ internal - log a sort failure without rethrowing so remaining partitions still run +sorterr:{[sc;dl;e] + / a single partition failure should not halt the whole run + .z.m.logerr[`sorterr;"failed to sort ",string[dl]," by ",(", " sv string sc),": ",e]; + :(); + }; + +/ internal - sort one partition directory by the given columns +sortcolumns:{[dloc;sortcols] + / split out of sortdir so the conditional body there stays a single statement + .z.m.loginfo[`sortcolumns;"sorting ",string[dloc]," by: ",", " sv string sortcols]; + .[xasc;(sortcols;dloc); + sorterr[sortcols;dloc]]; + }; + +/ internal - sort columns and apply attributes for a single on-disk partition directory +sortdir:{[sp;dloc] + / sort by the columns flagged sort=1b, then hand every row to applyattr (it skips non-attributes) + sortcols:exec column from sp where sort, not null column; + if[count sortcols; sortcolumns[dloc;sortcols]]; + applyattr[dloc;;]'[sp`column;sp`att]; + }; + +/ internal - log an attribute application failure without rethrowing +attrerr:{[dl;cn;at;e] + / logs failure and continues so other columns and partitions still get processed + .z.m.logerr[`attrerr;"unable to apply ",string[at]," attr to ",string[cn]," in ",string[dl],": ",e]; + :(); + }; + +applyattr:{[dloc;colname;att] + / apply a single kdb+ attribute to an on-disk column; logs and swallows errors so a run continues + / dloc: hsym (partition directory e.g. `:hdb/2024.01.01/trade); colname: symbol; att: symbol (p|s|g|u or empty) + / skip anything that is not a real attribute - covers the empty none-sentinel and any bad value + if[not att in `p`s`g`u; :()]; + .z.m.loginfo[`applyattr;"applying ",string[att]," attr to ",string[colname]," in ",string dloc]; + .[{@[x;y;z#]}; + (dloc;colname;att); + attrerr[dloc;colname;att]]; + }; + +savedown:{[dir;part;tabname;data] + / write an in-memory table to a date-partitioned hdb partition, then sort it per .z.m.sortconfig + / dir: hdb root (hsym); part: partition value (date/month/int); tabname: symbol; data: in-memory table + / enumerates syms against the hdb sym file; sorting and attributes are driven by .z.m.sortconfig + .z.m.loginfo[`savedown;"saving ",string[tabname]," partition ",string[part]," to ",string dir]; + path:` sv (.Q.par[dir;part;tabname];`); + path set .Q.en[dir;data]; + sort[tabname;path]; + .z.m.loginfo[`savedown;"finished saving ",string tabname]; + gc[]; + }; + +appenddown:{[dir;part;tabname;data] + / append rows to an existing on-disk partition (enumerates syms); does not sort + / call sort separately once the partition is complete, to avoid re-sorting on every append + / dir: hdb root (hsym); part: partition value; tabname: symbol; data: in-memory table + .z.m.loginfo[`appenddown;"appending ",string[tabname]," partition ",string[part]," in ",string dir]; + path:` sv (.Q.par[dir;part;tabname];`); + if[not count @[key;path;{`$()}]; + .z.m.logerr[`appenddown;err:"di.dbwrite: appenddown partition does not exist at ",string path]; + 'err; + ]; + .[path;();,;.Q.en[dir;data]]; + .z.m.loginfo[`appenddown;"finished appending ",string tabname]; + }; + +/ internal - render a memory-usage dict as a "key=val MB; ..." string +fmtmem:{[m] + / m: dict of MB values keyed by .Q.w field name + :"; " sv "=" sv' flip (string key m; (string value m),\:" MB"); + }; + +/ format current process memory stats as a loggable string +memstats:{[] + / convert .Q.w[] (bytes) to MB and render it via fmtmem + :"mem stats: ",fmtmem `long$.Q.w[]%1048576; + }; + +gc:{[] + / run .Q.gc[] and log before/after memory stats + .z.m.loginfo[`gc;"starting garbage collect. ",memstats[]]; + r:.Q.gc[]; + .z.m.loginfo[`gc;"garbage collection returned ",(string `long$r%1048576),"MB. ",memstats[]]; + }; \ No newline at end of file diff --git a/di/dbwrite/init.q b/di/dbwrite/init.q new file mode 100644 index 00000000..6bf98f85 --- /dev/null +++ b/di/dbwrite/init.q @@ -0,0 +1,6 @@ +/ dbwrite module - write, sort, and attribute utilities for on-disk data +/ used by processes that persist data to disk (rdb, wdb, tickerlogreplay) + +\l ::dbwrite.q + +export:([init;readcsv;setconfig;getconfig;sort;applyattr;savedown;appenddown]) diff --git a/di/dbwrite/test.csv b/di/dbwrite/test.csv new file mode 100644 index 00000000..f6df53fa --- /dev/null +++ b/di/dbwrite/test.csv @@ -0,0 +1,246 @@ +action,ms,bytes,lang,code,repeat,minver,comment +before,0,0,q,dbwrite:use`di.dbwrite,1,1,load module +before,0,0,q,mylog:`info`warn`error!({[c;m]};{[c;m]};{[c;m]}),1,1,define no-op binary mock logger {[c;m]} +before,0,0,q,"caplog:([] fn:`symbol$();ctx:`symbol$();msg:())",1,1,initialise log capture table +before,0,0,q,logcap:`info`warn`error!({[c;m] `caplog upsert(`info;c;m)};{[c;m] `caplog upsert(`warn;c;m)};{[c;m] `caplog upsert(`error;c;m)}),1,1,define capturing binary mock logger {[c;m]} +before,0,0,q,"rmrf:{[d] if[11h=type k:key d; rmrf each ` sv/:d,/:k]; @[hdel;d;{}]}",1,1,recursive delete helper for on-disk cleanup +before,0,0,q,"`:tmp_dbw_valid.csv 0: (""tabname,att,column,sort"";""trade,p,sym,1"";""trade,,time,0"";""default,p,sym,1"")",1,1,write valid sort csv +before,0,0,q,"`:tmp_dbw_badcols.csv 0: (""tabname,attr,col,sort"";""trade,p,sym,1"")",1,1,write csv with bad column names +before,0,0,q,"`:tmp_dbw_badatt.csv 0: (""tabname,att,column,sort"";""trade,z,sym,1"")",1,1,write csv with invalid attribute value +before,0,0,q,"`:tmp_dbw_3col.csv 0: (""tabname,att,sort"";""trade,p,1"")",1,1,write csv missing a column +before,0,0,q,"`:tmp_dbw_empty.csv 0: enlist ""tabname,att,column,sort""",1,1,write header-only sort csv +before,0,0,q,"`:tmp_dbw_nodfl.csv 0: (""tabname,att,column,sort"";""trade,p,sym,1"")",1,1,write sort csv with no default row +before,0,0,q,"`:tmp_dbw_5col.csv 0: (""tabname,att,column,sort,extra"";""trade,p,sym,1,foo"")",1,1,write csv with an extra column +before,0,0,q,"`:tmp_dbw_reorder.csv 0: (""tabname,att,sort,column"";""trade,p,1,sym"";""trade,,0,time"";""default,p,1,sym"")",1,1,write valid csv with columns reordered +before,0,0,q,`:tmp_dbw_nohdr.csv 0: (),1,1,write a zero-line csv (no header) +before,0,0,q,dbwrite.init[enlist[`log]!enlist mylog],1,1,init with no-op logger +before,0,0,q,cfg:([] tabname:`trade`trade`default; att:`p``p; column:`sym`time`sym; sort:101b),1,1,reusable valid config table (hand-built) +before,0,0,q,nodflcfg:([] tabname:enlist`trade; att:enlist`p; column:enlist`sym; sort:enlist 1b),1,1,reusable config with a trade row but no default + +comment,,,,,,,init - dependency injection validation (type errors / empty / null) +fail,0,0,q,dbwrite.init[(::)],1,1,init rejects :: as deps +fail,0,0,q,dbwrite.init[42],1,1,init rejects a non-dict deps value +fail,0,0,q,dbwrite.init[()!()],1,1,init rejects empty dict +fail,0,0,q,dbwrite.init[enlist[`other]!enlist mylog],1,1,init rejects dict missing log key +fail,0,0,q,dbwrite.init[enlist[`log]!enlist(::)],1,1,init rejects null log dep +fail,0,0,q,dbwrite.init[enlist[`log]!enlist 42],1,1,init rejects non-dict log value +fail,0,0,q,dbwrite.init[enlist[`log]!enlist(`info`warn!(mylog`info;mylog`warn))],1,1,init rejects log dict missing required key +true,0,0,q,"""di.dbwrite:""~11#@[{dbwrite.init[(::)]};(::);{x}]",1,1,error message is prefixed di.dbwrite +run,0,0,q,dbwrite.init[enlist[`log]!enlist mylog],1,1,re-init with valid log dep + +comment,,,,,,,readcsv - stores a config from a csv into module state +run,0,0,q,dbwrite.readcsv `:tmp_dbw_valid.csv,1,1,readcsv stores config from a valid csv without error +true,0,0,q,3=count dbwrite.getconfig[],1,1,stored config has 3 rows +true,0,0,q,`tabname`att`column`sort~cols dbwrite.getconfig[],1,1,stored config has the correct column schema +true,0,0,q,(dbwrite.getconfig[])~([] tabname:`trade`trade`default; att:`p``p; column:`sym`time`sym; sort:101b),1,1,stored config matches csv content +run,0,0,q,dbwrite.readcsv `:tmp_dbw_empty.csv,1,1,readcsv stores an empty config for a header-only csv +true,0,0,q,0=count dbwrite.getconfig[],1,1,empty csv produces empty stored config +run,0,0,q,dbwrite.readcsv `:tmp_dbw_reorder.csv,1,1,readcsv with reordered columns stores correct config +true,0,0,q,(`tabname`att`column`sort#dbwrite.getconfig[])~([] tabname:`trade`trade`default; att:`p``p; column:`sym`time`sym; sort:101b),1,1,readcsv is column-order independent (0: names columns from the header) +true,0,0,q,(asc cols dbwrite.getconfig[])~`att`column`sort`tabname,1,1,readcsv reads all four columns regardless of input order +run,0,0,q,"dbwrite.readcsv ""tmp_dbw_valid.csv""",1,1,readcsv accepts a string file path +true,0,0,q,3=count dbwrite.getconfig[],1,1,string path readcsv stores the correct number of rows +true,0,0,q,(dbwrite.getconfig[])~([] tabname:`trade`trade`default; att:`p``p; column:`sym`time`sym; sort:101b),1,1,string path readcsv stores the correct config + +comment,,,,,,,readcsv - input and header validation failures +fail,0,0,q,dbwrite.readcsv `:nonexistent_file.csv,1,1,readcsv errors on a missing file +fail,0,0,q,dbwrite.readcsv 42,1,1,readcsv errors on a non-symbol non-string file argument +fail,0,0,q,dbwrite.readcsv `:tmp_dbw_badcols.csv,1,1,readcsv rejects a csv with wrong column names +fail,0,0,q,dbwrite.readcsv `:tmp_dbw_3col.csv,1,1,readcsv rejects a csv missing a required column +fail,0,0,q,dbwrite.readcsv `:tmp_dbw_5col.csv,1,1,readcsv rejects a csv with an extra column +fail,0,0,q,dbwrite.readcsv `:tmp_dbw_nohdr.csv,1,1,readcsv rejects a csv with no header row +fail,0,0,q,dbwrite.readcsv `:tmp_dbw_badatt.csv,1,1,readcsv rejects a csv with invalid attribute values + +comment,,,,,,,setconfig - stores a hand-built table into module state +run,0,0,q,dbwrite.setconfig[cfg],1,1,setconfig accepts a valid hand-built table +true,0,0,q,(dbwrite.getconfig[])~cfg,1,1,getconfig returns the table stored by setconfig +run,0,0,q,dbwrite.setconfig[0#cfg],1,1,setconfig accepts an empty (but correctly typed) table +true,0,0,q,0=count dbwrite.getconfig[],1,1,empty table stored correctly + +comment,,,,,,,setconfig - config validation (type errors / bad content) +fail,0,0,q,dbwrite.setconfig[42],1,1,setconfig rejects a non-table value +fail,0,0,q,dbwrite.setconfig[([] tabname:enlist`trade; bad:enlist`p; column:enlist`sym; sort:enlist 1b)],1,1,setconfig rejects unrecognised config columns +fail,0,0,q,dbwrite.setconfig[([] tabname:enlist`trade; att:enlist`p; column:enlist`sym)],1,1,setconfig rejects a missing required config column +fail,0,0,q,dbwrite.setconfig[([] tabname:enlist`trade; att:enlist`z; column:enlist`sym; sort:enlist 1b)],1,1,setconfig rejects unknown attribute values +fail,0,0,q,dbwrite.setconfig[([] tabname:enlist`trade; att:enlist`p; column:enlist`sym; sort:enlist 1)],1,1,setconfig rejects a non-boolean sort column +fail,0,0,q,dbwrite.setconfig[([] tabname:enlist`; att:enlist`p; column:enlist`sym; sort:enlist 1b)],1,1,setconfig rejects a null tabname in config +fail,0,0,q,dbwrite.setconfig[([] tabname:enlist`trade; att:enlist`p; column:enlist`; sort:enlist 1b)],1,1,setconfig rejects a null column in config + +comment,,,,,,,sort - happy path using stored config (set via setconfig or readcsv) +run,0,0,q,dbwrite.setconfig[cfg],1,1,set config to hand-built cfg +run,0,0,q,dbwrite.sort[`trade;()],1,1,sort with a hand-built stored config and no partitions +run,0,0,q,dbwrite.readcsv `:tmp_dbw_valid.csv,1,1,set config via readcsv +run,0,0,q,dbwrite.sort[`trade;()],1,1,sort with a csv-loaded stored config and no partitions + +comment,,,,,,,sort - tabname type validation +fail,0,0,q,dbwrite.sort[42;enlist`:/],1,1,sort errors on a non-symbol table name +true,0,0,q,"""di.dbwrite:""~11#@[{dbwrite.sort[42;enlist`:/]};(::);{x}]",1,1,non-symbol table name gives a di.dbwrite:-prefixed error + +comment,,,,,,,sort - edge cases (empty config / null att / empty dirs / atom dir) +run,0,0,q,dbwrite.setconfig[0#cfg],1,1,set empty config +true,0,0,q,()~dbwrite.sort[`trade;enlist`:/],1,1,empty config yields no work and returns () +run,0,0,q,dbwrite.setconfig[([] tabname:enlist`trade; att:enlist`; column:enlist`sym; sort:enlist 0b)],1,1,set config with a null (no) attribute row +run,0,0,q,dbwrite.sort[`trade;enlist`:/],1,1,sort accepts a null (no) attribute row +run,0,0,q,dbwrite.setconfig[cfg],1,1,set config to cfg +run,0,0,q,dbwrite.sort[`trade;()],1,1,sort handles an empty partition list without error +run,0,0,q,dbwrite.sort[`trade;`:/],1,1,sort accepts a single atom hsym dir not just a list + +comment,,,,,,,sort - params resolution (table-specific / default / none / default fallback) +run,0,0,q,dbwrite.setconfig[cfg],1,1,set config to cfg +run,0,0,q,dbwrite.sort[`trade;enlist`:/],1,1,sort uses the table-specific params row +run,0,0,q,dbwrite.sort[`other;enlist`:/],1,1,sort falls back to the default params row +run,0,0,q,dbwrite.setconfig[nodflcfg],1,1,set config to nodflcfg (no default row) +true,0,0,q,()~dbwrite.sort[`other;enlist`:/],1,1,sort returns () when no params found and no default row +run,0,0,q,dbwrite.init[enlist[`log]!enlist mylog],1,1,reset module state (clears sortconfig to (::)) +run,0,0,q,dbwrite.sort[`anytable;()],1,1,sort with unset sortconfig falls back to the built-in default config + +comment,,,,,,,sort - on-disk end-to-end (explicit config sorts by sym and applies p#) +run,0,0,q,`:dbw_sort_tp/.d set `sym`price,1,1,write column order file +run,0,0,q,`:dbw_sort_tp/sym set `IBM`AAPL`MSFT,1,1,write unsorted sym column +run,0,0,q,`:dbw_sort_tp/price set 200 100 300f,1,1,write price column +run,0,0,q,dbwrite.setconfig[cfg],1,1,set config to cfg before sort +run,0,0,q,dbwrite.sort[`trade;`:dbw_sort_tp/],1,1,sort the trade partition by its config +true,0,0,q,`AAPL`IBM`MSFT~exec sym from get `:dbw_sort_tp/,1,1,sym sorted ascending on disk +true,0,0,q,`p=attr get `:dbw_sort_tp/sym,1,1,p attribute applied to sym on disk + +comment,,,,,,,sort - on-disk end-to-end (default config sorts by time) +run,0,0,q,`:dbw_def_tp/.d set `time`sym,1,1,write column order file +run,0,0,q,`:dbw_def_tp/time set 2024.01.01D09:00 2024.01.01D08:00,1,1,write unsorted time column +run,0,0,q,`:dbw_def_tp/sym set `IBM`AAPL,1,1,write sym column +run,0,0,q,dbwrite.init[enlist[`log]!enlist mylog],1,1,reset sortconfig to (::) for default-fallback test +run,0,0,q,dbwrite.sort[`anytable;`:dbw_def_tp/],1,1,sort using the built-in default config +true,0,0,q,(asc exec time from get `:dbw_def_tp/)~exec time from get `:dbw_def_tp/,1,1,time sorted ascending by the default config + +comment,,,,,,,sort - multiple partition dirs in one call and a non-fatal partition failure +run,0,0,q,`:dbw_md1/.d set `sym`px,1,1,write md1 column order +run,0,0,q,`:dbw_md1/sym set `c`a`b,1,1,write md1 unsorted sym +run,0,0,q,`:dbw_md1/px set 1 2 3f,1,1,write md1 price +run,0,0,q,`:dbw_md2/.d set `sym`px,1,1,write md2 column order +run,0,0,q,`:dbw_md2/sym set `b`c`a,1,1,write md2 unsorted sym +run,0,0,q,`:dbw_md2/px set 1 2 3f,1,1,write md2 price +run,0,0,q,dbwrite.setconfig[cfg],1,1,set config to cfg +run,0,0,q,dbwrite.sort[`trade;(`:dbw_md1/;`:dbw_md2/)],1,1,sort two partition dirs in one call +true,0,0,q,`a`b`c~exec sym from get `:dbw_md1/,1,1,first partition sorted +true,0,0,q,`a`b`c~exec sym from get `:dbw_md2/,1,1,second partition sorted +run,0,0,q,`:dbw_good/.d set `sym`px,1,1,write good-partition column order +run,0,0,q,`:dbw_good/sym set `c`a`b,1,1,write good-partition unsorted sym +run,0,0,q,`:dbw_good/px set 1 2 3f,1,1,write good-partition price +run,0,0,q,dbwrite.sort[`trade;(`:dbw_good/;`:/)],1,1,sort a good dir alongside a bad one +true,0,0,q,`a`b`c~exec sym from get `:dbw_good/,1,1,good partition still sorted despite the bad one failing + +comment,,,,,,,applyattr - applies an attribute on disk and is a no-op for non-attributes +run,0,0,q,`:dbw_attr_tp/.d set `sym`price,1,1,write column order file +run,0,0,q,`:dbw_attr_tp/sym set `IBM`AAPL`MSFT,1,1,write sym column +run,0,0,q,`:dbw_attr_tp/price set 200 100 300f,1,1,write price column +run,0,0,q,dbwrite.applyattr[`:dbw_attr_tp/;`sym;`p],1,1,apply p attr to sym +true,0,0,q,`p=attr get `:dbw_attr_tp/sym,1,1,p attribute applied to sym on disk +run,0,0,q,dbwrite.applyattr[`:dbw_attr_tp/;`price;`z],1,1,applyattr is a silent no-op for an invalid attribute +true,0,0,q,`=attr get `:dbw_attr_tp/price,1,1,invalid attribute leaves the column unattributed + +comment,,,,,,,savedown - write an in-memory table to an hdb partition then sort it +run,0,0,q,"sdtbl:([]time:2024.01.01D09:00 2024.01.01D08:00;sym:`IBM`AAPL;price:100 200f)",1,1,unsorted test table +run,0,0,q,dbwrite.init[enlist[`log]!enlist mylog],1,1,reset sortconfig to (::) for default-config savedown +run,0,0,q,dbwrite.savedown[`:dbw_hdb;2024.01.01;`trade;sdtbl],1,1,savedown with the default config (sort by time) +run,0,0,q,"sdpath:` sv (.Q.par[`:dbw_hdb;2024.01.01;`trade];`)",1,1,resolve the partition path +true,0,0,q,2=count get sdpath,1,1,two rows written to the partition +true,0,0,q,(asc exec time from get sdpath)~exec time from get sdpath,1,1,partition sorted by time after savedown +true,0,0,q,0type deps; + '"di.eodtime: deps must be a dict with `log key"]; + if[not `log in key deps; + '"di.eodtime: log dependency is required; pass at minimum `info!{[c;m]} keyed on `log"]; + if[99h<>type deps`log; + '"di.eodtime: log value must be a dict; pass at minimum `info!{[c;m]}"]; + if[not `info in key deps`log; + '"di.eodtime: log dict must have at minimum an `info key; got: ",(", " sv string key deps`log)]; + .z.m.loginfo:(deps`log)`info; + .z.m.rolltimezone:$[`rolltimezone in key deps;deps`rolltimezone;`$"GMT"]; + .z.m.datatimezone:$[`datatimezone in key deps;deps`datatimezone;`$"GMT"]; + .z.m.rolltimeoffset:$[`rolltimeoffset in key deps;deps`rolltimeoffset;0D00:00:00.000]; + .z.m.dailyadj:getdailyadjustment[]; + .z.m.d:getday[.z.p]; + .z.m.nextroll:getroll[.z.p]; + .z.m.loginfo[`eodtime;"initialised with rolltimezone=",string[.z.m.rolltimezone]," datatimezone=",string[.z.m.datatimezone]," rolltimeoffset=",string .z.m.rolltimeoffset]; + }; \ No newline at end of file diff --git a/di/eodtime/init.q b/di/eodtime/init.q new file mode 100644 index 00000000..027c0e12 --- /dev/null +++ b/di/eodtime/init.q @@ -0,0 +1,7 @@ +/ end-of-day time management - date resolution, roll scheduling and data timestamp adjustment + +tz:use`di.tz + +\l ::eodtime.q + +export:([init;getd;getnextroll;getdailyadj;getroll;getdailyadjustment;setnextroll;setdailyadj;setd]) diff --git a/di/eodtime/test.csv b/di/eodtime/test.csv new file mode 100644 index 00000000..1fdd0e36 --- /dev/null +++ b/di/eodtime/test.csv @@ -0,0 +1,92 @@ +action,ms,bytes,lang,code,repeat,minver,comment +comment,,,,,,,setup - load module and initialise with gmt defaults and log dep +before,0,0,q,eodtime:use`di.eodtime,1,1,load di.eodtime module +before,0,0,q,logdep:`info`warn`error!({[c;m]};{[c;m]};{[c;m]}),1,1,silent no-op log mock - binary {[c;m]} matches the log contract +before,0,0,q,eodtime.init[`rolltimezone`datatimezone`rolltimeoffset`log!(`$"GMT";`$"GMT";0D00:00:00.000;logdep)],1,1,init with gmt defaults and log dep + +comment,,,,,,,getd +true,0,0,q,-14h=type eodtime.getd[],1,1,getd returns date type + +comment,,,,,,,getnextroll +true,0,0,q,-12h=type eodtime.getnextroll[],1,1,getnextroll returns timestamp type +true,0,0,q,eodtime.getnextroll[]>.z.p,1,1,getnextroll returns future timestamp + +comment,,,,,,,getdailyadj +true,0,0,q,-16h=type eodtime.getdailyadj[],1,1,getdailyadj returns timespan type +true,0,0,q,0D=eodtime.getdailyadj[],1,1,getdailyadj returns 0D for gmt datatimezone + +comment,,,,,,,getdailyadjustment +true,0,0,q,-16h=type eodtime.getdailyadjustment[],1,1,getdailyadjustment returns timespan type +true,0,0,q,0D=eodtime.getdailyadjustment[],1,1,getdailyadjustment returns 0D for gmt datatimezone + +comment,,,,,,,getroll - gmt zero offset +true,0,0,q,-12h=type eodtime.getroll[2025.01.01D12:00:00.000000000],1,1,getroll returns timestamp type +true,0,0,q,2025.01.02D00:00:00.000000000=eodtime.getroll[2025.01.01D12:00:00.000000000],1,1,getroll returns next midnight for gmt zero offset + +comment,,,,,,,getroll - gmt with rolltimeoffset +run,0,0,q,eodtime.init[`rolltimezone`datatimezone`rolltimeoffset`log!(`$"GMT";`$"GMT";0D17:00:00.000;logdep)],1,1,reinit with 5pm gmt roll time +true,0,0,q,2025.01.01D17:00:00.000000000=eodtime.getroll[2025.01.01D12:00:00.000000000],1,1,getroll returns todays 5pm when before roll time +true,0,0,q,2025.01.02D17:00:00.000000000=eodtime.getroll[2025.01.01D18:00:00.000000000],1,1,getroll returns next day 5pm when past roll time + +comment,,,,,,,setnextroll +run,0,0,q,eodtime.init[`rolltimezone`datatimezone`rolltimeoffset`log!(`$"GMT";`$"GMT";0D00:00:00.000;logdep)],1,1,reinit with gmt defaults +run,0,0,q,eodtime.setnextroll[2025.06.01D00:00:00.000000000],1,1,setnextroll does not error +true,0,0,q,2025.06.01D00:00:00.000000000=eodtime.getnextroll[],1,1,getnextroll reflects setnextroll + +comment,,,,,,,setdailyadj +run,0,0,q,eodtime.setdailyadj[0D01:00:00.000000000],1,1,setdailyadj does not error +true,0,0,q,0D01:00:00.000000000=eodtime.getdailyadj[],1,1,getdailyadj reflects setdailyadj + +comment,,,,,,,setd +run,0,0,q,eodtime.setd[2025.06.01],1,1,setd does not error +true,0,0,q,2025.06.01=eodtime.getd[],1,1,getd reflects setd + +comment,,,,,,,utc-equivalent timezone shortcuts (GMT/UTC/Etc/GMT) +run,0,0,q,eodtime.init[`rolltimezone`datatimezone`rolltimeoffset`log!(`$"UTC";`$"Etc/GMT";0D00:00:00.000;logdep)],1,1,reinit with UTC/Etc/GMT shortcuts +true,0,0,q,0D=eodtime.getdailyadjustment[],1,1,Etc/GMT datatimezone returns 0D without lookup +true,0,0,q,2025.01.02D00:00:00.000000000=eodtime.getroll[2025.01.01D12:00:00.000000000],1,1,UTC rolltimezone returns correct roll + +comment,,,,,,,non-gmt rolltimezone (Europe/London) +run,0,0,q,eodtime.init[`rolltimezone`datatimezone`rolltimeoffset`log!(`$"Europe/London";`$"GMT";0D00:00:00.000;logdep)],1,1,reinit with London rolltimezone +true,0,0,q,-12h=type eodtime.getroll[2025.01.01D12:00:00.000000000],1,1,getroll returns timestamp type for non-gmt rolltimezone +true,0,0,q,2025.01.02D00:00:00.000000000=eodtime.getroll[2025.01.01D12:00:00.000000000],1,1,getroll correct for london winter (utc+0) +true,0,0,q,2025.07.01D23:00:00.000000000=eodtime.getroll[2025.07.01D12:00:00.000000000],1,1,getroll correct for london summer (utc+1 - midnight local is 23:00 utc) + +comment,,,,,,,getroll - dst transition overnight (europe/london clocks go back 2025.10.26 01:00 utc) +run,0,0,q,eodtime.init[`rolltimezone`datatimezone`rolltimeoffset`log!(`$"Europe/London";`$"GMT";0D05:00:00.000;logdep)],1,1,reinit with 5am london roll time +true,0,0,q,2025.10.26D05:00:00.000000000=eodtime.getroll[2025.10.25D12:00:00.000000000],1,1,rollover uses tomorrows post-transition offset (gmt) not todays bst offset + +comment,,,,,,,non-gmt datatimezone (Europe/London - DST aware) +run,0,0,q,eodtime.init[`rolltimezone`datatimezone`rolltimeoffset`log!(`$"GMT";`$"Europe/London";0D00:00:00.000;logdep)],1,1,reinit with London datatimezone +true,0,0,q,-16h=type eodtime.getdailyadjustment[],1,1,Europe/London datatimezone returns timespan type +true,0,0,q,0D<=eodtime.getdailyadjustment[],1,1,Europe/London datatimezone returns non-negative offset + +comment,,,,,,,init - empty config (just log dep) applies all defaults +run,0,0,q,eodtime.init[enlist[`log]!enlist logdep],1,1,init with just log dep applies all defaults +true,0,0,q,0D=eodtime.getdailyadj[],1,1,default datatimezone (GMT) produces zero daily adjustment +true,0,0,q,0D=eodtime.getdailyadjustment[],1,1,default datatimezone (GMT) produces zero computed adjustment +true,0,0,q,2025.01.02D00:00:00.000000000=eodtime.getroll[2025.01.01D12:00:00.000000000],1,1,default rolltimeoffset (0D) gives midnight roll + +comment,,,,,,,init - non-dict configs throws +fail,0,0,q,eodtime.init[(::)],1,1,errors when configs is not a dictionary + +comment,,,,,,,error cases +fail,0,0,q,eodtime.getroll[`notvalid],1,1,getroll throws on non-timestamp input + +comment,,,,,,,init - dep validation +fail,0,0,q,eodtime.init[()!()],1,1,errors when log dependency is missing +fail,0,0,q,eodtime.init[enlist[`log]!enlist 42],1,1,errors when log value is not a dictionary +run,0,0,q,eodtime.init[enlist[`log]!enlist `info`warn!(logdep`info;logdep`warn)],1,1,init succeeds with info and warn but no error key - only info is required +run,0,0,q,.test.err:@[{eodtime.init[()!()]};(::);{x}],1,1,capture error string from init with missing log dep +true,0,0,q,.test.err like "di.eodtime:*",1,1,error is prefixed di.eodtime: + +comment,,,,,,,init - accepts any dict already shaped to the binary {[c;m]} contract (e.g. di.log's logdict) +run,0,0,q,sixlvllog:`trace`debug`info`warn`error`fatal!({[c;m]};{[c;m]};{[c;m]};{[c;m]};{[c;m]};{[c;m]}),1,1,hand-built six-level dict matching di.log's logdict shape +run,0,0,q,eodtime.init[`log`rolltimezone!(sixlvllog;`$"GMT")],1,1,superset log dict accepted as-is - only info/warn/error are used + +comment,,,,,,,log call verification - capturing logger sees init message with binary {[c;m]} args +run,0,0,q,caplog:`info`warn`error!({[c;m] `.test.cap upsert(`info;m)};{[c;m] `.test.cap upsert(`warn;m)};{[c;m] `.test.cap upsert(`error;m)}),1,1,capturing log mock - binary {[c;m]} +run,0,0,q,.test.cap:([] fn:`symbol$();msg:()),1,1,initialise log capture table +run,0,0,q,eodtime.init[enlist[`log]!enlist caplog],1,1,init with capturing logger +true,0,0,q,1=count select from .test.cap where fn=`info,1,1,one info entry captured on init +true,0,0,q,any (.test.cap`msg) like "initialised*",1,1,log message confirms initialisation diff --git a/di/log/init.q b/di/log/init.q new file mode 100644 index 00000000..cd68a4e4 --- /dev/null +++ b/di/log/init.q @@ -0,0 +1,4 @@ +// structured logger - default log dependency for di.* modules +\l ::log.q + +export:([createlog;logdict;trace;debug;info;warn;error;fatal]) diff --git a/di/log/log.md b/di/log/log.md new file mode 100644 index 00000000..ae955c88 --- /dev/null +++ b/di/log/log.md @@ -0,0 +1,193 @@ +# Log + +`log.q` is the default logging implementation for `di.*` modules. It writes formatted lines to +stdout and satisfies the log dependency contract expected by modules such as `di.email`. It also +provides `createlog`, a factory for rich structured logger instances with level filtering, +multiple output sinks, and configurable format templates. + +## Usage + +```q +logger:use`di.log +logger.trace[`mymodule;"entering function"] +logger.debug[`mymodule;"value is 42"] +logger.info[`mymodule;"starting up"] +logger.warn[`mymodule;"disk usage above 80%"] +logger.error[`mymodule;"connection failed"] +logger.fatal[`mymodule;"unrecoverable error, shutting down"] +``` + +Output format: + +``` +2026-04-09T12:00:00.000000000 [TRACE] [mymodule] entering function +2026-04-09T12:00:00.001000000 [DEBUG] [mymodule] value is 42 +2026-04-09T12:00:00.002000000 [INFO] [mymodule] starting up +2026-04-09T12:00:00.003000000 [WARN] [mymodule] disk usage above 80% +2026-04-09T12:00:00.004000000 [ERROR] [mymodule] connection failed +2026-04-09T12:00:00.005000000 [FATAL] [mymodule] unrecoverable error, shutting down +``` + +## Injecting into other modules + +All `di.*` modules that accept a log dependency expect a dictionary with keys `` `info`warn`error ``, each a function with signature `{[ctx;msg]}`. + +```q +logger:use`di.log +logdep:`info`warn`error!(logger.info;logger.warn;logger.error) + +email:use`di.email +email.init[enlist[`log]!enlist logdep] +``` + +You can extend the injected dictionary with `trace`, `debug`, and `fatal` for modules that support them: + +```q +logdep:`trace`debug`info`warn`error`fatal!(logger.trace;logger.debug;logger.info;logger.warn;logger.error;logger.fatal) +``` + +`di.log` has no dependencies of its own, so there's no `init` to call on it first — use +`logdict` to skip building the dict by hand. It's the dependency already wrapped as +`` `log!enlist logdict ``, ready to pass straight into any `di.*` module's `init`: + +```q +mylog:use`di.log +email:use`di.email +email.init[mylog.logdict] +``` + +## createlog + +`createlog` is a factory that returns an independent logger instance with level filtering, multiple output sinks, and configurable format templates. Each call to `createlog` produces a separate instance with its own state. + +Instance-level functions (`trace`..`fatal`) take the same `{[ctx;msg]}` signature as the plain top-level functions, so an instance satisfies the log dependency contract directly. None of the built-in format templates have a slot for `ctx`, so it's folded into the message text as `[ctx] msg`. + +```q +logger:use`di.log +mylog:logger.createlog[] + +mylog.setlvl `warn / suppress trace, debug, info +mylog.info[`mymodule;"this is suppressed"] / returns () silently +mylog.warn[`mymodule;"this appears"] + +mylog.setfmt `syslog / switch to syslog format +mylog.addfmt[`compact;"$l $m"] / add a custom format +mylog.setfmt `compact + +mylog.add[2i;`error`fatal] / add stderr sink for error and fatal +mylog.remove[1i;`trace] / remove stdout from trace level +``` + +### Sinks + +A sink is a handle (integer file descriptor or function) passed to `add`. If a function is provided it is called with the formatted line string. Built-in handles follow standard q conventions: `1i` is stdout, `2i` is stderr. Since a sink can be any open handle, `hopen` a real file to log to disk: + +```q +fh:hopen`:/path/to/app.log +mylog.add[fh;lvls] / route every level to the file +mylog.remove[1i;lvls] / optionally drop the default stdout sink +mylog.info[`mymodule;"this now goes to app.log"] +``` + +Or use a plain function sink to capture output in memory: + +```q +buf:(); +capture:{[msg] buf,:enlist msg}; / function sink - captures output +mylog.add[capture;`info`warn`error] +mylog.info[`mymodule;"captured"] +buf / ("2026-...captured\n") +mylog.remove[capture;`info] +``` + +### Format templates + +Three built-in formats are available: + +| Name | Template | Example output | +|---|---|---| +| `basic` (default) | `$p $l PID[$i] HOST[$h] $m` | `2026-04-09T12:00:00.000000000 INFO PID[1234] HOST[myhost] message` | +| `syslog` | `<$s> $m` | `<6> message` | +| `raw` | `$m` | `message` | + +Template variables: `$p` timestamp, `$l` level, `$i` PID, `$h` hostname, `$m` message, `$s` syslog severity number. + +## API + +### `trace` +Parameters: `[ctx; msg]` + +Write a trace-level message to stdout. + +- `ctx` — symbol context tag (e.g. `` `mymodule ``) +- `msg` — string message + +### `debug` +Parameters: `[ctx; msg]` + +Write a debug-level message to stdout. + +### `info` +Parameters: `[ctx; msg]` + +Write an info-level message to stdout. + +### `warn` +Parameters: `[ctx; msg]` + +Write a warning-level message to stdout. + +### `error` +Parameters: `[ctx; msg]` + +Write an error-level message to stdout. + +### `fatal` +Parameters: `[ctx; msg]` + +Write a fatal-level message to stdout. + +### `logdict` +A dictionary, not a function. + +`` `log!enlist(`trace`debug`info`warn`error`fatal!(...))`` — the log dependency dict pre-wrapped +exactly as any `di.*` module's `init` expects `deps`, so it can be passed straight through +without building it by hand. It includes all six levels, a superset of the `info`/`warn`/`error` +contract minimum, since the underlying `createlog[]` instance computes them anyway — a module +that supports the optional extended levels gets them for free. Backed by its own independent +`createlog[]` instance, separate from the plain `trace`..`fatal` functions above. + +### `createlog` +Parameters: none + +Returns an independent logger instance as a dictionary of functions. Each call returns a new instance with isolated state. + +Returned keys: `` `trace`debug`info`warn`error`fatal`add`remove`setfmt`getfmt`addfmt`setlvl`getlvl `` + +| Function | Parameters | Description | +|---|---|---| +| `trace`..`fatal` | `[ctx;msg]` | Write a message at the given level (filtered by active level) | +| `setlvl` | `[lvl]` | Set minimum level; one of `` `trace`debug`info`warn`error`fatal `` | +| `getlvl` | `[_]` | Return current minimum level | +| `setfmt` | `[name]` | Switch to a named format template | +| `getfmt` | `[_]` | Return current format name | +| `addfmt` | `[name;template]` | Register a new named format template | +| `add` | `[handle;lvls]` | Add a sink for one or more levels; returns the handle | +| `remove` | `[handle;lvl]` | Remove a sink from a level. `handle` may be a bare handle (removes every sink registered with it) or a `(handle;fn)` pair matching what was passed to `add` (removes only that exact sink) | + +## Log dependency contract + +The log dependency contract used across `di.*` modules requires a dictionary: + +```q +`info`warn`error!({[ctx;msg] ...};{[ctx;msg] ...};{[ctx;msg] ...}) +``` + +`di.log` satisfies this contract. You can also supply any custom implementation with the same signatures. + +## Testing + +```q +q)k4unit:use`di.k4unit +q)k4unit.moduletest`di.log +``` diff --git a/di/log/log.q b/di/log/log.q new file mode 100644 index 00000000..c923f7c3 --- /dev/null +++ b/di/log/log.q @@ -0,0 +1,193 @@ +/ structured logger for di.* modules +/ provides info, warn, error, trace, debug, fatal with signature {[ctx;msg]} +/ ctx is a symbol context tag, msg is a string +/ also provides createlog, a factory for rich structured logger instances + +/ os-aware newline +nl:$[.z.o in `w32`w64;"\r\n";"\n"]; + +/ log levels in priority order +lvls:`trace`debug`info`warn`error`fatal; + +/ syslog severity per level (rfc5424) +sysloglvl:lvls!7 7 6 4 3 2i; + +/ built-in format templates for createlog instances +fmts:`basic`syslog`raw!("$p $l PID[$i] HOST[$h] $m";"<$s> $m";"$m"); + +/ format pattern handlers for createlog instances; each takes {[level;msg]} +/ level is an uppercase string (e.g. "INFO"), msg is the formatted message string +pattern:"plihms~"!( + {[x;y] string .z.p}; + {[x;y] x}; + {[x;y] string .z.i}; + {[x;y] string .z.h}; + {[x;y] y}; + {[x;y] string sysloglvl`$lower x}; + {[x;y] "$"}); + +/ split a format template on delimiter, returning (textparts; substitutionfunctions) +/ escaped delimiter (e.g. $$) is replaced by $~ which resolves to literal $ +fmtprep:{[del;rep;fmt] + fmt:ssr[fmt;del,del;del,"~"]; + parts:del vs fmt; + fns:rep@first each 1_parts; + (enlist[first parts],1_/:1_parts;fns) + }; + +/ apply a prepared format returning the assembled line string +/ level is an uppercase string, msg is the formatted message string +fmtapply:{[prep;level;msg] + textparts:prep 0; + fns:prep 1; + vals:fns .\:(level;msg); + raze first[textparts],vals,'1_textparts + }; + +/ apply printf-style variable substitution to a message +/ msg is a plain string or (fmtstring;arg1;arg2;...) +/ %s converts to string, %r uses .Q.s1, %% becomes a literal percent +fmtmsg:{[msg] + $[10h=abs type msg; + msg; + [fmt:first msg; + args:1_msg; + if[not 10h=abs type fmt;'"format string must be a string"]; + fmt:ssr[fmt;"%%";"\000"]; + parts:"%" vs fmt; + nspecs:count[parts]-1; + if[nspecs<>count args;'`$"expected ",string[nspecs]," argument(s) for format string, got ",string count args]; + subs:"sr"!({$[10h=abs type x;x;-11h=type x;string x;'`type]};.Q.s1); + result:first[parts],raze {[subs;args;parts;i] + part:parts[1+i]; + code:first part; + rest:1_part; + if[not code in key subs;'`$"unsupported format char: ",enlist code]; + (subs[code] args[i]),rest + }[subs;args;parts;] each til count[parts]-1; + ssr[result;"\000";"%"] + ] + ]}; + +/ format and write a log line to stdout using the dependency-contract format +logline:{[level;ctx;msg] + -1 (string .z.p)," [",level,"] [",string[ctx],"] ",msg; + }; + +/ instance counter and per-instance state; always accessed via .z.m so nested closures (e.g. +/ createlog's returned methods) reliably resolve to this module's namespace. .z.M does not +/ resolve correctly inside nested closures here and must not be used for this state. +i:0; +inst:()!(); + +/ factory helper: returns a {[ctx;msg]} log function for the given level and instance +/ each entry in sink is a (handle;sender) pair; sender is called with the formatted text +makelevel:{[id;gv;lvl] + {[id;gv;lvl;ctx;msg] + if[(lvls?lvl) version strings +true,0,0,q,expdeps~(tml.parsetoml fdeps)`dependencies,1,1,a deps.toml [dependencies] table parses to the symbol!version-string dict di.depcheck readdeps expects +true,0,0,q,11h=type key (tml.parsetoml fdeps)`dependencies,1,1,dependency names are symbols (quoted dotted keys unquoted) +true,0,0,q,all 10h=type each value (tml.parsetoml fdeps)`dependencies,1,1,version values stay strings (policy-free) for meetsmin comparison +comment,,,,,,,getapimeta +true,0,0,q,"(asc (key tml) except `getapimeta)~asc exec name from tml.getapimeta[]",1,1,getapimeta documents exactly the callable exports (plumbing omitted) +true,0,0,q,not `getapimeta in exec name from tml.getapimeta[],1,1,getapimeta does not register itself (plumbing not in the api) +true,0,0,q,`name`public`descrip`params`return~cols tml.getapimeta[],1,1,getapimeta rows carry the registry columns +after,0,0,q,teardowntoml[],1,1,remove the temp fixture dir diff --git a/di/toml/test.q b/di/toml/test.q new file mode 100644 index 00000000..2fcf18b3 --- /dev/null +++ b/di/toml/test.q @@ -0,0 +1,30 @@ +/ di.toml test fixtures (loaded by test.csv). di.toml is pure - no mocks needed; the TOML text is +/ defined here as q string literals (far cleaner than CSV-escaping all the quotes inline) plus a +/ small temp .toml file fixture for parsefile. + +fflat:"a = 1\nb = 2.5\nc = true\nd = false\ns = \"hello\""; +fqkey:"\"my key\" = 42"; +fcomments:"# whole line\na = 1 # trailing\ns = \"a # b\" # real comment"; +fsection:"top = 1\n[sec]\nk = 10\nname = \"rdb\""; +farrays:"nums = [1, 2, 3]\nstrs = [\"xx\", \"yy\"]\nempty = []"; +fescapes:"s = \"a\\nb\\tc\\\"d\""; +fblank:"\n \n# only a comment\n"; +fqdotkey:"\"a.b\" = 1"; / a QUOTED key with a dot - a literal key, allowed +expesc:"a\nb\tc\"d"; / expected unescaped value of fescapes' s +fbs:"p = \"a\\\\nb\""; / TOML p = "a\\nb" (escaped backslash then n) +expbs:"a\\nb"; / expected: a, backslash, n, b (4 chars) - NOT a newline +femptystr:"k = \"\""; / TOML k = "" (a legitimate empty string) +finvesc:"s = \"a\\xb\""; / TOML s = "a\xb" (\x is not a valid escape) +fescq:"d = \"a\\\"b\" # c"; / TOML d = "a\"b" # c (escaped quote in value + comment) +expescq:"a\"b"; / expected value of d: a, ", b (comment stripped) +fescarr:"a = [\"x\\\"y\", \"zz\"]"; / TOML a = ["x\"y", "zz"] (escaped quote in an array elem) +expescarr:("x\"y";"zz"); / expected: two elements x"y and zz (comma not mis-split) +fqsec:"[\"a.b\"]\nx = 1"; / TOML ["a.b"] - a QUOTED section name (dot is literal) +ftab:"s = \"x\ty\""; / TOML s = "xy" (a literal tab inside the string) +exptab:"x\ty"; / expected: x, tab, y - the tab preserved, not spaced +fdeps:"# peer deps\n[dependencies]\n\"di.servers\" = \"0.3.0\" # injected\n\"di.dbwrite\" = \"0.1.0\""; / a di.depcheck deps.toml +expdeps:`di.servers`di.dbwrite!("0.3.0";"0.1.0"); / di.depcheck's readdeps expects [dependencies] -> symbol!version-string + +TDIR:"/tmp/ditomltest"; +setuptoml:{[] system "mkdir -p ",TDIR; (`$":",TDIR,"/x.toml") 0: ("dir = \":appdb\"";"rows = 100");}; +teardowntoml:{[] system "rm -rf ",TDIR;}; diff --git a/di/toml/toml.md b/di/toml/toml.md new file mode 100644 index 00000000..176e0e51 --- /dev/null +++ b/di/toml/toml.md @@ -0,0 +1,104 @@ +# di.toml + +A small, scoped-down TOML parser for the modular TorQ world. It reads settings `.toml` files (and +in-memory TOML text) into q dicts. Its reason for existing is to back di.config's `.toml` +resolution tier — di.config lazily loads di.toml only when a `.toml` file is actually parsed. + +**Pure module** — no `init`, no logger, no injected dependencies. di.config invokes it *during +config resolution*, before any logger exists, so parse errors are **signalled** with a clear +`'di.toml: …'` message rather than logged. + +**Fail-loud** — because this is a *shared* utility (any module that parses a file may use it), +di.toml never silently returns nulls or garbage for input it can't correctly handle: a missing +file, an unparseable value (bare word / datetime), a dotted key, or a malformed/array-of-tables +section header all **signal**. A caller that treats a *missing file* as acceptable (a config +cascade probing optional tiers) must **guard existence itself** before calling — the way di.config +does (`if[0=count key hsym`$path; :()!()]`) — di.toml does not paper over it. + +## Scope (v1) + +Supported — what real settings files need: + +- `key = value` pairs; keys bare or double-quoted (`"my key" = 1`) +- `#` comments, whole-line or trailing — **quote-aware**, so a `#` inside a `"…"` string is not a + comment +- one level of `[section]` nesting — a section becomes a **sub-dict** keyed by the section name +- scalar values: double-quoted strings (`\"` `\\` `\n` `\t` escapes), integers, floats, + `true`/`false`, and flat arrays of any of those + +Deliberately **not** supported (out of scope for settings): nested inline tables (`{a=1,b=2}`), +dotted keys (`a.b.c`), single-quoted literal strings, multi-line strings, datetimes, and +array-of-tables (`[[x]]`). + +## Type policy + +- **Strings are policy-free.** TOML has no symbol type, so every TOML string parses to a q **char + string** (`10h`), never a symbol. Callers that want a symbol coerce at the point of use (`` `$ `` + is a no-op on an already-a-symbol value, so the same consumer code works for `.q` settings too). + This is the contract di.config and its consumers rely on. +- **Integers parse to `long`** (matching what `value` gives a `.q` settings line like `rows:100`), + floats to `float`, `true`/`false` to `boolean`. + +## Exported functions + +| Function | Signature | Description | +|---|---|---| +| `parsetoml` | `parsetoml[text]` | Parse a TOML string into a dict (one level of `[section]` nesting). Blank and comment-only lines contribute nothing. | +| `parsefile` | `parsefile[path]` | Read and parse a `.toml` file into a dict. A **missing file signals** — a caller that treats missing as acceptable (a config cascade) guards existence itself first (as di.config does). | +| `getapimeta` | `getapimeta[]` | This module's api metadata — one row per **callable** function (`parsetoml`, `parsefile`); `getapimeta` itself is plumbing and is deliberately not listed. For `di.torq` to register with `di.api`. | + +Export is conservative — the internal helpers (`trimstr`, `isquoted`, `instrmask`, `firstunquoted`, +`stripcomment`, `splitassign`, `unquotekey`, `parsekey`, `unescape`, `parsescalar`, `splitcommas`, +`parsevalue`, `sectname`, `addkv`, `addline`) are not exported. No `version` export / `VERSION` file yet — deferred +to the di.depcheck rollout, as in di.config/di.servers. + +## di.config integration contract + +di.config's `parsefile` delegates a `.toml` path via `` (use`di.toml)[`parsefile] path ``, expecting +a **flat dict** of `setting → value` back (section values are sub-dicts). This module satisfies that +directly. Once di.toml is on `QPATH` alongside di.config, the `.toml` half of di.config's cascade +(and `.toml > .q` within a tier) works, and di.config's guard-rail error (`the di.toml module was +not found on QPATH`) no longer fires. + +## Out-of-scope constructs are rejected, not mis-parsed + +Anything the scoped grammar can't correctly represent **signals** rather than silently producing +wrong data: + +- a bare/unquoted non-numeric value (a word, a datetime) +- an **invalid bare key or section name** — one with a space, `:`, or any char outside `A-Za-z0-9_-` + (this is what makes most *non-TOML* lines that happen to contain `=` — a shell `export FOO=5`, a q + `x:a=5` — fail loud instead of parsing to a bogus symbol key). Keys and section names are held to + the *same* rule (a shared `parsekey`) +- an **empty key or section name** (`= 5`, `[]`) +- a **dotted key or section** (`a.b`, `[a.b]` — nesting) — but a *quoted* `"a.b"` (key or section) is + a legitimate literal name and is kept, unquoted +- a **duplicate key / section** (or a key/section-name clash) — a TOML error, not silent last-wins +- an **empty (missing) value** `k =`, and an **unterminated / malformed array** (`[` without a `]`) +- an **unknown or dangling string escape** (`\x`, a trailing `\`) +- an **array-of-tables** `[[x]]`, or a malformed section header (missing `]`) +- a non-char input, or a 1-char input (a char atom) — handled/validated at the `parsetoml` entry so + it never surfaces as a cryptic `'type` + +This is what makes di.toml safe as a shared parser — a consumer feeding a richer or malformed file +gets a clear `'di.toml: …'` error, not quiet garbage. + +## Known limitations + +None that mis-parse. Everything outside the supported grammar — including TOML number forms not +covered (underscored `1_000`, hex/octal/binary `0x`/`0o`/`0b`, `inf`/`nan`) — is **rejected with a +clear signal**, never silently mis-parsed. The scope is deliberately the settings-file subset; widen +it (and add tests) if a real settings file needs more. + +## Tests + +```q +k4unit:use`di.k4unit +k4unit.moduletest`di.toml +``` + +`test.csv` (+ `test.q` fixtures) covers scalars and their types, policy-free strings, quoted keys, +quote-aware comments, one-level sections, arrays (int/string/empty), the `\n`/`\t`/`\"` escapes, +blank/comment-only input, a missing `=` signalling, `parsefile` (present + missing file), and +`getapimeta`. TOML text lives in `test.q` as q string literals (cleaner than CSV-escaping the quotes +inline). Runs from the repo root (`test.q` is loaded relative). diff --git a/di/toml/toml.q b/di/toml/toml.q new file mode 100644 index 00000000..e8a096b7 --- /dev/null +++ b/di/toml/toml.q @@ -0,0 +1,128 @@ +/ scoped-down TOML parser for the modular torq world - reads settings .toml files (and text) into a +/ q dict. supports: key=value (bare or "quoted" keys), quote-aware # comments, one level of [section] +/ nesting, and scalars (quoted strings, integer->long, float, true/false, flat arrays). not supported: +/ inline tables, dotted keys, single-quoted/multi-line strings, datetimes, array-of-tables. +/ fail-loud: anything it cannot correctly parse signals 'di.toml: ...', never silent garbage. +/ strings parse to q char strings, never symbols (TOML has none) - callers coerce with `$ at use. +/ pure: no init/logger/deps - di.config loads it during config resolution, before a logger exists. +/ parsefile signals on a MISSING file; a caller for whom that is acceptable (a cascade probing +/ optional tiers) guards existence itself first, as di.config does: if[0=count key hsym`$path;:()!()]. +/ reserved-word trap: cut/trim/parse/ss/sv/vs/ssr are builtins - hence trimstr/parsetoml/etc. + +keychars:.Q.a,.Q.A,.Q.n,"_-"; / chars allowed in a bare key; anything else must be "quoted" + +trimstr:{[s] i:where not s in " \t"; $[count i;(first i)_(1+last i)#s;""]}; / strip leading/trailing space+tab; keep internal +isquoted:{[s] (1 list of scalars (empty -> ()); else a single scalar. empty and unterminated + / values are rejected. + v:trimstr v; + if[0=count v;'"di.toml: missing value"]; + if["["=first v; + if[not "]"=last v;'"di.toml: malformed array (must be [ ... ]): ",v]; + inner:trimstr 1_-1_v; + :$[0=count inner;();parsescalar each splitcommas inner]]; + parsescalar v}; + +addkv:{[d;k;v] + / add k->v, signalling on a duplicate (a repeated key/section, or a key/section clash - TOML errors). + / catenation (not ,:) keeps the value list general so mixed value types never hit a type-widen error. + if[k in key d;'"di.toml: duplicate key: ",string k]; + d,(enlist k)!enlist v}; + +addline:{[acc;line] + / fold step; acc is (top;cursect;sect). a [section] flushes the open section into top and opens a + / fresh one; a key=value goes into the current section, or top before any section. + top:acc 0; cursect:acc 1; sect:acc 2; + $[(first line)="["; + [hdr:trimstr line; + if[not "]"=last hdr;'"di.toml: malformed section header (missing ]): ",line]; + if["[["~2 sublist hdr;'"di.toml: array-of-tables [[...]] is not supported: ",line]; + nm:sectname hdr; + if[not null cursect;top:addkv[top;cursect;sect]]; + (top;nm;()!())]; + [kv:splitassign line; + k:parsekey kv 0; + v:parsevalue kv 1; + $[null cursect;(addkv[top;k;v];cursect;sect);(top;cursect;addkv[sect;k;v])]]]}; + +parsetoml:{[text] + / TOML text -> dict (one level of [section] nesting). accepts a string (or the char atom q makes of + / a 1-char literal); other types signal - else a 1-char input is an atom and ssr/vs throw 'type. + if[not 10h=abs type text;'"di.toml: parsetoml expects a char string; got type ",string type text]; + lines:trimstr each stripcomment each "\n" vs $[10h=type text;text;enlist text]; + acc:addline/[(()!();`;()!());lines where 0value (sections nest)"); + (`parsefile; 1b; "read and parse a .toml file into a dict (signals if missing)"; "[string: file path]"; "dict: setting->value")); + }; From 40f803506794649e451dbbafb550ea7f9d41f702 Mon Sep 17 00:00:00 2001 From: ascottDI Date: Wed, 5 Aug 2026 15:32:31 +0100 Subject: [PATCH 5/9] refactor to fit merged modules rather than mocked. Handlers module still mocked awaiting merge --- di/servers/servers.md | 28 +++++++++++++++++----------- di/servers/servers.q | 15 ++++++++++++--- di/servers/test.csv | 17 ++++++++++------- di/servers/test.q | 17 +++++++++-------- 4 files changed, 48 insertions(+), 29 deletions(-) diff --git a/di/servers/servers.md b/di/servers/servers.md index c4d893ce..df8c5203 100644 --- a/di/servers/servers.md +++ b/di/servers/servers.md @@ -23,8 +23,8 @@ duplicate `di.timer.addjob` id would throw). `init` does **not** open connection | key | kind | meaning | |---|---|---| -| `log` | injectable | binary `` `info`warn`error `` `{[c;m]}` logger dict | -| `timer` | injectable | di.timer contract; `addjob` = the 6-arg `custom` form `{[id;func;params;period;mode;opts]}` | +| `log` | injectable | binary `` `info`warn`error `` `{[c;m]}` logger dict — di.log's `logdict``log` satisfies this directly (it carries all six levels; the extra `trace`/`debug`/`fatal` are ignored) | +| `timer` | injectable | the di.timer export dict; di.servers calls `` timer[`addjob][`custom] `` — the 6-arg `{[id;func;params;period;mode;opts]}` variant (`addjob` is a variant dict of `custom`/`default`/`simple`) | | `handlers` | injectable | di.handlers contract; `register[event;phase;nm;pri;func]` | | `proctype`/`procname` | config | this process's own identity (required); used to exclude self from `process.csv` | | `connections` | config | proctypes this process should dial (symbols, or strings from a `.toml` cascade — normalised). Optional; default = none | @@ -100,17 +100,23 @@ priority-ordered fan-out. ## Open items / not yet done -- **Live-peer integration tests are in place** (`test.q` + `test.csv`, 33 checks). They spawn a - genuinely separate `q` peer (a self-connect returns pseudo-handle `0`, not a real socket) and +- **Live-peer integration tests are in place** (`test.q` + `test.csv`, 37 checks), and now wire the + **real merged `di.timer` and `di.log`** — only `di.handlers` (not yet merged) is mocked. They spawn + a genuinely separate `q` peer (a self-connect returns pseudo-handle `0`, not a real socket) and cover: `startup` connecting to a live peer and logging a failed dial while excluding self, `gethandlebytype` returning a live remote handle (`2=h"1+1"`), the retry cycle recovering an - ungraceful kill (`cleanup`+reopen), and `waitfortype` connected-vs-timeout — plus the mockable - surface (init validation, dep-wiring, idempotency, input validation, `getapimeta`). Because - `retry`/`cleanup` are internal, the retry cycle is driven by invoking the callback the **mock - timer captured** at `addjob` (the actually-wired path), not a direct export. -- **Provider modules not in kdbx-modules yet.** `di.log` (feature-logging branch) and - `di.handlers` aren't here yet, so the injected contracts are mocked in tests. The handlers - mock uses the real `register[event;phase;nm;pri;func]` shape from `handlers.q`. + ungraceful kill (`cleanup`+reopen), and `waitfortype` connected-vs-timeout — plus init validation, + dep-wiring, idempotency, input validation, and `getapimeta`. `init` schedules `serversretry` in the + real `di.timer` (asserted via `` timer.getalljobs[] ``); because `retry`/`cleanup` are internal, the + retry cycle is driven by invoking the exact func di.servers handed the timer (`` firejob `` reads it + back from `` getalljobs[] `` — the actually-wired path), not a direct export. Idempotent re-init is + a genuine test here: the real timer's `` addjob[`custom] `` throws on a duplicate id, so a + non-idempotent `init` would fail outright. A final check re-inits against di.log's real `logdict` to + prove the injected-log contract holds end-to-end. +- **`di.handlers` not in kdbx-modules yet**, so only that injected contract is mocked. The handlers + mock uses the real `register[event;phase;nm;pri;func]` shape from `handlers.q`; the `.z.pc` observer + path is therefore exercised only via the explicit `retry`→`cleanup` sweep, not a live auto-fired + `.z.pc` (which real di.handlers would install). - **`config`processcsv` and the assembled `connections` list** depend on di.torq's config wiring — coordinate when di.torq's servers dep is built. - Scoped-out (v1): discovery service, password/access-list files, non-TorQ tracking, and the diff --git a/di/servers/servers.q b/di/servers/servers.q index 150f830c..76051bdb 100644 --- a/di/servers/servers.q +++ b/di/servers/servers.q @@ -55,6 +55,12 @@ init:{[deps] '"di.servers: log dict must have `info`warn`error keys; got: ",(", " sv string key deps`log)]; if[99h<>type deps`timer; '"di.servers: timer value must be a dict (see di.timer)"]; + if[not `addjob in key deps`timer; + '"di.servers: timer dict must expose `addjob (see di.timer)"]; + if[99h<>type deps[`timer]`addjob; + '"di.servers: timer`addjob must be a variant dict (see di.timer addjob.custom/default/simple)"]; + if[not `custom in key deps[`timer]`addjob; + '"di.servers: timer`addjob must expose the `custom variant [id;func;params;period;mode;opts]"]; if[99h<>type deps`handlers; '"di.servers: handlers value must be a dict (see di.handlers)"]; if[not all `proctype`procname in key deps; @@ -76,9 +82,12 @@ init:{[deps] / (param `wh`, not `w`, so it does not shadow the SERVERS column w.) pcfunc:{[wh] .z.m.SERVERS:update endp:.z.p,w:0Ni from .z.m.SERVERS where w=wh; }; (.z.m.handlers[`register])[`.z.pc;`;`servers;0j;pcfunc]; - / di.timer mode-1h period is in SECONDS, so 10 = a 10-second retry (a bare 10000 here would be - / ~2.8h - the latent typo that made dead-handle recovery effectively never fire in early POCs). - (.z.m.timer[`addjob])[`serversretry;retry;();10;1;()!()]; + / di.timer's addjob is a VARIANT DICT; take `custom - the fully-configurable 6-arg form + / [id;func;params;period;mode;opts]. mode-1h period is in SECONDS, so 10 = a 10s retry (a bare + / 10000 would be ~2.8h - the latent typo that made dead-handle recovery never fire in early POCs). + / retry is passed BY VALUE (a lambda, not a symbol) so di.timer stores and runs it directly; its + / compile-time .z.m rewrite means it still updates di.servers' SERVERS when the timer fires it. + (.z.m.timer[`addjob][`custom])[`serversretry;retry;();10;1;()!()]; .z.m.registered:1b; ]; .z.m.loginfo[`init;"di.servers initialised"]; diff --git a/di/servers/test.csv b/di/servers/test.csv index 652476e8..3502d6ba 100644 --- a/di/servers/test.csv +++ b/di/servers/test.csv @@ -7,16 +7,16 @@ before,0,0,q,spawnpeer[],1,1,launch a genuinely separate q peer to act as otherp comment,,,,,,,init - dependency + config validation (plain signal; logger not wired yet) fail,0,0,q,svc.init[()!()],1,1,init with no deps errors fail,0,0,q,svc.init[enlist[`log]!enlist mocklog],1,1,init missing timer/handlers errors -fail,0,0,q,"svc.init[`log`timer`handlers`proctype`procname!(42;mocktimer;mockhandlers;`selfproc;`selfinst)]",1,1,init rejects a non-dict log value -fail,0,0,q,"svc.init[`log`timer`handlers`proctype`procname!((enlist`info)!enlist{[c;m]};mocktimer;mockhandlers;`selfproc;`selfinst)]",1,1,init rejects a log dict missing warn/error -fail,0,0,q,"svc.init[`log`timer`handlers!(mocklog;mocktimer;mockhandlers)]",1,1,init rejects deps without proctype/procname identity +fail,0,0,q,"svc.init[`log`timer`handlers`proctype`procname!(42;rtmr;mockhandlers;`selfproc;`selfinst)]",1,1,init rejects a non-dict log value +fail,0,0,q,"svc.init[`log`timer`handlers`proctype`procname!((enlist`info)!enlist{[c;m]};rtmr;mockhandlers;`selfproc;`selfinst)]",1,1,init rejects a log dict missing warn/error +fail,0,0,q,"svc.init[`log`timer`handlers!(mocklog;rtmr;mockhandlers)]",1,1,init rejects deps without proctype/procname identity comment,,,,,,,init - wiring and idempotency run,0,0,q,svc.init[svrdeps[`otherproc`deadproc]],1,1,init with mock deps + config (dial otherproc and deadproc) true,0,0,q,"1=count select from handlercalls where event=`.z.pc,name=`servers",1,1,init registered its .z.pc observer via the handlers dep -true,0,0,q,1=count select from timercalls where id=`serversretry,1,1,init scheduled the retry job via the timer dep -true,0,0,q,10=first exec period from timercalls where id=`serversretry,1,1,retry period is 10s (di.timer mode-1h period in seconds - guards the old 10000 typo) +true,0,0,q,1=count select from rtmr.getalljobs[] where id=`serversretry,1,1,init scheduled the retry job via the timer dep +true,0,0,q,10=first exec period from rtmr.getalljobs[] where id=`serversretry,1,1,retry period is 10s (di.timer mode-1h period in seconds - guards the old 10000 typo) run,0,0,q,svc.init[svrdeps[`otherproc`deadproc]],1,1,re-init must be idempotent -true,0,0,q,1=count select from timercalls where id=`serversretry,1,1,idempotent re-init did NOT add a second retry job +true,0,0,q,1=count select from rtmr.getalljobs[] where id=`serversretry,1,1,idempotent re-init did NOT add a second retry job true,0,0,q,"1=count select from handlercalls where event=`.z.pc,name=`servers",1,1,idempotent re-init did NOT re-register the pc handler comment,,,,,,,startup - connect to a live peer and a dead one (self excluded) run,0,0,q,svc.startup[],1,1,open connections to otherproc (live) and deadproc (nothing listening) @@ -42,8 +42,11 @@ comment,,,,,,,input validation + getapimeta fail,0,0,q,"svc.getservers[""nosuch""]",1,1,getservers rejects a non-symbol proctype fail,0,0,q,"svc.gethandlebytype[`hdb;""any""]",1,1,gethandlebytype rejects a non-symbol selection fail,0,0,q,"svc.waitfortype[`hdb;""x"";200]",1,1,waitfortype rejects a non-integer timeout -fail,0,0,q,"svc.init[`log`timer`handlers`proctype`procname!(mocklog;mocktimer;mockhandlers;""rdb"";`p)]",1,1,init rejects a non-symbol proctype +fail,0,0,q,"svc.init[`log`timer`handlers`proctype`procname!(mocklog;rtmr;mockhandlers;""rdb"";`p)]",1,1,init rejects a non-symbol proctype true,0,0,q,(asc (key svc) except `init`getapimeta)~asc exec name from svc.getapimeta[],1,1,getapimeta documents exactly the callable exports (plumbing omitted) true,0,0,q,not any `init`getapimeta in exec name from svc.getapimeta[],1,1,getapimeta omits plumbing (init/getapimeta not registered) true,0,0,q,`name`public`descrip`params`return~cols svc.getapimeta[],1,1,getapimeta rows carry the registry columns +comment,,,,,,,real di.log integration (the merged logger itself - not the recording mock) +true,0,0,q,all `info`warn`error in key (use`di.log)[`logdict]`log,1,1,di.log logdict provides the info/warn/error keys the injected-log contract requires +run,0,0,q,"svc.init[`log`timer`handlers`proctype`procname!((use`di.log)[`logdict]`log;rtmr;mockhandlers;`selfproc;`selfinst)]",1,1,init accepts di.log's real logdict end-to-end (idempotent re-init against the merged logger) after,0,0,q,teardownfixture[],1,1,kill the peer and remove the fixture dir diff --git a/di/servers/test.q b/di/servers/test.q index 94474039..7de66186 100644 --- a/di/servers/test.q +++ b/di/servers/test.q @@ -10,12 +10,11 @@ mocklog:`info`warn`error!( {[c;m]`logrows upsert(`warn;c;m)}; {[c;m]`logrows upsert(`error;c;m)}); -/ timer mock: records (id;period) for the wiring asserts AND captures each job's func by id, so a -/ test can fire the retry cycle exactly as the real timer would (retry/cleanup are INTERNAL - not -/ exported - so they are driven only via this captured callback). -timercalls:([]id:`symbol$();period:`long$()); -timerjobs:(`symbol$())!(); -mocktimer:enlist[`addjob]!enlist {[id;func;params;period;mode;opts] timerjobs[id]:func; `timercalls upsert (id;period);}; +/ REAL di.timer - di.servers is its first consumer, so we integrate against the merged module (not a +/ mock's guess at the addjob contract) to prove the addjob[`custom] wiring end-to-end. di.servers does +/ NOT call timer.init, so no .z.ts cycle starts here: init adds the serversretry job to di.timer's jobs +/ table, and firejob invokes the stored func manually - exactly what the timer's cycle would do. +rtmr:use`di.timer; / handlers mock: records (event;name) with di.handlers' register[event;phase;nm;pri;func] shape. it / does NOT actually bind .z.pc - so the only cleanup path exercised here is the explicit retry-> @@ -27,7 +26,9 @@ mockhandlers:`register`remove`list!( {[ev]}); warnlogged:{[s] any (exec msg from logrows where lvl=`warn) like "*",s,"*"}; -firejob:{[id] timerjobs[id][]}; +/ fire a scheduled job's stored func exactly as the real di.timer's cycle would (retry/cleanup are +/ INTERNAL - not exported - so this is the only handle on them: the func di.servers gave the timer). +firejob:{[jid] (first exec func from rtmr.getalljobs[] where id=jid)[]}; / --- real peer process fixture --- FIXDIR:"/tmp/diserverstest"; @@ -67,4 +68,4 @@ setupfixture:{[] teardownfixture:{[] killpeer[]; system "rm -rf ",FIXDIR;}; / build the deps dict di.torq would assemble: injectables + this process's config slice. -svrdeps:{[conns] `log`timer`handlers`proctype`procname`connections`processcsv!(mocklog;mocktimer;mockhandlers;`selfproc;`selfinst;conns;FIXDIR,"/process.csv")}; +svrdeps:{[conns] `log`timer`handlers`proctype`procname`connections`processcsv!(mocklog;rtmr;mockhandlers;`selfproc;`selfinst;conns;FIXDIR,"/process.csv")}; From cd427284d8e7153a8408995c156b179ce47edab0 Mon Sep 17 00:00:00 2001 From: ascottDI Date: Thu, 6 Aug 2026 10:46:51 +0100 Subject: [PATCH 6/9] cleaning up orphaned code --- di/servers/servers.md | 13 +++++++------ di/servers/servers.q | 35 ++++++++++++++++++++--------------- di/servers/test.csv | 6 ++++++ di/servers/test.q | 4 ++++ 4 files changed, 37 insertions(+), 21 deletions(-) diff --git a/di/servers/servers.md b/di/servers/servers.md index df8c5203..9a793f78 100644 --- a/di/servers/servers.md +++ b/di/servers/servers.md @@ -43,7 +43,7 @@ h "1+1" | Function | Signature | Description | |---|---|---| | `init` | `init[deps]` | Wire deps + config, record identity, install the `.z.pc` handler + retry job. Idempotent. | -| `startup` | `startup[]` | Read `process.csv` (`processcsv`), drop self, connect to each row whose proctype is in `connections`. A failed connection is logged (not raised) and left as `w:0Ni` for `retry`. No-op if no connections configured. | +| `startup` | `startup[]` | Read `process.csv` (`processcsv`), drop self, connect to each row whose proctype is in `connections`. A failed connection is logged (not raised) and left as `w:0Ni` for `retry`. No-op if no connections configured. **Idempotent**: skips procs already tracked in `SERVERS`, so a repeat call (or a grown `process.csv`) adds only new rows — never a duplicate or a leaked second handle. `process.csv` must be the strict v1 4-column `host,port,proctype,procname` layout — a reordered or wider header is **rejected loudly** (the reader is positional, so it would otherwise misparse silently). | | `getservers` | `getservers[proctype]` | Live (`w` non-null) `SERVERS` rows for a proctype. | | `gethandlebytype` | `gethandlebytype[proctype;selection]` | One live handle via `` `any``/`roundrobin`/`last``; `0Ni` if none. Bumps usage stats. | | `waitfortype` | `waitfortype[proctype;timeoutms;pollms]` | Block until a live connection exists or timeout; `1b`/`0b`. Caller decides if a timeout is fatal. `startup` must have run first. | @@ -88,9 +88,9 @@ priority-ordered fan-out. - **Three-flat-var logging** — `.z.m.loginfo`/`.z.m.logwarn`/`.z.m.logerr`, matching `consistency.md`, `di.compression` and `di.config`. (The project hasn't globally frozen this vs. the single-dict form — flag before changing.) -- **`raiseerror` (log-then-signal)** for all post-init domain errors (`formathp` unknown - ipctype, `selector` unknown selection, missing `process.csv`). `init`'s own dependency - validation is the one exception (plain `'` — no logger yet). +- **`raiseerror` (log-then-signal)** for all post-init domain errors (`selector` unknown + selection, missing or malformed `process.csv`). `init`'s own dependency validation is the one + exception (plain `'` — no logger yet). - **`getapimeta`** exported; a test asserts it documents exactly the module's *callable* exports — `init`/`getapimeta` are plumbing (di.torq calls them by convention) and are deliberately omitted from the registry rows, matching di.toml and the skill convention. No @@ -119,8 +119,9 @@ priority-ordered fan-out. `.z.pc` (which real di.handlers would install). - **`config`processcsv` and the assembled `connections` list** depend on di.torq's config wiring — coordinate when di.torq's servers dep is built. -- Scoped-out (v1): discovery service, password/access-list files, non-TorQ tracking, and the - `tcps`/`unix` socket types end-to-end (only `tcp` is wired through `startup`). +- Scoped-out (v1): discovery service, password/access-list files, non-TorQ tracking, and + `tcps`/`unix` socket types. Only `tcp` is supported; `formathp` builds a `tcp` handle with no + socket-type arg — a future `SOCKETTYPE` config reintroduces that (with a test) when needed. ## Tests diff --git a/di/servers/servers.q b/di/servers/servers.q index 76051bdb..c582f0eb 100644 --- a/di/servers/servers.q +++ b/di/servers/servers.q @@ -93,15 +93,11 @@ init:{[deps] .z.m.loginfo[`init;"di.servers initialised"]; }; -formathp:{[host;port;ipctype] - / internal - build a connection-handle symbol for `tcp`/`tcps`/`unix. only `tcp is exercised by - / startup in v1; the others exist for a future SOCKETTYPE-style config. - h:string host; - p:string port; - $[ipctype=`tcp; lower `$":",h,":",p; - ipctype=`tcps;lower `$":tcps://",h,":",p; - ipctype=`unix;lower `$":unix://",p; - raiseerror[`formathp;"unknown ipctype ",string ipctype]] +formathp:{[host;port] + / internal - build the tcp connection-handle symbol from a process.csv row. v1 is tcp only; a + / future SOCKETTYPE config would reintroduce tcps/unix handling (and a type arg) when there is a + / real requirement and a test - we do not ship unexercised branches. + lower `$":",(string host),":",string port }; opencon:{[hpup] @@ -114,12 +110,17 @@ opencon:{[hpup] }; readprocesscsv:{[path] - / internal - read the static process.csv phone book (host,port,proctype,procname). the PATH is - / supplied by the caller (from config`processcsv); di.servers reads no env itself, holding - / di.config's env-free boundary - di.torq resolves the path and puts it in config. + / internal - read the static process.csv phone book. the PATH comes from config`processcsv (di.torq + / resolves it; di.servers reads no env). v1 is a STRICT 4-column host,port,proctype,procname layout: + / validate the header up front and FAIL LOUD, because ("SISS";",") is positional and would otherwise + / silently misread a reordered or wider file (e.g. a real 13-column TorQ process.csv) into garbage. fsym:`$":",path; if[0=count key fsym;raiseerror[`readprocesscsv;"process.csv not found at ",path]]; - ("SISS";enlist",") 0: fsym + lines:read0 fsym; + if[0=count lines;raiseerror[`readprocesscsv;"process.csv is empty at ",path]]; + if[not `host`port`proctype`procname~`$trim each "," vs first lines; + raiseerror[`readprocesscsv;"process.csv header must be exactly host,port,proctype,procname (v1 4-column phone book); got: ",first lines]]; + ("SISS";enlist",") 0: lines }; startup:{[] @@ -139,9 +140,13 @@ startup:{[] procs:update isme:(proctype=pt)&procname=pn from procs; procs:select from procs where not isme; procs:select from procs where proctype in conns; - if[0=count procs;.z.m.loginfo[`servers;"no process.csv rows match the configured connections"];:()]; + / idempotent: skip any proc already tracked in SERVERS. a repeat startup (or a process.csv that has + / grown since) then adds only NEW rows - never a duplicate row or a leaked second handle to a proc + / already connected. reconnecting a dropped peer is retry's job, not startup's. + procs:select from procs where not procname in exec procname from .z.m.SERVERS; + if[0=count procs;.z.m.loginfo[`servers;"no new process.csv rows to connect"];:()]; {[row] - hpup:formathp[row`host;row`port;`tcp]; + hpup:formathp[row`host;row`port]; w:opencon[hpup]; if[not null w;.z.m.loginfo[`servers;"connected to ",(string row`proctype),"/",(string row`procname)," at ",string hpup]]; / catenate+reassign, NOT `tablename insert - a symbol-based insert into `.z.m.SERVERS` misses diff --git a/di/servers/test.csv b/di/servers/test.csv index 3502d6ba..950ecfda 100644 --- a/di/servers/test.csv +++ b/di/servers/test.csv @@ -38,6 +38,12 @@ true,0,0,q,1=count svc.getservers[`otherproc],1,1,retry reconnected to otherproc comment,,,,,,,waitfortype - connected vs timeout true,0,0,q,svc.waitfortype[`otherproc;2000;200],1,1,returns 1b immediately for an already-connected proctype true,0,0,q,0b~svc.waitfortype[`deadproc;700;200],1,1,returns 0b on timeout for a proctype that never comes up +comment,,,,,,,startup is idempotent - a repeat call must not duplicate rows or leak a second handle +run,0,0,q,svc.startup[],1,1,call startup a second time with the same config +true,0,0,q,1=count svc.getservers[`otherproc],1,1,repeat startup added no duplicate otherproc row (guard skips already-tracked procs) +comment,,,,,,,readprocesscsv fails loud on a reordered/wider header instead of silently misparsing +run,0,0,q,"svc.init[`log`timer`handlers`proctype`procname`connections`processcsv!(mocklog;rtmr;mockhandlers;`selfproc;`selfinst;`rdb;writebadcsv[])]",1,1,re-init pointing at a process.csv whose header is reordered +fail,0,0,q,svc.startup[],1,1,startup signals on the bad header rather than silently misreading columns comment,,,,,,,input validation + getapimeta fail,0,0,q,"svc.getservers[""nosuch""]",1,1,getservers rejects a non-symbol proctype fail,0,0,q,"svc.gethandlebytype[`hdb;""any""]",1,1,gethandlebytype rejects a non-symbol selection diff --git a/di/servers/test.q b/di/servers/test.q index 7de66186..47890bf8 100644 --- a/di/servers/test.q +++ b/di/servers/test.q @@ -67,5 +67,9 @@ setupfixture:{[] teardownfixture:{[] killpeer[]; system "rm -rf ",FIXDIR;}; +/ write a process.csv whose header is REORDERED vs the assumed host,port,proctype,procname (the exact +/ shape that used to silently misparse) and return its path - used to prove readprocesscsv fails loud. +writebadcsv:{[] (`$":",p:FIXDIR,"/bad.csv") 0: ("port,host,proctype,procname"; "5010,localhost,rdb,rdb1"); p}; + / build the deps dict di.torq would assemble: injectables + this process's config slice. svrdeps:{[conns] `log`timer`handlers`proctype`procname`connections`processcsv!(mocklog;rtmr;mockhandlers;`selfproc;`selfinst;conns;FIXDIR,"/process.csv")}; From 395861f5c557783677d81b39023e8286583eefb3 Mon Sep 17 00:00:00 2001 From: ascottDI Date: Thu, 6 Aug 2026 15:24:26 +0100 Subject: [PATCH 7/9] adding in the version control --- di/servers/init.q | 8 +++++++- di/servers/servers.md | 17 ++++++++++++----- di/servers/test.csv | 8 ++++++-- di/servers/version.txt | 1 + 4 files changed, 26 insertions(+), 8 deletions(-) create mode 100644 di/servers/version.txt diff --git a/di/servers/init.q b/di/servers/init.q index 97ceb222..31fe018f 100644 --- a/di/servers/init.q +++ b/di/servers/init.q @@ -1,3 +1,9 @@ / connection management and handle-by-type lookup for the modular torq world. \l ::servers.q -export:([init;startup;getservers;gethandlebytype;waitfortype;getapimeta]) +/ module version: fallback default here, exported so di.depcheck can read di.servers' version to +/ satisfy other modules' declared minimums. +version:"0.1.0"; +/ version.txt in the module folder is the source of truth and takes priority over the fallback above; +/ read module-relative at load (`:::` resolves to di/servers). a missing/empty file keeps the fallback. +version:@[{trim first read0 x};`:::version.txt;{[e]version}]; +export:([init;startup;getservers;gethandlebytype;waitfortype;getapimeta;version]) diff --git a/di/servers/servers.md b/di/servers/servers.md index 9a793f78..a776a278 100644 --- a/di/servers/servers.md +++ b/di/servers/servers.md @@ -47,10 +47,11 @@ h "1+1" | `getservers` | `getservers[proctype]` | Live (`w` non-null) `SERVERS` rows for a proctype. | | `gethandlebytype` | `gethandlebytype[proctype;selection]` | One live handle via `` `any``/`roundrobin`/`last``; `0Ni` if none. Bumps usage stats. | | `waitfortype` | `waitfortype[proctype;timeoutms;pollms]` | Block until a live connection exists or timeout; `1b`/`0b`. Caller decides if a timeout is fatal. `startup` must have run first. | -| `getapimeta` | `getapimeta[]` | This module's api metadata, one row per **callable** API function (`init`/`getapimeta` plumbing omitted), for `di.torq` to register with `di.api`. | +| `getapimeta` | `getapimeta[]` | This module's api metadata, one row per **callable** API function (`init`/`getapimeta`/`version` plumbing omitted), for `di.torq` to register with `di.api`. | +| `version` | `version` | The module's semver string (`"0.1.0"`) — metadata, not a function. Sourced from `version.txt` (priority) with an `init.q` fallback; read by `di.depcheck` to satisfy other modules' declared minimum-version requirements. | Export is deliberately conservative — only functions `di.torq` or a consumer actually calls -(so `di.api` lists exactly these). The rest are **internal**: `retry` (the scheduled +(so `di.api` lists exactly these), plus the `version` metadata string. The rest are **internal**: `retry` (the scheduled `serversretry` job — passed to the timer *by value* at init, so it needs no export; it first runs `cleanup` to sweep ungracefully-vanished handles, then reopens every dead handle), `cleanup`, `formathp`, `opencon`, `readprocesscsv`, `retryrows`, `selector`, `updatestats`, @@ -92,9 +93,15 @@ priority-ordered fan-out. selection, missing or malformed `process.csv`). `init`'s own dependency validation is the one exception (plain `'` — no logger yet). - **`getapimeta`** exported; a test asserts it documents exactly the module's *callable* - exports — `init`/`getapimeta` are plumbing (di.torq calls them by convention) and are - deliberately omitted from the registry rows, matching di.toml and the skill convention. No - `version` export / VERSION file yet — deferred to the di.depcheck rollout, as in di.config. + exports — `init`/`getapimeta`/`version` are plumbing/metadata (di.torq calls or reads them by + convention) and are deliberately omitted from the registry rows, matching di.toml and the skill + convention. +- **`version` export** — a bare exported semver string (`"0.1.0"`, numeric `major.minor.patch`), + read by di.depcheck to satisfy other modules' declared minimum-version requirements. The source + of truth is **`version.txt`** in the module folder: `init.q` reads it module-relative at load + (`:::version.txt`) and it takes **priority** over the compiled-in fallback in `init.q`; a missing + or empty `version.txt` falls back to that default. Bump the release version by editing + `version.txt` alone. - **Env-free** — di.servers reads no environment variable; the `process.csv` path arrives via `config`processcsv` (di.torq resolves it), holding di.config's env-free boundary. diff --git a/di/servers/test.csv b/di/servers/test.csv index 950ecfda..45d1a2b4 100644 --- a/di/servers/test.csv +++ b/di/servers/test.csv @@ -49,9 +49,13 @@ fail,0,0,q,"svc.getservers[""nosuch""]",1,1,getservers rejects a non-symbol proc fail,0,0,q,"svc.gethandlebytype[`hdb;""any""]",1,1,gethandlebytype rejects a non-symbol selection fail,0,0,q,"svc.waitfortype[`hdb;""x"";200]",1,1,waitfortype rejects a non-integer timeout fail,0,0,q,"svc.init[`log`timer`handlers`proctype`procname!(mocklog;rtmr;mockhandlers;""rdb"";`p)]",1,1,init rejects a non-symbol proctype -true,0,0,q,(asc (key svc) except `init`getapimeta)~asc exec name from svc.getapimeta[],1,1,getapimeta documents exactly the callable exports (plumbing omitted) -true,0,0,q,not any `init`getapimeta in exec name from svc.getapimeta[],1,1,getapimeta omits plumbing (init/getapimeta not registered) +true,0,0,q,(asc (key svc) except `init`getapimeta`version)~asc exec name from svc.getapimeta[],1,1,getapimeta documents exactly the callable exports (version/plumbing omitted) +true,0,0,q,not any `init`getapimeta`version in exec name from svc.getapimeta[],1,1,getapimeta omits plumbing (init/getapimeta/version not registered) true,0,0,q,`name`public`descrip`params`return~cols svc.getapimeta[],1,1,getapimeta rows carry the registry columns +comment,,,,,,,module metadata - exported version +true,0,0,q,10h=type svc.version,1,1,version is a string +true,0,0,q,0 Date: Tue, 11 Aug 2026 20:03:39 +0100 Subject: [PATCH 8/9] Update version convention, di.server name changed to di.servers, restore getservers ALL/null contract to couple correctly with di.heartbeat and prevent self-connections --- di/servers/{version.txt => VERSION} | 0 di/servers/init.q | 11 +++++------ di/servers/servers.md | 4 ++-- di/servers/servers.q | 25 +++++++++++++++++++++---- di/servers/test.csv | 16 +++++++++++++++- di/servers/test.q | 10 ++++++++++ 6 files changed, 53 insertions(+), 13 deletions(-) rename di/servers/{version.txt => VERSION} (100%) diff --git a/di/servers/version.txt b/di/servers/VERSION similarity index 100% rename from di/servers/version.txt rename to di/servers/VERSION diff --git a/di/servers/init.q b/di/servers/init.q index 31fe018f..2f6ca55b 100644 --- a/di/servers/init.q +++ b/di/servers/init.q @@ -1,9 +1,8 @@ / connection management and handle-by-type lookup for the modular torq world. \l ::servers.q -/ module version: fallback default here, exported so di.depcheck can read di.servers' version to -/ satisfy other modules' declared minimums. -version:"0.1.0"; -/ version.txt in the module folder is the source of truth and takes priority over the fallback above; -/ read module-relative at load (`:::` resolves to di/servers). a missing/empty file keeps the fallback. -version:@[{trim first read0 x};`:::version.txt;{[e]version}]; +/ module version, read from the VERSION file rather than hardcoded, so a release bump touches one +/ plain-text file. read module-relative at load (`:::` resolves to di/servers), and BEFORE the export +/ line since export:([...]) evaluates each name. NB `version` stays in the export: di.depcheck +/ resolves a dependency's version from the export dict +version:first read0`:::VERSION export:([init;startup;getservers;gethandlebytype;waitfortype;getapimeta;version]) diff --git a/di/servers/servers.md b/di/servers/servers.md index a776a278..5227fd5f 100644 --- a/di/servers/servers.md +++ b/di/servers/servers.md @@ -43,8 +43,8 @@ h "1+1" | Function | Signature | Description | |---|---|---| | `init` | `init[deps]` | Wire deps + config, record identity, install the `.z.pc` handler + retry job. Idempotent. | -| `startup` | `startup[]` | Read `process.csv` (`processcsv`), drop self, connect to each row whose proctype is in `connections`. A failed connection is logged (not raised) and left as `w:0Ni` for `retry`. No-op if no connections configured. **Idempotent**: skips procs already tracked in `SERVERS`, so a repeat call (or a grown `process.csv`) adds only new rows — never a duplicate or a leaked second handle. `process.csv` must be the strict v1 4-column `host,port,proctype,procname` layout — a reordered or wider header is **rejected loudly** (the reader is positional, so it would otherwise misparse silently). | -| `getservers` | `getservers[proctype]` | Live (`w` non-null) `SERVERS` rows for a proctype. | +| `startup` | `startup[]` | Read `process.csv` (`processcsv`), drop self, connect to each row whose proctype is in `connections`. A failed connection is logged (not raised) and left as `w:0Ni` for `retry`. No-op if no connections configured. **Idempotent**: skips procs already tracked in `SERVERS`, so a repeat call (or a grown `process.csv`) adds only new rows — never a duplicate or a leaked second handle. `process.csv` must be the strict v1 4-column `host,port,proctype,procname` layout — a reordered or wider header is **rejected loudly** (the reader is positional, so it would otherwise misparse silently). **Self-exclusion is an exact `(proctype;procname)` match against the identity from config**, so a `process.csv` that disagrees would leave this process dialling itself; any surviving row carrying this process's `procname` is therefore skipped with a warning naming both proctypes. | +| `getservers` | ``getservers[proctype]`` | Live (`w` non-null) `SERVERS` rows. Accepts a **symbol**, a **symbol list** (rows for any of them), or `` ` `` for **every** proctype — the contract legacy TorQ's `.servers.getservers` (`trackservers.q:75`) and sibling `di.serverselect.getservers` both implement, so a consumer written against either works here unchanged. | | `gethandlebytype` | `gethandlebytype[proctype;selection]` | One live handle via `` `any``/`roundrobin`/`last``; `0Ni` if none. Bumps usage stats. | | `waitfortype` | `waitfortype[proctype;timeoutms;pollms]` | Block until a live connection exists or timeout; `1b`/`0b`. Caller decides if a timeout is fatal. `startup` must have run first. | | `getapimeta` | `getapimeta[]` | This module's api metadata, one row per **callable** API function (`init`/`getapimeta`/`version` plumbing omitted), for `di.torq` to register with `di.api`. | diff --git a/di/servers/servers.q b/di/servers/servers.q index c582f0eb..92af0c5a 100644 --- a/di/servers/servers.q +++ b/di/servers/servers.q @@ -140,6 +140,17 @@ startup:{[] procs:update isme:(proctype=pt)&procname=pn from procs; procs:select from procs where not isme; procs:select from procs where proctype in conns; + / SELF-CONNECTION GUARD. the exclusion above is an exact (proctype;procname) match against the + / identity from config. if process.csv disagrees - a drifted proctype for this procname - the self + / row is not recognised and this process dials ITSELF. procname is process.csv's unique key (the + / idempotency filter below relies on that too), so a surviving row carrying our procname IS us: + / drop it and say why, rather than opening a self-connection nobody would think to look for. + / checked AFTER the connections filter: a row that could never be connected to was never at risk + if[count mismatched:select from procs where procname=pn; + .z.m.logwarn[`startup;"process.csv lists procname ",(string pn)," as proctype ", + (string first mismatched`proctype),", but this process is configured as proctype ",(string pt), + " - identity drift, skipping that row rather than connecting to myself"]; + procs:select from procs where not procname=pn]; / idempotent: skip any proc already tracked in SERVERS. a repeat startup (or a process.csv that has / grown since) then adds only NEW rows - never a duplicate row or a leaked second handle to a proc / already connected. reconnecting a dropped peer is retry's job, not startup's. @@ -183,9 +194,15 @@ retry:{[] }; getservers:{[pt] - / every live (non-null handle) SERVERS row for a proctype. - if[not -11h=type pt;raiseerror[`getservers;"proctype must be a symbol"]]; - select from .z.m.SERVERS where proctype=pt, not null w + / every live (non-null handle) SERVERS row for a proctype. ` matches EVERY proctype and a list + / matches any of them - the contract legacy TorQ's .servers.getservers (trackservers.q:75) and + / di.serverselect.getservers both implement, and which any consumer written against either expects. + / matching with `in` rather than `=` is what makes both shapes work; a bare symbol still behaves + / exactly as before, so every existing caller is unaffected + if[not 11h=abs type pt;raiseerror[`getservers;"proctype must be a symbol or symbol list"]]; + $[`~pt; + select from .z.m.SERVERS where not null w; + select from .z.m.SERVERS where proctype in pt, not null w] }; selector:{[tab;selection] @@ -246,7 +263,7 @@ getapimeta:{[] / listed - the registry describes the callable api, not plumbing. names are bare (di.torq qualifies). :flip `name`public`descrip`params`return!flip( (`startup; 1b; "open connections to configured proctypes from process.csv (reads init config)"; "[]"; "null"); - (`getservers; 1b; "live SERVERS rows for a proctype"; "[symbol: proctype]"; "table: live server rows"); + (`getservers; 1b; "live SERVERS rows for a proctype, a list of proctypes, or ` for all"; "[symbol|list: proctype, or ` for all]"; "table: live server rows"); (`gethandlebytype; 1b; "one live handle for a proctype via any/roundrobin/last selection"; "[symbol: proctype; symbol: selection]"; "int: handle, 0Ni if none"); (`waitfortype; 1b; "block until a proctype connects or timeout elapses"; "[symbol: proctype; long: timeoutms; long: pollms]"; "boolean: 1b connected, 0b timed out")); }; diff --git a/di/servers/test.csv b/di/servers/test.csv index 45d1a2b4..11f13c66 100644 --- a/di/servers/test.csv +++ b/di/servers/test.csv @@ -24,6 +24,13 @@ true,0,0,q,0=count svc.getservers[`selfproc],1,1,self is excluded even though it true,0,0,q,1=count svc.getservers[`otherproc],1,1,otherproc connected successfully true,0,0,q,0=count svc.getservers[`deadproc],1,1,deadproc failed (nothing listening) so it is not live true,0,0,q,"warnlogged[""failed to open connection""]",1,1,the failed dial was logged at warn +comment,,,,,,,"getservers contract - ` matches EVERY proctype and a list matches any of them (legacy trackservers.q:75 and di.serverselect both implement this; = against an atom was a regression against both)" +true,0,0,q,"(count svc.getservers[`])=sum count each svc.getservers each `selfproc`otherproc`deadproc",1,1,` returns exactly the union of the per-proctype results - match-all is not a special case with its own filter +true,0,0,q,1=count svc.getservers[`otherproc`deadproc],1,1,a list returns live rows for any listed proctype - deadproc is not live so contributes none +true,0,0,q,1=count svc.getservers[`otherproc`nosuchtype],1,1,an unknown proctype in the list is simply absent rather than an error +true,0,0,q,0=count svc.getservers[`nosuchtype`alsonone],1,1,a list matching nothing returns an empty table not an error +true,0,0,q,svc.getservers[`otherproc]~svc.getservers[enlist`otherproc],1,1,a one-element list is identical to the bare symbol - the existing atom path is unchanged +fail,0,0,q,svc.getservers[42],1,1,getservers still rejects a non-symbol non-list comment,,,,,,,gethandlebytype - a genuinely live remote handle run,0,0,q,h:svc.gethandlebytype[`otherproc;`any],1,1,get a handle to the peer true,0,0,q,"2=h ""1+1""",1,1,the handle is a live remote connection to a separate process @@ -46,6 +53,13 @@ run,0,0,q,"svc.init[`log`timer`handlers`proctype`procname`connections`processcsv fail,0,0,q,svc.startup[],1,1,startup signals on the bad header rather than silently misreading columns comment,,,,,,,input validation + getapimeta fail,0,0,q,"svc.getservers[""nosuch""]",1,1,getservers rejects a non-symbol proctype + +comment,,,,,,,"startup must not dial ITSELF when process.csv disagrees with the configured identity" +run,0,0,q,"svc.init[`log`timer`handlers`proctype`procname`connections`processcsv!(mocklog;rtmr;mockhandlers;`selfproc;`selfinst;`rdb;writedriftcsv[])]",1,1,re-init against a csv listing OUR procname (selfinst) under a different proctype +run,0,0,q,delete from `logrows,1,1,clear the log capture +run,0,0,q,svc.startup[],1,1,startup - the exact (proctype;procname) match cannot recognise the drifted self row +true,0,0,q,0=count select from .m.di.0servers.SERVERS where procname=`selfinst,1,1,no row was added for this process - the guard skipped it rather than opening a connection to itself +true,0,0,q,"warnlogged[""connecting to myself""]",1,1,and the drift was reported rather than skipped silently fail,0,0,q,"svc.gethandlebytype[`hdb;""any""]",1,1,gethandlebytype rejects a non-symbol selection fail,0,0,q,"svc.waitfortype[`hdb;""x"";200]",1,1,waitfortype rejects a non-integer timeout fail,0,0,q,"svc.init[`log`timer`handlers`proctype`procname!(mocklog;rtmr;mockhandlers;""rdb"";`p)]",1,1,init rejects a non-symbol proctype @@ -55,7 +69,7 @@ true,0,0,q,`name`public`descrip`params`return~cols svc.getapimeta[],1,1,getapime comment,,,,,,,module metadata - exported version true,0,0,q,10h=type svc.version,1,1,version is a string true,0,0,q,0 Date: Thu, 13 Aug 2026 16:10:38 +0100 Subject: [PATCH 9/9] following automated reviewer comments --- di/servers/init.q | 8 ++++++-- di/servers/servers.md | 4 ++-- 2 files changed, 8 insertions(+), 4 deletions(-) diff --git a/di/servers/init.q b/di/servers/init.q index 2f6ca55b..df5fd89c 100644 --- a/di/servers/init.q +++ b/di/servers/init.q @@ -3,6 +3,10 @@ / module version, read from the VERSION file rather than hardcoded, so a release bump touches one / plain-text file. read module-relative at load (`:::` resolves to di/servers), and BEFORE the export / line since export:([...]) evaluates each name. NB `version` stays in the export: di.depcheck -/ resolves a dependency's version from the export dict -version:first read0`:::VERSION +/ resolves a dependency's version from the export dict. +/ trim so a trailing newline/CRLF cannot pad the semver; fail loud with a clear message if VERSION is +/ missing/unreadable/empty (it is a required module file - better than a raw OS error, a silent empty +/ value, or a misleading 0.0.0 that would corrupt depcheck's version comparison). +version:@[{trim first read0 x};`:::VERSION;{'"di.servers: VERSION file missing or unreadable"}]; +if[0=count version;'"di.servers: VERSION file is empty"]; export:([init;startup;getservers;gethandlebytype;waitfortype;getapimeta;version]) diff --git a/di/servers/servers.md b/di/servers/servers.md index 5227fd5f..d008304e 100644 --- a/di/servers/servers.md +++ b/di/servers/servers.md @@ -43,12 +43,12 @@ h "1+1" | Function | Signature | Description | |---|---|---| | `init` | `init[deps]` | Wire deps + config, record identity, install the `.z.pc` handler + retry job. Idempotent. | -| `startup` | `startup[]` | Read `process.csv` (`processcsv`), drop self, connect to each row whose proctype is in `connections`. A failed connection is logged (not raised) and left as `w:0Ni` for `retry`. No-op if no connections configured. **Idempotent**: skips procs already tracked in `SERVERS`, so a repeat call (or a grown `process.csv`) adds only new rows — never a duplicate or a leaked second handle. `process.csv` must be the strict v1 4-column `host,port,proctype,procname` layout — a reordered or wider header is **rejected loudly** (the reader is positional, so it would otherwise misparse silently). **Self-exclusion is an exact `(proctype;procname)` match against the identity from config**, so a `process.csv` that disagrees would leave this process dialling itself; any surviving row carrying this process's `procname` is therefore skipped with a warning naming both proctypes. | +| `startup` | `startup[]` | Read `process.csv` (`processcsv`), drop self, connect to each row whose proctype is in `connections`. A failed connection is logged (not raised) and left as `w:0Ni` for `retry`. No-op if no connections configured. **Idempotent**: skips procs already tracked in `SERVERS`, so a repeat call (or a grown `process.csv`) adds only new rows — never a duplicate or a leaked second handle. `process.csv` must be the strict v1 4-column `host,port,proctype,procname` layout — a reordered or wider header is **rejected loudly** (the reader is positional, so it would otherwise misparse silently). **Self-exclusion is an exact `(proctype;procname)` match against the identity from config**, so a `process.csv` that disagrees would leave this process dialling itself; any row carrying this process's `procname` **that survives the connections filter (i.e. one this process would otherwise dial)** is therefore skipped with a warning naming both proctypes. A drifted row whose proctype isn't a connection type is already filtered out and poses no self-connection risk, so it is not warned about. | | `getservers` | ``getservers[proctype]`` | Live (`w` non-null) `SERVERS` rows. Accepts a **symbol**, a **symbol list** (rows for any of them), or `` ` `` for **every** proctype — the contract legacy TorQ's `.servers.getservers` (`trackservers.q:75`) and sibling `di.serverselect.getservers` both implement, so a consumer written against either works here unchanged. | | `gethandlebytype` | `gethandlebytype[proctype;selection]` | One live handle via `` `any``/`roundrobin`/`last``; `0Ni` if none. Bumps usage stats. | | `waitfortype` | `waitfortype[proctype;timeoutms;pollms]` | Block until a live connection exists or timeout; `1b`/`0b`. Caller decides if a timeout is fatal. `startup` must have run first. | | `getapimeta` | `getapimeta[]` | This module's api metadata, one row per **callable** API function (`init`/`getapimeta`/`version` plumbing omitted), for `di.torq` to register with `di.api`. | -| `version` | `version` | The module's semver string (`"0.1.0"`) — metadata, not a function. Sourced from `version.txt` (priority) with an `init.q` fallback; read by `di.depcheck` to satisfy other modules' declared minimum-version requirements. | +| `version` | `version` | The module's semver string (`"0.1.0"`) — metadata, not a function. Read at load from the plain-text `VERSION` file in the module folder; read by `di.depcheck` to satisfy other modules' declared minimum-version requirements. | Export is deliberately conservative — only functions `di.torq` or a consumer actually calls (so `di.api` lists exactly these), plus the `version` metadata string. The rest are **internal**: `retry` (the scheduled