From 8309da61cf9ca6a6fc907468f8841b48ea6b0e64 Mon Sep 17 00:00:00 2001 From: alowrydi Date: Mon, 20 Jul 2026 17:08:20 +0100 Subject: [PATCH 01/11] Initial draft for di.depcheck --- di/depcheck/depcheck.md | 211 +++++++++++++++++++++++++++++++++ di/depcheck/depcheck.q | 256 ++++++++++++++++++++++++++++++++++++++++ di/depcheck/deps.q | 4 + di/depcheck/init.q | 6 + di/depcheck/test.csv | 80 +++++++++++++ 5 files changed, 557 insertions(+) create mode 100644 di/depcheck/depcheck.md create mode 100644 di/depcheck/depcheck.q create mode 100644 di/depcheck/deps.q create mode 100644 di/depcheck/init.q create mode 100644 di/depcheck/test.csv diff --git a/di/depcheck/depcheck.md b/di/depcheck/depcheck.md new file mode 100644 index 00000000..5b0d3652 --- /dev/null +++ b/di/depcheck/depcheck.md @@ -0,0 +1,211 @@ +# di.depcheck + +Dependency, version, core-contract, and `.z.ts`-ownership auditing for kdb-x modules. It is the modernised +successor to legacy TorQ's `checkdependency`/`runchk`/`checkvers` (`torq.q`): a per-module `deps.q` replaces the +old CSV registry, real numeric semver replaces the 5-component digit-walk, and — new — it enforces the shape of +the shared "core dependency contracts" (Logging / Timer / Handlers) that the DI dependency-injection pattern +relies on. + +--- + +## How it works + +di.depcheck is a **post-load audit, not a pre-load gate**. It is intended to run once, after a host process +(`di.torq`) has already `use`d every module it needs for that process. It never calls `use` on anything it +checks — every check reads other modules' state purely by introspecting the session namespace the kdb-x `use` +loader already populates: + +- Every loaded module lands under `` `.m.di `` keyed by a short form (`di.timer` → `` `0timer ``, giving + `` `.m.di.0timer ``), and that module's full `export` dict is itself readable there + (`` .m.di.0timer.export ``). This is how di.depcheck reads another module's exports and `version` without + importing it. +- "Not found" in a failure report means a *declared* dependency was never `use`d into this session by whatever + loaded di.depcheck's caller — it is not a QPATH filesystem scan. + +### Checks performed + +1. **Presence & minimum version** — for every currently-loaded module that ships a `deps.q` + (`` deps:`di.tplog`di.pubsub!("0.2.0";"0.3.0") ``, symbol-keyed by dependency module name, string-valued by + minimum version), each declared dependency is checked: is it loaded, and if so does its exported `version` + satisfy the declared minimum (real numeric `major.minor.patch` comparison, not a digit-walk)? A module with no + `deps.q` is skipped, not treated as an error — most modules don't ship one yet. +2. **Core dependency contracts** — for whichever of `di.log`, `di.timer`, or `di.handlers` are loaded, their + export dict is checked for the required keys of the contract they provide (Logging: `info`/`warn`/`error`; + Timer: `addjob`/`deletejobs`/`enablejobs`/`disablejobs`/`getactivejobs`/`cp`; Handlers: + `register`/`remove`/`list`). This is a fixed, named set — not a generic self-declaration registry, since no + such mechanism exists elsewhere in this codebase. +3. **`.z.ts` ownership** — warns if `.z.ts` is bound to something while di.timer either isn't loaded or doesn't + look initialised (its `enabled` state is used as a proxy for "di.timer's `init` ran and bound `.z.ts` + itself"). This does **not** prove nothing has overwritten `.z.ts` afterwards — an accepted limitation of a + warning-level check. Scope is deliberately limited to `.z.ts` only. +4. **kdb-x engine version** (optional, and currently unwired) — compares the running engine's `.z.K` (e.g. `5f`) + against an optional `` `minkdbxversion `` passed alongside `log` on the same `deps` dict. **This is the shape + of the check, not a live check yet**: if `` `minkdbxversion `` is absent from `deps` — which is every real + call site today, since no caller exists that supplies one — `kdbxcheck` returns immediately without looking + at `.z.K` at all. There is currently no config path that could supply a real minimum: di.config's cascade + isn't wired to di.torq yet (di.config's own docs list that as future work), and `deps.q`'s format is + per-dependency *module* versions, not an engine-level minimum. The comparison itself is unit-tested and + correct in isolation (see `test.csv`) — what's missing is a caller that has an opinion on what the minimum + should be. Uses `.z.K`, not `.z.v` — `.z.v`'s value on the box this module was developed on + (`"5.0.20260122"`) is a build-stamp string, not a confirmed match for the kdb-x product version, and + di.k4unit already has a working precedent for exactly this problem (`minver<=.z.K` gates which tests run). + +Presence and version failures, and core-contract failures, are **fail-fast**: `init` logs a single multi-line +report at `error` and then signals, so a caller sees a blocking error. The `.z.ts` and kdb-x-version checks are +**warning-only**: logged at `warn`, never signalled. + +### Report format + +``` +DEPENDENCY CHECK FAILED: + di.tplog requires minimum version 0.2.0, found 0.1.3 + di.pubsub requires minimum version 0.3.0, not found + +WARNING: + .z.ts has been directly assigned outside di.timer. This may cause timer conflicts. +``` + +A loaded dependency that exports no `version` at all gets its own distinct line (neither "found" nor "not +found" fits): `di.timer requires minimum version 1.0.0, but di.timer exports no version`. + +--- + +## Dependencies + +| Dependency | Key | Required | Description | +|---|---|---|---| +| logger | `` `log `` | yes | dict with `info`, `warn`, and `error`, each binary `{[c;m]}` where `c` is a symbol context and `m` is a string (per `consistency.md`) | +| kdb-x minimum version | `` `minkdbxversion `` | no | an optional float compared against `.z.K` | + +**No hard dependencies** on other `di.*` modules — the module works standalone, and ships its own (empty) +`deps.q`, dogfooding the convention it introduces. + +--- + +## Initialisation + +```q +depcheck:use`di.depcheck + +logdep:`info`warn`error!( + {[c;m] -1 string[c],": INFO ",m;}; + {[c;m] -1 string[c],": WARN ",m;}; + {[c;m] -2 string[c],": ERROR ",m;}); + +depcheck.init[enlist[`log]!enlist logdep] + +/ with an optional kdb-x minimum version: +depcheck.init[`log`minkdbxversion!(logdep;5.0)] +``` + +`init` must be called after every module the host process needs has already been `use`d (this is what it +audits) — typically the very last thing `di.torq` does during startup. + +--- + +## Exported Functions + +### `init[deps]` +Validate the required `log` dependency, then run every check against the current session and report. `deps` is +a dict with a `` `log `` key and an optional `` `minkdbxversion `` float. Throws on any presence, version, or +core-contract failure; logs (but does not throw on) `.z.ts`-ownership or kdb-x-version warnings. +```q +depcheck.init[enlist[`log]!enlist logdep] +``` + +### `version` +The module version string. +```q +depcheck.version / "0.1.0" +``` + +--- + +## Running Tests + +```q +k4unit:use`di.k4unit +k4unit.moduletest`di.depcheck +``` + +Tests drive real, already-shipped modules rather than synthetic fixtures, so the assertions track the codebase's +actual current state: + +- **di.timer** is merged to `main`, so it is loaded unconditionally and always exercised for real (missing + `version`; `cp` defined but not exported — both real, live gaps). +- **di.kafka** (PR #112) and **di.handlers** (PR #114, the positive control — the one module that does export + `version`) are still unmerged, so this suite must not assume they're present. Their loads are protected + (`.dc.havekafka`/`.dc.havehandlers`), and every assertion depending on them is written as + `(not haveflag) or realassertion` — a real check when the module is loaded, a no-op pass when it isn't. + Verified directly against a bare `main`-only checkout (no other branches merged in): all 34 tests pass there + too, exercising real di.timer coverage and no-op'ing the di.kafka/di.handlers-dependent assertions rather than + crashing on a missing module. +- **di.log** (PR #90, DI-Dexter fork) follows the same graceful pattern — checked for real when resolvable on + QPATH, a no-op pass otherwise. + +--- + +## Notes + +- **Known, flagged gaps**, not fixed here: + - Three real, shipped modules were found missing `version` during development: di.timer, di.kafka, and di.log + (the DI-Dexter fork PR #90 candidate). di.handlers is the only one that has it, and its own source comments + call that a placeholder pending this module's existence. + - di.compression imports `kx.log` directly rather than following the binary `{[c;m]}` three-flat-var + convention this module (and di.handlers/di.kafka/di.config/di.eodtime) uses — a pre-`consistency.md` outlier + worth a separate cleanup pass. + - The `.z.ts` ownership check is a best-effort proxy (see above) — it cannot detect something rebinding + `.z.ts` after di.timer's `init` runs. + - The 0.x.y semver tension: a passing `>=` check during 0.x.y development does not guarantee contract + compatibility, since every module in this workstream is currently pre-1.0. Implemented as literal `>=` + anyway, per the plan. + - The kdb-x-version check is **shape-only as of this PR, not operating against anything real yet**: it is + warning-only rather than fail-fast, and — more importantly — no caller exists today that passes + `` `minkdbxversion ``, so it is a no-op in every real invocation until di.torq (or some other caller) is + built and threads a real minimum through. Do not describe this PR as "implements the kdb-x version check" + to reviewers — it implements the comparison, unit-tested in isolation, with no live minimum source wired up. + - `test.csv` assumes di.timer is present (true of every real checkout of this repo, since it's merged to + `main`) — verified against a bare `main`-only checkout, but not against an arbitrarily minimal QPATH + containing only di.depcheck and di.k4unit. The module itself has no such assumption (verified standalone + against exactly that minimal QPATH); only the test suite's negative control does. + - The `checkdeps[]` case of two different loaded modules declaring the same dependency at different minimums + is manually verified, not committed as an automated test — it needs two real `deps.q` fixtures on disk, and + no real module ships a non-empty `deps.q` yet to build a portable, no-hardcoded-path test against. Verified + directly: `di.handlers` required simultaneously at a satisfied minimum (by one fixture consumer) and an + unsatisfied one (by another) correctly produced exactly one failure line, for the unsatisfied case only. + - A malformed `deps.q` — present but not a dict (wrong type; a plausible authoring mistake) — is reported as + its own failure line (`" deps.q is malformed - expected a dict, got type "`) rather than crashing. + Not committed as an automated test, for the same real-file-on-disk reason as the point above; manually + verified: a fixture with `deps:"a string"` produced exactly that report line and did not abort the walk. + +- **Two real bugs found and fixed**, both by directly constructing and running the edge case, not by reading the + code — both looked entirely reasonable on the page: + 1. **`checkdepversion`'s eager `or`.** Originally combined `(xp~(::)) or not \`version in key xp` as a single + condition. q's `or`/`and` are eager vector operators, not short-circuiting, so `key xp` was evaluated even + when `xp` was already known to be `(::)`, throwing `'type` (`key` doesn't accept a generic null). Fixed to + sequential `if[]` early returns, matching the pattern `checkonecontract` already used correctly for the + same situation. Constructed via direct `.m.di` namespace manipulation, since no real broken module could be + made to load successfully and then fail an export read. + 2. **A malformed `deps.q` crashed the entire audit, not just the one bad module.** `checkmoduledeps` handed + whatever `readdepsq` returned straight to `key`/`value`/`checkonedep'` with no type check. A `deps.q` that + defines `deps` as something other than a dict (e.g. a plain string) threw a raw `'dict` error out of that + `each` call — and since `each` doesn't isolate per-element errors, one badly-authored `deps.q` anywhere in + the loaded module set aborted `checkdeps[]` entirely, masking every real failure in every other module. + Fixed by validating `99h=type d` in `checkmoduledeps` and reporting malformed `deps.q` as its own clear + failure line instead. This one is the more serious of the two: for a tool whose entire purpose is to run + reliably at startup, an unhandled crash from one module's authoring mistake defeats the purpose more + thoroughly than any single check being wrong. + 3. **Dependency resolution was silently wrong for any non-`di.*` name.** `getexport`/`checkonedep`/ + `checkonecontract` all hardcoded the `` `.m.di `` namespace when checking whether a dependency was loaded. + A di.* module's `deps.q` can legitimately name an external vendor module as a hard dependency (e.g. + `kx.log`) — but vendor modules register under their own `` `.m. `` namespace (`kx.log` → + `` `.m.kx ``, confirmed directly: `use\`kx.log` populates `` `.m.kx ``, not `` `.m.di ``). A genuinely + loaded `kx.log` was reported as `"kx.log requires minimum version 1.0.0, not found"` — a silent false + negative, worse than a crash, since it looks like a correct, actionable result. Fixed by generalising + `shortmod`/introducing `modvendorns` to resolve a dependency's vendor namespace from its own name instead + of assuming `di.`. `checkdeps`'s outer walk of *which modules to audit as consumers* stays intentionally + scoped to `` key `.m.di `` — di.depcheck audits the di.* modularisation effort's dependency graph, not + arbitrary vendor modules' own internal needs; only *resolving a declared dependency's target* needed to + stop assuming di.*. `kx.log` itself isn't committed anywhere in this repo (confirmed via `git ls-tree` + across every branch — it's only vendored locally on this dev machine), so the regression test fabricates a + non-di.* module via direct namespace manipulation (`` `.m.zz.0widget ``) rather than depending on it. diff --git a/di/depcheck/depcheck.q b/di/depcheck/depcheck.q new file mode 100644 index 00000000..0c12f86a --- /dev/null +++ b/di/depcheck/depcheck.q @@ -0,0 +1,256 @@ +/ dependency, version, core-contract, and .z.ts ownership auditing for already-loaded kdb-x modules +/ this module never calls `use` on anything it checks - every check reads other modules' state purely by +/ introspecting the session namespace the kdb-x `use` loader already populates (`.m.di.0`), so it stays +/ standalone with no hard di.* dependency of its own + +/ module version - di.depcheck is the first module to carry one, dogfooding the convention it introduces +version:"0.1.0"; + +/ the fixed set of core dependency contracts di.depcheck knows how to validate, keyed by the module that provides +/ each one - not a generic self-declaration registry, since no such mechanism exists anywhere in this codebase yet +contracts:`di.log`di.timer`di.handlers!( + `info`warn`error; + `addjob`deletejobs`enablejobs`disablejobs`getactivejobs`cp; + `register`remove`list + ); + +/ ============================================================ +/ session-namespace introspection helpers +/ ============================================================ + +shortmod:{[modname] + / di.timer -> `0timer ; kx.log -> `0log - the short form the kdb-x `use` loader keys a module under, off its + / vendor's `.m. namespace. Assumes exactly one dot (vendor.name), matching every module name seen in + / this codebase so far + `$"0",last "." vs string modname + }; + +shorttofull:{[s] + / `0timer -> `di.timer - only ever applied to keys of `.m.di (checkdeps walks di.* modules specifically, see + / checkdeps), so the di. prefix is correct here and does not need vendor-generalising like shortmod/modvendorns + `$"di.",1_string s + }; + +modvendorns:{[modname] + / di.timer -> `.m.di ; kx.log -> `.m.kx - the top-level session namespace a module's vendor is keyed under. + / a di.* module's deps.q may legitimately declare a hard dependency on an external vendor module (e.g. + / kx.log), so dependency resolution (getexport/checkonedep/checkonecontract) must not hardcode `.m.di - only + / checkdeps's walk of which modules to audit as consumers is intentionally di.*-scoped + `$".m.",first "." vs string modname + }; + +getexport:{[modname] + / read another already-loaded module's export dict purely via session-namespace introspection - no `use`, no import + / returns (::) if the module isn't loaded, or if its export somehow can't be read + sn:shortmod modname; + vns:modvendorns modname; + if[not sn in key vns;:(::)]; + @[get;`$(string vns),".",(string sn),".export";{(::)}] + }; + +/ ============================================================ +/ deps.q loading +/ ============================================================ + +finddepsq:{[modname] + / locate /deps.q on QPATH (colon-separated, like PATH); returns its file path, or (::) if the module ships none + relpath:(ssr[string modname;".";"/"]),"/deps.q"; + roots:":" vs getenv`QPATH; + paths:{[relpath;root] hsym `$root,"/",relpath}[relpath;] each roots; + found:paths where not {[p] ()~key p} each paths; + $[0=count found;(::);first found] + }; + +readdepsq:{[modname] + / load 's deps.q - a single pure `deps:...` assignment, by convention (the only real precedent, di.merge, + / has no other content) - and capture its value without leaving a stray global `deps` behind + p:finddepsq modname; + if[p~(::);:(::)]; + @[system;"l ",1_string p;{[e] (::)}]; + d:@[get;`deps;{(::)}]; + delete deps from `.; + d + }; + +/ ============================================================ +/ semver comparison +/ ============================================================ + +parsesemver:{[v] + / parse a "major.minor.patch" string into a 3-long int vector; any parse failure yields nulls at that position + parts:"." vs v; + if[not 3=count parts;:3#0Ni]; + {@[{"I"$x};x;0Ni]} each parts + }; + +vercmp:{[a;b] + / -1/0/1 comparing semver strings a and b by (major,minor,patch); a null component sorts lowest + / real numeric semver comparison only - no pre-release/build-metadata support, matching every version string seen + / in this codebase so far (all plain X.Y.Z), unlike legacy TorQ's 5-component digit-walk + pa:parsesemver a; + pb:parsesemver b; + diffs:pa<>pb; + $[not any diffs;0i;[i:diffs?1b;$[pa[i]= check does not guarantee contract compatibility - minor bumps may + / carry breaking changes pre-1.0 across this workstream. Implementing literal >= anyway, per the plan; this is a + / known, accepted gap, not something this function tries to solve + not -1i=vercmp[a;b] + }; + +/ ============================================================ +/ dependency presence/version checks +/ ============================================================ + +checkfoundversion:{[dep;minver;foundver] + / dep is loaded and exports a version - compares it against the declared minimum + $[vergte[foundver;minver];();enlist string[dep]," requires minimum version ",minver,", found ",foundver] + }; + +checkdepversion:{[dep;minver] + / dep is confirmed loaded - checks its exported version, if any, against minver + / NOTE: sequential if[] early returns, not a single `or`-combined condition - q's `or`/`and` are eager vector + / operators, not short-circuiting, so `(xp~(::)) or not `version in key xp` would evaluate `key xp` even when + / xp is (::) and throw 'type. Caught by direct testing, not by reading the code - see depcheck.md + xp:getexport dep; + noversionmsg:enlist string[dep]," requires minimum version ",minver,", but ",string[dep]," exports no version"; + if[xp~(::);:noversionmsg]; + if[not `version in key xp;:noversionmsg]; + checkfoundversion[dep;minver;xp`version] + }; + +checkonedep:{[dep;minver] + / checks a single declared (dependency;minimum-version) pair against the current session + / returns () on pass, or an enlisted failure line matching the plan's exact report format + / not vendor-restricted to di.* - a deps.q may name an external vendor module (e.g. kx.log) as a hard + / dependency, so presence is checked against dep's own vendor namespace, not hardcoded to `.m.di + depshort:shortmod dep; + vns:modvendorns dep; + $[not depshort in key vns; + enlist string[dep]," requires minimum version ",minver,", not found"; + checkdepversion[dep;minver]] + }; + +checkmoduledeps:{[modshort] + / checks one already-loaded module's declared deps.q (if any) against the current session + / a malformed deps.q (present but not a dict) is reported as its own failure rather than left to throw a raw + / q type error out of checkonedep'[key d;value d] - a crash there would abort the whole checkdeps[] walk (each + / does not isolate per-element errors), masking every other module's real failures behind one bad file + modname:shorttofull modshort; + d:readdepsq modname; + $[d~(::);(); + not 99h=type d;enlist string[modname]," deps.q is malformed - expected a dict, got type ",string type d; + raze checkonedep'[key d;value d]] + }; + +checkdeps:{[] + / walks every loaded di.* module's deps.q and checks each declared dependency for presence and minimum version + / "not found" here means a declared dependency was never `use`d into this session - this is a post-load audit, + / not a QPATH filesystem scan (see depcheck.md for why) + / if two different loaded modules declare the same dependency at different minimums, each is checked + / independently and both lines are emitted if both fail - checkonedep is a pure function of (dep;minver) with + / no shared state across calls, so this needs no special handling. Manually verified with two real deps.q + / fixtures on disk (di.handlers required at both a satisfied and an unsatisfied minimum simultaneously) since + / no real module ships a non-empty deps.q yet to build a committed, portable test against - see depcheck.md + raze checkmoduledeps each key `.m.di + }; + +/ ============================================================ +/ core dependency contract checks +/ ============================================================ + +checkonecontract:{[provider;required] + / if provider is loaded, checks its export dict contains every key its known contract requires + / vendor-agnostic like checkonedep/getexport, though every current contracts entry happens to be di.*-prefixed + sn:shortmod provider; + vns:modvendorns provider; + if[not sn in key vns;:()]; + xp:getexport provider; + if[xp~(::);:enlist string[provider]," is loaded but its export dict could not be read"]; + missing:required where not required in key xp; + $[0=count missing;();enlist string[provider]," is missing required contract key(s): ",", " sv string missing] + }; + +checkcontracts:{[] + / checks whichever of the known core-dependency providers (di.log/di.timer/di.handlers) are loaded in this session + raze checkonecontract'[key contracts;value contracts] + }; + +/ ============================================================ +/ .z.ts ownership check +/ ============================================================ + +ztscheck:{[] + / warns if .z.ts is bound to something while di.timer either isn't loaded or doesn't look initialised + / LIMITATION (accepted, warning-level only): di.timer's `enabled` flag being 1b is evidence its init ran and bound + / .z.ts itself - it does NOT prove nothing has overwritten .z.ts since. di.timer assigns .z.ts directly (not via + / di.handlers, which explicitly excludes .z.ts from its own scope), so there is no ownership marker to check + / instead. Scope is deliberately limited to .z.ts only - whether this should ever cover other .z.* events is an + / open question for di.handlers' owner, not decided here + bound:not (::)~@[get;`.z.ts;{(::)}]; + if[not bound;:()]; + timerowns:(`0timer in key `.m.di) and 1b~@[get;`.m.di.0timer.enabled;0b]; + if[timerowns;:()]; + enlist ".z.ts has been directly assigned outside di.timer. This may cause timer conflicts." + }; + +/ ============================================================ +/ kdb-x engine version check +/ ============================================================ + +kdbxcheck:{[deps] + / compares the running kdb-x engine's major.minor (.z.K) against an optional minimum passed via deps`minkdbxversion + / uses .z.K, matching di.k4unit's own precedent (`minver<=.z.K` gates which tests run) rather than parsing .z.v, + / whose value on this box ("5.0.20260122") did not match the plan's assumed kdb-x product-semver shape and isn't + / confirmed to be the same number as the product version shown in the kdb-x startup banner - see depcheck.md + / warning-level, not fail-fast: there is no config cascade yet for di.depcheck to source a minimum from, so this + / is opt-in via an extra key on the same `deps dict rather than a new config channel + minversion:$[`minkdbxversion in key deps;deps`minkdbxversion;0Nf]; + if[null minversion;:()]; + if[.z.K>=minversion;:()]; + enlist "kdb-x engine version ",(string .z.K)," is below the configured minimum ",(string minversion),"." + }; + +/ ============================================================ +/ report formatting +/ ============================================================ + +buildreport:{[header;lines] + / formats a bulleted, indented block under a header line, matching the plan's exact "HEADER:\n line1\n line2" shape + header,":\n",sv["\n";" ",/:lines] + }; + +/ ============================================================ +/ public api +/ ============================================================ + +init:{[deps] + / initialise di.depcheck - validate the required log dependency, then audit the current session: dependency + / presence/version, core-dependency-contract shape, .z.ts ownership, and (optionally) a minimum kdb-x engine + / version. deps: a dict with a required `log key (binary `info`warn`error functions, per consistency.md) and an + / optional `minkdbxversion float + if[99h<>type deps;'"di.depcheck: deps must be a dict with `log key"]; + if[not `log in key deps;'"di.depcheck: log dependency is required; pass `info`warn`error functions - see di.log"]; + if[99h<>type deps`log;'"di.depcheck: log value must be a dict; pass `info`warn`error functions"]; + if[not all `info`warn`error in key deps`log; + '"di.depcheck: log dict must have `info`warn`error keys; got: ",(", " sv string key deps`log)]; + .z.m.loginfo:(deps`log)`info; + .z.m.logwarn:(deps`log)`warn; + .z.m.logerr:(deps`log)`error; + + failures:checkdeps[],checkcontracts[]; + warnings:ztscheck[],kdbxcheck[deps]; + + if[count failures; + report:buildreport["DEPENDENCY CHECK FAILED";failures]; + .z.m.logerr[`depcheck;report]; + '"di.depcheck: ",report]; + + if[count warnings; + .z.m.logwarn[`depcheck;buildreport["WARNING";warnings]]]; + + .z.m.loginfo[`depcheck;"dependency check complete: ",(string count failures)," failure(s), ",(string count warnings)," warning(s)"]; + }; diff --git a/di/depcheck/deps.q b/di/depcheck/deps.q new file mode 100644 index 00000000..66222c30 --- /dev/null +++ b/di/depcheck/deps.q @@ -0,0 +1,4 @@ +/ hard module dependencies and their minimum versions, validated by di.depcheck +/ di.depcheck has no hard dependencies - its only runtime dependency (log) is injected via init as a dictionary of +/ functions, following the same convention di.depcheck itself checks other modules against +deps:(`$())!(); diff --git a/di/depcheck/init.q b/di/depcheck/init.q new file mode 100644 index 00000000..35b1519e --- /dev/null +++ b/di/depcheck/init.q @@ -0,0 +1,6 @@ +/ di.depcheck - dependency presence/version, core-contract, and .z.ts ownership auditing for kdb-x modules +/ intended to run once, post-load, after a host process (di.torq) has `use`d every module it needs - see depcheck.md + +\l ::depcheck.q + +export:([init;version]) diff --git a/di/depcheck/test.csv b/di/depcheck/test.csv new file mode 100644 index 00000000..073c5b0a --- /dev/null +++ b/di/depcheck/test.csv @@ -0,0 +1,80 @@ +action,ms,bytes,lang,code,repeat,minver,comment +comment,,,,,,,setup - load module and a capturing logger +before,0,0,q,depcheck:use`di.depcheck,1,,load di.depcheck module +before,0,0,q,.dc.captbl:([]lvl:`symbol$();ctx:`symbol$();msg:()),1,,log capture table for assertions +before,0,0,q,caplog:`info`warn`error!({[c;m] `.dc.captbl insert (`info;c;m)};{[c;m] `.dc.captbl insert (`warn;c;m)};{[c;m] `.dc.captbl insert (`error;c;m)}),1,,capturing binary logger {[c;m]} + +comment,,,,,,,init - dependency validation +fail,0,0,q,depcheck.init[(::)],1,,init rejects a non-dict deps +fail,0,0,q,depcheck.init[()!()],1,,init rejects missing log key +fail,0,0,q,depcheck.init[enlist[`log]!enlist 42],1,,init rejects a non-dict log value +fail,0,0,q,depcheck.init[enlist[`log]!enlist `info`warn!(caplog`info;caplog`warn)],1,,init rejects a log dict missing the error key +run,0,0,q,.dc.errstr:@[{depcheck.init[()!()]};(::);{x}],1,,capture the error string from a bad init +true,0,0,q,.dc.errstr like "di.depcheck:*",1,,init error is prefixed di.depcheck: + +comment,,,,,,,module metadata - exported version +true,0,0,q,10h=type depcheck.version,1,,version is a string +true,0,0,q,0 Date: Tue, 21 Jul 2026 14:37:51 +0100 Subject: [PATCH 02/11] Fix version-comparison and dropped-warning logging bugs and add integration tests --- di/depcheck/depcheck.md | 229 ++++++++++++++++++++----------- di/depcheck/depcheck.q | 30 +++- di/depcheck/test.csv | 32 ++++- di/depcheck/test_integration.csv | 34 +++++ 4 files changed, 238 insertions(+), 87 deletions(-) create mode 100644 di/depcheck/test_integration.csv diff --git a/di/depcheck/depcheck.md b/di/depcheck/depcheck.md index 5b0d3652..42ac27aa 100644 --- a/di/depcheck/depcheck.md +++ b/di/depcheck/depcheck.md @@ -8,53 +8,40 @@ relies on. --- -## How it works - -di.depcheck is a **post-load audit, not a pre-load gate**. It is intended to run once, after a host process -(`di.torq`) has already `use`d every module it needs for that process. It never calls `use` on anything it -checks — every check reads other modules' state purely by introspecting the session namespace the kdb-x `use` -loader already populates: - -- Every loaded module lands under `` `.m.di `` keyed by a short form (`di.timer` → `` `0timer ``, giving - `` `.m.di.0timer ``), and that module's full `export` dict is itself readable there - (`` .m.di.0timer.export ``). This is how di.depcheck reads another module's exports and `version` without - importing it. -- "Not found" in a failure report means a *declared* dependency was never `use`d into this session by whatever - loaded di.depcheck's caller — it is not a QPATH filesystem scan. - -### Checks performed - -1. **Presence & minimum version** — for every currently-loaded module that ships a `deps.q` - (`` deps:`di.tplog`di.pubsub!("0.2.0";"0.3.0") ``, symbol-keyed by dependency module name, string-valued by - minimum version), each declared dependency is checked: is it loaded, and if so does its exported `version` - satisfy the declared minimum (real numeric `major.minor.patch` comparison, not a digit-walk)? A module with no - `deps.q` is skipped, not treated as an error — most modules don't ship one yet. -2. **Core dependency contracts** — for whichever of `di.log`, `di.timer`, or `di.handlers` are loaded, their - export dict is checked for the required keys of the contract they provide (Logging: `info`/`warn`/`error`; - Timer: `addjob`/`deletejobs`/`enablejobs`/`disablejobs`/`getactivejobs`/`cp`; Handlers: - `register`/`remove`/`list`). This is a fixed, named set — not a generic self-declaration registry, since no - such mechanism exists elsewhere in this codebase. -3. **`.z.ts` ownership** — warns if `.z.ts` is bound to something while di.timer either isn't loaded or doesn't - look initialised (its `enabled` state is used as a proxy for "di.timer's `init` ran and bound `.z.ts` - itself"). This does **not** prove nothing has overwritten `.z.ts` afterwards — an accepted limitation of a - warning-level check. Scope is deliberately limited to `.z.ts` only. -4. **kdb-x engine version** (optional, and currently unwired) — compares the running engine's `.z.K` (e.g. `5f`) - against an optional `` `minkdbxversion `` passed alongside `log` on the same `deps` dict. **This is the shape - of the check, not a live check yet**: if `` `minkdbxversion `` is absent from `deps` — which is every real - call site today, since no caller exists that supplies one — `kdbxcheck` returns immediately without looking - at `.z.K` at all. There is currently no config path that could supply a real minimum: di.config's cascade - isn't wired to di.torq yet (di.config's own docs list that as future work), and `deps.q`'s format is - per-dependency *module* versions, not an engine-level minimum. The comparison itself is unit-tested and - correct in isolation (see `test.csv`) — what's missing is a caller that has an opinion on what the minimum - should be. Uses `.z.K`, not `.z.v` — `.z.v`'s value on the box this module was developed on - (`"5.0.20260122"`) is a build-stamp string, not a confirmed match for the kdb-x product version, and - di.k4unit already has a working precedent for exactly this problem (`minver<=.z.K` gates which tests run). - -Presence and version failures, and core-contract failures, are **fail-fast**: `init` logs a single multi-line -report at `error` and then signals, so a caller sees a blocking error. The `.z.ts` and kdb-x-version checks are -**warning-only**: logged at `warn`, never signalled. - -### Report format +## Features + +- A **post-load audit, not a pre-load gate** — runs once, after a host process (`di.torq`) has already `use`d + every module it needs. It never calls `use` on anything it checks: every check reads other modules' state + purely by introspecting the session namespace the kdb-x `use` loader already populates. Every loaded module + lands under `` `.m.di `` keyed by a short form (`di.timer` → `` `0timer ``, giving `` `.m.di.0timer ``), and + that module's full `export` dict is itself readable there (`` .m.di.0timer.export ``) — this is how + di.depcheck reads another module's exports and `version` without importing it. "Not found" in a failure + report means a *declared* dependency was never `use`d into this session — it is not a QPATH filesystem scan. +- **Presence & minimum version** — for every currently-loaded module that ships a `deps.q` + (`` deps:`di.tplog`di.pubsub!("0.2.0";"0.3.0") ``, symbol-keyed by dependency module name, string-valued by + minimum version), each declared dependency is checked: is it loaded, and if so does its exported `version` + satisfy the declared minimum (real numeric `major.minor.patch` comparison, not a digit-walk)? A module with no + `deps.q` is skipped, not treated as an error — most modules don't ship one yet. +- **Core dependency contracts** — for whichever of `di.log`, `di.timer`, or `di.handlers` are loaded, their + export dict is checked for the required keys of the contract they provide (Logging: `info`/`warn`/`error`; + Timer: `addjob`/`deletejobs`/`enablejobs`/`disablejobs`/`getactivejobs`/`cp`; Handlers: + `register`/`remove`/`list`). This is a fixed, named set — not a generic self-declaration registry, since no + such mechanism exists elsewhere in this codebase. +- **`.z.ts` ownership** — warns if `.z.ts` is bound to something while di.timer either isn't loaded or doesn't + look initialised (its `enabled` state is used as a proxy for "di.timer's `init` ran and bound `.z.ts` itself"). + This does **not** prove nothing has overwritten `.z.ts` afterwards — an accepted limitation of a warning-level + check. Scope is deliberately limited to `.z.ts` only. +- **kdb-x engine version** (optional, and currently unwired) — compares the running engine's `.z.K` (e.g. `5f`) + against an optional `` `minkdbxversion `` passed alongside `log` on the same `deps` dict. This is the shape of + the check, not a live check yet: no caller exists today that supplies `` `minkdbxversion ``, so it is a no-op + in every real invocation. Uses `.z.K`, not `.z.v` — `.z.v`'s build-stamp string isn't a confirmed match for the + kdb-x product version, and di.k4unit already has a working precedent for exactly this problem + (`minver<=.z.K` gates which tests run). +- Presence and version failures, and core-contract failures, are **fail-fast**: `init` logs a single multi-line + report at `error` and then signals, so a caller sees a blocking error. The `.z.ts` and kdb-x-version checks are + **warning-only**: logged at `warn`, never signalled. + +Report format: ``` DEPENDENCY CHECK FAILED: @@ -84,19 +71,13 @@ found" fits): `di.timer requires minimum version 1.0.0, but di.timer exports no ## Initialisation -```q -depcheck:use`di.depcheck +`init[deps]` takes a single dictionary combining the required `log` dependency with an optional kdb-x minimum +version override. -logdep:`info`warn`error!( - {[c;m] -1 string[c],": INFO ",m;}; - {[c;m] -1 string[c],": WARN ",m;}; - {[c;m] -2 string[c],": ERROR ",m;}); - -depcheck.init[enlist[`log]!enlist logdep] - -/ with an optional kdb-x minimum version: -depcheck.init[`log`minkdbxversion!(logdep;5.0)] -``` +| Key | Required | Description | +|---|---|---| +| `` `log `` | yes | Log dep — `info`/`warn`/`error`, each binary `{[c;m]}` | +| `` `minkdbxversion `` | no | Minimum kdb-x engine version, compared against `.z.K`. Default: unchecked | `init` must be called after every module the host process needs has already been `use`d (this is what it audits) — typically the very last thing `di.torq` does during startup. @@ -121,27 +102,73 @@ depcheck.version / "0.1.0" --- +## Usage Example + +```q +/ log dep must already match the binary {[c;m]} contract - write your own, or use di.log: +/ logging:use`di.log +/ depcheck.init[enlist[`log]!enlist logging.logdict] +logdep:`info`warn`error!({[c;m]};{[c;m]};{[c;m]}) + +/ typical usage: di.torq loads every module the process needs first... +timer:use`di.timer +handlers:use`di.handlers + +/ ...then depcheck audits the fully-loaded session, last +depcheck:use`di.depcheck +depcheck.init[enlist[`log]!enlist logdep] + +/ with an optional kdb-x minimum version: +depcheck.init[`log`minkdbxversion!(logdep;5.0)] +``` + +--- + ## Running Tests +**Unit suite** (`test.csv`) — introspects session state and fabricated namespace fixtures, no child processes. +`moduletest` loads and runs it: + ```q k4unit:use`di.k4unit k4unit.moduletest`di.depcheck ``` -Tests drive real, already-shipped modules rather than synthetic fixtures, so the assertions track the codebase's -actual current state: - -- **di.timer** is merged to `main`, so it is loaded unconditionally and always exercised for real (missing - `version`; `cp` defined but not exported — both real, live gaps). -- **di.kafka** (PR #112) and **di.handlers** (PR #114, the positive control — the one module that does export - `version`) are still unmerged, so this suite must not assume they're present. Their loads are protected - (`.dc.havekafka`/`.dc.havehandlers`), and every assertion depending on them is written as - `(not haveflag) or realassertion` — a real check when the module is loaded, a no-op pass when it isn't. - Verified directly against a bare `main`-only checkout (no other branches merged in): all 34 tests pass there - too, exercising real di.timer coverage and no-op'ing the di.kafka/di.handlers-dependent assertions rather than - crashing on a missing module. -- **di.log** (PR #90, DI-Dexter fork) follows the same graceful pattern — checked for real when resolvable on - QPATH, a no-op pass otherwise. +Run in a fresh q session. `moduletest` doesn't reset its internal test table between calls, so calling it twice +in the same session duplicates this file's rows and produces spurious failures in the fixture that creates and +removes a scratch `deps.q` on disk. + +Tests drive real, already-shipped modules rather than synthetic fixtures, so the assertions track the +codebase's actual current state. **di.timer** is merged to `main`, so it is loaded unconditionally and always +exercised for real (missing `version`; `cp` defined but not exported — both real, live gaps). **di.kafka**, +**di.handlers** (the positive control — the one module that does export `version`), and **di.log** are all +still on open, unmerged PRs, so their loads are protected and every assertion depending on them gracefully +no-ops when the module isn't resolvable on QPATH — verified directly against a bare `main`-only checkout (no +other branches merged in), where every test still passes, exercising real di.timer coverage and no-op'ing the +rest rather than crashing on a missing module. `checkmoduledeps`/`finddepsq`/`readdepsq` are also exercised +directly against real on-disk `deps.q` files, not only indirectly via `checkdeps[]` — including di.depcheck's +own real shipped (empty) `deps.q`, di.timer's real absence of one, and a genuinely malformed one written to a +scratch module directory derived from `getenv\`QPATH\`` at runtime (no hardcoded absolute path, no new committed +fixture file, cleaned up afterwards). + +**Integration suite** (`test_integration.csv`) — spins up a real, separate child kdb-x process (needed because, +unlike a plain q peer, the child must itself be kdb-x-capable to `use\`di.timer\`/use\`di.depcheck\``) and proves +two things the unit suite structurally cannot: that a genuine zero-failure success path completes cleanly end +to end in a real process (unreachable in `test.csv`, since di.timer is always loaded there as a live negative +control and permanently fails its own contract), and that a real signalled failure actually terminates a real +host process with a real non-zero exit code and the real report visible in real captured output, not a mock +logger. The child binary is resolved via `` /proc//exe `` of the *currently running* process rather than +`QHOME` — on this dev machine `QHOME` resolves to a pre-kdb-x q build with no `use` keyword at all, so guessing +from `QHOME` the way `di.handlers`' integration test does (which only ever needs a plain q peer, not a +kdb-x-capable one) would silently pick the wrong binary. Skips cleanly if the resolved binary doesn't exist. +`moduletest` only ever loads `test.csv`, so load and run this suite directly, in a fresh session: + +```q +k4unit:use`di.k4unit +.m.di.0k4unit.KUltf .Q.dd[hsym`$.Q.m.mp`di.depcheck;`test_integration.csv] +.m.di.0k4unit.KUrt[] +k4unit.getresults[] / one row per assertion; ok=1 is a pass +``` --- @@ -154,8 +181,8 @@ actual current state: - di.compression imports `kx.log` directly rather than following the binary `{[c;m]}` three-flat-var convention this module (and di.handlers/di.kafka/di.config/di.eodtime) uses — a pre-`consistency.md` outlier worth a separate cleanup pass. - - The `.z.ts` ownership check is a best-effort proxy (see above) — it cannot detect something rebinding - `.z.ts` after di.timer's `init` runs. + - The `.z.ts` ownership check is a best-effort proxy (see Features above) — it cannot detect something + rebinding `.z.ts` after di.timer's `init` runs. - The 0.x.y semver tension: a passing `>=` check during 0.x.y development does not guarantee contract compatibility, since every module in this workstream is currently pre-1.0. Implemented as literal `>=` anyway, per the plan. @@ -175,11 +202,15 @@ actual current state: unsatisfied one (by another) correctly produced exactly one failure line, for the unsatisfied case only. - A malformed `deps.q` — present but not a dict (wrong type; a plausible authoring mistake) — is reported as its own failure line (`" deps.q is malformed - expected a dict, got type "`) rather than crashing. - Not committed as an automated test, for the same real-file-on-disk reason as the point above; manually - verified: a fixture with `deps:"a string"` produced exactly that report line and did not abort the walk. - -- **Two real bugs found and fixed**, both by directly constructing and running the edge case, not by reading the - code — both looked entirely reasonable on the page: + This is now committed as an automated test (a real, malformed `deps.q` written to a scratch module directory + at runtime, see "Running Tests" above) — closing what was previously a manually-verified-only gap. + - This kdb-x build's `like` throws `` 'nyi `` on any pattern with more than one `*`-delimited literal segment + (e.g. `` "*a*b*" ``), regardless of whether the string being matched contains a newline. Every `like` + assertion in `test.csv` uses a single literal segment (`` "*single clause*" ``) for this reason — worth + knowing before adding a new one. + +- **Five real bugs found and fixed**, all by directly constructing and running the edge case, not by reading the + code — every one looked entirely reasonable on the page: 1. **`checkdepversion`'s eager `or`.** Originally combined `(xp~(::)) or not \`version in key xp` as a single condition. q's `or`/`and` are eager vector operators, not short-circuiting, so `key xp` was evaluated even when `xp` was already known to be `(::)`, throwing `'type` (`key` doesn't accept a generic null). Fixed to @@ -209,3 +240,41 @@ actual current state: stop assuming di.*. `kx.log` itself isn't committed anywhere in this repo (confirmed via `git ls-tree` across every branch — it's only vendored locally on this dev machine), so the regression test fabricates a non-di.* module via direct namespace manipulation (`` `.m.zz.0widget ``) rather than depending on it. + 4. **`parsesemver` silently mis-handled any non-numeric version component**, found during a later adversarial + re-pass over the shipped code, again by direct testing rather than reading. It only guarded the wrong + *count* of dot-separated parts; an individual part that partially parsed (any leading digits followed by + non-numeric text, e.g. a pre-release tag `"3-rc1"`) silently became `0Ni` (null) at just that position, + rather than failing the whole version. Since q's integer null sorts as the lowest possible value, this + produced two confirmed wrong answers, not merely an "unsupported" gap: `vergte["1.2.3-rc1";"1.2.0"]` + returned `0b` — a real, newer version reported as failing an *older* minimum, solely because its unparsed + suffix nulled out the patch component that would otherwise have decided the comparison; and + `vergte["1.2.3";"abc"]` returned `1b` — a typo'd `deps.q` minimum silently treated as "no real requirement + at all," with nothing surfaced anywhere. Fixed by collapsing *any* unparseable component to the same + `(0Ni;0Ni;0Ni)` "malformed" sentinel used for the wrong-part-count case (`ismalformed`), and having + `checkfoundversion` check for it explicitly on both sides (declared minimum and exported found-version) + before ever handing either to the numeric comparison — surfacing a distinct, correctly-worded failure line + instead of a silent, misleading pass or fail. See `checkonedep`/`checkfoundversion` and the malformed-semver + tests in `test.csv`. + 5. **Warnings were silently dropped from the log whenever a failure also occurred.** `init` computed `warnings` + up front but only logged them in a branch positioned *after* the failures check, and the failures check + signals (`` ' ``) on any failure — so if a real warning (e.g. `.z.ts` misuse, a stale kdb-x version) + happened to coincide with a real failure in the same `init` call, execution never reached the + `.z.m.logwarn` line at all: the warning was computed, then silently discarded, with nothing about it ever + logged anywhere. Found by directly asking "is this thoroughly tested and does it have thorough logging" and + reading the actual `init` body, not by running anything new — the bug was visible directly in the code once + someone looked at execution order rather than just at each `if[]` block in isolation. Fixed by moving the + warnings-logging line to run unconditionally before the failures check, so a coinciding warning is always + logged regardless of whether `init` also throws. Regression test: `init` is now called end-to-end with a + real failure (di.timer's contract gap) and a real, simultaneously-computed warning + (`` `minkdbxversion `` set above `.z.K`), and the captured log table is asserted to contain *both* an + `error` row and a `warn` row with the real kdbxcheck text — previously that `warn` row would never have + appeared. + +- **Two further gaps noted but not changed** (defensive-completeness items, not bugs — no real file or call site + in this codebase currently triggers either): + - `readdepsq` assumes a `deps.q` is a *single pure* `deps:...` assignment, per every real precedent seen so + far. This is unenforced: a `deps.q` that also defines stray globals beyond `deps` would leak them into the + root namespace permanently, since only `deps` itself is ever explicitly deleted after being captured. + - `init`'s validation of its own injected `log` dependency checks key presence only (`info`/`warn`/`error` + present), not function arity — the same shape-only limitation already called out above for how + `checkcontracts` audits *other* modules' contracts applies equally to di.depcheck validating its own. diff --git a/di/depcheck/depcheck.q b/di/depcheck/depcheck.q index 0c12f86a..c8886f36 100644 --- a/di/depcheck/depcheck.q +++ b/di/depcheck/depcheck.q @@ -77,10 +77,21 @@ readdepsq:{[modname] / ============================================================ parsesemver:{[v] - / parse a "major.minor.patch" string into a 3-long int vector; any parse failure yields nulls at that position + / parse a "major.minor.patch" string into a 3-long int vector; a version that does not parse cleanly as three + / all-numeric parts collapses to a single (0Ni;0Ni;0Ni) "malformed" sentinel, checked via ismalformed, rather + / than left to silently participate in numeric comparison one component at a time - a lone bad component (e.g. + / a pre-release tag like "1.2.3-rc1") used to null out only itself, which could silently make a real, newer + / version compare as lower than it should, or make a typo'd deps.q minver like "abc" silently compare as no + / real minimum at all. Caught by direct testing, not by reading the code - see depcheck.md parts:"." vs v; if[not 3=count parts;:3#0Ni]; - {@[{"I"$x};x;0Ni]} each parts + nums:{@[{"I"$x};x;0Ni]} each parts; + $[any null nums;3#0Ni;nums] + }; + +ismalformed:{[v] + / true if v does not parse as a clean major.minor.patch triple - see parsesemver + (3#0Ni)~parsesemver v }; vercmp:{[a;b] @@ -107,6 +118,12 @@ vergte:{[a;b] checkfoundversion:{[dep;minver;foundver] / dep is loaded and exports a version - compares it against the declared minimum + / a malformed minver (a deps.q authoring typo) or malformed foundver (a module exporting a non-semver string) + / is reported explicitly here rather than silently entering vergte's numeric comparison - see parsesemver + if[ismalformed minver; + :enlist string[dep]," has a declared minimum version of ",minver,", which is not a valid major.minor.patch version"]; + if[ismalformed foundver; + :enlist string[dep]," exports version ",foundver,", which is not a valid major.minor.patch version"]; $[vergte[foundver;minver];();enlist string[dep]," requires minimum version ",minver,", found ",foundver] }; @@ -244,13 +261,16 @@ init:{[deps] failures:checkdeps[],checkcontracts[]; warnings:ztscheck[],kdbxcheck[deps]; + / warnings are logged unconditionally, before the failures check below - not gated behind "no failures", so a + / real warning is never silently dropped just because a failure also happened to signal in the same call. + / caught by direct testing, not by reading the code - see depcheck.md + if[count warnings; + .z.m.logwarn[`depcheck;buildreport["WARNING";warnings]]]; + if[count failures; report:buildreport["DEPENDENCY CHECK FAILED";failures]; .z.m.logerr[`depcheck;report]; '"di.depcheck: ",report]; - if[count warnings; - .z.m.logwarn[`depcheck;buildreport["WARNING";warnings]]]; - .z.m.loginfo[`depcheck;"dependency check complete: ",(string count failures)," failure(s), ",(string count warnings)," warning(s)"]; }; diff --git a/di/depcheck/test.csv b/di/depcheck/test.csv index 073c5b0a..82e59339 100644 --- a/di/depcheck/test.csv +++ b/di/depcheck/test.csv @@ -61,6 +61,28 @@ true,0,0,q,0=count .m.di.0depcheck.checkonedep[`zz.widget;"1.0.0"],1,,a loaded n true,0,0,q,".m.di.0depcheck.checkonedep[`zz.widget;""3.0.0""]~enlist ""zz.widget requires minimum version 3.0.0, found 2.0.0""",1,,a loaded non-di.* dependency is version-checked for real not falsely reported not-found true,0,0,q,".m.di.0depcheck.checkonedep[`zz.doesnotexist;""1.0.0""]~enlist ""zz.doesnotexist requires minimum version 1.0.0, not found""",1,,a genuinely absent non-di.* vendor namespace still safely reports not found, no crash +comment,,,,,,,malformed semver handling - caught during manual adversarial re-scrutiny (not by reading the code): a +comment,,,,,,,pre-release-tagged found version used to silently null out its patch component and compare as lower than +comment,,,,,,,it really is; a typo'd deps.q minver used to silently compare as no real minimum at all. See parsesemver. +before,0,0,q,.m.zz.0badfound.export:enlist[`version]!enlist "1.2.3-rc1",1,,fabricate a loaded module exporting a version with a pre-release tag +true,0,0,q,"(3#0Ni)~.m.di.0depcheck.parsesemver[""1.2.3-rc1""]",1,,a version with a non-numeric component parses to the malformed sentinel not a partial triple +true,0,0,q,".m.di.0depcheck.ismalformed[""1.2.3-rc1""]",1,,ismalformed recognises the pre-release-tagged string +true,0,0,q,not .m.di.0depcheck.ismalformed["1.2.3"],1,,ismalformed does not flag a well-formed version +true,0,0,q,".m.di.0depcheck.checkonedep[`zz.badfound;""1.2.0""]~enlist ""zz.badfound exports version 1.2.3-rc1, which is not a valid major.minor.patch version""",1,,a pre-release-tagged found version is reported as invalid rather than silently compared as lower +true,0,0,q,".m.di.0depcheck.checkonedep[`zz.widget;""abc""]~enlist ""zz.widget has a declared minimum version of abc, which is not a valid major.minor.patch version""",1,,a malformed deps.q minver is reported as invalid rather than silently treated as no real minimum + +comment,,,,,,,buildreport - internal report-formatting helper, exercised directly not just indirectly via init's report +true,0,0,q,.m.di.0depcheck.buildreport["X";("a";"b")]~"X:\n a\n b",1,,buildreport formats a header line followed by bulleted indented entries + +comment,,,,,,,checkmoduledeps - internal per-module deps.q auditor, exercised directly against real on-disk deps.q files +comment,,,,,,,rather than only indirectly via checkdeps[] - closes a real gap: finddepsq/readdepsq's actual QPATH file +comment,,,,,,,discovery and load mechanics had never been exercised by any committed test, only by manual scratch testing +true,0,0,q,()~.m.di.0depcheck.checkmoduledeps[`0timer],1,,di.timer genuinely ships no deps.q - the real not-found branch against a real module, no crash +true,0,0,q,()~.m.di.0depcheck.checkmoduledeps[`0depcheck],1,,di.depcheck's own real shipped empty deps.q is found and loaded via the real QPATH file-lookup and system load - proves the real file-discovery-and-load path actually works, not just the in-memory branches +run,0,0,q,".dc.qroot:first "":"" vs getenv`QPATH; system ""mkdir -p "",.dc.qroot,""/di/zzmalformeddep""; (hsym `$.dc.qroot,""/di/zzmalformeddep/deps.q"") 0: enlist ""deps:42""",1,,"create a real on-disk malformed deps.q (deps defined as an int, not a dict) under a scratch module directory derived from QPATH at runtime - no hardcoded absolute path, no new committed fixture file. deliberately a run action not before: before actions across every accumulated load of this file in one k4unit session run as a single upfront batch, while run/true/fail replay per load in file order - a before here would create the fixture once but the paired cleanup below (a run action) would fire once per load, deleting it out from under a second load's assertion" +true,0,0,q,any .m.di.0depcheck.checkmoduledeps[`0zzmalformeddep] like "*deps.q is malformed*",1,,a genuinely malformed deps.q found on disk is reported as its own failure line rather than crashing checkdeps - previously only manually verified never committed +run,0,0,q,"system ""rm -rf "",.dc.qroot,""/di/zzmalformeddep""",1,,remove the scratch fixture directory + comment,,,,,,,checkcontracts - core dependency contract shape, exercised against live loaded modules true,0,0,q,any .m.di.0depcheck.checkcontracts[] like "di.timer is missing required contract key(s): cp",1,,real di.timer fails the Timer contract - cp is defined but not exported - always present on main true,0,0,q,not any .m.di.0depcheck.checkcontracts[] like "di.handlers is missing*",1,,real di.handlers satisfies the Handlers contract in full when present; vacuously true (no entry to match) when absent @@ -72,9 +94,15 @@ run,0,0,q,.z.ts:{[x]},1,,directly assign .z.ts outside di.timer for the negative true,0,0,q,.m.di.0depcheck.ztscheck[]~enlist ".z.ts has been directly assigned outside di.timer. This may cause timer conflicts.",1,,warns when .z.ts is bound outside di.timer run,0,0,q,system"x .z.ts",1,,restore .z.ts to the kdb-x built-in default after the test -comment,,,,,,,end-to-end init - real loaded modules (di.timer missing cp) drive a real failing report +comment,,,,,,,end-to-end init - real loaded modules (di.timer missing cp) drive a real failing report. minkdbxversion +comment,,,,,,,is also set above .z.K so a real warning is computed alongside the real failure - regression test for a +comment,,,,,,,logging bug found during adversarial re-scrutiny: warnings were only ever logged when init did NOT also +comment,,,,,,,fail, since the failures branch signalled (threw) before the warnings-logging line was reached, silently +comment,,,,,,,dropping any real warning that happened to coincide with a real failure. Fixed by logging warnings first. run,0,0,q,.dc.captbl:0#.dc.captbl,1,,reset log capture -run,0,0,q,.dc.initerr:@[{depcheck.init[enlist[`log]!enlist caplog]};(::);{x}],1,,init throws end-to-end because real di.timer fails the Timer contract +run,0,0,q,.dc.initerr:@[{depcheck.init[`log`minkdbxversion!(caplog;999f)]};(::);{x}],1,,init throws end-to-end while a real kdbxcheck warning is also computed true,0,0,q,.dc.initerr like "di.depcheck: DEPENDENCY CHECK FAILED:*",1,,the signalled error is the di.depcheck-prefixed report block true,0,0,q,.dc.initerr like "*di.timer is missing required contract key(s): cp*",1,,the report names the real di.timer contract gap true,0,0,q,any `error = exec lvl from .dc.captbl,1,,the failing report was also logged at error before being signalled +true,0,0,q,any `warn = exec lvl from .dc.captbl,1,,the coinciding warning was also logged at warn - not silently dropped - regression test for the logging fix +true,0,0,q,(exec first msg from .dc.captbl where lvl=`warn) like "*is below the configured minimum*",1,,the logged warning is the real kdbxcheck text, not an empty or wrong message diff --git a/di/depcheck/test_integration.csv b/di/depcheck/test_integration.csv new file mode 100644 index 00000000..4c9a63f0 --- /dev/null +++ b/di/depcheck/test_integration.csv @@ -0,0 +1,34 @@ +action,ms,bytes,lang,code,repeat,minver,comment +comment,,,,,,,"integration test - spins up a real, separate child kdb-x process, loads real modules via QPATH," +comment,,,,,,,"and calls depcheck.init[] for real, proving two things test.csv structurally cannot: that a real" +comment,,,,,,,"signalled failure actually terminates a real host process (non-zero exit, real error on stderr)," +comment,,,,,,,and that a genuine zero-failure success path completes cleanly end to end - unreachable in test.csv +comment,,,,,,,"since di.timer, always loaded there as a live negative control, permanently fails its own contract" +comment,,,,,,,"the child must be kdb-x-capable (needs use/QPATH), unlike a plain q peer, so this deliberately reuses" +comment,,,,,,,the currently-running process's own binary via /proc//exe rather than guessing from QHOME - +comment,,,,,,,QHOME on this dev machine resolves to a pre-kdb-x q build with no use keyword at all +comment,,,,,,,note k4unit runs every before row first then the asserts - so results are captured in befores and checked in trues +,,,,,,, +before,0,0,q,"qbin:first system ""readlink -f /proc/"",(string .z.i),""/exe""",1,,"resolve the currently-running process's own binary - guaranteed kdb-x-capable, no QHOME guess, no hardcoded path" +before,0,0,q,.it.haveqbin:not ()~key hsym `$qbin,1,,confirm the resolved binary actually exists on disk +before,0,0,q,if[not .it.haveqbin;exit 0],1,,skip this entire suite cleanly if the binary could not be resolved +before,0,0,q,".it.dir:(first "":"" vs getenv`QPATH),""/di/zzintegration""",1,,scratch directory derived from QPATH at runtime - no hardcoded path +before,0,0,q,"system ""mkdir -p "",.it.dir",1,,create the scratch directory +before,0,0,q,"(hsym `$.it.dir,""/success.q"") 0: enlist ""depcheck:use`di.depcheck; logdep:`info`warn`error!({[c;m] -1 m};{[c;m] -1 m};{[c;m] -2 m}); depcheck.init[enlist[`log]!enlist logdep]; exit 0;""",1,,"write a real child script loading ONLY di.depcheck - no di.timer, so no real contract gap exists - a genuine zero-failure success case" +before,0,0,q,"(hsym `$.it.dir,""/failure.q"") 0: enlist ""timer:use`di.timer; depcheck:use`di.depcheck; logdep:`info`warn`error!({[c;m] -1 m};{[c;m] -1 m};{[c;m] -2 m}); depcheck.init[enlist[`log]!enlist logdep]; exit 0;""",1,,"write a real child script that also loads di.timer - its real, permanent Timer-contract gap (missing cp) drives a real failure" +before,0,0,q,".it.successresult:system qbin,"" "",.it.dir,""/success.q -q < /dev/null > "",.it.dir,""/success.out 2>&1; echo EXITCODE:$?""",1,,"run the success child to completion synchronously, capturing combined stdout+stderr and its real exit code" +before,0,0,q,".it.successexit:""I""$9_last .it.successresult",1,,parse the real exit code +before,0,0,q,".it.successout:sv[""\n"";read0 hsym `$.it.dir,""/success.out""]",1,,read the child's real captured output +before,0,0,q,".it.failureresult:system qbin,"" "",.it.dir,""/failure.q -q < /dev/null > "",.it.dir,""/failure.out 2>&1; echo EXITCODE:$?""",1,,"run the failure child to completion synchronously, capturing combined stdout+stderr and its real exit code" +before,0,0,q,".it.failureexit:""I""$9_last .it.failureresult",1,,parse the real exit code +before,0,0,q,".it.failureout:sv[""\n"";read0 hsym `$.it.dir,""/failure.out""]",1,,read the child's real captured output +before,0,0,q,"system ""rm -rf "",.it.dir",1,,remove the scratch fixture directory +,,,,,,, +comment,,,,,,,genuine zero-failure success path - unreachable in test.csv since di.timer always fails its own contract there +true,0,0,q,0=.it.successexit,1,,the child process loading only di.depcheck exited cleanly with a real 0 exit code +true,0,0,q,".it.successout like ""*dependency check complete: 0 failure(s), 0 warning(s)*""",1,,the real captured log shows the genuine zero-failure summary line - proves the true success path works end to end in a real process +,,,,,,, +comment,,,,,,,a real signalled failure actually terminates a real host process +true,0,0,q,0<>.it.failureexit,1,,the child process crashed with a real non-zero exit code when a real dependency check failed +true,0,0,q,".it.failureout like ""*DEPENDENCY CHECK FAILED*""",1,,the real crash output contains the real error-level report - not a captured mock +true,0,0,q,".it.failureout like ""*di.timer is missing required contract key(s): cp*""",1,,the real report names the real di.timer contract gap From bd066ae94dd9032d8cd4c8c8ccf766e26d80f47a Mon Sep 17 00:00:00 2001 From: alowrydi Date: Mon, 3 Aug 2026 12:09:37 +0100 Subject: [PATCH 03/11] Refactoring to fit di.toml standard from feature-toml --- di/depcheck/depcheck.md | 226 +++++++--------------------------------- di/depcheck/depcheck.q | 160 +++++++++++++++++++++++++--- di/depcheck/init.q | 2 +- di/depcheck/test.csv | 77 +++++++++++++- 4 files changed, 261 insertions(+), 204 deletions(-) diff --git a/di/depcheck/depcheck.md b/di/depcheck/depcheck.md index 42ac27aa..44c46011 100644 --- a/di/depcheck/depcheck.md +++ b/di/depcheck/depcheck.md @@ -1,47 +1,20 @@ # di.depcheck -Dependency, version, core-contract, and `.z.ts`-ownership auditing for kdb-x modules. It is the modernised -successor to legacy TorQ's `checkdependency`/`runchk`/`checkvers` (`torq.q`): a per-module `deps.q` replaces the -old CSV registry, real numeric semver replaces the 5-component digit-walk, and — new — it enforces the shape of -the shared "core dependency contracts" (Logging / Timer / Handlers) that the DI dependency-injection pattern -relies on. +Dependency, version, core-contract, and `.z.ts`-ownership auditing for kdb-x modules. Runs once at process startup — after every module has been loaded — and reports any declared dependency that is missing, out of date, or fails the shared core-dependency contracts. It is the modernised successor to legacy TorQ's `checkdependency`/`runchk`/`checkvers`: a per-module manifest replaces the old CSV registry, and real numeric semver replaces the 5-component digit-walk. --- ## Features -- A **post-load audit, not a pre-load gate** — runs once, after a host process (`di.torq`) has already `use`d - every module it needs. It never calls `use` on anything it checks: every check reads other modules' state - purely by introspecting the session namespace the kdb-x `use` loader already populates. Every loaded module - lands under `` `.m.di `` keyed by a short form (`di.timer` → `` `0timer ``, giving `` `.m.di.0timer ``), and - that module's full `export` dict is itself readable there (`` .m.di.0timer.export ``) — this is how - di.depcheck reads another module's exports and `version` without importing it. "Not found" in a failure - report means a *declared* dependency was never `use`d into this session — it is not a QPATH filesystem scan. -- **Presence & minimum version** — for every currently-loaded module that ships a `deps.q` - (`` deps:`di.tplog`di.pubsub!("0.2.0";"0.3.0") ``, symbol-keyed by dependency module name, string-valued by - minimum version), each declared dependency is checked: is it loaded, and if so does its exported `version` - satisfy the declared minimum (real numeric `major.minor.patch` comparison, not a digit-walk)? A module with no - `deps.q` is skipped, not treated as an error — most modules don't ship one yet. -- **Core dependency contracts** — for whichever of `di.log`, `di.timer`, or `di.handlers` are loaded, their - export dict is checked for the required keys of the contract they provide (Logging: `info`/`warn`/`error`; - Timer: `addjob`/`deletejobs`/`enablejobs`/`disablejobs`/`getactivejobs`/`cp`; Handlers: - `register`/`remove`/`list`). This is a fixed, named set — not a generic self-declaration registry, since no - such mechanism exists elsewhere in this codebase. -- **`.z.ts` ownership** — warns if `.z.ts` is bound to something while di.timer either isn't loaded or doesn't - look initialised (its `enabled` state is used as a proxy for "di.timer's `init` ran and bound `.z.ts` itself"). - This does **not** prove nothing has overwritten `.z.ts` afterwards — an accepted limitation of a warning-level - check. Scope is deliberately limited to `.z.ts` only. -- **kdb-x engine version** (optional, and currently unwired) — compares the running engine's `.z.K` (e.g. `5f`) - against an optional `` `minkdbxversion `` passed alongside `log` on the same `deps` dict. This is the shape of - the check, not a live check yet: no caller exists today that supplies `` `minkdbxversion ``, so it is a no-op - in every real invocation. Uses `.z.K`, not `.z.v` — `.z.v`'s build-stamp string isn't a confirmed match for the - kdb-x product version, and di.k4unit already has a working precedent for exactly this problem - (`minver<=.z.K` gates which tests run). -- Presence and version failures, and core-contract failures, are **fail-fast**: `init` logs a single multi-line - report at `error` and then signals, so a caller sees a blocking error. The `.z.ts` and kdb-x-version checks are - **warning-only**: logged at `warn`, never signalled. - -Report format: +- **A post-load audit, not a pre-load gate** — it runs after the host process (`di.torq`) has already `use`d every module it needs, and never calls `use` on anything it checks. Each module's state is read by introspecting the session namespace the kdb-x loader populates: every loaded module lands under `` `.m.di `` keyed by a short form (`di.timer` → `` `.m.di.0timer ``), and that module's `export` dict is readable there directly. "Not found" in a report means a *declared* dependency was never loaded — it is not a filesystem scan. +- **Presence & minimum version** — for each loaded module that ships a manifest (symbol-keyed by dependency name, string-valued by minimum version), every declared dependency is checked: is it loaded, and does its exported `version` satisfy the declared minimum (real numeric `major.minor.patch` comparison)? A module with no manifest is skipped, not failed. +- **Dual-format manifests (`deps.q` and/or `deps.toml`)** — a module's dependencies are read from **both** a `deps.q` (a q dict literal) and a `deps.toml` (a `[dependencies]` section), wherever each exists, merged with **`.toml` winning on a clash** — mirroring di.config's `parsetier` so both formats can coexist mid-migration. `di.toml` is loaded **lazily and only when a `deps.toml` actually exists**; a module with only `deps.q` never triggers it. +- **Transitive manifest-graph walk** — beyond each loaded module's direct deps, `checkgraph` walks the graph **on disk** (reading each peer's own manifest whether or not it is loaded), cycle-guarded via a visited set, and reports a **presence** failure for any dependency reached at depth ≥ 2 that resolves nowhere on QPATH. It loads no module code. Transitive *version* checking is deferred (see Notes); direct (depth-1) deps keep the stronger presence=*loaded*/version=*exported* check and are excluded from the walk so the two never double-report. +- **Core dependency contracts** — for whichever of `di.log`, `di.timer`, or `di.handlers` are loaded, the export dict is checked for the required keys of the contract it provides (Logging: `info`/`warn`/`error`; Timer: `addjob`/`deletejobs`/`enablejobs`/`disablejobs`/`getactivejobs`/`cp`; Handlers: `register`/`remove`/`list`). The single-contract check is also exported directly as `checkcontract[provider;requiredkeys]` for a contract this module doesn't know by name. +- **`.z.ts` ownership** — warns if `.z.ts` is bound while di.timer is absent or uninitialised (its `enabled` flag is the proxy for "di.timer's `init` ran and bound `.z.ts`"). Warning-level only, and cannot detect a later rebind — an accepted limitation. +- **kdb-x engine version** (optional) — compares the running engine's `.z.K` against an optional `` `minkdbxversion `` passed alongside `log` on the same `deps` dict. Warning-level; no caller supplies a minimum today, so it is a no-op in every real invocation. + +Presence, version, and core-contract failures are **fail-fast** — `init` logs a single multi-line report at `error`, then signals, so the caller sees a blocking error. The `.z.ts` and kdb-x-version checks are **warning-only** — logged at `warn`, never signalled. ``` DEPENDENCY CHECK FAILED: @@ -52,8 +25,7 @@ WARNING: .z.ts has been directly assigned outside di.timer. This may cause timer conflicts. ``` -A loaded dependency that exports no `version` at all gets its own distinct line (neither "found" nor "not -found" fits): `di.timer requires minimum version 1.0.0, but di.timer exports no version`. +A loaded dependency that exports no `version` gets its own line — `di.timer requires minimum version 1.0.0, but di.timer exports no version` — and a transitive-only dependency missing from QPATH gets `di.zzc is required transitively by di.zzb but was not found on QPATH`. --- @@ -64,34 +36,35 @@ found" fits): `di.timer requires minimum version 1.0.0, but di.timer exports no | logger | `` `log `` | yes | dict with `info`, `warn`, and `error`, each binary `{[c;m]}` where `c` is a symbol context and `m` is a string (per `consistency.md`) | | kdb-x minimum version | `` `minkdbxversion `` | no | an optional float compared against `.z.K` | -**No hard dependencies** on other `di.*` modules — the module works standalone, and ships its own (empty) -`deps.q`, dogfooding the convention it introduces. +**No hard dependencies** on other `di.*` modules — the module works standalone, and ships its own (empty) `deps.q`, dogfooding the convention it introduces. + +**`di.toml` is a soft, lazy dependency** — it is not declared in `deps.q` and not loaded at import time. It is resolved once (cached) and called **only** when a module being audited ships a `deps.toml` file. A process whose modules use only `deps.q` never loads it, so it is not required to be on QPATH in that case. If a `deps.toml` *does* exist and `di.toml` is missing or fails to parse it, that is reported as one aggregated failure line (see Notes) rather than throwing. + +The `log` dependency must be passed to `init` inside the `deps` dict keyed on `` `log ``, and must already match the binary `{[c;m]}` contract — the module validates key presence but does not detect or adapt other shapes (e.g. a monadic `kx.log` instance). To use `di.log`, pass its `logdict`. --- ## Initialisation -`init[deps]` takes a single dictionary combining the required `log` dependency with an optional kdb-x minimum -version override. +`init[deps]` takes a single dictionary combining the required `log` dependency with an optional kdb-x minimum version. | Key | Required | Description | |---|---|---| | `` `log `` | yes | Log dep — `info`/`warn`/`error`, each binary `{[c;m]}` | | `` `minkdbxversion `` | no | Minimum kdb-x engine version, compared against `.z.K`. Default: unchecked | -`init` must be called after every module the host process needs has already been `use`d (this is what it -audits) — typically the very last thing `di.torq` does during startup. +`init` must be called **after** every module the host process needs has already been `use`d — typically the very last thing `di.torq` does during startup. It audits the fully-loaded session, so anything loaded later is not seen. --- ## Exported Functions ### `init[deps]` -Validate the required `log` dependency, then run every check against the current session and report. `deps` is -a dict with a `` `log `` key and an optional `` `minkdbxversion `` float. Throws on any presence, version, or -core-contract failure; logs (but does not throw on) `.z.ts`-ownership or kdb-x-version warnings. +Validate the required `log` dependency, then run every check against the current session and report. Throws on any presence, version, or core-contract failure; logs (but does not throw on) `.z.ts`-ownership or kdb-x-version warnings. ```q depcheck.init[enlist[`log]!enlist logdep] +/ with an optional minimum kdb-x version: +depcheck.init[`log`minkdbxversion!(logdep;5.0)] ``` ### `version` @@ -100,6 +73,14 @@ The module version string. depcheck.version / "0.1.0" ``` +### `checkcontract[provider;requiredkeys]` +Standalone version of the per-contract check that `checkcontracts[]` runs automatically for the three known core dependencies — checks whether `provider`'s export dict (if it's loaded) contains every key in `requiredkeys`. Never calls `use`, matching this module's introspection-only design. Returns `()` on a pass, or if `provider` isn't loaded at all; returns an enlisted failure line naming the missing keys otherwise. `requiredkeys` accepts either a symbol vector or a single bare symbol atom. +```q +depcheck.checkcontract[`di.timer;`addjob`deletejobs] / () - di.timer really exports both +depcheck.checkcontract[`di.timer;`addjob`cp] / enlist "di.timer is missing required contract key(s): cp" +depcheck.checkcontract[`di.timer;`cp] / bare atom works the same as enlist`cp +``` + --- ## Usage Example @@ -110,59 +91,27 @@ depcheck.version / "0.1.0" / depcheck.init[enlist[`log]!enlist logging.logdict] logdep:`info`warn`error!({[c;m]};{[c;m]};{[c;m]}) -/ typical usage: di.torq loads every module the process needs first... +/ di.torq loads every module the process needs first... timer:use`di.timer handlers:use`di.handlers / ...then depcheck audits the fully-loaded session, last depcheck:use`di.depcheck depcheck.init[enlist[`log]!enlist logdep] - -/ with an optional kdb-x minimum version: -depcheck.init[`log`minkdbxversion!(logdep;5.0)] ``` --- ## Running Tests -**Unit suite** (`test.csv`) — introspects session state and fabricated namespace fixtures, no child processes. -`moduletest` loads and runs it: - ```q k4unit:use`di.k4unit k4unit.moduletest`di.depcheck ``` -Run in a fresh q session. `moduletest` doesn't reset its internal test table between calls, so calling it twice -in the same session duplicates this file's rows and produces spurious failures in the fixture that creates and -removes a scratch `deps.q` on disk. - -Tests drive real, already-shipped modules rather than synthetic fixtures, so the assertions track the -codebase's actual current state. **di.timer** is merged to `main`, so it is loaded unconditionally and always -exercised for real (missing `version`; `cp` defined but not exported — both real, live gaps). **di.kafka**, -**di.handlers** (the positive control — the one module that does export `version`), and **di.log** are all -still on open, unmerged PRs, so their loads are protected and every assertion depending on them gracefully -no-ops when the module isn't resolvable on QPATH — verified directly against a bare `main`-only checkout (no -other branches merged in), where every test still passes, exercising real di.timer coverage and no-op'ing the -rest rather than crashing on a missing module. `checkmoduledeps`/`finddepsq`/`readdepsq` are also exercised -directly against real on-disk `deps.q` files, not only indirectly via `checkdeps[]` — including di.depcheck's -own real shipped (empty) `deps.q`, di.timer's real absence of one, and a genuinely malformed one written to a -scratch module directory derived from `getenv\`QPATH\`` at runtime (no hardcoded absolute path, no new committed -fixture file, cleaned up afterwards). - -**Integration suite** (`test_integration.csv`) — spins up a real, separate child kdb-x process (needed because, -unlike a plain q peer, the child must itself be kdb-x-capable to `use\`di.timer\`/use\`di.depcheck\``) and proves -two things the unit suite structurally cannot: that a genuine zero-failure success path completes cleanly end -to end in a real process (unreachable in `test.csv`, since di.timer is always loaded there as a live negative -control and permanently fails its own contract), and that a real signalled failure actually terminates a real -host process with a real non-zero exit code and the real report visible in real captured output, not a mock -logger. The child binary is resolved via `` /proc//exe `` of the *currently running* process rather than -`QHOME` — on this dev machine `QHOME` resolves to a pre-kdb-x q build with no `use` keyword at all, so guessing -from `QHOME` the way `di.handlers`' integration test does (which only ever needs a plain q peer, not a -kdb-x-capable one) would silently pick the wrong binary. Skips cleanly if the resolved binary doesn't exist. -`moduletest` only ever loads `test.csv`, so load and run this suite directly, in a fresh session: +Run in a fresh q session — `moduletest` doesn't reset its internal result table between calls, so a second call duplicates this file's rows. The unit suite drives real, already-shipped modules rather than synthetic fixtures: **di.timer** (merged to `main`) is always loaded and exercises real gaps (no `version`; `cp` defined but unexported), while **di.kafka**, **di.handlers** (the positive control — the one module that does export `version`), **di.log**, and **di.toml** (on its own `feature-toml` branch) are all on unmerged PRs, so every assertion depending on them gracefully no-ops when the module isn't resolvable on QPATH — the suite passes standalone on a bare `feature-depcheck` checkout, exercising the real path only when the module happens to be present. The manifest readers, dual-format merge, and transitive walk are exercised against scratch modules written under a QPATH root at runtime — the q-only / toml-only / both-formats-clashing / neither cases, a deliberately-broken `deps.toml`, and an A→B→C chain with a B→A cycle — all cleaned up afterwards, with no hardcoded paths and no committed fixtures. The di.toml-dependent assertions among these no-op when di.toml is absent. +The **integration suite** (`test_integration.csv`) spins up a real, separate child kdb-x process (the child must itself be kdb-x-capable to `use` the modules) to prove two things the unit suite structurally cannot: that a genuine zero-failure run completes cleanly end to end, and that a real signalled failure terminates a real host process with a non-zero exit code and the real report in captured output. `moduletest` only ever loads `test.csv`, so load and run this suite directly: ```q k4unit:use`di.k4unit .m.di.0k4unit.KUltf .Q.dd[hsym`$.Q.m.mp`di.depcheck;`test_integration.csv] @@ -174,107 +123,10 @@ k4unit.getresults[] / one row per assertion; ok=1 is a pass ## Notes -- **Known, flagged gaps**, not fixed here: - - Three real, shipped modules were found missing `version` during development: di.timer, di.kafka, and di.log - (the DI-Dexter fork PR #90 candidate). di.handlers is the only one that has it, and its own source comments - call that a placeholder pending this module's existence. - - di.compression imports `kx.log` directly rather than following the binary `{[c;m]}` three-flat-var - convention this module (and di.handlers/di.kafka/di.config/di.eodtime) uses — a pre-`consistency.md` outlier - worth a separate cleanup pass. - - The `.z.ts` ownership check is a best-effort proxy (see Features above) — it cannot detect something - rebinding `.z.ts` after di.timer's `init` runs. - - The 0.x.y semver tension: a passing `>=` check during 0.x.y development does not guarantee contract - compatibility, since every module in this workstream is currently pre-1.0. Implemented as literal `>=` - anyway, per the plan. - - The kdb-x-version check is **shape-only as of this PR, not operating against anything real yet**: it is - warning-only rather than fail-fast, and — more importantly — no caller exists today that passes - `` `minkdbxversion ``, so it is a no-op in every real invocation until di.torq (or some other caller) is - built and threads a real minimum through. Do not describe this PR as "implements the kdb-x version check" - to reviewers — it implements the comparison, unit-tested in isolation, with no live minimum source wired up. - - `test.csv` assumes di.timer is present (true of every real checkout of this repo, since it's merged to - `main`) — verified against a bare `main`-only checkout, but not against an arbitrarily minimal QPATH - containing only di.depcheck and di.k4unit. The module itself has no such assumption (verified standalone - against exactly that minimal QPATH); only the test suite's negative control does. - - The `checkdeps[]` case of two different loaded modules declaring the same dependency at different minimums - is manually verified, not committed as an automated test — it needs two real `deps.q` fixtures on disk, and - no real module ships a non-empty `deps.q` yet to build a portable, no-hardcoded-path test against. Verified - directly: `di.handlers` required simultaneously at a satisfied minimum (by one fixture consumer) and an - unsatisfied one (by another) correctly produced exactly one failure line, for the unsatisfied case only. - - A malformed `deps.q` — present but not a dict (wrong type; a plausible authoring mistake) — is reported as - its own failure line (`" deps.q is malformed - expected a dict, got type "`) rather than crashing. - This is now committed as an automated test (a real, malformed `deps.q` written to a scratch module directory - at runtime, see "Running Tests" above) — closing what was previously a manually-verified-only gap. - - This kdb-x build's `like` throws `` 'nyi `` on any pattern with more than one `*`-delimited literal segment - (e.g. `` "*a*b*" ``), regardless of whether the string being matched contains a newline. Every `like` - assertion in `test.csv` uses a single literal segment (`` "*single clause*" ``) for this reason — worth - knowing before adding a new one. - -- **Five real bugs found and fixed**, all by directly constructing and running the edge case, not by reading the - code — every one looked entirely reasonable on the page: - 1. **`checkdepversion`'s eager `or`.** Originally combined `(xp~(::)) or not \`version in key xp` as a single - condition. q's `or`/`and` are eager vector operators, not short-circuiting, so `key xp` was evaluated even - when `xp` was already known to be `(::)`, throwing `'type` (`key` doesn't accept a generic null). Fixed to - sequential `if[]` early returns, matching the pattern `checkonecontract` already used correctly for the - same situation. Constructed via direct `.m.di` namespace manipulation, since no real broken module could be - made to load successfully and then fail an export read. - 2. **A malformed `deps.q` crashed the entire audit, not just the one bad module.** `checkmoduledeps` handed - whatever `readdepsq` returned straight to `key`/`value`/`checkonedep'` with no type check. A `deps.q` that - defines `deps` as something other than a dict (e.g. a plain string) threw a raw `'dict` error out of that - `each` call — and since `each` doesn't isolate per-element errors, one badly-authored `deps.q` anywhere in - the loaded module set aborted `checkdeps[]` entirely, masking every real failure in every other module. - Fixed by validating `99h=type d` in `checkmoduledeps` and reporting malformed `deps.q` as its own clear - failure line instead. This one is the more serious of the two: for a tool whose entire purpose is to run - reliably at startup, an unhandled crash from one module's authoring mistake defeats the purpose more - thoroughly than any single check being wrong. - 3. **Dependency resolution was silently wrong for any non-`di.*` name.** `getexport`/`checkonedep`/ - `checkonecontract` all hardcoded the `` `.m.di `` namespace when checking whether a dependency was loaded. - A di.* module's `deps.q` can legitimately name an external vendor module as a hard dependency (e.g. - `kx.log`) — but vendor modules register under their own `` `.m. `` namespace (`kx.log` → - `` `.m.kx ``, confirmed directly: `use\`kx.log` populates `` `.m.kx ``, not `` `.m.di ``). A genuinely - loaded `kx.log` was reported as `"kx.log requires minimum version 1.0.0, not found"` — a silent false - negative, worse than a crash, since it looks like a correct, actionable result. Fixed by generalising - `shortmod`/introducing `modvendorns` to resolve a dependency's vendor namespace from its own name instead - of assuming `di.`. `checkdeps`'s outer walk of *which modules to audit as consumers* stays intentionally - scoped to `` key `.m.di `` — di.depcheck audits the di.* modularisation effort's dependency graph, not - arbitrary vendor modules' own internal needs; only *resolving a declared dependency's target* needed to - stop assuming di.*. `kx.log` itself isn't committed anywhere in this repo (confirmed via `git ls-tree` - across every branch — it's only vendored locally on this dev machine), so the regression test fabricates a - non-di.* module via direct namespace manipulation (`` `.m.zz.0widget ``) rather than depending on it. - 4. **`parsesemver` silently mis-handled any non-numeric version component**, found during a later adversarial - re-pass over the shipped code, again by direct testing rather than reading. It only guarded the wrong - *count* of dot-separated parts; an individual part that partially parsed (any leading digits followed by - non-numeric text, e.g. a pre-release tag `"3-rc1"`) silently became `0Ni` (null) at just that position, - rather than failing the whole version. Since q's integer null sorts as the lowest possible value, this - produced two confirmed wrong answers, not merely an "unsupported" gap: `vergte["1.2.3-rc1";"1.2.0"]` - returned `0b` — a real, newer version reported as failing an *older* minimum, solely because its unparsed - suffix nulled out the patch component that would otherwise have decided the comparison; and - `vergte["1.2.3";"abc"]` returned `1b` — a typo'd `deps.q` minimum silently treated as "no real requirement - at all," with nothing surfaced anywhere. Fixed by collapsing *any* unparseable component to the same - `(0Ni;0Ni;0Ni)` "malformed" sentinel used for the wrong-part-count case (`ismalformed`), and having - `checkfoundversion` check for it explicitly on both sides (declared minimum and exported found-version) - before ever handing either to the numeric comparison — surfacing a distinct, correctly-worded failure line - instead of a silent, misleading pass or fail. See `checkonedep`/`checkfoundversion` and the malformed-semver - tests in `test.csv`. - 5. **Warnings were silently dropped from the log whenever a failure also occurred.** `init` computed `warnings` - up front but only logged them in a branch positioned *after* the failures check, and the failures check - signals (`` ' ``) on any failure — so if a real warning (e.g. `.z.ts` misuse, a stale kdb-x version) - happened to coincide with a real failure in the same `init` call, execution never reached the - `.z.m.logwarn` line at all: the warning was computed, then silently discarded, with nothing about it ever - logged anywhere. Found by directly asking "is this thoroughly tested and does it have thorough logging" and - reading the actual `init` body, not by running anything new — the bug was visible directly in the code once - someone looked at execution order rather than just at each `if[]` block in isolation. Fixed by moving the - warnings-logging line to run unconditionally before the failures check, so a coinciding warning is always - logged regardless of whether `init` also throws. Regression test: `init` is now called end-to-end with a - real failure (di.timer's contract gap) and a real, simultaneously-computed warning - (`` `minkdbxversion `` set above `.z.K`), and the captured log table is asserted to contain *both* an - `error` row and a `warn` row with the real kdbxcheck text — previously that `warn` row would never have - appeared. - -- **Two further gaps noted but not changed** (defensive-completeness items, not bugs — no real file or call site - in this codebase currently triggers either): - - `readdepsq` assumes a `deps.q` is a *single pure* `deps:...` assignment, per every real precedent seen so - far. This is unenforced: a `deps.q` that also defines stray globals beyond `deps` would leak them into the - root namespace permanently, since only `deps` itself is ever explicitly deleted after being captured. - - `init`'s validation of its own injected `log` dependency checks key presence only (`info`/`warn`/`error` - present), not function arity — the same shape-only limitation already called out above for how - `checkcontracts` audits *other* modules' contracts applies equally to di.depcheck validating its own. +- **`di.toml` coupling.** `deps.toml` is inert data parsed by the `di.toml` module (no evaluation), whereas `deps.q` is a q dict literal read by executing it. Module keys in a `[dependencies]` section must be **quoted** — `"di.timer" = "0.2.0"` — because di.toml rejects unquoted dotted keys. di.depcheck reads only string version values via `parsefile`, so it is unaffected by di.toml's scalar value-typing. All format-specific reading lives behind `finddepsq`/`readdepsq`/`finddepstoml`/`readdepstoml`/`readdeps`; every other function consumes only their merged, format-agnostic dict, so dropping a format later is a change to those readers alone. Which format the repo ultimately standardises on is an open cross-team decision (with the TorqX POC), deliberately not resolved here. +- **Deliberate divergence from di.config.** di.config's `requiretoml` throws and aborts its entire settings cascade the instant a `.toml` tier can't be read, because it must hand back one complete, correct config. di.depcheck does the **opposite** on the same event — it catches, folds one clearly-worded line (matching `requiretoml`'s wording) into the aggregate report, and keeps walking — because its whole purpose is to surface *every* problem across *many* modules in one pass. Two right answers to two different jobs, not an inconsistency. +- **Transitive presence is walked; transitive version is not, yet.** `checkgraph` reports a missing transitive dependency at depth ≥ 2, but cannot check the *version* of an unloaded module without either loading it (which the walk must not do) or a per-module `VERSION` file. `VERSION` files are not yet a repo-wide convention — modules carry an inline exported `version` — so adopting them is a coordinated rollout; the walk gains transitive version checking for free once they land. +- **`.z.ts` ownership is a best-effort proxy.** di.timer's `enabled` flag evidences that its `init` bound `.z.ts`; it cannot detect something rebinding `.z.ts` afterwards. Scope is deliberately limited to `.z.ts` only. +- **The kdb-x-version check is shape-only for now.** The comparison is implemented and unit-tested in isolation, but no caller supplies `` `minkdbxversion `` yet, so it is a no-op in every real invocation until di.torq (or another caller) threads a real minimum through. It is warning-only, not fail-fast. Uses `.z.K`, not `.z.v`, matching di.k4unit's existing precedent. +- **Semver is numeric `X.Y.Z` only** — no pre-release/build-metadata support, matching every version string in this codebase. A malformed declared minimum or exported found-version is reported as its own distinct failure line rather than silently mis-compared. During 0.x.y development a passing `>=` check does not guarantee contract compatibility. +- **Modules currently missing a `version` export** — di.timer, di.kafka, and di.log were all found without one; di.handlers is the only module that exports it (and its own comments call that a placeholder pending this module). Adding `version` everywhere is a coordinated repo-wide rollout, of which di.depcheck is the consumer side. di.compression separately imports `kx.log` directly rather than following the binary `{[c;m]}` convention — a pre-`consistency.md` outlier worth its own cleanup. diff --git a/di/depcheck/depcheck.q b/di/depcheck/depcheck.q index c8886f36..0a56b6f2 100644 --- a/di/depcheck/depcheck.q +++ b/di/depcheck/depcheck.q @@ -34,7 +34,7 @@ shorttofull:{[s] modvendorns:{[modname] / di.timer -> `.m.di ; kx.log -> `.m.kx - the top-level session namespace a module's vendor is keyed under. / a di.* module's deps.q may legitimately declare a hard dependency on an external vendor module (e.g. - / kx.log), so dependency resolution (getexport/checkonedep/checkonecontract) must not hardcode `.m.di - only + / kx.log), so dependency resolution (getexport/checkonedep/checkcontract) must not hardcode `.m.di - only / checkdeps's walk of which modules to audit as consumers is intentionally di.*-scoped `$".m.",first "." vs string modname }; @@ -72,6 +72,70 @@ readdepsq:{[modname] d }; +/ ============================================================ +/ deps.toml loading (lazy, file-existence-gated di.toml) +/ ============================================================ + +finddepstoml:{[modname] + / locate /deps.toml on QPATH - sibling of finddepsq; returns its file path, or (::) if the module ships none + relpath:(ssr[string modname;".";"/"]),"/deps.toml"; + roots:":" vs getenv`QPATH; + paths:{[relpath;root] hsym `$root,"/",relpath}[relpath;] each roots; + found:paths where not {[p] ()~key p} each paths; + $[0=count found;(::);first found] + }; + +resolvetoml:{[] + / lazily resolve di.toml once and cache it in module state, returning its export dict. throws if di.toml is not + / resolvable on QPATH - caught by readdepstoml and turned into one aggregated failure line, never an abort. only ever + / called when a real deps.toml file has already been found, so a module with no deps.toml never triggers a di.toml load + cached:@[get;`.z.m.tomlmod;{(::)}]; + if[not cached~(::);:cached]; + m:use`di.toml; + .z.m.tomlmod:m; + m + }; + +readdepstoml:{[modname] + / read /deps.toml if it exists, returning (failures;depsdict). di.toml is touched ONLY when the file exists + / (file-existence-gated, exactly like di.config's parsefile checks existence before dispatching on extension). a missing + / or broken di.toml degrades to one aggregated failure line worded like di.config's requiretoml (name the file/module, + / name the underlying cause, one sentence) - but deliberately does NOT throw: di.config's requiretoml aborts its whole + / cascade because it must return one complete config, whereas di.depcheck must surface every module's problems in one + / pass, so it catches, records a line, and keeps walking (see depcheck.md) + p:finddepstoml modname; + if[p~(::);:(();()!())]; + path:1_string p; + @[{[mn;pth] + d:(resolvetoml[])[`parsefile] pth; + / a well-formed manifest has a [dependencies] section (a dict); absent -> nothing declared; present but not a + / dict (e.g. `dependencies = "x"` written as a scalar) is a clear authoring error, reported not merged (a + / non-dict here would otherwise throw out of readdeps's merge and abort the whole walk) + $[not `dependencies in key d;(();()!()); + 99h=type d`dependencies;(();d`dependencies); + (enlist "di.depcheck: deps.toml for ",(string mn), + " has a malformed [dependencies] section - expected a table of module = \"version\" entries";()!())] + }[modname;]; + path; + {[mn;e] (enlist "di.depcheck: cannot read deps.toml for ",(string mn), + " - the di.toml module was not found on QPATH or failed to parse it; di.toml is required to read .toml manifests (underlying: ",e,")"; + ()!())}[modname;]] + }; + +readdeps:{[modname] + / merged manifest reader: reads deps.q and deps.toml where each exists and merges them, with deps.toml winning on a key + / clash - mirroring di.config's live parsetier ((parsefile base,".q"),parsefile base,".toml"). returns (failures;dict): + / failures aggregates a malformed-deps.q line and/or an unreadable-deps.toml line; dict is the merged symbol->minversion + / mapping (empty if neither format is present). all format-specific reading lives here and in finddepsq/finddepstoml - + / every downstream function (checkonedep/checkmoduledeps/checkdeps/checkgraph) consumes only this already-merged dict + dq:readdepsq modname; + qmalformed:(not dq~(::)) and not 99h=type dq; + qdict:$[qmalformed or dq~(::);()!();dq]; + tr:readdepstoml modname; + malfail:$[qmalformed;enlist string[modname]," deps.q is malformed - expected a dict, got type ",string type dq;()]; + (malfail,tr 0;qdict,tr 1) + }; + / ============================================================ / semver comparison / ============================================================ @@ -144,6 +208,12 @@ checkonedep:{[dep;minver] / returns () on pass, or an enlisted failure line matching the plan's exact report format / not vendor-restricted to di.* - a deps.q may name an external vendor module (e.g. kx.log) as a hard / dependency, so presence is checked against dep's own vendor namespace, not hardcoded to `.m.di + / a manifest version must be a string (quoted in deps.toml, a q string in deps.q). a non-string value - an unquoted + / deps.toml version parsed as a float/int by di.toml, or a symbol/number in deps.q - is an authoring error reported + / as its own clear line, rather than left to corrupt the concatenated message or throw out of parsesemver's `vs`. + / 10h=abs type accepts a char vector or a lone char atom (both string-ish), rejecting int/float/symbol + if[not 10h=abs type minver; + :enlist string[dep]," has a non-string minimum version in its manifest (got type ",(string type minver),") - versions must be quoted strings"]; depshort:shortmod dep; vns:modvendorns dep; $[not depshort in key vns; @@ -152,15 +222,14 @@ checkonedep:{[dep;minver] }; checkmoduledeps:{[modshort] - / checks one already-loaded module's declared deps.q (if any) against the current session - / a malformed deps.q (present but not a dict) is reported as its own failure rather than left to throw a raw - / q type error out of checkonedep'[key d;value d] - a crash there would abort the whole checkdeps[] walk (each - / does not isolate per-element errors), masking every other module's real failures behind one bad file + / checks one already-loaded module's declared deps (deps.q and/or deps.toml) against the current session. returns + / aggregated failure lines: manifest read-failures (a malformed deps.q or an unreadable deps.toml, both pre-collected + / by readdeps rather than thrown) plus each declared dependency's presence/version line. one bad manifest never aborts + / the checkdeps[] walk and masks other modules' real failures - readdeps catches, checkonedep is a pure per-pair function modname:shorttofull modshort; - d:readdepsq modname; - $[d~(::);(); - not 99h=type d;enlist string[modname]," deps.q is malformed - expected a dict, got type ",string type d; - raze checkonedep'[key d;value d]] + r:readdeps modname; + merged:r 1; + (r 0),$[0=count merged;();raze checkonedep'[key merged;value merged]] }; checkdeps:{[] @@ -175,25 +244,86 @@ checkdeps:{[] raze checkmoduledeps each key `.m.di }; +/ ============================================================ +/ transitive dependency-manifest graph walk +/ ============================================================ + +resolvemodule:{[modname] + / reimplements the kdb-x `use` loader's QPATH search (colon-separated roots, first match wins; a dotted module name's + / dots become path segments) to test whether a module is INSTALLED on QPATH without loading it - returns the resolved + / module directory (hsym) or (::) if nothing matches. adapted from TorqX di.depcheck's resolvemodule; deliberately does + / not reach into kdb-x's undocumented .Q.m.* internals. used only for transitive presence, distinct from checkonedep's + / loaded-check: a module can be installed-on-QPATH yet not loaded-into-the-session + relpath:ssr[string modname;".";"/"]; + roots:":" vs getenv`QPATH; + exts:(".q";".k";".q_";".k_"); + dirs:{[rp;root] root,"/",rp}[relpath;] each roots; + hit:{[dir;exts] any {[d;e] 0= 2 that does not resolve on QPATH. depth-1 (deps directly + / declared by a loaded module) stays with checkdeps (presence=loaded, version=exported); checkgraph excludes those via + / the directdeps set so the two never double-report. satisfies consistency.md's on-disk / loads-no-module-code walk + / without a pre-load pass. LIMITATION: transitive VERSION checking of an unloaded module is not done here - a found + / version needs loading (forbidden) or a per-module VERSION file (deferred until a repo-wide rollout) - see depcheck.md + roots:shorttofull each key `.m.di; + directdeps:distinct raze {[r] key (readdeps r) 1} each roots; + acc:`visited`fails!(`symbol$();()); + acc:{[covered;directdeps;acc;root] visit[covered;directdeps;acc;root]}[roots;directdeps]/[acc;roots]; + acc`fails + }; + / ============================================================ / core dependency contract checks / ============================================================ -checkonecontract:{[provider;required] - / if provider is loaded, checks its export dict contains every key its known contract requires - / vendor-agnostic like checkonedep/getexport, though every current contracts entry happens to be di.*-prefixed +/ generic, exported primitive: checks whether a loaded module's export dict contains every key a given contract +/ requires. vendor-agnostic like checkonedep/getexport, though every current contracts entry happens to be +/ di.*-prefixed. usable for any provider/contract pair, not just the three known core dependencies checkcontracts[] +/ audits automatically below. never calls `use` - introspection only, matching this module's whole design; unlike +/ its TorqX counterpart of the same name, which does call `use` for real +/ requiredkeys accepts a single symbol atom as well as a vector - every internal caller (contracts, below) already +/ passes a vector, but this is now a public entry point for a caller auditing a contract of exactly one key, which +/ is naturally written as a bare symbol rather than remembering to `enlist` it - a real boundary this module didn't +/ have before it was exported. caught by direct testing, not by reading the code +checkcontract:{[provider;requiredkeys] + requiredkeys:$[-11h=type requiredkeys;enlist requiredkeys;requiredkeys]; sn:shortmod provider; vns:modvendorns provider; if[not sn in key vns;:()]; xp:getexport provider; if[xp~(::);:enlist string[provider]," is loaded but its export dict could not be read"]; - missing:required where not required in key xp; + missing:requiredkeys where not requiredkeys in key xp; $[0=count missing;();enlist string[provider]," is missing required contract key(s): ",", " sv string missing] }; checkcontracts:{[] / checks whichever of the known core-dependency providers (di.log/di.timer/di.handlers) are loaded in this session - raze checkonecontract'[key contracts;value contracts] + raze checkcontract'[key contracts;value contracts] }; / ============================================================ @@ -258,7 +388,7 @@ init:{[deps] .z.m.logwarn:(deps`log)`warn; .z.m.logerr:(deps`log)`error; - failures:checkdeps[],checkcontracts[]; + failures:checkdeps[],checkgraph[],checkcontracts[]; warnings:ztscheck[],kdbxcheck[deps]; / warnings are logged unconditionally, before the failures check below - not gated behind "no failures", so a diff --git a/di/depcheck/init.q b/di/depcheck/init.q index 35b1519e..b89b5cc8 100644 --- a/di/depcheck/init.q +++ b/di/depcheck/init.q @@ -3,4 +3,4 @@ \l ::depcheck.q -export:([init;version]) +export:([init;version;checkcontract]) diff --git a/di/depcheck/test.csv b/di/depcheck/test.csv index 82e59339..bb325fc5 100644 --- a/di/depcheck/test.csv +++ b/di/depcheck/test.csv @@ -35,6 +35,7 @@ comment,,,,,,,this suite must pass standalone against a bare main checkout not j before,0,0,q,timer:use`di.timer,1,,load the real di.timer module - always present on main - a live negative control before,0,0,q,.dc.havekafka:@[{kafka:use`di.kafka;1b};`;0b],1,,attempt di.kafka - PR #112 unmerged - absent on a bare main checkout before,0,0,q,.dc.havehandlers:@[{handlers:use`di.handlers;1b};`;0b],1,,attempt di.handlers - PR #114 unmerged - absent on a bare main checkout +before,0,0,q,.dc.havetoml:@[{use`di.toml;1b};`;0b],1,,attempt di.toml - PR #116, on its own feature-toml branch - absent on a bare feature-depcheck checkout comment,,,,,,,checkonedep - dependency presence and version, exercised directly against live loaded modules true,0,0,q,".m.di.0depcheck.checkonedep[`di.doesnotexist;""1.0.0""]~enlist ""di.doesnotexist requires minimum version 1.0.0, not found""",1,,an undeclared/unloaded dependency is reported not found @@ -50,7 +51,7 @@ comment,,,,,,,loaded-but-unreadable-export edge case - constructed via direct na comment,,,,,,,module can be made to load successfully and then fail an export read. Caught checkdepversion using an comment,,,,,,,eager `or` over a `~(::)` check during manual smoke testing - not by reading the code - see depcheck.q before,0,0,q,.m.di.0fakemod.something:1,1,,fabricate a loaded module whose export dict was never actually set -true,0,0,q,".m.di.0depcheck.checkonecontract[`di.fakemod;`x`y]~enlist ""di.fakemod is loaded but its export dict could not be read""",1,,checkonecontract reports the unreadable-export case distinctly +true,0,0,q,".m.di.0depcheck.checkcontract[`di.fakemod;`x`y]~enlist ""di.fakemod is loaded but its export dict could not be read""",1,,checkcontract reports the unreadable-export case distinctly true,0,0,q,".m.di.0depcheck.checkonedep[`di.fakemod;""1.0.0""]~enlist ""di.fakemod requires minimum version 1.0.0, but di.fakemod exports no version""",1,,checkonedep no longer throws 'type on this case - regression test for the `or`-eagerness bug comment,,,,,,,non-di.* vendor dependency resolution - a deps.q may legitimately name an external vendor module (e.g. @@ -87,6 +88,19 @@ comment,,,,,,,checkcontracts - core dependency contract shape, exercised against true,0,0,q,any .m.di.0depcheck.checkcontracts[] like "di.timer is missing required contract key(s): cp",1,,real di.timer fails the Timer contract - cp is defined but not exported - always present on main true,0,0,q,not any .m.di.0depcheck.checkcontracts[] like "di.handlers is missing*",1,,real di.handlers satisfies the Handlers contract in full when present; vacuously true (no entry to match) when absent +comment,,,,,,,checkcontract - generic exported primitive underneath checkcontracts[], reached via the real used export +comment,,,,,,,dict (not the internal namespace path) since the point of exporting it is that callers reach it this way +true,0,0,q,0=count depcheck.checkcontract[`di.timer;`addjob`deletejobs],1,,passing case - real di.timer genuinely exports both required keys +true,0,0,q,"depcheck.checkcontract[`di.timer;`addjob`cp]~enlist ""di.timer is missing required contract key(s): cp""",1,,failing case - only the genuinely missing key (cp) is reported not the present one (addjob) +true,0,0,q,0=count depcheck.checkcontract[`di.doesnotexist;`x`y],1,,not-loaded case - a module never used at all returns no failure not a crash, matching checkonedep's presence-check being the place absence is reported instead + +comment,,,,,,,checkcontract - bare symbol atom for requiredkeys (a single-key contract written the natural way, +comment,,,,,,,without remembering to enlist it) - found by direct testing against the newly-exported function, not +comment,,,,,,,by reading the code. every internal caller (contracts dict values) already passes a vector, so this +comment,,,,,,,edge case did not exist before checkcontract became a public entry point for arbitrary future callers +true,0,0,q,"depcheck.checkcontract[`di.timer;`cp]~enlist ""di.timer is missing required contract key(s): cp""",1,,a bare atom failing key normalises to the same vector-shaped result as an enlisted one +true,0,0,q,0=count depcheck.checkcontract[`di.timer;`addjob],1,,a bare atom passing key normalises correctly too - not just the failing case + comment,,,,,,,ztscheck - .z.ts ownership, exercised against real .z.ts state run,0,0,q,system"x .z.ts",1,,reset .z.ts to the kdb-x built-in default before testing true,0,0,q,0=count .m.di.0depcheck.ztscheck[],1,,no warning when .z.ts is unbound @@ -106,3 +120,64 @@ true,0,0,q,.dc.initerr like "*di.timer is missing required contract key(s): cp*" true,0,0,q,any `error = exec lvl from .dc.captbl,1,,the failing report was also logged at error before being signalled true,0,0,q,any `warn = exec lvl from .dc.captbl,1,,the coinciding warning was also logged at warn - not silently dropped - regression test for the logging fix true,0,0,q,(exec first msg from .dc.captbl where lvl=`warn) like "*is below the configured minimum*",1,,the logged warning is the real kdbxcheck text, not an empty or wrong message + +comment,,,,,,,dual-format manifest reading - deps.q and/or deps.toml merged with .toml winning on a clash (mirrors +comment,,,,,,,di.config's live parsetier). scratch module dirs are derived from QPATH at runtime - no hardcoded path, +comment,,,,,,,no committed fixture - and created/removed with run actions (not before) so each replays in file order +run,0,0,q,".dc.qr:first "":"" vs getenv`QPATH; system ""mkdir -p "",.dc.qr,""/di/zzqonly "",.dc.qr,""/di/zztomlonly "",.dc.qr,""/di/zzboth""",1,,create scratch module directories from QPATH at runtime +run,0,0,q,"(hsym `$.dc.qr,""/di/zzqonly/deps.q"") 0: enlist ""deps:enlist[`di.timer]!enlist \""1.0.0\""""",1,,zzqonly ships only a deps.q +run,0,0,q,"(hsym `$.dc.qr,""/di/zztomlonly/deps.toml"") 0: (""[dependencies]"";""\""di.timer\"" = \""1.0.0\"""")",1,,zztomlonly ships only a deps.toml +run,0,0,q,"(hsym `$.dc.qr,""/di/zzboth/deps.q"") 0: enlist ""deps:enlist[`di.timer]!enlist \""1.0.0\""""",1,,zzboth deps.q declares di.timer 1.0.0 +run,0,0,q,"(hsym `$.dc.qr,""/di/zzboth/deps.toml"") 0: (""[dependencies]"";""\""di.timer\"" = \""9.9.9\"""")",1,,zzboth deps.toml declares di.timer 9.9.9 - the clashing value that should win +true,0,0,q,"((.m.di.0depcheck.readdeps `di.zzqonly) 1)~enlist[`di.timer]!enlist ""1.0.0""",1,,a q-only module's deps.q is read +true,0,0,q,"$[.dc.havetoml;((.m.di.0depcheck.readdeps `di.zztomlonly) 1)~enlist[`di.timer]!enlist ""1.0.0"";1b]",1,,a toml-only module's deps.toml is read via di.toml (no-op pass when di.toml is absent) +true,0,0,q,"$[.dc.havetoml;((.m.di.0depcheck.readdeps `di.zzboth) 1)~enlist[`di.timer]!enlist ""9.9.9"";1b]",1,,when both formats declare the same key the .toml value wins on the clash (no-op pass when di.toml is absent) +true,0,0,q,0=count (.m.di.0depcheck.readdeps `di.zzqonly) 0,1,,a clean q-only manifest yields no read-failures +true,0,0,q,0=count (.m.di.0depcheck.readdeps `di.doesnotexist) 1,1,,a module with neither format yields an empty merged manifest +run,0,0,q,"system ""rm -rf "",.dc.qr,""/di/zzqonly "",.dc.qr,""/di/zztomlonly "",.dc.qr,""/di/zzboth""",1,,remove the dual-format scratch fixtures + +comment,,,,,,,a broken deps.toml degrades to one aggregated failure line worded like di.config's requiretoml - never a +comment,,,,,,,throw (deliberately unlike di.config's requiretoml which aborts its whole cascade - see depcheck.md) +run,0,0,q,"system ""mkdir -p "",.dc.qr,""/di/zzbroken""; (hsym `$.dc.qr,""/di/zzbroken/deps.toml"") 0: enlist ""garbage no equals""",1,,create a scratch module with a malformed deps.toml +true,0,0,q,"any ((.m.di.0depcheck.readdeps `di.zzbroken) 0) like ""di.depcheck: cannot read deps.toml*""",1,,a malformed deps.toml is reported as one clear di.toml-attributed failure line, not thrown +run,0,0,q,"system ""rm -rf "",.dc.qr,""/di/zzbroken""",1,,remove the broken-toml scratch fixture + +comment,,,,,,,transitive graph walk - a loaded root (zzga, fabricated) -> zzgb (installed on QPATH) -> zzgc (absent). +comment,,,,,,,zzgb also declares zzga (a cycle), which the visited-set guard must terminate. zzgb itself is a direct dep +comment,,,,,,,of the loaded root so it is checkdeps' domain and must NOT be double-reported by checkgraph +run,0,0,q,.m.di.0zzga.marker:1,1,,fabricate a loaded module di.zzga so it becomes a walk root +run,0,0,q,"system ""mkdir -p "",.dc.qr,""/di/zzga "",.dc.qr,""/di/zzgb""; (hsym `$.dc.qr,""/di/zzga/init.q"") 0: enlist ""export:([])""; (hsym `$.dc.qr,""/di/zzga/deps.q"") 0: enlist ""deps:enlist[`di.zzgb]!enlist \""1.0.0\""""",1,,install zzga on QPATH declaring zzgb +run,0,0,q,"(hsym `$.dc.qr,""/di/zzgb/init.q"") 0: enlist ""export:([])""; (hsym `$.dc.qr,""/di/zzgb/deps.q"") 0: enlist ""deps:`di.zzgc`di.zzga!(\""1.0.0\"";\""1.0.0\"")""",1,,install zzgb on QPATH declaring zzgc (missing) and zzga (cycle) +true,0,0,q,"any (.m.di.0depcheck.checkgraph[]) like ""di.zzgc is required transitively by di.zzgb*""",1,,checkgraph surfaces a depth>=2 dependency that does not resolve on QPATH +true,0,0,q,"not any (.m.di.0depcheck.checkgraph[]) like ""di.zzgb is required transitively*""",1,,zzgb (a direct dep of the loaded root) is not double-reported - the cycle also terminated without error +run,0,0,q,"system ""rm -rf "",.dc.qr,""/di/zzga "",.dc.qr,""/di/zzgb""",1,,remove the graph-walk on-disk fixtures (the fabricated .m.di.0zzga is harmless with no deps on disk) + +comment,,,,,,,resolvemodule - QPATH presence resolution without loading, distinct from checkonedep's loaded-check +true,0,0,q,not (::)~.m.di.0depcheck.resolvemodule `di.timer,1,,a real installed module resolves on QPATH +true,0,0,q,(::)~.m.di.0depcheck.resolvemodule `di.doesnotexistanywhere,1,,an absent module resolves to (::) + +comment,,,,,,,a deps.toml that exists but declares no [dependencies] section - the else branch of readdepstoml +run,0,0,q,"system ""mkdir -p "",.dc.qr,""/di/zznodeps""; (hsym `$.dc.qr,""/di/zznodeps/deps.toml"") 0: enlist ""name = \""foo\""""",1,,create a deps.toml with a top-level key but no [dependencies] section +true,0,0,q,0=count (.m.di.0depcheck.readdeps `di.zznodeps) 1,1,,a deps.toml with no [dependencies] section yields an empty merged manifest not a crash +true,0,0,q,"$[.dc.havetoml;0=count (.m.di.0depcheck.readdeps `di.zznodeps) 0;1b]",1,,and no read-failures either - a valid file with nothing declared is not an error (needs di.toml to read the file; no-op pass when absent) +run,0,0,q,"system ""rm -rf "",.dc.qr,""/di/zznodeps""",1,,remove the no-dependencies scratch fixture + +comment,,,,,,,malformed [dependencies] - a deps.toml where `dependencies` is a scalar not a section. must degrade to a +comment,,,,,,,clean failure line with an empty merged dict, NOT throw out of readdeps's merge and abort the whole walk +run,0,0,q,"system ""mkdir -p "",.dc.qr,""/di/zzscalar""; (hsym `$.dc.qr,""/di/zzscalar/deps.toml"") 0: enlist ""dependencies = \""oops\""""",1,,create a deps.toml whose dependencies key is a scalar string +true,0,0,q,"$[.dc.havetoml;any ((.m.di.0depcheck.readdeps `di.zzscalar) 0) like ""di.depcheck: deps.toml for di.zzscalar has a malformed*"";1b]",1,,a non-dict [dependencies] is reported as one clear failure line (no-op pass when di.toml is absent) +true,0,0,q,0=count (.m.di.0depcheck.readdeps `di.zzscalar) 1,1,,and the merged manifest is empty so the merge never throws +run,0,0,q,"system ""rm -rf "",.dc.qr,""/di/zzscalar""",1,,remove the malformed-dependencies scratch fixture + +comment,,,,,,,non-string version value - an unquoted deps.toml version parses to a float; a deps.q version may be an int +comment,,,,,,,or symbol. either is an authoring error reported as its own clear line, never a corrupted concatenated +comment,,,,,,,message or a throw out of parsesemver's `vs`. a single-char string version must NOT trip this guard +run,0,0,q,"system ""mkdir -p "",.dc.qr,""/di/zzfloatver""; (hsym `$.dc.qr,""/di/zzfloatver/deps.toml"") 0: (""[dependencies]"";""\""di.timer\"" = 1.0"")",1,,deps.toml with an UNQUOTED version - di.toml parses it as a float +true,0,0,q,"$[.dc.havetoml;any (.m.di.0depcheck.checkmoduledeps `0zzfloatver) like ""di.timer has a non-string minimum version*"";1b]",1,,an unquoted (float) deps.toml version is reported as a non-string version, not a garbage line or a crash (needs di.toml to parse the float; no-op pass when absent) +run,0,0,q,"system ""rm -rf "",.dc.qr,""/di/zzfloatver""",1,,remove the float-version fixture +run,0,0,q,"system ""mkdir -p "",.dc.qr,""/di/zzintver""; (hsym `$.dc.qr,""/di/zzintver/deps.q"") 0: enlist ""deps:enlist[`di.timer]!enlist 5""",1,,deps.q with an int version value +true,0,0,q,"any (.m.di.0depcheck.checkmoduledeps `0zzintver) like ""di.timer has a non-string minimum version*""",1,,an int deps.q version is reported as a non-string version too +run,0,0,q,"system ""rm -rf "",.dc.qr,""/di/zzintver""",1,,remove the int-version fixture +run,0,0,q,"system ""mkdir -p "",.dc.qr,""/di/zzonechar""; (hsym `$.dc.qr,""/di/zzonechar/deps.q"") 0: enlist ""deps:enlist[`di.timer]!enlist \""1\""""",1,,deps.q with a legitimate single-char string version +true,0,0,q,"not any (.m.di.0depcheck.checkmoduledeps `0zzonechar) like ""di.timer has a non-string minimum*""",1,,a single-char string version is valid and must NOT be flagged as non-string - the guard does not over-trigger +run,0,0,q,"system ""rm -rf "",.dc.qr,""/di/zzonechar""",1,,remove the single-char-version fixture From 1f6909157930f5c282b4b2a99ecf5f67a2342ed9 Mon Sep 17 00:00:00 2001 From: alowrydi Date: Tue, 4 Aug 2026 16:24:19 +0100 Subject: [PATCH 04/11] Initial di.permissions draft --- di/permissions/deps.q | 7 + di/permissions/init.q | 13 + di/permissions/permissions.md | 431 +++++++++++ di/permissions/permissions.q | 1094 +++++++++++++++++++++++++++ di/permissions/test.csv | 496 ++++++++++++ di/permissions/test_integration.csv | 38 + 6 files changed, 2079 insertions(+) create mode 100644 di/permissions/deps.q create mode 100644 di/permissions/init.q create mode 100644 di/permissions/permissions.md create mode 100644 di/permissions/permissions.q create mode 100644 di/permissions/test.csv create mode 100644 di/permissions/test_integration.csv diff --git a/di/permissions/deps.q b/di/permissions/deps.q new file mode 100644 index 00000000..8fd4c189 --- /dev/null +++ b/di/permissions/deps.q @@ -0,0 +1,7 @@ +/ hard module dependencies and their minimum versions, validated by di.depcheck +/ di.permissions has NO hard dependencies - it is a standalone module: +/ - log and handlers are injected via init as dictionaries of functions +/ - lamq's variable introspection is handled internally rather than via di.api, which is +/ registry-only and does not expose varnames/allns. it does not reproduce TorQ's namespace walk: +/ the query is tokenised first and only those tokens tested, which is O(tokens) not O(all names) +deps:(`$())!(); diff --git a/di/permissions/init.q b/di/permissions/init.q new file mode 100644 index 00000000..94d4f0e1 --- /dev/null +++ b/di/permissions/init.q @@ -0,0 +1,13 @@ +/ di.permissions - role-based access control and authentication for a KDB-X process +/ consolidates TorQ's permissions.q (.pm), writeaccess.q (.readonly), ldap.q (.ldap) and common/execas.q +/ owns the exec phase of the message-handling .z.* events via the injected di.handlers dependency + +\l ::permissions.q + +/ NB: export:([...]) EVALUATES each name, so it can only list names that already exist - the export +/ list and the implementation therefore cannot drift apart in this direction. +/ init and getapimeta are framework plumbing di.torq calls by convention; every other name here has a +/ getapimeta row, which the test suite asserts +export:([init;teardown;version;getapimeta;status; + allowed;requ;val;valp;execas; + admin;loadpermissions;unblock]) diff --git a/di/permissions/permissions.md b/di/permissions/permissions.md new file mode 100644 index 00000000..ab3767f3 --- /dev/null +++ b/di/permissions/permissions.md @@ -0,0 +1,431 @@ +# di.permissions + +Role-based access control and authentication for a KDB-X process. It owns the `exec` phase of every +message-handling `.z.*` event via the injected `di.handlers` dependency, permission-checks each +incoming query against a user's roles and groups, and optionally enforces a whole-process read-only +mode. + +Consolidates five TorQ files: `code/handlers/permissions.q` (`.pm`), `writeaccess.q` (`.readonly`), +`ldap.q` (`.ldap`), and `code/common/execas.q`. TorQ's `controlaccess.q` tiered engine is **deferred** — +see [Engine scope](#engine-scope). + +--- + +## Features + +- **Users, groups and roles.** Roles grant the right to call *functions* (gated by a paramcheck + lambda); groups grant read/write access to *tables and variables*. Group membership is transitive — + a group may itself be a member of another group. +- **Query interception.** Select/update/delete, bare variable references, named function calls, + `.q`-keyword calls (including joins, whose table arguments are checked recursively) and lambda + expressions are each classified and checked appropriately. +- **Virtual tables.** A named view of a table with an implicit where-clause spliced into any select + against it, so a group can be granted a filtered slice rather than the whole table. +- **Pluggable authentication.** `local` (md5 hash) and `ldap` backends, selected per user by the + `authtype` on their user row. +- **Read-only mode.** A runtime flag that routes evaluation through `reval` instead of `eval`. +- **Result size cap.** Serialized results larger than `maxsize` are refused. +- **Anonymous access.** Optional auto-provisioning of public users, torn down on disconnect. + +--- + +## Dependencies + +| Dependency | Key | Required | Description | +|---|---|---|---| +| logger | `` `log `` | yes | dict with `info`, `warn`, `error`, each binary `{[c;m]}` — symbol context, string message | +| handlers | `` `handlers `` | yes | dict with `register`, `remove`, `list` — see `di.handlers` | +| ldap bind | `` `ldapbind `` | **no** | `{[session;dict]}` returning a dict with a `` `ReturnCode `` key (`0i` = success). Replaces the native LDAP library entirely when supplied — see [LDAP coverage](#ldap-coverage--the-bind-path-is-exercised-via-an-injected-ldapbind) | + +**No hard dependencies on other `di.*` modules** — `deps.q` is empty and the module is standalone. + +Both dependencies are **required and never defaulted**; `init` throws immediately if either is absent, +malformed, or missing keys. There is no fallback logger. That matters more here than elsewhere: legacy +`permissions.q` logs nothing at all, so every rejected login and denied query is currently silent, and +a silent fallback would make that silence look deliberate. + +> **Note on `di.api`.** TorQ's `lamq` enumerated every variable in every root namespace via +> `.api.varnames`/`.api.allns`, then intersected that list with the tokens in the query. `di.api` is +> registry-only and does not expose those functions, by design — module code lives in each module's +> private `.z.m`, so a root-namespace scan would find nothing useful. +> +> This module does **not** reimplement that walk. It inverts the algorithm: tokenise the query first, +> then test only those tokens for being defined root variables. Same result, but O(tokens) rather than +> O(all names) — measured at 0.065 ms per lambda query against 2.9 ms for the walk on a process with +> 5000 root names. + +--- + +## Initialisation + +```q +perms:use`di.permissions +handlers:use`di.handlers + +logdep:`info`warn`error!( + {[c;m] -1 string[c],": INFO ",m;}; + {[c;m] -1 string[c],": WARN ",m;}; + {[c;m] -2 string[c],": ERROR ",m;}); + +handlers.init[enlist[`log]!enlist logdep]; +handlersdep:`register`remove`list!(handlers.register;handlers.remove;handlers.list); + +perms.init[`enabled`readonly!(1b;0b);`log`handlers!(logdep;handlersdep)]; +``` + +`init` must be called before any other function. It is **idempotent**: a second call re-wires the +dependencies and config and reclaims the same handler registrations, leaving grant data intact. + +When `enabled` is `0b` (the default) `init` wires the logger, logs that it is disabled, and stops — no +handlers are registered and nothing is published at root. + +--- + +## Configuration + +| Key | Default | Description | +|---|---|---| +| `enabled` | `0b` | master switch; when off, nothing is registered or published | +| `engine` | `` `rbac `` | authorization engine. Only `rbac` is implemented — `` `tiered `` is rejected | +| `maxsize` | `200000000` | maximum serialized size of any returned result | +| `runmode` | `1b` | `1b` executes the query, `0b` returns a boolean verdict only | +| `permissivemode` | `0b` | when `1b`, an object with no grants at all is readable by default | +| `readonly` | `0b` | route evaluation through `reval`, blocking writes | +| `public` | `0b` | allow anonymous users to be auto-provisioned on login | +| `ignorelist` | `()` | message heads that bypass the check on `.z.ps` — see below | +| `grantdirs` | `()` | directories holding grant files, loaded by `loadpermissions` | +| `proctype` | `` ` `` | process type, selects `{proctype}.q` in the grant cascade | +| `procname` | `` ` `` | process name, selects `{procname}.q` in the grant cascade | +| `publishroot` | `1b` | expose the legacy `.pm.*` names at root. Set `0b` if you have no legacy grant files — the module still enforces, it just leaves the root namespace untouched | +| `ldapenabled` | `0b` | enable the LDAP backend and load its native library | +| `ldaplibpath` | `""` | path to the LDAP `.so`; falls back to `$KDBLIB` | +| `ldapdebug` | `0i` | log LDAP chatter at info level | +| `ldapservers` | `` enlist `$"ldap://localhost:0" `` | LDAP server URIs | +| `ldapversion` | `3` | LDAP protocol version | +| `ldapblocktime` | `0D00:30:00` | how long a locked-out user stays locked out; null means forever | +| `ldapchecklimit` | `3` | failed attempts before lockout | +| `ldapchecktime` | `0D00:05` | window in which a repeat login skips the server | +| `ldapbuilddnsuf` | `""` | suffix used when building the bind DN | +| `ldapbuilddn` | `{"uid=",string[x],",",…}` | function building the bind DN from a username | + +Unrecognised keys are **warned about**, not silently dropped. Every key is uniquely named so it +survives `di.config`'s flat cascade — note the `ldap*` prefixes, which exist because legacy ships four +separate `enabled` settings that would otherwise collapse onto one another. + +### ⚠ `ignorelist` defaults to empty, unlike TorQ + +TorQ's `zpsignore.q` ships **enabled** with `` (`upd;"upd";`.u.upd;".u.upd") ``, exempting those from +permission checks on `.z.ps`. Silently exempting `upd` is not a safe default for an access-control +module, so this ships empty. **A process that receives `.u.upd`-shaped feed traffic must set it +explicitly**, or that traffic will be permission-checked and rejected: + +```q +perms.init[`enabled`ignorelist!(1b;(`upd;"upd";`.u.upd;".u.upd"));deps] +``` + +It is a **mixed** list — the head of an incoming message is matched against both symbol and string +forms. It applies to `.z.ps` only, matching TorQ; `.z.pg` is never exempted. + +--- + +## Exported functions + +### `init[config;deps]` +Wire dependencies, resolve config, and (when enabled) publish root names, load grants and register +handlers. Idempotent. + +### `teardown[]` +Release everything `init` installed: handler registrations, `.h.val`, and the published `.pm.*` root +names. Grant data survives, so a later `init` re-registers and re-publishes cleanly. + +### `allowed[user;query]` +Would this user be permitted to run this query? Never executes it. +```q +perms.allowed[`alice;"select from trade"] / 1b +``` + +> **`allowed` is a true predicate.** It returns a boolean and never executes the query. Two earlier +> caveats have been fixed: it now descends into `.q`-keyword joins (so it agrees with `requ` rather +> than permitting joins `requ` refuses), and a forbidden lambda expression returns `0b` instead of +> raising. +> +> **One caveat remains:** it **ignores `permissivemode`**, pinning it off regardless of config — +> inherited from TorQ, which fixes `allowed:mainexpr[;;0b;0b]`. On a permissive-mode process `allowed` +> will therefore deny things `requ` permits. `requ` is the authority. + +### `requ[user;query]` +Permission-check a query as a user and execute it. Passes the query through untouched when the module +is disabled. + +### `val[expr]` / `valp[expr]` +Evaluate a parse tree / a string or parse tree, under `reval` when read-only mode is on. TorQ binds +these at **load** time (`val:$[readonly;reval;eval]`), so read-only could not be toggled without a +restart; here the choice resolves per call. + +### `execas[query;user]` +Run a query as another user, subject to that user's permissions. + +### `admin` +The grant-administration sub-API. `admin.wildcard` is the wildcard object (`` `$"*" ``) — grant against +it for superuser rights. + +| Group | Functions | +|---|---| +| users | `adduser` `removeuser` `cloneuser` | +| groups | `addgroup` `removegroup` `addtogroup` `removefromgroup` | +| roles | `addrole` `removerole` `assignrole` `unassignrole` | +| functions | `addfunction` `removefunction` `grantfunction` `revokefunction` | +| tables | `grantaccess` `revokeaccess` | +| virtual tables | `createvirtualtable` `removevirtualtable` | +| anonymous | `addpublic` `removepublic` | + +```q +perms.admin.addrole[`reader;"may select"]; +perms.admin.grantfunction[`select;`reader;{1b}]; +perms.admin.addgroup[`traders;"trading desk"]; +perms.admin.grantaccess[`trade;`traders;`read]; +perms.admin.adduser[`alice;`local;`md5;md5 "secret"]; +perms.admin.assignrole[`alice;`reader]; +perms.admin.addtogroup[`alice;`traders]; +``` + +> **Paramchecks must be functions.** They are applied to the call's parameter dict under protection, +> and any non-boolean result is coerced to `0b` — so a literal `1b` stored as a paramcheck **fails +> closed**. `grantfunction` rejects a non-function outright. + +### `loadpermissions[]` +Load the grant cascade — `default` → `proctype` → `procname` — from every configured `grantdirs` +directory. Missing files are skipped with an info log. + +### `unblock[user]` +Clear a user's cached LDAP state — both a lockout and any cached successful authentication. + +> TorQ's equivalent returns early unless the user is actually *blocked*, which leaves no way to force +> re-authentication for a user who is merely cached: after a password change their cached success +> stands until `ldapchecktime` elapses. This clears `success` too, so the next login always reaches +> the server. + +### `status[]` +What the module is currently enforcing: engine, read-only state, permissive mode, run mode, maxsize, +public access, and LDAP availability. Legacy has no equivalent introspection. + +### `version` +The module version string. + +### `getapimeta[]` +This module's api metadata — one `` `name`public`descrip`params`return `` row per callable export, for +`di.torq` to collect and register with `di.api`. `init` and `getapimeta` are deliberately absent: they +are framework plumbing `di.torq` calls by convention rather than discovers. Needs no `init`. + +--- + +## Input validation + +Every public entry point validates its arguments and reports failures the same way as everything else +— prefixed `di.permissions:`, naming the offending argument and the expected type, and logged at +`error` before being signalled: + +```q +perms.allowed["alice";"1+1"] +/ 'di.permissions: allowed: user must be a symbol, got 10h +perms.admin.grantaccess[`t;`g;`sideways] +/ 'di.permissions: grantaccess: level must be `read or `write, got `sideways +``` + +This matters because the alternative is a raw `'type` or `'length` thrown from a downstream `upsert` — +unprefixed, unlogged, and giving no indication which argument was wrong. **Malformed client queries +are covered too**: an unparseable string yields +`di.permissions: mainexpr: could not parse query: …` rather than q's bare parse error, so a client +probing with garbage still leaves an audit trail. + +--- + +## Handler registration + +| Event | Phase | Behaviour | +|---|---|---| +| `.z.pw` | `exec` | authenticate; dispatches on the user's `authtype` | +| `.z.pg` | `exec` | permission check, read-only selection, maxsize | +| `.z.ps` | `exec` | as `.z.pg`, plus the ignore-list bypass | +| `.z.pi` | `exec` | console input, console-formatted results | +| `.z.pp` | `exec` | HTTP POST refused outright | +| `.z.ws` | `exec` | websocket messages refused outright | +| `.z.pc` | `` ` `` | simple observer — anonymous user cleanup | +| `.h.val` | — | assigned directly; **not** a `.z.*` event | + +All registrations use the stable name `` `di.permissions ``, so re-init reclaims rather than collides. + +**`.z.ph` is deliberately not claimed.** HTTP GET permissioning happens via `.h.val` on kdb+ 3.5+, +which is what `permissions.q` itself does; claiming `.z.ph`'s `exec` would replace KDB-X's built-in +HTTP handler wholesale, including its response formatting. + +`exec` ownership is not a preference. `di.handlers` rejects a `pre`/`post` registration when no `exec` +owner exists, and on a bare process nothing owns `.z.pg` — so a `pre`-only design cannot register at +all. It is also the only way to reproduce the three structurally different composition idioms legacy +used on `.z.pw` alone (flat replace, gate-and-call-through, AND-compose) within a single-owner model. + +--- + +## Root-name publication + +`use` mangles module code into a private namespace, so anything an evaluated config file must reach +has to be published at a real root name — the convention TorqX applies for `.gw.*`, `.u.upd` and +`.hdb.reload`. + +Legacy grant files (`config/permissions/*.q`) are **executable q** calling `.pm.addrole`, +`.pm.grantfunction`, `.pm.ALL` and so on at root. `init` therefore publishes eight names — the seven +grant-script functions plus `ALL` — under `.pm`, so a legacy grant file loads unmodified: + +``` +.pm.ALL .pm.adduser .pm.addgroup .pm.addrole +.pm.addtogroup .pm.assignrole .pm.grantaccess .pm.grantfunction +``` + +Published **permanently during `init`**, but **only when `enabled`**, and removed by `teardown`. + +> This last point diverges from TorQ, which defines `.pm.*` regardless of `enabled`. It is safe +> because `gateway.q` guards on existence (`` `.pm.valp ~ key `.pm.valp ``) and falls back cleanly, and +> it is better: a disabled permissions module should not advertise admin functions that gate nothing. + +The query API (`requ`, `allowed`, `val`, `valp`, `execas`) is **not** published at root — `di.gateway` +holds a module handle and calls it in-process. + +--- + +## Engine scope + +TorQ ships two independent authorization systems. Only `permissions.q`'s **RBAC** engine is ported. +`controlaccess.q`'s tiered model (superuser/poweruser/defaultuser, host allowlisting, per-user token +lists) is deferred, and `` engine:`tiered `` is rejected with a clear message. + +The evidence: **nothing outside `controlaccess.q` references `.access.*` anywhere in TorQ**, against +`.pm`'s three real external callers (`gateway.q`, `execas.q`, `apidetails.q`). Both engines ship +disabled. The `engine` config key exists from v1 so the tiered engine can land later without reshaping +this schema. + +--- + +## Running tests + +> **Note on suite structure.** Rows share accumulated state (users, groups and grants created by +> earlier rows), so an individual assertion cannot be run in isolation and an early failure cascades — +> a property of the k4unit CSV format rather than a choice here. When debugging, read upward from the +> first failing row, not just the row itself. +> +> The suite **is** verified re-runnable: calling `moduletest` twice in one session passes both times. +> That was not true initially — grant data and LDAP cache state surviving `teardown` broke ten +> assertions on a second run, which is what surfaced the `unblock` gap documented above. + +**Unit suite** (`test.csv`) — 111 rows, 67 assertions, no sockets and no native library required: +```q +k4unit:use`di.k4unit +k4unit.moduletest`di.permissions +``` + +### LDAP coverage — the bind path is exercised via an injected `ldapbind` + +`deps` accepts an **optional `ldapbind`** — a function `{[session;dict]}` returning a dict with a +`` `ReturnCode `` key. When supplied it replaces the native library outright, and the `.so` is never +resolved. This exists so the caching and lockout logic can be exercised without a directory server: + +```q +fakebind:{[sess;d] enlist[`ReturnCode]!enlist 0i} / 0i = success, anything else = failure +perms.init[`enabled`ldapenabled!(1b;1b);`log`handlers`ldapbind!(logdep;handlersdep;fakebind)] +``` + +This is a **`deps` injection, not a config value** — deps are process wiring code the module already +trusts completely (the injected `log` and `handlers` could subvert it just as thoroughly), so it adds +no trust boundary that did not already exist. Operator-editable settings files cannot reach it. + +Now covered by the suite: a successful bind; the skip-the-server path (a repeat login inside +`ldapchecktime` with the same password does **not** reach the server, verified by counting calls); a +different password going back to the server; failed attempts accumulating to `ldapchecklimit` and +locking the user out; a locked-out user being refused **without** reaching the server; `unblock` +clearing a lockout; `ldapblocktime` expiry releasing one without an explicit unblock; and a bind that +throws failing closed rather than propagating. + +The bind's **return shape is validated**: it must be a dictionary containing a `` `ReturnCode `` key. +This is a safety check, not tidiness — `result[\`ReturnCode]` on an integer is *handle apply* in q, so +a bind mistakenly returning `42` would attempt an IPC write to file descriptor 42. Anything of the +wrong shape now fails closed with a clear message. + +> **⚠ Security note on the cache.** After a successful bind, a repeat login by the same user with the +> same password inside `ldapchecktime` (default 5 minutes) is served **from the cache without +> contacting the server**. That is TorQ's behaviour and it is deliberate — it exists to spare the +> directory server load — but it means **revoking an account server-side is not effective until the +> window elapses**. Set `ldapchecktime` to `0D00:00` to disable the optimisation and force every login +> to the server. + +**The native path has since been verified against a real `kdbldap.so`** found on this machine: all +four symbols bind at the arities this module uses (`kdbldap_init`/2, `kdbldap_set_option`/3, +`kdbldap_bind_s`/4, `kdbldap_err2string`/1), `initialise` opens a session, a real `kdbldap_bind_s` +executes, and its failure is decoded by the native `err2string` into +`"Can't contact LDAP server"`. The lockout bookkeeping was driven by those genuine failures — three +attempts, then lockout, then refusal without contacting the server. + +Still not covered, and genuinely needing a live directory: a **successful** bind, and +directory-specific behaviour such as referrals or TLS negotiation. + +**Integration suite** (`test_integration.csv`) — stands up a real child q process on an OS-assigned +port and drives a real connection. It exists because **`reval`'s read-only restriction is not applied +when `.z.w=0`**: at the console `reval parse "g::1"` happily sets `g`, but over a real handle the same +call throws `'noupdate`. k4unit runs in-process at handle 0, so a unit test asserting a blocked write +would fail against correct code. `moduletest` only loads `test.csv`, so run this one directly: +```q +k4unit:use`di.k4unit +.m.di.0k4unit.KUltf .Q.dd[hsym`$.Q.m.mp`di.permissions;`test_integration.csv] +.m.di.0k4unit.KUrt[] +k4unit.getresults[] +``` +Run it in a fresh session — running it after `moduletest` re-runs the still-loaded unit tests against +dirty module state. + +--- + +## Migrating from `.pm` / `.access` + +- `.pm.allowed`, `.pm.requ`, `.pm.val`, `.pm.valp`, `.pm.execas` → call through the module handle. + `val`/`valp` keep the same *shape* (unary functions), so a consumer that copies the function value + is unaffected; only the read-only decision moved from load time to call time. +- `.pm.cando` is **dropped** — it had no callers anywhere and differed from `allowed` only by parsing + first, which `allowed` now does itself. +- Grant scripts need **no change** — the names they call are republished at root. +- `.access.*` has no equivalent; see [Engine scope](#engine-scope). +- The `-public` command-line flag is replaced by the `public` config key. +- **Rejection messages changed prefix.** TorQ emits `"pm: no read permission on [x]"`; this module + emits `"di.permissions: : no read permission on [x]"`. Any log scraping or alerting that + greps for `pm:` needs updating — the new prefix does not contain it. +- **`.pm.*` at root is a compatibility shim, not the API.** The eight published names exist so legacy + grant files load unmodified. New code should hold a module handle and call `perms.admin.*`; the root + namespace looks like legacy TorQ but is a strict subset of it. + +--- + +## Notes + +- `init` must be called before any other function; every other export checks and errors clearly. +- All errors raised after `init` are logged at `error` before being signalled. +- Console input (`.z.pi`) routes through the same permission check as a network query — being local is + not an exemption. The `.z.w=0` bypass inside the query path is what keeps the console usable. +- `.z.pp` and `.z.ws` are refused outright rather than permission-checked, matching TorQ. +- Group membership is chased to a fixed point for *authorization* checks, so nested groups work. +- **Public-user detection deliberately does not do that.** It keeps TorQ's first-row lookup + (`` `public~(1!usergroup)[u]`groupname ``), which is correct by construction — an anonymous user is + provisioned into exactly one group. Generalising it to a full membership check would be a privilege + change rather than a fix: a real user who is in `public` alongside other groups would then be + rejected when presenting a valid password, or, with an empty password, have their user row upserted + over and their role demoted to `publicuser`. The narrower check is the safer contract. +- LDAP is entirely optional: with `ldapenabled:0b` (the default, matching TorQ's shipped settings) the + native library is never resolved and the whole test suite runs with no `.so` present. +- **Config values are type-checked at `init`.** A mistyped setting (`maxsize:"big"`, `readonly:"yes"`, + a non-timespan `ldapblocktime`) is rejected immediately, naming every offending key, rather than + surfacing later as a confusing runtime error far from its cause. +- **A failed `init` installs nothing.** The one step that depends on external state — resolving the + LDAP native library — runs before anything is published or registered, so a missing `.so` leaves the + process untouched rather than half-configured. +- **A grant made against a virtual table's *name* survives `removevirtualtable`.** The two are + independent objects, so `allowed` will still permit the name until the grant is revoked separately; + execution then fails because the name no longer resolves. Revoke the grant as well as removing the + view. +- `grantdirs` accepts a single directory as a bare string or a list of directories. `ignorelist` is a + mixed list, matching how message heads arrive. diff --git a/di/permissions/permissions.q b/di/permissions/permissions.q new file mode 100644 index 00000000..6addf5d7 --- /dev/null +++ b/di/permissions/permissions.q @@ -0,0 +1,1094 @@ +/ role-based access control and authentication +/ ported from TorQ code/handlers/permissions.q (.pm), writeaccess.q (.readonly), ldap.q (.ldap) +/ and code/common/execas.q; controlaccess.q's tiered engine is deliberately deferred (see the engine +/ config key, which ships from v1 so the tiered engine can land later without reshaping this schema) + +/ module version - placeholder only; no project-wide version / di.depcheck convention exists yet +version:"0.1.0"; + +/ ============================================================ +/ constants (load-time) +/ ============================================================ + +/ wildcard object - grants against it mean "any function" / "any table", i.e. superuser +/ NB: TorQ calls this .pm.ALL; the style guide requires lowercase and `all` is a reserved word, so it +/ is `wildcard` internally and republished at root as .pm.ALL for legacy grant scripts (see publishroot) +wildcard:`$"*"; + +/ the admin functions a legacy grant file (config/permissions/*.q) calls at root, plus the wildcard +/ constant. these are the names publishroot must expose - see the root-publication section +grantscriptnames:`adduser`addgroup`addrole`addtogroup`assignrole`grantaccess`grantfunction; + +/ the engines this module knows about; only rbac is implemented in v1 +knownengines:`rbac`tiered; + +/ rejection messages, keyed by reason. text ported from permissions.q, minus its "pm: " prefix - +/ every one of these is signalled through raiseerror, which composes "di.permissions: : " itself, +/ so keeping the legacy prefix would double it up +err:(`symbol$())!(); +err[`func]:{"user role does not permit running function [",string[x],"]"}; +err[`selt]:{"no read permission on [",string[x],"]"}; +err[`selx]:{"unsupported select statement, superuser only"}; +err[`updt]:{"no write permission on [",string[x],"]"}; +err[`expr]:{"unsupported expression, superuser only"}; +err[`quer]:{"free text queries not permissioned for this user"}; +err[`size]:{"returned value exceeds maximum permitted size"}; + +/ ============================================================ +/ schema (load-time templates - the live copies are .z.m.*, populated in init) +/ ============================================================ + +userschema:([id:`symbol$()]authtype:`symbol$();hashtype:`symbol$();password:()); +groupinfoschema:([name:`symbol$()]description:()); +roleinfoschema:([name:`symbol$()]description:()); +usergroupschema:([]user:`symbol$();groupname:`symbol$()); +userroleschema:([]user:`symbol$();role:`symbol$()); +functiongroupschema:([]function:`symbol$();fgroup:`symbol$()); +accessschema:([]object:`symbol$();entity:`symbol$();level:`symbol$()); +functionschema:([]object:`symbol$();role:`symbol$();paramcheck:()); +virtualtableschema:([name:`symbol$()]table:`symbol$();whereclause:()); +publictrackschema:([name:`symbol$()]handle:`int$()); + +/ ldap login-attempt cache +/ NB: TorQ's cache also carries server/port columns, populated from .ldap.server and .ldap.port - +/ neither of which is defined anywhere in ldap.q (the setting is `servers`, plural), so that upsert +/ throws on the first login attempt. nothing ever reads the two columns back, so they are dropped +/ here rather than fixed: they are write-only columns fed by a broken write +ldapcacheschema:([user:`symbol$()]pass:();time:`timestamp$(); + attempts:`long$();success:`boolean$();blocked:`boolean$()); + +/ ============================================================ +/ config defaults +/ ============================================================ + +/ every key this module accepts, with its default. init warns on anything else rather than dropping it +/ silently. keys are uniquely named so they survive di.config's flat cascade - notably the ldap block is +/ prefixed, because legacy ships four separate `enabled` settings (.pm .access .readonly .ldap) that +/ would otherwise collapse onto one another +/ ignorelist is a MIXED list (symbols and strings) - TorQ's zpsignore.q matches the head of an async +/ message against both forms, e.g. (`upd;"upd";`.u.upd;".u.upd"), so it cannot be a typed symbol vector. +/ it defaults EMPTY here, unlike zpsignore.q which ships enabled with that list: silently exempting +/ upd from permission checks is not a safe default for an access-control module. a process that takes +/ .u.upd-shaped feed traffic must set it explicitly - see permissions.md +/ publishroot: expose the legacy .pm.* names at root so an unmodified TorQ grant file loads. defaults +/ ON for migration compatibility, but a deployment with no legacy grant files can set it 0b and leave +/ the root namespace untouched - the module's own API is reached through the module handle regardless +configdefaults:`enabled`engine`maxsize`runmode`permissivemode`readonly`public`ignorelist`grantdirs`proctype`procname`publishroot! + (0b;`rbac;200000000;1b;0b;0b;0b;();();`;`;1b); + +/ ldap settings. ldapenabled defaults OFF, matching TorQ's shipped config/settings/default.q rather +/ than ldap.q's own file default of (.z.o~`l64) - so the suite runs with no native library present +/ every ldap setting is read from .z.m.config at call time (one storage location, no bare-name +/ ambiguity) - including by the default ldapbuilddn below, which is why it is explicit about it +ldapconfigdefaults:`ldapenabled`ldaplibpath`ldapdebug`ldapservers`ldapversion`ldapblocktime`ldapchecklimit`ldapchecktime`ldapbuilddnsuf`ldapbuilddn! + (0b;"";0i;enlist `$"ldap://localhost:0";3;0D00:30:00;3;0D00:05;"";{"uid=",string[x],",",.z.m.config`ldapbuilddnsuf}); + +/ ============================================================ +/ internal helpers +/ ============================================================ + +requiresym:{[ctx;nm;x] + / validate a public-api argument that must be a symbol + / without this a wrong type escapes as a raw 'type or 'length from a downstream upsert - unprefixed, + / unlogged, and with no indication which argument was wrong + if[-11h<>type x; + raiseerror[ctx;nm," must be a symbol, got ",.Q.s1 type x]]; + }; + +requirequery:{[ctx;q] + / a query must be a string or a parse tree - not an atom, and not a bare symbol + if[type[q] within -19 -1h; + raiseerror[ctx;"query must be a string or a parse tree, got an atom of type ",.Q.s1 type q]]; + }; + +raiseerror:{[ctx;msg] + / log an error under ctx then signal it, so a failure is observable in the log and not only as a throw + .z.m.logerr[ctx;msg]; + '"di.permissions: ",string[ctx],": ",msg; + }; + +initialised:{[] + / has init run? a direct (module-rewritten) reference detects prior setup without touching root + :@[{.z.m.enabled;1b};::;0b]; + }; + +requireinit:{[ctx] + / every exported function except init depends on init having wired the logger and the tables + if[not initialised[]; + '"di.permissions: ",string[ctx],": init must be called before any other function"]; + }; + +/ ============================================================ +/ init +/ ============================================================ + +validatedeps:{[deps] + / log and handlers are both required and never defaulted - there is no fallback logger. + / legacy permissions.q logs nothing at all, so every rejected login and denied query is currently + / silent; a silent fallback here would make that silence look deliberate + if[99h<>type deps; + '"di.permissions: deps must be a dict with `log and `handlers keys - see di.log, di.handlers"]; + if[not all `log`handlers in key deps; + '"di.permissions: log and handlers dependencies are required; pass `log (`info`warn`error) ", + "and `handlers (`register`remove`list) - see di.log, di.handlers; got: ",(", " sv string key deps)]; + if[99h<>type deps`log; + '"di.permissions: log value must be a dict; pass `info`warn`error functions - see di.log"]; + if[not all `info`warn`error in key deps`log; + '"di.permissions: log dict must have `info`warn`error keys; got: ",(", " sv string key deps`log)]; + if[99h<>type deps`handlers; + '"di.permissions: handlers value must be a dict; pass `register`remove`list functions - see di.handlers"]; + if[not all `register`remove`list in key deps`handlers; + '"di.permissions: handlers dict must have `register`remove`list keys; got: ",(", " sv string key deps`handlers)]; + / ldapbind is OPTIONAL - when supplied it replaces the native bind entirely (see ldap.bind) + if[`ldapbind in key deps; + if[not type[deps`ldapbind] within 100 112h; + '"di.permissions: ldapbind must be a function taking (session;dict) and returning a dict with a `ReturnCode key"]]; + }; + +resolveconfig:{[config] + / merge the caller's config over the known-key defaults, warning about anything unrecognised rather + / than dropping it silently + defaults:configdefaults,ldapconfigdefaults; + if[config~(::); :defaults]; + if[99h<>type config; + '"di.permissions: config must be a dict of settings, or (::) for defaults"]; + if[count unknown:(key config) except key defaults; + .z.m.logwarn[`init;"ignoring unrecognised config key(s): ",", " sv string unknown]]; + :defaults,(key[defaults] inter key config)#config; + }; + +/ expected value shapes, grouped by check. a mistyped setting must fail at init with a clear message +/ rather than surfacing later as a confusing runtime error far from its cause +/ engine is deliberately absent - validateengine gives a better message for it +/ ignorelist and grantdirs are deliberately absent - both accept several shapes and are normalised +boolconfigkeys:`enabled`readonly`permissivemode`runmode`public`ldapenabled`publishroot; +intconfigkeys:`maxsize`ldapversion`ldapchecklimit`ldapdebug; +symconfigkeys:`proctype`procname; +strconfigkeys:`ldaplibpath`ldapbuilddnsuf; +spanconfigkeys:`ldapblocktime`ldapchecktime; + +validateconfig:{[cfg] + / type-check every setting whose shape is fixed, reporting all offenders of a kind at once + / NB: the parameter is `ks`, NOT `keys` - a parameter named `keys` throws 'nyi when the function is + / called, even though (`keys in .Q.res) is 0b. .Q.res is not exhaustive; test, don't trust it + chk:{[cfg;ks;ok;what] + bad:ks where not ok each cfg ks; + if[count bad; + raiseerror[`init;"config key(s) ",(", " sv string bad)," must be ",what]]; + }; + chk[cfg;boolconfigkeys;{-1h=type x};"a boolean (1b or 0b)"]; + chk[cfg;intconfigkeys;{type[x] in -6 -7h};"an integer"]; + chk[cfg;symconfigkeys;{-11h=type x};"a symbol"]; + chk[cfg;strconfigkeys;{10h=type x};"a string"]; + chk[cfg;spanconfigkeys;{-16h=type x};"a timespan (e.g. 0D00:30:00)"]; + chk[cfg;enlist`ldapbuilddn;{type[x] within 100 112h};"a function taking a username"]; + chk[cfg;enlist`ldapservers;{11h=abs type x};"a symbol or symbol list"]; + }; + +validateengine:{[eng] + / v1 implements rbac only. tiered (TorQ's controlaccess.q) is deferred, but the key ships now so it + / can land later without reshaping the config schema + if[not -11h=type eng; + raiseerror[`init;"engine must be a symbol, one of: ",", " sv string knownengines]]; + if[eng~`tiered; + raiseerror[`init;"engine `tiered (TorQ controlaccess.q) is not implemented in this version; use `rbac"]]; + if[not eng in knownengines; + raiseerror[`init;"unknown engine ",string[eng],"; expected one of: ",", " sv string knownengines]]; + }; + +resettables:{[] + / install fresh copies of every schema - called on first init only, so a re-init preserves grants + .z.m.user:userschema; + .z.m.groupinfo:groupinfoschema; + .z.m.roleinfo:roleinfoschema; + .z.m.usergroup:usergroupschema; + .z.m.userrole:userroleschema; + .z.m.functiongroup:functiongroupschema; + .z.m.access:accessschema; + .z.m.function:functionschema; + .z.m.virtualtable:virtualtableschema; + .z.m.publictrack:publictrackschema; + .z.m.ldapcache:ldapcacheschema; + / ldap.initialise normally sets these; default them so an injected bind (which skips the native + / library entirely) still has a session value to pass through. + / ldapready is an EXPLICIT flag: inferring availability from ldapsession merely existing would + / report the native library as ready the moment this default was added + .z.m.ldapsession:0i; + .z.m.ldapready:0b; + }; + +init:{[config;deps] + / wire the injected dependencies, resolve config, and (when enabled) install this module as the + / owner of the message-handling .z.* events + / config: a dict of settings, or (::) for defaults. deps: a dict with `log and `handlers keys + / example: perms.init[`enabled`readonly!(1b;1b);`log`handlers!(logdep;handlersdep)] + / idempotent - a second call re-wires deps and config and reclaims the same handler registrations, + / leaving existing grant data intact + validatedeps[deps]; + .z.m.loginfo:(deps`log)`info; + .z.m.logwarn:(deps`log)`warn; + .z.m.logerr:(deps`log)`error; + .z.m.register:(deps`handlers)`register; + .z.m.removehandler:(deps`handlers)`remove; + .z.m.listhandlers:(deps`handlers)`list; + cfg:resolveconfig[config]; + validateconfig[cfg]; + validateengine[cfg`engine]; + if[not initialised[];resettables[]]; + .z.m.config:cfg; + .z.m.enabled:cfg`enabled; + .z.m.engine:cfg`engine; + .z.m.maxsize:cfg`maxsize; + .z.m.runmode:cfg`runmode; + .z.m.permissivemode:cfg`permissivemode; + .z.m.readonly:cfg`readonly; + .z.m.public:cfg`public; + .z.m.ignorelist:cfg`ignorelist; + / set before the disabled bail below, so status[] can report it either way + .z.m.ldapbind:$[`ldapbind in key deps;deps`ldapbind;(::)]; + if[not .z.m.enabled; + .z.m.loginfo[`init;"di.permissions loaded but disabled - no handlers registered, nothing published at root"]; + :(::)]; + / an injected bind replaces the native library outright, so the .so is never resolved in that case. + / resolve the native library FIRST, because it is the one step that can fail on external state. + / doing it before anything is installed means a missing .so leaves the process untouched rather than + / half-configured with root names published and no handlers registered + if[(cfg`ldapenabled) and (::)~.z.m.ldapbind;ldap.initialise[ldap.resolvelibpath[]]]; + / root names next: grant files call .pm.addrole etc. on their first line + $[cfg`publishroot;publishroot[]; + .z.m.loginfo[`init;"publishroot is 0b - .pm.* not exposed at root; legacy grant files will not load"]]; + / a grant file is arbitrary q and may throw. unwind the root publication and mark the module + / disabled before rethrowing, so a failed init never leaves .pm.* published with no handlers + / registered and status[] reporting enabled + @[loadpermissions;::;{[e] + unpublishroot[]; + .z.m.enabled:0b; + raiseerror[`init;"grant file failed to load, root names unpublished: ",e]}]; + registerhandlers[]; + .z.m.loginfo[`init;"di.permissions initialised - engine ",string[.z.m.engine],", readonly ",("disabled";"enabled").z.m.readonly]; + }; + +status:{[] + / a snapshot of what this module is currently enforcing - legacy has no equivalent introspection + requireinit[`status]; + :`enabled`engine`readonly`permissivemode`runmode`maxsize`public`publishroot`ldapenabled`ldapavailable! + (.z.m.enabled;.z.m.engine;.z.m.readonly;.z.m.permissivemode;.z.m.runmode;.z.m.maxsize;.z.m.public; + .z.m.config`publishroot; + .z.m.config`ldapenabled;$[(::)~.z.m.ldapbind;@[{.z.m.ldapready};::;0b];1b]); + }; + +/ ============================================================ +/ admin api - grant data management (the admin.* dotted group) +/ ============================================================ +/ these are what a legacy config/permissions/*.q grant file calls, and what publishroot exposes at +/ .pm.* so such a file loads unmodified. every one is an idempotent table mutation + +/ the wildcard object, exposed so a caller can grant superuser: admin.grantfunction[admin.wildcard;...] +/ TorQ exposes the same constant as .pm.ALL, which is what publishroot republishes it as +admin.wildcard:wildcard; + +admin.adduser:{[u;authtype;hashtype;password] + / register a user with an authentication method and a hashed password + requireinit[`adduser]; + requiresym[`adduser;"user id";u]; + requiresym[`adduser;"authtype";authtype]; + requiresym[`adduser;"hashtype";hashtype]; + if[u in key .z.m.groupinfo;raiseerror[`adduser;"cannot add user with same name as existing group: ",string u]]; + .z.m.user:.z.m.user upsert (u;authtype;hashtype;password); + }; + +admin.removeuser:{[u] + requireinit[`removeuser]; + .z.m.user:.[.z.m.user;();_;u]; + }; + +admin.addgroup:{[n;d] + requireinit[`addgroup]; + requiresym[`addgroup;"group name";n]; + if[n in key .z.m.user;raiseerror[`addgroup;"cannot add group with same name as existing user: ",string n]]; + .z.m.groupinfo:.z.m.groupinfo upsert (n;d); + }; + +admin.removegroup:{[n] + requireinit[`removegroup]; + .z.m.groupinfo:.[.z.m.groupinfo;();_;n]; + }; + +admin.addrole:{[n;d] + requireinit[`addrole]; + requiresym[`addrole;"role name";n]; + .z.m.roleinfo:.z.m.roleinfo upsert (n;d); + }; + +admin.removerole:{[n] + requireinit[`removerole]; + .z.m.roleinfo:.[.z.m.roleinfo;();_;n]; + }; + +admin.addtogroup:{[u;g] + / add a user to a group, giving them the group's table-level access + requireinit[`addtogroup]; + if[not g in key .z.m.groupinfo;raiseerror[`addtogroup;"no such group, call admin.addgroup first: ",string g]]; + / NB: upsert, not join. TorQ writes `usergroup,:(u;g)`, whose amend-in-place semantics insert a row; + / the explicit `.z.m.x:.z.m.x,(...)` rewrite this module needs is NOT equivalent - on an empty table + / it flattens to a plain list. upsert on an unkeyed table appends, and the guard above stops duplicates + if[not (u;g) in .z.m.usergroup;.z.m.usergroup:.z.m.usergroup upsert (u;g)]; + }; + +admin.removefromgroup:{[u;g] + requireinit[`removefromgroup]; + if[(u;g) in .z.m.usergroup;.z.m.usergroup:.[.z.m.usergroup;();_;.z.m.usergroup?(u;g)]]; + }; + +admin.assignrole:{[u;r] + / assign a user a role, giving them the role's function-level access + requireinit[`assignrole]; + if[not r in key .z.m.roleinfo;raiseerror[`assignrole;"no such role, call admin.addrole first: ",string r]]; + if[not (u;r) in .z.m.userrole;.z.m.userrole:.z.m.userrole upsert (u;r)]; + }; + +admin.unassignrole:{[u;r] + requireinit[`unassignrole]; + if[(u;r) in .z.m.userrole;.z.m.userrole:.[.z.m.userrole;();_;.z.m.userrole?(u;r)]]; + }; + +admin.addfunction:{[f;g] + / put a function into a function group, so a grant against the group covers it + requireinit[`addfunction]; + if[not (f;g) in .z.m.functiongroup;.z.m.functiongroup:.z.m.functiongroup upsert (f;g)]; + }; + +admin.removefunction:{[f;g] + requireinit[`removefunction]; + if[(f;g) in .z.m.functiongroup;.z.m.functiongroup:.[.z.m.functiongroup;();_;.z.m.functiongroup?(f;g)]]; + }; + +admin.grantaccess:{[o;e;l] + / grant an entity (user or group) read or write access to a table or variable + requireinit[`grantaccess]; + requiresym[`grantaccess;"object";o]; + requiresym[`grantaccess;"entity";e]; + / NB: type-check BEFORE the membership test - ("read" in `read`write) throws a raw 'type, which + / would escape unprefixed and unlogged before this check could report anything useful + requiresym[`grantaccess;"level";l]; + / an unrecognised level is silently useless - it is stored but can never match a check + if[not l in `read`write; + raiseerror[`grantaccess;"level must be `read or `write, got ",.Q.s1 l]]; + if[not (o;e;l) in .z.m.access;.z.m.access:.z.m.access upsert (o;e;l)]; + }; + +admin.revokeaccess:{[o;e;l] + requireinit[`revokeaccess]; + if[(o;e;l) in .z.m.access;.z.m.access:.[.z.m.access;();_;.z.m.access?(o;e;l)]]; + }; + +admin.grantfunction:{[o;r;p] + / grant a role the right to call a function, gated by paramcheck p + / p MUST be a function - a paramcheck is applied to the call's parameter dict and any non-boolean + / result is coerced to 0b, so a literal 1b stored here fails closed rather than granting access + requireinit[`grantfunction]; + if[not type[p] within 100 112h;raiseerror[`grantfunction;"paramcheck must be a function; a literal fails closed"]]; + if[not (o;r;p) in .z.m.function;.z.m.function:.z.m.function upsert (o;r;p)]; + }; + +admin.revokefunction:{[o;r] + requireinit[`revokefunction]; + t:`object`role#.z.m.function; + if[(o;r) in t;.z.m.function:.[.z.m.function;();_;t?(o;r)]]; + }; + +admin.createvirtualtable:{[n;t;w] + / expose a named view of a table with an implicit where-clause spliced into any select against it + requireinit[`createvirtualtable]; + if[not n in key .z.m.virtualtable;.z.m.virtualtable:.z.m.virtualtable upsert (n;t;w)]; + }; + +admin.removevirtualtable:{[n] + requireinit[`removevirtualtable]; + if[n in key .z.m.virtualtable;.z.m.virtualtable:.[.z.m.virtualtable;();_;n]]; + }; + +admin.addpublic:{[u;w] + / track an auto-provisioned anonymous user against the handle that created it + requireinit[`addpublic]; + .z.m.publictrack:.z.m.publictrack upsert (u;w); + }; + +admin.removepublic:{[u] + requireinit[`removepublic]; + .z.m.publictrack:.[.z.m.publictrack;();_;u]; + }; + +admin.cloneuser:{[u;unew;p] + / copy a user's auth method plus group and role membership onto a new user id + requireinit[`cloneuser]; + if[not u in key .z.m.user;raiseerror[`cloneuser;"no such user to clone: ",string u]]; + ul:raze exec authtype,hashtype from .z.m.user where id=u; + admin.adduser[unew;ul 0;ul 1;value (string ul 1)," string `",p]; + admin.addtogroup[unew;] each exec groupname from .z.m.usergroup where user=u; + admin.assignrole[unew;] each exec role from .z.m.userrole where user=u; + }; + +/ ============================================================ +/ rbac engine - permission checks +/ ============================================================ + +rbac.pdict:{[f;a] + / build a parameter-name -> value dict for a call, so a paramcheck can inspect arguments by name + / handles bare calls, select, and projections (rebuilding the full argument list from the + / projection's captured args plus the new ones) + d:enlist[`]!enlist[::]; + d:d,$[not ca:count a; (); + f~`select; (); + (1=count a) and (99h=type first a); first a; + 104h=type value f; [fnfp:value value f; (value[fnfp 0][1])!fnfp[1],a]; + 101h<>type fp:value[value[f]][1]; fp!a; + ((),(`$string til ca))!a + ]; + :d; + }; + +rbac.fchk:{[u;f;a] + / may user u call function f with args a? + / any one satisfied paramcheck is sufficient - a wildcard (superuser) grant therefore trumps a + / failed paramcheck from another role + r:exec role from .z.m.userrole where user=u; + o:wildcard,f,exec fgroup from .z.m.functiongroup where function=f; + c:exec paramcheck from .z.m.function where (object in o),role in r; + k:@[;rbac.pdict[f;a];::] each c; + k:`boolean$@[k;where not -1h=type each k;:;0b]; + :max k; + }; + +rbac.achk:{[u;t;rw;pr] + / may user u read/write table t? pr is permissive mode - an object with no grants at all is allowed + if[rbac.fchk[u;wildcard;()]; :1b]; + if[pr and not t in key 1!.z.m.access; :1b]; + t:wildcard,t; + / groups can contain groups - chase membership to a fixed point + g:raze over (exec groupname by user from .z.m.usergroup)\[u]; + :exec 0=5; + }; + +rbac.xdq:{[x] + / is the head of this expression a .q keyword? + :first[x] in .q; + }; + +rbac.qexe:{[x] + / evaluate a parse tree and enforce the result-size cap + v:val x; + if[.z.m.maxsize<-22!v;raiseerror[`qexe;err[`size][]]]; + :v; + }; + +rbac.exe:{[x] + / evaluate an expression, choosing parse-tree vs string evaluation by the head's type + v:$[(104<>a)&100type s;:0b]; + if[null s;:0b]; + / a view IS a readable object and must be permission-checked - but `get` would EVALUATE it, which + / would let an unpermissioned caller trigger arbitrary view computation before any check runs. + / recognise it by name instead. views[] lists every view in the process: only a root-level :: + / creates a lazy view, a namespaced one (.ns.x::) is evaluated eagerly at definition and is an + / ordinary variable thereafter. TorQ never evaluated anything here, and neither does this + if[s in views[];:1b]; + :@[{100h>type get x};s;0b]; + }; + +rbac.lamq:{[u;e;b;pr] + / permission-check a lambda-shaped expression by finding which defined root variables it references + / and checking read access on each, reporting every disallowed reference at once + / NB: this tokenises FIRST and tests only those tokens. TorQ enumerates every variable in every root + / namespace and intersects, which is O(all names) per query - measured at 2.9ms on a process with + / 5000 root names versus 0.085ms on a small one, a 34x cliff on exactly the RDB/HDB shapes this + / module targets. Testing the handful of tokens actually referenced is O(tokens) and equivalent + pq:`$distinct -4!raze(rbac.str rbac.flatten e),'" "; + rqt:pq where rbac.isdefinedvar each pq; + / public objects are always readable + rqt:rqt except distinct exec object from .z.m.access where entity=`public; + prohibited:rqt where not rbac.achk[u;;`read;pr] each rqt; + / a dry run reports a verdict; only a real execution raises. TorQ raises either way, which makes + / `allowed` - documented as returning a boolean - throw instead for lambda-shaped queries + if[count prohibited; + $[b;raiseerror[`lamq;" | " sv err[`selt] each prohibited];:0b]]; + :$[b;rbac.exe e;1b]; + }; + +/ ============================================================ +/ rbac engine - top-level classifier +/ ============================================================ + +rbac.isvar:{[x] + / is x a symbol naming an existing non-function variable? + :$[-11h<>type x;0b;100h>type @[get;x;{[n;e] raiseerror[`isvar;err[`selt] n]}[x]]]; + }; + +rbac.mainexpr:{[u;e;b;pr] + / classify an expression and permission-check it accordingly + ie:e; + / guard the parse: client input is untrusted, and an unparseable query would otherwise escape as a + / bare q error (e.g. ' ) - unprefixed, unlogged, and with no audit trail of who sent it + e:$[10=type e;@[parse;e;{[m] raiseerror[`mainexpr;"could not parse query: ",m]}];e]; + / a bare variable reference - read check, through any virtual-table indirection + if[rbac.isvar f:first e; + if[not rbac.achk[u;f;`read;pr];$[b;raiseerror[`mainexpr;err[`selt] f];:0b]]; + :$[b;rbac.qexe $[f in key .z.m.virtualtable;exec (?;table;enlist whereclause;0b;()) from .z.m.virtualtable f;e];1b]]; + / a named function call + if[-11h=type f; + if[not rbac.fchk[u;f;1_e];$[b;raiseerror[`mainexpr;err[`func] f];:0b]]; + :$[b;rbac.exe ie;1b]]; + / select / update / delete + if[rbac.isq e;:rbac.query[u;e;b;pr]]; + / .q keyword call + if[rbac.xdq e;:rbac.dotqf[u;e;b;pr]]; + / lambda - value any dict args before razing + if[any (100 104h) in type each raze @[e;where 99h=type'[e];value];:rbac.lamq[u;ie;b;pr]]; + / unrecognised - superuser only + if[not (rbac.fchk[u;wildcard;()] or rbac.fchk[u;`$string first e;()]);$[b;raiseerror[`mainexpr;err[`expr] f];:0b]]; + :$[b;rbac.exe ie;1b]; + }; + +/ execute-and-check. TorQ binds these as load-time projections over runmode/permissivemode; here they +/ read config at CALL time, so both settings are tunable at runtime (same load-time-binding fix as +/ val/valp below) +rbac.expr:{[u;e] :rbac.mainexpr[u;e;.z.m.runmode;.z.m.permissivemode]}; + +/ ============================================================ +/ query normalisation and the public entry points +/ ============================================================ + +rbac.destringf:{[x] + :$[(s:`$x) in key `.q;.q s;s~`insert;insert;any (100h;104h)=type first f:@[parse;x;0];f;s]; + }; + +rbac.parsequery:{[q] + / normalise a string or .q-keyword-headed query into its resolved parse tree + :$[10=type q;q;10h=abs type f:first q;rbac.destringf[f],1_q;q]; + }; + +val:{[x] + / evaluate a parse tree, under reval when read-only mode is on + / TorQ fixes this at LOAD time (val:$[readonly;reval;eval]), so read-only cannot be toggled without + / a restart; resolving per call fixes that. gateway.q copies this function value, so the shape it + / consumes is unchanged + / ⚠ TESTING: reval's read-only restriction is NOT applied when .z.w=0 (the console). Verified on + / KDB-X 5f/2025.11.17: at the console `reval parse "g::1"` happily sets g, but over a real IPC + / handle the same call throws 'noupdate. k4unit runs in-process at handle 0, so a unit test that + / asserts a write is blocked will FAIL against correct code - assert the selection instead, and + / cover actual enforcement in an integration test with a child process (see di.handlers' pattern) + requireinit[`val]; + :$[.z.m.readonly;reval x;eval x]; + }; + +valp:{[x] + / evaluate a string or parse tree, under reval when read-only mode is on + requireinit[`valp]; + :$[.z.m.readonly;reval parse x;value x]; + }; + +allowed:{[u;q] + / would user u be permitted to run q? a dry-run verdict - never executes + / NB: TorQ pins permissive mode OFF here (allowed:mainexpr[;;0b;0b]) regardless of config, so on a + / permissive-mode process it denies things requ then permits. a pre-check that disagrees with the + / execution it precedes is a defect, not a feature - di.gateway gates on this - so it reads the + / configured value and the two agree + requireinit[`allowed]; + requiresym[`allowed;"user";u]; + requirequery[`allowed;q]; + :rbac.mainexpr[u;rbac.parsequery q;0b;.z.m.permissivemode]; + }; + +requ:{[u;q] + / permission-check q as user u and execute it; passes through untouched when disabled + requireinit[`requ]; + requiresym[`requ;"user";u]; + requirequery[`requ;q]; + q:rbac.parsequery q; + :$[.z.m.enabled;rbac.expr[u;q];valp q]; + }; + +execas:{[f;u] + / run f as user u, subject to that user's permissions + / TorQ's execas.q guards on .pm.requ existing; inside the module requ always exists and itself + / handles the disabled case, so the guard is redundant and dropped + requireinit[`execas]; + requiresym[`execas;"user";u]; + requirequery[`execas;f]; + :requ[u;f]; + }; + +/ ============================================================ +/ ldap authentication backend (the ldap.* dotted group) +/ ============================================================ +/ ported from TorQ code/handlers/ldap.q. the native library is OPTIONAL: nothing here is touched +/ unless ldapenabled is set, so the module and its whole test suite run with no .so present + +ldap.resolvelibpath:{[] + / the native library, from the ldaplibpath setting, falling back to $KDBLIB like TorQ + / TorQ has no @[value;...] guard on .ldap.lib, so it is overridable only via $KDBLIB; making it a + / real setting (with the same fallback) follows di.kafka's libpath pattern + p:.z.m.config`ldaplibpath; + :$[0r;raiseerror[`ldap;"error initialising ldap: ",.z.m.ldaperr2string r]]; + s:.z.m.ldapsetoption[.z.m.ldapsession;`LDAP_OPT_PROTOCOL_VERSION;.z.m.config`ldapversion]; + if[0<>s;raiseerror[`ldap;"error setting ldap protocol version: ",.z.m.ldaperr2string s]]; + .z.m.ldapready:1b; + .z.m.loginfo[`ldap;"ldap initialised against ",", " sv string .z.m.config`ldapservers]; + }; + +ldap.bind:{[sess;customdict] + / thin wrapper over the native bind, merging caller overrides onto the default dn/cred/mech dict + defaultkeys:`dn`cred`mech; + if[customdict~(::);customdict:()!()]; + if[99h<>type customdict;raiseerror[`ldap;"bind overrides must be (::) or a dictionary"]]; + upddict:(defaultkeys!```),customdict; + / dispatch through the injected bind when one was supplied, else the native library. + / the injected form exists so the caching and lockout logic below can be exercised without a real + / directory server. it is a DEPS injection, not a config value: deps are process wiring code that + / this module already trusts completely (the injected log and handlers could subvert it just as + / easily), so this adds no trust boundary that did not already exist + r:$[(::)~.z.m.ldapbind;.z.m.ldapbindnative[sess;;;]. upddict defaultkeys;.z.m.ldapbind[sess;upddict]]; + / validate the shape before returning. an unchecked non-dict is dangerous, not merely wrong: + / r[`ReturnCode] on an integer is HANDLE APPLY, so a bind returning 42 attempts an IPC write to + / file descriptor 42. fail closed with a clear message instead + if[99h<>type r; + raiseerror[`ldap;"bind returned a ",(.Q.s1 type r)," - expected a dictionary with a `ReturnCode key"]]; + if[not `ReturnCode in key r; + raiseerror[`ldap;"bind returned a dictionary with no `ReturnCode key; got: ",(", " sv string key r)]]; + :r; + }; + +ldap.errstring:{[rc] + / describe a bind return code, falling back to the raw code when no native library is loaded + :$[(::)~.z.m.ldapbind;.z.m.ldaperr2string rc;"return code ",string rc]; + }; + +ldap.blocked:{[usr;incache] + / is this user currently locked out? clears an expired lockout as a side effect + / a null blocktime means the lockout never expires + if[not incache`blocked;:0b]; + if[null .z.m.config`ldapblocktime; + ldap.debuglog["authentication attempts for user ",(string usr)," are blocked"]; + :1b]; + bt:incache[`time]+.z.m.config`ldapblocktime; + if[.z.p.z.p-.z.m.config`ldapchecktime;incache[`pass]~np); + enlist[`ReturnCode]!enlist 0i; + .[ldap.bind;(.z.m.ldapsession;`dn`cred!(dn;pass));enlist[`ReturnCode]!enlist -2i]]; + ok:authorised[`ReturnCode]~0i; + .z.m.ldapcache:.z.m.ldapcache upsert (usr;np;.z.p;$[ok;0;1+0^incache`attempts];ok;0b); + $[ok; + ldap.debuglog["successfully authenticated user ",dn]; + ldap.debuglog["failed to authenticate user ",dn,": ",ldap.errstring authorised`ReturnCode]]; + if[(.z.m.config`ldapchecklimit)<=.z.m.ldapcache[usr]`attempts; + .z.m.ldapcache:update blocked:1b from .z.m.ldapcache where user=usr; + .z.m.logwarn[`ldap;"attempt limit reached, user ",dn," has been locked out"]]; + :ok; + }; + +unblock:{[usr] + / clear a user's ldap lockout - the admin escape hatch + requireinit[`unblock]; + requiresym[`unblock;"user";usr]; + if[not usr in key .z.m.ldapcache; + .z.m.loginfo[`unblock;"no ldap login record for user ",string usr]; + :(::)]; + / clear the whole cached state, not only a lockout. TorQ returns early unless the user is blocked, + / which leaves no way to force re-authentication for a user who is merely cached - after a password + / change their cached success stands until ldapchecktime elapses. clearing `success` here means the + / next login always reaches the server + wasblocked:.z.m.ldapcache[usr]`blocked; + .z.m.ldapcache:update attempts:0,success:0b,blocked:0b from .z.m.ldapcache where user=usr; + .z.m.loginfo[`unblock;$[wasblocked;"unblocked user ";"cleared cached ldap state for user "],string usr]; + }; + +/ ============================================================ +/ authentication backends (the auth.* dotted group) +/ ============================================================ +/ one function per authtype, so a user row's authtype selects its backend + +auth.local:{[u;p] + / compare a hashed password against the stored hash. md5 is the only hashtype TorQ supports + ud:.z.m.user u; + :$[`md5~ud`hashtype;(md5 p)~ud`password;0b]; + }; + +auth.ldap:{[u;p] + / delegate to the ldap backend, which is a no-op returning 0b when ldap is disabled + :$[.z.m.config`ldapenabled;ldap.login[u;p];0b]; + }; + +/ ============================================================ +/ connection lifecycle - the bodies di.handlers registers +/ ============================================================ + +authenticate:{[u;p] + / the .z.pw body: authenticate a connecting user, optionally auto-provisioning an anonymous one + / ⚠ BUG FIX vs TorQ: legacy gates the anonymous path on `if["B"$(.Q.opt .z.x)[`public][0;0]]`. + / with no -public flag that is `if[`boolean$()]`, which throws 'type - so on any process started + / without -public, login THROWS for every unknown user instead of returning 0b. replaced with the + / `public` boolean config key. (droppublic's `any` form was accidentally safe; this one was not) + requireinit[`authenticate]; + / public detection deliberately keeps TorQ's FIRST-ROW semantics: 1! keys usergroup on user and a + / lookup returns only the first matching row. that is correct by construction here - the + / provisioning branch below puts an anonymous user in exactly one group - and generalising it to a + / full membership check would be a privilege change, not a fix: a real user who happens to be in + / `public alongside other groups would then either be REJECTED when presenting a valid password, or + / (with an empty password) have their user row upserted over and their role demoted to publicuser. + / that is an account-takeover path, so the narrower legacy check is the safer contract + known:u in key .z.m.user; + ingrouppublic:`public~(1!.z.m.usergroup)[u]`groupname; + if[(not known) or ingrouppublic; + if[not .z.m.public; + .z.m.logwarn[`authenticate;"rejected unknown user ",(string u)," (public access disabled)"]; + :0b]; + if[not ""~p; + .z.m.logwarn[`authenticate;"rejected anonymous login for ",(string u)," (a password was supplied)"]; + :0b]; + admin.adduser[u;`local;`md5;md5 p]; + admin.assignrole[u;`publicuser]; + admin.addtogroup[u;`public]; + admin.addpublic[u;.z.w]; + .z.m.loginfo[`authenticate;"provisioned anonymous user ",string u]; + :1b]; + ud:.z.m.user u; + if[not ud[`authtype] in key auth; + .z.m.logwarn[`authenticate;"rejected ",(string u),": unknown authtype ",string ud`authtype]; + :0b]; + ok:auth[ud`authtype][u;p]; + if[not ok;.z.m.logwarn[`authenticate;"failed authentication for user ",string u]]; + :ok; + }; + +droppublic:{[w] + / the .z.pc body: tear down an auto-provisioned anonymous user when its connection closes + requireinit[`droppublic]; + if[not .z.m.public;:(::)]; + tracked:exec name from .z.m.publictrack where handle=w; + if[0=count tracked;:(::)]; + u:first tracked; + admin.removeuser[u]; + admin.unassignrole[u;`publicuser]; + admin.removefromgroup[u;`public]; + admin.removepublic[u]; + .z.m.loginfo[`droppublic;"dropped anonymous user ",string u]; + }; + +/ ============================================================ +/ handler bodies - registered with di.handlers, never exported +/ ============================================================ +/ these are passed to register BY VALUE, so they need no public name + +hooks.sync:{[x] + / .z.pg exec: permission-check and run a synchronous message + / handle 0 (the console / this process itself) bypasses checking entirely, as in TorQ + :$[.z.w=0;value x;requ[.z.u;x]]; + }; + +hooks.async:{[x] + / .z.ps exec: as hooks.sync, but first honouring the ignore-list + / this is zpsignore.q's behaviour folded inline. di.handlers always calls the exec owner after + / folding the pre phase - there is no skip-exec path - so the bypass has to live here, not as a phase. + / TorQ applies it to .z.ps ONLY, and that is preserved: .z.pg is not exempted + if[any first[x]~/:.z.m.ignorelist;:value x]; + :$[.z.w=0;value x;requ[.z.u;x]]; + }; + +hooks.console:{[x] + / .z.pi exec: console input. blank lines skip the check; results are console-formatted, as in TorQ + / NB: TorQ routes console input through the same permission check as a network query - being local + / is not an exemption; the .z.w=0 bypass inside hooks.sync is what makes the console usable + :$[x in (1#"\n";"");.Q.s value x;.Q.s $[.z.w=0;value x;requ[.z.u;x]]]; + }; + +hooks.rejectpost:{[x] + / .z.pp exec: TorQ disables HTTP POST outright when permissions are on - not a check, a refusal + raiseerror[`http;"HTTP POST requests are not permitted"]; + }; + +hooks.rejectws:{[x] + / .z.ws exec: TorQ disables websocket messages outright when permissions are on + raiseerror[`websocket;"websocket access is not permitted"]; + }; + +/ event -> exec body. .z.pw is binary, the rest unary; .z.pc is a simple observer, registered separately +execbodies:`.z.pw`.z.pg`.z.ps`.z.pi`.z.pp`.z.ws! + (authenticate;hooks.sync;hooks.async;hooks.console;hooks.rejectpost;hooks.rejectws); + +/ ============================================================ +/ root-name publication +/ ============================================================ +/ `use` mangles module code into a private namespace, so anything an evaluated config file or a remote +/ caller must reach has to be published at a real root name - the convention TorqX applies for .gw.*, +/ .u.upd, .hdb.reload and .logroll.rollnow. +/ config/permissions/*.q grant files call .pm.addrole, .pm.grantfunction, .pm.ALL etc. at root, so +/ without this a legacy grant file fails on its first line. +/ published permanently during init (a shim scoped to one loadpermissions call would not be a shim), +/ but ONLY when enabled - a disabled module should not advertise admin functions that gate nothing. +/ NB: this diverges from TorQ, which defines .pm.* regardless of enabled; safe because gateway.q +/ guards on existence (`.pm.valp ~ key `.pm.valp) and falls back cleanly + +publishroot:{[] + / expose the wildcard constant and the grant-script admin functions at .pm.* + set[`.pm.ALL;wildcard]; + {[n] set[` sv `.pm,n;admin n]} each grantscriptnames; + .z.m.loginfo[`publishroot;"published ",(string 1+count grantscriptnames)," names under .pm for legacy grant scripts"]; + }; + +unpublishroot:{[] + / remove every name publishroot created, leaving no root residue behind + present:(key `.pm) inter `ALL,grantscriptnames; + if[count present;![`.pm;();0b;present]]; + }; + +/ ============================================================ +/ grant data loading +/ ============================================================ + +loadgrantfile:{[path] + / load one grant file at ROOT (not via `use`) so its .pm.* calls resolve against the published names + if[()~key hsym `$path; + .z.m.loginfo[`loadpermissions;"grant file not found, skipping: ",path]; + :(::)]; + .z.m.loginfo[`loadpermissions;"loading grant file ",path]; + system "l ",path; + }; + +loadpermissions:{[] + / load the grant cascade: default -> proctype -> procname, under each configured directory + / mirrors TorQ's `.proc.loadconfig[dir;] each `default,proctype,procname` + requireinit[`loadpermissions]; + / normalise grantdirs: a bare string is a single directory, not a list of one-char directories. + / without this, (),"path" degrades to a char list and every char is treated as a directory + dirs:.z.m.config`grantdirs; + dirs:$[10h=type dirs;enlist dirs;(),dirs]; + if[0=count dirs; + .z.m.loginfo[`loadpermissions;"no grantdirs configured, nothing to load"]; + :(::)]; + names:`default,(.z.m.config`proctype),.z.m.config`procname; + names:names where not null names; + / NB: nested each, NOT `cross`. cross joins with `,` so a path STRING is concatenated with the + / symbol rather than paired with it - ("/tmp/gp" cross `default) is an 8-item mixed list, and + / dot-applying that to a binary function rank-errors + {[nms;d] {[d;n] loadgrantfile[d,"/",(string n),".q"]}[d;] each nms}[names;] each dirs; + }; + +/ ============================================================ +/ registration and teardown +/ ============================================================ + +registerhandlers:{[] + / claim the exec phase of every message-handling event, plus a .z.pc observer for cleanup + / registered under a stable name so a re-init reclaims the same events rather than colliding + {[e] .z.m.register[e;`exec;`di.permissions;0;execbodies e]} each key execbodies; + .z.m.register[`.z.pc;`;`di.permissions;0;droppublic]; + / .h.val is where HTTP GET permissioning actually happens on kdb+ 3.5+; it is not a .z.* event, so + / di.handlers' register would reject the symbol - assign it directly, keeping the original to restore. + / .z.ph is deliberately NOT claimed: an exec owner there replaces the built-in handler wholesale + / capture the ORIGINAL .h.val once only - init is idempotent and re-runs this, so an unguarded + / capture on a second init would record our own hooks.sync as the "original" and teardown would + / restore that instead of kdb+'s built-in + if[not @[{.z.m.hvaloriginal;1b};::;0b];.z.m.hvaloriginal:@[get;`.h.val;{(::)}]]; + set[`.h.val;hooks.sync]; + .z.m.loginfo[`init;"registered exec on ",(", " sv string key execbodies),", observer on .z.pc, and .h.val"]; + }; + +teardown:{[] + / release everything init installed: handler registrations, .h.val, and the published root names + / leaves grant data intact, so a subsequent init re-registers and re-publishes cleanly + requireinit[`teardown]; + if[not .z.m.enabled; + .z.m.loginfo[`teardown;"di.permissions is disabled, nothing to release"]; + :(::)]; + / NB: dot-apply, not @. `@[f;(a;b;c);h]` passes the three-element LIST as one argument to a ternary + / function, which rank-errors straight into the handler - so every removal silently "succeeded" + {[e] .[.z.m.removehandler;(e;`exec;`di.permissions);{[e2] .z.m.logwarn[`teardown;"could not remove exec handler: ",e2]}]} each key execbodies; + .[.z.m.removehandler;(`.z.pc;`;`di.permissions);{[e2] .z.m.logwarn[`teardown;"could not remove .z.pc observer: ",e2]}]; + $[(::)~.z.m.hvaloriginal;@[{![`.h;();0b;enlist`val];};::;{[e2]}];set[`.h.val;.z.m.hvaloriginal]]; + unpublishroot[]; + .z.m.enabled:0b; + .z.m.loginfo[`teardown;"di.permissions released - handlers, .h.val and .pm.* root names removed"]; + }; + +/ ============================================================ +/ api metadata +/ ============================================================ + +getapimeta:{[] + / this module's api metadata, one row per CALLABLE export, for di.torq to collect and register with + / di.api. init and getapimeta are omitted: di.torq calls those two by convention rather than + / discovering them, so they are plumbing, not API. teardown is NOT plumbing - it is an ordinary + / lifecycle operation a caller needs documented, so it gets a normal row. + / names are bare; di.torq applies the process-wide qualification + :flip `name`public`descrip`params`return!flip( + (`teardown; 1b; "release handler registrations, .h.val and the published .pm.* root names"; + "[]"; "null"); + (`version; 1b; "module version string"; + "[]"; "string: version"); + (`status; 1b; "what this module is currently enforcing - engine, readonly, ldap state"; + "[]"; "dict: setting -> value"); + (`allowed; 1b; "would this user be permitted to run this query? never executes it"; + "[symbol: user; string|parse tree: query]"; "boolean: permitted"); + (`requ; 1b; "permission-check a query as a user and execute it"; + "[symbol: user; string|parse tree: query]"; "any: the query result"); + (`val; 1b; "evaluate a parse tree, under reval when read-only mode is on"; + "[parse tree: expression]"; "any: the result"); + (`valp; 1b; "evaluate a string or parse tree, under reval when read-only mode is on"; + "[string|parse tree: expression]"; "any: the result"); + (`execas; 1b; "run a query as another user, subject to that user's permissions"; + "[string|parse tree: query; symbol: user]"; "any: the query result"); + (`admin; 1b; "grant administration sub-api - see the admin.* rows below for members"; + "[dict of functions, keyed by name]"; "dict: the admin functions"); + (`loadpermissions; 1b; "load the grant cascade (default, proctype, procname) from grantdirs"; + "[]"; "null"); + (`unblock; 1b; "clear a user's ldap lockout"; + "[symbol: user]"; "null"); + / the admin sub-api, enumerated so di.api can discover all of it rather than one opaque entry. + / public:0b - these are real callables registered in di.api's full (f) view but kept out of the + / public (p) summary, which lists `admin itself. the suite asserts the non-dotted names still + / match the export list exactly + (`admin.adduser; 0b; "register a user with an authentication method and a hashed password"; + "[symbol: user; symbol: authtype (local|ldap); symbol: hashtype (md5); string: hashed password]"; "null"); + (`admin.removeuser; 0b; "remove a user entirely"; "[symbol: user]"; "null"); + (`admin.cloneuser; 0b; "copy a user's auth method plus group and role membership onto a new id"; + "[symbol: source; symbol: new user; string: password]"; "null"); + (`admin.addgroup; 0b; "create a group, which grants table and variable access"; "[symbol: group; string: description]"; "null"); + (`admin.removegroup; 0b; "remove a group"; "[symbol: group]"; "null"); + (`admin.addtogroup; 0b; "add a user (or a group) to a group; membership is transitive"; + "[symbol: user or group; symbol: group]"; "null"); + (`admin.removefromgroup; 0b; "remove a user or group from a group"; "[symbol: user or group; symbol: group]"; "null"); + (`admin.addrole; 0b; "create a role, which grants the right to call functions"; "[symbol: role; string: description]"; "null"); + (`admin.removerole; 0b; "remove a role"; "[symbol: role]"; "null"); + (`admin.assignrole; 0b; "give a user a role"; "[symbol: user; symbol: role]"; "null"); + (`admin.unassignrole; 0b; "take a role away from a user"; "[symbol: user; symbol: role]"; "null"); + (`admin.addfunction; 0b; "put a function into a function group"; "[symbol: function; symbol: function group]"; "null"); + (`admin.removefunction; 0b; "take a function out of a function group"; "[symbol: function; symbol: function group]"; "null"); + (`admin.grantaccess; 0b; "grant an entity read or write access to a table or variable"; + "[symbol: object; symbol: entity; symbol: read or write]"; "null"); + (`admin.revokeaccess; 0b; "revoke a previously granted access"; "[symbol: object; symbol: entity; symbol: read or write]"; "null"); + (`admin.grantfunction; 0b; "grant a role the right to call a function, gated by a paramcheck"; + "[symbol: object; symbol: role; function: paramcheck]"; "null"); + (`admin.revokefunction; 0b; "revoke a function grant from a role"; "[symbol: object; symbol: role]"; "null"); + (`admin.createvirtualtable; 0b; "expose a filtered view of a table under a new name"; + "[symbol: name; symbol: table; list: where clause]"; "null"); + (`admin.removevirtualtable; 0b; "remove a virtual table"; "[symbol: name]"; "null"); + (`admin.addpublic; 0b; "track an auto provisioned anonymous user against its handle"; "[symbol: user; int: handle]"; "null"); + (`admin.removepublic; 0b; "stop tracking an anonymous user"; "[symbol: user]"; "null"); + (`admin.wildcard; 0b; "the wildcard object - grant against it for superuser rights"; "[]"; "symbol: the wildcard")); + }; diff --git a/di/permissions/test.csv b/di/permissions/test.csv new file mode 100644 index 00000000..d2693c97 --- /dev/null +++ b/di/permissions/test.csv @@ -0,0 +1,496 @@ +action,ms,bytes,lang,code,repeat,minver,comment +comment,,,,,,,setup - load the module with a capturing logger and a capturing handlers mock +before,0,0,q,perms:use`di.permissions,1,1,load di.permissions +before,0,0,q,.pt.cap:([]lvl:`symbol$();ctx:`symbol$();msg:()),1,1,log capture table for assertions +before,0,0,q,caplog:`info`warn`error!({[c;m] `.pt.cap insert (`info;c;m)};{[c;m] `.pt.cap insert (`warn;c;m)};{[c;m] `.pt.cap insert (`error;c;m)}),1,1,capturing binary logger {[c;m]} +before,0,0,q,.pt.reg:([]event:`symbol$();phase:`symbol$();nm:`symbol$();pri:`long$();f:()),1,1,handler registration capture table +before,0,0,q,"mockh:`register`remove`list!({[e;p;n;pr;f] `.pt.reg insert (e;p;n;pr;f)};{[e;p;n] delete from `.pt.reg where event=e,phase=p,nm=n; (::)};{[e] select from .pt.reg where event=e})",1,1,handlers mock capturing every registration +before,0,0,q,deps:`log`handlers!(caplog;mockh),1,1,the injected dependency dict + +comment,,,,,,,init - dependency validation (log and handlers are required and never defaulted) +fail,0,0,q,perms.init[(::);(::)],1,1,init rejects a non-dict deps +fail,0,0,q,perms.init[(::);()!()],1,1,init rejects deps missing both keys +fail,0,0,q,perms.init[(::);enlist[`log]!enlist caplog],1,1,init rejects deps missing the handlers key +fail,0,0,q,perms.init[(::);enlist[`handlers]!enlist mockh],1,1,init rejects deps missing the log key +fail,0,0,q,perms.init[(::);`log`handlers!(42;mockh)],1,1,init rejects a non-dict log value +fail,0,0,q,perms.init[(::);`log`handlers!((enlist[`info]!enlist caplog`info);mockh)],1,1,init rejects a log dict missing warn and error +fail,0,0,q,perms.init[(::);`log`handlers!(caplog;enlist[`register]!enlist mockh`register)],1,1,init rejects a handlers dict missing remove and list +run,0,0,q,.pt.errstr:@[{perms.init[(::);()!()]};(::);{x}],1,1,capture the error string from a bad init +true,0,0,q,".pt.errstr like ""di.permissions:*""",1,1,init errors are prefixed di.permissions: +true,0,0,q,".pt.errstr like ""*di.log*""",1,1,the error names which module supplies the missing dependency + +comment,,,,,,,init - config validation +fail,0,0,q,perms.init[enlist[`engine]!enlist `tiered;deps],1,1,the tiered engine is not implemented and is rejected +fail,0,0,q,perms.init[enlist[`engine]!enlist `nosuch;deps],1,1,an unknown engine is rejected +fail,0,0,q,perms.init[enlist[`engine]!enlist 42;deps],1,1,a non-symbol engine is rejected +fail,0,0,q,perms.init[42;deps],1,1,a non-dict config is rejected +run,0,0,q,delete from `.pt.cap,1,1,clear the log capture +run,0,0,q,perms.init[enlist[`nosuchkey]!enlist 1b;deps],1,1,an unrecognised config key is accepted +true,0,0,q,1=count select from .pt.cap where lvl=`warn,1,1,an unrecognised config key warns rather than being dropped silently +true,0,0,q,"any (exec msg from .pt.cap where lvl=`warn) like\: ""*nosuchkey*""",1,1,the warning names the offending key + +comment,,,,,,,disabled by default - nothing registered and nothing published at root +run,0,0,q,perms.init[(::);deps],1,1,init with defaults (enabled is 0b) +true,0,0,q,0=count .pt.reg,1,1,a disabled module registers no handlers +true,0,0,q,0=count (key `.pm) except `,1,1,a disabled module publishes nothing at root +true,0,0,q,not perms.status[][`enabled],1,1,status reports the module as disabled + +comment,,,,,,,module metadata +true,0,0,q,10h=type perms.version,1,1,version is a string +true,0,0,q,0 "",.pi.dir,""/child.log 2>&1 &""",1,1,spawn the child in the background +before,0,0,q,"{[f] do[50;if[not ()~key hsym `$f;:(::)];system ""sleep 0.1""]}[.pi.portfile]",1,1,wait up to 5 seconds for the child to publish its port +before,0,0,q,".pi.port:@[{""J""$first read0 hsym `$x};.pi.portfile;{[e] 0N}]",1,1,read the OS-assigned port +before,0,0,q,".pi.h:$[null .pi.port;0Ni;@[hopen;`$""::"",string .pi.port;{[e] 0Ni}]]",1,1,connect to the child - NB :: not : which would open a FILE +before,0,0,q,.pi.up:not null .pi.h,1,1,did the child come up? +before,0,0,q,".pi.childlog:@[{read0 hsym `$x};.pi.dir,""/child.log"";{[e] enlist ""no child log""}]",1,1,keep the child log so a failure to start is diagnosable +comment,,,,,,,a skip must be VISIBLE - a suite reporting 0 failures while testing nothing is worse than a failure +true,0,0,q,.pi.up,1,1,the child process started; if this fails read .pi.childlog for why +comment,,,,,,,read only enforcement over a REAL handle - the whole reason this suite exists +before,0,0,q,".pi.read:$[.pi.up;@[{.pi.h""select from ([]a:1 2 3)""};::;{[e] `ERR}];`SKIP]",1,1,a permitted read through the child +before,0,0,q,".pi.write:$[.pi.up;@[{.pi.h""gg::42""};::;{[e] e}];""SKIP""]",1,1,a global write through the child +before,0,0,q,".pi.ggafter:$[.pi.up;@[{.pi.h""gg""};::;{[e] 0N}];0N]",1,1,read gg back to prove the write never landed +true,0,0,q,$[.pi.up;3=count .pi.read;0b],1,1,a permitted read succeeds through the real handler chain +true,0,0,q,$[.pi.up;10h=type .pi.write;0b],1,1,the write was refused - an error string came back not a result +true,0,0,q,"$[.pi.up;.pi.write like ""*noupdate*"";0b]",1,1,the refusal is revals noupdate - read only is genuinely enforced +true,0,0,q,$[.pi.up;0=.pi.ggafter;0b],1,1,gg is untouched - the write did not land +comment,,,,,,,cleanup +before,0,0,q,"if[.pi.up;@[{.pi.h""exit 0""};::;{[e]}]]",1,1,ask the child to exit +before,0,0,q,if[.pi.up;@[hclose;.pi.h;{[e]}]],1,1,close the handle +before,0,0,q,"system ""rm -rf "",.pi.dir",1,1,remove the scratch directory From 95488a3d450af167d11a07e06eabb9fbc15b5d32 Mon Sep 17 00:00:00 2001 From: alowrydi Date: Thu, 6 Aug 2026 10:23:47 +0100 Subject: [PATCH 05/11] Fixes to cloneuser password hashing & valp readonly bug, VERSION convention, admin input validation, permission-check select clauses --- di/permissions/VERSION | 1 + di/permissions/init.q | 6 ++ di/permissions/permissions.md | 70 +++++++++++--- di/permissions/permissions.q | 145 +++++++++++++++++++++++++--- di/permissions/test.csv | 94 ++++++++++++++++++ di/permissions/test_integration.csv | 14 ++- 6 files changed, 304 insertions(+), 26 deletions(-) create mode 100644 di/permissions/VERSION diff --git a/di/permissions/VERSION b/di/permissions/VERSION new file mode 100644 index 00000000..6c6aa7cb --- /dev/null +++ b/di/permissions/VERSION @@ -0,0 +1 @@ +0.1.0 \ No newline at end of file diff --git a/di/permissions/init.q b/di/permissions/init.q index 94d4f0e1..e2498140 100644 --- a/di/permissions/init.q +++ b/di/permissions/init.q @@ -4,6 +4,12 @@ \l ::permissions.q +/ module version, read from the VERSION file rather than hardcoded in the implementation - the +/ convention Jamie Grant's TorqX modules use, so a release bump touches one plain-text file. +/ NB `version` STAYS in the export: di.depcheck reads it from the export dict (checkdepversion), +/ and reports "exports no version" - failing the dependency check - if a module drops it +version:first read0`:::VERSION + / NB: export:([...]) EVALUATES each name, so it can only list names that already exist - the export / list and the implementation therefore cannot drift apart in this direction. / init and getapimeta are framework plumbing di.torq calls by convention; every other name here has a diff --git a/di/permissions/permissions.md b/di/permissions/permissions.md index ab3767f3..8c604643 100644 --- a/di/permissions/permissions.md +++ b/di/permissions/permissions.md @@ -18,7 +18,9 @@ see [Engine scope](#engine-scope). a group may itself be a member of another group. - **Query interception.** Select/update/delete, bare variable references, named function calls, `.q`-keyword calls (including joins, whose table arguments are checked recursively) and lambda - expressions are each classified and checked appropriately. + expressions are each classified and checked appropriately. A select is checked on its **where, by + and columns clauses as well as its target table**, using the same predicate as a bare reference — + see [Clause checking](#clause-checking). - **Virtual tables.** A named view of a table with an implicit where-clause spliced into any select against it, so a group can be granted a filtered slice rather than the whole table. - **Pluggable authentication.** `local` (md5 hash) and `ldap` backends, selected per user by the @@ -34,7 +36,7 @@ see [Engine scope](#engine-scope). | Dependency | Key | Required | Description | |---|---|---|---| | logger | `` `log `` | yes | dict with `info`, `warn`, `error`, each binary `{[c;m]}` — symbol context, string message | -| handlers | `` `handlers `` | yes | dict with `register`, `remove`, `list` — see `di.handlers` | +| handlers | `` `handlers `` | yes | dict with `register` and `remove` — see `di.handlers`. A full `di.handlers` dict (which also carries `list`) is fine; only the two keys this module calls are required | | ldap bind | `` `ldapbind `` | **no** | `{[session;dict]}` returning a dict with a `` `ReturnCode `` key (`0i` = success). Replaces the native LDAP library entirely when supplied — see [LDAP coverage](#ldap-coverage--the-bind-path-is-exercised-via-an-injected-ldapbind) | **No hard dependencies on other `di.*` modules** — `deps.q` is empty and the module is standalone. @@ -99,7 +101,7 @@ handlers are registered and nothing is published at root. | `publishroot` | `1b` | expose the legacy `.pm.*` names at root. Set `0b` if you have no legacy grant files — the module still enforces, it just leaves the root namespace untouched | | `ldapenabled` | `0b` | enable the LDAP backend and load its native library | | `ldaplibpath` | `""` | path to the LDAP `.so`; falls back to `$KDBLIB` | -| `ldapdebug` | `0i` | log LDAP chatter at info level | +| `ldapdebug` | `0b` | log LDAP chatter at info level | | `ldapservers` | `` enlist `$"ldap://localhost:0" `` | LDAP server URIs | | `ldapversion` | `3` | LDAP protocol version | | `ldapblocktime` | `0D00:30:00` | how long a locked-out user stays locked out; null means forever | @@ -144,14 +146,11 @@ Would this user be permitted to run this query? Never executes it. perms.allowed[`alice;"select from trade"] / 1b ``` -> **`allowed` is a true predicate.** It returns a boolean and never executes the query. Two earlier +> **`allowed` is a true predicate.** It returns a boolean and never executes the query. Three earlier > caveats have been fixed: it now descends into `.q`-keyword joins (so it agrees with `requ` rather -> than permitting joins `requ` refuses), and a forbidden lambda expression returns `0b` instead of -> raising. -> -> **One caveat remains:** it **ignores `permissivemode`**, pinning it off regardless of config — -> inherited from TorQ, which fixes `allowed:mainexpr[;;0b;0b]`. On a permissive-mode process `allowed` -> will therefore deny things `requ` permits. `requ` is the authority. +> than permitting joins `requ` refuses), a forbidden lambda expression returns `0b` instead of raising, +> and it honours `permissivemode` rather than pinning it off as TorQ does (`allowed:mainexpr[;;0b;0b]`), +> so it no longer denies things `requ` permits. ### `requ[user;query]` Permission-check a query as a user and execute it. Passes the query through untouched when the module @@ -210,7 +209,15 @@ What the module is currently enforcing: engine, read-only state, permissive mode public access, and LDAP availability. Legacy has no equivalent introspection. ### `version` -The module version string. +The module version string, e.g. `"0.1.0"`. + +Read at load time from the **`VERSION` file** in the module directory (`version:first read0`:::VERSION` +in `init.q`) rather than being hardcoded as a q literal, so a release bump touches one plain-text file. +This follows the TorqX module convention. + +`version` remains in the **export dictionary**. `di.depcheck` resolves a dependency's version from its +export dict (`checkdepversion`) and reports `"… exports no version"` — failing the dependency check — +if a module omits it. Moving the *value* to a file does not move the *export*. ### `getapimeta[]` This module's api metadata — one `` `name`public`descrip`params`return `` row per callable export, for @@ -238,8 +245,46 @@ are covered too**: an unparseable string yields `di.permissions: mainexpr: could not parse query: …` rather than q's bare parse error, so a client probing with garbage still leaves an audit trail. +"Every" is enforced rather than asserted: the suite drives one wrong-typed call at **each** admin +entry point and requires all of them to raise a `di.permissions:`-prefixed error, so an admin function +added later cannot quietly skip validation. It also checks that the rejection reached the injected +logger, not merely that it was thrown. + --- +## Clause checking + +TorQ permission-checks a select on its **target table only** (`permissions.q`'s `query` inspects +nothing but `first q[1]`). A select's where, by and columns clauses can name *other* tables, and those +executed unchecked — so a user granted any single table could read any other: + +```q +/ alice is granted `open and has NO grant on `secret +select p:first secret`pin from open / TorQ: returns 1234. here: refused +select p:count secret from open / TorQ: returns the row count. here: refused +select from open where id in exec id from secret / TorQ: executes. here: refused +``` + +This module checks every readable object named anywhere in the where, by and columns clauses, and +refuses the query unless the user has read access to each. `allowed` applies the same check, so it +agrees with `requ` rather than permitting a query `requ` would refuse. + +**The predicate is `rbac.isdefinedvar` — the same one a bare reference and a lambda expression use.** +That is the point: all three paths agree on what counts as a readable object, so an object cannot be +readable through a select clause while a bare reference to it is refused. An earlier revision checked +only *table*-valued symbols, which left exactly that inconsistency: + +```q +.pt.secretvec / refused +select p:first .pt.secretvec from .pt.trade / returned its contents +``` + +**The target table's own column names are removed first.** Inside a select, a symbol matching a column +denotes that column, not a same-named global — so checking it would deny ordinary queries. Without the +exclusion, `select id,v from open` is refused on any process that also happens to define globals `id` +or `v`; with it, that query is clean and an ungranted global of a colliding name still cannot be read +(the suite asserts both, via a `zz` column and a `zz` root global). + ## Handler registration | Event | Phase | Behaviour | @@ -316,7 +361,8 @@ this schema. > That was not true initially — grant data and LDAP cache state surviving `teardown` broke ten > assertions on a second run, which is what surfaced the `unblock` gap documented above. -**Unit suite** (`test.csv`) — 111 rows, 67 assertions, no sockets and no native library required: +**Unit suite** (`test.csv`) — no sockets and no native library required. Run it for the current row +and assertion counts rather than trusting a figure quoted here: ```q k4unit:use`di.k4unit k4unit.moduletest`di.permissions diff --git a/di/permissions/permissions.q b/di/permissions/permissions.q index 6addf5d7..8363f042 100644 --- a/di/permissions/permissions.q +++ b/di/permissions/permissions.q @@ -3,8 +3,8 @@ / and code/common/execas.q; controlaccess.q's tiered engine is deliberately deferred (see the engine / config key, which ships from v1 so the tiered engine can land later without reshaping this schema) -/ module version - placeholder only; no project-wide version / di.depcheck convention exists yet -version:"0.1.0"; +/ NB: the module version is NOT defined here - it is read from the VERSION file in init.q, so a +/ release bump touches one plain-text file rather than a q string literal / ============================================================ / constants (load-time) @@ -81,7 +81,7 @@ configdefaults:`enabled`engine`maxsize`runmode`permissivemode`readonly`public`ig / every ldap setting is read from .z.m.config at call time (one storage location, no bare-name / ambiguity) - including by the default ldapbuilddn below, which is why it is explicit about it ldapconfigdefaults:`ldapenabled`ldaplibpath`ldapdebug`ldapservers`ldapversion`ldapblocktime`ldapchecklimit`ldapchecktime`ldapbuilddnsuf`ldapbuilddn! - (0b;"";0i;enlist `$"ldap://localhost:0";3;0D00:30:00;3;0D00:05;"";{"uid=",string[x],",",.z.m.config`ldapbuilddnsuf}); + (0b;"";0b;enlist `$"ldap://localhost:0";3;0D00:30:00;3;0D00:05;"";{"uid=",string[x],",",.z.m.config`ldapbuilddnsuf}); / ============================================================ / internal helpers @@ -95,6 +95,18 @@ requiresym:{[ctx;nm;x] raiseerror[ctx;nm," must be a symbol, got ",.Q.s1 type x]]; }; +requirestring:{[ctx;nm;x] + / validate a public-api argument that must be a string (char vector) + if[10h<>type x; + raiseerror[ctx;nm," must be a string, got ",.Q.s1 type x]]; + }; + +requireint:{[ctx;nm;x] + / validate a public-api argument that must be an integer handle + if[not type[x] in -6 -7h; + raiseerror[ctx;nm," must be an integer, got ",.Q.s1 type x]]; + }; + requirequery:{[ctx;q] / a query must be a string or a parse tree - not an atom, and not a bare symbol if[type[q] within -19 -1h; @@ -130,15 +142,19 @@ validatedeps:{[deps] '"di.permissions: deps must be a dict with `log and `handlers keys - see di.log, di.handlers"]; if[not all `log`handlers in key deps; '"di.permissions: log and handlers dependencies are required; pass `log (`info`warn`error) ", - "and `handlers (`register`remove`list) - see di.log, di.handlers; got: ",(", " sv string key deps)]; + "and `handlers (`register`remove) - see di.log, di.handlers; got: ",(", " sv string key deps)]; if[99h<>type deps`log; '"di.permissions: log value must be a dict; pass `info`warn`error functions - see di.log"]; if[not all `info`warn`error in key deps`log; '"di.permissions: log dict must have `info`warn`error keys; got: ",(", " sv string key deps`log)]; if[99h<>type deps`handlers; - '"di.permissions: handlers value must be a dict; pass `register`remove`list functions - see di.handlers"]; - if[not all `register`remove`list in key deps`handlers; - '"di.permissions: handlers dict must have `register`remove`list keys; got: ",(", " sv string key deps`handlers)]; + '"di.permissions: handlers value must be a dict; pass `register`remove functions - see di.handlers"]; + / only `register`remove are required - this module never calls `list (nor `version). requiring a key + / the code path does not exercise is friction with no safety payoff. NB this is unrelated to + / di.depcheck's handlers contract, which checks that di.handlers EXPORTS register/remove/list - + / a statement about the provider's export dict, not about any consumer's injected-deps dict + if[not all `register`remove in key deps`handlers; + '"di.permissions: handlers dict must have `register`remove keys; got: ",(", " sv string key deps`handlers)]; / ldapbind is OPTIONAL - when supplied it replaces the native bind entirely (see ldap.bind) if[`ldapbind in key deps; if[not type[deps`ldapbind] within 100 112h; @@ -161,8 +177,9 @@ resolveconfig:{[config] / rather than surfacing later as a confusing runtime error far from its cause / engine is deliberately absent - validateengine gives a better message for it / ignorelist and grantdirs are deliberately absent - both accept several shapes and are normalised -boolconfigkeys:`enabled`readonly`permissivemode`runmode`public`ldapenabled`publishroot; -intconfigkeys:`maxsize`ldapversion`ldapchecklimit`ldapdebug; +/ ldapdebug is a bare on/off flag, not a level - ldap.debuglog reads it as `if[...]`, nothing grades it +boolconfigkeys:`enabled`readonly`permissivemode`runmode`public`ldapenabled`publishroot`ldapdebug; +intconfigkeys:`maxsize`ldapversion`ldapchecklimit; symconfigkeys:`proctype`procname; strconfigkeys:`ldaplibpath`ldapbuilddnsuf; spanconfigkeys:`ldapblocktime`ldapchecktime; @@ -230,7 +247,6 @@ init:{[config;deps] .z.m.logerr:(deps`log)`error; .z.m.register:(deps`handlers)`register; .z.m.removehandler:(deps`handlers)`remove; - .z.m.listhandlers:(deps`handlers)`list; cfg:resolveconfig[config]; validateconfig[cfg]; validateengine[cfg`engine]; @@ -299,6 +315,7 @@ admin.adduser:{[u;authtype;hashtype;password] admin.removeuser:{[u] requireinit[`removeuser]; + requiresym[`removeuser;"user id";u]; .z.m.user:.[.z.m.user;();_;u]; }; @@ -311,6 +328,7 @@ admin.addgroup:{[n;d] admin.removegroup:{[n] requireinit[`removegroup]; + requiresym[`removegroup;"group name";n]; .z.m.groupinfo:.[.z.m.groupinfo;();_;n]; }; @@ -322,12 +340,15 @@ admin.addrole:{[n;d] admin.removerole:{[n] requireinit[`removerole]; + requiresym[`removerole;"role name";n]; .z.m.roleinfo:.[.z.m.roleinfo;();_;n]; }; admin.addtogroup:{[u;g] / add a user to a group, giving them the group's table-level access requireinit[`addtogroup]; + requiresym[`addtogroup;"user";u]; + requiresym[`addtogroup;"group name";g]; if[not g in key .z.m.groupinfo;raiseerror[`addtogroup;"no such group, call admin.addgroup first: ",string g]]; / NB: upsert, not join. TorQ writes `usergroup,:(u;g)`, whose amend-in-place semantics insert a row; / the explicit `.z.m.x:.z.m.x,(...)` rewrite this module needs is NOT equivalent - on an empty table @@ -337,29 +358,39 @@ admin.addtogroup:{[u;g] admin.removefromgroup:{[u;g] requireinit[`removefromgroup]; + requiresym[`removefromgroup;"user";u]; + requiresym[`removefromgroup;"group name";g]; if[(u;g) in .z.m.usergroup;.z.m.usergroup:.[.z.m.usergroup;();_;.z.m.usergroup?(u;g)]]; }; admin.assignrole:{[u;r] / assign a user a role, giving them the role's function-level access requireinit[`assignrole]; + requiresym[`assignrole;"user";u]; + requiresym[`assignrole;"role";r]; if[not r in key .z.m.roleinfo;raiseerror[`assignrole;"no such role, call admin.addrole first: ",string r]]; if[not (u;r) in .z.m.userrole;.z.m.userrole:.z.m.userrole upsert (u;r)]; }; admin.unassignrole:{[u;r] requireinit[`unassignrole]; + requiresym[`unassignrole;"user";u]; + requiresym[`unassignrole;"role";r]; if[(u;r) in .z.m.userrole;.z.m.userrole:.[.z.m.userrole;();_;.z.m.userrole?(u;r)]]; }; admin.addfunction:{[f;g] / put a function into a function group, so a grant against the group covers it requireinit[`addfunction]; + requiresym[`addfunction;"function";f]; + requiresym[`addfunction;"function group";g]; if[not (f;g) in .z.m.functiongroup;.z.m.functiongroup:.z.m.functiongroup upsert (f;g)]; }; admin.removefunction:{[f;g] requireinit[`removefunction]; + requiresym[`removefunction;"function";f]; + requiresym[`removefunction;"function group";g]; if[(f;g) in .z.m.functiongroup;.z.m.functiongroup:.[.z.m.functiongroup;();_;.z.m.functiongroup?(f;g)]]; }; @@ -379,6 +410,9 @@ admin.grantaccess:{[o;e;l] admin.revokeaccess:{[o;e;l] requireinit[`revokeaccess]; + requiresym[`revokeaccess;"object";o]; + requiresym[`revokeaccess;"entity";e]; + requiresym[`revokeaccess;"level";l]; if[(o;e;l) in .z.m.access;.z.m.access:.[.z.m.access;();_;.z.m.access?(o;e;l)]]; }; @@ -393,6 +427,8 @@ admin.grantfunction:{[o;r;p] admin.revokefunction:{[o;r] requireinit[`revokefunction]; + requiresym[`revokefunction;"object";o]; + requiresym[`revokefunction;"role";r]; t:`object`role#.z.m.function; if[(o;r) in t;.z.m.function:.[.z.m.function;();_;t?(o;r)]]; }; @@ -400,31 +436,45 @@ admin.revokefunction:{[o;r] admin.createvirtualtable:{[n;t;w] / expose a named view of a table with an implicit where-clause spliced into any select against it requireinit[`createvirtualtable]; + requiresym[`createvirtualtable;"name";n]; + requiresym[`createvirtualtable;"table";t]; if[not n in key .z.m.virtualtable;.z.m.virtualtable:.z.m.virtualtable upsert (n;t;w)]; }; admin.removevirtualtable:{[n] requireinit[`removevirtualtable]; + requiresym[`removevirtualtable;"name";n]; if[n in key .z.m.virtualtable;.z.m.virtualtable:.[.z.m.virtualtable;();_;n]]; }; admin.addpublic:{[u;w] / track an auto-provisioned anonymous user against the handle that created it requireinit[`addpublic]; + requiresym[`addpublic;"user";u]; + requireint[`addpublic;"handle";w]; .z.m.publictrack:.z.m.publictrack upsert (u;w); }; admin.removepublic:{[u] requireinit[`removepublic]; + requiresym[`removepublic;"user";u]; .z.m.publictrack:.[.z.m.publictrack;();_;u]; }; admin.cloneuser:{[u;unew;p] / copy a user's auth method plus group and role membership onto a new user id requireinit[`cloneuser]; + requiresym[`cloneuser;"source user";u]; + requiresym[`cloneuser;"new user id";unew]; + requirestring[`cloneuser;"password";p]; if[not u in key .z.m.user;raiseerror[`cloneuser;"no such user to clone: ",string u]]; ul:raze exec authtype,hashtype from .z.m.user where id=u; - admin.adduser[unew;ul 0;ul 1;value (string ul 1)," string `",p]; + / NB: hash directly. TorQ builds the string (string hashtype)," string `",p and EVALUATES it, which + / throws on any password containing a space ('word) or a backtick ('type), and evaluates + / caller-supplied text in an auth path. md5 p is identical for a well-formed password and total + if[not `md5~ul 1; + raiseerror[`cloneuser;"cannot clone user with unsupported hashtype ",string[ul 1],"; only md5 is supported"]]; + admin.adduser[unew;ul 0;ul 1;md5 p]; admin.addtogroup[unew;] each exec groupname from .z.m.usergroup where user=u; admin.assignrole[unew;] each exec role from .z.m.userrole where user=u; }; @@ -494,11 +544,56 @@ rbac.qexe:{[x] rbac.exe:{[x] / evaluate an expression, choosing parse-tree vs string evaluation by the head's type + / NB: only heads of type 102/103/105-112 (operators, iterators, compositions) reach val - a symbol + / head is 11h and a lambda head is 100h, so both fall through to valp, as does a string. that is + / fine because valp handles all three shapes; it did NOT before the parse guard was added there v:$[(104<>a)&100=5 in a DIFFERENT function. assert it here rather than trust that coupling: on a shorter + / list 2_q silently yields (), so refs is empty, so nothing is checked and the query is PERMITTED. + / a silent fail-OPEN is the one direction a permission check must never take, so fail loudly instead + if[5>count q; + raiseerror[`query;"malformed query tree - expected at least 5 elements, got ",string count q]]; + tgt:$[11h=abs type q 1;first q 1;`]; + refs:distinct rbac.symsin 2_q; + refs:refs except @[{cols get x};tgt;`$()]; + refs:refs where rbac.isdefinedvar each refs; + / public objects are always readable, as in lamq + refs:refs except distinct exec object from .z.m.access where entity=`public; + bad:refs where not rbac.achk[u;;`read;pr] each refs; + if[count bad; + $[b;raiseerror[`query;" | " sv err[`selt] each bad];:0b]]; + :1b; + }; + rbac.query:{[u;q;b;pr] / permission-check a select/update/delete-shaped query / b: execute (1b) vs return a boolean verdict (0b). pr: permissive mode @@ -506,6 +601,7 @@ rbac.query:{[u;q;b;pr] / update or delete in place - needs write access on the target if[((!)~q 0) and 11h=type q 1; if[not rbac.achk[u;first q 1;`write;pr];$[b;raiseerror[`query;err[`updt][first q 1]];:0b]]; + if[not rbac.checkclauses[u;q;b;pr];:0b]; :$[b;rbac.qexe q;1b]]; / nested query - recurse into the inner select if[rbac.isq q 1;:$[b;rbac.qexe @[q;1;rbac.expr[u]];1b]]; @@ -517,6 +613,7 @@ rbac.query:{[u;q;b;pr] q:@[q;1;:;vt`table]; q:@[q;2;:;enlist first[q 2],vt`whereclause]]; if[not rbac.achk[u;t;`read;pr];$[b;raiseerror[`query;err[`selt][t]];:0b]]; + if[not rbac.checkclauses[u;q;b;pr];:0b]; :$[b;rbac.qexe q;1b]]; / anything else - superuser only if[not rbac.fchk[u;wildcard;()];$[b;raiseerror[`query;err[`selx][]];:0b]]; @@ -659,8 +756,22 @@ val:{[x] valp:{[x] / evaluate a string or parse tree, under reval when read-only mode is on + / ⚠ BUG FIX vs TorQ: legacy is `valp:$[readonly;{reval parse x};value]` (permissions.q:9). `parse` + / requires a STRING and throws 'type on a list, so under readonly every parse-tree input threw + / instead of evaluating. rbac.exe routes BOTH symbol heads (type 11h) and lambda heads (100h) here, + / which is the standard sync/async IPC call shape h(`func;arg) - so a readonly process (canonically + / an HDB) rejected the most common client idiom with a bare 'type requireinit[`valp]; - :$[.z.m.readonly;reval parse x;value x]; + if[not .z.m.readonly;:value x]; + / read-only. a STRING parses then evaluates, exactly as legacy did - `parse` inserts the literal + / markers so eval and value agree on it. + / a PARSE TREE must keep `value`'s semantics, which resolve the head but NOT the arguments. handing + / it to `reval` directly would use EVAL semantics, which resolve a symbol argument to the variable it + / names: (`echo;`secret) would return the contents of `secret` to a caller with no grant on it, since + / only the head is permission-checked. that is a read bypass, and it would exist ONLY in read-only + / mode - strictly more permissive than the same call with readonly off, which is the wrong direction. + / applying value to the tree as a literal inside reval keeps value's semantics and reval's write ban + :$[10h=type x;reval parse x;reval (value;enlist x)]; }; allowed:{[u;q] @@ -871,8 +982,11 @@ authenticate:{[u;p] if[not ud[`authtype] in key auth; .z.m.logwarn[`authenticate;"rejected ",(string u),": unknown authtype ",string ud`authtype]; :0b]; + / log both outcomes: a warn-only trail records rejections but leaves successful logins invisible, + / so the audit cannot answer "who connected". ldap successes were logged only under ldapdebug ok:auth[ud`authtype][u;p]; - if[not ok;.z.m.logwarn[`authenticate;"failed authentication for user ",string u]]; + $[ok;.z.m.loginfo[`authenticate;"authenticated user ",string u]; + .z.m.logwarn[`authenticate;"failed authentication for user ",string u]]; :ok; }; @@ -997,6 +1111,11 @@ registerhandlers:{[] / claim the exec phase of every message-handling event, plus a .z.pc observer for cleanup / registered under a stable name so a re-init reclaims the same events rather than colliding {[e] .z.m.register[e;`exec;`di.permissions;0;execbodies e]} each key execbodies; + / priority 0 on .z.pc - lower runs first, so this cleanup precedes any other observer. that is the + / faithful port: TorQ chains .z.pc as {droppublic[y];@[x;y]} (permissions.q:260), cleanup first and + / the prior handler after. it is also the right layering - TorqX registers its gateway .z.po/.z.pc + / connection bookkeeping at priority 10, so the security teardown lands ahead of it, and nothing in + / that bookkeeping depends on the user record droppublic removes .z.m.register[`.z.pc;`;`di.permissions;0;droppublic]; / .h.val is where HTTP GET permissioning actually happens on kdb+ 3.5+; it is not a .z.* event, so / di.handlers' register would reject the symbol - assign it directly, keeping the original to restore. diff --git a/di/permissions/test.csv b/di/permissions/test.csv index d2693c97..3ffe50cf 100644 --- a/di/permissions/test.csv +++ b/di/permissions/test.csv @@ -38,6 +38,11 @@ true,0,0,q,not perms.status[][`enabled],1,1,status reports the module as disable comment,,,,,,,module metadata true,0,0,q,10h=type perms.version,1,1,version is a string true,0,0,q,0 Date: Thu, 6 Aug 2026 11:55:42 +0100 Subject: [PATCH 06/11] fix two latent bugs, repair the test harness, improve docs --- di/permissions/permissions.md | 175 +++++++++++++++++++++++++++- di/permissions/permissions.q | 54 ++++++--- di/permissions/test.csv | 60 +++++++++- di/permissions/test_integration.csv | 8 +- 4 files changed, 275 insertions(+), 22 deletions(-) diff --git a/di/permissions/permissions.md b/di/permissions/permissions.md index 8c604643..79f5afc9 100644 --- a/di/permissions/permissions.md +++ b/di/permissions/permissions.md @@ -114,6 +114,12 @@ Unrecognised keys are **warned about**, not silently dropped. Every key is uniqu survives `di.config`'s flat cascade — note the `ldap*` prefixes, which exist because legacy ships four separate `enabled` settings that would otherwise collapse onto one another. +> **⚠ Breaking config change: `ldapdebug` is now a boolean.** It was an int (`0i`); it is now `0b`. +> `init` type-checks every setting whose shape is fixed, so a caller passing `ldapdebug:1i` **will now +> fail `init`** with `config key(s) ldapdebug must be a boolean (1b or 0b)`. The value was only ever +> read as an on/off flag (`if[.z.m.config`ldapdebug;…]` in `ldap.debuglog`) — nothing graded it as a +> level — so the int type was a lie the validator now refuses. Set `ldapdebug:1b` instead. + ### ⚠ `ignorelist` defaults to empty, unlike TorQ TorQ's `zpsignore.q` ships **enabled** with `` (`upd;"upd";`.u.upd;".u.upd") ``, exempting those from @@ -134,16 +140,28 @@ forms. It applies to `.z.ps` only, matching TorQ; `.z.pg` is never exempted. ### `init[config;deps]` Wire dependencies, resolve config, and (when enabled) publish root names, load grants and register -handlers. Idempotent. +handlers. Idempotent — see [Initialisation](#initialisation) for the full worked example. +```q +perms.init[`enabled`readonly!(1b;0b);`log`handlers!(logdep;handlersdep)] +/ or with defaults only (module loads but stays disabled): +perms.init[(::);`log`handlers!(logdep;handlersdep)] +``` ### `teardown[]` Release everything `init` installed: handler registrations, `.h.val`, and the published `.pm.*` root names. Grant data survives, so a later `init` re-registers and re-publishes cleanly. +```q +count (key `.pm) except ` / 8 - the published root names +perms.teardown[] +count (key `.pm) except ` / 0 - all removed +perms.status[][`enabled] / 0b - and the module reports itself disabled +``` ### `allowed[user;query]` Would this user be permitted to run this query? Never executes it. ```q -perms.allowed[`alice;"select from trade"] / 1b +perms.allowed[`alice;"select from trade"] / 1b - alices group is granted read on trade +perms.allowed[`alice;"select from secret"] / 0b - no grant, and nothing was executed ``` > **`allowed` is a true predicate.** It returns a boolean and never executes the query. Three earlier @@ -154,15 +172,43 @@ perms.allowed[`alice;"select from trade"] / 1b ### `requ[user;query]` Permission-check a query as a user and execute it. Passes the query through untouched when the module -is disabled. +is disabled. This is the authority — `allowed` is the dry run. +```q +perms.requ[`alice;"select from trade"] +/ sym px +/ --------- +/ a 1 +/ a 2 +/ b 3 + +perms.requ[`alice;"select from secret"] +/ 'di.permissions: query: no read permission on [secret] +``` ### `val[expr]` / `valp[expr]` Evaluate a parse tree / a string or parse tree, under `reval` when read-only mode is on. TorQ binds these at **load** time (`val:$[readonly;reval;eval]`), so read-only could not be toggled without a restart; here the choice resolves per call. +```q +perms.valp "2+2" / 4 +perms.val parse "2+2" / 4 +``` +Both are used by `di.gateway`, which copies the function value; neither performs a permission check on +its own — that is `requ`'s job. ### `execas[query;user]` Run a query as another user, subject to that user's permissions. +```q +perms.execas["select from trade";`alice] +/ sym px +/ --------- +/ a 1 +/ a 2 +/ b 3 + +perms.execas["select from secret";`alice] +/ 'di.permissions: query: no read permission on [secret] +``` ### `admin` The grant-administration sub-API. `admin.wildcard` is the wildcard object (`` `$"*" ``) — grant against @@ -186,6 +232,9 @@ perms.admin.grantaccess[`trade;`traders;`read]; perms.admin.adduser[`alice;`local;`md5;md5 "secret"]; perms.admin.assignrole[`alice;`reader]; perms.admin.addtogroup[`alice;`traders]; + +perms.admin.wildcard / `* - the wildcard object +perms.admin.grantfunction[perms.admin.wildcard;`admin;{1b}]; / superuser: any function ``` > **Paramchecks must be functions.** They are applied to the call's parameter dict under protection, @@ -194,10 +243,19 @@ perms.admin.addtogroup[`alice;`traders]; ### `loadpermissions[]` Load the grant cascade — `default` → `proctype` → `procname` — from every configured `grantdirs` -directory. Missing files are skipped with an info log. +directory. Missing files are skipped with an info log. Called automatically by `init`; call it again to +reload after editing a grant file. +```q +perms.loadpermissions[] +/ with no grantdirs configured this logs "nothing to load" and returns +``` ### `unblock[user]` Clear a user's cached LDAP state — both a lockout and any cached successful authentication. +```q +perms.unblock[`alice] +/ a user with no LDAP record logs "no ldap login record for user alice" and returns +``` > TorQ's equivalent returns early unless the user is actually *blocked*, which leaves no way to force > re-authentication for a user who is merely cached: after a password change their cached success @@ -207,9 +265,27 @@ Clear a user's cached LDAP state — both a lockout and any cached successful au ### `status[]` What the module is currently enforcing: engine, read-only state, permissive mode, run mode, maxsize, public access, and LDAP availability. Legacy has no equivalent introspection. +```q +perms.status[] +/ enabled | 1b +/ engine | `rbac +/ readonly | 0b +/ permissivemode| 0b +/ runmode | 1b +/ maxsize | 200000000 +/ public | 0b +/ publishroot | 1b +/ ldapenabled | 0b +/ ldapavailable | 0b +``` +`ldapavailable` reports whether a bind is actually reachable — the native library having loaded, or an +`ldapbind` having been injected — not merely that `ldapenabled` is set. ### `version` -The module version string, e.g. `"0.1.0"`. +The module version string. +```q +perms.version / "0.1.0" +``` Read at load time from the **`VERSION` file** in the module directory (`version:first read0`:::VERSION` in `init.q`) rather than being hardcoded as a q literal, so a release bump touches one plain-text file. @@ -223,6 +299,19 @@ if a module omits it. Moving the *value* to a file does not move the *export*. This module's api metadata — one `` `name`public`descrip`params`return `` row per callable export, for `di.torq` to collect and register with `di.api`. `init` and `getapimeta` are deliberately absent: they are framework plumbing `di.torq` calls by convention rather than discovers. Needs no `init`. +```q +cols perms.getapimeta[] / `name`public`descrip`params`return +count perms.getapimeta[] / 33 - 11 top-level exports plus the 22 admin.* members + +3 sublist select name,public,descrip from perms.getapimeta[] +/ name public descrip +/ --------------------------------------------------------------------- +/ teardown 1b "release handler registrations, .h.val and the publi.. +/ version 1b "module version string" +/ status 1b "what this module is currently enforcing - engine, r.. +``` +The `admin.*` members carry `public:0b` — they are real callables registered in `di.api`'s full view +but kept out of the public summary, which lists `admin` itself. --- @@ -350,6 +439,56 @@ this schema. --- +## Usage Example + +```q +/ log dep must already match the binary {[c;m]} contract - write your own, or use di.log: +/ logging:use`di.log +/ logdep:logging.logdict +logdep:`info`warn`error!({[c;m]};{[c;m]};{[c;m]}) + +/ di.handlers is INJECTED, not imported - build its dep dict and hand it over +perms:use`di.permissions +handlers:use`di.handlers +handlers.init[enlist[`log]!enlist logdep]; +handlersdep:`register`remove`list!(handlers.register;handlers.remove;handlers.list); + +perms.init[enlist[`enabled]!enlist 1b;`log`handlers!(logdep;handlersdep)]; + +/ set up a reader who may select from trade and nothing else +trade:([]sym:`a`a`b;px:1 2 3.0); +secret:([]pin:1234 5678); +perms.admin.addrole[`reader;"may select"]; +perms.admin.grantfunction[`select;`reader;{1b}]; +perms.admin.addgroup[`traders;"trading desk"]; +perms.admin.grantaccess[`trade;`traders;`read]; +perms.admin.adduser[`alice;`local;`md5;md5 "secret"]; +perms.admin.assignrole[`alice;`reader]; +perms.admin.addtogroup[`alice;`traders]; + +/ dry run - allowed never throws, it returns a verdict +perms.allowed[`alice;"select from trade"] / 1b +perms.allowed[`alice;"select from secret"] / 0b + +/ execute - requ returns the result, or RAISES on a refusal +perms.requ[`alice;"select from trade"] / the three rows +/ perms.requ[`alice;"select from secret"] / 'di.permissions: query: no read permission on [secret] +/ ^ left commented out: a refusal raises, which would stop this block. catch it if you want to run it: +@[perms.requ[`alice;];"select from secret";{[e] -1 "refused: ",e;}]; + +/ what is being enforced right now +perms.status[] + +/ release everything on the way out +perms.teardown[]; +``` + +From here a real process would load its grants from files rather than by hand — set `grantdirs` and +let `init` run the `default` → `proctype` → `procname` cascade. See +[Root-name publication](#root-name-publication) for why an unmodified TorQ grant file works unchanged. + +--- + ## Running tests > **Note on suite structure.** Rows share accumulated state (users, groups and grants created by @@ -416,7 +555,15 @@ directory-specific behaviour such as referrals or TLS negotiation. port and drives a real connection. It exists because **`reval`'s read-only restriction is not applied when `.z.w=0`**: at the console `reval parse "g::1"` happily sets `g`, but over a real handle the same call throws `'noupdate`. k4unit runs in-process at handle 0, so a unit test asserting a blocked write -would fail against correct code. `moduletest` only loads `test.csv`, so run this one directly: +would fail against correct code. + +> **⚠ The invocation below is a workaround, not the supported interface.** `di.k4unit.moduletest` +> hardcodes the filename `test.csv` (`di/k4unit/init.q:9`) and its `KUltf`/`KUrt` primitives are not +> exported, so there is **no supported way to run a second suite in a module**. The lines below reach +> into `di.k4unit`'s private, loader-mangled namespace and will break if that mangling changes. +> **Upstream ask: a `moduletest[module;filename]` overload on `di.k4unit`**, after which this reduces +> to ``k4unit.moduletest[`di.permissions;`test_integration.csv]``. + ```q k4unit:use`di.k4unit .m.di.0k4unit.KUltf .Q.dd[hsym`$.Q.m.mp`di.permissions;`test_integration.csv] @@ -426,6 +573,22 @@ k4unit.getresults[] Run it in a fresh session — running it after `moduletest` re-runs the still-loaded unit tests against dirty module state. +> **What this suite does and does not prove.** It verifies read-only enforcement and parse-tree +> handling over a **real child process and a real IPC handle** — the things that cannot be tested +> in-process, because `reval` does not enforce at `.z.w=0`. +> +> It wires **real `di.handlers` only when that module is on `QPATH`**, falling back to a minimal +> inline stand-in (a bare `set[ev;f]`) otherwise. `di.handlers` lives on the `feature-handlers` +> branch, so a checkout of `feature-permissions` alone runs against the stand-in and does **not** +> exercise real phase/`exec`-ownership dispatch — that is covered by `di.handlers`' own suite, not +> this one. Check out both branches to exercise the real wiring. +> +> Which path ran is **reported, not assumed**: the suite asserts the child returned a boolean for +> `realhandlers`, and the value is visible in the results. This exists because the "prefer real +> `di.handlers`" branch was silently dead for weeks — `h.init` on a function-local throws +> `'h.init` (module dot-sugar resolves only against a *global* name), and the protected apply +> swallowed it, so the stand-in always ran while the suite reported green. + --- ## Migrating from `.pm` / `.access` diff --git a/di/permissions/permissions.q b/di/permissions/permissions.q index 8363f042..1a5b2ed8 100644 --- a/di/permissions/permissions.q +++ b/di/permissions/permissions.q @@ -101,6 +101,19 @@ requirestring:{[ctx;nm;x] raiseerror[ctx;nm," must be a string, got ",.Q.s1 type x]]; }; +normdescription:{[ctx;x] + / normalise a description to a char VECTOR, and reject anything that is not text. + / ⚠ this is load-bearing, not cosmetic. roleinfo/groupinfo declare `description:()` - a general + / list - and q COLLAPSES a general column to a typed vector as soon as it holds only atoms. a + / one-character description like "x" is a char ATOM, so it turns the column into a char vector, + / and the next multi-character description then throws a bare 'type. same for a symbol description. + / normalising to a vector here means the column only ever receives vectors and stays general + if[-10h=type x;:enlist x]; + if[10h<>type x; + raiseerror[ctx;"description must be a string, got ",.Q.s1 type x]]; + :x; + }; + requireint:{[ctx;nm;x] / validate a public-api argument that must be an integer handle if[not type[x] in -6 -7h; @@ -280,10 +293,29 @@ init:{[config;deps] unpublishroot[]; .z.m.enabled:0b; raiseerror[`init;"grant file failed to load, root names unpublished: ",e]}]; + ensurepublicscaffolding[]; registerhandlers[]; .z.m.loginfo[`init;"di.permissions initialised - engine ",string[.z.m.engine],", readonly ",("disabled";"enabled").z.m.readonly]; }; +ensurepublicscaffolding:{[] + / the anonymous-user path assigns `publicuser and adds to `public. both assignrole and addtogroup + / REFUSE an undefined role/group (as TorQ's do), so with public enabled and neither defined, every + / anonymous login THROWS out of .z.pw instead of connecting or being cleanly refused. + / TorQ has the same gap; it was simply unreachable there because the -public flag threw 'type first + / (see authenticate). fixing that made this reachable, so the module now provides its own + / scaffolding - created only when ABSENT, so a grant file's own definitions and descriptions win. + / the role and group are created EMPTY: an anonymous user gets nothing until an operator grants to + / them, so this is fail-closed, not a privilege grant + if[not .z.m.public;:(::)]; + if[not `publicuser in key .z.m.roleinfo; + admin.addrole[`publicuser;"anonymous users - created by di.permissions, holds no grants by default"]; + .z.m.loginfo[`init;"public access is enabled and no publicuser role was defined - created an empty one"]]; + if[not `public in key .z.m.groupinfo; + admin.addgroup[`public;"anonymous users - created by di.permissions, holds no grants by default"]; + .z.m.loginfo[`init;"public access is enabled and no public group was defined - created an empty one"]]; + }; + status:{[] / a snapshot of what this module is currently enforcing - legacy has no equivalent introspection requireinit[`status]; @@ -322,6 +354,7 @@ admin.removeuser:{[u] admin.addgroup:{[n;d] requireinit[`addgroup]; requiresym[`addgroup;"group name";n]; + d:normdescription[`addgroup;d]; if[n in key .z.m.user;raiseerror[`addgroup;"cannot add group with same name as existing user: ",string n]]; .z.m.groupinfo:.z.m.groupinfo upsert (n;d); }; @@ -335,6 +368,7 @@ admin.removegroup:{[n] admin.addrole:{[n;d] requireinit[`addrole]; requiresym[`addrole;"role name";n]; + d:normdescription[`addrole;d]; .z.m.roleinfo:.z.m.roleinfo upsert (n;d); }; @@ -566,20 +600,12 @@ rbac.symsin:{[x] }; rbac.checkclauses:{[u;q;b;pr] - / a select is permission-checked on its TARGET table only, but its where, by and columns clauses can - / name OTHER objects: `select p:first secretvec from open` returned secretvec's contents to a user - / with no grant on it, even though the identical BARE reference is refused. TorQ has the same gap - - / permissions.q's query checks nothing but first q[1]. - / 2_q is (where;by;columns); the target at q[1] is already checked by the caller. - / the predicate is rbac.isdefinedvar - THE SAME ONE a bare reference and lamq use - so all three - / paths agree on what counts as a readable object, rather than this one having its own narrower idea. - / the target's own COLUMN NAMES are removed FIRST: inside a select a symbol matching a column denotes - / that column, not a same-named global, so checking it would deny ordinary queries. measured against - / `select id,v from t` with globals `id`/`v` also defined - flagged without this, clean with it - / 2_q assumes the (?/!;target;where;by;columns) shape, which rbac.isq guarantees by gating on - / count>=5 in a DIFFERENT function. assert it here rather than trust that coupling: on a shorter - / list 2_q silently yields (), so refs is empty, so nothing is checked and the query is PERMITTED. - / a silent fail-OPEN is the one direction a permission check must never take, so fail loudly instead + / read-check every object named in a select's where, by and columns clauses - the target at q[1] is + / already checked by the caller, 2_q is the rest. uses rbac.isdefinedvar, the SAME predicate a bare + / reference and lamq use, minus the target's own column names. + / full rationale, the leak it closes and the false-positive measurements: permissions.md, "Clause checking" + / the guard below is defence in depth: rbac.isq gates count>=5 in a DIFFERENT function, and on a short + / list 2_q silently yields () - nothing checked, query PERMITTED. fail loudly rather than fail open if[5>count q; raiseerror[`query;"malformed query tree - expected at least 5 elements, got ",string count q]]; tgt:$[11h=abs type q 1;first q 1;`]; diff --git a/di/permissions/test.csv b/di/permissions/test.csv index 3ffe50cf..11967ddd 100644 --- a/di/permissions/test.csv +++ b/di/permissions/test.csv @@ -278,12 +278,58 @@ comment,,,,,,,"=== EVERY admin entry point validates its arguments through raise comment,,,,,,,a raw 'type from a downstream upsert bypasses the log entirely - no audit trail of who sent what comment,,,,,,,this asserts the whole surface at once so a newly added admin function cannot quietly skip it run,0,0,q,".pt.pfx:{[f] r:@[f;::;{x}]; $[10h=type r;0=5 before rbac.query runs so no client input can present a short tree. +comment,,,,,,,"the guard exists because 2_q on a short list yields () - refs empty, nothing checked, query" +comment,,,,,,,PERMITTED. a silent fail-open is the one direction a permission check must never take +run,0,0,q,".pt.mns:first (key `.m.di) where (string key `.m.di) like ""*permissions*""",1,1,derive this modules mangled namespace rather than hardcoding it +run,0,0,q,".pt.cc:get `$"".m.di."",(string .pt.mns),"".rbac.checkclauses""",1,1,reach the private checkclauses - there is no public path to a malformed tree +true,0,0,q,100h=type .pt.cc,1,1,resolved the internal function +fail,0,0,q,.pt.cc[`alice;(?;`t);1b;0b],1,1,a tree shorter than 5 elements RAISES rather than silently permitting +fail,0,0,q,.pt.cc[`alice;(?;`t);0b;0b],1,1,and the dry run form raises too - a malformed tree is not a permission verdict +run,0,0,q,".pt.ccerr:@[{.pt.cc[`alice;(?;`t);1b;0b]};(::);{x}]",1,1,capture the refusal text +true,0,0,q,"0 Date: Thu, 6 Aug 2026 12:14:48 +0100 Subject: [PATCH 07/11] trimming down excessive comments --- di/permissions/permissions.md | 144 ++++++------ di/permissions/permissions.q | 408 ++++++++++------------------------ 2 files changed, 193 insertions(+), 359 deletions(-) diff --git a/di/permissions/permissions.md b/di/permissions/permissions.md index 79f5afc9..1fb91d51 100644 --- a/di/permissions/permissions.md +++ b/di/permissions/permissions.md @@ -6,7 +6,7 @@ incoming query against a user's roles and groups, and optionally enforces a whol mode. Consolidates five TorQ files: `code/handlers/permissions.q` (`.pm`), `writeaccess.q` (`.readonly`), -`ldap.q` (`.ldap`), and `code/common/execas.q`. TorQ's `controlaccess.q` tiered engine is **deferred** — +`ldap.q` (`.ldap`), and `code/common/execas.q`. TorQ's `controlaccess.q` tiered engine is **deferred** - see [Engine scope](#engine-scope). --- @@ -14,12 +14,12 @@ see [Engine scope](#engine-scope). ## Features - **Users, groups and roles.** Roles grant the right to call *functions* (gated by a paramcheck - lambda); groups grant read/write access to *tables and variables*. Group membership is transitive — + lambda); groups grant read/write access to *tables and variables*. Group membership is transitive - a group may itself be a member of another group. - **Query interception.** Select/update/delete, bare variable references, named function calls, `.q`-keyword calls (including joins, whose table arguments are checked recursively) and lambda expressions are each classified and checked appropriately. A select is checked on its **where, by - and columns clauses as well as its target table**, using the same predicate as a bare reference — + and columns clauses as well as its target table**, using the same predicate as a bare reference - see [Clause checking](#clause-checking). - **Virtual tables.** A named view of a table with an implicit where-clause spliced into any select against it, so a group can be granted a filtered slice rather than the whole table. @@ -35,11 +35,11 @@ see [Engine scope](#engine-scope). | Dependency | Key | Required | Description | |---|---|---|---| -| logger | `` `log `` | yes | dict with `info`, `warn`, `error`, each binary `{[c;m]}` — symbol context, string message | -| handlers | `` `handlers `` | yes | dict with `register` and `remove` — see `di.handlers`. A full `di.handlers` dict (which also carries `list`) is fine; only the two keys this module calls are required | -| ldap bind | `` `ldapbind `` | **no** | `{[session;dict]}` returning a dict with a `` `ReturnCode `` key (`0i` = success). Replaces the native LDAP library entirely when supplied — see [LDAP coverage](#ldap-coverage--the-bind-path-is-exercised-via-an-injected-ldapbind) | +| logger | `` `log `` | yes | dict with `info`, `warn`, `error`, each binary `{[c;m]}` - symbol context, string message | +| handlers | `` `handlers `` | yes | dict with `register` and `remove` - see `di.handlers`. A full `di.handlers` dict (which also carries `list`) is fine; only the two keys this module calls are required | +| ldap bind | `` `ldapbind `` | **no** | `{[session;dict]}` returning a dict with a `` `ReturnCode `` key (`0i` = success). Replaces the native LDAP library entirely when supplied - see [LDAP coverage](#ldap-coverage) | -**No hard dependencies on other `di.*` modules** — `deps.q` is empty and the module is standalone. +**No hard dependencies on other `di.*` modules** - `deps.q` is empty and the module is standalone. Both dependencies are **required and never defaulted**; `init` throws immediately if either is absent, malformed, or missing keys. There is no fallback logger. That matters more here than elsewhere: legacy @@ -48,12 +48,12 @@ a silent fallback would make that silence look deliberate. > **Note on `di.api`.** TorQ's `lamq` enumerated every variable in every root namespace via > `.api.varnames`/`.api.allns`, then intersected that list with the tokens in the query. `di.api` is -> registry-only and does not expose those functions, by design — module code lives in each module's +> registry-only and does not expose those functions, by design - module code lives in each module's > private `.z.m`, so a root-namespace scan would find nothing useful. > > This module does **not** reimplement that walk. It inverts the algorithm: tokenise the query first, > then test only those tokens for being defined root variables. Same result, but O(tokens) rather than -> O(all names) — measured at 0.065 ms per lambda query against 2.9 ms for the walk on a process with +> O(all names) - measured at 0.065 ms per lambda query against 2.9 ms for the walk on a process with > 5000 root names. --- @@ -78,7 +78,7 @@ perms.init[`enabled`readonly!(1b;0b);`log`handlers!(logdep;handlersdep)]; `init` must be called before any other function. It is **idempotent**: a second call re-wires the dependencies and config and reclaims the same handler registrations, leaving grant data intact. -When `enabled` is `0b` (the default) `init` wires the logger, logs that it is disabled, and stops — no +When `enabled` is `0b` (the default) `init` wires the logger, logs that it is disabled, and stops - no handlers are registered and nothing is published at root. --- @@ -88,17 +88,17 @@ handlers are registered and nothing is published at root. | Key | Default | Description | |---|---|---| | `enabled` | `0b` | master switch; when off, nothing is registered or published | -| `engine` | `` `rbac `` | authorization engine. Only `rbac` is implemented — `` `tiered `` is rejected | +| `engine` | `` `rbac `` | authorization engine. Only `rbac` is implemented - `` `tiered `` is rejected | | `maxsize` | `200000000` | maximum serialized size of any returned result | | `runmode` | `1b` | `1b` executes the query, `0b` returns a boolean verdict only | | `permissivemode` | `0b` | when `1b`, an object with no grants at all is readable by default | | `readonly` | `0b` | route evaluation through `reval`, blocking writes | | `public` | `0b` | allow anonymous users to be auto-provisioned on login | -| `ignorelist` | `()` | message heads that bypass the check on `.z.ps` — see below | +| `ignorelist` | `()` | message heads that bypass the check on `.z.ps` - see below | | `grantdirs` | `()` | directories holding grant files, loaded by `loadpermissions` | | `proctype` | `` ` `` | process type, selects `{proctype}.q` in the grant cascade | | `procname` | `` ` `` | process name, selects `{procname}.q` in the grant cascade | -| `publishroot` | `1b` | expose the legacy `.pm.*` names at root. Set `0b` if you have no legacy grant files — the module still enforces, it just leaves the root namespace untouched | +| `publishroot` | `1b` | expose the legacy `.pm.*` names at root. Set `0b` if you have no legacy grant files - the module still enforces, it just leaves the root namespace untouched | | `ldapenabled` | `0b` | enable the LDAP backend and load its native library | | `ldaplibpath` | `""` | path to the LDAP `.so`; falls back to `$KDBLIB` | | `ldapdebug` | `0b` | log LDAP chatter at info level | @@ -108,19 +108,19 @@ handlers are registered and nothing is published at root. | `ldapchecklimit` | `3` | failed attempts before lockout | | `ldapchecktime` | `0D00:05` | window in which a repeat login skips the server | | `ldapbuilddnsuf` | `""` | suffix used when building the bind DN | -| `ldapbuilddn` | `{"uid=",string[x],",",…}` | function building the bind DN from a username | +| `ldapbuilddn` | `{"uid=",string[x],",",...}` | function building the bind DN from a username | Unrecognised keys are **warned about**, not silently dropped. Every key is uniquely named so it -survives `di.config`'s flat cascade — note the `ldap*` prefixes, which exist because legacy ships four +survives `di.config`'s flat cascade - note the `ldap*` prefixes, which exist because legacy ships four separate `enabled` settings that would otherwise collapse onto one another. -> **⚠ Breaking config change: `ldapdebug` is now a boolean.** It was an int (`0i`); it is now `0b`. +> **NB Breaking config change: `ldapdebug` is now a boolean.** It was an int (`0i`); it is now `0b`. > `init` type-checks every setting whose shape is fixed, so a caller passing `ldapdebug:1i` **will now > fail `init`** with `config key(s) ldapdebug must be a boolean (1b or 0b)`. The value was only ever -> read as an on/off flag (`if[.z.m.config`ldapdebug;…]` in `ldap.debuglog`) — nothing graded it as a -> level — so the int type was a lie the validator now refuses. Set `ldapdebug:1b` instead. +> read as an on/off flag (`if[.z.m.config`ldapdebug;...]` in `ldap.debuglog`) - nothing graded it as a +> level - so the int type was a lie the validator now refuses. Set `ldapdebug:1b` instead. -### ⚠ `ignorelist` defaults to empty, unlike TorQ +### `ignorelist` defaults to empty, unlike TorQ TorQ's `zpsignore.q` ships **enabled** with `` (`upd;"upd";`.u.upd;".u.upd") ``, exempting those from permission checks on `.z.ps`. Silently exempting `upd` is not a safe default for an access-control @@ -131,7 +131,7 @@ explicitly**, or that traffic will be permission-checked and rejected: perms.init[`enabled`ignorelist!(1b;(`upd;"upd";`.u.upd;".u.upd"));deps] ``` -It is a **mixed** list — the head of an incoming message is matched against both symbol and string +It is a **mixed** list - the head of an incoming message is matched against both symbol and string forms. It applies to `.z.ps` only, matching TorQ; `.z.pg` is never exempted. --- @@ -140,7 +140,7 @@ forms. It applies to `.z.ps` only, matching TorQ; `.z.pg` is never exempted. ### `init[config;deps]` Wire dependencies, resolve config, and (when enabled) publish root names, load grants and register -handlers. Idempotent — see [Initialisation](#initialisation) for the full worked example. +handlers. Idempotent - see [Initialisation](#initialisation) for the full worked example. ```q perms.init[`enabled`readonly!(1b;0b);`log`handlers!(logdep;handlersdep)] / or with defaults only (module loads but stays disabled): @@ -172,7 +172,7 @@ perms.allowed[`alice;"select from secret"] / 0b - no grant, and nothing was ex ### `requ[user;query]` Permission-check a query as a user and execute it. Passes the query through untouched when the module -is disabled. This is the authority — `allowed` is the dry run. +is disabled. This is the authority - `allowed` is the dry run. ```q perms.requ[`alice;"select from trade"] / sym px @@ -194,7 +194,7 @@ perms.valp "2+2" / 4 perms.val parse "2+2" / 4 ``` Both are used by `di.gateway`, which copies the function value; neither performs a permission check on -its own — that is `requ`'s job. +its own - that is `requ`'s job. ### `execas[query;user]` Run a query as another user, subject to that user's permissions. @@ -211,7 +211,7 @@ perms.execas["select from secret";`alice] ``` ### `admin` -The grant-administration sub-API. `admin.wildcard` is the wildcard object (`` `$"*" ``) — grant against +The grant-administration sub-API. `admin.wildcard` is the wildcard object (`` `$"*" ``) - grant against it for superuser rights. | Group | Functions | @@ -238,11 +238,11 @@ perms.admin.grantfunction[perms.admin.wildcard;`admin;{1b}]; / superuser: any ``` > **Paramchecks must be functions.** They are applied to the call's parameter dict under protection, -> and any non-boolean result is coerced to `0b` — so a literal `1b` stored as a paramcheck **fails +> and any non-boolean result is coerced to `0b` - so a literal `1b` stored as a paramcheck **fails > closed**. `grantfunction` rejects a non-function outright. ### `loadpermissions[]` -Load the grant cascade — `default` → `proctype` → `procname` — from every configured `grantdirs` +Load the grant cascade - `default` -> `proctype` -> `procname` - from every configured `grantdirs` directory. Missing files are skipped with an info log. Called automatically by `init`; call it again to reload after editing a grant file. ```q @@ -251,7 +251,7 @@ perms.loadpermissions[] ``` ### `unblock[user]` -Clear a user's cached LDAP state — both a lockout and any cached successful authentication. +Clear a user's cached LDAP state - both a lockout and any cached successful authentication. ```q perms.unblock[`alice] / a user with no LDAP record logs "no ldap login record for user alice" and returns @@ -278,8 +278,8 @@ perms.status[] / ldapenabled | 0b / ldapavailable | 0b ``` -`ldapavailable` reports whether a bind is actually reachable — the native library having loaded, or an -`ldapbind` having been injected — not merely that `ldapenabled` is set. +`ldapavailable` reports whether a bind is actually reachable - the native library having loaded, or an +`ldapbind` having been injected - not merely that `ldapenabled` is set. ### `version` The module version string. @@ -292,11 +292,11 @@ in `init.q`) rather than being hardcoded as a q literal, so a release bump touch This follows the TorqX module convention. `version` remains in the **export dictionary**. `di.depcheck` resolves a dependency's version from its -export dict (`checkdepversion`) and reports `"… exports no version"` — failing the dependency check — +export dict (`checkdepversion`) and reports `"... exports no version"` - failing the dependency check - if a module omits it. Moving the *value* to a file does not move the *export*. ### `getapimeta[]` -This module's api metadata — one `` `name`public`descrip`params`return `` row per callable export, for +This module's api metadata - one `` `name`public`descrip`params`return `` row per callable export, for `di.torq` to collect and register with `di.api`. `init` and `getapimeta` are deliberately absent: they are framework plumbing `di.torq` calls by convention rather than discovers. Needs no `init`. ```q @@ -310,7 +310,7 @@ count perms.getapimeta[] / 33 - 11 top-level exports plus the 22 admin.* membe / version 1b "module version string" / status 1b "what this module is currently enforcing - engine, r.. ``` -The `admin.*` members carry `public:0b` — they are real callables registered in `di.api`'s full view +The `admin.*` members carry `public:0b` - they are real callables registered in `di.api`'s full view but kept out of the public summary, which lists `admin` itself. --- @@ -318,7 +318,7 @@ but kept out of the public summary, which lists `admin` itself. ## Input validation Every public entry point validates its arguments and reports failures the same way as everything else -— prefixed `di.permissions:`, naming the offending argument and the expected type, and logged at +- prefixed `di.permissions:`, naming the offending argument and the expected type, and logged at `error` before being signalled: ```q @@ -328,10 +328,10 @@ perms.admin.grantaccess[`t;`g;`sideways] / 'di.permissions: grantaccess: level must be `read or `write, got `sideways ``` -This matters because the alternative is a raw `'type` or `'length` thrown from a downstream `upsert` — +This matters because the alternative is a raw `'type` or `'length` thrown from a downstream `upsert` - unprefixed, unlogged, and giving no indication which argument was wrong. **Malformed client queries are covered too**: an unparseable string yields -`di.permissions: mainexpr: could not parse query: …` rather than q's bare parse error, so a client +`di.permissions: mainexpr: could not parse query: ...` rather than q's bare parse error, so a client probing with garbage still leaves an audit trail. "Every" is enforced rather than asserted: the suite drives one wrong-typed call at **each** admin @@ -345,7 +345,7 @@ logger, not merely that it was thrown. TorQ permission-checks a select on its **target table only** (`permissions.q`'s `query` inspects nothing but `first q[1]`). A select's where, by and columns clauses can name *other* tables, and those -executed unchecked — so a user granted any single table could read any other: +executed unchecked - so a user granted any single table could read any other: ```q / alice is granted `open and has NO grant on `secret @@ -358,7 +358,7 @@ This module checks every readable object named anywhere in the where, by and col refuses the query unless the user has read access to each. `allowed` applies the same check, so it agrees with `requ` rather than permitting a query `requ` would refuse. -**The predicate is `rbac.isdefinedvar` — the same one a bare reference and a lambda expression use.** +**The predicate is `rbac.isdefinedvar` - the same one a bare reference and a lambda expression use.** That is the point: all three paths agree on what counts as a readable object, so an object cannot be readable through a select clause while a bare reference to it is refused. An earlier revision checked only *table*-valued symbols, which left exactly that inconsistency: @@ -369,7 +369,7 @@ select p:first .pt.secretvec from .pt.trade / returned its contents ``` **The target table's own column names are removed first.** Inside a select, a symbol matching a column -denotes that column, not a same-named global — so checking it would deny ordinary queries. Without the +denotes that column, not a same-named global - so checking it would deny ordinary queries. Without the exclusion, `select id,v from open` is refused on any process that also happens to define globals `id` or `v`; with it, that query is clean and an ungranted global of a colliding name still cannot be read (the suite asserts both, via a `zz` column and a `zz` root global). @@ -384,8 +384,8 @@ or `v`; with it, that query is clean and an ungranted global of a colliding name | `.z.pi` | `exec` | console input, console-formatted results | | `.z.pp` | `exec` | HTTP POST refused outright | | `.z.ws` | `exec` | websocket messages refused outright | -| `.z.pc` | `` ` `` | simple observer — anonymous user cleanup | -| `.h.val` | — | assigned directly; **not** a `.z.*` event | +| `.z.pc` | `` ` `` | simple observer - anonymous user cleanup | +| `.h.val` | - | assigned directly; **not** a `.z.*` event | All registrations use the stable name `` `di.permissions ``, so re-init reclaims rather than collides. @@ -394,7 +394,7 @@ which is what `permissions.q` itself does; claiming `.z.ph`'s `exec` would repla HTTP handler wholesale, including its response formatting. `exec` ownership is not a preference. `di.handlers` rejects a `pre`/`post` registration when no `exec` -owner exists, and on a bare process nothing owns `.z.pg` — so a `pre`-only design cannot register at +owner exists, and on a bare process nothing owns `.z.pg` - so a `pre`-only design cannot register at all. It is also the only way to reproduce the three structurally different composition idioms legacy used on `.z.pw` alone (flat replace, gate-and-call-through, AND-compose) within a single-owner model. @@ -403,12 +403,12 @@ used on `.z.pw` alone (flat replace, gate-and-call-through, AND-compose) within ## Root-name publication `use` mangles module code into a private namespace, so anything an evaluated config file must reach -has to be published at a real root name — the convention TorqX applies for `.gw.*`, `.u.upd` and +has to be published at a real root name - the convention TorqX applies for `.gw.*`, `.u.upd` and `.hdb.reload`. Legacy grant files (`config/permissions/*.q`) are **executable q** calling `.pm.addrole`, -`.pm.grantfunction`, `.pm.ALL` and so on at root. `init` therefore publishes eight names — the seven -grant-script functions plus `ALL` — under `.pm`, so a legacy grant file loads unmodified: +`.pm.grantfunction`, `.pm.ALL` and so on at root. `init` therefore publishes eight names - the seven +grant-script functions plus `ALL` - under `.pm`, so a legacy grant file loads unmodified: ``` .pm.ALL .pm.adduser .pm.addgroup .pm.addrole @@ -421,7 +421,7 @@ Published **permanently during `init`**, but **only when `enabled`**, and remove > because `gateway.q` guards on existence (`` `.pm.valp ~ key `.pm.valp ``) and falls back cleanly, and > it is better: a disabled permissions module should not advertise admin functions that gate nothing. -The query API (`requ`, `allowed`, `val`, `valp`, `execas`) is **not** published at root — `di.gateway` +The query API (`requ`, `allowed`, `val`, `valp`, `execas`) is **not** published at root - `di.gateway` holds a module handle and calls it in-process. --- @@ -483,8 +483,8 @@ perms.status[] perms.teardown[]; ``` -From here a real process would load its grants from files rather than by hand — set `grantdirs` and -let `init` run the `default` → `proctype` → `procname` cascade. See +From here a real process would load its grants from files rather than by hand - set `grantdirs` and +let `init` run the `default` -> `proctype` -> `procname` cascade. See [Root-name publication](#root-name-publication) for why an unmodified TorQ grant file works unchanged. --- @@ -492,24 +492,24 @@ let `init` run the `default` → `proctype` → `procname` cascade. See ## Running tests > **Note on suite structure.** Rows share accumulated state (users, groups and grants created by -> earlier rows), so an individual assertion cannot be run in isolation and an early failure cascades — +> earlier rows), so an individual assertion cannot be run in isolation and an early failure cascades - > a property of the k4unit CSV format rather than a choice here. When debugging, read upward from the > first failing row, not just the row itself. > > The suite **is** verified re-runnable: calling `moduletest` twice in one session passes both times. -> That was not true initially — grant data and LDAP cache state surviving `teardown` broke ten +> That was not true initially - grant data and LDAP cache state surviving `teardown` broke ten > assertions on a second run, which is what surfaced the `unblock` gap documented above. -**Unit suite** (`test.csv`) — no sockets and no native library required. Run it for the current row +**Unit suite** (`test.csv`) - no sockets and no native library required. Run it for the current row and assertion counts rather than trusting a figure quoted here: ```q k4unit:use`di.k4unit k4unit.moduletest`di.permissions ``` -### LDAP coverage — the bind path is exercised via an injected `ldapbind` +### LDAP coverage -`deps` accepts an **optional `ldapbind`** — a function `{[session;dict]}` returning a dict with a +`deps` accepts an **optional `ldapbind`** - a function `{[session;dict]}` returning a dict with a `` `ReturnCode `` key. When supplied it replaces the native library outright, and the `.so` is never resolved. This exists so the caching and lockout logic can be exercised without a directory server: @@ -518,7 +518,7 @@ fakebind:{[sess;d] enlist[`ReturnCode]!enlist 0i} / 0i = success, anything perms.init[`enabled`ldapenabled!(1b;1b);`log`handlers`ldapbind!(logdep;handlersdep;fakebind)] ``` -This is a **`deps` injection, not a config value** — deps are process wiring code the module already +This is a **`deps` injection, not a config value** - deps are process wiring code the module already trusts completely (the injected `log` and `handlers` could subvert it just as thoroughly), so it adds no trust boundary that did not already exist. Operator-editable settings files cannot reach it. @@ -530,14 +530,14 @@ clearing a lockout; `ldapblocktime` expiry releasing one without an explicit unb throws failing closed rather than propagating. The bind's **return shape is validated**: it must be a dictionary containing a `` `ReturnCode `` key. -This is a safety check, not tidiness — `result[\`ReturnCode]` on an integer is *handle apply* in q, so +This is a safety check, not tidiness - `result[\`ReturnCode]` on an integer is *handle apply* in q, so a bind mistakenly returning `42` would attempt an IPC write to file descriptor 42. Anything of the wrong shape now fails closed with a clear message. -> **⚠ Security note on the cache.** After a successful bind, a repeat login by the same user with the +> **NB Security note on the cache.** After a successful bind, a repeat login by the same user with the > same password inside `ldapchecktime` (default 5 minutes) is served **from the cache without -> contacting the server**. That is TorQ's behaviour and it is deliberate — it exists to spare the -> directory server load — but it means **revoking an account server-side is not effective until the +> contacting the server**. That is TorQ's behaviour and it is deliberate - it exists to spare the +> directory server load - but it means **revoking an account server-side is not effective until the > window elapses**. Set `ldapchecktime` to `0D00:00` to disable the optimisation and force every login > to the server. @@ -545,19 +545,19 @@ wrong shape now fails closed with a clear message. four symbols bind at the arities this module uses (`kdbldap_init`/2, `kdbldap_set_option`/3, `kdbldap_bind_s`/4, `kdbldap_err2string`/1), `initialise` opens a session, a real `kdbldap_bind_s` executes, and its failure is decoded by the native `err2string` into -`"Can't contact LDAP server"`. The lockout bookkeeping was driven by those genuine failures — three +`"Can't contact LDAP server"`. The lockout bookkeeping was driven by those genuine failures - three attempts, then lockout, then refusal without contacting the server. Still not covered, and genuinely needing a live directory: a **successful** bind, and directory-specific behaviour such as referrals or TLS negotiation. -**Integration suite** (`test_integration.csv`) — stands up a real child q process on an OS-assigned +**Integration suite** (`test_integration.csv`) - stands up a real child q process on an OS-assigned port and drives a real connection. It exists because **`reval`'s read-only restriction is not applied when `.z.w=0`**: at the console `reval parse "g::1"` happily sets `g`, but over a real handle the same call throws `'noupdate`. k4unit runs in-process at handle 0, so a unit test asserting a blocked write would fail against correct code. -> **⚠ The invocation below is a workaround, not the supported interface.** `di.k4unit.moduletest` +> **NB The invocation below is a workaround, not the supported interface.** `di.k4unit.moduletest` > hardcodes the filename `test.csv` (`di/k4unit/init.q:9`) and its `KUltf`/`KUrt` primitives are not > exported, so there is **no supported way to run a second suite in a module**. The lines below reach > into `di.k4unit`'s private, loader-mangled namespace and will break if that mangling changes. @@ -570,22 +570,22 @@ k4unit:use`di.k4unit .m.di.0k4unit.KUrt[] k4unit.getresults[] ``` -Run it in a fresh session — running it after `moduletest` re-runs the still-loaded unit tests against +Run it in a fresh session - running it after `moduletest` re-runs the still-loaded unit tests against dirty module state. > **What this suite does and does not prove.** It verifies read-only enforcement and parse-tree -> handling over a **real child process and a real IPC handle** — the things that cannot be tested +> handling over a **real child process and a real IPC handle** - the things that cannot be tested > in-process, because `reval` does not enforce at `.z.w=0`. > > It wires **real `di.handlers` only when that module is on `QPATH`**, falling back to a minimal > inline stand-in (a bare `set[ev;f]`) otherwise. `di.handlers` lives on the `feature-handlers` > branch, so a checkout of `feature-permissions` alone runs against the stand-in and does **not** -> exercise real phase/`exec`-ownership dispatch — that is covered by `di.handlers`' own suite, not +> exercise real phase/`exec`-ownership dispatch - that is covered by `di.handlers`' own suite, not > this one. Check out both branches to exercise the real wiring. > > Which path ran is **reported, not assumed**: the suite asserts the child returned a boolean for > `realhandlers`, and the value is visible in the results. This exists because the "prefer real -> `di.handlers`" branch was silently dead for weeks — `h.init` on a function-local throws +> `di.handlers`" branch was silently dead for weeks - `h.init` on a function-local throws > `'h.init` (module dot-sugar resolves only against a *global* name), and the protected apply > swallowed it, so the stand-in always ran while the suite reported green. @@ -593,17 +593,17 @@ dirty module state. ## Migrating from `.pm` / `.access` -- `.pm.allowed`, `.pm.requ`, `.pm.val`, `.pm.valp`, `.pm.execas` → call through the module handle. +- `.pm.allowed`, `.pm.requ`, `.pm.val`, `.pm.valp`, `.pm.execas` -> call through the module handle. `val`/`valp` keep the same *shape* (unary functions), so a consumer that copies the function value is unaffected; only the read-only decision moved from load time to call time. -- `.pm.cando` is **dropped** — it had no callers anywhere and differed from `allowed` only by parsing +- `.pm.cando` is **dropped** - it had no callers anywhere and differed from `allowed` only by parsing first, which `allowed` now does itself. -- Grant scripts need **no change** — the names they call are republished at root. +- Grant scripts need **no change** - the names they call are republished at root. - `.access.*` has no equivalent; see [Engine scope](#engine-scope). - The `-public` command-line flag is replaced by the `public` config key. - **Rejection messages changed prefix.** TorQ emits `"pm: no read permission on [x]"`; this module emits `"di.permissions: : no read permission on [x]"`. Any log scraping or alerting that - greps for `pm:` needs updating — the new prefix does not contain it. + greps for `pm:` needs updating - the new prefix does not contain it. - **`.pm.*` at root is a compatibility shim, not the API.** The eight published names exist so legacy grant files load unmodified. New code should hold a module handle and call `perms.admin.*`; the root namespace looks like legacy TorQ but is a strict subset of it. @@ -614,12 +614,12 @@ dirty module state. - `init` must be called before any other function; every other export checks and errors clearly. - All errors raised after `init` are logged at `error` before being signalled. -- Console input (`.z.pi`) routes through the same permission check as a network query — being local is +- Console input (`.z.pi`) routes through the same permission check as a network query - being local is not an exemption. The `.z.w=0` bypass inside the query path is what keeps the console usable. - `.z.pp` and `.z.ws` are refused outright rather than permission-checked, matching TorQ. - Group membership is chased to a fixed point for *authorization* checks, so nested groups work. - **Public-user detection deliberately does not do that.** It keeps TorQ's first-row lookup - (`` `public~(1!usergroup)[u]`groupname ``), which is correct by construction — an anonymous user is + (`` `public~(1!usergroup)[u]`groupname ``), which is correct by construction - an anonymous user is provisioned into exactly one group. Generalising it to a full membership check would be a privilege change rather than a fix: a real user who is in `public` alongside other groups would then be rejected when presenting a valid password, or, with an empty password, have their user row upserted @@ -629,8 +629,8 @@ dirty module state. - **Config values are type-checked at `init`.** A mistyped setting (`maxsize:"big"`, `readonly:"yes"`, a non-timespan `ldapblocktime`) is rejected immediately, naming every offending key, rather than surfacing later as a confusing runtime error far from its cause. -- **A failed `init` installs nothing.** The one step that depends on external state — resolving the - LDAP native library — runs before anything is published or registered, so a missing `.so` leaves the +- **A failed `init` installs nothing.** The one step that depends on external state - resolving the + LDAP native library - runs before anything is published or registered, so a missing `.so` leaves the process untouched rather than half-configured. - **A grant made against a virtual table's *name* survives `removevirtualtable`.** The two are independent objects, so `allowed` will still permit the name until the grant is revoked separately; diff --git a/di/permissions/permissions.q b/di/permissions/permissions.q index 1a5b2ed8..6071d87d 100644 --- a/di/permissions/permissions.q +++ b/di/permissions/permissions.q @@ -1,30 +1,19 @@ -/ role-based access control and authentication -/ ported from TorQ code/handlers/permissions.q (.pm), writeaccess.q (.readonly), ldap.q (.ldap) -/ and code/common/execas.q; controlaccess.q's tiered engine is deliberately deferred (see the engine -/ config key, which ships from v1 so the tiered engine can land later without reshaping this schema) +/ role-based access control and authentication. ported from TorQ's permissions.q, writeaccess.q, +/ ldap.q and common/execas.q - see permissions.md for scope, omissions and migration notes. +/ the version lives in the VERSION file and is read by init.q -/ NB: the module version is NOT defined here - it is read from the VERSION file in init.q, so a -/ release bump touches one plain-text file rather than a q string literal - -/ ============================================================ / constants (load-time) -/ ============================================================ -/ wildcard object - grants against it mean "any function" / "any table", i.e. superuser -/ NB: TorQ calls this .pm.ALL; the style guide requires lowercase and `all` is a reserved word, so it -/ is `wildcard` internally and republished at root as .pm.ALL for legacy grant scripts (see publishroot) +/ grants against this mean "any function" / "any table", i.e. superuser. republished as .pm.ALL wildcard:`$"*"; -/ the admin functions a legacy grant file (config/permissions/*.q) calls at root, plus the wildcard -/ constant. these are the names publishroot must expose - see the root-publication section +/ the admin functions a legacy grant file calls at root - publishroot exposes exactly these grantscriptnames:`adduser`addgroup`addrole`addtogroup`assignrole`grantaccess`grantfunction; / the engines this module knows about; only rbac is implemented in v1 knownengines:`rbac`tiered; -/ rejection messages, keyed by reason. text ported from permissions.q, minus its "pm: " prefix - -/ every one of these is signalled through raiseerror, which composes "di.permissions: : " itself, -/ so keeping the legacy prefix would double it up +/ rejection messages, keyed by reason. no prefix - raiseerror composes one err:(`symbol$())!(); err[`func]:{"user role does not permit running function [",string[x],"]"}; err[`selt]:{"no read permission on [",string[x],"]"}; @@ -34,9 +23,7 @@ err[`expr]:{"unsupported expression, superuser only"}; err[`quer]:{"free text queries not permissioned for this user"}; err[`size]:{"returned value exceeds maximum permitted size"}; -/ ============================================================ / schema (load-time templates - the live copies are .z.m.*, populated in init) -/ ============================================================ userschema:([id:`symbol$()]authtype:`symbol$();hashtype:`symbol$();password:()); groupinfoschema:([name:`symbol$()]description:()); @@ -49,48 +36,29 @@ functionschema:([]object:`symbol$();role:`symbol$();paramcheck:()); virtualtableschema:([name:`symbol$()]table:`symbol$();whereclause:()); publictrackschema:([name:`symbol$()]handle:`int$()); -/ ldap login-attempt cache -/ NB: TorQ's cache also carries server/port columns, populated from .ldap.server and .ldap.port - -/ neither of which is defined anywhere in ldap.q (the setting is `servers`, plural), so that upsert -/ throws on the first login attempt. nothing ever reads the two columns back, so they are dropped -/ here rather than fixed: they are write-only columns fed by a broken write +/ ldap login-attempt cache. TorQ's server/port columns are dropped - write-only, and fed by a +/ write that throws (it reads .ldap.server, which ldap.q never defines) ldapcacheschema:([user:`symbol$()]pass:();time:`timestamp$(); attempts:`long$();success:`boolean$();blocked:`boolean$()); -/ ============================================================ / config defaults -/ ============================================================ - -/ every key this module accepts, with its default. init warns on anything else rather than dropping it -/ silently. keys are uniquely named so they survive di.config's flat cascade - notably the ldap block is -/ prefixed, because legacy ships four separate `enabled` settings (.pm .access .readonly .ldap) that -/ would otherwise collapse onto one another -/ ignorelist is a MIXED list (symbols and strings) - TorQ's zpsignore.q matches the head of an async -/ message against both forms, e.g. (`upd;"upd";`.u.upd;".u.upd"), so it cannot be a typed symbol vector. -/ it defaults EMPTY here, unlike zpsignore.q which ships enabled with that list: silently exempting -/ upd from permission checks is not a safe default for an access-control module. a process that takes -/ .u.upd-shaped feed traffic must set it explicitly - see permissions.md -/ publishroot: expose the legacy .pm.* names at root so an unmodified TorQ grant file loads. defaults -/ ON for migration compatibility, but a deployment with no legacy grant files can set it 0b and leave -/ the root namespace untouched - the module's own API is reached through the module handle regardless + +/ every accepted key with its default; init warns on anything else. keys are uniquely named to +/ survive di.config's flat cascade. see permissions.md for what each one does. +/ NB ignorelist is a MIXED list (symbols and strings) - zpsignore.q matches an async message head +/ against both forms - so it cannot be a typed symbol vector configdefaults:`enabled`engine`maxsize`runmode`permissivemode`readonly`public`ignorelist`grantdirs`proctype`procname`publishroot! (0b;`rbac;200000000;1b;0b;0b;0b;();();`;`;1b); -/ ldap settings. ldapenabled defaults OFF, matching TorQ's shipped config/settings/default.q rather -/ than ldap.q's own file default of (.z.o~`l64) - so the suite runs with no native library present -/ every ldap setting is read from .z.m.config at call time (one storage location, no bare-name -/ ambiguity) - including by the default ldapbuilddn below, which is why it is explicit about it +/ ldap settings; ldapenabled defaults OFF so the suite runs with no native library present. +/ every ldap setting is read from .z.m.config at call time, including by the default ldapbuilddn ldapconfigdefaults:`ldapenabled`ldaplibpath`ldapdebug`ldapservers`ldapversion`ldapblocktime`ldapchecklimit`ldapchecktime`ldapbuilddnsuf`ldapbuilddn! (0b;"";0b;enlist `$"ldap://localhost:0";3;0D00:30:00;3;0D00:05;"";{"uid=",string[x],",",.z.m.config`ldapbuilddnsuf}); -/ ============================================================ / internal helpers -/ ============================================================ requiresym:{[ctx;nm;x] / validate a public-api argument that must be a symbol - / without this a wrong type escapes as a raw 'type or 'length from a downstream upsert - unprefixed, - / unlogged, and with no indication which argument was wrong if[-11h<>type x; raiseerror[ctx;nm," must be a symbol, got ",.Q.s1 type x]]; }; @@ -102,12 +70,9 @@ requirestring:{[ctx;nm;x] }; normdescription:{[ctx;x] - / normalise a description to a char VECTOR, and reject anything that is not text. - / ⚠ this is load-bearing, not cosmetic. roleinfo/groupinfo declare `description:()` - a general - / list - and q COLLAPSES a general column to a typed vector as soon as it holds only atoms. a - / one-character description like "x" is a char ATOM, so it turns the column into a char vector, - / and the next multi-character description then throws a bare 'type. same for a symbol description. - / normalising to a vector here means the column only ever receives vectors and stays general + / normalise a description to a char VECTOR - load-bearing, not cosmetic. description:() is a + / general column and q collapses one to a typed vector once it holds only atoms; a 1-char + / description is a char ATOM, so it would collapse the column and the next longer one throws 'type if[-10h=type x;:enlist x]; if[10h<>type x; raiseerror[ctx;"description must be a string, got ",.Q.s1 type x]]; @@ -143,14 +108,10 @@ requireinit:{[ctx] '"di.permissions: ",string[ctx],": init must be called before any other function"]; }; -/ ============================================================ / init -/ ============================================================ validatedeps:{[deps] - / log and handlers are both required and never defaulted - there is no fallback logger. - / legacy permissions.q logs nothing at all, so every rejected login and denied query is currently - / silent; a silent fallback here would make that silence look deliberate + / log and handlers are both required and never defaulted - there is no fallback logger if[99h<>type deps; '"di.permissions: deps must be a dict with `log and `handlers keys - see di.log, di.handlers"]; if[not all `log`handlers in key deps; @@ -162,10 +123,8 @@ validatedeps:{[deps] '"di.permissions: log dict must have `info`warn`error keys; got: ",(", " sv string key deps`log)]; if[99h<>type deps`handlers; '"di.permissions: handlers value must be a dict; pass `register`remove functions - see di.handlers"]; - / only `register`remove are required - this module never calls `list (nor `version). requiring a key - / the code path does not exercise is friction with no safety payoff. NB this is unrelated to - / di.depcheck's handlers contract, which checks that di.handlers EXPORTS register/remove/list - - / a statement about the provider's export dict, not about any consumer's injected-deps dict + / only register/remove are required - this module calls no others. unrelated to di.depcheck's + / handlers contract, which checks the PROVIDER's export dict, not a consumer's deps dict if[not all `register`remove in key deps`handlers; '"di.permissions: handlers dict must have `register`remove keys; got: ",(", " sv string key deps`handlers)]; / ldapbind is OPTIONAL - when supplied it replaces the native bind entirely (see ldap.bind) @@ -186,11 +145,8 @@ resolveconfig:{[config] :defaults,(key[defaults] inter key config)#config; }; -/ expected value shapes, grouped by check. a mistyped setting must fail at init with a clear message -/ rather than surfacing later as a confusing runtime error far from its cause -/ engine is deliberately absent - validateengine gives a better message for it -/ ignorelist and grantdirs are deliberately absent - both accept several shapes and are normalised -/ ldapdebug is a bare on/off flag, not a level - ldap.debuglog reads it as `if[...]`, nothing grades it +/ expected value shapes, grouped by check. engine, ignorelist and grantdirs are deliberately +/ absent - engine gets a better message from validateengine, the other two are normalised boolconfigkeys:`enabled`readonly`permissivemode`runmode`public`ldapenabled`publishroot`ldapdebug; intconfigkeys:`maxsize`ldapversion`ldapchecklimit; symconfigkeys:`proctype`procname; @@ -199,8 +155,8 @@ spanconfigkeys:`ldapblocktime`ldapchecktime; validateconfig:{[cfg] / type-check every setting whose shape is fixed, reporting all offenders of a kind at once - / NB: the parameter is `ks`, NOT `keys` - a parameter named `keys` throws 'nyi when the function is - / called, even though (`keys in .Q.res) is 0b. .Q.res is not exhaustive; test, don't trust it + / NB the parameter is `ks`, NOT `keys` - a `keys` parameter throws 'nyi at CALL time, and + / (`keys in .Q.res) is 0b, so .Q.res will not warn you chk:{[cfg;ks;ok;what] bad:ks where not ok each cfg ks; if[count bad; @@ -216,8 +172,7 @@ validateconfig:{[cfg] }; validateengine:{[eng] - / v1 implements rbac only. tiered (TorQ's controlaccess.q) is deferred, but the key ships now so it - / can land later without reshaping the config schema + / v1 implements rbac only; the key ships now so tiered can land without reshaping the schema if[not -11h=type eng; raiseerror[`init;"engine must be a symbol, one of: ",", " sv string knownengines]]; if[eng~`tiered; @@ -239,21 +194,16 @@ resettables:{[] .z.m.virtualtable:virtualtableschema; .z.m.publictrack:publictrackschema; .z.m.ldapcache:ldapcacheschema; - / ldap.initialise normally sets these; default them so an injected bind (which skips the native - / library entirely) still has a session value to pass through. - / ldapready is an EXPLICIT flag: inferring availability from ldapsession merely existing would - / report the native library as ready the moment this default was added + / defaulted so an injected bind (which skips the native library) still has a session to pass. + / ldapready is EXPLICIT - inferring readiness from ldapsession existing would always report ready .z.m.ldapsession:0i; .z.m.ldapready:0b; }; init:{[config;deps] - / wire the injected dependencies, resolve config, and (when enabled) install this module as the - / owner of the message-handling .z.* events - / config: a dict of settings, or (::) for defaults. deps: a dict with `log and `handlers keys - / example: perms.init[`enabled`readonly!(1b;1b);`log`handlers!(logdep;handlersdep)] - / idempotent - a second call re-wires deps and config and reclaims the same handler registrations, - / leaving existing grant data intact + / wire deps, resolve config, and (when enabled) claim the message-handling .z.* events. + / config: a dict of settings, or (::) for defaults. deps: a dict with `log and `handlers. + / idempotent - a second call reclaims the same registrations and leaves grant data intact validatedeps[deps]; .z.m.loginfo:(deps`log)`info; .z.m.logwarn:(deps`log)`warn; @@ -278,17 +228,14 @@ init:{[config;deps] if[not .z.m.enabled; .z.m.loginfo[`init;"di.permissions loaded but disabled - no handlers registered, nothing published at root"]; :(::)]; - / an injected bind replaces the native library outright, so the .so is never resolved in that case. - / resolve the native library FIRST, because it is the one step that can fail on external state. - / doing it before anything is installed means a missing .so leaves the process untouched rather than - / half-configured with root names published and no handlers registered + / resolve the native library FIRST - it is the one step that can fail on external state, so a + / missing .so leaves the process untouched rather than half-configured. an injected bind skips it if[(cfg`ldapenabled) and (::)~.z.m.ldapbind;ldap.initialise[ldap.resolvelibpath[]]]; / root names next: grant files call .pm.addrole etc. on their first line $[cfg`publishroot;publishroot[]; .z.m.loginfo[`init;"publishroot is 0b - .pm.* not exposed at root; legacy grant files will not load"]]; - / a grant file is arbitrary q and may throw. unwind the root publication and mark the module - / disabled before rethrowing, so a failed init never leaves .pm.* published with no handlers - / registered and status[] reporting enabled + / a grant file is arbitrary q and may throw - unwind root publication before rethrowing, so a + / failed init never leaves .pm.* published with nothing registered @[loadpermissions;::;{[e] unpublishroot[]; .z.m.enabled:0b; @@ -299,14 +246,9 @@ init:{[config;deps] }; ensurepublicscaffolding:{[] - / the anonymous-user path assigns `publicuser and adds to `public. both assignrole and addtogroup - / REFUSE an undefined role/group (as TorQ's do), so with public enabled and neither defined, every - / anonymous login THROWS out of .z.pw instead of connecting or being cleanly refused. - / TorQ has the same gap; it was simply unreachable there because the -public flag threw 'type first - / (see authenticate). fixing that made this reachable, so the module now provides its own - / scaffolding - created only when ABSENT, so a grant file's own definitions and descriptions win. - / the role and group are created EMPTY: an anonymous user gets nothing until an operator grants to - / them, so this is fail-closed, not a privilege grant + / the anonymous path assigns `publicuser and adds to `public, and assignrole/addtogroup REFUSE an + / undefined name - so without these every anonymous login throws out of .z.pw. created only when + / ABSENT (a grant file's own definitions win) and EMPTY, so an anonymous user is fail-closed if[not .z.m.public;:(::)]; if[not `publicuser in key .z.m.roleinfo; admin.addrole[`publicuser;"anonymous users - created by di.permissions, holds no grants by default"]; @@ -325,14 +267,8 @@ status:{[] .z.m.config`ldapenabled;$[(::)~.z.m.ldapbind;@[{.z.m.ldapready};::;0b];1b]); }; -/ ============================================================ / admin api - grant data management (the admin.* dotted group) -/ ============================================================ -/ these are what a legacy config/permissions/*.q grant file calls, and what publishroot exposes at -/ .pm.* so such a file loads unmodified. every one is an idempotent table mutation - -/ the wildcard object, exposed so a caller can grant superuser: admin.grantfunction[admin.wildcard;...] -/ TorQ exposes the same constant as .pm.ALL, which is what publishroot republishes it as +/ what a legacy grant file calls, and what publishroot exposes at .pm.*. all idempotent admin.wildcard:wildcard; admin.adduser:{[u;authtype;hashtype;password] @@ -384,9 +320,7 @@ admin.addtogroup:{[u;g] requiresym[`addtogroup;"user";u]; requiresym[`addtogroup;"group name";g]; if[not g in key .z.m.groupinfo;raiseerror[`addtogroup;"no such group, call admin.addgroup first: ",string g]]; - / NB: upsert, not join. TorQ writes `usergroup,:(u;g)`, whose amend-in-place semantics insert a row; - / the explicit `.z.m.x:.z.m.x,(...)` rewrite this module needs is NOT equivalent - on an empty table - / it flattens to a plain list. upsert on an unkeyed table appends, and the guard above stops duplicates + / NB upsert, not join: `.z.m.x:.z.m.x,(...)` flattens an EMPTY table to a plain list if[not (u;g) in .z.m.usergroup;.z.m.usergroup:.z.m.usergroup upsert (u;g)]; }; @@ -433,10 +367,8 @@ admin.grantaccess:{[o;e;l] requireinit[`grantaccess]; requiresym[`grantaccess;"object";o]; requiresym[`grantaccess;"entity";e]; - / NB: type-check BEFORE the membership test - ("read" in `read`write) throws a raw 'type, which - / would escape unprefixed and unlogged before this check could report anything useful + / NB type-check BEFORE the membership test - ("read" in `read`write) throws a raw 'type requiresym[`grantaccess;"level";l]; - / an unrecognised level is silently useless - it is stored but can never match a check if[not l in `read`write; raiseerror[`grantaccess;"level must be `read or `write, got ",.Q.s1 l]]; if[not (o;e;l) in .z.m.access;.z.m.access:.z.m.access upsert (o;e;l)]; @@ -451,9 +383,9 @@ admin.revokeaccess:{[o;e;l] }; admin.grantfunction:{[o;r;p] - / grant a role the right to call a function, gated by paramcheck p - / p MUST be a function - a paramcheck is applied to the call's parameter dict and any non-boolean - / result is coerced to 0b, so a literal 1b stored here fails closed rather than granting access + / grant a role the right to call a function, gated by paramcheck p. + / p MUST be a function - a non-boolean paramcheck result is coerced to 0b, so a literal 1b would + / fail closed rather than grant access requireinit[`grantfunction]; if[not type[p] within 100 112h;raiseerror[`grantfunction;"paramcheck must be a function; a literal fails closed"]]; if[not (o;r;p) in .z.m.function;.z.m.function:.z.m.function upsert (o;r;p)]; @@ -503,9 +435,8 @@ admin.cloneuser:{[u;unew;p] requirestring[`cloneuser;"password";p]; if[not u in key .z.m.user;raiseerror[`cloneuser;"no such user to clone: ",string u]]; ul:raze exec authtype,hashtype from .z.m.user where id=u; - / NB: hash directly. TorQ builds the string (string hashtype)," string `",p and EVALUATES it, which - / throws on any password containing a space ('word) or a backtick ('type), and evaluates - / caller-supplied text in an auth path. md5 p is identical for a well-formed password and total + / NB hash directly - TorQ builds and EVALUATES a string here, which throws on a password + / containing a space or a backtick, and evaluates caller-supplied text in an auth path if[not `md5~ul 1; raiseerror[`cloneuser;"cannot clone user with unsupported hashtype ",string[ul 1],"; only md5 is supported"]]; admin.adduser[unew;ul 0;ul 1;md5 p]; @@ -513,9 +444,7 @@ admin.cloneuser:{[u;unew;p] admin.assignrole[unew;] each exec role from .z.m.userrole where user=u; }; -/ ============================================================ / rbac engine - permission checks -/ ============================================================ rbac.pdict:{[f;a] / build a parameter-name -> value dict for a call, so a paramcheck can inspect arguments by name @@ -554,13 +483,10 @@ rbac.achk:{[u;t;rw;pr] :exec 0=5; }; @@ -577,20 +503,18 @@ rbac.qexe:{[x] }; rbac.exe:{[x] - / evaluate an expression, choosing parse-tree vs string evaluation by the head's type - / NB: only heads of type 102/103/105-112 (operators, iterators, compositions) reach val - a symbol - / head is 11h and a lambda head is 100h, so both fall through to valp, as does a string. that is - / fine because valp handles all three shapes; it did NOT before the parse guard was added there + / evaluate an expression, choosing parse-tree vs string evaluation by the head's type. + / NB only operator/iterator/composition heads reach val; symbol (11h) and lambda (100h) heads fall + / through to valp, as does a string - valp handles all three v:$[(104<>a)&100type s;:0b]; if[null s;:0b]; - / a view IS a readable object and must be permission-checked - but `get` would EVALUATE it, which - / would let an unpermissioned caller trigger arbitrary view computation before any check runs. - / recognise it by name instead. views[] lists every view in the process: only a root-level :: - / creates a lazy view, a namespaced one (.ns.x::) is evaluated eagerly at definition and is an - / ordinary variable thereafter. TorQ never evaluated anything here, and neither does this + / a view must be permission-checked, but `get` would EVALUATE it - letting an unpermissioned + / caller trigger arbitrary computation before any check runs. recognise it by name instead if[s in views[];:1b]; :@[{100h>type get x};s;0b]; }; rbac.lamq:{[u;e;b;pr] - / permission-check a lambda-shaped expression by finding which defined root variables it references - / and checking read access on each, reporting every disallowed reference at once - / NB: this tokenises FIRST and tests only those tokens. TorQ enumerates every variable in every root - / namespace and intersects, which is O(all names) per query - measured at 2.9ms on a process with - / 5000 root names versus 0.085ms on a small one, a 34x cliff on exactly the RDB/HDB shapes this - / module targets. Testing the handful of tokens actually referenced is O(tokens) and equivalent + / read-check every defined root variable a lambda references, reporting all failures at once. + / NB tokenises FIRST and tests only those tokens - O(tokens), not TorQ's O(all root names) pq:`$distinct -4!raze(rbac.str rbac.flatten e),'" "; rqt:pq where rbac.isdefinedvar each pq; / public objects are always readable rqt:rqt except distinct exec object from .z.m.access where entity=`public; prohibited:rqt where not rbac.achk[u;;`read;pr] each rqt; - / a dry run reports a verdict; only a real execution raises. TorQ raises either way, which makes - / `allowed` - documented as returning a boolean - throw instead for lambda-shaped queries + / a dry run reports a verdict; only a real execution raises (TorQ raises either way) if[count prohibited; $[b;raiseerror[`lamq;" | " sv err[`selt] each prohibited];:0b]]; :$[b;rbac.exe e;1b]; }; -/ ============================================================ / rbac engine - top-level classifier -/ ============================================================ rbac.isvar:{[x] / is x a symbol naming an existing non-function variable? @@ -748,14 +657,10 @@ rbac.mainexpr:{[u;e;b;pr] :$[b;rbac.exe ie;1b]; }; -/ execute-and-check. TorQ binds these as load-time projections over runmode/permissivemode; here they -/ read config at CALL time, so both settings are tunable at runtime (same load-time-binding fix as -/ val/valp below) +/ execute-and-check. reads runmode/permissivemode at CALL time, so both are tunable at runtime rbac.expr:{[u;e] :rbac.mainexpr[u;e;.z.m.runmode;.z.m.permissivemode]}; -/ ============================================================ / query normalisation and the public entry points -/ ============================================================ rbac.destringf:{[x] :$[(s:`$x) in key `.q;.q s;s~`insert;insert;any (100h;104h)=type first f:@[parse;x;0];f;s]; @@ -767,45 +672,30 @@ rbac.parsequery:{[q] }; val:{[x] - / evaluate a parse tree, under reval when read-only mode is on - / TorQ fixes this at LOAD time (val:$[readonly;reval;eval]), so read-only cannot be toggled without - / a restart; resolving per call fixes that. gateway.q copies this function value, so the shape it - / consumes is unchanged - / ⚠ TESTING: reval's read-only restriction is NOT applied when .z.w=0 (the console). Verified on - / KDB-X 5f/2025.11.17: at the console `reval parse "g::1"` happily sets g, but over a real IPC - / handle the same call throws 'noupdate. k4unit runs in-process at handle 0, so a unit test that - / asserts a write is blocked will FAIL against correct code - assert the selection instead, and - / cover actual enforcement in an integration test with a child process (see di.handlers' pattern) + / evaluate a parse tree, under reval when read-only mode is on. resolved per CALL, not at load + / time as TorQ does, so read-only is togglable without a restart. + / NB TESTING: reval does NOT enforce at .z.w=0 (the console), only over a real handle - so a unit + / test asserting a blocked write FAILS against correct code. enforcement lives in the integration suite requireinit[`val]; :$[.z.m.readonly;reval x;eval x]; }; valp:{[x] - / evaluate a string or parse tree, under reval when read-only mode is on - / ⚠ BUG FIX vs TorQ: legacy is `valp:$[readonly;{reval parse x};value]` (permissions.q:9). `parse` - / requires a STRING and throws 'type on a list, so under readonly every parse-tree input threw - / instead of evaluating. rbac.exe routes BOTH symbol heads (type 11h) and lambda heads (100h) here, - / which is the standard sync/async IPC call shape h(`func;arg) - so a readonly process (canonically - / an HDB) rejected the most common client idiom with a bare 'type + / evaluate a string or parse tree, under reval when read-only mode is on. + / NB parse only a STRING - it throws 'type on a list, and rbac.exe routes parse trees here, which + / is the standard h(`func;arg) IPC shape requireinit[`valp]; if[not .z.m.readonly;:value x]; - / read-only. a STRING parses then evaluates, exactly as legacy did - `parse` inserts the literal - / markers so eval and value agree on it. - / a PARSE TREE must keep `value`'s semantics, which resolve the head but NOT the arguments. handing - / it to `reval` directly would use EVAL semantics, which resolve a symbol argument to the variable it - / names: (`echo;`secret) would return the contents of `secret` to a caller with no grant on it, since - / only the head is permission-checked. that is a read bypass, and it would exist ONLY in read-only - / mode - strictly more permissive than the same call with readonly off, which is the wrong direction. + / NB a PARSE TREE must keep `value` semantics (head resolved, arguments NOT). handing it straight + / to `reval` uses EVAL semantics, which resolve a symbol argument to its variable: (`echo;`secret) + / would return secret's contents to a caller with no grant on it, since only the head is checked. / applying value to the tree as a literal inside reval keeps value's semantics and reval's write ban :$[10h=type x;reval parse x;reval (value;enlist x)]; }; allowed:{[u;q] - / would user u be permitted to run q? a dry-run verdict - never executes - / NB: TorQ pins permissive mode OFF here (allowed:mainexpr[;;0b;0b]) regardless of config, so on a - / permissive-mode process it denies things requ then permits. a pre-check that disagrees with the - / execution it precedes is a defect, not a feature - di.gateway gates on this - so it reads the - / configured value and the two agree + / would user u be permitted to run q? a dry-run verdict - never executes. + / NB reads the CONFIGURED permissive mode (TorQ pins it off), so allowed and requ agree requireinit[`allowed]; requiresym[`allowed;"user";u]; requirequery[`allowed;q]; @@ -823,38 +713,29 @@ requ:{[u;q] execas:{[f;u] / run f as user u, subject to that user's permissions - / TorQ's execas.q guards on .pm.requ existing; inside the module requ always exists and itself - / handles the disabled case, so the guard is redundant and dropped requireinit[`execas]; requiresym[`execas;"user";u]; requirequery[`execas;f]; :requ[u;f]; }; -/ ============================================================ / ldap authentication backend (the ldap.* dotted group) -/ ============================================================ -/ ported from TorQ code/handlers/ldap.q. the native library is OPTIONAL: nothing here is touched -/ unless ldapenabled is set, so the module and its whole test suite run with no .so present +/ the native library is OPTIONAL - nothing here is touched unless ldapenabled is set ldap.resolvelibpath:{[] - / the native library, from the ldaplibpath setting, falling back to $KDBLIB like TorQ - / TorQ has no @[value;...] guard on .ldap.lib, so it is overridable only via $KDBLIB; making it a - / real setting (with the same fallback) follows di.kafka's libpath pattern + / the native library, from the ldaplibpath setting, falling back to $KDBLIB p:.z.m.config`ldaplibpath; :$[0type customdict;raiseerror[`ldap;"bind overrides must be (::) or a dictionary"]]; upddict:(defaultkeys!```),customdict; - / dispatch through the injected bind when one was supplied, else the native library. - / the injected form exists so the caching and lockout logic below can be exercised without a real - / directory server. it is a DEPS injection, not a config value: deps are process wiring code that - / this module already trusts completely (the injected log and handlers could subvert it just as - / easily), so this adds no trust boundary that did not already exist + / dispatch through the injected bind when supplied, else the native library. the injected form + / lets the caching and lockout logic be exercised without a directory server r:$[(::)~.z.m.ldapbind;.z.m.ldapbindnative[sess;;;]. upddict defaultkeys;.z.m.ldapbind[sess;upddict]]; - / validate the shape before returning. an unchecked non-dict is dangerous, not merely wrong: - / r[`ReturnCode] on an integer is HANDLE APPLY, so a bind returning 42 attempts an IPC write to - / file descriptor 42. fail closed with a clear message instead + / NB validate the shape: r[`ReturnCode] on an INTEGER is handle apply, so a bind returning 42 + / would attempt an IPC write to file descriptor 42. fail closed instead if[99h<>type r; raiseerror[`ldap;"bind returned a ",(.Q.s1 type r)," - expected a dictionary with a `ReturnCode key"]]; if[not `ReturnCode in key r; @@ -946,18 +823,14 @@ unblock:{[usr] if[not usr in key .z.m.ldapcache; .z.m.loginfo[`unblock;"no ldap login record for user ",string usr]; :(::)]; - / clear the whole cached state, not only a lockout. TorQ returns early unless the user is blocked, - / which leaves no way to force re-authentication for a user who is merely cached - after a password - / change their cached success stands until ldapchecktime elapses. clearing `success` here means the - / next login always reaches the server + / clear the whole cached state, not only a lockout - clearing `success` means the next login + / always reaches the server, which is what makes this usable after a password change wasblocked:.z.m.ldapcache[usr]`blocked; .z.m.ldapcache:update attempts:0,success:0b,blocked:0b from .z.m.ldapcache where user=usr; .z.m.loginfo[`unblock;$[wasblocked;"unblocked user ";"cleared cached ldap state for user "],string usr]; }; -/ ============================================================ / authentication backends (the auth.* dotted group) -/ ============================================================ / one function per authtype, so a user row's authtype selects its backend auth.local:{[u;p] @@ -971,24 +844,16 @@ auth.ldap:{[u;p] :$[.z.m.config`ldapenabled;ldap.login[u;p];0b]; }; -/ ============================================================ / connection lifecycle - the bodies di.handlers registers -/ ============================================================ authenticate:{[u;p] - / the .z.pw body: authenticate a connecting user, optionally auto-provisioning an anonymous one - / ⚠ BUG FIX vs TorQ: legacy gates the anonymous path on `if["B"$(.Q.opt .z.x)[`public][0;0]]`. - / with no -public flag that is `if[`boolean$()]`, which throws 'type - so on any process started - / without -public, login THROWS for every unknown user instead of returning 0b. replaced with the - / `public` boolean config key. (droppublic's `any` form was accidentally safe; this one was not) + / the .z.pw body: authenticate a connecting user, optionally auto-provisioning an anonymous one. + / anonymous access is gated by the `public` config key, replacing TorQ's -public command-line read requireinit[`authenticate]; - / public detection deliberately keeps TorQ's FIRST-ROW semantics: 1! keys usergroup on user and a - / lookup returns only the first matching row. that is correct by construction here - the - / provisioning branch below puts an anonymous user in exactly one group - and generalising it to a - / full membership check would be a privilege change, not a fix: a real user who happens to be in - / `public alongside other groups would then either be REJECTED when presenting a valid password, or - / (with an empty password) have their user row upserted over and their role demoted to publicuser. - / that is an account-takeover path, so the narrower legacy check is the safer contract + / NB FIRST-ROW lookup is deliberate, not a bug. correct by construction - the branch below puts an + / anonymous user in exactly one group - and a full membership check would be a privilege CHANGE: + / a real user also in `public would be rejected with a valid password, or have their row upserted + / over and role demoted. that is an account-takeover path. see permissions.md known:u in key .z.m.user; ingrouppublic:`public~(1!.z.m.usergroup)[u]`groupname; if[(not known) or ingrouppublic; @@ -1008,8 +873,7 @@ authenticate:{[u;p] if[not ud[`authtype] in key auth; .z.m.logwarn[`authenticate;"rejected ",(string u),": unknown authtype ",string ud`authtype]; :0b]; - / log both outcomes: a warn-only trail records rejections but leaves successful logins invisible, - / so the audit cannot answer "who connected". ldap successes were logged only under ldapdebug + / log both outcomes - a warn-only trail cannot answer "who connected" ok:auth[ud`authtype][u;p]; $[ok;.z.m.loginfo[`authenticate;"authenticated user ",string u]; .z.m.logwarn[`authenticate;"failed authentication for user ",string u]]; @@ -1030,10 +894,8 @@ droppublic:{[w] .z.m.loginfo[`droppublic;"dropped anonymous user ",string u]; }; -/ ============================================================ / handler bodies - registered with di.handlers, never exported -/ ============================================================ -/ these are passed to register BY VALUE, so they need no public name +/ passed to register BY VALUE, so they need no public name hooks.sync:{[x] / .z.pg exec: permission-check and run a synchronous message @@ -1042,18 +904,15 @@ hooks.sync:{[x] }; hooks.async:{[x] - / .z.ps exec: as hooks.sync, but first honouring the ignore-list - / this is zpsignore.q's behaviour folded inline. di.handlers always calls the exec owner after - / folding the pre phase - there is no skip-exec path - so the bypass has to live here, not as a phase. - / TorQ applies it to .z.ps ONLY, and that is preserved: .z.pg is not exempted + / .z.ps exec: as hooks.sync, but first honouring the ignore-list (zpsignore.q, folded inline - + / di.handlers has no skip-exec path, so the bypass cannot be a phase). .z.ps ONLY, as in TorQ if[any first[x]~/:.z.m.ignorelist;:value x]; :$[.z.w=0;value x;requ[.z.u;x]]; }; hooks.console:{[x] - / .z.pi exec: console input. blank lines skip the check; results are console-formatted, as in TorQ - / NB: TorQ routes console input through the same permission check as a network query - being local - / is not an exemption; the .z.w=0 bypass inside hooks.sync is what makes the console usable + / .z.pi exec: console input. blank lines skip the check; results are console-formatted. + / console input IS permission-checked - the .z.w=0 bypass in hooks.sync is what keeps it usable :$[x in (1#"\n";"");.Q.s value x;.Q.s $[.z.w=0;value x;requ[.z.u;x]]]; }; @@ -1071,18 +930,11 @@ hooks.rejectws:{[x] execbodies:`.z.pw`.z.pg`.z.ps`.z.pi`.z.pp`.z.ws! (authenticate;hooks.sync;hooks.async;hooks.console;hooks.rejectpost;hooks.rejectws); -/ ============================================================ / root-name publication -/ ============================================================ -/ `use` mangles module code into a private namespace, so anything an evaluated config file or a remote -/ caller must reach has to be published at a real root name - the convention TorqX applies for .gw.*, -/ .u.upd, .hdb.reload and .logroll.rollnow. -/ config/permissions/*.q grant files call .pm.addrole, .pm.grantfunction, .pm.ALL etc. at root, so -/ without this a legacy grant file fails on its first line. -/ published permanently during init (a shim scoped to one loadpermissions call would not be a shim), -/ but ONLY when enabled - a disabled module should not advertise admin functions that gate nothing. -/ NB: this diverges from TorQ, which defines .pm.* regardless of enabled; safe because gateway.q -/ guards on existence (`.pm.valp ~ key `.pm.valp) and falls back cleanly +/ `use` mangles module code into a private namespace, so anything an evaluated grant file must +/ reach has to be published at a real root name - legacy files call .pm.addrole etc on line one. +/ published during init but ONLY when enabled, unlike TorQ; safe because gateway.q guards on +/ existence. see permissions.md, "Root-name publication" publishroot:{[] / expose the wildcard constant and the grant-script admin functions at .pm.* @@ -1097,9 +949,7 @@ unpublishroot:{[] if[count present;![`.pm;();0b;present]]; }; -/ ============================================================ / grant data loading -/ ============================================================ loadgrantfile:{[path] / load one grant file at ROOT (not via `use`) so its .pm.* calls resolve against the published names @@ -1112,10 +962,8 @@ loadgrantfile:{[path] loadpermissions:{[] / load the grant cascade: default -> proctype -> procname, under each configured directory - / mirrors TorQ's `.proc.loadconfig[dir;] each `default,proctype,procname` requireinit[`loadpermissions]; - / normalise grantdirs: a bare string is a single directory, not a list of one-char directories. - / without this, (),"path" degrades to a char list and every char is treated as a directory + / NB normalise: a bare string is ONE directory - (),"path" would make each char a directory dirs:.z.m.config`grantdirs; dirs:$[10h=type dirs;enlist dirs;(),dirs]; if[0=count dirs; @@ -1123,32 +971,25 @@ loadpermissions:{[] :(::)]; names:`default,(.z.m.config`proctype),.z.m.config`procname; names:names where not null names; - / NB: nested each, NOT `cross`. cross joins with `,` so a path STRING is concatenated with the - / symbol rather than paired with it - ("/tmp/gp" cross `default) is an 8-item mixed list, and - / dot-applying that to a binary function rank-errors + / NB nested each, NOT cross - cross joins with `,`, concatenating the path STRING with the + / symbol instead of pairing them, giving a mixed list that rank-errors on dot-apply {[nms;d] {[d;n] loadgrantfile[d,"/",(string n),".q"]}[d;] each nms}[names;] each dirs; }; -/ ============================================================ / registration and teardown -/ ============================================================ registerhandlers:{[] - / claim the exec phase of every message-handling event, plus a .z.pc observer for cleanup - / registered under a stable name so a re-init reclaims the same events rather than colliding + / claim the exec phase of every message-handling event, plus a .z.pc observer for cleanup. + / a stable name means a re-init reclaims the same events rather than colliding {[e] .z.m.register[e;`exec;`di.permissions;0;execbodies e]} each key execbodies; - / priority 0 on .z.pc - lower runs first, so this cleanup precedes any other observer. that is the - / faithful port: TorQ chains .z.pc as {droppublic[y];@[x;y]} (permissions.q:260), cleanup first and - / the prior handler after. it is also the right layering - TorqX registers its gateway .z.po/.z.pc - / connection bookkeeping at priority 10, so the security teardown lands ahead of it, and nothing in - / that bookkeeping depends on the user record droppublic removes + / priority 0 - lower runs first, so this cleanup precedes any other observer, matching TorQ's + / {droppublic[y];@[x;y]} order. TorqX registers gateway bookkeeping at 10, behind this .z.m.register[`.z.pc;`;`di.permissions;0;droppublic]; - / .h.val is where HTTP GET permissioning actually happens on kdb+ 3.5+; it is not a .z.* event, so - / di.handlers' register would reject the symbol - assign it directly, keeping the original to restore. - / .z.ph is deliberately NOT claimed: an exec owner there replaces the built-in handler wholesale - / capture the ORIGINAL .h.val once only - init is idempotent and re-runs this, so an unguarded - / capture on a second init would record our own hooks.sync as the "original" and teardown would - / restore that instead of kdb+'s built-in + / .h.val is where HTTP GET permissioning happens on kdb+ 3.5+. not a .z.* event, so di.handlers + / would reject it - assign directly. .z.ph is NOT claimed: an exec owner there replaces the + / built-in handler wholesale. + / NB capture the original ONCE - init is idempotent, so an unguarded capture on a second init + / would record our own hooks.sync as the "original" if[not @[{.z.m.hvaloriginal;1b};::;0b];.z.m.hvaloriginal:@[get;`.h.val;{(::)}]]; set[`.h.val;hooks.sync]; .z.m.loginfo[`init;"registered exec on ",(", " sv string key execbodies),", observer on .z.pc, and .h.val"]; @@ -1161,8 +1002,8 @@ teardown:{[] if[not .z.m.enabled; .z.m.loginfo[`teardown;"di.permissions is disabled, nothing to release"]; :(::)]; - / NB: dot-apply, not @. `@[f;(a;b;c);h]` passes the three-element LIST as one argument to a ternary - / function, which rank-errors straight into the handler - so every removal silently "succeeded" + / NB dot-apply, not @ - `@[f;(a;b;c);h]` passes the LIST as one argument, rank-errors into the + / handler, and every removal silently "succeeds" {[e] .[.z.m.removehandler;(e;`exec;`di.permissions);{[e2] .z.m.logwarn[`teardown;"could not remove exec handler: ",e2]}]} each key execbodies; .[.z.m.removehandler;(`.z.pc;`;`di.permissions);{[e2] .z.m.logwarn[`teardown;"could not remove .z.pc observer: ",e2]}]; $[(::)~.z.m.hvaloriginal;@[{![`.h;();0b;enlist`val];};::;{[e2]}];set[`.h.val;.z.m.hvaloriginal]]; @@ -1171,16 +1012,11 @@ teardown:{[] .z.m.loginfo[`teardown;"di.permissions released - handlers, .h.val and .pm.* root names removed"]; }; -/ ============================================================ / api metadata -/ ============================================================ getapimeta:{[] - / this module's api metadata, one row per CALLABLE export, for di.torq to collect and register with - / di.api. init and getapimeta are omitted: di.torq calls those two by convention rather than - / discovering them, so they are plumbing, not API. teardown is NOT plumbing - it is an ordinary - / lifecycle operation a caller needs documented, so it gets a normal row. - / names are bare; di.torq applies the process-wide qualification + / one row per CALLABLE export, for di.torq to register with di.api. init and getapimeta are + / omitted as plumbing; teardown is a real lifecycle operation and gets a row. names are bare :flip `name`public`descrip`params`return!flip( (`teardown; 1b; "release handler registrations, .h.val and the published .pm.* root names"; "[]"; "null"); @@ -1204,10 +1040,8 @@ getapimeta:{[] "[]"; "null"); (`unblock; 1b; "clear a user's ldap lockout"; "[symbol: user]"; "null"); - / the admin sub-api, enumerated so di.api can discover all of it rather than one opaque entry. - / public:0b - these are real callables registered in di.api's full (f) view but kept out of the - / public (p) summary, which lists `admin itself. the suite asserts the non-dotted names still - / match the export list exactly + / the admin sub-api, enumerated rather than hidden behind one opaque entry. public:0b - real + / callables in di.api's full view, kept out of the public summary, which lists `admin itself (`admin.adduser; 0b; "register a user with an authentication method and a hashed password"; "[symbol: user; symbol: authtype (local|ldap); symbol: hashtype (md5); string: hashed password]"; "null"); (`admin.removeuser; 0b; "remove a user entirely"; "[symbol: user]"; "null"); From 60493f65aae17a625fdae9d1d97293b8cc74fc8f Mon Sep 17 00:00:00 2001 From: alowrydi Date: Thu, 6 Aug 2026 12:46:42 +0100 Subject: [PATCH 08/11] typo fix --- di/permissions/permissions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/di/permissions/permissions.md b/di/permissions/permissions.md index 1fb91d51..ba1fc832 100644 --- a/di/permissions/permissions.md +++ b/di/permissions/permissions.md @@ -5,7 +5,7 @@ message-handling `.z.*` event via the injected `di.handlers` dependency, permiss incoming query against a user's roles and groups, and optionally enforces a whole-process read-only mode. -Consolidates five TorQ files: `code/handlers/permissions.q` (`.pm`), `writeaccess.q` (`.readonly`), +Consolidates four TorQ files: `code/handlers/permissions.q` (`.pm`), `writeaccess.q` (`.readonly`), `ldap.q` (`.ldap`), and `code/common/execas.q`. TorQ's `controlaccess.q` tiered engine is **deferred** - see [Engine scope](#engine-scope). From f965162b93121b10185ea8f946a63d1f3c9ac510 Mon Sep 17 00:00:00 2001 From: alowrydi Date: Thu, 6 Aug 2026 13:08:01 +0100 Subject: [PATCH 09/11] Remove accidental depcheck module artefact --- di/depcheck/depcheck.md | 132 ---------- di/depcheck/depcheck.q | 406 ------------------------------- di/depcheck/deps.q | 4 - di/depcheck/init.q | 6 - di/depcheck/test.csv | 183 -------------- di/depcheck/test_integration.csv | 34 --- 6 files changed, 765 deletions(-) delete mode 100644 di/depcheck/depcheck.md delete mode 100644 di/depcheck/depcheck.q delete mode 100644 di/depcheck/deps.q delete mode 100644 di/depcheck/init.q delete mode 100644 di/depcheck/test.csv delete mode 100644 di/depcheck/test_integration.csv diff --git a/di/depcheck/depcheck.md b/di/depcheck/depcheck.md deleted file mode 100644 index 44c46011..00000000 --- a/di/depcheck/depcheck.md +++ /dev/null @@ -1,132 +0,0 @@ -# di.depcheck - -Dependency, version, core-contract, and `.z.ts`-ownership auditing for kdb-x modules. Runs once at process startup — after every module has been loaded — and reports any declared dependency that is missing, out of date, or fails the shared core-dependency contracts. It is the modernised successor to legacy TorQ's `checkdependency`/`runchk`/`checkvers`: a per-module manifest replaces the old CSV registry, and real numeric semver replaces the 5-component digit-walk. - ---- - -## Features - -- **A post-load audit, not a pre-load gate** — it runs after the host process (`di.torq`) has already `use`d every module it needs, and never calls `use` on anything it checks. Each module's state is read by introspecting the session namespace the kdb-x loader populates: every loaded module lands under `` `.m.di `` keyed by a short form (`di.timer` → `` `.m.di.0timer ``), and that module's `export` dict is readable there directly. "Not found" in a report means a *declared* dependency was never loaded — it is not a filesystem scan. -- **Presence & minimum version** — for each loaded module that ships a manifest (symbol-keyed by dependency name, string-valued by minimum version), every declared dependency is checked: is it loaded, and does its exported `version` satisfy the declared minimum (real numeric `major.minor.patch` comparison)? A module with no manifest is skipped, not failed. -- **Dual-format manifests (`deps.q` and/or `deps.toml`)** — a module's dependencies are read from **both** a `deps.q` (a q dict literal) and a `deps.toml` (a `[dependencies]` section), wherever each exists, merged with **`.toml` winning on a clash** — mirroring di.config's `parsetier` so both formats can coexist mid-migration. `di.toml` is loaded **lazily and only when a `deps.toml` actually exists**; a module with only `deps.q` never triggers it. -- **Transitive manifest-graph walk** — beyond each loaded module's direct deps, `checkgraph` walks the graph **on disk** (reading each peer's own manifest whether or not it is loaded), cycle-guarded via a visited set, and reports a **presence** failure for any dependency reached at depth ≥ 2 that resolves nowhere on QPATH. It loads no module code. Transitive *version* checking is deferred (see Notes); direct (depth-1) deps keep the stronger presence=*loaded*/version=*exported* check and are excluded from the walk so the two never double-report. -- **Core dependency contracts** — for whichever of `di.log`, `di.timer`, or `di.handlers` are loaded, the export dict is checked for the required keys of the contract it provides (Logging: `info`/`warn`/`error`; Timer: `addjob`/`deletejobs`/`enablejobs`/`disablejobs`/`getactivejobs`/`cp`; Handlers: `register`/`remove`/`list`). The single-contract check is also exported directly as `checkcontract[provider;requiredkeys]` for a contract this module doesn't know by name. -- **`.z.ts` ownership** — warns if `.z.ts` is bound while di.timer is absent or uninitialised (its `enabled` flag is the proxy for "di.timer's `init` ran and bound `.z.ts`"). Warning-level only, and cannot detect a later rebind — an accepted limitation. -- **kdb-x engine version** (optional) — compares the running engine's `.z.K` against an optional `` `minkdbxversion `` passed alongside `log` on the same `deps` dict. Warning-level; no caller supplies a minimum today, so it is a no-op in every real invocation. - -Presence, version, and core-contract failures are **fail-fast** — `init` logs a single multi-line report at `error`, then signals, so the caller sees a blocking error. The `.z.ts` and kdb-x-version checks are **warning-only** — logged at `warn`, never signalled. - -``` -DEPENDENCY CHECK FAILED: - di.tplog requires minimum version 0.2.0, found 0.1.3 - di.pubsub requires minimum version 0.3.0, not found - -WARNING: - .z.ts has been directly assigned outside di.timer. This may cause timer conflicts. -``` - -A loaded dependency that exports no `version` gets its own line — `di.timer requires minimum version 1.0.0, but di.timer exports no version` — and a transitive-only dependency missing from QPATH gets `di.zzc is required transitively by di.zzb but was not found on QPATH`. - ---- - -## Dependencies - -| Dependency | Key | Required | Description | -|---|---|---|---| -| logger | `` `log `` | yes | dict with `info`, `warn`, and `error`, each binary `{[c;m]}` where `c` is a symbol context and `m` is a string (per `consistency.md`) | -| kdb-x minimum version | `` `minkdbxversion `` | no | an optional float compared against `.z.K` | - -**No hard dependencies** on other `di.*` modules — the module works standalone, and ships its own (empty) `deps.q`, dogfooding the convention it introduces. - -**`di.toml` is a soft, lazy dependency** — it is not declared in `deps.q` and not loaded at import time. It is resolved once (cached) and called **only** when a module being audited ships a `deps.toml` file. A process whose modules use only `deps.q` never loads it, so it is not required to be on QPATH in that case. If a `deps.toml` *does* exist and `di.toml` is missing or fails to parse it, that is reported as one aggregated failure line (see Notes) rather than throwing. - -The `log` dependency must be passed to `init` inside the `deps` dict keyed on `` `log ``, and must already match the binary `{[c;m]}` contract — the module validates key presence but does not detect or adapt other shapes (e.g. a monadic `kx.log` instance). To use `di.log`, pass its `logdict`. - ---- - -## Initialisation - -`init[deps]` takes a single dictionary combining the required `log` dependency with an optional kdb-x minimum version. - -| Key | Required | Description | -|---|---|---| -| `` `log `` | yes | Log dep — `info`/`warn`/`error`, each binary `{[c;m]}` | -| `` `minkdbxversion `` | no | Minimum kdb-x engine version, compared against `.z.K`. Default: unchecked | - -`init` must be called **after** every module the host process needs has already been `use`d — typically the very last thing `di.torq` does during startup. It audits the fully-loaded session, so anything loaded later is not seen. - ---- - -## Exported Functions - -### `init[deps]` -Validate the required `log` dependency, then run every check against the current session and report. Throws on any presence, version, or core-contract failure; logs (but does not throw on) `.z.ts`-ownership or kdb-x-version warnings. -```q -depcheck.init[enlist[`log]!enlist logdep] -/ with an optional minimum kdb-x version: -depcheck.init[`log`minkdbxversion!(logdep;5.0)] -``` - -### `version` -The module version string. -```q -depcheck.version / "0.1.0" -``` - -### `checkcontract[provider;requiredkeys]` -Standalone version of the per-contract check that `checkcontracts[]` runs automatically for the three known core dependencies — checks whether `provider`'s export dict (if it's loaded) contains every key in `requiredkeys`. Never calls `use`, matching this module's introspection-only design. Returns `()` on a pass, or if `provider` isn't loaded at all; returns an enlisted failure line naming the missing keys otherwise. `requiredkeys` accepts either a symbol vector or a single bare symbol atom. -```q -depcheck.checkcontract[`di.timer;`addjob`deletejobs] / () - di.timer really exports both -depcheck.checkcontract[`di.timer;`addjob`cp] / enlist "di.timer is missing required contract key(s): cp" -depcheck.checkcontract[`di.timer;`cp] / bare atom works the same as enlist`cp -``` - ---- - -## Usage Example - -```q -/ log dep must already match the binary {[c;m]} contract - write your own, or use di.log: -/ logging:use`di.log -/ depcheck.init[enlist[`log]!enlist logging.logdict] -logdep:`info`warn`error!({[c;m]};{[c;m]};{[c;m]}) - -/ di.torq loads every module the process needs first... -timer:use`di.timer -handlers:use`di.handlers - -/ ...then depcheck audits the fully-loaded session, last -depcheck:use`di.depcheck -depcheck.init[enlist[`log]!enlist logdep] -``` - ---- - -## Running Tests - -```q -k4unit:use`di.k4unit -k4unit.moduletest`di.depcheck -``` - -Run in a fresh q session — `moduletest` doesn't reset its internal result table between calls, so a second call duplicates this file's rows. The unit suite drives real, already-shipped modules rather than synthetic fixtures: **di.timer** (merged to `main`) is always loaded and exercises real gaps (no `version`; `cp` defined but unexported), while **di.kafka**, **di.handlers** (the positive control — the one module that does export `version`), **di.log**, and **di.toml** (on its own `feature-toml` branch) are all on unmerged PRs, so every assertion depending on them gracefully no-ops when the module isn't resolvable on QPATH — the suite passes standalone on a bare `feature-depcheck` checkout, exercising the real path only when the module happens to be present. The manifest readers, dual-format merge, and transitive walk are exercised against scratch modules written under a QPATH root at runtime — the q-only / toml-only / both-formats-clashing / neither cases, a deliberately-broken `deps.toml`, and an A→B→C chain with a B→A cycle — all cleaned up afterwards, with no hardcoded paths and no committed fixtures. The di.toml-dependent assertions among these no-op when di.toml is absent. - -The **integration suite** (`test_integration.csv`) spins up a real, separate child kdb-x process (the child must itself be kdb-x-capable to `use` the modules) to prove two things the unit suite structurally cannot: that a genuine zero-failure run completes cleanly end to end, and that a real signalled failure terminates a real host process with a non-zero exit code and the real report in captured output. `moduletest` only ever loads `test.csv`, so load and run this suite directly: -```q -k4unit:use`di.k4unit -.m.di.0k4unit.KUltf .Q.dd[hsym`$.Q.m.mp`di.depcheck;`test_integration.csv] -.m.di.0k4unit.KUrt[] -k4unit.getresults[] / one row per assertion; ok=1 is a pass -``` - ---- - -## Notes - -- **`di.toml` coupling.** `deps.toml` is inert data parsed by the `di.toml` module (no evaluation), whereas `deps.q` is a q dict literal read by executing it. Module keys in a `[dependencies]` section must be **quoted** — `"di.timer" = "0.2.0"` — because di.toml rejects unquoted dotted keys. di.depcheck reads only string version values via `parsefile`, so it is unaffected by di.toml's scalar value-typing. All format-specific reading lives behind `finddepsq`/`readdepsq`/`finddepstoml`/`readdepstoml`/`readdeps`; every other function consumes only their merged, format-agnostic dict, so dropping a format later is a change to those readers alone. Which format the repo ultimately standardises on is an open cross-team decision (with the TorqX POC), deliberately not resolved here. -- **Deliberate divergence from di.config.** di.config's `requiretoml` throws and aborts its entire settings cascade the instant a `.toml` tier can't be read, because it must hand back one complete, correct config. di.depcheck does the **opposite** on the same event — it catches, folds one clearly-worded line (matching `requiretoml`'s wording) into the aggregate report, and keeps walking — because its whole purpose is to surface *every* problem across *many* modules in one pass. Two right answers to two different jobs, not an inconsistency. -- **Transitive presence is walked; transitive version is not, yet.** `checkgraph` reports a missing transitive dependency at depth ≥ 2, but cannot check the *version* of an unloaded module without either loading it (which the walk must not do) or a per-module `VERSION` file. `VERSION` files are not yet a repo-wide convention — modules carry an inline exported `version` — so adopting them is a coordinated rollout; the walk gains transitive version checking for free once they land. -- **`.z.ts` ownership is a best-effort proxy.** di.timer's `enabled` flag evidences that its `init` bound `.z.ts`; it cannot detect something rebinding `.z.ts` afterwards. Scope is deliberately limited to `.z.ts` only. -- **The kdb-x-version check is shape-only for now.** The comparison is implemented and unit-tested in isolation, but no caller supplies `` `minkdbxversion `` yet, so it is a no-op in every real invocation until di.torq (or another caller) threads a real minimum through. It is warning-only, not fail-fast. Uses `.z.K`, not `.z.v`, matching di.k4unit's existing precedent. -- **Semver is numeric `X.Y.Z` only** — no pre-release/build-metadata support, matching every version string in this codebase. A malformed declared minimum or exported found-version is reported as its own distinct failure line rather than silently mis-compared. During 0.x.y development a passing `>=` check does not guarantee contract compatibility. -- **Modules currently missing a `version` export** — di.timer, di.kafka, and di.log were all found without one; di.handlers is the only module that exports it (and its own comments call that a placeholder pending this module). Adding `version` everywhere is a coordinated repo-wide rollout, of which di.depcheck is the consumer side. di.compression separately imports `kx.log` directly rather than following the binary `{[c;m]}` convention — a pre-`consistency.md` outlier worth its own cleanup. diff --git a/di/depcheck/depcheck.q b/di/depcheck/depcheck.q deleted file mode 100644 index 0a56b6f2..00000000 --- a/di/depcheck/depcheck.q +++ /dev/null @@ -1,406 +0,0 @@ -/ dependency, version, core-contract, and .z.ts ownership auditing for already-loaded kdb-x modules -/ this module never calls `use` on anything it checks - every check reads other modules' state purely by -/ introspecting the session namespace the kdb-x `use` loader already populates (`.m.di.0`), so it stays -/ standalone with no hard di.* dependency of its own - -/ module version - di.depcheck is the first module to carry one, dogfooding the convention it introduces -version:"0.1.0"; - -/ the fixed set of core dependency contracts di.depcheck knows how to validate, keyed by the module that provides -/ each one - not a generic self-declaration registry, since no such mechanism exists anywhere in this codebase yet -contracts:`di.log`di.timer`di.handlers!( - `info`warn`error; - `addjob`deletejobs`enablejobs`disablejobs`getactivejobs`cp; - `register`remove`list - ); - -/ ============================================================ -/ session-namespace introspection helpers -/ ============================================================ - -shortmod:{[modname] - / di.timer -> `0timer ; kx.log -> `0log - the short form the kdb-x `use` loader keys a module under, off its - / vendor's `.m. namespace. Assumes exactly one dot (vendor.name), matching every module name seen in - / this codebase so far - `$"0",last "." vs string modname - }; - -shorttofull:{[s] - / `0timer -> `di.timer - only ever applied to keys of `.m.di (checkdeps walks di.* modules specifically, see - / checkdeps), so the di. prefix is correct here and does not need vendor-generalising like shortmod/modvendorns - `$"di.",1_string s - }; - -modvendorns:{[modname] - / di.timer -> `.m.di ; kx.log -> `.m.kx - the top-level session namespace a module's vendor is keyed under. - / a di.* module's deps.q may legitimately declare a hard dependency on an external vendor module (e.g. - / kx.log), so dependency resolution (getexport/checkonedep/checkcontract) must not hardcode `.m.di - only - / checkdeps's walk of which modules to audit as consumers is intentionally di.*-scoped - `$".m.",first "." vs string modname - }; - -getexport:{[modname] - / read another already-loaded module's export dict purely via session-namespace introspection - no `use`, no import - / returns (::) if the module isn't loaded, or if its export somehow can't be read - sn:shortmod modname; - vns:modvendorns modname; - if[not sn in key vns;:(::)]; - @[get;`$(string vns),".",(string sn),".export";{(::)}] - }; - -/ ============================================================ -/ deps.q loading -/ ============================================================ - -finddepsq:{[modname] - / locate /deps.q on QPATH (colon-separated, like PATH); returns its file path, or (::) if the module ships none - relpath:(ssr[string modname;".";"/"]),"/deps.q"; - roots:":" vs getenv`QPATH; - paths:{[relpath;root] hsym `$root,"/",relpath}[relpath;] each roots; - found:paths where not {[p] ()~key p} each paths; - $[0=count found;(::);first found] - }; - -readdepsq:{[modname] - / load 's deps.q - a single pure `deps:...` assignment, by convention (the only real precedent, di.merge, - / has no other content) - and capture its value without leaving a stray global `deps` behind - p:finddepsq modname; - if[p~(::);:(::)]; - @[system;"l ",1_string p;{[e] (::)}]; - d:@[get;`deps;{(::)}]; - delete deps from `.; - d - }; - -/ ============================================================ -/ deps.toml loading (lazy, file-existence-gated di.toml) -/ ============================================================ - -finddepstoml:{[modname] - / locate /deps.toml on QPATH - sibling of finddepsq; returns its file path, or (::) if the module ships none - relpath:(ssr[string modname;".";"/"]),"/deps.toml"; - roots:":" vs getenv`QPATH; - paths:{[relpath;root] hsym `$root,"/",relpath}[relpath;] each roots; - found:paths where not {[p] ()~key p} each paths; - $[0=count found;(::);first found] - }; - -resolvetoml:{[] - / lazily resolve di.toml once and cache it in module state, returning its export dict. throws if di.toml is not - / resolvable on QPATH - caught by readdepstoml and turned into one aggregated failure line, never an abort. only ever - / called when a real deps.toml file has already been found, so a module with no deps.toml never triggers a di.toml load - cached:@[get;`.z.m.tomlmod;{(::)}]; - if[not cached~(::);:cached]; - m:use`di.toml; - .z.m.tomlmod:m; - m - }; - -readdepstoml:{[modname] - / read /deps.toml if it exists, returning (failures;depsdict). di.toml is touched ONLY when the file exists - / (file-existence-gated, exactly like di.config's parsefile checks existence before dispatching on extension). a missing - / or broken di.toml degrades to one aggregated failure line worded like di.config's requiretoml (name the file/module, - / name the underlying cause, one sentence) - but deliberately does NOT throw: di.config's requiretoml aborts its whole - / cascade because it must return one complete config, whereas di.depcheck must surface every module's problems in one - / pass, so it catches, records a line, and keeps walking (see depcheck.md) - p:finddepstoml modname; - if[p~(::);:(();()!())]; - path:1_string p; - @[{[mn;pth] - d:(resolvetoml[])[`parsefile] pth; - / a well-formed manifest has a [dependencies] section (a dict); absent -> nothing declared; present but not a - / dict (e.g. `dependencies = "x"` written as a scalar) is a clear authoring error, reported not merged (a - / non-dict here would otherwise throw out of readdeps's merge and abort the whole walk) - $[not `dependencies in key d;(();()!()); - 99h=type d`dependencies;(();d`dependencies); - (enlist "di.depcheck: deps.toml for ",(string mn), - " has a malformed [dependencies] section - expected a table of module = \"version\" entries";()!())] - }[modname;]; - path; - {[mn;e] (enlist "di.depcheck: cannot read deps.toml for ",(string mn), - " - the di.toml module was not found on QPATH or failed to parse it; di.toml is required to read .toml manifests (underlying: ",e,")"; - ()!())}[modname;]] - }; - -readdeps:{[modname] - / merged manifest reader: reads deps.q and deps.toml where each exists and merges them, with deps.toml winning on a key - / clash - mirroring di.config's live parsetier ((parsefile base,".q"),parsefile base,".toml"). returns (failures;dict): - / failures aggregates a malformed-deps.q line and/or an unreadable-deps.toml line; dict is the merged symbol->minversion - / mapping (empty if neither format is present). all format-specific reading lives here and in finddepsq/finddepstoml - - / every downstream function (checkonedep/checkmoduledeps/checkdeps/checkgraph) consumes only this already-merged dict - dq:readdepsq modname; - qmalformed:(not dq~(::)) and not 99h=type dq; - qdict:$[qmalformed or dq~(::);()!();dq]; - tr:readdepstoml modname; - malfail:$[qmalformed;enlist string[modname]," deps.q is malformed - expected a dict, got type ",string type dq;()]; - (malfail,tr 0;qdict,tr 1) - }; - -/ ============================================================ -/ semver comparison -/ ============================================================ - -parsesemver:{[v] - / parse a "major.minor.patch" string into a 3-long int vector; a version that does not parse cleanly as three - / all-numeric parts collapses to a single (0Ni;0Ni;0Ni) "malformed" sentinel, checked via ismalformed, rather - / than left to silently participate in numeric comparison one component at a time - a lone bad component (e.g. - / a pre-release tag like "1.2.3-rc1") used to null out only itself, which could silently make a real, newer - / version compare as lower than it should, or make a typo'd deps.q minver like "abc" silently compare as no - / real minimum at all. Caught by direct testing, not by reading the code - see depcheck.md - parts:"." vs v; - if[not 3=count parts;:3#0Ni]; - nums:{@[{"I"$x};x;0Ni]} each parts; - $[any null nums;3#0Ni;nums] - }; - -ismalformed:{[v] - / true if v does not parse as a clean major.minor.patch triple - see parsesemver - (3#0Ni)~parsesemver v - }; - -vercmp:{[a;b] - / -1/0/1 comparing semver strings a and b by (major,minor,patch); a null component sorts lowest - / real numeric semver comparison only - no pre-release/build-metadata support, matching every version string seen - / in this codebase so far (all plain X.Y.Z), unlike legacy TorQ's 5-component digit-walk - pa:parsesemver a; - pb:parsesemver b; - diffs:pa<>pb; - $[not any diffs;0i;[i:diffs?1b;$[pa[i]= check does not guarantee contract compatibility - minor bumps may - / carry breaking changes pre-1.0 across this workstream. Implementing literal >= anyway, per the plan; this is a - / known, accepted gap, not something this function tries to solve - not -1i=vercmp[a;b] - }; - -/ ============================================================ -/ dependency presence/version checks -/ ============================================================ - -checkfoundversion:{[dep;minver;foundver] - / dep is loaded and exports a version - compares it against the declared minimum - / a malformed minver (a deps.q authoring typo) or malformed foundver (a module exporting a non-semver string) - / is reported explicitly here rather than silently entering vergte's numeric comparison - see parsesemver - if[ismalformed minver; - :enlist string[dep]," has a declared minimum version of ",minver,", which is not a valid major.minor.patch version"]; - if[ismalformed foundver; - :enlist string[dep]," exports version ",foundver,", which is not a valid major.minor.patch version"]; - $[vergte[foundver;minver];();enlist string[dep]," requires minimum version ",minver,", found ",foundver] - }; - -checkdepversion:{[dep;minver] - / dep is confirmed loaded - checks its exported version, if any, against minver - / NOTE: sequential if[] early returns, not a single `or`-combined condition - q's `or`/`and` are eager vector - / operators, not short-circuiting, so `(xp~(::)) or not `version in key xp` would evaluate `key xp` even when - / xp is (::) and throw 'type. Caught by direct testing, not by reading the code - see depcheck.md - xp:getexport dep; - noversionmsg:enlist string[dep]," requires minimum version ",minver,", but ",string[dep]," exports no version"; - if[xp~(::);:noversionmsg]; - if[not `version in key xp;:noversionmsg]; - checkfoundversion[dep;minver;xp`version] - }; - -checkonedep:{[dep;minver] - / checks a single declared (dependency;minimum-version) pair against the current session - / returns () on pass, or an enlisted failure line matching the plan's exact report format - / not vendor-restricted to di.* - a deps.q may name an external vendor module (e.g. kx.log) as a hard - / dependency, so presence is checked against dep's own vendor namespace, not hardcoded to `.m.di - / a manifest version must be a string (quoted in deps.toml, a q string in deps.q). a non-string value - an unquoted - / deps.toml version parsed as a float/int by di.toml, or a symbol/number in deps.q - is an authoring error reported - / as its own clear line, rather than left to corrupt the concatenated message or throw out of parsesemver's `vs`. - / 10h=abs type accepts a char vector or a lone char atom (both string-ish), rejecting int/float/symbol - if[not 10h=abs type minver; - :enlist string[dep]," has a non-string minimum version in its manifest (got type ",(string type minver),") - versions must be quoted strings"]; - depshort:shortmod dep; - vns:modvendorns dep; - $[not depshort in key vns; - enlist string[dep]," requires minimum version ",minver,", not found"; - checkdepversion[dep;minver]] - }; - -checkmoduledeps:{[modshort] - / checks one already-loaded module's declared deps (deps.q and/or deps.toml) against the current session. returns - / aggregated failure lines: manifest read-failures (a malformed deps.q or an unreadable deps.toml, both pre-collected - / by readdeps rather than thrown) plus each declared dependency's presence/version line. one bad manifest never aborts - / the checkdeps[] walk and masks other modules' real failures - readdeps catches, checkonedep is a pure per-pair function - modname:shorttofull modshort; - r:readdeps modname; - merged:r 1; - (r 0),$[0=count merged;();raze checkonedep'[key merged;value merged]] - }; - -checkdeps:{[] - / walks every loaded di.* module's deps.q and checks each declared dependency for presence and minimum version - / "not found" here means a declared dependency was never `use`d into this session - this is a post-load audit, - / not a QPATH filesystem scan (see depcheck.md for why) - / if two different loaded modules declare the same dependency at different minimums, each is checked - / independently and both lines are emitted if both fail - checkonedep is a pure function of (dep;minver) with - / no shared state across calls, so this needs no special handling. Manually verified with two real deps.q - / fixtures on disk (di.handlers required at both a satisfied and an unsatisfied minimum simultaneously) since - / no real module ships a non-empty deps.q yet to build a committed, portable test against - see depcheck.md - raze checkmoduledeps each key `.m.di - }; - -/ ============================================================ -/ transitive dependency-manifest graph walk -/ ============================================================ - -resolvemodule:{[modname] - / reimplements the kdb-x `use` loader's QPATH search (colon-separated roots, first match wins; a dotted module name's - / dots become path segments) to test whether a module is INSTALLED on QPATH without loading it - returns the resolved - / module directory (hsym) or (::) if nothing matches. adapted from TorqX di.depcheck's resolvemodule; deliberately does - / not reach into kdb-x's undocumented .Q.m.* internals. used only for transitive presence, distinct from checkonedep's - / loaded-check: a module can be installed-on-QPATH yet not loaded-into-the-session - relpath:ssr[string modname;".";"/"]; - roots:":" vs getenv`QPATH; - exts:(".q";".k";".q_";".k_"); - dirs:{[rp;root] root,"/",rp}[relpath;] each roots; - hit:{[dir;exts] any {[d;e] 0= 2 that does not resolve on QPATH. depth-1 (deps directly - / declared by a loaded module) stays with checkdeps (presence=loaded, version=exported); checkgraph excludes those via - / the directdeps set so the two never double-report. satisfies consistency.md's on-disk / loads-no-module-code walk - / without a pre-load pass. LIMITATION: transitive VERSION checking of an unloaded module is not done here - a found - / version needs loading (forbidden) or a per-module VERSION file (deferred until a repo-wide rollout) - see depcheck.md - roots:shorttofull each key `.m.di; - directdeps:distinct raze {[r] key (readdeps r) 1} each roots; - acc:`visited`fails!(`symbol$();()); - acc:{[covered;directdeps;acc;root] visit[covered;directdeps;acc;root]}[roots;directdeps]/[acc;roots]; - acc`fails - }; - -/ ============================================================ -/ core dependency contract checks -/ ============================================================ - -/ generic, exported primitive: checks whether a loaded module's export dict contains every key a given contract -/ requires. vendor-agnostic like checkonedep/getexport, though every current contracts entry happens to be -/ di.*-prefixed. usable for any provider/contract pair, not just the three known core dependencies checkcontracts[] -/ audits automatically below. never calls `use` - introspection only, matching this module's whole design; unlike -/ its TorqX counterpart of the same name, which does call `use` for real -/ requiredkeys accepts a single symbol atom as well as a vector - every internal caller (contracts, below) already -/ passes a vector, but this is now a public entry point for a caller auditing a contract of exactly one key, which -/ is naturally written as a bare symbol rather than remembering to `enlist` it - a real boundary this module didn't -/ have before it was exported. caught by direct testing, not by reading the code -checkcontract:{[provider;requiredkeys] - requiredkeys:$[-11h=type requiredkeys;enlist requiredkeys;requiredkeys]; - sn:shortmod provider; - vns:modvendorns provider; - if[not sn in key vns;:()]; - xp:getexport provider; - if[xp~(::);:enlist string[provider]," is loaded but its export dict could not be read"]; - missing:requiredkeys where not requiredkeys in key xp; - $[0=count missing;();enlist string[provider]," is missing required contract key(s): ",", " sv string missing] - }; - -checkcontracts:{[] - / checks whichever of the known core-dependency providers (di.log/di.timer/di.handlers) are loaded in this session - raze checkcontract'[key contracts;value contracts] - }; - -/ ============================================================ -/ .z.ts ownership check -/ ============================================================ - -ztscheck:{[] - / warns if .z.ts is bound to something while di.timer either isn't loaded or doesn't look initialised - / LIMITATION (accepted, warning-level only): di.timer's `enabled` flag being 1b is evidence its init ran and bound - / .z.ts itself - it does NOT prove nothing has overwritten .z.ts since. di.timer assigns .z.ts directly (not via - / di.handlers, which explicitly excludes .z.ts from its own scope), so there is no ownership marker to check - / instead. Scope is deliberately limited to .z.ts only - whether this should ever cover other .z.* events is an - / open question for di.handlers' owner, not decided here - bound:not (::)~@[get;`.z.ts;{(::)}]; - if[not bound;:()]; - timerowns:(`0timer in key `.m.di) and 1b~@[get;`.m.di.0timer.enabled;0b]; - if[timerowns;:()]; - enlist ".z.ts has been directly assigned outside di.timer. This may cause timer conflicts." - }; - -/ ============================================================ -/ kdb-x engine version check -/ ============================================================ - -kdbxcheck:{[deps] - / compares the running kdb-x engine's major.minor (.z.K) against an optional minimum passed via deps`minkdbxversion - / uses .z.K, matching di.k4unit's own precedent (`minver<=.z.K` gates which tests run) rather than parsing .z.v, - / whose value on this box ("5.0.20260122") did not match the plan's assumed kdb-x product-semver shape and isn't - / confirmed to be the same number as the product version shown in the kdb-x startup banner - see depcheck.md - / warning-level, not fail-fast: there is no config cascade yet for di.depcheck to source a minimum from, so this - / is opt-in via an extra key on the same `deps dict rather than a new config channel - minversion:$[`minkdbxversion in key deps;deps`minkdbxversion;0Nf]; - if[null minversion;:()]; - if[.z.K>=minversion;:()]; - enlist "kdb-x engine version ",(string .z.K)," is below the configured minimum ",(string minversion),"." - }; - -/ ============================================================ -/ report formatting -/ ============================================================ - -buildreport:{[header;lines] - / formats a bulleted, indented block under a header line, matching the plan's exact "HEADER:\n line1\n line2" shape - header,":\n",sv["\n";" ",/:lines] - }; - -/ ============================================================ -/ public api -/ ============================================================ - -init:{[deps] - / initialise di.depcheck - validate the required log dependency, then audit the current session: dependency - / presence/version, core-dependency-contract shape, .z.ts ownership, and (optionally) a minimum kdb-x engine - / version. deps: a dict with a required `log key (binary `info`warn`error functions, per consistency.md) and an - / optional `minkdbxversion float - if[99h<>type deps;'"di.depcheck: deps must be a dict with `log key"]; - if[not `log in key deps;'"di.depcheck: log dependency is required; pass `info`warn`error functions - see di.log"]; - if[99h<>type deps`log;'"di.depcheck: log value must be a dict; pass `info`warn`error functions"]; - if[not all `info`warn`error in key deps`log; - '"di.depcheck: log dict must have `info`warn`error keys; got: ",(", " sv string key deps`log)]; - .z.m.loginfo:(deps`log)`info; - .z.m.logwarn:(deps`log)`warn; - .z.m.logerr:(deps`log)`error; - - failures:checkdeps[],checkgraph[],checkcontracts[]; - warnings:ztscheck[],kdbxcheck[deps]; - - / warnings are logged unconditionally, before the failures check below - not gated behind "no failures", so a - / real warning is never silently dropped just because a failure also happened to signal in the same call. - / caught by direct testing, not by reading the code - see depcheck.md - if[count warnings; - .z.m.logwarn[`depcheck;buildreport["WARNING";warnings]]]; - - if[count failures; - report:buildreport["DEPENDENCY CHECK FAILED";failures]; - .z.m.logerr[`depcheck;report]; - '"di.depcheck: ",report]; - - .z.m.loginfo[`depcheck;"dependency check complete: ",(string count failures)," failure(s), ",(string count warnings)," warning(s)"]; - }; diff --git a/di/depcheck/deps.q b/di/depcheck/deps.q deleted file mode 100644 index 66222c30..00000000 --- a/di/depcheck/deps.q +++ /dev/null @@ -1,4 +0,0 @@ -/ hard module dependencies and their minimum versions, validated by di.depcheck -/ di.depcheck has no hard dependencies - its only runtime dependency (log) is injected via init as a dictionary of -/ functions, following the same convention di.depcheck itself checks other modules against -deps:(`$())!(); diff --git a/di/depcheck/init.q b/di/depcheck/init.q deleted file mode 100644 index b89b5cc8..00000000 --- a/di/depcheck/init.q +++ /dev/null @@ -1,6 +0,0 @@ -/ di.depcheck - dependency presence/version, core-contract, and .z.ts ownership auditing for kdb-x modules -/ intended to run once, post-load, after a host process (di.torq) has `use`d every module it needs - see depcheck.md - -\l ::depcheck.q - -export:([init;version;checkcontract]) diff --git a/di/depcheck/test.csv b/di/depcheck/test.csv deleted file mode 100644 index bb325fc5..00000000 --- a/di/depcheck/test.csv +++ /dev/null @@ -1,183 +0,0 @@ -action,ms,bytes,lang,code,repeat,minver,comment -comment,,,,,,,setup - load module and a capturing logger -before,0,0,q,depcheck:use`di.depcheck,1,,load di.depcheck module -before,0,0,q,.dc.captbl:([]lvl:`symbol$();ctx:`symbol$();msg:()),1,,log capture table for assertions -before,0,0,q,caplog:`info`warn`error!({[c;m] `.dc.captbl insert (`info;c;m)};{[c;m] `.dc.captbl insert (`warn;c;m)};{[c;m] `.dc.captbl insert (`error;c;m)}),1,,capturing binary logger {[c;m]} - -comment,,,,,,,init - dependency validation -fail,0,0,q,depcheck.init[(::)],1,,init rejects a non-dict deps -fail,0,0,q,depcheck.init[()!()],1,,init rejects missing log key -fail,0,0,q,depcheck.init[enlist[`log]!enlist 42],1,,init rejects a non-dict log value -fail,0,0,q,depcheck.init[enlist[`log]!enlist `info`warn!(caplog`info;caplog`warn)],1,,init rejects a log dict missing the error key -run,0,0,q,.dc.errstr:@[{depcheck.init[()!()]};(::);{x}],1,,capture the error string from a bad init -true,0,0,q,.dc.errstr like "di.depcheck:*",1,,init error is prefixed di.depcheck: - -comment,,,,,,,module metadata - exported version -true,0,0,q,10h=type depcheck.version,1,,version is a string -true,0,0,q,0 zzgb (installed on QPATH) -> zzgc (absent). -comment,,,,,,,zzgb also declares zzga (a cycle), which the visited-set guard must terminate. zzgb itself is a direct dep -comment,,,,,,,of the loaded root so it is checkdeps' domain and must NOT be double-reported by checkgraph -run,0,0,q,.m.di.0zzga.marker:1,1,,fabricate a loaded module di.zzga so it becomes a walk root -run,0,0,q,"system ""mkdir -p "",.dc.qr,""/di/zzga "",.dc.qr,""/di/zzgb""; (hsym `$.dc.qr,""/di/zzga/init.q"") 0: enlist ""export:([])""; (hsym `$.dc.qr,""/di/zzga/deps.q"") 0: enlist ""deps:enlist[`di.zzgb]!enlist \""1.0.0\""""",1,,install zzga on QPATH declaring zzgb -run,0,0,q,"(hsym `$.dc.qr,""/di/zzgb/init.q"") 0: enlist ""export:([])""; (hsym `$.dc.qr,""/di/zzgb/deps.q"") 0: enlist ""deps:`di.zzgc`di.zzga!(\""1.0.0\"";\""1.0.0\"")""",1,,install zzgb on QPATH declaring zzgc (missing) and zzga (cycle) -true,0,0,q,"any (.m.di.0depcheck.checkgraph[]) like ""di.zzgc is required transitively by di.zzgb*""",1,,checkgraph surfaces a depth>=2 dependency that does not resolve on QPATH -true,0,0,q,"not any (.m.di.0depcheck.checkgraph[]) like ""di.zzgb is required transitively*""",1,,zzgb (a direct dep of the loaded root) is not double-reported - the cycle also terminated without error -run,0,0,q,"system ""rm -rf "",.dc.qr,""/di/zzga "",.dc.qr,""/di/zzgb""",1,,remove the graph-walk on-disk fixtures (the fabricated .m.di.0zzga is harmless with no deps on disk) - -comment,,,,,,,resolvemodule - QPATH presence resolution without loading, distinct from checkonedep's loaded-check -true,0,0,q,not (::)~.m.di.0depcheck.resolvemodule `di.timer,1,,a real installed module resolves on QPATH -true,0,0,q,(::)~.m.di.0depcheck.resolvemodule `di.doesnotexistanywhere,1,,an absent module resolves to (::) - -comment,,,,,,,a deps.toml that exists but declares no [dependencies] section - the else branch of readdepstoml -run,0,0,q,"system ""mkdir -p "",.dc.qr,""/di/zznodeps""; (hsym `$.dc.qr,""/di/zznodeps/deps.toml"") 0: enlist ""name = \""foo\""""",1,,create a deps.toml with a top-level key but no [dependencies] section -true,0,0,q,0=count (.m.di.0depcheck.readdeps `di.zznodeps) 1,1,,a deps.toml with no [dependencies] section yields an empty merged manifest not a crash -true,0,0,q,"$[.dc.havetoml;0=count (.m.di.0depcheck.readdeps `di.zznodeps) 0;1b]",1,,and no read-failures either - a valid file with nothing declared is not an error (needs di.toml to read the file; no-op pass when absent) -run,0,0,q,"system ""rm -rf "",.dc.qr,""/di/zznodeps""",1,,remove the no-dependencies scratch fixture - -comment,,,,,,,malformed [dependencies] - a deps.toml where `dependencies` is a scalar not a section. must degrade to a -comment,,,,,,,clean failure line with an empty merged dict, NOT throw out of readdeps's merge and abort the whole walk -run,0,0,q,"system ""mkdir -p "",.dc.qr,""/di/zzscalar""; (hsym `$.dc.qr,""/di/zzscalar/deps.toml"") 0: enlist ""dependencies = \""oops\""""",1,,create a deps.toml whose dependencies key is a scalar string -true,0,0,q,"$[.dc.havetoml;any ((.m.di.0depcheck.readdeps `di.zzscalar) 0) like ""di.depcheck: deps.toml for di.zzscalar has a malformed*"";1b]",1,,a non-dict [dependencies] is reported as one clear failure line (no-op pass when di.toml is absent) -true,0,0,q,0=count (.m.di.0depcheck.readdeps `di.zzscalar) 1,1,,and the merged manifest is empty so the merge never throws -run,0,0,q,"system ""rm -rf "",.dc.qr,""/di/zzscalar""",1,,remove the malformed-dependencies scratch fixture - -comment,,,,,,,non-string version value - an unquoted deps.toml version parses to a float; a deps.q version may be an int -comment,,,,,,,or symbol. either is an authoring error reported as its own clear line, never a corrupted concatenated -comment,,,,,,,message or a throw out of parsesemver's `vs`. a single-char string version must NOT trip this guard -run,0,0,q,"system ""mkdir -p "",.dc.qr,""/di/zzfloatver""; (hsym `$.dc.qr,""/di/zzfloatver/deps.toml"") 0: (""[dependencies]"";""\""di.timer\"" = 1.0"")",1,,deps.toml with an UNQUOTED version - di.toml parses it as a float -true,0,0,q,"$[.dc.havetoml;any (.m.di.0depcheck.checkmoduledeps `0zzfloatver) like ""di.timer has a non-string minimum version*"";1b]",1,,an unquoted (float) deps.toml version is reported as a non-string version, not a garbage line or a crash (needs di.toml to parse the float; no-op pass when absent) -run,0,0,q,"system ""rm -rf "",.dc.qr,""/di/zzfloatver""",1,,remove the float-version fixture -run,0,0,q,"system ""mkdir -p "",.dc.qr,""/di/zzintver""; (hsym `$.dc.qr,""/di/zzintver/deps.q"") 0: enlist ""deps:enlist[`di.timer]!enlist 5""",1,,deps.q with an int version value -true,0,0,q,"any (.m.di.0depcheck.checkmoduledeps `0zzintver) like ""di.timer has a non-string minimum version*""",1,,an int deps.q version is reported as a non-string version too -run,0,0,q,"system ""rm -rf "",.dc.qr,""/di/zzintver""",1,,remove the int-version fixture -run,0,0,q,"system ""mkdir -p "",.dc.qr,""/di/zzonechar""; (hsym `$.dc.qr,""/di/zzonechar/deps.q"") 0: enlist ""deps:enlist[`di.timer]!enlist \""1\""""",1,,deps.q with a legitimate single-char string version -true,0,0,q,"not any (.m.di.0depcheck.checkmoduledeps `0zzonechar) like ""di.timer has a non-string minimum*""",1,,a single-char string version is valid and must NOT be flagged as non-string - the guard does not over-trigger -run,0,0,q,"system ""rm -rf "",.dc.qr,""/di/zzonechar""",1,,remove the single-char-version fixture diff --git a/di/depcheck/test_integration.csv b/di/depcheck/test_integration.csv deleted file mode 100644 index 4c9a63f0..00000000 --- a/di/depcheck/test_integration.csv +++ /dev/null @@ -1,34 +0,0 @@ -action,ms,bytes,lang,code,repeat,minver,comment -comment,,,,,,,"integration test - spins up a real, separate child kdb-x process, loads real modules via QPATH," -comment,,,,,,,"and calls depcheck.init[] for real, proving two things test.csv structurally cannot: that a real" -comment,,,,,,,"signalled failure actually terminates a real host process (non-zero exit, real error on stderr)," -comment,,,,,,,and that a genuine zero-failure success path completes cleanly end to end - unreachable in test.csv -comment,,,,,,,"since di.timer, always loaded there as a live negative control, permanently fails its own contract" -comment,,,,,,,"the child must be kdb-x-capable (needs use/QPATH), unlike a plain q peer, so this deliberately reuses" -comment,,,,,,,the currently-running process's own binary via /proc//exe rather than guessing from QHOME - -comment,,,,,,,QHOME on this dev machine resolves to a pre-kdb-x q build with no use keyword at all -comment,,,,,,,note k4unit runs every before row first then the asserts - so results are captured in befores and checked in trues -,,,,,,, -before,0,0,q,"qbin:first system ""readlink -f /proc/"",(string .z.i),""/exe""",1,,"resolve the currently-running process's own binary - guaranteed kdb-x-capable, no QHOME guess, no hardcoded path" -before,0,0,q,.it.haveqbin:not ()~key hsym `$qbin,1,,confirm the resolved binary actually exists on disk -before,0,0,q,if[not .it.haveqbin;exit 0],1,,skip this entire suite cleanly if the binary could not be resolved -before,0,0,q,".it.dir:(first "":"" vs getenv`QPATH),""/di/zzintegration""",1,,scratch directory derived from QPATH at runtime - no hardcoded path -before,0,0,q,"system ""mkdir -p "",.it.dir",1,,create the scratch directory -before,0,0,q,"(hsym `$.it.dir,""/success.q"") 0: enlist ""depcheck:use`di.depcheck; logdep:`info`warn`error!({[c;m] -1 m};{[c;m] -1 m};{[c;m] -2 m}); depcheck.init[enlist[`log]!enlist logdep]; exit 0;""",1,,"write a real child script loading ONLY di.depcheck - no di.timer, so no real contract gap exists - a genuine zero-failure success case" -before,0,0,q,"(hsym `$.it.dir,""/failure.q"") 0: enlist ""timer:use`di.timer; depcheck:use`di.depcheck; logdep:`info`warn`error!({[c;m] -1 m};{[c;m] -1 m};{[c;m] -2 m}); depcheck.init[enlist[`log]!enlist logdep]; exit 0;""",1,,"write a real child script that also loads di.timer - its real, permanent Timer-contract gap (missing cp) drives a real failure" -before,0,0,q,".it.successresult:system qbin,"" "",.it.dir,""/success.q -q < /dev/null > "",.it.dir,""/success.out 2>&1; echo EXITCODE:$?""",1,,"run the success child to completion synchronously, capturing combined stdout+stderr and its real exit code" -before,0,0,q,".it.successexit:""I""$9_last .it.successresult",1,,parse the real exit code -before,0,0,q,".it.successout:sv[""\n"";read0 hsym `$.it.dir,""/success.out""]",1,,read the child's real captured output -before,0,0,q,".it.failureresult:system qbin,"" "",.it.dir,""/failure.q -q < /dev/null > "",.it.dir,""/failure.out 2>&1; echo EXITCODE:$?""",1,,"run the failure child to completion synchronously, capturing combined stdout+stderr and its real exit code" -before,0,0,q,".it.failureexit:""I""$9_last .it.failureresult",1,,parse the real exit code -before,0,0,q,".it.failureout:sv[""\n"";read0 hsym `$.it.dir,""/failure.out""]",1,,read the child's real captured output -before,0,0,q,"system ""rm -rf "",.it.dir",1,,remove the scratch fixture directory -,,,,,,, -comment,,,,,,,genuine zero-failure success path - unreachable in test.csv since di.timer always fails its own contract there -true,0,0,q,0=.it.successexit,1,,the child process loading only di.depcheck exited cleanly with a real 0 exit code -true,0,0,q,".it.successout like ""*dependency check complete: 0 failure(s), 0 warning(s)*""",1,,the real captured log shows the genuine zero-failure summary line - proves the true success path works end to end in a real process -,,,,,,, -comment,,,,,,,a real signalled failure actually terminates a real host process -true,0,0,q,0<>.it.failureexit,1,,the child process crashed with a real non-zero exit code when a real dependency check failed -true,0,0,q,".it.failureout like ""*DEPENDENCY CHECK FAILED*""",1,,the real crash output contains the real error-level report - not a captured mock -true,0,0,q,".it.failureout like ""*di.timer is missing required contract key(s): cp*""",1,,the real report names the real di.timer contract gap From c736395dee8b436ab2e4226737126561177bfef5 Mon Sep 17 00:00:00 2001 From: alowrydi Date: Thu, 6 Aug 2026 15:19:38 +0100 Subject: [PATCH 10/11] align init to single deps parameter convention --- di/permissions/permissions.md | 48 +++++---- di/permissions/permissions.q | 23 +++-- di/permissions/test.csv | 153 +++++++++++++++------------- di/permissions/test_integration.csv | 4 +- 4 files changed, 132 insertions(+), 96 deletions(-) diff --git a/di/permissions/permissions.md b/di/permissions/permissions.md index ba1fc832..db33108d 100644 --- a/di/permissions/permissions.md +++ b/di/permissions/permissions.md @@ -72,9 +72,23 @@ logdep:`info`warn`error!( handlers.init[enlist[`log]!enlist logdep]; handlersdep:`register`remove`list!(handlers.register;handlers.remove;handlers.list); -perms.init[`enabled`readonly!(1b;0b);`log`handlers!(logdep;handlersdep)]; +perms.init[(`log`handlers!(logdep;handlersdep)),`enabled`readonly!(1b;0b)]; ``` +`init` takes a **single dictionary** carrying both the dependencies and the configuration, the same +call shape every `di.*` module takes. Dependency keys (`` `log ``, `` `handlers ``, `` `ldapbind ``) +and configuration keys share one flat namespace; no configuration key collides with a dependency key, +and the dependency keys are stripped before the configuration is stored, so they never appear in +`status[]` or trigger the unrecognised-key warning. + +> **Watch the join.** `` enlist[`k]!enlist somedict `` puts a *table* on the value side (a one-element +> list of dictionaries is a table), so joining two of them throws `` 'mismatch ``. Join `di.log`'s +> `logdict` with **one** multi-key dictionary rather than chaining single-key ones: +> ```q +> perms.init[logging.logdict,`handlers`enabled`readonly!(handlersdep;1b;0b)] / works +> perms.init[logging.logdict,(enlist[`handlers]!enlist handlersdep),...] / 'mismatch +> ``` + `init` must be called before any other function. It is **idempotent**: a second call re-wires the dependencies and config and reclaims the same handler registrations, leaving grant data intact. @@ -128,7 +142,7 @@ module, so this ships empty. **A process that receives `.u.upd`-shaped feed traf explicitly**, or that traffic will be permission-checked and rejected: ```q -perms.init[`enabled`ignorelist!(1b;(`upd;"upd";`.u.upd;".u.upd"));deps] +perms.init[deps,`enabled`ignorelist!(1b;(`upd;"upd";`.u.upd;".u.upd"))] ``` It is a **mixed** list - the head of an incoming message is matched against both symbol and string @@ -138,13 +152,13 @@ forms. It applies to `.z.ps` only, matching TorQ; `.z.pg` is never exempted. ## Exported functions -### `init[config;deps]` +### `init[deps]` Wire dependencies, resolve config, and (when enabled) publish root names, load grants and register handlers. Idempotent - see [Initialisation](#initialisation) for the full worked example. ```q -perms.init[`enabled`readonly!(1b;0b);`log`handlers!(logdep;handlersdep)] +perms.init[(`log`handlers!(logdep;handlersdep)),`enabled`readonly!(1b;0b)] / or with defaults only (module loads but stays disabled): -perms.init[(::);`log`handlers!(logdep;handlersdep)] +perms.init[`log`handlers!(logdep;handlersdep)] ``` ### `teardown[]` @@ -453,7 +467,7 @@ handlers:use`di.handlers handlers.init[enlist[`log]!enlist logdep]; handlersdep:`register`remove`list!(handlers.register;handlers.remove;handlers.list); -perms.init[enlist[`enabled]!enlist 1b;`log`handlers!(logdep;handlersdep)]; +perms.init[(`log`handlers!(logdep;handlersdep)),enlist[`enabled]!enlist 1b]; / set up a reader who may select from trade and nothing else trade:([]sym:`a`a`b;px:1 2 3.0); @@ -515,7 +529,7 @@ resolved. This exists so the caching and lockout logic can be exercised without ```q fakebind:{[sess;d] enlist[`ReturnCode]!enlist 0i} / 0i = success, anything else = failure -perms.init[`enabled`ldapenabled!(1b;1b);`log`handlers`ldapbind!(logdep;handlersdep;fakebind)] +perms.init[(`log`handlers`ldapbind!(logdep;handlersdep;fakebind)),`enabled`ldapenabled!(1b;1b)] ``` This is a **`deps` injection, not a config value** - deps are process wiring code the module already @@ -577,17 +591,17 @@ dirty module state. > handling over a **real child process and a real IPC handle** - the things that cannot be tested > in-process, because `reval` does not enforce at `.z.w=0`. > -> It wires **real `di.handlers` only when that module is on `QPATH`**, falling back to a minimal -> inline stand-in (a bare `set[ev;f]`) otherwise. `di.handlers` lives on the `feature-handlers` -> branch, so a checkout of `feature-permissions` alone runs against the stand-in and does **not** -> exercise real phase/`exec`-ownership dispatch - that is covered by `di.handlers`' own suite, not -> this one. Check out both branches to exercise the real wiring. +> It wires the **real `di.handlers`**, which ships alongside this module on the same branch, so it +> does exercise real phase and `exec`-ownership dispatch - not a stand-in. A minimal inline stand-in +> (a bare `set[ev;f]`) exists only as a fallback for a `QPATH` that lacks `di.handlers`. > -> Which path ran is **reported, not assumed**: the suite asserts the child returned a boolean for -> `realhandlers`, and the value is visible in the results. This exists because the "prefer real -> `di.handlers`" branch was silently dead for weeks - `h.init` on a function-local throws -> `'h.init` (module dot-sugar resolves only against a *global* name), and the protected apply -> swallowed it, so the stand-in always ran while the suite reported green. +> Which path ran is **asserted, not assumed**: the suite requires the child to report +> `realhandlers` as `1b`, so a silent fall-through to the stand-in **fails** the suite rather than +> passing quietly. That assertion exists because the "prefer real `di.handlers`" branch was silently +> dead for weeks - `h.init` on a function-local throws `'h.init` (module dot-sugar resolves only +> against a *global* name), and the protected apply swallowed it, so the stand-in always ran while +> the suite reported green. If you run this on a `QPATH` without `di.handlers`, expect that one row +> to fail; that is the point of it. --- diff --git a/di/permissions/permissions.q b/di/permissions/permissions.q index 6071d87d..b50ab2f9 100644 --- a/di/permissions/permissions.q +++ b/di/permissions/permissions.q @@ -110,6 +110,10 @@ requireinit:{[ctx] / init +/ the keys of the single init dict that are dependencies rather than config - everything else in +/ that dict is a config setting. no config key shares a name with one of these +depkeys:`log`handlers`ldapbind; + validatedeps:{[deps] / log and handlers are both required and never defaulted - there is no fallback logger if[99h<>type deps; @@ -133,13 +137,12 @@ validatedeps:{[deps] '"di.permissions: ldapbind must be a function taking (session;dict) and returning a dict with a `ReturnCode key"]]; }; -resolveconfig:{[config] - / merge the caller's config over the known-key defaults, warning about anything unrecognised rather - / than dropping it silently +resolveconfig:{[deps] + / take the config half of the single init dict and merge it over the known-key defaults, warning + / about anything unrecognised rather than dropping it silently. depkeys are dependencies, not + / config, so they are dropped before the unknown-key check and never reach .z.m.config defaults:configdefaults,ldapconfigdefaults; - if[config~(::); :defaults]; - if[99h<>type config; - '"di.permissions: config must be a dict of settings, or (::) for defaults"]; + config:(key[deps] except depkeys)#deps; if[count unknown:(key config) except key defaults; .z.m.logwarn[`init;"ignoring unrecognised config key(s): ",", " sv string unknown]]; :defaults,(key[defaults] inter key config)#config; @@ -200,9 +203,11 @@ resettables:{[] .z.m.ldapready:0b; }; -init:{[config;deps] +init:{[deps] / wire deps, resolve config, and (when enabled) claim the message-handling .z.* events. - / config: a dict of settings, or (::) for defaults. deps: a dict with `log and `handlers. + / deps: ONE dict carrying `log and `handlers (required), `ldapbind (optional), and any config + / settings alongside them - the same call shape every di.* module takes, so di.torq can wire it. + / e.g. perms.init[(`log`handlers!(logdep;handlersdep)),`enabled`readonly!(1b;1b)] / idempotent - a second call reclaims the same registrations and leaves grant data intact validatedeps[deps]; .z.m.loginfo:(deps`log)`info; @@ -210,7 +215,7 @@ init:{[config;deps] .z.m.logerr:(deps`log)`error; .z.m.register:(deps`handlers)`register; .z.m.removehandler:(deps`handlers)`remove; - cfg:resolveconfig[config]; + cfg:resolveconfig[deps]; validateconfig[cfg]; validateengine[cfg`engine]; if[not initialised[];resettables[]]; diff --git a/di/permissions/test.csv b/di/permissions/test.csv index 11967ddd..87729263 100644 --- a/di/permissions/test.csv +++ b/di/permissions/test.csv @@ -8,29 +8,46 @@ before,0,0,q,"mockh:`register`remove`list!({[e;p;n;pr;f] `.pt.reg insert (e;p;n; before,0,0,q,deps:`log`handlers!(caplog;mockh),1,1,the injected dependency dict comment,,,,,,,init - dependency validation (log and handlers are required and never defaulted) -fail,0,0,q,perms.init[(::);(::)],1,1,init rejects a non-dict deps -fail,0,0,q,perms.init[(::);()!()],1,1,init rejects deps missing both keys -fail,0,0,q,perms.init[(::);enlist[`log]!enlist caplog],1,1,init rejects deps missing the handlers key -fail,0,0,q,perms.init[(::);enlist[`handlers]!enlist mockh],1,1,init rejects deps missing the log key -fail,0,0,q,perms.init[(::);`log`handlers!(42;mockh)],1,1,init rejects a non-dict log value -fail,0,0,q,perms.init[(::);`log`handlers!((enlist[`info]!enlist caplog`info);mockh)],1,1,init rejects a log dict missing warn and error -fail,0,0,q,perms.init[(::);`log`handlers!(caplog;enlist[`register]!enlist mockh`register)],1,1,init rejects a handlers dict missing remove and list -run,0,0,q,.pt.errstr:@[{perms.init[(::);()!()]};(::);{x}],1,1,capture the error string from a bad init +fail,0,0,q,perms.init[(::)],1,1,init rejects a non-dict deps +fail,0,0,q,perms.init[()!()],1,1,init rejects deps missing both keys +fail,0,0,q,perms.init[enlist[`log]!enlist caplog],1,1,init rejects deps missing the handlers key +fail,0,0,q,perms.init[enlist[`handlers]!enlist mockh],1,1,init rejects deps missing the log key +fail,0,0,q,perms.init[`log`handlers!(42;mockh)],1,1,init rejects a non-dict log value +fail,0,0,q,perms.init[`log`handlers!((enlist[`info]!enlist caplog`info);mockh)],1,1,init rejects a log dict missing warn and error +fail,0,0,q,perms.init[`log`handlers!(caplog;enlist[`register]!enlist mockh`register)],1,1,init rejects a handlers dict missing remove and list +run,0,0,q,.pt.errstr:@[{perms.init[()!()]};(::);{x}],1,1,capture the error string from a bad init true,0,0,q,".pt.errstr like ""di.permissions:*""",1,1,init errors are prefixed di.permissions: true,0,0,q,".pt.errstr like ""*di.log*""",1,1,the error names which module supplies the missing dependency comment,,,,,,,init - config validation -fail,0,0,q,perms.init[enlist[`engine]!enlist `tiered;deps],1,1,the tiered engine is not implemented and is rejected -fail,0,0,q,perms.init[enlist[`engine]!enlist `nosuch;deps],1,1,an unknown engine is rejected -fail,0,0,q,perms.init[enlist[`engine]!enlist 42;deps],1,1,a non-symbol engine is rejected -fail,0,0,q,perms.init[42;deps],1,1,a non-dict config is rejected +fail,0,0,q,"perms.init[deps,enlist[`engine]!enlist `tiered]",1,1,the tiered engine is not implemented and is rejected +fail,0,0,q,"perms.init[deps,enlist[`engine]!enlist `nosuch]",1,1,an unknown engine is rejected +fail,0,0,q,"perms.init[deps,enlist[`engine]!enlist 42]",1,1,a non-symbol engine is rejected +fail,0,0,q,perms.init[42],1,1,a non-dict argument is rejected run,0,0,q,delete from `.pt.cap,1,1,clear the log capture -run,0,0,q,perms.init[enlist[`nosuchkey]!enlist 1b;deps],1,1,an unrecognised config key is accepted +run,0,0,q,"perms.init[deps,enlist[`nosuchkey]!enlist 1b]",1,1,an unrecognised config key is accepted true,0,0,q,1=count select from .pt.cap where lvl=`warn,1,1,an unrecognised config key warns rather than being dropped silently true,0,0,q,"any (exec msg from .pt.cap where lvl=`warn) like\: ""*nosuchkey*""",1,1,the warning names the offending key +run,0,0,q,delete from `.pt.cap,1,1,clear the log capture +run,0,0,q,"perms.init[deps,enlist[`enabled]!enlist 1b]",1,1,init with the dependency keys alongside a known config key +true,0,0,q,0=count select from .pt.cap where lvl=`warn,1,1,dependency keys are never reported as an unrecognised config key +run,0,0,q,".pt.mns0:first (key `.m.di) where (string key `.m.di) like ""*permissions*""",1,1,locate the module private namespace +run,0,0,q,".pt.cfg0:get `$"".m.di."",(string .pt.mns0),"".config""",1,1,read the resolved config back out of module state +true,0,0,q,"not any `log`handlers`ldapbind in key .pt.cfg0",1,1,dependency keys never reach the stored config +run,0,0,q,perms.teardown[],1,1,release before the next block + +comment,,,,,,,init - the single-dict call shape di.torq wires every module through +run,0,0,q,logging:use`di.log,1,1,load di.log for its ready-made logdict dependency +true,0,0,q,"(enlist`log)~key logging.logdict",1,1,logdict is a one-key dependency dict ready to pass straight to init +run,0,0,q,"perms.init[logging.logdict,`handlers`enabled!(mockh;1b)]",1,1,the single-dict shape di.torq wires each module through +true,0,0,q,0 Date: Tue, 11 Aug 2026 15:46:58 +0100 Subject: [PATCH 11/11] Trimming version comment --- di/permissions/init.q | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/di/permissions/init.q b/di/permissions/init.q index e2498140..8c3f6d7e 100644 --- a/di/permissions/init.q +++ b/di/permissions/init.q @@ -4,10 +4,7 @@ \l ::permissions.q -/ module version, read from the VERSION file rather than hardcoded in the implementation - the -/ convention Jamie Grant's TorqX modules use, so a release bump touches one plain-text file. -/ NB `version` STAYS in the export: di.depcheck reads it from the export dict (checkdepversion), -/ and reports "exports no version" - failing the dependency check - if a module drops it +/ Module version version:first read0`:::VERSION / NB: export:([...]) EVALUATES each name, so it can only list names that already exist - the export