Feature tplog

[ascottDI]

di.tplog

Summary

Adds di.tplog, a di.* module for tickerplant log files: create/open a log, append to it, roll to
the next day, replay it on startup, and repair a corrupt one. It is the modular replacement for the
inline log handling in TorQ's code/processes/tickerplant.q (.u.ld / .u.endofday) together with
the recovery utilities in code/common/tplogutils.q, folded into a single import surface.

Self-contained — no hard use dependencies; the one runtime dependency, a logger, is injected via
init, so the module loads and is tested standalone.

Motivation

TorQ keeps tickerplant log handling in two places: the create/append/roll/replay code inline in
tickerplant.q, and the corruption check/repair in tplogutils.q. Both the tickerplant (which
writes and rolls the log) and a replaying subscriber (which reads and, if needed, repairs it) need
these operations, so the extraction brings them under one module rather than leaving consumers to
import two partial pieces. The logic is re-expressed on the framework's conventions — an injected
logger, module-local state, strict input validation — while preserving the observable behaviour.

Design

No handler involvement

di.tplog registers no .z.* handlers. It is called directly by the tickerplant
(open/write/roll) and by subscribers on startup (replay/replayupto), so it sits outside the
di.handlers observer/decider model and takes no handler dependency.

Corruption detection on KDB-X

The corruption handling was written against measured KDB-X -11! behaviour, which differs from
classic kdb+ and from the assumptions in the TorqX POC:

• -11!(-2;logfile) counts a clean log without running upd, and throws on any corruption — it does
  not return the classic (goodcount;bytes) pair. Corruption is detected by trapping that throw
  (corruptp), and open counts with -11!(-2) before replaying, so a corrupt log fails fast before
  any partial replay mutates state.
• -11!(-1;logfile) counts too but executes upd, so it is unsafe for detection and is not used.

replay and replayupto check with the non-executing corruptp, repair to a .good file if
required (a byte-scan that never calls upd), then replay once — so a good message before a
corruption point is never processed twice. A naive trap-and-retry would partially replay before
throwing and then replay again.

Dependencies (injected via init)

Dependency	Key	Required	Description
logger	`log	yes	dict of binary {[ctx;msg]} functions with at least info, warn, error

init validates the dependency strictly and signals immediately if it is missing or malformed; there
is no fallback. di.log's logdictlog(aninfo..fataldict) is a conforming superset of the required keys and is passed through without adaptation. There are no config keys and no hardusedependencies, sodeps.q` is empty.

Public API

Function	Signature	Description
init	[deps]	Wire the injected logger. Must be called first.
logname	[dir;date]	Log-file handle for a dir string and date (<dir>/tp<date>).
open	[dir;date]	Open, creating if absent; replay an existing log once through the root upd, or fail fast if corrupt. Returns (handle;count).
write	[handle;msg]	Append one message to an open handle.
roll	[handle;dir;olddate]	Close the handle and open the olddate+1 log. Returns (handle;count).
replay	[logfile]	Replay through the root upd, repairing first if corrupt. Returns the count.
replayupto	[logfile;n]	Replay the first n messages only (repair-aware).
check	[logfile]	Return logfile if usable, else a repaired `<logfile>.good.
repair	[logfile]	Recover readable messages into `<logfile>.good.
version	—	Module version ("0.1.0").

Public functions validate their argument types and route failures through a log-then-signal helper.
getapimeta[] exposes the callable API for central registration with di.api; the framework plumbing
(init, getapimeta, version) is intentionally excluded.

Changes from the TorQ implementation

Change	Rationale
Lifecycle and repair in one module	TorQ splits the two across tickerplant.q (inline) and tplogutils.q; consumers need both, so they are folded into one import surface.
Corruption detection rewritten for KDB-X -11!	Classic kdb+ returns a partial-count pair on a corrupt log; this build throws instead, so detection traps the throw and open counts with the non-executing -11!(-2) before replaying.
check drops lastmsgtoreplay	Its skip-repair optimisation relied on the partial good-count -11!(-2) no longer returns; the argument would be dead weight, so check takes just the logfile and repairs on any corruption.
Injected logger	Replaces TorQ's .lg.* calls; the log dependency is required, with no stdout fallback.
Repair output handle closed	TorQ left the .good write handle open; it is closed after the scan.

Testing

test.csv / test.q (k4unit), 25 checks, hermetic (temp log files under /tmp, cleaned up
afterwards). A capturing logger is injected so the log contract is asserted end to end.

Coverage: the metadata/version contract; strict init dependency validation (a fail row per guard);
public-input validation; logname; the open/write/roll lifecycle; fail-fast open on a corrupt log;
replay recovering a corrupt log while processing each recovered message exactly once (the
double-processing regression); replayupto replaying only the first n; and check/repair on clean
and corrupt logs, asserting the warning is logged.

k4unit:use`di.k4unit
k4unit.moduletest`di.tplog

Limitations

• repair is tuned to the (upd;trade;…) message shape inherited from tplogutils; logs of other
  tables are recovered only if their messages share that prefix. Corruption detection is
  schema-agnostic — only the repair byte-scan is trade-specific.
• The filename convention is fixed (<dir>/tp<date>, one log per date); sharing a directory between
  multiple logical logs would need a prefix argument on logname.

Files

di/tplog/init.q       entry point, version read, export list
di/tplog/tplog.q      implementation
di/tplog/tplog.md     module documentation
di/tplog/VERSION      module version (single source of truth)
di/tplog/deps.q       dependency manifest (empty — no hard use deps)
di/tplog/test.csv     k4unit tests (25)
di/tplog/test.q       test fixtures and helpers

The module version lives in a plain-text VERSION file, read in init.q (version:trim first read0\:::VERSION) and exported for di.depcheck, matching the convention used by di.serversanddi.clienttracking`.