A small, local, calibrated decision layer for security operations. It returns typed verdicts, and it abstains when it isn't sure.
Haruspex reads a situation and a bounded set of options and returns a typed answer:
- boolean: yes/no plus
P(yes) - choice: one option from a set
- score: a level on an ordered rubric
Every verdict carries the full probability distribution over its options and, once calibrated, a prediction set: the options the true answer is among, at a chosen error rate.
It runs locally on CPU, CUDA or Apple MPS, and serves a simple HTTP decision API.
Haruspex reuses the published decision encoder
fastino/GLiNER2.5-Decide (Apache-2.0,
non-autoregressive). It does not train a model from scratch. It adds four things:
- A typed decision contract (boolean / choice / score) over the encoder.
- Domain specialization. Each security domain gets a small LoRA adapter, which is reattached to the shared base at load time.
- Deterministic routing. Questions that are lookups rather than judgments, such as CVE applicability and cloud scanner findings, go to rule-based resolvers. A resolver answers only when the evidence pins the answer down. Otherwise it defers, so a gap in its rules causes an abstention, not a wrong verdict.
- One evidence format. Defender, Sentinel, Elastic ECS, Windows Security/Sysmon, AWS CloudTrail, Suricata, Zeek and Okta records are normalised into one schema before the model reads them, so a new log source needs a mapper, not a new model.
- Selective calibration. A verdict is accepted only when its confidence clears a threshold set for its decision type. That threshold is certified on a held-out calibration split with Learn-then-Test (an exact binomial test, Bonferroni-corrected over a fixed grid). With probability at least 1−δ, the error among accepted answers stays within your target risk. When the calibration data can't certify any threshold, Haruspex abstains on that decision type.
- Novelty gate. An input unlike anything the adapter was trained on abstains and is marked for escalation, instead of getting a confident guess.
Each security domain trains a separate LoRA adapter on domain-specific data:
| Domain | Use | Training data |
|---|---|---|
web |
General web security decisions | 400+ labeled rows |
dast |
DAST finding adjudication (real vs noise) | 104 rows: SQLi, Supabase RLS, rate limiting, cache deception, CORS reflection, null-byte traversal, GraphQL introspection, debug data leak, SVG injection + noise patterns |
code |
Source-level taint analysis — not shipped: at chance on independent code sets (OWASP, PrimeVul), see docs/benchmark.md | NIST Juliet (experimental) |
mobile |
MASTG mobile security | MASTG control assessments |
cloud |
Cloud/IaC posture findings | Terraform, CloudFormation, Kubernetes |
network |
AD/network lateral movement | BloodHound, Kerberos, relay |
llm |
LLM/AI prompt injection triage | Jailbreak, tool SSRF, prompt leakage |
soc |
SOC alert triage | Defender, Sentinel, Sysmon, Suricata |
Haruspex serves one /v1/evaluate endpoint that praetor and vigil both call:
- Vigil routes template-relevance ranking to
domain: "guide"and finding adjudication todomain: "dast". - Praetor routes
assess_findingenrichment to the domain matching the vuln type (web/cloud/llm/mobile/network) and reads the probability as a second opinion alongside its deterministic verdict. - MCP server (
haruspex-mcp) exposesdecideandlist_domainstools for direct agent access.
Self-hosted and keyless — the agent needs only the base URL (http://127.0.0.1:3000).
pip install -e . # inference
pip install -e ".[train]" # + LoRA training
pip install -e ".[serve]" # + HTTP serverfrom haruspex import Haruspex
m = Haruspex.load() # base encoder; pass an adapter dir or HF repo id to specialize
print(m.decide({"type": "boolean",
"state": "Target: WordPress 6.2, PHP 8.1, nginx",
"question": "Is a WordPress xmlrpc.php template relevant?",
"criteria": {"true": "WordPress target", "false": "not WordPress"}}))Device auto-selects (CUDA → MPS → CPU); override with device= or --device.
# 1. split: train / dev / cal / test
python scripts/split_data.py data/security.jsonl --test-frac 0.2 --cal-frac 0.15 --dev-frac 0.15
# 2. train a LoRA adapter; choose epochs / rank / seed by eval on *_dev only
python -m haruspex.train --train data/security_train.jsonl --out runs/sec --epochs 8 --seed 0
# 3. fit the calibrator on the cal split (temperature on dev; threshold + prediction sets on cal)
python -m haruspex.calibrate --model runs/sec --data data/security_cal.jsonl \
--target-risk 0.2 --delta 0.1 --temp-data data/security_dev.jsonl
# 3b. fit the novelty gate (abstain on unfamiliar inputs)
python -m haruspex.ood --model runs/sec --train data/security_train.jsonl \
--cal data/security_cal.jsonl --alpha 0.01
# 4. report the final number once, on test
python -m haruspex.eval --model runs/sec --data data/security_test.jsonl
# 5. serve
python -m haruspex.serve --model runs/sec --port 3000Training ends with a sanity gate: a fixed discrimination probe that refuses to save a
collapsed checkpoint (one where every output is about 0.5). Retry with another --seed
if it fires. The default learning rate is deliberately low; higher rates collapse on some
seeds. On smaller GPUs, add --batch 2 --grad-accum 4 --grad-checkpoint --max-len 192.
Confident mistakes are the main failure of a single adapter. Train the same recipe with a
few --seed values and list them in one manifest; they share one base model, and their
disagreement lowers confidence where a single adapter would be sure and wrong:
{"base": "fastino/GLiNER2.5-Decide", "adapters": ["m0", "m1", "m2"]}(m0…m2 are the final/ directories of the seed runs.) Calibrate the ensemble directory
like any other adapter. Each verdict then also reports agreement across members.
Decision rows are JSONL, one decision per line; the format is in
data/schema.md. Validate a file before training:
python scripts/validate_data.py data/security.jsonl --buildIncluded packs: security.jsonl (offensive and defensive: scanner adjudication, triage,
severity, routing, CVE applicability, phishing, detection rules), plus mobile, cloud,
network and llm.
Converters for public datasets (their output is not committed):
| script | source | decision |
|---|---|---|
convert_guide.py |
Microsoft GUIDE (Defender XDR incidents) | incident triage grade |
convert_otrf.py |
OTRF Security-Datasets (Windows Security / Sysmon) | ATT&CK tactic of attacker events |
convert_nvd.py |
NVD (after fetch_nvd.py) |
CVSS v3 severity from a description |
convert_owasp_benchmark.py |
OWASP Benchmark (GPL-2.0) | external test only, never trained on |
Splits are content-hashed: a row's split depends only on the row itself, so adding data never moves a training row into test. Each split has one job:
| split | used for |
|---|---|
train |
fitting the adapter |
dev |
choosing hyperparameters and temperature |
cal |
fitting the calibrator, and nothing else |
test |
the final reported number |
A state may be raw JSON from a supported source; it is normalised and rendered before
the model reads it (haruspex/evidence.py):
- Microsoft Defender XDR alerts (Graph security
alerts_v2) - Microsoft Sentinel
SecurityAlertrows - Elastic Common Schema documents
- Windows Security and Sysmon events, flat or
evtx_dump-style nested - AWS CloudTrail records (a bare record or a
{"Records": [...]}file) - Suricata EVE JSON (alert, dns, http, tls, flow)
- Zeek JSON logs (conn, dns, http, ssl, notice;
id.*dotted or nested) - Okta System Log events
- a list of any of these, read as one event window
The published adapters were trained on Microsoft and Windows evidence only. CloudTrail, Suricata, Zeek and Okta are rendered into the same schema, but expect the novelty gate to abstain on them more often until an adapter is trained on that telemetry.
Other JSON is passed through as-is; the novelty gate then decides whether the adapter should answer at all. Long states are read in overlapping windows, and a verdict stands only when every window agrees.
POST /v1/evaluate (also /v1/systemone; type "noul" is accepted as boolean)
{ "state": <str|obj>, "domain": "web"|"mobile"|...,
"questions": { "<name>": { "type": "boolean"|"choice"|"score",
"instructions": "...",
"criteria": {"true":..,"false":..} | {opt: desc} | [levels] } } }
Each answer carries its verdict, the full probabilities, accept / abstain and a
prediction_set (when a calibrator is loaded), and a source. All questions in one
request are scored as a single batch.
| source | meaning |
|---|---|
model |
the encoder answered |
cve_lookup |
the CVE resolver answered |
cloud_rule |
the cloud resolver answered |
model_declined |
abstained; route the question to an external lookup |
Other endpoints are GET /health, GET /v1/models and GET /metrics. /health
reports per-domain readiness (adapter, calibrated, novelty) and warns when a domain
would answer with an uncalibrated bare base model — start the server with
--model/--adapters, never bare, and gate clients on ok/warnings.
Limits: the body is capped at 512 KiB (checked before parsing), state at 64 K
characters, and a request at 64 questions. Malformed input returns 400; a request over
any limit returns 413.
Optional settings:
HARUSPEX_API_KEYrequires a bearer key, compared in constant time.HARUSPEX_AUDIT_LOGwrites one JSONL line per request; the rawstateis never logged.--ssl-keyfile/--ssl-certfileserve HTTPS.
haruspex-mcp (pip install -e ".[mcp]") is a stdio MCP server with decide and
list_domains tools, sharing the HTTP server's routing and limits. Every result carries
escalate and an action: when Haruspex abstains, the agent is told to hand the question
to a stronger model or a human.
{"mcpServers": {"haruspex": {"command": "/path/to/.venv/bin/haruspex-mcp",
"args": ["--adapters", "web=/abs/runs/sec,soc=/abs/runs/soc"]}}}Each domain gets its own adapter on the shared base, served behind one endpoint. The
request's domain field picks the adapter. When domain is omitted, the domain is
inferred from the request text; otherwise the default adapter answers.
haruspex-serve --adapters web=runs/sec,mobile=runs/mobile,cloud=runs/cloud --port 3000
haruspex-serve --adapters web=TyrusRC/haruspex,mobile=TyrusRC/haruspex#mobile # from the Hubrepo#subfolder loads one adapter from a multi-domain repository. Published adapters
in TyrusRC/haruspex:
| load as | domain |
|---|---|
TyrusRC/haruspex |
web / general security (offensive and defensive) |
TyrusRC/haruspex#soc |
SOC incident triage, Windows / Sysmon events |
TyrusRC/haruspex#mobile |
Android / iOS |
TyrusRC/haruspex#cloud |
cloud and IaC findings |
TyrusRC/haruspex#network |
internal network / Active Directory |
TyrusRC/haruspex#llm |
LLM application security |
TyrusRC/haruspex#general |
every domain at once; the fallback when none fits |
Route each domain to its own adapter: the general adapter's calibrator is pooled over all domains, so its risk target holds on average, not per domain.
scripts/bench.py runs Haruspex and any OpenAI-compatible model on the same rows. The
other model sees the same option descriptions, decodes greedily with output constrained
to the valid labels, and can take few-shot examples drawn from the train split. The
script reports accuracy with 95 % intervals, per-category accuracy, and an exact paired
McNemar test. It also runs the cascade — Haruspex answers what it accepts, the LLM gets
only the abstentions — and reports how many LLM calls that leaves.
python scripts/bench.py --data data/security_test.jsonl --model runs/sec \
--llm-url http://127.0.0.1:8088 --shots 3 --shots-from data/security_train.jsonlHeadline result (in-distribution security triage, 188 rows): the Haruspex→8B cascade
reaches Llama-3.1-8B's accuracy (0.904, statistically identical) while calling the LLM on
only 16 % of items — the same result at ~6× less LLM compute, 84 % decided locally at
34 ms. Out-of-distribution the larger LLM is the more robust detector: on independent
prompt-injection sets Llama-3.1-8B holds 0.82–0.93 across two sources, while a 0.5B fine-tuned
on one of them (deepset) wins that benchmark (0.966 vs 0.819) but collapses to 0.59 on the
other (vs the 8B's 0.925) — single-benchmark overfitting, not a real capability, so that adapter
is not shipped. Full results, both wins and losses, are in docs/benchmark.md.
The intended shape is a cascade: Haruspex as the fast, calibrated first pass, and a general
LLM or a human for whatever it declines. Use Haruspex after a tool has produced candidates, to triage and rank them. For example,
for each finding, ask "is this real and reportable?" and "how severe?". Treat a no or
an abstention as "not proven yet", never as "drop it". Don't use it to gate which targets
get tested.
- It decides bounded questions; open-ended judgment belongs to a general model.
- Recalibrate on your own data before relying on the risk guarantee in a new domain.
- Full fine-tuning is disabled until gliner2 can reload its full-mode checkpoints.
- Prediction-set coverage is marginal over the calibration mix, not guaranteed per input.
- Ensembles run their members one after another, so latency grows with their size.
- SOC triage and Windows event logs are covered; other DFIR artefacts (memory, disk, timelines) are not yet.
- The novelty gate compares surface text, not meaning: a familiar format with new content may pass it.
- A decision that needs facts from two distant parts of a long log at once is beyond the windowed reader; escalate those.
Run python -m unittest discover tests before opening a pull request.
The decision-model framing follows TypeSafe's Jev ("System One") and its open reproductions. The base encoder is GLiNER2.5-Decide by fastino, built on GLiNER2 and DeBERTa.