Skip to content
TyrusRCPublic

About

A small, open-weight decision model

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

86 Commits

Folders and files

Repository files navigation

Haruspex

A small, local, calibrated decision layer for security operations. It returns typed verdicts, and it abstains when it isn't sure.

License

Haruspex reads a situation and a bounded set of options and returns a typed answer:

  • boolean: yes/no plus P(yes)
  • choice: one option from a set
  • score: a level on an ordered rubric

Every verdict carries the full probability distribution over its options and, once calibrated, a prediction set: the options the true answer is among, at a chosen error rate.

It runs locally on CPU, CUDA or Apple MPS, and serves a simple HTTP decision API.

How it works

Haruspex reuses the published decision encoder fastino/GLiNER2.5-Decide (Apache-2.0, non-autoregressive). It does not train a model from scratch. It adds four things:

  1. A typed decision contract (boolean / choice / score) over the encoder.
  2. Domain specialization. Each security domain gets a small LoRA adapter, which is reattached to the shared base at load time.
  3. Deterministic routing. Questions that are lookups rather than judgments, such as CVE applicability and cloud scanner findings, go to rule-based resolvers. A resolver answers only when the evidence pins the answer down. Otherwise it defers, so a gap in its rules causes an abstention, not a wrong verdict.
  4. One evidence format. Defender, Sentinel, Elastic ECS, Windows Security/Sysmon, AWS CloudTrail, Suricata, Zeek and Okta records are normalised into one schema before the model reads them, so a new log source needs a mapper, not a new model.
  5. Selective calibration. A verdict is accepted only when its confidence clears a threshold set for its decision type. That threshold is certified on a held-out calibration split with Learn-then-Test (an exact binomial test, Bonferroni-corrected over a fixed grid). With probability at least 1−δ, the error among accepted answers stays within your target risk. When the calibration data can't certify any threshold, Haruspex abstains on that decision type.
  6. Novelty gate. An input unlike anything the adapter was trained on abstains and is marked for escalation, instead of getting a confident guess.

Domain adapters

Each security domain trains a separate LoRA adapter on domain-specific data:

Domain Use Training data
web General web security decisions 400+ labeled rows
dast DAST finding adjudication (real vs noise) 104 rows: SQLi, Supabase RLS, rate limiting, cache deception, CORS reflection, null-byte traversal, GraphQL introspection, debug data leak, SVG injection + noise patterns
code Source-level taint analysis — not shipped: at chance on independent code sets (OWASP, PrimeVul), see docs/benchmark.md NIST Juliet (experimental)
mobile MASTG mobile security MASTG control assessments
cloud Cloud/IaC posture findings Terraform, CloudFormation, Kubernetes
network AD/network lateral movement BloodHound, Kerberos, relay
llm LLM/AI prompt injection triage Jailbreak, tool SSRF, prompt leakage
soc SOC alert triage Defender, Sentinel, Sysmon, Suricata

Integration

Haruspex serves one /v1/evaluate endpoint that praetor and vigil both call:

  • Vigil routes template-relevance ranking to domain: "guide" and finding adjudication to domain: "dast".
  • Praetor routes assess_finding enrichment to the domain matching the vuln type (web/cloud/llm/mobile/network) and reads the probability as a second opinion alongside its deterministic verdict.
  • MCP server (haruspex-mcp) exposes decide and list_domains tools for direct agent access.

Self-hosted and keyless — the agent needs only the base URL (http://127.0.0.1:3000).

Install

pip install -e .              # inference
pip install -e ".[train]"     # + LoRA training
pip install -e ".[serve]"     # + HTTP server

Quickstart

from haruspex import Haruspex

m = Haruspex.load()   # base encoder; pass an adapter dir or HF repo id to specialize
print(m.decide({"type": "boolean",
                "state": "Target: WordPress 6.2, PHP 8.1, nginx",
                "question": "Is a WordPress xmlrpc.php template relevant?",
                "criteria": {"true": "WordPress target", "false": "not WordPress"}}))

Device auto-selects (CUDA → MPS → CPU); override with device= or --device.

Train, calibrate, evaluate, serve

# 1. split: train / dev / cal / test
python scripts/split_data.py data/security.jsonl --test-frac 0.2 --cal-frac 0.15 --dev-frac 0.15

# 2. train a LoRA adapter; choose epochs / rank / seed by eval on *_dev only
python -m haruspex.train --train data/security_train.jsonl --out runs/sec --epochs 8 --seed 0

# 3. fit the calibrator on the cal split (temperature on dev; threshold + prediction sets on cal)
python -m haruspex.calibrate --model runs/sec --data data/security_cal.jsonl \
    --target-risk 0.2 --delta 0.1 --temp-data data/security_dev.jsonl

# 3b. fit the novelty gate (abstain on unfamiliar inputs)
python -m haruspex.ood --model runs/sec --train data/security_train.jsonl \
    --cal data/security_cal.jsonl --alpha 0.01

# 4. report the final number once, on test
python -m haruspex.eval --model runs/sec --data data/security_test.jsonl

# 5. serve
python -m haruspex.serve --model runs/sec --port 3000

Training ends with a sanity gate: a fixed discrimination probe that refuses to save a collapsed checkpoint (one where every output is about 0.5). Retry with another --seed if it fires. The default learning rate is deliberately low; higher rates collapse on some seeds. On smaller GPUs, add --batch 2 --grad-accum 4 --grad-checkpoint --max-len 192.

Seed ensembles

Confident mistakes are the main failure of a single adapter. Train the same recipe with a few --seed values and list them in one manifest; they share one base model, and their disagreement lowers confidence where a single adapter would be sure and wrong:

{"base": "fastino/GLiNER2.5-Decide", "adapters": ["m0", "m1", "m2"]}

(m0…m2 are the final/ directories of the seed runs.) Calibrate the ensemble directory like any other adapter. Each verdict then also reports agreement across members.

Data

Decision rows are JSONL, one decision per line; the format is in data/schema.md. Validate a file before training:

python scripts/validate_data.py data/security.jsonl --build

Included packs: security.jsonl (offensive and defensive: scanner adjudication, triage, severity, routing, CVE applicability, phishing, detection rules), plus mobile, cloud, network and llm.

Converters for public datasets (their output is not committed):

script source decision
convert_guide.py Microsoft GUIDE (Defender XDR incidents) incident triage grade
convert_otrf.py OTRF Security-Datasets (Windows Security / Sysmon) ATT&CK tactic of attacker events
convert_nvd.py NVD (after fetch_nvd.py) CVSS v3 severity from a description
convert_owasp_benchmark.py OWASP Benchmark (GPL-2.0) external test only, never trained on

Splits are content-hashed: a row's split depends only on the row itself, so adding data never moves a training row into test. Each split has one job:

split used for
train fitting the adapter
dev choosing hyperparameters and temperature
cal fitting the calibrator, and nothing else
test the final reported number

Log sources

A state may be raw JSON from a supported source; it is normalised and rendered before the model reads it (haruspex/evidence.py):

  • Microsoft Defender XDR alerts (Graph security alerts_v2)
  • Microsoft Sentinel SecurityAlert rows
  • Elastic Common Schema documents
  • Windows Security and Sysmon events, flat or evtx_dump-style nested
  • AWS CloudTrail records (a bare record or a {"Records": [...]} file)
  • Suricata EVE JSON (alert, dns, http, tls, flow)
  • Zeek JSON logs (conn, dns, http, ssl, notice; id.* dotted or nested)
  • Okta System Log events
  • a list of any of these, read as one event window

The published adapters were trained on Microsoft and Windows evidence only. CloudTrail, Suricata, Zeek and Okta are rendered into the same schema, but expect the novelty gate to abstain on them more often until an adapter is trained on that telemetry.

Other JSON is passed through as-is; the novelty gate then decides whether the adapter should answer at all. Long states are read in overlapping windows, and a verdict stands only when every window agrees.

HTTP API

POST /v1/evaluate        (also /v1/systemone; type "noul" is accepted as boolean)
{ "state": <str|obj>, "domain": "web"|"mobile"|...,
  "questions": { "<name>": { "type": "boolean"|"choice"|"score",
                             "instructions": "...",
                             "criteria": {"true":..,"false":..} | {opt: desc} | [levels] } } }

Each answer carries its verdict, the full probabilities, accept / abstain and a prediction_set (when a calibrator is loaded), and a source. All questions in one request are scored as a single batch.

source meaning
model the encoder answered
cve_lookup the CVE resolver answered
cloud_rule the cloud resolver answered
model_declined abstained; route the question to an external lookup

Other endpoints are GET /health, GET /v1/models and GET /metrics. /health reports per-domain readiness (adapter, calibrated, novelty) and warns when a domain would answer with an uncalibrated bare base model — start the server with --model/--adapters, never bare, and gate clients on ok/warnings.

Limits: the body is capped at 512 KiB (checked before parsing), state at 64 K characters, and a request at 64 questions. Malformed input returns 400; a request over any limit returns 413.

Optional settings:

  • HARUSPEX_API_KEY requires a bearer key, compared in constant time.
  • HARUSPEX_AUDIT_LOG writes one JSONL line per request; the raw state is never logged.
  • --ssl-keyfile / --ssl-certfile serve HTTPS.

Agents (MCP)

haruspex-mcp (pip install -e ".[mcp]") is a stdio MCP server with decide and list_domains tools, sharing the HTTP server's routing and limits. Every result carries escalate and an action: when Haruspex abstains, the agent is told to hand the question to a stronger model or a human.

{"mcpServers": {"haruspex": {"command": "/path/to/.venv/bin/haruspex-mcp",
  "args": ["--adapters", "web=/abs/runs/sec,soc=/abs/runs/soc"]}}}

Per-domain adapters

Each domain gets its own adapter on the shared base, served behind one endpoint. The request's domain field picks the adapter. When domain is omitted, the domain is inferred from the request text; otherwise the default adapter answers.

haruspex-serve --adapters web=runs/sec,mobile=runs/mobile,cloud=runs/cloud --port 3000
haruspex-serve --adapters web=TyrusRC/haruspex,mobile=TyrusRC/haruspex#mobile  # from the Hub

repo#subfolder loads one adapter from a multi-domain repository. Published adapters in TyrusRC/haruspex:

load as domain
TyrusRC/haruspex web / general security (offensive and defensive)
TyrusRC/haruspex#soc SOC incident triage, Windows / Sysmon events
TyrusRC/haruspex#mobile Android / iOS
TyrusRC/haruspex#cloud cloud and IaC findings
TyrusRC/haruspex#network internal network / Active Directory
TyrusRC/haruspex#llm LLM application security
TyrusRC/haruspex#general every domain at once; the fallback when none fits

Route each domain to its own adapter: the general adapter's calibrator is pooled over all domains, so its risk target holds on average, not per domain.

Benchmarking

scripts/bench.py runs Haruspex and any OpenAI-compatible model on the same rows. The other model sees the same option descriptions, decodes greedily with output constrained to the valid labels, and can take few-shot examples drawn from the train split. The script reports accuracy with 95 % intervals, per-category accuracy, and an exact paired McNemar test. It also runs the cascade — Haruspex answers what it accepts, the LLM gets only the abstentions — and reports how many LLM calls that leaves.

python scripts/bench.py --data data/security_test.jsonl --model runs/sec \
    --llm-url http://127.0.0.1:8088 --shots 3 --shots-from data/security_train.jsonl

Headline result (in-distribution security triage, 188 rows): the Haruspex→8B cascade reaches Llama-3.1-8B's accuracy (0.904, statistically identical) while calling the LLM on only 16 % of items — the same result at ~6× less LLM compute, 84 % decided locally at 34 ms. Out-of-distribution the larger LLM is the more robust detector: on independent prompt-injection sets Llama-3.1-8B holds 0.82–0.93 across two sources, while a 0.5B fine-tuned on one of them (deepset) wins that benchmark (0.966 vs 0.819) but collapses to 0.59 on the other (vs the 8B's 0.925) — single-benchmark overfitting, not a real capability, so that adapter is not shipped. Full results, both wins and losses, are in docs/benchmark.md.

Integration guidance

The intended shape is a cascade: Haruspex as the fast, calibrated first pass, and a general LLM or a human for whatever it declines. Use Haruspex after a tool has produced candidates, to triage and rank them. For example, for each finding, ask "is this real and reportable?" and "how severe?". Treat a no or an abstention as "not proven yet", never as "drop it". Don't use it to gate which targets get tested.

Limitations

  • It decides bounded questions; open-ended judgment belongs to a general model.
  • Recalibrate on your own data before relying on the risk guarantee in a new domain.
  • Full fine-tuning is disabled until gliner2 can reload its full-mode checkpoints.
  • Prediction-set coverage is marginal over the calibration mix, not guaranteed per input.
  • Ensembles run their members one after another, so latency grows with their size.
  • SOC triage and Windows event logs are covered; other DFIR artefacts (memory, disk, timelines) are not yet.
  • The novelty gate compares surface text, not meaning: a familiar format with new content may pass it.
  • A decision that needs facts from two distant parts of a long log at once is beyond the windowed reader; escalate those.

Contributing

Run python -m unittest discover tests before opening a pull request.

License

Apache-2.0.

Acknowledgements

The decision-model framing follows TypeSafe's Jev ("System One") and its open reproductions. The base encoder is GLiNER2.5-Decide by fastino, built on GLiNER2 and DeBERTa.

About

A small, open-weight decision model

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages