Skip to content
View hibou04-ops's full-sized avatar
🎯
Focusing
🎯
Focusing
  • Independent
  • Seoul, South Korea
  • 16:53 (UTC +09:00)
  • LinkedIn in/hibou04

Block or report hibou04-ops

Block user

Prevent this user from interacting with your repositories and sending you notifications. Learn more about blocking users.

You must be logged in to block users.

Content in all repositories owned by your account will be closed.
Maximum 250 characters. Please don’t include any personal information such as legal names or email addresses. This note will only be visible to you.
Report abuse

Contact GitHub support about this user’s behavior. Learn more about reporting abuse.

Report abuse
hibou04-ops/README.md

Deterministic Verification Gates for AI Agents & LLM Pipelines

"Stop auditing stochastic models with more stochastic models."
Offline, zero-API-cost checks for hallucinated file:line citations, train/test split leakage, LLM-judge noise illusions, and prompt/sweep overfitting before they ship.

Public Primitives Verification Ledger Runtime Contract Korean Docs


⚑ 10-Second Look: What Happens When an Agent Lies or Overfits?

A green self-report can hide a broken citation, a contaminated split, or a gain smaller than measurement noise. Put a deterministic check between the claim and your CI decision:

The commands below were exercised with the reproducible inputs in the expandable setup. Colored lines summarize actual fixture results; they are not verbatim stdout. - / + are diff highlighting. Install once; the checks shown then run offline without API keys.

$ python -m pip install antemortem==0.12.4 mini-antemortem-cli==0.11.5 mini-omega-lock==0.8.6 omegaprompt==2.2.8 omega-lock==0.3.13 antemortem-structural-lab==0.8.2

# 1. Catch a coding agent citing a line beyond EOF, then check its claimed AST symbol:
$ antemortem metrics agent_patch_claim.json --repo . --fail-over 0
- [FAIL] Fabricated Citation: src/core/validator.py:142 (file has only 98 lines; exit 4)
$ python -c "from pathlib import Path; from antemortem.citations import verify_ast_symbol_grounding as verify; ok=verify(Path('src/core/validator.py'),40,75,'enforce_holdout_split'); print('AST symbol grounded:',ok); raise SystemExit(0 if ok else 1)"
- [FAIL] AST Symbol Mismatch: enforce_holdout_split is absent from lines 40–75 (exit 1)

# 2. Catch silent train/test leakage before evaluation:
$ mini-antemortem-cli check --target-provider openai --judge-provider anthropic --train tune_set.jsonl --test test_set.jsonl --rubric rubric.json --variants variants.json --validation-mode disjoint --fail-on-severity high
- [FAIL] Split Leakage: 14 normalized test inputs duplicate train content despite disjoint IDs (exit 1)

# 3. Block a claimed gain below the measured empirical noise ceiling, using stored deltas:
$ python -c "import json; from mini_omega_lock import compute_empirical_noise_ceiling as ceiling; noise=ceiling(json.load(open('judge_deltas.json'))); gain=0.021; print(f'gain={gain:.3f}; empirical_noise_ceiling={noise:.3f}'); raise SystemExit(1 if gain <= noise else 0)"
- [BLOCKED] Claimed gain 0.021 <= empirical noise ceiling 0.038 (exit 1)

# 4. Require the supplied holdout scores to survive the configured gate:
$ omega-lock gate --train train_scores.json --holdout holdout_scores.json
+ [PASS] Supplied holdout fixture passed the gate (exit 0)

The contract behind the preview: antemortem-cli is the repository; antemortem is its distribution and executable. AST symbol grounding is a separate Python helper. The mini-omega-lock offline command uses its Python API because preflight requires an inference endpoint. Its empirical noise ceiling (compute_minimum_detectable_effect is a compatibility alias) is not a power-based MDE or a calibrated coverage guarantee. The explicit gain <= noise comparison is the example's rejection policy. The final PASS applies to the supplied score arrays; that command does not issue a receipt or establish independent OOS generalization.

Reproduce these failures locally β€” exact fixture setup

Save the following as make_showcase_fixture.py in an empty working directory and run python make_showcase_fixture.py, then cd showcase-fixture. Run the commands above one at a time, without copying $ or colored summary lines. Nonzero exits are expected for the deliberately failing cases. Provider names in the static linter are configuration metadata; this check makes no provider requests.

import json
from pathlib import Path

root = Path("showcase-fixture")
root.mkdir(exist_ok=False)
(root / "src/core").mkdir(parents=True)
(root / "src/core/validator.py").write_text("value = 1\n" * 98, encoding="utf-8")
data = {
    "agent_patch_claim.json": {
        "classifications": [{"id": "t1", "label": "GHOST", "citation": "src/core/validator.py:142"}],
        "decision": "SAFE_TO_PROCEED",
    },
    "rubric.json": {"dimensions": [{"name": "correctness", "description": "Correct answer", "weight": 1.0, "scale": [1, 5]}]},
    "variants.json": {"system_prompts": ["Return a concise factual answer.", "Explain all reasoning before responding."]},
    "judge_deltas.json": [0.038, -0.038, 0.038, -0.038],
    "train_scores.json": [0.6, 0.7, 0.8, 0.9],
    "holdout_scores.json": [0.59, 0.69, 0.79, 0.89],
}
for name, value in data.items():
    (root / name).write_text(json.dumps(value), encoding="utf-8")
for name, prefix in [("tune_set.jsonl", "train"), ("test_set.jsonl", "test")]:
    rows = [{"id": f"{prefix}-{i}", "input": f"Question {i}" if prefix == "train" else f"  Question   {i}  ", "reference": str(i)} for i in range(14)]
    (root / name).write_text("\n".join(json.dumps(row) for row in rows) + "\n", encoding="utf-8")

🧭 Start Here: Pick the Gate for Your Failure Mode

Don't guess which package to installβ€”match the failure in your AI workflow. These six pinned releases are on PyPI and use Apache-2.0; installation names and versions below match the artifact ledger.

If your AI workflow suffers from... Install & Run (PyPI Β· Apache-2.0) What it checks deterministically Interfaces
Coding agent hallucinating file:line or symbols antemortem-cli Β· pip install antemortem==0.12.4 Physical line-range citation verification against local sources; separate Python AST symbol helper CLI Β· Python API Β· MCP Β· GitHub Action
Silent leakage or rigged prompt-eval setups mini-antemortem-cli Β· pip install mini-antemortem-cli==0.11.5 Preflight static linter with 11 evaluation trap rules, including split overlap, bias, and homogeneous variants CLI Β· CI Gate
A/B prompt "wins" that may be judge noise mini-omega-lock Β· pip install mini-omega-lock==0.8.6 Empirical noise ceiling from measured deltas; endpoint-based measurement preflight CLI: preflight Β· Python API
Prompts that fit development sets but fail held-out checks omegaprompt Β· pip install omegaprompt==2.2.8 Prompt evaluation, calibration, configuration and held-out split checks; documented promptfoo / DSPy workflows CLI Β· MCP Β· CI Gate
Hyperparameter / sweep overfitting (Optuna, etc.) omega-lock Β· pip install omega-lock==0.3.13 Walk-forward / holdout checks, score-degradation guards, and separate SHA-256 integrity-receipt workflows CLI Β· Python API
Fragile code paths needing counterexample probes antemortem-structural-lab Β· pip install antemortem-structural-lab==0.8.2 Experimental structural AST probing, contract/Oracle-driven execution feedback, and deterministic counterexample replay CLI Β· Python API

Local verification can run without a model service after installation. Evaluation and inference paths may use configured providers. For MCP, install the repository's documented extras and use its supported entry point.


πŸ—οΈ Two-Tier Systems Architecture (Open Primitives + In-Memory R&D Core)

This workspace is an independent personal systems engineering & verification research lab structured into two tiers:

  • Tier 1 β€” Public Modular Primitives (6 Packages Β· Apache-2.0 Β· PyPI Live): Standalone CLI, Python and selected MCP tools for developer workflows and CI pipelines.
  • Tier 2 β€” Private Research Harnesses (2 Harnesses Β· All Rights Reserved β€” Personal R&D): Closed-source systems research on in-memory pipeline fusion (omega-citadel) and adversarial boundary mutation testing (omega-crucible).
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ TIER 1: PUBLIC MODULAR PRIMITIVES (Apache-2.0 Β· PyPI Β· CLI / API / MCP / CI)       β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ mini-antemortem-cli       β”‚ antemortem-cli            β”‚ antemortem-structural-lab  β”‚
β”‚ 11 Eval Trap Rules        β”‚ File:Line + AST helper    β”‚ Counterexample Probes      β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ mini-omega-lock           β”‚ omegaprompt               β”‚ omega-lock                 β”‚
β”‚ Empirical Noise Ceiling   β”‚ Prompt Evaluation Gates   β”‚ Walk-Forward / Holdout     β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
             β”‚                            β”‚                            β”‚
             β–Ό       [ Zero-IPC / Zero-Subprocess Research Fusion ]      β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ TIER 2: PRIVATE RESEARCH HARNESSES (Personal R&D Β· All Rights Reserved)            β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ β€’ omega-citadel (v1.1.2 pin) β€” In-Memory 5-Stage Governance Engine                 β”‚
β”‚   β”œβ”€ Schema + AST grounding + noise filter + holdout lock + SHA-256 receipts       β”‚
β”‚   └─ Historical medians: 14.2 / 19.3654 ms; 79.1x / 63.50x (v1.0.1 / v1.1.1)       β”‚
β”‚                                                                                    β”‚
β”‚ β€’ omega-crucible (v1.0.3 pin) β€” Adversarial Meta-Verification Harness              β”‚
β”‚   β”œβ”€ 1,067 synthetic leakage, drift, and citation-hallucination mutations          β”‚
β”‚   └─ 1,067 / 1,067 detected: 163.77 ms (v1.0.0); 173.2306 ms (v1.0.3)              β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
Private R&D Harness Artifact Pin Systems Research Scope (Personal R&D) Measured Deterministic Benchmark
omega-citadel v1.1.2 In-Memory 5-Stage Governance Engine: Combines prompt/schema contracts, AST grounding, noise filtering, holdout locking, and SHA-256 integrity receipts in one Python runtime. Historical 20-pair records: 14.2 ms / 79.1x at v1.0.1; 19.3654 ms / 63.50x at v1.1.1. The earlier display 14.2 ms – 19.3 ms truncates the latter median.
omega-crucible v1.0.3 Adversarial Meta-Verification Harness: Injects synthetic split leakage, schema drift, and hallucinated file:line citations into the harness's defenders. Historical v1.0.0: 1,067 / 1,067 (100.0%) synthetic mutation detections in 163.77 ms. Separate v1.0.3 clean-wheel record: 1,067 / 1,067 in 173.2306 ms.

Benchmark contract: Windows x64 / CPython 3.12.10; fixed-payload research harnesses. The latency ratio compares five cold subprocess stages with one warm in-memory process, including process-start and parsing overhead. It is not a measurement of the six public packages' actual CLI chain. Historical measurements retain their original versions; the finite authored mutation corpus is not evidence of universal robustness. Private source artifacts remain private.

Research state: Structural Lab and the private research retain M0_HOLD; utility and independent OOS remain NOT_ESTABLISHED. Engineering PASS counts, line/AST checks and integrity hashes do not establish natural-language truth or real-world generalization.


πŸ” Cryptographic Artifact Attestation & SHA-256 Ledger

omega-citadel and omega-crucible remain private All Rights Reserved personal R&D artifacts. The recorded wheels, source archives and compiled-bytecode locked-runtime.zip bundles below are pinned for authorized integrity checks. The public section pins both PyPI files for each of the six releases. The 12 public artifact hashes below were rechecked against downloaded PyPI bytes and live metadata on 2026-11-06. The five private artifact hashes retain their earlier verified ledger entries.

Copy the following block verbatim into SHA256SUMS.txt beside the corresponding files:

# Tier 2: Private personal R&D release artifacts
79e59e88f69d0f976ff48be42a0a0417c20f2aac95a41d9a075c2d3825a0b8fb  omega_citadel-1.1.2-py3-none-any.whl
fbf374e0ea9f8888bc00c64bf57fb13062ee57908df68150c7a2ccddc4ac45bc  omega_citadel-1.1.2.tar.gz
2604f342a2d375a4fd643cd510853524a534587a750d127edf8431f52849174f  omega-citadel-1.1.2-locked-runtime.zip
9b159fe454f591a0a5459a7c48a30570ebe4b2a2f7d7c1ca52090a0b9345bc82  omega_crucible-1.0.3-py3-none-any.whl
1f5282884bd2316e5cfa4a8733ed417d395c5ea71c60fc20ab70fec735b7cc34  omega-crucible-1.0.3-locked-runtime.zip

# Tier 1: Six public PyPI releases (wheel + source archive)
395ab96a6979ae5130397e507c07f48c8a338f5697df459f19df3101ab413323  antemortem-0.12.4-py3-none-any.whl
b37622de01a23a4601827493ad0d0a9e207993524363674f323b1e13aa1ef5ff  antemortem-0.12.4.tar.gz
f74b4ac55fd41af93816487ac2a694c71aaab2d473ae50d30ea0d2daab751747  mini_antemortem_cli-0.11.5-py3-none-any.whl
1e55f0de559b2f70baca105f5c8730f3f268ae2d454b112d3bcdd3b8e679afa7  mini_antemortem_cli-0.11.5.tar.gz
75b32424c7b93b5ee6ee026909bba166cf93a06b4ff9696fe31485c583395a28  mini_omega_lock-0.8.6-py3-none-any.whl
31bcdcc776d077766857ad0942f0ed6ba97b08204c71ba69f4e6a84f64af2506  mini_omega_lock-0.8.6.tar.gz
9fab54398337cff17c24a302ea26c0da439e93c06af24cfb36ae026dde93ba90  omegaprompt-2.2.8-py3-none-any.whl
46654b03634ce08acfcf9ca107f1f9efa8beda592c0b515d6ba8199265f1bedc  omegaprompt-2.2.8.tar.gz
a50129730e46f04c38add465b2856d24cab68e870f961af704ee96f64cf2fd3b  omega_lock-0.3.13-py3-none-any.whl
fdff29e4fcf7ea28688733028eda40bdaaf964ad5d94be014116428229dd549d  omega_lock-0.3.13.tar.gz
8f7fa2662f5adc97b8670031caa1fb332911571d910857145d6bb5613ade001e  antemortem_structural_lab-0.8.2-py3-none-any.whl
1053aef3e97fecfb22151995c4e1800602e1c6b730fef1431ade39861615c1ff  antemortem_structural_lab-0.8.2.tar.gz
  • Verify artifact integrity locally: Authorized auditors with all listed files run:

    sha256sum -c SHA256SUMS.txt

    For public artifacts only, copy the Tier 1 lines into a separate checksum file. Checksums establish byte identity against this ledger; they are not digital signatures or evidence of artifact safety.

  • Public download sources: antemortem 0.12.4 Β· mini-antemortem-cli 0.11.5 Β· mini-omega-lock 0.8.6 Β· omegaprompt 2.2.8 Β· omega-lock 0.3.13 Β· antemortem-structural-lab 0.8.2.

  • Ecosystem Verification Ledger: 4,306 PASS is the historical cross-run ledger (2,160 + 1,561 + 230 + 355) spanning public-package checks, Structural Lab, and private-harness runs. It is not a fresh synchronized test total for the eight artifact pins on this page. The fail-closed examples above preserve explicit rejection exits.

  • ν•œκ΅­μ–΄ λ¬Έμ„œ (Korean Documentation): README.ko.md

Pinned Loading

  1. antemortem-structural-lab antemortem-structural-lab Public

    Contract and oracle-driven counterexample discovery with metered search policies, shrinking and reproducible research ledgers.

    Python

  2. omega-lock omega-lock Public

    Fail-closed holdout validation for Optuna and ML evaluation with absolute and relative score-degradation checks.

    Python 2

  3. antemortem-cli antemortem-cli Public

    Deterministic code-citation verification and CI gates with evidence recomputation and total citation coverage checks.

    Python 1

  4. omegaprompt omegaprompt Public

    Provider-neutral LLM evaluation and prompt calibration with explicit noise measurement and fail-closed preflight contracts.

    Python 1

  5. mini-omega-lock mini-omega-lock Public

    Standalone measurement-reliability probes and empirical noise-ceiling estimation for repeated LLM evaluations.

    Python 1

  6. mini-antemortem-cli mini-antemortem-cli Public

    Standalone static preflight checks for LLM evaluation configurations, leakage detection and canonical risk rules.

    Python 1