docs: add tokenizer and state contract Codex skills

.agents/skills/insight-state-contract/SKILL.md

@@ -0,0 +1,79 @@
+---
+name: insight-state-contract
+description: Verify Insight_Extractor JSON state round trips, malformed-state errors, failure atomicity, legacy categories, path semantics, and lazy model restoration. Use for persistence changes in cgfixit/Insight_Extractor.
+---
+
+# Insight state contract
+
+Work in the user's active Insight_Extractor checkout. Confirm its Git root and
+origin; read `AGENTS.md` and delegated guidance before edits. Resolve paths below
+from that root and use its Python 3.12+ virtual environment. Diagnose for a check
+request; implement changes only when the task authorizes a fix.
+
+## Establish the current contract
+
+Read `save_state`, `load_state`, `_reset_keyword_runtime`, embedding refresh callers,
+and `extract` in `src/insight_extractor/extractor.py`; inspect `config.py`,
+`exceptions.py`, CLI error handling in `__main__.py`, and state tests in
+`tests/unit/test_extractor.py`. Use actual signatures and serialized fields.
+
+At creation the state includes keyword bank, frequencies, categories, stem mode,
+similarity threshold, model name, and version 1. `child_safety` is mapped to
+`ai_safety`. Missing files return False. Malformed JSON is wrapped in StateLoadError,
+but schema failures can leak other exceptions and partially mutate the object.
+Reconfirm these observations; do not encode defects as the desired contract.
+
+## Exercise isolated fixtures
+
+Use `tmp_path` or `TemporaryDirectory`, explicit UTF-8, and synthetic text. Never
+load or overwrite the user's real `insight_extractor_state.json`. Set
+`HF_HUB_OFFLINE=1`, `TRANSFORMERS_OFFLINE=1`, and `HF_HUB_DISABLE_TELEMETRY=1`.
+
+- Round-trip supported fields and enum values, Unicode keywords, and the legacy
+  category. Inspect optional/missing fields, an empty bank, and unsupported versions;
+  distinguish current compatibility behavior from a proposed schema change.
+- Probe malformed JSON, invalid UTF-8, a list/null/scalar root, wrong field types,
+  invalid enum values, and invalid numeric settings. Cover read errors with injected
+  filesystem failures instead of platform-specific permission tricks.
+- Compare absolute Path, relative Path, and deprecated relative str paths from a
+  scratch working directory with a distinct output_dir. The live implementation may
+  resolve str relative to output_dir and Path relative to cwd; do not silently unify
+  them in an unrelated fix.
+
+## Failure atomicity
+
+Initialize an extractor with known state and injected fake model/tokenizer objects.
+Before each invalid load, snapshot keyword values, Counter values, categories,
+settings, compiled pattern behavior, embedding values/dirty flag, and identities
+of model/tokenizer/runtime objects. Copy mutable values; retaining only references
+can hide in-place mutation. Supply some valid changed fields followed by an invalid
+field so the test detects partial application, not just early parse failure.
+
+After failure, assert the established error contract and unchanged state/runtime.
+Also verify the source file was not modified. Treat a leaked exception or changed
+object as a finding. File-write atomicity is a separate property; do not claim that
+testing load rollback proves save_state survives interrupted writes.
+
+## Lazy restoration
+
+Replace the actual model-construction boundary with a sentinel that raises if called.
+Load a valid state with a different model name: cached model and tokenizer should
+be invalidated, embeddings cleared/marked dirty as appropriate, and regex runtime
+restored without encoding or downloads. Check same-model and empty-bank cases too.
+Run regex/dynamic matching immediately after load. Only then inject fake `_model`
+and `_tokenizer` for a semantic call and verify embeddings correspond to restored
+keywords. Injecting before load is insufficient when a changed model name clears them.
+
+## When fixing is requested
+
+Validate/prepare incoming state before publishing it to the live extractor, including
+fallible enum conversion and regex compilation. Preserve legacy compatibility, lazy
+loading, filenames, and version unless the task explicitly changes the contract.
+Prefer a focused fix over a new persistence framework. Add regressions for the exact
+schema failure and rollback problem, plus the successful lazy-restoration path.
+
+Run `python -m pytest tests/unit/test_extractor.py -q --tb=short`, then repository
+preflight for a code fix. Report a matrix of valid load, malformed input, rollback,
+path behavior, and lazy restoration with concrete observed results and skipped
+checks. Do not claim the existing optional integration suite proves these contracts.
+Commit/push only within authorization already present in the task.

.agents/skills/insight-state-contract/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "Insight State Contract"
+  short_description: "Check state rollback and lazy model restoration"
+  default_prompt: "Use $insight-state-contract to verify state validation, failure atomicity and lazy restoration."

.agents/skills/insight-tokenizer-check/SKILL.md

@@ -0,0 +1,65 @@
+---
+name: insight-tokenizer-check
+description: Check Insight_Extractor tokenizer chunk termination, invalid token budgets or overlap, token coverage, and lazy loading. Use for tokenizer bugs or regressions in cgfixit/Insight_Extractor, not general NLP tuning.
+---
+
+# Insight tokenizer check
+
+Work in the user's active Insight_Extractor checkout. Confirm the Git root and
+origin; read its `AGENTS.md` and delegated guidance before edits. Resolve all paths
+below from that root and use its Python 3.12+ virtual environment. A check request
+authorizes diagnosis; implement a fix only when the user requests one.
+
+## Trace the actual boundary
+
+Read `src/insight_extractor/tokenizer.py`, `tests/unit/test_tokenizer.py`, and callers
+of `chunk_text`, `tokenize_sentences`, and `extract_semantic_keywords` in
+`src/insight_extractor/extractor.py`. Inspect live signatures before proposing tests.
+At creation, `chunk_text` advances by `max_tokens - overlap`, while sentence splitting
+passes overlap 50. Treat these as starting clues, not proof they remain unchanged.
+
+## Reproduce without hanging the test runner
+
+Set `HF_HUB_OFFLINE=1`, `TRANSFORMERS_OFFLINE=1`, and `HF_HUB_DISABLE_TELEMETRY=1`.
+Inject a deterministic object into `SentenceTokenizer._tokenizer`; its `encode`
+should return identifiable integer tokens, and `decode` should record token slices.
+Reuse the existing fake's interface; never load HuggingFace assets for these checks.
+
+Before probing suspect arguments, bound the experiment. Prefer a fake `decode`
+that raises a dedicated diagnostic exception after a small fixed call budget on a
+tiny input (for example, eight calls for six tokens). Report that exception as a
+nontermination signal, never as valid production rejection. For a defect that can
+loop before decode, use `subprocess.run([...], timeout=5)` with the same interpreter;
+the subprocess must terminate on timeout. Do not use an unbounded thread or sleep.
+
+Check distinct cases:
+
+- Ordinary chunking, overlap zero, final partial chunk, and exact budget boundary.
+- Zero/negative budgets, negative overlap, overlap equal to or above the budget,
+  and non-integer arguments where supported validation is under review.
+- Empty and short text as well as input exceeding the budget: the short-input
+  shortcut can conceal invalid arguments.
+- `tokenize_sentences` with budgets below, equal to, and above 50 on long sentences.
+
+Assert termination, each chunk's token budget, ordered token coverage with only the
+requested overlap, and no dropped tail when overlap is zero. Decoded text need not
+round-trip byte-for-byte with a real tokenizer; compare recorded token slices.
+Separate current observed behavior from the desired validation contract.
+
+## When fixing is requested
+
+Define whether invalid input should raise or be normalized, using existing contracts
+and the user's request. Put validation before tokenization and short-input returns;
+make the sentence caller's overlap compatible with valid small budgets. Preserve
+default behavior and avoid unrelated sentence-scoring changes. Add focused regression
+tests that fail safely on the old code, including the zero-overlap tail case.
+
+Patch constructors at their actual import site when checking laziness; the runtime
+module need not expose `AutoTokenizer` because its top-level import is TYPE_CHECKING-only.
+Verify construction remains lazy and injected-tokenizer calls never load a model.
+
+Run `python -m pytest tests/unit/test_tokenizer.py -q --tb=short`, then repository
+preflight for a code fix. Report arguments/input, bounded reproduction, expected
+versus actual results, checks run, and remaining failures. Fake-tokenizer success
+does not establish real-model compatibility. Do not commit, push, or download models
+without authorization already present in the task.

.agents/skills/insight-tokenizer-check/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "Insight Tokenizer Check"
+  short_description: "Check chunk termination, overlap and token coverage"
+  default_prompt: "Use $insight-tokenizer-check to check tokenizer termination and token coverage without downloading models."

.codex/AGENTS.md

@@ -80,5 +80,7 @@ Read `.codex/README.md` and `.codex/codex_custom_instructions.md`, then use the
 - `insight-verify-no-model` — validate pipeline changes without downloading BERT;
 - `insight-add-entity-pattern` — add a static regex entity end to end;
 - `insight-optimize` — measured, minimal optimization of a real hot path.
+- `insight-tokenizer-check` — bounded chunking, overlap, token coverage, and lazy-loading checks;
+- `insight-state-contract` — malformed state, rollback, path semantics, and lazy restoration.
 
 For fuller implementation checklists, consult the matching file under `.claude/skills/`.

.codex/README.md

@@ -40,6 +40,13 @@ not proof that a plugin or hooks are installed.
 
 The skills reuse the existing `.claude/skills/` checklists where those are more detailed. The Codex versions are the entrypoints and add Windows-friendly commands plus explicit stop conditions.
 
+## Focused verification skills
+
+- `insight-tokenizer-check`: bounded checks for chunking hangs, invalid budgets, overlap, token coverage, and lazy imports.
+- `insight-state-contract`: state round trips, malformed schemas, failure atomicity, legacy categories, paths, and lazy model restoration.
+
+These two skills are self-contained under `.agents/skills/`; they diagnose by default and implement fixes when requested. Invoke them with `$insight-tokenizer-check` or `$insight-state-contract`, or select them through `/skills` where supported. Personal copies can also be installed under `~/.codex/skills`; keep them synchronized with these repository versions. Same-name personal and repository skills are not merged.
+
 ## Safe defaults
 
 - Keep model loading lazy.