From e851019216a5616a63d341a9b5a3952e606ee2e3 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 15 Aug 2026 03:34:36 +0000 Subject: [PATCH 1/2] fix: remove stale pre-4.53 transformers comment from pyproject.toml MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - pyproject.toml: the dependency comment block said transformers was "Capped at <4.45 until upstream fix" with a remediation snippet for transformers==4.44.2/sentence-transformers==3.0.1, directly contradicting the dependencies list two lines below (transformers>=4.53.0,<5.0.0). Root cause: the >=4.45 cap comment was never removed when the floor was raised to >=4.53.0 in commit 9e5f8ff — it shipped self-contradictory from that same commit. Replaced with a DEPENDENCY NOTES block matching requirements.txt's house style, keeping only the notes that are still true. - .claude/skills/deps-security-refresh/SKILL.md, .claude/skills/insight-optimize/SKILL.md: both described accelerate==0.34.2 as "the current" constraints.txt pin, but that was fixed to accelerate==1.14.0 in commit 312096b (2026-07-08). Reworded to past-tense/"watch for regression" framing so a future agent reading these skills doesn't mistake fixed history for a live bug. - docs/SECURITY_AUDIT.md: re-ran pip-audit against a clean constraints-pinned install as part of this verification; logged one net-new, not-yet-triaged advisory (PYSEC-2026-2290) alongside the already-flagged dedicated-retriage note. No verdict assigned, no pin changed — full triage stays a separate ask-first task per this repo's rules. requirements.txt, constraints.txt, and pyproject.toml's actual version bounds were already mutually consistent (constraints.txt's == pins all satisfy the >= / < ranges in requirements.txt and pyproject.toml). Verified: clean Python 3.12 + 3.13 venvs, `pip install -r requirements.txt -c constraints.txt && pip install -e ".[dev]"`, transformers/accelerate/sentence-transformers import triangle, ruff check, ruff format --check, mypy --strict, and pytest tests/unit/ (118/118 passed on both interpreters). Integration tests skipped — no model download in this environment. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01DQngYCdcQQHKr18ABhe4ut --- .claude/skills/deps-security-refresh/SKILL.md | 12 ++++++++---- .claude/skills/insight-optimize/SKILL.md | 9 +++++---- docs/SECURITY_AUDIT.md | 7 +++++++ pyproject.toml | 14 ++++++-------- 4 files changed, 26 insertions(+), 16 deletions(-) diff --git a/.claude/skills/deps-security-refresh/SKILL.md b/.claude/skills/deps-security-refresh/SKILL.md index 19ec240..e8e30b6 100644 --- a/.claude/skills/deps-security-refresh/SKILL.md +++ b/.claude/skills/deps-security-refresh/SKILL.md @@ -70,10 +70,14 @@ If the environment allows a model download, also instantiate actual crash site. If not possible, say so in the report (skips are reported, never hidden). -Known failure this check catches (verified 2026-07, Linux/py3.12): the current -constraints set `transformers==4.53.0` + `accelerate==0.34.2` fails at import with -`ImportError: cannot import name 'TorchTensorParallelPlugin'` — transformers 4.53's -trainer needs `accelerate>=1.3.0`. CI masks it by installing without constraints. +Known failure this check catches (fixed 2026-07-08 in commit 312096b — verify it stays +fixed on every bump): `constraints.txt` previously pinned `transformers==4.53.0` + +`accelerate==0.34.2`, which fails at import with `ImportError: cannot import name +'TorchTensorParallelPlugin'` — transformers 4.53's trainer needs `accelerate>=1.3.0`. +`constraints.txt` now pins `accelerate==1.14.0`. CI never caught this because it +installs without constraints (`pip install -e ".[dev]"`), so a constraints-pinned +install is the only thing that exercises this triangle — re-run this check after any +accelerate/transformers/sentence-transformers bump. ## Step 4 — Scan diff --git a/.claude/skills/insight-optimize/SKILL.md b/.claude/skills/insight-optimize/SKILL.md index 9a7470c..1797746 100644 --- a/.claude/skills/insight-optimize/SKILL.md +++ b/.claude/skills/insight-optimize/SKILL.md @@ -77,10 +77,11 @@ the README alone — this repo once shipped a phantom-API test suite, commit `40 4. **Dependency-risk** — the transformers/sentence-transformers/accelerate triangle: `transformers>=4.53.0,<5.0.0` (floor clears 18 CVEs per `docs/SECURITY_AUDIT.md`); `accelerate` is load-bearing despite zero imports (sentence-transformers needs it — - `ModelLoadError: name 'init_empty_weights' is not defined` without it); the known - broken pin `transformers==4.53.0` + `accelerate==0.34.2` → - `ImportError: cannot import name 'TorchTensorParallelPlugin'` (fixing the pin is - ask-first); `pip-audit` surface with reachability verdicts in + `ModelLoadError: name 'init_empty_weights' is not defined` without it); watch for + regression to the previously broken pin `transformers==4.53.0` + `accelerate==0.34.2` + (fixed 2026-07-08 in commit 312096b; `constraints.txt` now pins `accelerate==1.14.0`) + → `ImportError: cannot import name 'TorchTensorParallelPlugin'` (lowering the floor + again is ask-first); `pip-audit` surface with reachability verdicts in `docs/SECURITY_AUDIT.md`. 5. **Maintainability** — the lazy-loading invariant above; the duplicated sentence-split regex `r"(?<=[.!?])\s+"` in `tokenizer.tokenize_sentences` **and** diff --git a/docs/SECURITY_AUDIT.md b/docs/SECURITY_AUDIT.md index b498e73..7e29534 100644 --- a/docs/SECURITY_AUDIT.md +++ b/docs/SECURITY_AUDIT.md @@ -118,6 +118,13 @@ identify `CVE-2026-4372`'s affected component and reachability verdict, and conf whether the `PYSEC-*` IDs are duplicates of the already-documented `CVE-2025-149xx` series or net-new. +**2026-08-15 re-check:** re-ran `pip-audit` against a clean constraints-pinned install +(`transformers==4.53.0`) as part of a dependency-consistency verification pass. Same +`PYSEC-2025-211`..`218` and `PYSEC-2026-2288`/`2289` findings as above, plus one net-new +advisory not previously listed here: **`PYSEC-2026-2290`** (fixed in `5.5.0`), not yet +triaged. Rolled into the same dedicated `transformers` CVE re-triage noted above — no +verdict assigned, no pin changed. + --- ## SAST Findings (bandit) diff --git a/pyproject.toml b/pyproject.toml index 32685ec..cec8081 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -34,14 +34,12 @@ classifiers = [ "Typing :: Typed", ] -# IMPORTANT — transformers upper bound: -# transformers >= 4.45 breaks sentence-transformers BERT model loading -# with NameError ('init_empty_weights'). Capped at <4.45 until upstream fix. -# Fix for existing installs: pip install "transformers==4.44.2" "sentence-transformers==3.0.1" -# accelerate is required — provides init_empty_weights used by sentence-transformers. -# transformers pinned >=4.53.0 clears all CVEs with available upstream fixes. -# accelerate floor raised to >=1.3.0 — transformers==4.53.0's trainer module imports -# TorchTensorParallelPlugin from accelerate.utils, absent before accelerate 1.3.0. +# DEPENDENCY NOTES (kept in sync with requirements.txt / constraints.txt): +# - accelerate is required — provides init_empty_weights used by sentence-transformers. +# Missing accelerate causes: ModelLoadError: name 'init_empty_weights' is not defined +# - transformers pinned >=4.53.0 clears all CVEs with available upstream fixes. +# - accelerate floor raised to >=1.3.0 — transformers==4.53.0's trainer module imports +# TorchTensorParallelPlugin from accelerate.utils, absent before accelerate 1.3.0. dependencies = [ "sentence-transformers>=3.4.1,<4.0.0", "scikit-learn>=1.5.0", From cf8dec84e0dc6090bc56633f08c985f13c9f1573 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 15 Aug 2026 03:36:30 +0000 Subject: [PATCH 2/2] feat: add verify-deps skill for dependency-file drift checks MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Codifies the checks performed in this session as a repeatable skill: cross-check requirements.txt/constraints.txt/pyproject.toml version bounds, grep the repo for stale "current pin" claims left behind after a fix (the pyproject.toml <4.45 comment and the two skill docs' stale accelerate==0.34.2 references fixed in the prior commit are the motivating examples), verify the clean-venv install + compatibility triangle, run the four CI gates, and spot-check pip-audit against docs/SECURITY_AUDIT.md's documented findings. Deliberately scoped narrower than deps-security-refresh: this skill makes no version-pin changes and does no CVE reachability triage — it only catches and fixes documentation/comment drift once the pins themselves are already correct. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01DQngYCdcQQHKr18ABhe4ut --- .claude/skills/verify-deps/SKILL.md | 181 ++++++++++++++++++++++++++++ 1 file changed, 181 insertions(+) create mode 100644 .claude/skills/verify-deps/SKILL.md diff --git a/.claude/skills/verify-deps/SKILL.md b/.claude/skills/verify-deps/SKILL.md new file mode 100644 index 0000000..d1a0f78 --- /dev/null +++ b/.claude/skills/verify-deps/SKILL.md @@ -0,0 +1,181 @@ +--- +name: verify-deps +description: Verify that requirements.txt, constraints.txt, and pyproject.toml are mutually consistent, and that every place in the repo describing current pin versions (comments in those files, README's Known Issue section, docs/SECURITY_AUDIT.md, other skill docs) still matches reality — no unbumped pins, no stale "current" claims left over from a prior fix. Use when asked to "verify deps", "check dependency consistency/accuracy", before a release, or periodically as a drift check. Distinct from deps-security-refresh (bumping pins / triaging new CVEs) — this skill makes no version changes, only catches and fixes documentation drift. +--- + +# verify-deps — dependency-file consistency and drift check + +This is a **read-and-reconcile** skill, not a bump skill. Nothing in this repo's +dependency *pins* should change when you run it — only stale comments, docs, and skill +text that no longer match the pins should. Two real bugs motivate this: + +- `pyproject.toml` shipped a comment claiming transformers was "Capped at <4.45" in the + same commit (`9e5f8ff`) that its `dependencies` list already said + `transformers>=4.53.0,<5.0.0` — self-contradictory from day one, never caught because + nobody diffed the comment against the line six lines below it. +- Two skill docs (`deps-security-refresh`, `insight-optimize`) described + `accelerate==0.34.2` as "the current" `constraints.txt` pin over a month after it was + fixed to `1.14.0` in commit `312096b` — the fix shipped, the *narrative about the fix* + didn't get updated everywhere it was duplicated. + +Both bugs were **accurate when written, wrong by the time anyone read them again**. +That's what this skill hunts for. + +## Step 1 — Cross-check the three source-of-truth files + +Read all three in full: + +- `requirements.txt` — ranges (`>=`, `<`) + `DEPENDENCY NOTES` comment block +- `constraints.txt` — exact `==` pins + comment block +- `pyproject.toml` `[project] dependencies` — ranges, must be **identical** to + `requirements.txt`'s ranges (same packages, same bounds, same order is nice-to-have + but not required) + +For every package, check by hand (don't eyeball — actually compare the numbers): + +1. Does `pyproject.toml`'s range match `requirements.txt`'s range exactly? +2. Does `constraints.txt`'s `==` pin satisfy both ranges (`>=` floor and `<` ceiling)? +3. Is any package present in one file but missing from another? + +If you find a mismatch, the fix is almost always to `constraints.txt` or one comment — +**do not** widen/narrow a range to make a mismatch disappear; that's a behavior change +and is ask-first per CLAUDE.md §6. Report the mismatch and stop if the right fix isn't +obviously "the comment is wrong." + +## Step 2 — Hunt for stale version claims repo-wide + +The bug pattern is always the same shape: a comment or doc asserts "the current pin is +X" and X used to be true. Grep the whole repo — not just the three dependency files — +for both the *current* pinned versions and every version this repo has ever pinned and +moved away from: + +```bash +grep -rn "4\.44\.2\|3\.0\.1\|<4\.45\|0\.26\.0\|0\.34\.2" --include='*.md' --include='*.toml' --include='*.txt' . +``` + +(That list is this repo's known historical pins as of 2026-08 — `transformers` capped +at `<4.45`, `transformers==4.44.2`, `sentence-transformers==3.0.1`, +`accelerate==0.26.0`, `accelerate==0.34.2`. If `constraints.txt` has moved further since +this skill was written, extend the grep with whatever pin it now shows.) + +For every hit, read the surrounding paragraph and classify it: + +- **Historical / diff context** (a changelog entry, a "Hardened Manifest Changes + Applied" diff block, a "previously pinned" or "fixed in commit X" sentence) — fine, + leave it. +- **Asserted as current state** ("the current constraints set...", "pins X", with no + past-tense marker) — this is the bug. Reword to past tense, name the fixing commit if + `git log -S""` finds one, and state what the pin actually is now. + +Known places this has bitten before — always check these even if the grep above comes +back clean, since new drift accumulates in the same spots: + +- `pyproject.toml`'s dependency comment block +- `README.md`'s "Known Issue" sections (`ModelLoadError`, `TorchTensorParallelPlugin`) +- `docs/SECURITY_AUDIT.md`'s executive summary and addenda +- `.claude/skills/deps-security-refresh/SKILL.md` and + `.claude/skills/insight-optimize/SKILL.md` — both narrate specific pin numbers + +## Step 3 — Verify the install and the compatibility triangle + +Don't take the files' word for it — prove they actually resolve together, in a clean +Python 3.12 venv (and 3.13 if available, matching the CI matrix): + +```bash +python3.12 -m venv /tmp/depcheck && /tmp/depcheck/bin/python -m pip install --upgrade pip +/tmp/depcheck/bin/python -m pip install -r requirements.txt -c constraints.txt +/tmp/depcheck/bin/python -m pip install -e . +/tmp/depcheck/bin/python - << 'EOF' +import accelerate, sentence_transformers, transformers +print("transformers", transformers.__version__) +print("sentence-transformers", sentence_transformers.__version__) +print("accelerate", accelerate.__version__) +from accelerate import init_empty_weights +from accelerate.utils import TorchTensorParallelPlugin +from sentence_transformers import SentenceTransformer +print("triangle OK") +EOF +``` + +CI never runs this combination — `.github/workflows/ci.yml` installs with +`pip install -e ".[dev]"`, **no** `-c constraints.txt`, so a broken constraints pin (like +`accelerate==0.34.2` was) is invisible to CI and only surfaces for someone who follows +the README's documented install command. This step is the only thing that catches that +class of bug — don't skip it because "CI is green." + +## Step 4 — Confirm the four CI gates still pass + +Comment-only edits shouldn't break anything, but prove it — install dev deps in the same +venv and run all four gates, on 3.12 and 3.13 if both are available: + +```bash +/tmp/depcheck/bin/python -m pip install -e ".[dev]" +/tmp/depcheck/bin/ruff check src/ tests/ +/tmp/depcheck/bin/ruff format --check src/ tests/ +/tmp/depcheck/bin/mypy src/insight_extractor +/tmp/depcheck/bin/pytest tests/unit/ -v --tb=short +``` + +## Step 5 — Optional: spot-check pip-audit against the pinned environment + +This is a lighter touch than `deps-security-refresh`'s full CVE triage — you're checking +whether `docs/SECURITY_AUDIT.md`'s finding set has silently gone stale, not doing new +reachability analysis. + +```bash +/tmp/depcheck/bin/pip install pip-audit +/tmp/depcheck/bin/pip-audit +``` + +Compare the finding IDs against `docs/SECURITY_AUDIT.md`'s tables. If the set is +unchanged, nothing to do. If there's a **net-new** ID not listed anywhere in the doc, +add one factual line noting it exists (ID + fix version, no verdict) to the existing +"not yet triaged" addendum — do **not** write a reachability verdict or touch a pin; +assigning a verdict is a `deps-security-refresh` job and is out of scope here. If +findings *disappeared* from what's documented, leave the doc alone (it's describing +history, not a live claim) unless the disappearance means a documented "unreachable" CVE +is now moot — flag that for the owner rather than deleting the row yourself. + +## Step 6 — Commit + +Only if Steps 1-2 found real drift (most runs, especially soon after this skill is +created, should find nothing — say so and stop): + +``` +fix: + +- : — root cause / when it went stale (commit if found via + `git log -S`) +- : — ... + + + +Verified: clean venv install (3.12[, 3.13]), compatibility triangle, ruff check, ruff +format --check, mypy --strict, pytest tests/unit/ (/ passed). Integration tests +skipped — no model download in this environment. +``` + +No pin in `requirements.txt`, `constraints.txt`, or `pyproject.toml` should appear in +this diff unless Step 1 found an actual mismatch between the three files (not just a +stale comment) — that case is rare and worth double-checking against CLAUDE.md §6 +before touching a version number. + +## Full checklist + +- [ ] Every package's range identical between `requirements.txt` and `pyproject.toml` +- [ ] Every `constraints.txt` `==` pin satisfies both files' ranges +- [ ] No package missing from any of the three files +- [ ] Repo-wide grep for historical pin numbers run; every hit classified + historical-context vs. asserted-as-current +- [ ] `pyproject.toml` comment block, README Known-Issue section, + `docs/SECURITY_AUDIT.md`, and both dependency-aware skill docs checked by hand +- [ ] Clean-venv install with `-c constraints.txt` succeeds +- [ ] transformers/accelerate/sentence-transformers triangle imports cleanly +- [ ] Four CI gates green (3.12, and 3.13 if available) +- [ ] `pip-audit` spot-checked against documented findings; any net-new ID logged + factually, no verdict assigned +- [ ] `git diff --stat` contains no pin changes unless Step 1 found a real mismatch +- [ ] Report states plainly if nothing was found to fix — a clean pass is a valid, + useful outcome, not a failure to find something