From 27ae4ca29aa0fdb125cc5cc7e21da3f402ad6cba Mon Sep 17 00:00:00 2001 From: Aman Sharma Date: Tue, 28 Jul 2026 13:00:45 +0200 Subject: [PATCH] Bumping version via claude hook --- avoiding-stale-dependency-declarations.md | 120 ++++++++++++++++++++++ 1 file changed, 120 insertions(+) create mode 100644 avoiding-stale-dependency-declarations.md diff --git a/avoiding-stale-dependency-declarations.md b/avoiding-stale-dependency-declarations.md new file mode 100644 index 0000000..ce69551 --- /dev/null +++ b/avoiding-stale-dependency-declarations.md @@ -0,0 +1,120 @@ +--- +title: "Avoiding stale dependency declarations via Claude's hook" +--- + +# Avoiding stale dependency declarations via Claude's hook + +Coding agents like Claude Code frequently write out dependency manifests +(`pom.xml`, `requirements.txt`, `pyproject.toml`, `package.json`) from +memory. Since the model's knowledge is frozen at training time, an exact +version pin it writes can already be outdated by the time it lands in your +repository, sometimes by many releases. + +[`yul`](https://github.com/chains-project/yul) is a small Claude Code +`PreToolUse` hook that addresses this at write time. Whenever Claude writes +or edits a manifest file, `yul` checks any newly added or changed +dependency that is pinned with an exact version against the real registry +(Maven Central, PyPI, npm). If the pin is outdated, it blocks the write +(exit code 2) and reports the current version on stderr, so Claude sees it +immediately and retries with the up-to-date version. Untouched dependencies +and non-exact pins (ranges, carets) pass through untouched, and +resolver/network errors fail open so the hook never blocks on its own +errors. + +## Methodology + +To see how often this actually changes what ends up on disk, we ran 20 +small scaffolding prompts against Claude Code, each twice: once with the +`yul` hook installed, once without. Everything else about the prompt and +environment was identical between the two runs. + +- 20 cases total, 5 each for Maven, npm, pip (`requirements.txt`), and + `pyproject.toml`. +- Each ecosystem's 5 cases split into two manifest states + - 4 cases where the manifest is being written from scratch + - 1 case where the manifest already exists and is being edited. + +## Results + +The full set of prompts and per-case transcripts is available in the +[`benchmark`](https://github.com/chains-project/yul/tree/55a771629e97567c9a644b7db699c39b235f9e75/benchmark) +directory of the `yul` repository. + +Reason codes, for cases where nothing was updated: + +- `range-pin` — dependency was written as a range/caret, not an exact pin; `yul` only checks exact pins by design. +- `cli-install` — Claude ran the package manager's own install command (e.g. `npm install`) instead of writing the manifest itself, so nothing reached `yul` to check. +- `rate-limited` — an exact stale pin *was* written, but the registry lookup itself failed (Maven Central `429`) and `yul` fails open on resolver errors. + +| Case | Ecosystem | Via | Version update | Reason | +|---|---|---|---|---| +| maven-01-pdfbox | Maven | Write | `pdfbox` `3.0.3` → `3.0.8` | - | +| maven-02-json | Maven | Write, Edit | `jackson-databind` `2.17.2` → `2.22.1`, `junit-jupiter` `5.10.3` → `6.1.2` | - | +| maven-03-okhttp | Maven | Write | `okhttp` `4.12.0` → `okhttp-jvm 5.4.0`, `jackson-databind` `2.17.2` → `2.22.1` | - | +| maven-04-logback | Maven | Write | `logback-classic` `1.5.6` → `1.6.0`, `logstash-logback-encoder` `7.4` → `9.0` | - | +| maven-05-jackson-existing | Maven | Edit | `jackson-databind` `2.17.2` → `2.22.1` | - | +| npm-01-express | npm | Bash | - | `cli-install` | +| npm-02-axios | npm | Bash | - | `cli-install` | +| npm-03-lodash | npm | Write | - | `range-pin` | +| npm-04-react | npm | Write | - | `range-pin` | +| npm-05-axios-existing | npm | Bash | - | `cli-install` | +| pip-01-requests | pip | Write | `requests` `2.32.3` → `2.34.2` | - | +| pip-02-flask | pip | Write | `Flask` `3.0.3` → `3.1.3` | - | +| pip-03-pandas | pip | Write | - | `range-pin` | +| pip-04-numpy | pip | Write | - | `range-pin` | +| pip-05-requests-existing | pip | Edit | `requests` `2.31.0` → `2.34.2` | - | +| pyproject-01-fastapi | pyproject.toml | Write | - | `range-pin` | +| pyproject-02-pydantic | pyproject.toml | Write, Edit | - | `range-pin` | +| pyproject-03-httpx | pyproject.toml | Write | - | `range-pin` | +| pyproject-04-sqlalchemy | pyproject.toml | Write | - | `range-pin` | +| pyproject-05-httpx-existing | pyproject.toml | Edit | `httpx` `0.27.2` → `0.28.1` | - | + +**`yul` fired in 9/20 cases, updating a stale pin before it hit disk.** All +3 "existing manifest" cases fired, via `Edit`. In the remaining 11 cases +there was nothing stale to catch: `range-pin` (8 cases) and `cli-install` +(3 cases). + +## Discussion + +### range-pin + +The four `pyproject.toml` cases (fastapi, pydantic, httpx, sqlalchemy) are +non-events: Claude wrote ranges like `>=0.111` or `>=2.0,<3.0` that already +cover the latest release, so there's nothing to nudge. + +The npm cases show a range isn't automatically current, though: +`npm-lodash` pinned `^4.17.21` under the hook vs. `^4.18.1` (the actual +latest) without it, and `npm-react` pinned `^18.3.1` in both runs while +latest is `19.2.8` - a caret only protects against drift within the major +version Claude already believes is current. + +Of the two `pip` cases, `numpy>=1.26,<2.0` is a non-event like the +`pyproject.toml` ones. `pandas>=2.2,<3` is not: latest pandas is `3.0.5`, +which its own upper bound excludes. `yul` doesn't check ranges at all, so +this passes through untouched either way - but it's a reminder that a +range pin can go stale too, just via its ceiling instead of a fixed +version. + +### cli-install + +All three cases (`npm-express`, `npm-axios`, and the existing-manifest +`npm-axios-existing`) ran `npm install ` via `Bash` instead of writing +`package.json` directly. npm resolves and pins the latest release itself +at install time, before any `Write`/`Edit` reaches the hook - so there's +nothing stale for `yul` to catch, and nothing for it to intercept either. + +## Related work + +- J. Spracklen, R. Wijewickrama, A. H. M. N. Sakib, A. Maiti, B. Viswanath, + and M. Jadliwala, "We Have a Package for You! A Comprehensive Analysis of + Package Hallucinations by Code Generating LLMs," in *Proc. 34th USENIX + Security Symposium*, 2025. +- C. Wang, J. Wu, X. Ling, T. Luo, and C. Zhao, "Correct Code, Vulnerable + Dependencies: A Large Scale Measurement Study of LLM-Specified Library + Versions," arXiv:2605.06279, 2026. + +## Future work + +The remaining gap is about pin style (ranges pass through by design) and +about dependencies added via the package manager's own CLI, which never go +through `Write` or `Edit` at all.