Python: report installed dependency versions, and pick range bounds by kind (ANE-3089)

[spatten]

Overview

A Python dependency declared as a version range was reported at whichever bound the author happened to write first. Given a requirements.txt line of cryptography<60.0.0, >=46.0.3, the CLI reported pip+cryptography$60.0.0 — the exclusive upper bound, which is the one version the range explicitly forbids, and which is not a release that exists. Writing the same range as cryptography>=46.0.3, <60.0.0 reported 46.0.3.

There are two separate causes, and this PR fixes both.

The first is in verConstraintToRevision, which turns a version constraint into the single revision a locator can carry. Its CAnd case took the leftmost candidate that produced a version, and every bound operator was equally willing to answer, so the result depended on source order. It now ranks candidates by what kind of bound produced them — exact, then inclusive lower, inclusive upper, exclusive lower, exclusive upper — and keeps the best rather than the first. This is not Python-specific: Poetry's comma operator and Elixir's and build the same conjunctions.

The second is bigger, and is why the reported version was wrong even in the "good" ordering. When python and pip are available, the setuptools strategy already runs pip list and pip show to find transitive dependencies and edges. It had the installed version of every direct dependency in hand and discarded it, reporting a bound from the manifest instead. In the scan directory I used to reproduce this, the installed cryptography was 44.0.1 — neither of the two answers the CLI gave. The edges to cffi and pycparser came from pip show cryptography, so the true version was sitting in the very data used to build them. Direct requirements now take the installed version, with extras and environment markers preserved.

While doing that I hoisted toCanonicalName out of Strategy.Python.Poetry.Common into Strategy.Python.Util, where both the Poetry and setuptools strategies can reach it, and made it collapse runs of separators the way PEP 503 specifies. Package-name matching in the setuptools strategy was previously a bare toLower, so zope.interface and zope-interface did not match.

Acceptance criteria

When users scan a Python project whose requirements.txt or setup.py declares version ranges:

• The reported version no longer depends on the order the bounds were written in.
• With python and pip available, dependencies are reported at the version actually installed in the environment.
• A requirement that declares no version at all — a bare requests line — now gets a version, where previously it produced a locator with no revision. This is common in real requirements.txt files and is probably the widest-reaching effect of this PR.
• Without python or pip, which covers --static-only-analysis and container scans, a range falls back to the lowest version it allows rather than an upper bound.

Testing plan

Automated coverage is in test/Srclib/ConverterSpec.hs (new — this function had no tests at all) and test/Python/ReqTxtSpec.hs. Below is the manual check, which is what actually demonstrates the behavior.

Setup. Create a scan directory with these two files:

setup.py:

from setuptools import setup

setup(
    name="repro-ane-3089",
    version="0.0.1",
)

requirements.txt:

cryptography<60.0.0, >=46.0.3

Install a version of cryptography that differs from what the file declares, so that the manifest and the environment disagree and you can see which one is reported:

python -m venv .venv && source .venv/bin/activate
pip install 'cryptography==44.0.1'

1. Upper bound written first, with pip available. Run fossa analyze --output | jq -r '.sourceUnits[0].Build.Dependencies[].locator'.

Expect pip+cryptography$44.0.1 — the installed version. On master this reports pip+cryptography$60.0.0.

2. Lower bound written first. Change the line to cryptography>=46.0.3, <60.0.0 and run the same command.

Expect pip+cryptography$44.0.1 again. The reported version must not change when only the bound order changes. On master this reports pip+cryptography$46.0.3, so the two orderings disagree.

3. No version declared at all. Change the line to just cryptography and run the same command.

Expect pip+cryptography$44.0.1. On master this reports pip+cryptography$ with no revision.

4. Without pip, both bound orders. Run fossa analyze --output --static-only-analysis against each of the two orderings from steps 1 and 2.

Expect pip+cryptography$46.0.3 for both — no environment to consult, so the CLI falls back to the lowest version the range allows. On master the upper-bound-first ordering reports 60.0.0 here.

5. Transitive dependencies still resolve. In any of the runs above, check the full dependency list.

Expect cffi and pycparser to appear with concrete versions, and pip+cryptography$44.0.1 to have an imports edge to pip+cffi$.... This confirms the direct dependency and the graph node carrying the edges are the same node, rather than the package appearing twice.

Risks

Reporting the installed version overrides an exact pin. If requirements.txt says requests==2.25.1 and the environment has 2.30.0, this now reports 2.30.0. In a healthy scan — pinned file, clean venv, pip install -r actually run — the two agree and nothing changes. It only differs when the environment has drifted from the manifest, and in that case the environment is what ships. I think this is right, and it is consistent with the CLI already trusting pip for the entire transitive closure, but it is the judgment call in this PR most worth a second opinion.

Ranking bounds for COr is arbitrary rather than principled. For a conjunction the branches are complementary bounds on one range, so ranking them means something. For a disjunction they are alternative ranges and nothing in the constraint says which is installed. I applied the same ranking anyway so the result is at least order-independent, and said so in a comment. Open to just leaving COr left-biased instead.

The PEP 503 change alters Poetry matching. toCanonicalName now collapses separator runs, so foo__bar canonicalizes to foo-bar instead of foo--bar. Poetry is the only existing caller. Rare, and the old result looks like a bug, but it is a behavior change outside the ticket's scope.

Metrics

Not readily trackable today. The effect worth knowing is how often a reported Python version changes for a given project between CLI versions, which would need a comparison across scans on the server side rather than anything the CLI can emit.

References

• ANE-3089: Pip version bounds reported inconsistently.

Two follow-ups I deliberately left out, both worth their own tickets:

• Warn when the installed version does not satisfy the declared range. This is exactly the state of my repro directory — 44.0.1 installed against >=46.0.3 means that environment was never pip install -r'd, which makes the transitive graph suspect too. It needs a PEP 440 comparator, and neither semver nor versions in our dependency set implements PEP 440's ordering rules for epochs and pre/post/dev releases. Writing one is larger than this whole PR.
• COr binds tighter than CAnd in the Poetry and Elixir constraint parsers. makeExprParser takes its operator table in descending precedence, and both tables list the OR operator first — src/Strategy/Python/Poetry/PyProject.hs:389 and src/Strategy/Elixir/MixTree.hs:454. In both Poetry and Hex, AND binds tighter, so >=2.0,<3.0 || >=3.2,<4.0 should parse as two ranges and currently does not. The current grouping is asserted as correct in test/Python/Poetry/PyProjectSpec.hs:207 and test/Elixir/MixTreeSpec.hs:479, so fixing it means rewriting passing tests.

Checklist

• I added tests for this PR's change (or explained in the PR description why tests don't make sense).
• If this PR introduced a user-visible change, I added documentation into docs/.
• If this PR added docs, I added links as appropriate to the user manual's ToC in docs/README.ms and gave consideration to how discoverable or not my documentation is. — no new pages, only edits to the existing setuptools.md, which is already linked.
• If this change is externally visible, I updated Changelog.md. If this PR did not mark a release, I added my changes into an ## Unreleased section at the top.
• If I made changes to .fossa.yml or fossa-deps.{json.yml}, I updated docs/references/files/*.schema.json AND I have updated example files used by fossa init command. — n/a.
• If I made changes to a subcommand's options, I updated docs/references/subcommands/<subcommand>.md. — n/a, no option changes.

🤖 Generated with Claude Code