fix(pycg)!: decide shard outcomes by convergence, not wall clock - #147
Open
rahlk wants to merge 3 commits into
Open
fix(pycg)!: decide shard outcomes by convergence, not wall clock#147rahlk wants to merge 3 commits into
rahlk wants to merge 3 commits into
Conversation
Sharded PyCG dropped shards that exceeded a wall-clock timeout, and a
dropped shard contributed zero edges. Which shards ran slow depends on
machine load and Ray scheduling, so byte-identical invocations produced
different call graphs: three runs over one 2,364-file fixture gave 48,595
/ 43,431 / 40,224 edges, an 11% spread with PyCG's own contribution
swinging 44%.
The clock bound was redundant. PyCG's fixpoint loop is
while (max_iter < 0 or iter_cnt < max_iter) and not has_converged():
so --pycg-max-iter (default 50) already guarantees termination. Bounding
it a second time by the clock added nothing but the load-dependence.
A shard is now a runaway when its fixpoint stopped at max_iter instead of
converging -- a function of the input alone. Adaptive decomposition is
unchanged, since that is what recovers recall. A runaway that cannot be
split further now keeps the edges it did derive: a capped fixpoint is a
sound under-approximation, so discarding them was pure recall loss on top
of the non-determinism. _PYCG_DECOMP_FLOOR drops 10 -> 1, because a floor
of 10 left small runaways unsplittable and forced them down that path.
Reading convergence needs care in two places, both covered by tests.
Asking cg.has_converged() after analyze() returns is wrong: analyze runs a
CallGraphProcessor pass past the loop, so a post-hoc call compares state
that pass has already moved. The loop's last recorded value is also wrong:
when the cap stops the loop the `and` short-circuits, leaving the False
that admitted the final pass and mislabelling a shard that converged on
exactly pass max_iter. So the cap is raised by one and the loop is cut off
from inside the check -- one extra question, not an extra pass.
BREAKING CHANGE: --pycg-shard-timeout is removed. It is the defect, and
after this change it controls nothing. Use --pycg-max-iter to trade recall
against runtime, deterministically. Note that --pycg-max-iter -1 now has no
wall-clock net behind it, so a divergent shard can run indefinitely there.
Closes #145
Exhausting --pycg-max-iter meant "runaway", so the shard was re-partitioned and re-analysed. That is backwards: hitting the cap means PyCG returned a sound under-approximation, and splitting such a shard makes the answer worse, because every cut severs the calls crossing it. Measured on one 100-file shard of a 2,364-file ORM-heavy project: bounding the fixpoint and keeping the shard whole gave 110,490 edges in 95s, where budget-driven halving gave 15,468 edges in 600s. Re-splitting also pays whole extra rounds of re-analysis -- with a low --pycg-max-iter every shard hits the cap, and one such run took 2h50m without finishing. A capped shard now contributes its edges directly; only a shard that raised is decomposed. Both classifications remain pure functions of the input, so the reproducibility 55965a4 restored is unaffected. Also documents that lowering --pycg-max-iter does not reliably bound runtime (per-pass cost dominates: a 26-file shard needs 5 passes and yields the identical 93 edges at 3 and at 50), adds a README section explaining why sharding exists and what it costs, and regenerates the --help block, which still advertised the --pycg-shard-timeout flag 55965a4 removed. Scope: this removes the load-dependent shard-dropping mechanism, the 11% effect #145 was filed for. Output is not yet byte-identical across runs -- a separate, ~100x smaller source remains in Jedi's overload resolution for open(), tracked as #146. Refs #145, #146
Two corrections to claims made in this branch. Timings: the same 100-file shard, same --pycg-max-iter, same fence and same machine, has completed in ~8 minutes on one run and failed to complete in 81 minutes on another (98% CPU throughout, resident set flat, no output). PyCG's cost on this workload is bimodal, not merely noisy, so the wall-clock figures previously quoted (95s / 600s) are not measurements anyone should rely on -- and the halving figure was a harness cap rather than a completion. The argument for keeping a capped shard whole rests on edge counts, which are deterministic properties of the analysis: 110,490 edges whole versus 15,468 halved, because every cut severs the calls crossing it. That claim stands without any timing. Termination: 55965a4 argued the wall-clock bound was redundant because --pycg-max-iter already guarantees termination. That is wrong. The cap bounds fixpoint *iterations* and is only consulted at pass boundaries, so a single pathological pass escapes it entirely -- which is what the 81-minute run was doing. Removing the timeout removes the only wall-clock bound that existed. The timeout had to go because it made the output depend on machine load, but nothing replaces it, and the docs now say so instead of implying otherwise. Refs #145
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #145.
Problem
Sharded PyCG decided which shards to keep by wall-clock timeout, and a
timed-out shard contributed zero edges. Which shards run slow depends on
machine load and Ray scheduling, so byte-identical invocations produced
different call graphs — measured on a 2,364-file dependency-closed Odoo subset
with
--no-venv --pycg-shard --ray:-a 2-a 2(repeat)-a 4An 11% spread overall; PyCG's own contribution swings 44%.
The issue as filed asked for the incompleteness to be reported in the payload.
That was rejected: a self-describing non-deterministic output is still
non-deterministic. This removes the load-dependence at its source.
The fix (2 commits)
1.
fix(pycg)!: decide shard outcomes by convergence, not wall clockPyCG's loop is
while (max_iter < 0 or iter_cnt < max_iter) and not has_converged(), so shardoutcomes can be decided on convergence instead of the clock.
Correction to an earlier claim in this branch. The first commit message
argues the clock bound was redundant because
--pycg-max-iteralreadyguarantees termination. That is wrong, and the commit message is left as-is for
history with a follow-up commit correcting it. The cap bounds fixpoint
iterations and is only consulted at pass boundaries, so a single pathological
pass escapes it — one shard ran >81 minutes at
max_iter=3withoutcompleting (98% CPU throughout, resident set flat), while the identical shard
completed in ~8 minutes on another run. Removing the timeout removes the only
wall-clock bound that existed. It had to go, because it made the output depend on
machine load — but nothing replaces it yet.
ray.cancelremoved from all fourcollection paths.
contributing zero.
_PYCG_DECOMP_FLOOR10 → 1: a floor of 10 left small runaways unsplittable.2.
fix(pycg): keep capped shards instead of re-splitting themExhausting
--pycg-max-itermeant "runaway → re-split". That is backwards:hitting the cap means PyCG returned a sound under-approximation, and splitting
makes the answer worse because every cut severs the calls crossing it.
Measured on one 100-file shard — keeping it whole gave 110,490 edges;
budget-driven halving of the same shard gave 15,468. Edge counts are
deterministic properties of the analysis; wall-clock on this workload is not, so
no timings are quoted (see Measurement caveat below). Re-splitting also pays
extra rounds of full re-analysis: with a low
--pycg-max-iterevery shard hitsthe cap.
Only a shard that raised is decomposed now.
Reading convergence correctly
_analyze_with_convergence()is not a post-hochas_converged()call. Two traps,both covered by tests:
analyze()returns is wrong —analyzeruns aCallGraphProcessorpass past the loop, so a post-hoc call compares statethat pass has already moved and reports divergence for a converged shard.
the
andshort-circuits, leaving theFalsethat admitted the final pass,mislabelling a shard that converged on exactly pass
max_iter.So the cap is raised by one and the loop cut off from inside the check: one extra
question, not an extra pass.
Breaking change
--pycg-shard-timeoutis removed. It is the defect, and after this change itcontrols nothing. Use
--pycg-max-iterto trade recall against runtime. Note--pycg-max-iter -1now has no wall-clock net, so a divergent shard can runindefinitely there.
Scope — what this does NOT claim
This removes the load-dependent shard-dropping mechanism, the 11% effect #145 was
filed for. Output is not yet byte-identical across runs. A separate, ~100x
smaller source remains in Jedi's overload resolution for
open():f.read()resolves to
_TextIOBase.reador_BufferedIOBase.readdepending on the run,accounting for 0.1–0.3% of edges on the Flask fixture. Filed as #146 with a
full four-way matrix (sequential/Ray ×
max_iter3/50) showing it is independentof sharding, Ray, and the fixpoint cap.
I was not able to obtain a completed Odoo run demonstrating the fix
end-to-end at scale — every attempt was killed, capped, or interrupted. The fix
rests on the mechanism being structurally removed plus tests asserting it, not on
a before/after Odoo comparison.
Measurement caveat
PyCG's cost on this workload is bimodal, not merely noisy: the same 100-file
shard, same
--pycg-max-iter, same fence, same machine, completed in ~8 minuteson one run and failed to complete in 81 on another. Earlier drafts of this PR
quoted wall-clock figures (95s / 600s); those are not measurements to rely on,
and the halving figure was a harness cap rather than a completion. All timing
claims have been removed. What remains — edge counts and the structural removal
of the timeout — is deterministic and verifiable.
I was also unable to obtain a completed Odoo run demonstrating the fix
end-to-end at scale; every attempt was killed, capped, or interrupted.
Tests
test/test_pycg_shard_determinism.py(20 tests), including:SIGALRM,signal.alarm,ray.cancelorshard_timeoutremains reachable — the timeout was reachable from fourcollection paths, so a check scoped to one function would pass while another
still dropped shards by the clock;
_analyze_with_convergenceagainst a fake with PyCG's exactloop shape: converged, capped, the exactly-at-cap boundary, the post-loop-pass
trap, unbounded
max_iter=-1, degeneratemax_iter=0, cap restoration, andrestoration when
analyze()raises;change applied to one runner only would silently leave
--rayon the oldbehaviour.
Full suite: 306 passed, 6 skipped.
Docs
README.md: new Why sharding? section under Analysis levels explainingthe cost curve and the ~75% edge-cut price, in plain language.
README.md:--helpblock regenerated viascripts/update_readme.py— itstill advertised
--pycg-shard-timeout, which commit 1 removes.CHANGELOG.md: both changes, with the scope caveat and the L2 call graph is not byte-reproducible: Jedi overload resolution for open() varies run to run #146 link.