feat: save and load a trained map without pickle - #11
Merged
Conversation
Static analysis for review passes, pinned exactly like the rest of the tooling. Nothing here gates a merge: ruff and mypy remain the enforced checks. bandit reports no issues across src/ and benchmarks/; pip-audit reports no known vulnerabilities. bandit excludes tests/ rather than disabling B101, so `assert` is still flagged in src/, where it would be wrong -- asserts vanish under `python -O`, so validation has to raise. CI names the extras it needs instead of using --all-extras, so review tools no CI job runs are not installed on every matrix entry. `examples` is named explicitly because tests/test_end_to_end.py imports matplotlib. Two candidates were rejected rather than added: - `osv-scanner` on PyPI is version 0.0.1, registered 2026-07-16, with no author, no home page, and the summary "Reserved name placeholder. No functionality." The real tool is a Go binary from Google. Installing a code-free package squatting the expected name is the slopsquat case the dependency rules exist for. - `semgrep` pins `mcp==1.23.3`, which carries PYSEC-2026-3481/3482/3483 (fixed in 1.27.2/1.28.1). The exact pin makes them unremediable from here and 1.172.0 is already the latest. Against that, bandit finds nothing in src/, and this is a pure-numpy library with no network, auth, crypto, SQL or templating -- most of what semgrep's rulesets target. Revisit if the pin is relaxed. - `guarddog` is genuine but first released 2022-11-28, after this project, so it fails the rule that a dependency should predate what it is added to.
Until now the only way to keep a trained map was `pickle`, which is arbitrary code execution on load. `save_npz` and `load_npz` write one `.npz` holding the models as an array and everything else as JSON beside them: the configuration, the seed, the generator state and the last training report. A reloaded map continues the same random stream, so training it further produces what never stopping would have produced. Both the seed and the generator's current state are saved, because the seed alone restarts the stream and a resumed run would then diverge from an uninterrupted one without saying so. A test asserts the two are identical, and a second test asserts that re-seeding really would differ, so the first cannot pass with the state discarded. Strategies are saved by name, resolved through registries: DECAY_FUNCTIONS and DISTANCE_FUNCTIONS are added alongside the existing NEIGHBORHOOD_FUNCTIONS, keyed by each function's own name. A map built from the shipped functions round-trips completely. One built with a caller's own function records the name for provenance and raises ArtifactError naming the argument to pass it back through, which the loader accepts. A functools.partial has no name, so it is treated as custom rather than reloaded as the wrapped function with different arguments. The seed and the generator state are separate keys in the metadata. Collapsing them conflates "where the stream began" with "where it has reached", and assigning an inner PCG state where a full bit_generator.state belongs fails with "state must be for a PCG64 RNG". Also adds SOMConfig and a frozen TrainingReport, exposed as `som.config()` and `som.last_report`. `train()` still returns the quantization error. `final_learning_rate` is None for batch training, which has no step size: Eq. (8) is a weighted mean, and reporting the unused initial value would read as though it had been applied. The training loops now return the values they finished on rather than having `train` recompute them, so the report states what ran. On the load path: allow_pickle=False is passed explicitly, names resolve only through the registries so nothing from the file is imported or evaluated, and the member list, metadata JSON, format version, weight dimensions and weight shape are all validated before a map is constructed. Every failure, including a malformed generator state that NumPy rejects, surfaces as ArtifactError. A test builds a payload that really would create a file, proves it executes when unpickled, and shows the loader refusing a file containing it with nothing created. What that does not buy is stated in the docs as plainly as what it does: an `.npz` is a zip, so a hostile file can still attempt resource exhaustion through decompression. `tqdm` is now bound to None when absent rather than left undefined, so a type checker reads the guarded use as reachable instead of possibly-unbound at every call site. The public-method surface test now includes classmethods. It used `inspect.isfunction`, which is False for a classmethod accessed through the class, so it silently ignored `load_npz` and would have ignored any classmethod added later.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
save_npz/load_npzwith no pickle, plusTrainingReportandSOMConfig. Last feature PR for0.4.0. Two commits: the analysis tooling is separate from the feature so each reviews on its own.
The problem this had to solve
A map holds four callables — the neighborhood, two decays and the distance — and a callable cannot
go into a file without
pickle. Only the neighborhood had a name registry. So "save a map" had noobvious answer, which is why this one got a design pass before any code.
What a map actually contains, measured rather than assumed:
input_len, rates, radii, cyclic, seednp.random.Generatorbit_generator.stateis JSON-ableDecisions
One file, not a file plus a sidecar. A JSON string stores inside an
.npzand loads fine underallow_pickle=False. Provenance that can be separated from its artifact will be.Strategies resolve by name.
DECAY_FUNCTIONSandDISTANCE_FUNCTIONSare added beside theexisting
NEIGHBORHOOD_FUNCTIONS, keyed by each function's own name. A map built from shippedfunctions round-trips completely; one built with your own function records the name for provenance and
refuses to load silently:
Both halves are tested — the error and that its suggested fix works. An error whose suggestion
doesn't work is worse than no suggestion.
A
functools.partialfalls out correctly: no name, so it is treated as custom rather than silentlyreloading
partial(exponential_decay, factor=3.0)asfactor=2.0.Both the seed and the generator state are saved, and this is the part worth reviewing closely. The
seed says where the stream began; the state says where it has reached. Only the state resumes it.
The test that proves it works
Stepwise mode deliberately — it draws samples from the generator, so a lost state changes which
samples arrive. Batch mode would pass this with the state thrown away.
And a second test asserts that re-seeding really would diverge, so the first cannot pass for the
wrong reason. Without it, discarding the state entirely would still go green if re-seeding happened to
give the same stream.
Security
The load path is the reason this PR exists in the shape it does.
Cannot happen: code execution (
allow_pickle=Falsepassed explicitly); loading an arbitraryfunction by name (names resolve only through the registries — nothing is imported or evaluated); a
partly-read file treated as valid (member list, JSON, format version, weight dimensions and weight
shape all checked before a map is built — every failure is
ArtifactError).Can still happen: resource exhaustion, because an
.npzis a zip. The docs say so as plainly asthey say the rest.
allow_pickle=Falsebuys "cannot execute code", not "safe to load anything".The headline test doesn't assert that — it demonstrates it. It builds a
__reduce__payload thatcreates a file, proves the payload is live, then shows the loader refusing a file containing it
with nothing created. Liveness is proven through
pickledirectly rather than by asking NumPy tounpickle, which keeps the enabled form of NumPy's pickle flag out of the repository entirely — the
architecture profile forbids it at the artifact boundary, and a test that had to switch that rule off
to run would be the worse test.
Reviewing the load path against the OWASP deserialisation rules turned up one real gap, now fixed: a
crafted generator state produced a raw NumPy
ValueError("state must be for a PCG64 RNG") instead ofArtifactError, so a bad file did not fail consistently. Four malformed shapes are now tested.Bugs found while building this
rngkey, sorng["state"]was the inner PCGstate rather than the full
bit_generator.state. Every round trip failed with "state must be for aPCG64 RNG".
supplied.get(key, resolve(name))evaluated the fallback eagerly, so passing a custom callableback still tried to resolve its name and raised — on exactly the argument that exists to accept it.
inspect.isfunction, which isFalsefor a classmethod accessed through the class, so it silently skippedload_npzand wouldhave skipped any classmethod added later. A surface pin with a hole in it is worse than none.
tqdmwas left undefined when absent rather than bound toNone, which reads to a type checkeras possibly-unbound at every use site however carefully the use is guarded. Pre-existing; surfaced
from IDE diagnostics.
Also
require_provenanceWARN that the architecture profile has carried since refactor: split the package into a functional core and a thin shell #7 is cleared. Thecount stays at 13 only by coincidence — the strategy protocols added one positional-arg WARN of
their own.
save-and-load.md, in the nav, including a worked example of reading an artifact'smetadata without this package installed, since it is plain JSON.
Verification
369 tests, 100% coverage, ruff /
ruff format/mypy --strict/mkdocs build --strictclean,twine checkpasses, architecture profile--strictpasses with 0 ERROR, bandit reports no issuesacross
src/andbenchmarks/,pip-auditreports no known vulnerabilities.Second commit: the analysis extra
bandit,pip-auditandpylint, pinned exactly. Nothing gates a merge — ruff and mypy stay theenforced checks. CI now names the extras it needs rather than
--all-extras, so tools no CI job runsare not installed on every matrix entry.
Two candidates were rejected rather than added, which is the part worth reading:
osv-scanneris a slopsquat risk. The PyPI package of that name is version 0.0.1, registered2026-07-16 (two weeks ago), with no author, no home page, and the summary "Reserved name
placeholder. No functionality." The real tool is a Go binary from Google. Not installed.
semgrepwas added, then removed. It pinsmcp==1.23.3exactly, and that version carriesPYSEC-2026-3481/3482/3483 (fixed in 1.27.2/1.28.1) — so the pin makes them unremediable from here,
and 1.172.0 is already the latest semgrep. Against that: bandit finds nothing in
src/, and this isa pure-numpy library with no network, auth, crypto, SQL or templating, which is most of what
semgrep's rulesets target. Low coverage gained for three CVEs that cannot be patched.
guarddogis genuine (DataDog) but first released 2022-11-28, after this project, so it fails therule that a dependency should predate what it is added to.