Benchmark against MiniSom and SOMPY, and track performance with asv - #21
Merged
Conversation
…iSom
Two independent implementations of Kohonen's equations agreeing is stronger
evidence than either one's own tests. That reasoning already justifies
tests/test_linalg_matches_sklearn.py, which failed on first run, turned out to
have the wrong reference, and then exposed a real 5.8% defect in this package's
linear initialization.
MiniSom is the closest peer, and the two look more interchangeable than they
are: activate, winner, quantization, quantization_error and get_weights all
line up while some of the mathematics behind them does not. Measured:
asymptotic_decay identical, at exactly 0.0
gaussian 1.1e-16, one unit in the last place
sequential training 2.8e-16 relative, after up to 200 iterations
batch training 1.3e-15 relative, looser because the two accumulate
Eq. (8) in a different order
mexican hat differs: (1-u)e^-u here, (1-2u)e^-u there, so the zero
crossing sits at sqrt(2)*sigma rather than sigma
bubble differs: max(|dx|,|dy|) <= round(sigma) here, strict
c-sigma < i < c+sigma there. At sigma=1, 9 nodes to 1
Both disagreements are convention choices, not defects. MiniSom's mexican hat
is isotropic, so it never had the separability bug fixed here in 0.2.0. They
are pinned so that neither is later "fixed" into the other.
Three controls are needed before the training runs can be compared at all, and
each is asserted rather than assumed. Identical injected initial models, since
the two RNGs and PCA initializations differ. The sequential mode rather than
the random one, because arange(T) % n and np.resize(arange(n), T) are the same
sequence where the random modes are genuinely different. And a constant
learning rate on train_batch_offline, which otherwise relaxes toward the
Eq. (8) mean by a decaying eta rather than assigning it.
Note that MiniSom's train_batch is stepwise training in sequential sample
order, not Kohonen batch; train_batch_offline is the Eq. (8) implementation.
minisom==2.3.6 joins the dev extra: MIT, a single minisom.py, and it declares
no dependencies of its own.
Both performance claims this package makes are against its own previous code. Neither says anything about the alternative a user is actually choosing between. The script prints two tables. "Same mathematics" applies the six controls that reduce the two libraries to the same computation, verifies the trained models agree to within 1e-12 relative, and only then reports a timing; the controls are the ones tests/test_minisom_agreement.py asserts, so the protocol cannot quietly stop holding. "Own initializer" holds the algorithm and every hyperparameter fixed and lets each library seed its own models, which isolates the one default the two do not already share. Measured on a 400-sample, 8-feature dataset, medians of 9 interleaved repeats: 20x20 batch 1.03x faster 40x40 batch 1.34x slower 60x60 batch 1.64x slower 20x20 sequential 1.12x faster 40x40 sequential 1.06x faster 60x60 sequential 1.08x faster Batch is the finding and it is not favourable. The gap tracks the ratio of nodes to samples, because batch_update walks every node in Python: 3600 nodes over 30 iterations is 108,000 einsum calls at 60x60, where MiniSom's node-side update is a single vectorised divide and its Python loop runs over the 400 samples instead. Fixing that means changing _core/_update.py, so it is not in this diff. An earlier version of this script reported sequential as 7.10x SLOWER, which was a defect in the harness rather than a result. SOM.train computes the quantization error over the whole dataset to fill in its TrainingReport and MiniSom's entry points do not, so timing train charged this package for a pass the other arm never made. On the sequential cases that pass is larger than the training itself: 30 steps touch 30 samples, the report touches all 400. Timing _train_batch and _train_stepwise directly, as bench_batch.py already does, gives the numbers above. Recorded in the module docstring because both figures were reproducible and only one of them measured training. compare() gains a repeats parameter so a cross-library case, which trains a whole map per repeat, need not use the 31 that a single update step can afford.
SOMPY's batch training is Kohonen Eq. (8) by way of the Helsinki SOM Toolbox, the same algorithm this package implements, so comparing the two is comparing implementations of one published equation. Getting there costs more machinery than MiniSom did, for one reason: SOMPY cannot be installed alongside this package at all. It uses np.Inf at four sites, three of them as default arguments evaluated when the class body executes. np.Inf was removed in NumPy 2.0, so `import sompy` raises AttributeError on every NumPy this package supports. Two further reasons to keep it at arm's length regardless: importing it reconfigures the root logger for the whole process, and it pulls in matplotlib and scikit-image through sompy.visualization. So the comparison runs across a process boundary. sompy_worker.py trains one map inside a hand-built .venv-sompy and times only that call; bench_vs_sompy.py launches one worker per repeat, which keeps the arms interleaved while leaving the half second of interpreter start-up outside every measurement. If the environment is absent the script prints the setup command and exits 0. Nothing installs anything. Four things about that environment were each found by an install failing, and all four are recorded next to the command: NumPy must be pinned below 2 in the same resolve as everything else, since SOMPY's own metadata says numpy >= 1.7 and anything installed afterwards drags NumPy 2 back in; Python 3.12 is the ceiling for NumPy 1.26 wheels; scikit-image, joblib and matplotlib are imported but absent from install_requires; and the pin is a commit SHA because the repository has no tags and is not on PyPI under a name that resolves to it. Measured, batch and gaussian only, medians of 9 interleaved repeats: 20x20 1.30x faster 40x40 1.74x faster 60x60 1.99x faster Trained models agree to 1.4e-07 relative and the quantization errors match to four decimals. That is looser than the 1e-12 MiniSom is held to, because SOMPY rounds its codebook to six decimals after every update and expands ||x-w||^2 as -2x.w + ||w||^2, the less stable arrangement. The measured figure is printed in the table rather than only asserted, so a reader sees what was tolerated. Two things the harness must not do, both recorded in the code. SOMPY's calculate_quantization_error returns mean(|x - m|) over every feature, an elementwise MAE rather than the mean Euclidean distance per sample, so the metric is recomputed here for both arms; SOMPY is inconsistent with itself about this, since the value it logs per epoch is an L2 distance. And the worker reports its own NumPy version in its output, because this process cannot read it: importlib.metadata would name the version SOMPY cannot run on. Cases stop at 60x60. SOMPY materialises the full (nnodes, nnodes) neighborhood, 104 MB there and 800 MB at 100x100, which is the tensor this package rejected in 0.3.0. The cap and its reason are printed with the results rather than left silent.
Both benchmark scripts this repository already had compare two implementations at one commit. Neither notices a regression introduced between commits, which is the failure mode a library actually ships. airspeed velocity is what numpy, scipy, pandas and scikit-learn all use for that, and it answers a different question from benchmarks/: this measures python-som against its own git history. Sixteen benchmarks over the parts of this package that have changed before and may change again: both training loops, the neighborhood kernel introduced in 0.4.0 alongside the per-node path it replaced, the matching functions that are the other half of a batch iteration, the SVD that replaced scikit-learn's PCA, and the artifact round trip. Two are peakmem rather than time: the kernel's size is the entire justification for the kernel, since the alternative reached 800 MB on a 100x100 map, and a regression there would otherwise be silent. The directory is named after scikit-learn's so it cannot be confused with benchmarks/, which holds hand-run scripts rather than an asv suite. Nothing gates on a timing. ci.yml gains a step that executes every benchmark once and records nothing, which catches benchmark code rotting unnoticed while leaving the numbers out of the merge decision; the same step checks that the comparison scripts still import and that bench_vs_sompy.py degrades cleanly when its interpreter is absent. Real measurement is workflow_dispatch only, because a shared runner cannot measure anything: an earlier bench_vs_minisom.py swung a comparison from 2.56x to 1.01x between two runs on an idle laptop. Two things about asv's CLI that are not obvious and cost a CI round trip to find out. It has no `check` subcommand, so executing every benchmark without recording is `run --quick --dry-run`. And `asv machine --yes` has to run first: without a machine profile asv exits 1 before running anything. .asv/ is gitignored, for two reasons. Results are per-machine, so a database from one laptop tells nobody anything about another. More importantly, `asv machine` records the host's name, CPU model, core count, RAM and kernel version into .asv/results/<machine>/machine.json, which is a hardware fingerprint of whoever ran it. The manual workflow uploads only the comparison text for the same reason. asv==0.6.6 goes in a new `bench` extra rather than in `dev`, for the reason `analysis` is separate: it pulls about ten transitive packages and no gating job needs them.
The measurements exist in benchmarks/ but nobody choosing between these three libraries reads a benchmark script. This is the page that answers the question, and most of its length goes to why the measurement is harder than it looks rather than to the numbers. Both results are published as measured. Against SOMPY this package is 1.3x to 2.0x faster; against MiniSom it is 1.3x to 1.6x SLOWER at batch training, with the gap growing as the map grows, and the page says why: batch_update walks every node in Python where MiniSom's node-side update is a single vectorised divide, so the ratio tracks nodes against samples. Stepwise is within 10%. The page leads with the differences rather than the timings, because three packages that look interchangeable are not: MiniSom's train_batch is stepwise training, only the gaussian is the same function in all three, and the bubble selects 9 nodes here against 1 in MiniSom at sigma=1. It states in the other direction too that MiniSom's mexican hat is isotropic and so never had the separability bug this package shipped and fixed in 0.2.0. Every difference is presented as a convention choice, because that is what each one is. Two warnings a reader needs and would not otherwise get: `pip install sompy` installs an unrelated 2016 package by a different author, and `asv machine` writes the host's name, CPU model, RAM and kernel version to disk. No hostname appears in the recorded machine description. CPU model, core count, OS and Python version are enough to read the numbers against. The README gains a Benchmarks subsection pointing here. The CHANGELOG entry goes under Unreleased and opens by saying nothing in the shipped package changed, since a reader scanning for behaviour changes should be able to stop there. Also folds the one 105-character line left in the 0.6.1 entry, rather than spending a separate change on it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Benchmarks python-som against MiniSom and SOMPY, and adds asv to track it against its own git history. No line under
src/changes, which is what makes a diff this wide safe to review; the five commits are ordered so it reads one concern at a time.Results, published as measured
The MiniSom batch result is not favourable and is published unchanged.
batch_updatewalks every node in Python, 108,000einsumcalls on a 60x60 map over 30 iterations, where MiniSom's node-side update is a single vectorised divide and its Python loop runs over the 400 samples instead. The ratio tracks nodes against samples. Vectorising it means changing_core/_update.py, so it is deliberately not in this diff.Why most of the work is fairness, not timing
The three packages look interchangeable and are not. MiniSom's
train_batchis stepwise training; Eq. (8) istrain_batch_offline. Only the gaussian is the same function in all three, and the bubble selects 9 nodes here against 1 in MiniSom at sigma=1. SOMPY'scalculate_quantization_errorreturns an elementwise MAE rather than the mean Euclidean distance.Eight controls reduce them to the same computation, and the trained models then agree: 2.8e-16 relative against MiniSom stepwise, 1.3e-15 batch, 1.4e-07 against SOMPY (which rounds its codebook to six decimals every iteration). That agreement is what makes the timings mean anything, and
tests/test_minisom_agreement.pyrechecks it on every CI run.A harness bug worth flagging
The first version of the harness reported MiniSom sequential as 7.10x faster. That was wrong.
SOM.trainscores the whole dataset to fill in itsTrainingReportand MiniSom's entry points do not, so timingtraincharged this package for a pass the other arm never made. With 30 steps against 400 samples, that pass was larger than the training itself. Timing the loops directly, the same case comes out 1.08x slower for MiniSom. Both numbers were reproducible; only one of them measured training. Recorded in the module docstring.SOMPY needs its own interpreter
It uses
np.Infat class-definition time, removed in NumPy 2.0, so it cannot be installed alongside this package at all. It gets a hand-built.venv-sompyand a subprocess worker, launched once per repeat so the arms still interleave. Nothing installs anything: the script prints the setup command and exits 0 if the environment is absent.asv
Sixteen benchmarks over the training loops, the kernel, matching, the SVD and artifacts. No timing gates anything: CI executes each once and discards the numbers. Real runs are
workflow_dispatchonly, because shared runners cannot measure anything..asv/is gitignored, sinceasv machinewrites the host's name, CPU model, RAM and kernel version.Verification
All five commits pass
ruff check,ruff format --check,mypy,pytest --cov(461 tests, 100%) andmkdocs build --strictindependently, not only at the tip. Prose scanned withllm-writing-patterns: zero severity-5 findings.Follow-ups, not here
batch_update's node loop, worth roughly 1.6x at 60x60.