test(imq): vet the image-quality family against OpenCV and CellProfiler - #457
Conversation
|
|
||
| Reference: Kumar, Chen, Doermann, "Sharpness estimation for document and scene images" (ICPR 2012), | ||
| as implemented in https://github.com/umang-singhal/pydom (dom/dom.py, DOM.get_sharpness, defaults | ||
| width=2, sharpness_threshold=2, edge_threshold=0.0001). The reference is transcribed here rather |
There was a problem hiding this comment.
This says the functions below are transcribed line-for-line from pydom, whose repository is GPL-3.0, into MIT-licensed Nyxus without preserving its license/copyright. Please avoid vendoring this copy (invoke upstream or reimplement independently from the paper), or resolve the licensing explicitly before merge.
Yeah this could be kinda bad legally.
| | `MIN`/`MAX_SATURATION` | ROI | constant (`min == max`) | VALID-but-divergent | not asserted; Nyxus and CellProfiler disagree (below) | | ||
| | `MIN`/`MAX_SATURATION` | mask | narrower than the bounding box | VALID-but-divergent | not asserted; Nyxus counts in-box out-of-mask zeros, CellProfiler does not | | ||
| | `POWER_SPECTRUM_SLOPE` | ROI short side | < 24 px | VALID-prod-only | `imq.regression_quality_roi` — the pin is the guard's return value | | ||
| | `POWER_SPECTRUM_SLOPE` | ROI short side | ≥ 24 px | NOT PINNED | the algorithm's only reachable cell, and it is defective (below) | |
There was a problem hiding this comment.
VALID-but-divergent and NOT PINNED are not SPEC §5.1 dispositions. These are reachable production cells: classify them as VALID-BUT-PRODUCTION-ONLY and add regression guards, or justify INVALID; recording the defects alone does not satisfy the matrix→test rule.
| # (2) the tiling LOCAL_FOCUS_SCORE is defined over, asserted rather than assumed | ||
| tiles = nyxus_tiles(img, SCALE) | ||
| ok = len(tiles) == EXPECTED_TILES | ||
| all_ok &= ok | ||
| print(f" {'OK ' if ok else 'FAIL'} FOCUS_SCORE: opencv={focus!r} " | ||
| f"nyxus={NYXUS['FOCUS_SCORE']!r}") | ||
|
|
||
| # (3) LOCAL_FOCUS_SCORE | ||
| tiles = nyxus_tiles(IMG, SCALE) | ||
| local = sum(float(cv_laplacian(t).var()) for t in tiles) / (SCALE * SCALE) | ||
| ok = abs(local - NYXUS["LOCAL_FOCUS_SCORE"]) <= TOL * max(1.0, abs(local)) | ||
| all_ok &= ok | ||
| print(f" {'OK ' if ok else 'FAIL'} LOCAL_FOCUS_SCORE: opencv={local!r} " | ||
| f"nyxus={NYXUS['LOCAL_FOCUS_SCORE']!r} " | ||
| f"({len(tiles)} tile(s) of {tiles[0].shape[1]}x{tiles[0].shape[0]}, /scale^2)") | ||
|
|
||
| print(f"\n{'ALL OPENCV-VET CHECKS PASSED' if all_ok else 'SOME CHECKS FAILED -- do not promote'}") | ||
| print(" %s tiling: get_local_focus_score() visits %d tile(s) of %dx%d, divisor scale^2 = %d" | ||
| % ("OK " if ok else "FAIL", len(tiles), tiles[0].shape[1], tiles[0].shape[0], | ||
| SCALE * SCALE)) | ||
|
|
||
| produced = { | ||
| "FOCUS_SCORE": float(cv_laplacian(img).var()), | ||
| "LOCAL_FOCUS_SCORE": sum(float(cv_laplacian(t).var()) for t in tiles) / (SCALE * SCALE), |
There was a problem hiding this comment.
LOCAL_FOCUS_SCORE is explicitly a partial-pipeline oracle, so SPEC §4 requires a negative control that measures the uncovered gap. EXPECTED_TILES = 1 only checks that the copied Nyxus tiling stays copied; also compute the four-tile result and show it differs from the pinned partial value.
| | `POWER_SPECTRUM_SLOPE` | ROI short side | < 24 px | VALID-prod-only | `imq.regression_quality_roi` — the pin is the guard's return value | | ||
| | `POWER_SPECTRUM_SLOPE` | ROI short side | ≥ 24 px | NOT PINNED | the algorithm's only reachable cell, and it is defective (below) | | ||
| | `SHARPNESS` | `width` | 2 | VALID-prod-only | `imq.regression_quality_roi` — the reference DOM measure does not reproduce it (below) | | ||
| | any | out-of-core (`osized_calculate`) | — | NOT COVERED | no assertion reaches either NT path (below) | |
There was a problem hiding this comment.
Blocker: Out-of-core is a reachable production execution mode, so NOT COVERED is not a terminal SPEC §5.1 disposition. Add guards/dispositions for each IMQ feature before claiming the family matrix: PowerSpectrum and Sharpness have empty osized_calculate() implementations, while the FocusScore NT path documented below can overrun its 900-element window on a 100×20 ROI and leaves LOCAL_FOCUS_SCORE unset.
There was a problem hiding this comment.
Agreed. fcf3880d already relabels that row VALID-BUT-PRODUCTION-ONLY on your reasoning — it was
pushed after your comment was written, from work that predates it — but that answers the vocabulary
and none of the substance. You asked for guards or dispositions per IMQ feature, and one row for the
family is not that.
All three of your claims check out, and two are worse than you put them:
- The empty
osized_calculate()bodies do not skip those features.
FeatureMethod::osized_scan_whole_image()calls the no-op and thensave_value(), which writes
slope_/sharpness_. Neither member has a default initializer, neither constructor assigns
one, andcleanup_instance()is an empty base with no override in any of the four classes. An
oversized ROI publishes an indeterminate double, not a zero. LOCAL_FOCUS_SCOREis the same case, so it is three features and not two.
local_focus_score_is assigned only atfocus_score.cpp:26, insidecalculate()— the in-RAM
path exclusively — whilesave_value()writes it regardless.- The 900-element overrun is real: a 100×20 ROI writes 2000 entries into it.
Your comment also caught an error of mine. The matrix says the constant-ROI case leaves both
saturations at 0. It does not: osized_calculate() early-returns, but the base calls save_value()
anyway, and the feature methods are long-lived singletons nothing resets between ROIs — so what is
published is the previous oversized ROI's values, or indeterminate on the first. Same shape as the
NGTDMFeature::n_levels static. I will fix that text.
On how to close it: every item above is a source defect, and guarding them per feature needs an
oversized-ROI harness this family's gtest fixture does not build — one that overlaps the 2D
out-of-core repair already in flight, so building a second here means two to reconcile later. I would
rather split the out-of-core row into its own PR carrying the harness, the per-feature dispositions
and the fixes together, and leave #457 as a vetting pass over the in-RAM paths with that row marked
open. #457 changes no source file; the four IMQ feature .cpp md5s are byte-identical to the
previous round, verified in the ASan gate.
If you would rather it not land with an open row, say so and I will hold it and do the out-of-core
work here instead — your call. Either way I will restore the per-feature detail above into
matrix/imq.md.
There was a problem hiding this comment.
Agreed — splitting the out-of-core repair into a separate PR is reasonable. Please define that follow-up’s scope explicitly (the oversized-ROI harness plus per-feature dispositions/guards and fixes for every defect listed above), and make PR #457’s description and matrix/imq.md equally clear that this PR vets only the in-RAM paths and deliberately leaves the out-of-core row open. Please also link the follow-up from #457 once it exists.
…ng it Answers comment 1 of the PR PolusAI#457 review. imq_sharpness_reference_dom.py carried six functions transcribed line for line from https://github.com/umang-singhal/pydom into this tree, each mirroring one upstream statement, with the upstream file named at the top and no licence or copyright preserved. That repository is GPL-3.0 and Nyxus is MIT, confirmed against the GitHub licence API rather than assumed from the README. The script now imports the installed package and calls its public API: get_sharpness for the score, and load/edges/sharpness_matrix for the intermediates the report tabulates. It installs from git, which the previous docstring had ruled out on the grounds that no usable PyPI name exists - true, and irrelevant, because upstream ships a setup.py: pip install git+https://github.com/umang-singhal/pydom.git Into the offline audit env only. The reference is not a Nyxus dependency and CI never invokes this script. An absent reference now exits with that install line and the reason it is not vendored, rather than a traceback. No number in the refutation moved. The reference SHARPNESS is 0.5459295115771082 as before, edgex/edgey 56/73, and the masked counts 28/16 - and reference_sharpness() asserts the intermediates recompose get_sharpness() before returning either, so the diagnostics provably describe the run the entry point produced rather than a plausible reconstruction of it. Divergence 5 is now measured rather than asserted. The raw matrices out of sharpness_matrix hold 50 and 28 pixels at or above the threshold and masking drops those to the 28 and 16 the score is built from, which is the difference the report claims. Divergences 2, 4 and 6 stay read off the reference's source - its public API exposes no hook that isolates them - and the script says so instead of implying they were measured. The report's three inlined upstream snippets are replaced by prose; only the Nyxus C++ snippet remains, and the Python port of sharpness.cpp stays, being this project's own MIT code. TOOLS.md is retitled from "vendored, not installed" to "installed from git, never vendored" and carries the general rule: a reference under a copyleft licence is invoked or reimplemented from the publication, never pasted in. imq_golden_regen.md's reproduction block installs it. No feature values change and no source file outside tests/vetting is touched.
… dispositions Answers comment 2 of the PR PolusAI#457 review. matrix/imq.md classified reachable production cells as VALID-but-divergent and NOT PINNED, neither of which is a SPEC 5.1 disposition, and recorded their defects instead of asserting them. Every verdict is now VALID, VALID-BUT-PRODUCTION-ONLY or INVALID, and each VALID-BUT-PRODUCTION-ONLY cell has a regression guard rather than a paragraph. Five cases, each on the smallest input that reaches its cell: MIN/MAX_SATURATION, constant ROI 0 and 1 MIN/MAX_SATURATION, mask narrower 11/16 and 1/16 than the bounding box POWER_SPECTRUM_SLOPE, short side 1.7837481542489078 at least 24 px The saturation bands are an absolute 0: every pin is k/16, exactly representable, so any other value is a change of behaviour and not a float wobble. The cells exist because CellProfiler computes a different quantity on each - 100% for both extrema on a constant ROI, mask-only counting on a narrow mask - so neither can carry an oracle claim, and what stays uncovered is the agreement, not the code path. The 24x24 pin needed measuring rather than assuming. It is the first assertion in the tree to drive PowerSpectrumFeature past its size guard, and power_spectrum_slope() reads raw_radii[i] from a loop bounded by the padded FFT size, so the pin could have been a snapshot of an out-of-bounds read. Temporary instrumentation, reverted: magnitude.size() 1024 against raw_radii.size() 24, largest index reached 3, no out-of-range read, 3 points surviving to the fit. The fixture is a deterministic modular ramp for a second reason - a smooth ramp leaves fewer than two surviving points, power_spectrum_slope() falls through to the same 0 the guard returns, and the pin could no longer tell the two code paths apart. Both facts are recorded at the pin. Two cells the review did not name are brought into the same vocabulary, since the objection applies equally: LOCAL_FOCUS_SCORE at scale != 2 becomes INVALID (calculate() hardcodes 2 and nothing else calls the method), and the out-of-core row becomes VALID-BUT-PRODUCTION-ONLY. That last one is reachable - phase3.cpp calls osized_scan_whole_image() on every registered feature method - and is the one cell whose guard is still outstanding, because reaching osized_calculate() needs an oversized-ROI harness the gtest fixture does not build. It says so in its own row rather than being relabelled into looking closed. scan_imq_coverage.py found two defects in the guards while they were being written, and both fixes are here. It matched golden-table keys to assertions by feature name, which reported every qualified key as unread; it now matches on the key literal the test passes, which is the stronger check anyway, because MIN_SATURATION is pinned three times across the family and a feature-name match collapses all three onto one. SCOPED_TRACE labels are stripped first, being the other SCREAMING_SNAKE literal in these functions. It also showed the power-spectrum case naming its feature inside a wrapped expression, invisible to the coverage artifact; all five cases now name feature and pin key on the assertion line, and assert_imq_regression_on says why. Supporting artifacts: calc_imq_feature_on() in test_imq_common.h so a case can supply its own ROI, two config recipes (imq.saturation_production_only, imq.power_spectrum_past_guard), one benchmark (bench_imq_matrix_cell_rois), three oracle_coverage.csv regression rows, a regenerated imq_coverage.csv, and the scope sections of imq_cellprofiler_vetting_report.md and not_covered.md rewritten - those cells are no longer uncovered. No feature values change and the four IMQ feature sources are byte-identical to the previous round.
…nnot reach Answers comment 3 of the PR PolusAI#457 review. LOCAL_FOCUS_SCORE is a partial-pipeline oracle - cv2 supplies the Laplacian and the population variance, while the tile extraction and the scale^2 divisor are Nyxus' own definition, reproduced in the generator - and SPEC 4 requires such a family to carry a negative control that measures the uncovered gap. gen_imq_opencv.py had EXPECTED_TILES = 1, which only checks that the reproduced tiling still matches the one Nyxus walks and says nothing about how much of the feature that leaves unvetted. all_tiles() walks every scale^2 tile, which is the tiling the /scale^2 divisor already assumes, and the generator compares the two: all 4 tiles 28.341145833333336 pinned, 1 tile 7.5763888888888902 gap 73% Asserted rather than printed, for the reason the control exists: if the gap ever vanished the one-tile pin would not be a partial value and the scope note would be overstating the defect, so that case has to fail too. The shape follows gen_gabor_skimage.py's D2 control, which is the worked example SPEC 4 names for this situation. The 73% is carried into not_covered.md section E and imq_opencv_vetting_report.md, both of which described the split without sizing it. No feature values change and no source file outside tests/vetting is touched.
vjaganat90
left a comment
There was a problem hiding this comment.
LGTM*
*with view of follow-up PR we agreed upon
All four IMQ oracle assertions were banded at rel=1e-3, and that band was never a measurement. Every call went through assert_feature() in test_feature_calculation_common.h, whose signature ends `double frac_tolerance = 1000`, and no IMQ call passed one. agrees_gt divides the golden by that argument, so 1000 is rel=1e-3 -- which is where the registry's tolerance column got its value. A tolerance that agrees with a helper's default is not evidence that anyone measured anything. Measured, against fresh runs of both oracles (see commit 2 for the generators): FOCUS_SCORE 7.1e-15 absolute, 2.0e-16 relative LOCAL_FOCUS_SCORE 3.6e-15 absolute, 4.7e-16 relative MIN_SATURATION 0 -- bit for bit MAX_SATURATION 2.8e-17 absolute, 1.7e-16 relative All four now assert with ASSERT_NEAR at SPEC 7's exact tier, which that row of the tolerance table defines as an ABSOLUTE 1e-9 band. This follows a747234, which made the same correction for 2D NGTDM after three consecutive reviews asked for it: the label and the assertion now say the same thing, and the registry's tolerance column reads abs=1e-9. MAX_SATURATION's golden moves, and it is the one value in this PR that does. It was pinned at 0.16666666666666666 -- Nyxus' own 16/96 -- under a comment reading "CellProfiler ... = 16/96; tolerance rel=1e-3 (agreement is exact)". CellProfiler's own double is 0.16666666666666669, one ulp higher, because it reports a PERCENTAGE and the generator divides by 100. Both tools count the same 16 of 96 pixels, so this is a unit conversion rather than a disagreement, but the pin of an oracle table should carry the oracle's digits and this one carried Nyxus'. MIN_SATURATION has no such gap: 18/96 = 0.1875 is exactly representable, and there the agreement really is bit for bit. SHARPNESS moves from the five-digit 2.19047 to the full %.17g 2.1904708385718963. The truncated pin sat 3.8e-7 relative from the value it was guarding. Its band is an absolute 2.2e-9 -- rel=1e-9 at that magnitude is 2.1904708385718963e-9, rounded up to two significant figures, and the comment says so rather than leaving the number to be re-derived and found not to match. POWER_SPECTRUM_SLOPE's band is abs=0, because its value is a literal returned by rps()'s early-return path rather than a computed quantity -- any other value is a change of behaviour, not a float wobble. That path is also the reason the pin is 0 at all, which the file now says at the site. Structure follows test_2d_neighbor_cellprofiler.h: a ref_vals_map per file, a named band constant, and one assert helper with a SCOPED_TRACE, replacing four bare literals in four function bodies. The shared fixture moves to a new test_imq_common.h -- one templated calc_imq_feature<F>() -- so the three assertion files stop reaching for roi_cache.h, pixel.h, environment.h, test_data.h and test_main_nyxus.h individually. Include hygiene in the direction the reviews asked for: what the fixture header supplies is not repeated, and the header is named as the source in a trailing comment. check_test_names.py gains TABLE_DIM_AGNOSTIC, the table-level twin of the existing DIM_AGNOSTIC list for files. The 6.3.1 table rule requires a _2d_/_3d_ token, and imq carries neither -- its registry dim IS imq, the same word as its family, so the only conforming name it could have had was imq_imq_opencv_ref_vals. Membership in the new set is the positive claim "this table's family has one implementation", so a 2D table still cannot slip in unmarked. Negative controls, all seven run and all seven behaved: each of the five bands rejects a perturbation just outside it (+2e-9 on the three oracle pins that have room, 1e-12 on POWER_SPECTRUM_SLOPE, +5e-9 on SHARPNESS), a deleted table key is reported by name rather than compared against a default-inserted 0, and the unperturbed tree passes. Verified: 877 gtest cases on Windows, 876 pass and 1 skips without USE_GPU. That is the base's own count at 2210bc4; this pass adds and removes no case, and that is provable rather than measured twice -- test_all.cc and every *_coverage.h are byte-identical to upstream, and none of the four IMQ headers contains a TEST( at all.
…rator
Both IMQ generators validated themselves. Each carried a literal NYXUS = {...} golden
dict and a transcribed IMG matrix of the im_quality fixture, and compared its fresh tool
run against those copies -- so "ALL CHECKS PASSED" meant the script agreed with its own
source, and a hand-edited header or a test_data.h change would have gone unnoticed by
both halves. tests/vetting/not_covered.md had recorded this shape as still live here after
the 2D neighbour pass closed the same thing in its two generators; this closes it.
It was hiding something. Running CellProfiler and comparing against the header rather than
against the dict shows CP's PercentMaximal/100 is 0.16666666666666669 where the pin read
0.16666666666666666. The old generator's TOL was 1e-6, three orders above the 2.8e-17 gap,
so it could not have seen it whatever it was comparing. Commit 1 moves the pin.
Both generators now:
- parse im_quality_intensity / im_quality_mask out of test_data.h and rebuild the ROI
image matrix the way Nyxus does -- the AABB of the masked pixels, with in-box
unassigned positions left at 0. That 0 is the ROI's observed minimum and is what
MIN_SATURATION counts, so a "tidied" transcription would move four goldens;
- parse the header's ref_vals_map by counting braces rather than with a non-greedy
regex, which is what ate the last entry of a table in an earlier parser;
- re-verify every pin at ROUND_TRIP_ABSTOL = 0, which is NOT the band the C++ test asserts
at. The pins are the tool's own digits at %.17g, so a 17-significant-digit literal parses
back to the same double and re-verifying it must round-trip exactly; a non-zero residual
means the tool's output moved and the pin has to be regenerated. Both scripts first shipped
this as the assertion's 1e-9, which is the conflation PolusAI#443's review made gen_ngtdm_mirp.py
stop doing -- at 1e-9 a hand-edited golden could drift nine orders and still be reported as
verified;
- carry the REVERSE check -- a value the oracle produces that the header pins nothing
for -- and exit non-zero on it;
- exit non-zero on a pin they cannot produce.
Two family-specific checks beyond that. gen_imq_opencv.py asserts the tiling
LOCAL_FOCUS_SCORE is defined over: get_local_focus_score() loops y < height - M with
M = height/scale, so at scale=2 it visits exactly ONE 4x6 tile while dividing by
scale^2 = 4. The generator requires len(tiles) == 1, so fixing that bound fails here
instead of silently redefining what the golden means. It also asserts cv2's filtered image
equals Nyxus' hand-rolled laplacian() cell for cell (max abs diff 0.0) before comparing
any scalar -- a variance is a scalar and many different filtered images share one, so a
matching variance alone is weak evidence that two implementations filter alike.
gen_imq_cellprofiler.py adds the range and identity checks mechanically: each saturation
in [0,1], each equal to its own integer count over the 96 pixels, MIN + MAX <= 1, and
scale invariance on max-normalized input -- which is the property that lets CP's [0,1]
floats vet Nyxus' integer PixIntens, so it is asserted rather than assumed. It also fails
if FOCUS_SCORE or LOCAL_FOCUS_SCORE ever appear in its table: CellProfiler publishes
features by those names and they are a different statistic (normalized variance of the raw
image, not variance of the Laplacian), so wiring them up would credit a vetting that does
not exist.
Both also print the versions they actually ran under rather than the literals in their
docstrings -- cv2 and numpy for one, cellprofiler / cellprofiler-core / centrosome / numpy via
importlib.metadata for the other. PolusAI#448's review found a generator labelling every run mirp
2.6.0 from a format string; the docstrings now say they describe the run behind the pins rather
than making a claim about a later one. Printed output matches every version they claimed,
including the centrosome 1.2.3 nobody had checked.
Eight negative controls, all behaved: a pin drifted by one ulp, a pin the generator cannot
produce, a value the oracle produces that nothing pins, the tile count changing under the
golden, a CellProfiler table pinning a name CP shares but does not vet, and the unperturbed
tree passing. The first of those is what proves the round-trip band is live -- at the old 1e-9
it passed.
Verified: both generators run green against the tree they feed. opencv 4.13.0 in the
existing nyxus_mirp env; CellProfiler 4.2.8 in nyxus_cellprofiler. Reproduction, including
the JVM PATH entries that turn a bare exit 127 into a working import, is in
tests/vetting/audit/imq_golden_regen.md.
…refutes
SHARPNESS' registry row has read candidate_oracle = "reference DOM sharpness (Kumar et
al. 2012)", flag = promote-after-deepdive, notes = "Correct (all-double DOM sharpness
port, no overflow bug); left regression pending an independent reference-DOM oracle to
confirm the median-blur/edge/contrast parameterization." The deep dive has now happened.
The parameterization is not what needed confirming.
Nyxus returns 2.1904708385718963 where the published reference returns
0.54592951157710823 on the same fixture -- a factor of four, and structural rather than
numerical. Six differences, each a statement about a specific line:
1. The aggregation is a different statistic. The reference COUNTS pixels whose
sharpness reaches sharpness_threshold=2 (28 and 16 here); Nyxus SUMS the sharpness
values (157.83 and 19.68) and has no threshold parameter at all. The reference's Rx
is a fraction of edge pixels, bounded by 1, which is what makes the paper's
0 < S < sqrt(2) bound hold; Nyxus' 2.19 is already above sqrt(2).
2. Sy is summed over row windows like Sx; the reference sums it down COLUMNS.
3. The edge maps are assigned to opposite convolution axes. Measured: Nyxus edge_x=73,
edge_y=56 against the reference's edgex=56, edgey=73 -- the same two numbers,
exchanged.
4. Both smoothed images are normalized by the row-convolved one's maximum instead of
each by its own.
5. Sx/Sy are never masked by the edge maps before aggregating; the reference masks twice.
6. The last width=2 columns of Sx/Sy are never written.
Two smaller ones, recorded so they are not rediscovered: contrast() uses the forward
difference where the reference uses the backward one, offsetting the contrast field one
row and column against the DOM field the window sums pair it with; and median_blur() pads
by (rows, cols) rather than (ksize-1)/2, doing about nine times the work, with a
remove_padding() whose closing erase() is a no-op so a 768-element tail is left behind.
The measurement is executable. tests/vetting/audit/imq_sharpness_reference_dom.py vendors
the reference's six functions from umang-singhal/pydom (dom/dom.py), named at the top,
because the package is not installable from PyPI under a usable name -- `pip install dom`
fetches a domain-lookup CLI. Alongside them it carries a line-for-line port of
SharpnessFeature::sharpness(), and it ASSERTS that the port reproduces the golden pinned
in test_imq_regression.h, to 2.03e-16. That check is what makes the six differences a
statement about the shipped sharpness.cpp rather than about a Python script: if the C++
changes, the script fails and the report has to be re-derived. Its second check is that
the reference still disagrees -- if that one ever fails, SHARPNESS became promotable.
It prints its installed numpy and cv2 versions rather than the literals in its docstring, for
the reason PolusAI#448's review gave: a script that labels every run from a format string reports a
provenance it did not produce.
It lives in audit/ rather than oracles/ because `dom` is not a SPEC 4 oracle token and it
generates no golden. SHARPNESS stays status=regression, and its row now reads
candidate_oracle "... -- measured and refuted", flag=impl-defect. Promotion is not
blocked on finding an oracle; it is blocked on deciding which of the six differences are
bugs, which is PR/todo.md item 30.
…ects
IMQ had no audit artifacts of any kind -- no coverage CSV, no scanner, no config matrix,
no recipes, no vetting report, no golden-regeneration guide. This adds the set every other
family carries, and fills in what the measurements found.
The two recipe ids the registry has been naming, imq.laplacian_ksize1_zeropad and
imq.saturation_observed_extremum, were defined nowhere: grepping config_recipes.md for
"imq" returned nothing. Both are written, plus imq.regression_quality_roi for the two
snapshot rows, which had a blank recipe. glrlm.ibsi_ng128 is written too -- 32 GLRLM rows
have named it since that family was vetted, and test_2d_glrlm_common.h names it in a
comment, but its section was never added. And a prose sentence had been sitting in the
config_recipe column of 34 firstorder rows ("Not mode-specific. Any disagreement is a
definition, coordinate-frame, ..."); it moves to notes, where it was always a note.
Commit 5 adds the check that would have caught all four.
matrix/imq.md is the config matrix, and its headline is that the family has no
configuration axes at all: none of the four feature classes reads a NyxSetting, so every
knob is a compile-time default on a private static method and a recipe here names a
fixture rather than a settings bundle. Three things that look like knobs and are not:
`static int ksize` is declared in all four feature headers under a "User interface"
comment and is defined nowhere and referenced nowhere -- a translation unit that used it
would fail to link; FocusScoreFeature::kernel[9] is a MUTABLE static that the ksize != 1
branch overwrites in place and never restores, latent only because calculate() hardcodes
ksize=1, the same shape as the NGTDMFeature::n_levels static the 2D NGTDM pass fixed; and
Fsettings itself, which the tests pass because the helper takes one, not because a value
in it is read.
Four defects are recorded there and in PR/todo.md as items 28-31, with measurements:
- LOCAL_FOCUS_SCORE reaches one tile of four. get_local_focus_score() loops
y < height - M, so at scale=2 exactly one 4x6 tile is visited while the divisor stays
scale^2 = 4. The docs say "the mean and median values of the tiles are returned",
which the code returns neither of.
- POWER_SPECTRUM_SLOPE is pinned at the guard, not the algorithm. rps() returns early
unless floor(min(h,w)/8) >= 3 and the fixture is 8 px wide, so the pinned 0 is the
early return's value. Measured on a synthetic 32x32 ROI where the guard does not fire:
the radius axis is floor(sqrt(fft coefficient))+1 -- the FFT value, not the frequency
radius sqrt(kx^2+ky^2) -- and raw_radii is indexed by that bin inside a loop bounded
by the padded FFT size, 1024 against raw_radii's 32.
- SHARPNESS is not the DOM measure it is a port of (commit 3).
- Both out-of-core paths are uncovered, and get_percent_max_pixels_NT() uses two
independent ifs where the in-RAM path uses `else if`, so they disagree by construction
on a constant ROI -- on which osized_calculate() returns before either runs anyway,
leaving both saturations 0. One input, three answers. The in-RAM behaviour is measured:
a constant 4x4 ROI gives MIN_SATURATION=0, MAX_SATURATION=1 against CellProfiler's
100% for both.
None of the four blocks vetting, which is why this PR carries no src/ change: none sits
between Nyxus and its oracle. The assertions pin current behaviour, and the generator
asserts the tile count, so each fix surfaces as a moved golden rather than as a silent
redefinition.
Two registry corrections the review record asks for. All six rows had `target_test` set to the
same file as `current_test` -- a reorg destination pointing at where the assertion already is.
The family plan's contract is "current_test filled, target_test cleared", and the 2D NGTDM
rows rewritten under review in a747234 have it blank; all six are now cleared. And
LOCAL_FOCUS_SCORE is recorded as a PARTIAL-pipeline oracle: OpenCV supplies the per-tile
statistic, but which tiles and the scale^2 divisor are Nyxus' own definition reproduced in the
generator -- and that tiling is exactly where the family's open defect is. FOCUS_SCORE is not
in that position; cv2 computes the whole of it. The split is recorded the three ways PolusAI#437's
review established for GABOR: a not_covered.md section E row, the Notes column the scanner
emits, and the registry row's notes. The oracle column stays `opencv`, one SPEC 4 token per
row.
scan_imq_coverage.py regenerates audit/imq_coverage.csv from the tests and checks each
registry row against the tests of its OWN kind. It also runs three set differences the 3D
GLDM pass showed are worth having everywhere: every golden-table key must be read by an
assertion in the same file and vice versa (a pinned key nothing reads is where a bad
number lives -- it is how a 3D GLDM golden sat at another feature's value, 353x off);
every test function must be registered in test_all.cc and every test_name must resolve to
a declared case; and the six IMQ registry features must equal what featureset.cpp maps to
FeatureIMQ, compared as SETS in both directions rather than by count, because a count
cannot see a swap. Six negative controls, one per check, all behaved.
Also: benchmarks.md gains bench_imq_quality_roi, recording that the fixture's 18 zero
pixels are a coordinate typo in the literal that three features now depend on, and that
its 8 px width is what leaves POWER_SPECTRUM_SLOPE untested. TOOLS.md gains an opencv row
(no env of its own -- nyxus_mirp already carries cv2 4.13.0), the cv2 gotchas that matter
for a filter oracle, the vendoring note for the DOM reference, and the JVM PATH entries
that turn CellProfiler's bare exit 127 into a working import. not_covered.md records the
IMQ paths no assertion reaches and closes its own note that the two gen_imq_* generators
still carried pasted fixtures.
oracle_coverage.csv has three pointer columns. check_coverage.py validated two of them --
benchmark through validate_benchmarks, test_name through validate_test_names -- and had no
validator for config_recipe at all. A row could therefore name a SPEC 5 recipe that had
never been written and every checker in the tree stayed green, which is exactly what
happened: turning the check on found four dangling ids across three families.
imq.laplacian_ksize1_zeropad 4 rows, defined nowhere
imq.saturation_observed_extremum
glrlm.ibsi_ng128 32 rows -- the id test_2d_glrlm_common.h has named
in a comment since that family was vetted
"Not mode-specific. Any disagreement 38 rows -- a prose sentence in a pointer column
is a definition, coordinate-frame,
spacing, aggregation, or ..."
All four are closed by commit 4, which writes the three real recipes and moves the prose
to notes. validate_config_recipes fails on a fifth. A blank cell stays legal -- a row that
names no recipe claims nothing -- but a cell that names one has to resolve to a heading.
This is the third instance of the same class in this series, after a dangling target_test
and thirty GLCM rows describing a deleted file in `notes`: an unvalidated column is where
stale references live, and the fix is a validator rather than a one-off correction. The
self-test in tests/python/test_vetting_mechanics.py mirrors the two that already exist for
benchmark and test_name, and requires the checker to actually fail on a typo -- a lint that
cannot fail is not a lint. It also covers the file-missing return path, which the two older
self-tests leave untested in their own validators: PolusAI#445's S5 comment is that an unexercised
branch is one a refactor can turn into a silent pass.
The check is ordered after the corrections so every commit in this PR is green on its own.
Verified: check_coverage.py --check clean over all 767 rows; pytest
tests/python/test_vetting_mechanics.py 11 passed.
…ng it Answers comment 1 of the PR PolusAI#457 review. imq_sharpness_reference_dom.py carried six functions transcribed line for line from https://github.com/umang-singhal/pydom into this tree, each mirroring one upstream statement, with the upstream file named at the top and no licence or copyright preserved. That repository is GPL-3.0 and Nyxus is MIT, confirmed against the GitHub licence API rather than assumed from the README. The script now imports the installed package and calls its public API: get_sharpness for the score, and load/edges/sharpness_matrix for the intermediates the report tabulates. It installs from git, which the previous docstring had ruled out on the grounds that no usable PyPI name exists - true, and irrelevant, because upstream ships a setup.py: pip install git+https://github.com/umang-singhal/pydom.git Into the offline audit env only. The reference is not a Nyxus dependency and CI never invokes this script. An absent reference now exits with that install line and the reason it is not vendored, rather than a traceback. No number in the refutation moved. The reference SHARPNESS is 0.5459295115771082 as before, edgex/edgey 56/73, and the masked counts 28/16 - and reference_sharpness() asserts the intermediates recompose get_sharpness() before returning either, so the diagnostics provably describe the run the entry point produced rather than a plausible reconstruction of it. Divergence 5 is now measured rather than asserted. The raw matrices out of sharpness_matrix hold 50 and 28 pixels at or above the threshold and masking drops those to the 28 and 16 the score is built from, which is the difference the report claims. Divergences 2, 4 and 6 stay read off the reference's source - its public API exposes no hook that isolates them - and the script says so instead of implying they were measured. The report's three inlined upstream snippets are replaced by prose; only the Nyxus C++ snippet remains, and the Python port of sharpness.cpp stays, being this project's own MIT code. TOOLS.md is retitled from "vendored, not installed" to "installed from git, never vendored" and carries the general rule: a reference under a copyleft licence is invoked or reimplemented from the publication, never pasted in. imq_golden_regen.md's reproduction block installs it. No feature values change and no source file outside tests/vetting is touched.
… dispositions Answers comment 2 of the PR PolusAI#457 review. matrix/imq.md classified reachable production cells as VALID-but-divergent and NOT PINNED, neither of which is a SPEC 5.1 disposition, and recorded their defects instead of asserting them. Every verdict is now VALID, VALID-BUT-PRODUCTION-ONLY or INVALID, and each VALID-BUT-PRODUCTION-ONLY cell has a regression guard rather than a paragraph. Five cases, each on the smallest input that reaches its cell: MIN/MAX_SATURATION, constant ROI 0 and 1 MIN/MAX_SATURATION, mask narrower 11/16 and 1/16 than the bounding box POWER_SPECTRUM_SLOPE, short side 1.7837481542489078 at least 24 px The saturation bands are an absolute 0: every pin is k/16, exactly representable, so any other value is a change of behaviour and not a float wobble. The cells exist because CellProfiler computes a different quantity on each - 100% for both extrema on a constant ROI, mask-only counting on a narrow mask - so neither can carry an oracle claim, and what stays uncovered is the agreement, not the code path. The 24x24 pin needed measuring rather than assuming. It is the first assertion in the tree to drive PowerSpectrumFeature past its size guard, and power_spectrum_slope() reads raw_radii[i] from a loop bounded by the padded FFT size, so the pin could have been a snapshot of an out-of-bounds read. Temporary instrumentation, reverted: magnitude.size() 1024 against raw_radii.size() 24, largest index reached 3, no out-of-range read, 3 points surviving to the fit. The fixture is a deterministic modular ramp for a second reason - a smooth ramp leaves fewer than two surviving points, power_spectrum_slope() falls through to the same 0 the guard returns, and the pin could no longer tell the two code paths apart. Both facts are recorded at the pin. Two cells the review did not name are brought into the same vocabulary, since the objection applies equally: LOCAL_FOCUS_SCORE at scale != 2 becomes INVALID (calculate() hardcodes 2 and nothing else calls the method), and the out-of-core row becomes VALID-BUT-PRODUCTION-ONLY. That last one is reachable - phase3.cpp calls osized_scan_whole_image() on every registered feature method - and is the one cell whose guard is still outstanding, because reaching osized_calculate() needs an oversized-ROI harness the gtest fixture does not build. It says so in its own row rather than being relabelled into looking closed. scan_imq_coverage.py found two defects in the guards while they were being written, and both fixes are here. It matched golden-table keys to assertions by feature name, which reported every qualified key as unread; it now matches on the key literal the test passes, which is the stronger check anyway, because MIN_SATURATION is pinned three times across the family and a feature-name match collapses all three onto one. SCOPED_TRACE labels are stripped first, being the other SCREAMING_SNAKE literal in these functions. It also showed the power-spectrum case naming its feature inside a wrapped expression, invisible to the coverage artifact; all five cases now name feature and pin key on the assertion line, and assert_imq_regression_on says why. Supporting artifacts: calc_imq_feature_on() in test_imq_common.h so a case can supply its own ROI, two config recipes (imq.saturation_production_only, imq.power_spectrum_past_guard), one benchmark (bench_imq_matrix_cell_rois), three oracle_coverage.csv regression rows, a regenerated imq_coverage.csv, and the scope sections of imq_cellprofiler_vetting_report.md and not_covered.md rewritten - those cells are no longer uncovered. No feature values change and the four IMQ feature sources are byte-identical to the previous round.
…nnot reach Answers comment 3 of the PR PolusAI#457 review. LOCAL_FOCUS_SCORE is a partial-pipeline oracle - cv2 supplies the Laplacian and the population variance, while the tile extraction and the scale^2 divisor are Nyxus' own definition, reproduced in the generator - and SPEC 4 requires such a family to carry a negative control that measures the uncovered gap. gen_imq_opencv.py had EXPECTED_TILES = 1, which only checks that the reproduced tiling still matches the one Nyxus walks and says nothing about how much of the feature that leaves unvetted. all_tiles() walks every scale^2 tile, which is the tiling the /scale^2 divisor already assumes, and the generator compares the two: all 4 tiles 28.341145833333336 pinned, 1 tile 7.5763888888888902 gap 73% Asserted rather than printed, for the reason the control exists: if the gap ever vanished the one-tile pin would not be a partial value and the scope note would be overstating the defect, so that case has to fail too. The shape follows gen_gabor_skimage.py's D2 control, which is the worked example SPEC 4 names for this situation. The 73% is carried into not_covered.md section E and imq_opencv_vetting_report.md, both of which described the split without sizing it. No feature values change and no source file outside tests/vetting is touched.
The out-of-core row was already classified VALID-BUT-PRODUCTION-ONLY with its guard marked outstanding, but only the review thread said why it ships open. @vjaganat90 accepted splitting the repair into its own PR on three conditions: scope that follow-up explicitly, make matrix/imq.md and the PR description equally clear that PolusAI#457 vets the in-RAM paths and deliberately leaves that row open, and link the follow-up once it exists. This is the first two. matrix/imq.md gains a scope paragraph above the table, and the out-of-core section is rewritten from one prose paragraph into one bullet per defect, all of them read off the source and cited by file and line: - PowerSpectrumFeature::osized_calculate() and SharpnessFeature::osized_calculate() are empty {} bodies overriding the base's pure virtual, and FeatureMethod::osized_scan_whole_image() calls the no-op and then save_value() regardless; - FocusScoreFeature::osized_calculate() assigns focus_score_ only, so LOCAL_FOCUS_SCORE is the same case and it is three features rather than two; - none of the six result members has a default initializer, no constructor assigns one, and cleanup_instance() has no override in any of the four classes, so the first oversized ROI publishes an indeterminate double; - the early returns on a constant ROI leak the previous ROI's values, because the base saves regardless and feature methods are singletons nothing resets; - get_percent_max_pixels_NT() uses two independent ifs where the in-RAM path uses else if; - get_focus_score_NT() passes (width, height) where the row count belongs, takes the variance over a buffer larger than any pixel writes, overruns a 900-entry window on a 100x20 ROI, and steps the large-ROI branch wrong twice; - PowerSpectrumFeature::featureset names FOCUS_SCORE where the constructor provides POWER_SPECTRUM_SLOPE. A "what the follow-up carries" subsection scopes that PR: the oversized-ROI harness, one matrix row per feature replacing the family-wide row, and a fix for every defect above -- with the POWER_SPECTRUM_SLOPE radial-binning defect explicitly left out, since that one is in-RAM, already pinned, and needs an oracle rather than a harness. One correction rather than an addition. The file said the constant-ROI case leaves "both saturations at 0". It does not: osized_calculate() early-returns, but the base calls save_value() anyway and nothing resets the singleton between ROIs, so what is published is the previous oversized ROI's values, or an indeterminate double on the first. Same shape as the NGTDMFeature::n_levels static. Corrected here and in not_covered.md, which carried the same sentence. not_covered.md also gains the two rows its IMQ table was missing -- the empty osized_calculate() bodies, and LOCAL_FOCUS_SCORE never being assigned on that path -- and a note that those rows stay open on purpose. No test, no golden and no source file changes.
…ng it Answers comment 1 of the PR #457 review. imq_sharpness_reference_dom.py carried six functions transcribed line for line from https://github.com/umang-singhal/pydom into this tree, each mirroring one upstream statement, with the upstream file named at the top and no licence or copyright preserved. That repository is GPL-3.0 and Nyxus is MIT, confirmed against the GitHub licence API rather than assumed from the README. The script now imports the installed package and calls its public API: get_sharpness for the score, and load/edges/sharpness_matrix for the intermediates the report tabulates. It installs from git, which the previous docstring had ruled out on the grounds that no usable PyPI name exists - true, and irrelevant, because upstream ships a setup.py: pip install git+https://github.com/umang-singhal/pydom.git Into the offline audit env only. The reference is not a Nyxus dependency and CI never invokes this script. An absent reference now exits with that install line and the reason it is not vendored, rather than a traceback. No number in the refutation moved. The reference SHARPNESS is 0.5459295115771082 as before, edgex/edgey 56/73, and the masked counts 28/16 - and reference_sharpness() asserts the intermediates recompose get_sharpness() before returning either, so the diagnostics provably describe the run the entry point produced rather than a plausible reconstruction of it. Divergence 5 is now measured rather than asserted. The raw matrices out of sharpness_matrix hold 50 and 28 pixels at or above the threshold and masking drops those to the 28 and 16 the score is built from, which is the difference the report claims. Divergences 2, 4 and 6 stay read off the reference's source - its public API exposes no hook that isolates them - and the script says so instead of implying they were measured. The report's three inlined upstream snippets are replaced by prose; only the Nyxus C++ snippet remains, and the Python port of sharpness.cpp stays, being this project's own MIT code. TOOLS.md is retitled from "vendored, not installed" to "installed from git, never vendored" and carries the general rule: a reference under a copyleft licence is invoked or reimplemented from the publication, never pasted in. imq_golden_regen.md's reproduction block installs it. No feature values change and no source file outside tests/vetting is touched.
… dispositions Answers comment 2 of the PR #457 review. matrix/imq.md classified reachable production cells as VALID-but-divergent and NOT PINNED, neither of which is a SPEC 5.1 disposition, and recorded their defects instead of asserting them. Every verdict is now VALID, VALID-BUT-PRODUCTION-ONLY or INVALID, and each VALID-BUT-PRODUCTION-ONLY cell has a regression guard rather than a paragraph. Five cases, each on the smallest input that reaches its cell: MIN/MAX_SATURATION, constant ROI 0 and 1 MIN/MAX_SATURATION, mask narrower 11/16 and 1/16 than the bounding box POWER_SPECTRUM_SLOPE, short side 1.7837481542489078 at least 24 px The saturation bands are an absolute 0: every pin is k/16, exactly representable, so any other value is a change of behaviour and not a float wobble. The cells exist because CellProfiler computes a different quantity on each - 100% for both extrema on a constant ROI, mask-only counting on a narrow mask - so neither can carry an oracle claim, and what stays uncovered is the agreement, not the code path. The 24x24 pin needed measuring rather than assuming. It is the first assertion in the tree to drive PowerSpectrumFeature past its size guard, and power_spectrum_slope() reads raw_radii[i] from a loop bounded by the padded FFT size, so the pin could have been a snapshot of an out-of-bounds read. Temporary instrumentation, reverted: magnitude.size() 1024 against raw_radii.size() 24, largest index reached 3, no out-of-range read, 3 points surviving to the fit. The fixture is a deterministic modular ramp for a second reason - a smooth ramp leaves fewer than two surviving points, power_spectrum_slope() falls through to the same 0 the guard returns, and the pin could no longer tell the two code paths apart. Both facts are recorded at the pin. Two cells the review did not name are brought into the same vocabulary, since the objection applies equally: LOCAL_FOCUS_SCORE at scale != 2 becomes INVALID (calculate() hardcodes 2 and nothing else calls the method), and the out-of-core row becomes VALID-BUT-PRODUCTION-ONLY. That last one is reachable - phase3.cpp calls osized_scan_whole_image() on every registered feature method - and is the one cell whose guard is still outstanding, because reaching osized_calculate() needs an oversized-ROI harness the gtest fixture does not build. It says so in its own row rather than being relabelled into looking closed. scan_imq_coverage.py found two defects in the guards while they were being written, and both fixes are here. It matched golden-table keys to assertions by feature name, which reported every qualified key as unread; it now matches on the key literal the test passes, which is the stronger check anyway, because MIN_SATURATION is pinned three times across the family and a feature-name match collapses all three onto one. SCOPED_TRACE labels are stripped first, being the other SCREAMING_SNAKE literal in these functions. It also showed the power-spectrum case naming its feature inside a wrapped expression, invisible to the coverage artifact; all five cases now name feature and pin key on the assertion line, and assert_imq_regression_on says why. Supporting artifacts: calc_imq_feature_on() in test_imq_common.h so a case can supply its own ROI, two config recipes (imq.saturation_production_only, imq.power_spectrum_past_guard), one benchmark (bench_imq_matrix_cell_rois), three oracle_coverage.csv regression rows, a regenerated imq_coverage.csv, and the scope sections of imq_cellprofiler_vetting_report.md and not_covered.md rewritten - those cells are no longer uncovered. No feature values change and the four IMQ feature sources are byte-identical to the previous round.
…nnot reach Answers comment 3 of the PR #457 review. LOCAL_FOCUS_SCORE is a partial-pipeline oracle - cv2 supplies the Laplacian and the population variance, while the tile extraction and the scale^2 divisor are Nyxus' own definition, reproduced in the generator - and SPEC 4 requires such a family to carry a negative control that measures the uncovered gap. gen_imq_opencv.py had EXPECTED_TILES = 1, which only checks that the reproduced tiling still matches the one Nyxus walks and says nothing about how much of the feature that leaves unvetted. all_tiles() walks every scale^2 tile, which is the tiling the /scale^2 divisor already assumes, and the generator compares the two: all 4 tiles 28.341145833333336 pinned, 1 tile 7.5763888888888902 gap 73% Asserted rather than printed, for the reason the control exists: if the gap ever vanished the one-tile pin would not be a partial value and the scope note would be overstating the defect, so that case has to fail too. The shape follows gen_gabor_skimage.py's D2 control, which is the worked example SPEC 4 names for this situation. The 73% is carried into not_covered.md section E and imq_opencv_vetting_report.md, both of which described the split without sizing it. No feature values change and no source file outside tests/vetting is touched.
Last family of the vetting series. Six features, three assertion files plus a shared fixture header,
two oracles and one refuted candidate reference, no
src/change. Rebased ontoupstream/mainat
6f32f96f.The family was already labelled
vettedon four of its six features. Nothing about that label waswrong — but nothing about it had been measured either, and the pass found one mispinned golden,
four source defects and a checker gap that spans three families.
Nine registry rows, eleven gtest cases, two benchmarks, five config recipes.
Scope: this PR vets the in-RAM paths
Every cell
tests/vetting/matrix/imq.mdclassifies VALID or VALID-BUT-PRODUCTION-ONLY is one thatcalculate()reaches. The out-of-core cell is classified and deliberately left unguarded, andthe matrix says so above its own table rather than only in a review thread.
osized_calculate()is reachable production —phase3.cpp:117drives it for any ROI over the RAMlimit, and all four IMQ feature methods are registered in
feature_mgr_init.cpp— so it carries aSPEC §5.1 disposition rather than "not covered". Its regression guard is outstanding, and the row
says that instead of being closed by relabelling.
It is left open because closing it is a source change, not a vetting pass. All four IMQ classes are
defective on that path, each cited by file and line in the matrix:
PowerSpectrumFeature::osized_calculate()andSharpnessFeature::osized_calculate()are empty{}bodies overriding the base's pure virtual, andFeatureMethod::osized_scan_whole_image()calls the no-op and thensave_value()regardless;FocusScoreFeature::osized_calculate()setsfocus_score_only —local_focus_score_isassigned inside
calculate(), on the in-RAM path alone — so it is three features and not two;cleanup_instance()has no override in any of the four classes, so the first oversized ROIpublishes an indeterminate double rather than a zero;
regardless and feature methods are singletons nothing resets — the same shape as the
NGTDMFeature::n_levelsstatic;get_percent_max_pixels_NT()uses two independentifs where the in-RAM path useselse if;get_focus_score_NT()passes(width, height)where the row count belongs, takes the varianceover a buffer larger than the region any pixel writes, overruns a 900-entry window on a 100×20
ROI, and steps the large-ROI branch wrong twice;
PowerSpectrumFeature::featuresetnamesFOCUS_SCOREwhere the constructor providesPOWER_SPECTRUM_SLOPE.Guarding those per feature needs an oversized-ROI harness this family's gtest fixture does not
build, and that harness overlaps the one the 2D out-of-core repair needs on its own branch —
building a second here would leave two to reconcile.
The follow-up carries, together: the oversized-ROI harness; one matrix row per feature in place
of the single family-wide row, each with its own disposition and its own assertion; and a fix for
every defect above. Explicitly not in it: the
POWER_SPECTRUM_SLOPEradial-binning defect, whichis an in-RAM defect the out-of-core path merely inherits, is already pinned by
test_imq_power_spectrum_slope_large_roi_regression, and needs an oracle(
centrosome.radial_power_spectrum.rps) rather than a harness. It will be linked here when itopens.
What was cleaned up
The band was a default parameter. All four oracle assertions called
assert_feature()fromtest_feature_calculation_common.h, whose signature endsdouble frac_tolerance = 1000, and nonepassed a tolerance.
agrees_gtdivides the golden by that argument, so1000isrel=1e-3— whichis where the registry's tolerance column got its value. A tolerance that agrees with a helper's
default is not evidence of a measurement.
Measured against fresh runs of both oracles:
FOCUS_SCORELOCAL_FOCUS_SCOREMIN_SATURATIONMAX_SATURATIONAll four now assert with
ASSERT_NEARat SPEC §7's exact tier — that row of the tolerance table isan absolute 1e-9 band — following
a747234, which made the same correction for 2D NGTDM. Theregistry's tolerance column reads
abs=1e-9.Structure. A
ref_vals_mapper file, a named band constant, and one assert helper with aSCOPED_TRACE, replacing four bare literals in four function bodies — the shapetest_2d_neighbor_cellprofiler.huses. A newtest_imq_common.hcarries the templatedcalc_imq_feature<F>()andcalc_imq_feature_on(), so the three assertion files stop reaching forroi_cache.h,pixel.h,environment.h,test_data.handtest_main_nyxus.hindividually.check_test_names.pygainsTABLE_DIM_AGNOSTIC, the table-level twin of the existingDIM_AGNOSTIClist for files. The 6.3.1 table rule requires a_2d_/_3d_token, andimqcarries neither — its registry dim is
imq, the same word as its family, so the only conformingname it could have had was
imq_imq_opencv_ref_vals.What changed value, and why
MAX_SATURATION:0.16666666666666666→0.16666666666666669. The pin carried Nyxus' own16/96under a comment reading "CellProfiler … = 16/96; tolerance rel=1e-3 (agreement isexact)". Three things were wrong at once: the value was not CellProfiler's, the agreement was not
exact, and a
rel=1e-3band is five orders looser than "exact" would justify. Both tools count thesame 16 of 96 pixels; CellProfiler reports a percentage, so
100.0*16/96divided by 100 lands oneulp above
16/96. A unit conversion, not a disagreement — but the pin of an oracle table shouldcarry the oracle's digits.
The old generator could not have caught it: it compared CellProfiler against a literal
NYXUS = {...}dict in its own source atTOL=1e-6, three orders above the gap.SHARPNESS:2.19047→2.1904708385718963. Re-pinned at full%.17g. The truncated valuesat 3.8e-7 relative from what it was guarding.
No other golden moves, and no
src/file is touched.What the generators do now
Both carried a literal golden dict and a transcribed copy of the fixture, so "ALL CHECKS PASSED"
meant the script agreed with its own source.
tests/vetting/not_covered.mdhad recorded that shapeas still live here after the 2D neighbour pass closed the same thing; this closes it. Both now parse
the fixture out of
test_data.hand the pins out of the header they feed, carry the reverse check(a value the oracle produces that the header pins nothing for), and exit non-zero on any mismatch or
unproducible pin.
The pins are re-verified at a round-trip band of 0, not at the band the C++ asserts at. They are
the tool's own digits at
%.17g, so a 17-significant-digit literal parses back to the same doubleand must round-trip exactly; a non-zero residual means the tool's output moved. Both first shipped
this as the assertion's
1e-9— the conflation #443's review madegen_ngtdm_mirp.pystop doing,where a hand-edited golden could drift nine orders and still be reported as verified. Both also
print the versions they actually ran under rather than the literals in their docstrings, per
#448's review.
Three checks worth calling out:
gen_imq_opencv.pyasserts cv2's filtered image equals Nyxus' hand-rolledlaplacian()cellfor cell (
max abs diff 0.0) before comparing any scalar. A variance is a scalar and manydifferent filtered images share one, so a matching variance alone is weak evidence that two
implementations filter alike. With the convolution settled, the 7.1e-15 residual means only what
it looks like it means.
gen_imq_opencv.pyalso carries the SPEC §4 negative control forLOCAL_FOCUS_SCORE— see thenext section.
gen_imq_cellprofiler.pyfails ifFOCUS_SCOREorLOCAL_FOCUS_SCOREappear in its table.CellProfiler publishes features by those names, and they are a different statistic — normalized
variance of the raw image, not variance of the Laplacian — so pointing the CP oracle at the names
it shares would credit a vetting that does not exist.
One of the two oracle claims is narrower than it looks, and the gap is now sized
FOCUS_SCOREis wholly OpenCV's — cv2 computes the filter and the population variance.LOCAL_FOCUS_SCOREis not: cv2 supplies the per-tile statistic, but which tiles and thescale²divisor are Nyxus' own definition, reproduced in the generator — and that tiling is exactly where
the family's open defect is. Same shape #437's review made GABOR narrow, and it is recorded the same
three ways: a row in
not_covered.md§E, theNotescolumnscan_imq_coverage.pyemits, and theregistry row's
notes. Theoraclecolumn staysopencv, one SPEC §4 token per row.all_tiles()is the negative control SPEC §4 requires of a partial-pipeline oracle. It walks everyscale²tile — the tiling the/scale²divisor already assumes — and the generator asserts thegap rather than printing it:
Asserted, not printed, for the reason the control exists: if the gap ever vanished, the one-tile pin
would not be a partial value and the scope note would be overstating the defect, so that case has to
fail too. The shape follows
gen_gabor_skimage.py's D2 control, the worked example SPEC §4 names.Nothing external would close it — no tool implements Nyxus' tiling, because it is not a published
convention. What closes it is fixing the loop bound, at which point the divisor and the tile set
agree and the whole quantity is
mean(var(Laplacian(tile))), which cv2 does compute.What was measured and refuted
SHARPNESS' row readcandidate_oracle = "reference DOM sharpness (Kumar et al. 2012)",flag = promote-after-deepdive. The deep dive happened: Nyxus returns 2.1904708385718963 wherethe reference returns 0.54592951157710823 on the same fixture. Structural, not numerical — six
differences, of which the first decides the rest: the reference counts pixels above
sharpness_threshold=2; Nyxus sums the sharpness values and has no such parameter. Thereference's
Rxis a fraction of edge pixels, bounded by 1, which is what makes the paper's0 < S < sqrt(2)bound hold; Nyxus' 2.19 is already abovesqrt(2). The other five (a row-wiseSywhere the reference is column-wise, swapped edge axes — measured 73/56 against 56/73, the samenumbers exchanged — a shared normalization maximum, no final edge masking, and two unwritten
columns) are in
audit/imq_pydom_sharpness_vetting_report.md.The reference is invoked, never vendored. pydom is GPL-3.0 and this repository is MIT, so
tests/vetting/audit/imq_sharpness_reference_dom.pyimports the installed package and calls itspublic API —
get_sharpnessfor the score,load/edges/sharpness_matrixfor the intermediatesthe report tabulates — and
reference_sharpness()asserts the intermediates recompose the scorebefore returning either, so the diagnostics describe the run the entry point produced. It installs
into the offline audit env only and CI never invokes it:
What the script does carry is a port of Nyxus' own MIT-licensed
sharpness.cpp, and it hasexactly two checks that can fail: the port reproduces the pinned C++ golden to 2.03e-16 — which is
what makes the six differences a claim about the shipped C++ rather than about a Python script — and
the reference still disagrees. If that second one ever fails,
SHARPNESSbecame promotable, whichis the outcome the report is waiting for.
Row is now
candidate_oracle = "… — measured and refuted",flag = impl-defect. Status staysregression.The config matrix, and the cells no oracle reaches
matrix/imq.md's headline: the family has no configuration axes. None of the four featureclasses reads a
NyxSetting, so every knob is a compile-time default on a private static method,and a recipe here names a fixture rather than a settings bundle. The cells are therefore separated
by input, not by settings.
Every verdict is one of SPEC §5.1's three dispositions, and each
VALID-BUT-PRODUCTION-ONLY cell that
calculate()reaches carries a regression guard rather than aparagraph — five gtest cases, each on the smallest input that reaches its cell:
MIN/MAX_SATURATION, constant ROI (min == max)test_imq_{min,max}_saturation_constant_roi_regression— 0 and 1MIN/MAX_SATURATION, mask narrower than the AABBtest_imq_{min,max}_saturation_narrow_mask_regression— 11/16 and 1/16POWER_SPECTRUM_SLOPE, short side ≥ 24 pxtest_imq_power_spectrum_slope_large_roi_regression— 1.7837481542489078LOCAL_FOCUS_SCORE,scale ≠ 2calculate()hardcodes 2 and nothing else calls the methodThe saturation bands are an absolute 0: every pin is
k/16, exactly representable, so any othervalue is a change of behaviour and not a float wobble. Those cells exist because CellProfiler
computes a different quantity on each — 100% for both extrema on a constant ROI, mask-only counting
on a narrow mask — so neither can carry an oracle claim, and what stays uncovered is the
agreement, not the code path.
The 24×24 pin needed measuring rather than assuming. It is the first assertion in the tree to
drive
PowerSpectrumFeaturepast its size guard, andpower_spectrum_slope()readsraw_radii[i]from a loop bounded by the padded FFT size — so the pin could have been a snapshot of an
out-of-bounds read. Temporary instrumentation, reverted:
Largest index 3 against a 24-element vector, so the read stays in range and the pinned value is
defined. ASan+UBSan on that path agrees — 0 diagnostics. The index remains unbounded in the code;
this shows only that a 24×24 modular ramp keeps it at 3 of 24. The fixture is a deterministic
modular ramp for a second reason: a smooth ramp leaves fewer than two surviving points,
power_spectrum_slope()falls through to the same0the guard returns, and the pin could nolonger tell the two code paths apart.
What new audit artifacts were added
IMQ had none. This adds the set every other family carries:
matrix/imq.md— the config matrix, above.audit/imq_coverage.csv+audit/scan_imq_coverage.pyaudit/imq_opencv_vetting_report.md,audit/imq_cellprofiler_vetting_report.md,audit/imq_pydom_sharpness_vetting_report.mdaudit/imq_golden_regen.mdbenchmarks.mdgainsbench_imq_quality_roi, recording that the fixture's 18 zero pixels area coordinate typo in the literal (rows y=7..9 repeat rows 1..3's coordinates for x=3..8) that
three features now depend on, and that its 8 px width is what leaves
POWER_SPECTRUM_SLOPEpinnedat its guard; and
bench_imq_matrix_cell_rois, the three probe ROIs for the cells thatfixture cannot reach — a constant 4×4, a 4×4 whose mask covers 5 of the 16 pixels in its bounding
box, and a 24×24 modular ramp.
TOOLS.mdgains anopencvrow — no env of its own,nyxus_mirpalready carries cv2 4.13.0 —the cv2 gotchas that matter for a filter oracle (
ksize=1andksize=3are different filters;borderTypeis part of the recipe; compare the filtered image), the JVMPATHentries thatturn CellProfiler's bare exit 127 into a working import, and the general rule the DOM
reference produced: a reference under a copyleft licence is invoked or reimplemented from the
publication, never pasted in.
scan_imq_coverage.pyruns three set differences beyond the row mapping, all cheap and all liftedfrom the 3D GLDM pass: table keys against the assertions that read them (a pinned key nothing reads
is where a bad number lives — it is how a 3D GLDM golden sat at another feature's value, 353× off);
test functions against
TEST()registrations, both directions; and the six registry featuresagainst
featureset.cpp'sFeatureIMQmap, compared as sets in both directions, because acount cannot see a swap. It matches pins to assertions on the key literal the test passes rather
than on the feature name — the stronger check, and the necessary one now that
MIN_SATURATIONispinned three times across the family.
Four defects found, none of them blocking
Recorded with measurements in
tests/vetting/matrix/imq.md, and each carried into thedeferred-defect list this series keeps for behaviour changes. None sits between Nyxus and its
oracle, which is why this PR carries no
src/change; the assertions pin current behaviour soeach fix surfaces as a moved golden. Each needs its own branch, since every one of them changes a
public feature value.
LOCAL_FOCUS_SCOREreaches one tile of four.get_local_focus_score()loopsy < height - MwithM = height/scale, so atscale=2exactly one 4×6 tile is visited whilethe divisor stays
scale² = 4. The docs say "the mean and median values of the tiles arereturned", which the code returns neither of. The generator asserts the tile count is 1, so
fixing the bound fails there rather than silently redefining the golden — and the negative
control above sizes what that costs: 73%.
POWER_SPECTRUM_SLOPEis pinned twice, at the guard and at the algorithm behind it.rps()returns early unless
floor(min(h,w)/8) >= 3and the 8 px fixture never clears it, so that pinis the guard's return value and its band is
abs=0. Past the guard, measured on the committed24×24 ROI: the radius axis is
floor(sqrt(fft coefficient))+1— the FFT value, not thefrequency radius
sqrt(kx²+ky²)— andraw_radiiis indexed by that bin inside a loop boundedby the padded FFT size, 1024 against
raw_radii's 24.SHARPNESSis not the DOM measure (above).Two more that are latent rather than live:
static int ksizeis declared in all four IMQ featureheaders under a "User interface" comment and is defined nowhere and referenced nowhere (a
translation unit that used it would fail to link); and
FocusScoreFeature::kernel[9]is a mutablestatic the
ksize != 1branch overwrites in place and never restores — unreachable today becausecalculate()hardcodesksize=1, the same shape as theNGTDMFeature::n_levelsstatic the 2DNGTDM pass fixed.
The checker gap, and why it is in this PR
oracle_coverage.csvhas three pointer columns.check_coverage.pyvalidatedbenchmarkandtest_nameand had no validator forconfig_recipeat all, so a row could name a SPEC §5recipe that had never been written and every checker stayed green. Turning the check on found four
dangling ids across three families:
imq.laplacian_ksize1_zeropadimq.saturation_observed_extremumglrlm.ibsi_ng128test_2d_glrlm_common.h's comment since that family was vetted; section never writtenAll four are closed here — the three real recipes are written, the prose moves to
noteswhere itwas always a note — and
validate_config_recipesfails on a fifth. This is the third instance ofthe same class in the series, after a dangling
target_testand thirty GLCM rows describing adeleted file in
notes: an unvalidated column is where stale references live, and the fix is avalidator rather than a one-off correction.
mainindependently fixed four more of those prose rows while this branch was open — the radial andzernike ones now name
radial.shape2d_nativeandzernike.shape2d_native— which is why the countabove is 34 and not the 38 the branch started with.
The 32 GLRLM rows and the 34 prose rows are the only places this PR touches another family's data.
The
glrlm.ibsi_ng128section is transcribed fromtest_2d_glrlm_common.h,audit/glrlm_2d_pyradiomics_vetting_report.mdandaudit/glrlm_2d_ibsi_vetting_report.md; no GLRLMvalue or verdict changes.
How it was verified
Not "tests pass" — each gate below was run on this branch, at this base:
USE_GPU— the skip isTEST_2D_GABOR_GPU_RUNS_MECHANICS. The base's own count at6f32f96fis 879 (891TEST()blocks less the 13 behindUSE_ARROW/DICOM_SUPPORT/OMEZARR_SUPPORT, plus the one intest_3d_coverage_common.h); this pass adds exactly the five cell guards and removes none, and none of the four IMQ headers contains aTEST(at alltest_all.ccand the four IMQ feature sources before the build and again after the run — the last four are how the no-src/-change claim is made positively, since the synced tree is not a git checkout, and they are byte-identical to the ones the pre-review gate pinned. It assertsCMAKE_HOME_DIRECTORYis its own tree and printshostname, so a log copied between clones cannot vouch for itpytest tests/python/@pytest.mark.arrowfailing only because this build isALLEXTRAS=OFF. 89 is 88 + the one newcheck_coverageself-testgen_imq_opencv.py,gen_imq_cellprofiler.pyandimq_sharpness_reference_dom.pyall exit 0 against the tree in this PR — every pin re-verified, none unproducible, reverse check clean, negative control assertedcheck_test_names.py --check,check_coverage.py --check,scan_imq_coverage.py --checkall cleanEXPECT_TOTAL_CASESin the ASan gate was re-derived after the rebase, not carried forward. Thepre-rebase 882 was 793
TEST_NYXUScases plus 89 from six parameterized 3D coverage sweepsmainhas since retired into individually named cases. Derivation on this tree: 896
TEST(blocks atcolumn 0 in
test_all.cc, 13 of them behindUSE_ARROW/DICOM_SUPPORT/OMEZARR_SUPPORTwhich thegate's configure line turns off, plus the one case in
test_3d_coverage_common.h— 884. The samearithmetic reproduces the previous round's measured 793 exactly (805 − 13 + 1), and the run
reported 884, so it is a derivation that was then checked rather than a number that was fitted.
Negative controls, twenty-one in the original pass, all behaved. Seven on the assertions: each
band rejects a perturbation just outside it (+2e-9 on the three oracle pins with room, 1e-12 on
POWER_SPECTRUM_SLOPE's exact band, +5e-9 onSHARPNESS), a deleted table key is reported by namerather than compared against a default-inserted 0, and the unperturbed tree passes. Six on the
scanner: a pinned key nothing reads, an assertion with nothing pinned for it, an unregistered test
function, a row naming the wrong oracle, a feature the enum publishes that no row covers, and a
test_nameno gtest case answers to — each named by the checker, with the clean tree passing. Eighton the generators: a pin drifted by one ulp, a pin they cannot produce, a value the oracle produces
that nothing pins, the tile count changing under the golden, a CellProfiler table pinning a name CP
shares but does not vet, and all three scripts exiting 0 unperturbed. The review round added the
all_tiles()gap assertion, which is a negative control that ships rather than one that was runonce.
The finished branch was then re-audited against the framework protocol and against the review
comments this series has accumulated — #437, #438, #439, #440, #443, #445 and #448. Eight
findings; seven fixed, one (an
_invariantfile) weighed and declined with the reason. The two thatmattered are in this body: the generators' pin round-trip band, and
LOCAL_FOCUS_SCORE'spartial-pipeline oracle.
Review round
@vjaganat90's three inline comments and the out-of-core blocker are all answered, in that order:
functions had been transcribed line for line from pydom with no licence or copyright preserved.
The licence was confirmed against the GitHub licence API rather than assumed. De-vendored by
invoking upstream, the first remedy the comment named; no number in the refutation moved, and
divergence 5 got stronger out of it, since the raw
sharpness_matrixcounts are now printedbeside the masked ones.
three §5.1 tokens and each reachable cell has a guard — the five cases in the matrix table above,
plus two cells the comment did not name brought into the same vocabulary.
all_tiles(), asserted, sizingthe gap at 73%.
than stated — the empty bodies publish an indeterminate double rather than skipping, and
LOCAL_FOCUS_SCOREis in the same position, so it is three features and not two. The commentalso caught an error of mine, since corrected in the matrix and in
not_covered.md: theconstant-ROI case does not leave both saturations at 0; the base saves regardless and nothing
resets the singleton. Split into a follow-up by agreement, scoped in the Scope section above.
Types reaching the changelog if this is merged as a merge-commit rather than squashed: seven
test, onechoreand onedocs, all configured hidden. Nothing lands in the CHANGELOG and noversion moves either way, which is correct for a test-and-tooling PR.