From bbd6bd20af10ea3c78fddeed02dd6946fa00c5d9 Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Thu, 23 Jul 2026 10:40:06 +0200 Subject: [PATCH 001/282] MILAB-6496: lower per-sample mitool resource defaults MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The 64 GiB memory floor was applied on every run regardless of input size, and mitool's memory-from-limits launcher sizes the JVM -Xms at 50% of the grant — a ~32 GiB initial heap even on tiny datasets, which swaps on typical desktop RAM and stalls the "parsing reads" step. Lower the floor to 16 GiB; the size(reads)*4 term (cap 256 GiB) still scales large inputs up, so only small runs change. Also lower the CPU default from 16 to 8: 16 exceeded the core count on typical desktop machines. Both defaults now match peptide-extraction's mitool steps. --- .changeset/lower-mitool-mem-floor.md | 6 ++++++ workflow/src/main.tpl.tengo | 16 +++++++++++----- 2 files changed, 17 insertions(+), 5 deletions(-) create mode 100644 .changeset/lower-mitool-mem-floor.md diff --git a/.changeset/lower-mitool-mem-floor.md b/.changeset/lower-mitool-mem-floor.md new file mode 100644 index 0000000..cd165b9 --- /dev/null +++ b/.changeset/lower-mitool-mem-floor.md @@ -0,0 +1,6 @@ +--- +'@platforma-open/milaboratories.feature-integration.workflow': patch +'@platforma-open/milaboratories.feature-integration.ui': patch +--- + +Lower the per-sample mitool memory floor from 64 GiB to 16 GiB. The 64 GiB floor was applied on every run regardless of input size, and mitool's memory-from-limits launcher turns the grant into a JVM with `-Xms` = 50% of it — a ~32 GiB initial heap even for tiny datasets, which swaps on typical desktop RAM and stalls the "parsing reads" step. The `size("reads")*4` term still scales large inputs up (cap 256 GiB), so only small runs are affected. Also lower the per-sample mitool CPU default from 16 to 8 (matching peptide-extraction; 16 exceeded the core count on typical desktop machines) and fix the "mitool CPUs per sample" tooltip, which stated the default was 4. diff --git a/workflow/src/main.tpl.tengo b/workflow/src/main.tpl.tengo index fc19210..471a0d6 100644 --- a/workflow/src/main.tpl.tengo +++ b/workflow/src/main.tpl.tengo @@ -27,11 +27,17 @@ featurePropsTpl := assets.importTemplate(":fb-feature-properties") featurePropsSw := assets.importSoftware("@platforma-open/milaboratories.feature-integration.per-cell-metrics:feature-properties") -// Resource defaults mirror the MiXCR blocks (mixcr-analyze): 16 CPUs and a 64 GiB base per mitool -// process. FI's parse/refine are as compute- and RAM-heavy as mixcr alignment, so err large by -// default; users can still override via perProcessCPUs / perProcessMemGB (Advanced Settings). -defaultMitoolMemGB := 64 -defaultMitoolCPUs := 16 +// Per-sample mitool resource defaults. The parse/refine formulas (fb-parse / fb-refine) add +// size("reads")*4 on top of this base and cap at 256 GiB, so the base only sets the floor for small +// inputs while large runs still scale up on their own. Feature-barcode parse/refine are far lighter +// than the full VDJ alignment the mixcr-analyze formula was tuned for, so the floor is 16 GiB (cf. +// peptide-extraction's 32 GiB fixed mitool default). A 64 GiB floor made mitool's memory-from-limits +// launcher pick a ~32 GiB JVM -Xms even on tiny datasets, which swaps on typical desktop RAM and +// stalls "parsing reads". CPUs default to 8 (matching peptide-extraction's mitool steps); 16 exceeded +// the core count on typical desktop machines. Users can still override both via perProcessCPUs / +// perProcessMemGB (Advanced Settings). +defaultMitoolMemGB := 16 +defaultMitoolCPUs := 8 // Staging imports + exports the CSV to drive its upload before production needs it, and emits the // feature-name list for the control dropdown (see prerun.tpl). From 3e6be9e29b236e9b077e92c54d040604b0ef7007 Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Thu, 23 Jul 2026 10:44:32 +0200 Subject: [PATCH 002/282] MILAB-6496: clarify Settings tooltips Lead each optional field with when to use it (or to leave it blank), and tighten the control, sample-column, off-target, combine-mode, dominance, and min-UMI descriptions. --- .changeset/clarify-settings-tooltips.md | 5 ++ ui/src/pages/MainPage.vue | 81 +++++++++++++------------ 2 files changed, 46 insertions(+), 40 deletions(-) create mode 100644 .changeset/clarify-settings-tooltips.md diff --git a/.changeset/clarify-settings-tooltips.md b/.changeset/clarify-settings-tooltips.md new file mode 100644 index 0000000..5fed0ff --- /dev/null +++ b/.changeset/clarify-settings-tooltips.md @@ -0,0 +1,5 @@ +--- +'@platforma-open/milaboratories.feature-integration.ui': patch +--- + +Clarify Settings tooltips. Each optional field now leads with when to use it (or to leave it blank), and the control, sample-column, off-target, combine-mode, dominance, and min-UMI descriptions are tightened for readability. diff --git a/ui/src/pages/MainPage.vue b/ui/src/pages/MainPage.vue index ea5dd00..d794c5f 100644 --- a/ui/src/pages/MainPage.vue +++ b/ui/src/pages/MainPage.vue @@ -460,9 +460,9 @@ const gridOptions = { :style="{ flex: 1 }" > @@ -476,28 +476,13 @@ const gridOptions = { @update:model-value="setSampleColumn" > - - - + + + {{ combineColumnError }} @@ -547,10 +547,9 @@ const gridOptions = { label="Dominance threshold" > From 4c317f45303c2c03a0ede590df6434f36d96900d Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Thu, 23 Jul 2026 10:55:20 +0200 Subject: [PATCH 003/282] MILAB-6496: fix block changelog pointer to the generated block changelog MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit block.meta.changelog pointed at file:../CHANGELOG.md — the repo-root "Initial release" stub — so every published block-pack shipped the 1.0.0 stub and the desktop update view showed no release notes. Point it at file:./CHANGELOG.md, the changesets-generated block/CHANGELOG.md. Verified: the regenerated block-pack now carries the full 2.2.0 changelog. --- .changeset/fix-changelog-pointer.md | 5 +++++ block/package.json | 2 +- 2 files changed, 6 insertions(+), 1 deletion(-) create mode 100644 .changeset/fix-changelog-pointer.md diff --git a/.changeset/fix-changelog-pointer.md b/.changeset/fix-changelog-pointer.md new file mode 100644 index 0000000..81a1855 --- /dev/null +++ b/.changeset/fix-changelog-pointer.md @@ -0,0 +1,5 @@ +--- +'@platforma-open/milaboratories.feature-integration': patch +--- + +Fix the block changelog pointer. `block.meta.changelog` pointed at `file:../CHANGELOG.md` (the repo-root "Initial release" stub), so every published block-pack shipped the 1.0.0 stub and the desktop update view showed no release notes. Point it at `file:./CHANGELOG.md` — the changesets-generated block changelog. diff --git a/block/package.json b/block/package.json index d78915f..473ce19 100644 --- a/block/package.json +++ b/block/package.json @@ -48,7 +48,7 @@ "support": "mailto:support@milaboratories.com", "description": "Assign antigens to single cells from BEAM / LIBRA-seq style feature-barcode reads", "longDescription": "file:../docs/description.md", - "changelog": "file:../CHANGELOG.md", + "changelog": "file:./CHANGELOG.md", "tags": [ "single-cell", "vdj", From 31a64e949a14ee7c09cc66020f279d7d464dda3f Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Thu, 23 Jul 2026 16:05:01 +0200 Subject: [PATCH 004/282] MILAB-6496: retire 'decoy' term, fix stale off-target case tooltip MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Drop the retired 'Decoy' term from the off-target property/values tooltips (the field term is 'off-target antigen'). Correct the off-target-values tooltip: value matching trims surrounding spaces but is case-sensitive — it previously said matching ignores case, contradicting the shipped behaviour. --- .changeset/offtarget-tooltip-nomenclature.md | 5 +++++ ui/src/pages/MainPage.vue | 12 ++++++------ 2 files changed, 11 insertions(+), 6 deletions(-) create mode 100644 .changeset/offtarget-tooltip-nomenclature.md diff --git a/.changeset/offtarget-tooltip-nomenclature.md b/.changeset/offtarget-tooltip-nomenclature.md new file mode 100644 index 0000000..a2e1f46 --- /dev/null +++ b/.changeset/offtarget-tooltip-nomenclature.md @@ -0,0 +1,5 @@ +--- +'@platforma-open/milaboratories.feature-integration.ui': patch +--- + +Clarify off-target tooltips. The off-target property/values tooltips drop the retired "Decoy" term (in favour of "off-target antigen"), and the off-target-values tooltip now correctly states that value matching trims surrounding spaces but is case-sensitive — it previously claimed matching ignores case, which contradicted the shipped behaviour. diff --git a/ui/src/pages/MainPage.vue b/ui/src/pages/MainPage.vue index d794c5f..fc2bed7 100644 --- a/ui/src/pages/MainPage.vue +++ b/ui/src/pages/MainPage.vue @@ -493,10 +493,10 @@ const gridOptions = { @update:model-value="setOfftargetProperty" > @@ -510,8 +510,8 @@ const gridOptions = { > Date: Thu, 23 Jul 2026 16:21:24 +0200 Subject: [PATCH 005/282] MILAB-6496: mark the negative control on the feature axis Surface the chosen control feature as a dedicated hidden per-feature marker (pl7.app/feature/negativeControl, value "true") keyed on the shared feature axis, so VDJ Multiomic Integration can remove the control from its antigen metrics. emit_feature_properties.py writes the marker CSV; fb-feature-properties imports it as a hidden, non-filter column alongside the generic properties. No user-facing change. --- .changeset/emit-negative-control-marker.md | 6 +++ .../src/emit_feature_properties.py | 19 ++++++++ .../test/test_emit_feature_properties.py | 18 ++++++++ workflow/src/column-specs.lib.tengo | 25 ++++++++++- workflow/src/fb-feature-properties.tpl.tengo | 43 +++++++++++++------ workflow/src/main.tpl.tengo | 6 +++ 6 files changed, 104 insertions(+), 13 deletions(-) create mode 100644 .changeset/emit-negative-control-marker.md diff --git a/.changeset/emit-negative-control-marker.md b/.changeset/emit-negative-control-marker.md new file mode 100644 index 0000000..a4904bc --- /dev/null +++ b/.changeset/emit-negative-control-marker.md @@ -0,0 +1,6 @@ +--- +'@platforma-open/milaboratories.feature-integration.workflow': patch +'@platforma-open/milaboratories.feature-integration.per-cell-metrics': patch +--- + +Emit the negative control on the feature axis. The chosen control feature is now surfaced as a dedicated hidden per-feature marker (`pl7.app/feature/negativeControl`), so VDJ Multiomic Integration can remove the control from its antigen metrics (restriction index, antigen breadth, per-antigen fraction columns, and the dominant call). No user-facing change — the marker is hidden and is not offered as a per-feature property. diff --git a/software/per-cell-metrics/src/emit_feature_properties.py b/software/per-cell-metrics/src/emit_feature_properties.py index 53816c7..0cf943e 100644 --- a/software/per-cell-metrics/src/emit_feature_properties.py +++ b/software/per-cell-metrics/src/emit_feature_properties.py @@ -103,6 +103,12 @@ def main() -> None: default="", help="optional sample column for sample-aware mapping (a role column, excluded from properties)", ) + p.add_argument( + "--control-feature", + default="", + help="the negative-control feature name (from the block's control-feature dropdown). Emitted as a " + "dedicated per-feature marker so downstream can remove the control from its antigen metrics.", + ) p.add_argument("--output-prefix", default="result") args = p.parse_args() @@ -145,6 +151,19 @@ def main() -> None: with open(f"{args.output_prefix}_feature_property_meta.json", "w") as out: json.dump(meta, out) + # Negative-control marker (control-aware metrics). Emit the block's chosen control feature as a + # dedicated per-feature (feature,value) CSV with value "true", so the workflow surfaces it as a + # pl7.app/feature/negativeControl column keyed on the feature axis and VDJ Multiomic Integration + # removes the control ENTIRELY from its antigen metrics (restriction index, breadth, per-antigen + # fractions, dominant call) -- unlike an off-target, which stays in the metrics. Header-only when no + # control is designated. The name is emitted verbatim (trimmed); it is one of the panel's features. + control = args.control_feature.strip() + with open(f"{args.output_prefix}_negative_control.csv", "w", newline="") as out: + w = csv.writer(out) + w.writerow(["feature", "value"]) + if control: + w.writerow([control, "true"]) + if __name__ == "__main__": main() diff --git a/software/per-cell-metrics/test/test_emit_feature_properties.py b/software/per-cell-metrics/test/test_emit_feature_properties.py index b93203b..1724654 100644 --- a/software/per-cell-metrics/test/test_emit_feature_properties.py +++ b/software/per-cell-metrics/test/test_emit_feature_properties.py @@ -104,6 +104,24 @@ def test_missing_role_column_errors(tmp_path): assert r.returncode != 0 +def _control_rows(tmp_path): + with open(tmp_path / "r_negative_control.csv", newline="") as fh: + return list(csv.reader(fh)) + + +def test_control_feature_marker_emitted(tmp_path): + # --control-feature marks that feature "true" in the dedicated negative-control marker CSV, so the + # workflow surfaces it on the feature axis for VDJ Multiomic Integration to exclude from its metrics. + _run(tmp_path, "tag,feature\nAAAA,AGX\nGGGG,CTRL\n", "--control-feature", "CTRL") + assert _control_rows(tmp_path) == [["feature", "value"], ["CTRL", "true"]] + + +def test_no_control_feature_marker_header_only(tmp_path): + # No control designated -> marker CSV is header-only (no feature marked as the control). + _run(tmp_path, "tag,feature\nAAAA,AGX\nGGGG,BGX\n") + assert _control_rows(tmp_path) == [["feature", "value"]] + + def _rows(text): reader = csv.reader(io.StringIO(text)) header = next(reader) diff --git a/workflow/src/column-specs.lib.tengo b/workflow/src/column-specs.lib.tengo index d7e763d..16bbbdc 100644 --- a/workflow/src/column-specs.lib.tengo +++ b/workflow/src/column-specs.lib.tengo @@ -354,6 +354,28 @@ qcSummaryColumnsSpec := func(sampleAxisSpec) { } } +// negativeControlColumn: the xsv.importFile `columns` entry for the dedicated negative-control marker CSV +// (emit_feature_properties.py `_negative_control.csv`, a feature/value CSV with value "true" for +// the control). One String p-column (pl7.app/feature/negativeControl) keyed on the shared feature axis +// (the caller supplies `axes` = [featureAxis]). Hidden and NOT a discrete filter: it is not a user-facing +// property — it exists only so the control rides pl7.app/feature/featureId into VDJ Multiomic Integration, +// which removes the control from its antigen metrics (control-aware restriction index / breadth / fraction +// columns / dominant call). Distinct from the generic pl7.app/feature/property import (A-0026). +negativeControlColumn := func() { + return { + column: "value", + id: "negativeControl", + spec: { + name: "pl7.app/feature/negativeControl", + valueType: "String", + annotations: { + "pl7.app/label": "Negative control", + "pl7.app/table/visibility": "hidden" + } + } + } +} + export { cellAxis: cellAxis, featureAxis: featureAxis, @@ -361,5 +383,6 @@ export { perCellSummaryOutput: perCellSummaryOutput, qcFileMapOutput: qcFileMapOutput, qcSummaryColumnsSpec: qcSummaryColumnsSpec, - featurePropertyImportColumns: featurePropertyImportColumns + featurePropertyImportColumns: featurePropertyImportColumns, + negativeControlColumn: negativeControlColumn } diff --git a/workflow/src/fb-feature-properties.tpl.tengo b/workflow/src/fb-feature-properties.tpl.tengo index 6b0ade1..a4d33b4 100644 --- a/workflow/src/fb-feature-properties.tpl.tengo +++ b/workflow/src/fb-feature-properties.tpl.tengo @@ -34,12 +34,15 @@ self.body(func(inputs) { meta := json.decode(string(inputs.propertyMeta.getData())) propertyColumns := is_undefined(meta.columns) ? [] : meta.columns valuesByColumn := is_undefined(meta.valuesByColumn) ? {} : meta.valuesByColumn + // Negative-control marker (control-aware metrics): emit a dedicated hidden per-feature column marking + // the control feature, only when a control is designated (main.tpl gates this). + hasControl := !is_undefined(inputs.hasControl) && inputs.hasControl fb := pframes.pFrameBuilder() - // No extra columns -> no properties to import; publish an empty frame. Skipping the xsv import here is - // required (an empty `columns` list fails the import spec validation) and correct (nothing to key). - if len(propertyColumns) > 0 { + // Nothing to emit unless there are extra property columns (A-0026) and/or a negative-control marker. + // Publish an empty frame otherwise (an empty xsv `columns` list fails the import spec validation). + if len(propertyColumns) > 0 || hasControl { // Same trace step as the per-cell contract columns (deterministic; seeded from the input FASTQ // spec + blockId), so the properties carry consistent provenance / labels. trace := pSpec.makeTrace(inputs.traceSeedSpec, { @@ -50,16 +53,32 @@ self.body(func(inputs) { }) featureAxis := columnSpecs.featureAxis(blockId) - importColumns := columnSpecs.featurePropertyImportColumns(propertyColumns, valuesByColumn) - rawPf := xsv.importFile(inputs.propertiesFile, "csv", { - axes: [{ column: "feature", spec: featureAxis }], - columns: importColumns, - storageFormat: "Parquet", - partitionKeyLength: 0 - }, { splitDataAndSpec: true }) - for k, v in rawPf { - fb.add(k, trace.inject(v.spec), v.data) + if len(propertyColumns) > 0 { + importColumns := columnSpecs.featurePropertyImportColumns(propertyColumns, valuesByColumn) + rawPf := xsv.importFile(inputs.propertiesFile, "csv", { + axes: [{ column: "feature", spec: featureAxis }], + columns: importColumns, + storageFormat: "Parquet", + partitionKeyLength: 0 + }, { splitDataAndSpec: true }) + for k, v in rawPf { + fb.add(k, trace.inject(v.spec), v.data) + } + } + + // Dedicated negative-control marker column (hidden, not a filter). Rides pl7.app/feature/featureId + // into VDJ Multiomic Integration, which removes the control from its antigen metrics. + if hasControl { + ctrlPf := xsv.importFile(inputs.negativeControlFile, "csv", { + axes: [{ column: "feature", spec: featureAxis }], + columns: [columnSpecs.negativeControlColumn()], + storageFormat: "Parquet", + partitionKeyLength: 0 + }, { splitDataAndSpec: true }) + for k, v in ctrlPf { + fb.add(k, trace.inject(v.spec), v.data) + } } } diff --git a/workflow/src/main.tpl.tengo b/workflow/src/main.tpl.tengo index 471a0d6..710258a 100644 --- a/workflow/src/main.tpl.tengo +++ b/workflow/src/main.tpl.tengo @@ -323,13 +323,19 @@ wf.body(func(args) { arg("--csv-barcode-col").arg(args.barcodeSeqColumn). arg("--csv-feature-col").arg(args.featureNameColumn). arg("--sample-col").arg(featurePropsSampleColumn). + // Negative-control marker (control-aware metrics): pass the chosen control feature so the step emits + // a dedicated per-feature marker CSV; "" when no control is designated (marker header-only). + arg("--control-feature").arg(hasControl ? control : ""). arg("--output-prefix").arg("result"). saveFile("result_feature_properties.csv"). saveFileContent("result_feature_property_meta.json"). + saveFile("result_negative_control.csv"). run() featurePropsResult := render.create(featurePropsTpl, { propertiesFile: featurePropsRun.getFile("result_feature_properties.csv"), propertyMeta: featurePropsRun.getFileContent("result_feature_property_meta.json"), + negativeControlFile: featurePropsRun.getFile("result_negative_control.csv"), + hasControl: hasControl, blockId: blockId, traceSeedSpec: inputSpec }) From 8363948b7ae91d62883d3852104c032655d261a3 Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Thu, 23 Jul 2026 16:21:32 +0200 Subject: [PATCH 006/282] MILAB-6496: hide the Combine-mode column selector Not exposed to users for now. The control, its validation alert, and the workflow combine-mode logic are kept (wrapped in v-if=false) for later re-enable; with the selector hidden, combineColumn stays unset and every antigen uses the default sum mode. --- .changeset/hide-combine-mode-column.md | 5 ++++ ui/src/pages/MainPage.vue | 39 +++++++++++++++----------- 2 files changed, 27 insertions(+), 17 deletions(-) create mode 100644 .changeset/hide-combine-mode-column.md diff --git a/.changeset/hide-combine-mode-column.md b/.changeset/hide-combine-mode-column.md new file mode 100644 index 0000000..a7b9e25 --- /dev/null +++ b/.changeset/hide-combine-mode-column.md @@ -0,0 +1,5 @@ +--- +'@platforma-open/milaboratories.feature-integration.ui': patch +--- + +Hide the Combine-mode column selector. It is not exposed to users for now; the control, its validation, and the workflow's combine-mode logic are kept for later re-enable. With the selector hidden, `combineColumn` stays unset and every antigen uses the default "sum" mode. diff --git a/ui/src/pages/MainPage.vue b/ui/src/pages/MainPage.vue index fc2bed7..bf2beab 100644 --- a/ui/src/pages/MainPage.vue +++ b/ui/src/pages/MainPage.vue @@ -514,23 +514,28 @@ const gridOptions = { Matching trims surrounding spaces but is case-sensitive. - - - - - {{ combineColumnError }} - + + Specificity scores will not be computed without a negative control feature From 9f95daa756071cf25610b8f179899b594e7c38b0 Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Thu, 23 Jul 2026 16:39:58 +0200 Subject: [PATCH 007/282] MILAB-6496: rename cross-reactive label to 'Target cross-reactive' Make the co-binding label explicit: a cell binding two or more on-target antigens, distinct from unwanted nonspecific polyreactivity. Emitted value + tooltip + generator test updated in lockstep. Also retire the remaining internal 'Decoy' examples in favour of 'off-target'. --- .changeset/target-cross-reactive-label.md | 6 ++++++ software/per-cell-metrics/src/per_cell_metrics.py | 8 ++++---- software/test-data/manual/tests/test_panel.py | 2 +- ui/src/pages/MainPage.vue | 4 ++-- 4 files changed, 13 insertions(+), 7 deletions(-) create mode 100644 .changeset/target-cross-reactive-label.md diff --git a/.changeset/target-cross-reactive-label.md b/.changeset/target-cross-reactive-label.md new file mode 100644 index 0000000..28e1655 --- /dev/null +++ b/.changeset/target-cross-reactive-label.md @@ -0,0 +1,6 @@ +--- +'@platforma-open/milaboratories.feature-integration.per-cell-metrics': patch +'@platforma-open/milaboratories.feature-integration.ui': patch +--- + +Rename the co-binding cell label from "cross-reactive" to "Target cross-reactive", making explicit that it means a cell binding two or more on-target antigens — distinct from unwanted, nonspecific polyreactivity. Also drop the remaining internal "Decoy" examples (retired in favour of "off-target"). diff --git a/software/per-cell-metrics/src/per_cell_metrics.py b/software/per-cell-metrics/src/per_cell_metrics.py index a23308e..edb27e9 100644 --- a/software/per-cell-metrics/src/per_cell_metrics.py +++ b/software/per-cell-metrics/src/per_cell_metrics.py @@ -31,7 +31,7 @@ } -CROSS_REACTIVE = "cross-reactive" +CROSS_REACTIVE = "Target cross-reactive" def consensus_category( @@ -54,7 +54,7 @@ def consensus_category( control/off-target signal SUPPRESSES antigen dominance rather than being renormalised away — a cell swamped by them correctly fails the threshold instead of having its top on-target inflated to 100%. - ``offtargets`` designate features whose property (e.g. Type = Off-Target / Decoy) marks them as + ``offtargets`` designate features whose property (e.g. Type = Off-Target) marks them as binders the user does not want to call. When they are supplied and ``label_crossreactive`` is set, the overloaded "ambiguous" bucket is split: a cell whose on-target (non-excluded) signal collectively passes the threshold but is spread across >= 2 on-target features is called "cross-reactive" (a @@ -93,7 +93,7 @@ def offtarget_features( The off-target designation is property-driven: the user picks one imported per-feature property column (e.g. ``antigen_class``) and the set of its values that mark a feature as off-target (e.g. - {"Off-Target", "Decoy"}). This reads the tag->feature CSV — which carries those property columns — + {"Off-Target", "Off-target"}). This reads the tag->feature CSV — which carries those property columns — and returns the resolved set of off-target FEATURE names, so the dominant call can exclude them. Values are matched exactly, whitespace-trimmed but CASE-SENSITIVE (``strip()`` on both sides, no @@ -442,7 +442,7 @@ def main() -> None: p.add_argument( "--offtarget-values", default=None, - help="comma-separated values of --offtarget-col that mark a feature as off-target (e.g. 'Off-Target,Decoy')", + help="comma-separated values of --offtarget-col that mark a feature as off-target (e.g. 'Off-Target,Off-target')", ) p.add_argument("--output-prefix", default="result") args = p.parse_args() diff --git a/software/test-data/manual/tests/test_panel.py b/software/test-data/manual/tests/test_panel.py index 00b3638..e778e86 100644 --- a/software/test-data/manual/tests/test_panel.py +++ b/software/test-data/manual/tests/test_panel.py @@ -83,7 +83,7 @@ def test_crossreactive_two_even_antigens(): result = pcm.consensus_category( counts, threshold=0.6, control="ctrl", offtargets=frozenset(), label_crossreactive=True ) - assert result == "cross-reactive" + assert result == "Target cross-reactive" def test_generator_plants_crossreactive(tmp_path): diff --git a/ui/src/pages/MainPage.vue b/ui/src/pages/MainPage.vue index bf2beab..7a68cbb 100644 --- a/ui/src/pages/MainPage.vue +++ b/ui/src/pages/MainPage.vue @@ -496,8 +496,8 @@ const gridOptions = { For panels that tag antigens target/off-target — leave blank otherwise. Pick the per-feature property column holding those tags, then choose the off-target values below. Off-target features are dropped from the dominant call (like the negative control), and - cells binding two or more real targets are labelled cross-reactive instead of - "ambiguous". + cells binding two or more real targets are labelled Target cross-reactive instead + of "ambiguous". Date: Thu, 23 Jul 2026 16:57:08 +0200 Subject: [PATCH 008/282] MILAB-6496: drop dev design/calibration notes from version control design-and-schemas.md and real-data-calibration.md are internal build/design notes for the manual synthetic-data generator, superseded by the spec and the tracked generator README. Keep them on disk (gitignored) but out of the public repo. --- .gitignore | 3 + .../test-data/manual/design-and-schemas.md | 190 ------------------ .../test-data/manual/real-data-calibration.md | 85 -------- 3 files changed, 3 insertions(+), 275 deletions(-) delete mode 100644 software/test-data/manual/design-and-schemas.md delete mode 100644 software/test-data/manual/real-data-calibration.md diff --git a/.gitignore b/.gitignore index d9096dd..b717254 100644 --- a/.gitignore +++ b/.gitignore @@ -29,6 +29,9 @@ software/test-data/manual/** !software/test-data/manual/**/ !software/test-data/manual/**/*.py !software/test-data/manual/**/*.md +# ...except these dev design/calibration notes — kept on disk, not version-controlled. +software/test-data/manual/design-and-schemas.md +software/test-data/manual/real-data-calibration.md !software/test-data/manual/assets/whitelist_cells.txt # Local working docs / scratch — not version-controlled diff --git a/software/test-data/manual/design-and-schemas.md b/software/test-data/manual/design-and-schemas.md deleted file mode 100644 index 7c12d56..0000000 --- a/software/test-data/manual/design-and-schemas.md +++ /dev/null @@ -1,190 +0,0 @@ -# Multiomics synthetic data — design, schemas, and the join contract - -Background for the manual run in `README.md`: the experiment modeled, the pipeline, the axis contract the -data must satisfy, per-arm file schemas (verified against block code), the coherence model, and the -viability tests. (Consolidates the former `multiomics-manual-test-data-report.md` scoping report + -`multiomics-generator-spec.md` build spec.) - ---- - -## 1. The experiment: BEAM-Ab - -One GEM emulsion produces **three co-registered 10x 5′ v2 libraries from the same cells**, all sharing -**one 16 nt cell barcode** (from the 5′ gel-bead list `737K-august-2016`) — the *only* multiomic linking key. - -| Library | R1 | R2 | Purpose | -|---|---|---|---| -| Gene Expression (GEX) | 16 nt CB + 10 nt UMI | cDNA (5′) | transcriptome / cell typing | -| BCR V(D)J | 16 nt CB + 10 nt UMI | V(D)J contig | paired IGH + IGK/IGL → clonotype | -| Antigen Capture (BEAM) | 16 nt CB + 10 nt UMI | 15 nt antigen barcode @ pos 0 + adapter | per-cell antigen binding | - -UMIs are independent per library; the **cell barcode string** is the shared key. Specificity score -(Cell Ranger BEAM, which `feature-integration` reproduces): `(1 − beta.cdf(0.925, antigenUMI+1, controlUMI+3)) × 100`. - -**Why the import path (not raw FASTQ + Cell Ranger + MiXCR):** only the antigen arm has no import entry -point, so only it needs synthetic FASTQ. GEX (`import-sc-rnaseq-data`) accepts a count matrix; VDJ -(`import-vdj-data`) accepts an AIRR contig table and emits the clonotype key + linker directly. Three -lightweight assets off one shared barcode population — no aligners, no references. - ---- - -## 2. Pipeline - -``` - samples-and-data - ┌─────────────────────────┼──────────────────────────────┐ - GEX arm VDJ arm Antigen arm - import-sc-rnaseq-data import-vdj-data feature-integration - │ │ │ - rna-seq/countMatrix anchor: vdj/uniqueCellCount feature/umiCount - [cellId, geneId] linker: sc/cellLinker [sampleId, cellId, featureId] - │ [sampleId, cellId, scClonotypeKey] │ - cell-type-annotation │ │ - → rna-seq/cellType │ │ - └───────────────┬───────┴─────────────────────────────────┘ - ▼ - vdj-multiomic-integration - anchor = VDJ sc-clonotype dataset; REQUIRED: feature umiCount + cellLinker; - OPTIONAL: GEX countMatrix, cellType. Inner-join on [sampleId, cellId], group by scClonotypeKey. - ▼ - antibody-tcr-lead-selection → top-N antibody leads -``` - ---- - -## 3. The canonical cell barcode (the one rule) - -The convergence join is a **silent inner-join on `[sampleId, cellId]`** — any barcode mismatch drops -cells with no error. The canonical `cellId` = the **bare 16 nt** barcode. Verified per-arm normalization: - -| Arm | barcode handling | verified at | -|---|---|---| -| Antigen (`feature-integration`) | bare 16 nt from R1; de-novo corrected (error-free input → verbatim) | tag-pattern / mitool CELL | -| VDJ (`import-vdj-data`, `airr-sc`) | `cell_id` verbatim (`cellKeyMode:"direct"`) | `formats.lib.tengo:126-146` | -| GEX (`import-sc-rnaseq-data`) | strips `-\d+$` suffix → bare 16 nt | `clean_barcode_suffix` | - -Synthetic barcodes are random 16-mers (not real `737K` members), so keep them **error-free** and leave -cell-barcode whitelist correction **off** in the run (a whitelist would drop them all). The de-novo-error -scenario is for *standalone* `feature-integration` testing only — it would split cells across arms. - ---- - -## 4. The join-spine — axes that must align (byte-identical name + domain) - -| Axis / column | valueType | Key annotations | Produced by | Consumed by | -|---|---|---|---|---| -| `pl7.app/sampleId` | String | — | samples-and-data | all | -| `pl7.app/sc/cellId` | String | `parents=[sampleId]`, no domain | all three arms | the linker; **the multiomic key** | -| `pl7.app/vdj/scClonotypeKey` | String | domain: receptor/structure/runId | import-vdj-data | integration anchor + outputs; lead-selection | -| `pl7.app/vdj/uniqueCellCount` | Int/Long | **`isAnchor:"true"`**, `isAbundance` | import-vdj-data | integration `datasetOptions` anchor | -| `pl7.app/sc/cellLinker` | Int | **`isLinkerColumn:"true"`**, axes `[sampleId, cellId, scClonotypeKey]` | import-vdj-data | integration (REQUIRED linker) | -| `pl7.app/feature/umiCount` | Int | `isAbundance` | feature-integration | integration (REQUIRED feature) | -| `pl7.app/feature/featureId` | String | — | feature-integration | integration feature axis | -| `pl7.app/rna-seq/countMatrix` | Double | axes `[sampleId, cellId, geneId]` (geneId domain `{species}`) | import-sc-rnaseq-data | integration (OPTIONAL GEX) | -| `pl7.app/rna-seq/cellType` | String | axes `[sampleId, cellId]` | cell-type-annotation | integration (OPTIONAL annotation) | - -Integration mechanism: materialize `cellLinker` → `linker.csv [sampleId, cellId, scClonotypeKey]`; write -each per-cell input to its own CSV; inner-join each to the linker on `[sampleId, cellId]`; group by -`scClonotypeKey`. Outputs reuse `scClonotypeKey` verbatim, joining back onto the VDJ clonotype table. - ---- - -## 5. Per-arm file schemas (verified against block code) - -All three upload through **one Samples & Data block** as datasets keyed by the same `sampleId`(s). - -### 5.1 Antigen — `feature-integration` (paired FASTQ) -- **R1** (`*_R1.fastq.gz`): `[16 nt cell barcode][10 nt UMI]` = 26 nt. -- **R2** (`*_R2.fastq.gz`): `[15 nt antigen barcode @ pos 0][tail]`. -- **Panel CSV** (`tag,feature`): antigens + `negative_control`; barcodes pairwise Hamming ≥ 3. -- Error-free cell barcodes in the multiomics dataset. - -### 5.2 VDJ — `import-vdj-data`, format `airr-sc` (AIRR rearrangement TSV, one row per contig) -Columns present: `cell_id`, `locus`, `v_call`, `j_call`, `c_call`, `junction`, `junction_aa`, -`productive`, `duplicate_count`. -- `cell_id` = bare 16 nt (used verbatim). `junction` = CDR3 nt (ACGT, len %3==0 for productive). -- `v_call`/`j_call` = real IMGT gene names. `duplicate_count` (Int ≥1) = UMI support → primary abundance, - gets `isAnchor:"true"` (`infer-columns-airr.lib.tengo:141`). -- Each cell has ≥1 IGH + ≥1 IGK/IGL row. Clonotype key = CDR3-nt + V + J (+ C). Cells with identical - paired rows → same `scClonotypeKey` (lead clones); unique rows → singletons. -- Consumer settings: format = "AIRR single cell", chains = IG Heavy + IG Light. - -### 5.3 GEX — `import-sc-rnaseq-data` (CSV, genes-in-rows) -- First column = **gene IDs** (real Ensembl `ENSG…`); header = **cell barcodes**; body = non-neg integer counts. -- `detect_orientation` → genes-in-rows on an all-numeric body; `check_format` passes on gene-like first col. -- Include real marker genes per class (B: MS4A1/CD79A; plasmablast: MZB1/XBP1/PRDM1) so - `cell-type-annotation` is meaningful. No mapping file needed — species/format inferred. - ---- - -## 6. Coherence model — one cell, three consistent modalities - -Each synthetic cell gets a latent identity; all three arms derive from it, so the downstream result is -assertable. - -| Cell class | VDJ | Antigen (UMIs) | GEX program | -|---|---|---|---| -| **Lead B cells** (few dominant clones) | one of clones L1..L4 (paired IGH+IGK/L) | high on-target, low control → specificity ~100 | plasmablast: MZB1/XBP1/PRDM1 high | -| **Background B cells** | many singleton clones | antigen ≈ control (low) | naive-B: MS4A1/CD79A/TCL1A | -| **Non-B contaminants** (optional) | absent from VDJ | none / control-only | T/myeloid | - -Each lead clone binds exactly one antigen. **Every lead-clone cell appears in all three arms** (survives -the inner-join). Background/contaminant cells may be partial (realistic per-arm dropout). Seeded RNG. - ---- - -## 7. Viability tests (`validate_multiomics.py`, stdlib-only) - -- **Barcode alignment** (the test): the `cellId` set each arm will produce must overlap as intended; every - lead-clone cell ∈ all three sets. -- **Per-arm schema/geometry:** antigen R1=26/R2≥15, on-panel; VDJ AIRR header + heavy/light pairing + - junction validity; GEX orientation + Ensembl IDs + non-neg + no all-zero row/col + markers elevated. -- **Join simulation** (strongest offline proof): build the linker from VDJ pairing, join the antigen - per-cell UMIs on `cellId`, group by clonotype, apply the dominant-antigen + specificity rules; assert - the per-clonotype table is non-empty and L1..L4 show their intended antigen + high specificity. -- Realistic profile: **38/38 PASS**. - ---- - -## 8. Verification status (source review, 2026-07-01) - -A source-level review of all seven blocks (not just the docs) confirmed the join spine ties together. -Resolved items (previously "assumed"): - -1. **`import-vdj-data` per-clonotype anchor carries `isAnchor` — VERIFIED.** The SC anchor - `pl7.app/vdj/uniqueCellCount` (cell count) is freshly authored with `isAnchor:"true"` at the - per-clonotype stage (`process-single-cell.tpl.tengo:191`), so aggregation can't strip it, and it - matches the integration `datasetOptions` predicate (`[sampleId, scClonotypeKey]` + isAnchor). -2. **Samples & Data `Xsv` → import dropdowns — VERIFIED.** Xsv publishes `pl7.app/sequencing/data`/File - with `pl7.app/fileExtension` (csv/tsv); import-vdj matches tsv, import-sc matches csv/tsv. -3. **cellId join key — VERIFIED byte-identical** (`pl7.app/sc/cellId`, String, no domain) across FI, - the VDJ linker, the GEX countMatrix, and cellType. The integration join is a Python inner-join on - `[sampleId, cellId]` grouped by `scClonotypeKey`; cellType's cell axis is inherited from its input. - -Remaining risks / live checks: - -- **Build from the right source (CRITICAL):** `blocks/feature-integration` (stub, no outputs) and - `blocks/vdj-multiomic-integration` (README-only) are NOT the real code — build both from their - MILAB-6496 worktrees, else the convergence feature dropdown is empty and it can't run. -- **Silent inner-join on cellId:** a barcode mismatch drops cells with no error → keep barcodes - byte-identical bare-16nt and match the cell-whitelist setting to the profile. -- **Single-receptor only:** import-vdj emits one linker/anchor per receptor (receptor domain); a - TCR+BCR dataset publishes multiple `pl7.app/sc/cellLinker` columns and the integration's `addSingle` - expects exactly one. BEAM-Ab (BCR/IG) is safe. -- **lead-selection ranking (live check):** the integration's per-clonotype outputs (`restrictionIndex`, - `breadth`, `dominantFeature`, keyed on `scClonotypeKey`, no sample axis) are spec-compatible with - lead-selection's ranking discovery, but confirm they appear in its "Rank by" dropdown on a live run. -- **Backend assets:** import-sc needs `gene-annotations-assets:homo-sapiens`; cell-type needs the - CellTypist model assets — cached automatically online, required for a strictly-offline backend. - -**Tiering:** Tier 0 = VDJ + antigen → integration (feature + linker only). Tier 1 adds GEX + annotation -(the full run in `README.md`). Tier 2 = wider ecosystem on the same cells. - ---- - -## Key code references - -- Convergence: `vdj-multiomic-integration/.../model/src/index.ts`, `.../workflow/src/{main,aggregate}.tpl.tengo`, `.../software/aggregate-clonotypes/`. -- VDJ import: `blocks/import-vdj-data/workflow/src/{process-single-cell.tpl,infer-columns-airr.lib,formats.lib}.tengo`. -- GEX import: `blocks/import-sc-rnaseq-data/workflow/src/libs/pf-counts-conv.lib.tengo`. -- Antigen: this block (`feature-integration`) + `docs/dormant-features/cell-whitelist-correction-plan.md`. diff --git a/software/test-data/manual/real-data-calibration.md b/software/test-data/manual/real-data-calibration.md deleted file mode 100644 index 7536fb2..0000000 --- a/software/test-data/manual/real-data-calibration.md +++ /dev/null @@ -1,85 +0,0 @@ -# Real-data calibration — synthetic BEAM vs a real 5k BEAM-T reference - -Provenance + empirical basis for the `realistic` generator profile. The raw reference data is not in the -repo (10x public download); only this write-up is tracked. - -## Source -- 10x Genomics public dataset `5k_BEAM-T_Human_A0201_B0702_PBMC_5pv2_Multiplex` (~12 GB FASTQ tar, freely downloadable from 10x). -- **BEAM-T** (pMHC-multimer / TCR) — our synthetic is **BEAM-Ab** (BCR). Only *technical shapes* are - borrowed (read geometry, barcode error, UMI depth/duplication, panel separation); the biology - (cell types, antigen semantics, clone structure) is not. -- Measured 2026-07-01: one lane (6 M reads) per library, streamed; raw reads never persisted. - -## Read geometry — confirms ours -R1 = **26 nt** (16 CB + 10 UMI), R2 = **90 nt**, all three libraries. (Our R1 matches; real R2 is 90 nt -but the block only reads the first 15 nt as the feature, so our 25 nt R2 is fine.) - -## Antigen-capture shapes — measured vs synthetic -| Metric | Real BEAM-T | Default profile | **Realistic profile** (verified) | -|---|---|---|---| -| Dominant-feature UMIs/cell | median **632** (p10 18, p90 1490) | 8–30 | median **629** (18–1081) | -| Dominance fraction | median **1.00** (p10 0.79) | ~0.75 | median **0.99** (p10 0.70) | -| Background UMIs/cell | median **3** (p90 9) | higher | median **4** | -| PCR dup (reads/UMI) | median **1.3** (p90 2.0) | 1–4 | **1.30** | -| Reads/cell (antigen) | median **860** | ~60 | **~863** | -| Features detected/cell | median 3 (p90 4) | 3–4 | ~3–4 | - -## Panel barcodes — ours are authentic -Top real R2 15-mers are the **same standard 10x Antigen-Capture barcodes we use**: `GATTGGCTACTCAAT` -(90.2%), `CGGCTCACCGCGTCT` (4.9%), `CTATCTACCGGCTCG` (1.3%) + `CATGTCTACGTTAAG` (1.1%, one we don't -have). Pairwise Hamming among panel barcodes = **min 8** (our `≥3` design floor is safe). ~1–2% of -reads are Hamming-1 variants of the dominant barcode (feature-barcode seq errors → refine-tags snaps back). - -## Cross-library barcode overlap — validates the linking design -Top-5000 cell-barcode sets: **antigen∩gex 80%, antigen∩vdj 67%, gex∩vdj 69%, all-three 67%**. The same -16 nt barcode links the three libraries — and overlap is **partial** (~67–80%, not 100%), i.e. real -per-arm dropout. Our convergence inner-join is built for exactly this (cells missing from an arm drop). - -## GEX depth — already on the right path -~900 distinct UMIs/cell/lane (≈1800 across both lanes). Our synthetic GEX totals ~1800 counts/cell — -**matches**. (Genes/cell needs alignment; literature ~1–3.5k for 5′ PBMC.) The realistic profile bumps -genes 341→~1000 for more realistic genes/cell; UMI depth is unchanged (it was correct). - -## Cell-barcode error / correction structure -440k distinct raw barcodes / 6 M reads; top-5000 = **81.5%** of reads (≈ the 5k real cells). **50% of -distinct barcodes are singletons**; **~18.5% of reads are ambient/error** (not a real cell); **only -~14% of singleton junk is Hamming-1 of a real barcode** (→ correctable), the rest is ambient. So real -junk is *ambient-dominated*, not 1-bp-error-dominated. - -## What was applied — the `realistic` profile (defaults untouched) -| Generator | Flag | Output | Change | -|---|---|---|---| -| `antigen/generate.py` | `--profile realistic` | `realistic/` | UMI depth ↑, dup ↓, dominance ↑, background ↓ | -| `multiomics/generate_vdj.py` | `--realistic` | `vdj/realistic/` | reads realistic antigen consensus | -| `multiomics/generate_gex.py` | `--realistic` | `gex/realistic/` | 1000 genes (depth already matched) | -| `multiomics/validate_multiomics.py` | `--realistic` | — | validates the realistic chain (**38/38**) | - -Build the realistic multiomics chain: -```bash -# antigen (in antigen/) -python3 generate.py --profile realistic # + optionally --scenario all -# arms (in multiomics/) -python3 generate_vdj.py --realistic && python3 generate_gex.py --realistic -python3 validate_multiomics.py --realistic # 38/38 -``` - -## Recommendations NOT auto-applied (future, if wanted) -- **Ambient/error scenario:** make the `errors` fixture ambient-dominated — add a heavy tail of random - non-cell barcodes (~18% of reads), only ~14% of them Hamming-1 of a real cell. The current `errors` - scenario over-weights clean 1-bp errors. -- **Cross-library dropout scenario:** drop ~15–30% of cells per arm so triple-overlap ≈ 67% (tests the - inner-join drop). The default keeps 100% overlap for a clean, maximal join. -- **Dynamic range:** one antigen was 90% of the whole BEAM-T library. BEAM-Ab discovery is more even, so - we keep a spread — but a "single-dominant-antigen" scenario would mirror BEAM-T. - -## Scale (samples / panel / cells) — parameterized 2026-07-02 - -The depth/dominance calibration above is orthogonal to *scale*. Sample count, antigen-panel size, and -cells-per-sample are now CLI flags (`--samples` / `--panel-size` / `--cells-per-sample`, defaults -24 / 64 / 2000), so the fixture spans a toy bed to a cohort-scale run without touching this calibration. -Targets are corroborated by the BEAM dataset-scale survey (deep-research, adversarially verified -2026-06-30): cohort high-water ~22–50 donors; verified feature ceiling = 64 (BEAM-proper alone is ~6); -a real GEM well is 2k–10k cells. The panel keeps the 4 -real 10x anchor barcodes and synthesizes the rest (15-mers, Hamming ≥ 3). The `whitelist737k` cell pool -now samples the full 10x `737K-august-2016` inclusion list (737,280 barcodes) rather than the ~800 -harvested pool, so it scales with `--samples`/`--cells-per-sample`. From f7228208a21687c2d56357991d409a66cbb65697 Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Fri, 24 Jul 2026 09:39:08 +0200 Subject: [PATCH 009/282] MILAB-6496: wrap over-long --offtarget-values help (ruff E501) The 'decoy'->'off-target' scrub pushed the --offtarget-values help string to 122 chars; split it across two implicit-concatenated literals to satisfy the 120-col limit (ruff check E501, caught in CI). --- software/per-cell-metrics/src/per_cell_metrics.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/software/per-cell-metrics/src/per_cell_metrics.py b/software/per-cell-metrics/src/per_cell_metrics.py index edb27e9..f8251be 100644 --- a/software/per-cell-metrics/src/per_cell_metrics.py +++ b/software/per-cell-metrics/src/per_cell_metrics.py @@ -442,7 +442,8 @@ def main() -> None: p.add_argument( "--offtarget-values", default=None, - help="comma-separated values of --offtarget-col that mark a feature as off-target (e.g. 'Off-Target,Off-target')", + help="comma-separated values of --offtarget-col that mark a feature as off-target " + "(e.g. 'Off-Target,Off-target')", ) p.add_argument("--output-prefix", default="result") args = p.parse_args() From 76a21e537785e57af0f9e8b3335277ef9a6e012a Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 12:10:08 +0200 Subject: [PATCH 010/282] MILAB-6496: read the panel as a (tag, sample) table Blank barcodes are reported to the caller rather than filtered away, and a blank sample cell is fatal: the "*" sentinel means the panel declares no sample dimension at all, so reading an empty cell that way would widen one malformed row into a claim over every sample in the run. --- software/per-cell-metrics/src/panel.py | 95 ++++++++++++++++ software/per-cell-metrics/test/test_panel.py | 109 +++++++++++++++++++ 2 files changed, 204 insertions(+) create mode 100644 software/per-cell-metrics/src/panel.py create mode 100644 software/per-cell-metrics/test/test_panel.py diff --git a/software/per-cell-metrics/src/panel.py b/software/per-cell-metrics/src/panel.py new file mode 100644 index 0000000..256c770 --- /dev/null +++ b/software/per-cell-metrics/src/panel.py @@ -0,0 +1,95 @@ +"""The panel file as a (tag, sample) table. + +The panel is authoritative and cannot be checked against its subject, so it is +checked against the reads in both directions instead — per sample, because the +same barcode can carry a different antigen in a different sample's panel and a +global check would let a barcode undeclared in one sample pass on another's +declaration. + +A tag is the barcode sequence. The feature name is a declared property, not an +identity: a name only travels where every row for that tag agrees on it. +""" + +from __future__ import annotations + +import polars as pl + +# Stands for "every sample" when the panel carries no sample column. The unkeyed +# case is this rule with the sample component constant, not a separate rule. +ANY_SAMPLE = "*" + + +def read_panel(csv_path: str, roles: dict[str, str]) -> tuple[pl.DataFrame, list[int]]: + raw = pl.read_csv(csv_path, infer_schema_length=0) + barcode_col, sample_col = roles["barcode"], roles.get("sample") or "" + + for name, col in (("barcode", barcode_col), ("feature", roles["feature"])): + if col not in raw.columns: + raise SystemExit(f"panel file has no {name} column {col!r}; columns are {raw.columns}") + if sample_col and sample_col not in raw.columns: + raise SystemExit(f"panel file has no sample column {sample_col!r}; columns are {raw.columns}") + + panel = raw.with_row_index("_row").with_columns(pl.col(barcode_col).str.strip_chars().fill_null("").alias("tag")) + panel = panel.with_columns( + pl.col(sample_col).str.strip_chars().fill_null("").alias("sample") + if sample_col + else pl.lit(ANY_SAMPLE).alias("sample") + ) + + # A blank sample cell in a panel that HAS a sample column is fatal, never + # ANY_SAMPLE. "*" means the panel declares no sample dimension at all; + # reading an empty cell that way would widen one malformed row into a claim + # over every sample in the run. + if sample_col: + blank_sample = panel.filter(pl.col("sample") == "") + if blank_sample.height: + rows = ", ".join(str(r + 2) for r in blank_sample["_row"]) + raise SystemExit( + f"panel file has a blank {sample_col!r} on line(s) {rows}. Leave the column out " + "entirely to declare one panel over every sample; a blank cell is ambiguous." + ) + + # Blank barcodes are returned, not filtered away. Dropping a malformed row + # silently is the same failure the property no-silent-drop rule exists to + # prevent, and worse: nothing downstream can tell the panel was short. + dropped = [r + 2 for r in panel.filter(pl.col("tag") == "")["_row"]] + panel = panel.filter(pl.col("tag") != "").drop("_row") + + dupes = panel.group_by(["tag", "sample"]).len().filter(pl.col("len") > 1).sort(["tag", "sample"]) + if dupes.height: + offenders = ", ".join(f"{t}/{s}" for t, s in zip(dupes["tag"], dupes["sample"], strict=True)) + raise SystemExit( + f"panel file declares the same barcode twice for one sample: {offenders}. " + "Each (barcode, sample) pair must appear once." + ) + + drop = {barcode_col} | ({sample_col} if sample_col else set()) + kept = panel.select(["tag", "sample"] + [c for c in raw.columns if c not in drop]) + return kept, dropped + + +def property_columns(panel: pl.DataFrame) -> list[str]: + return [c for c in panel.columns if c not in ("tag", "sample")] + + +def consistent_properties( + panel: pl.DataFrame, columns: list[str] +) -> tuple[dict[str, dict[str, str]], list[tuple[str, str, list[str]]]]: + """Per tag, the properties holding one value across all its rows. + + Disagreements are returned rather than dropped. With barcode reuse across + panels an inconsistent declaration is the expected case, and dropping it + silently would break the panel file's own no-silent-drop rule. + """ + props: dict[str, dict[str, str]] = {} + inconsistent: list[tuple[str, str, list[str]]] = [] + for tag, rows in panel.group_by("tag", maintain_order=True): + name = tag[0] if isinstance(tag, tuple) else tag + props[name] = {} + for col in columns: + values = sorted({v.strip() for v in rows[col].to_list() if v and v.strip()}) + if len(values) == 1: + props[name][col] = values[0] + elif len(values) > 1: + inconsistent.append((name, col, values)) + return props, inconsistent diff --git a/software/per-cell-metrics/test/test_panel.py b/software/per-cell-metrics/test/test_panel.py new file mode 100644 index 0000000..d8ed5c7 --- /dev/null +++ b/software/per-cell-metrics/test/test_panel.py @@ -0,0 +1,109 @@ +import polars as pl +import pytest +from panel import consistent_properties, read_panel + +ROLES = {"barcode": "Sequence", "feature": "Name", "sample": "Samples"} + + +def _csv(tmp_path, rows, header): + p = tmp_path / "panel.csv" + p.write_text("\n".join([",".join(header)] + [",".join(r) for r in rows]) + "\n") + return str(p) + + +def test_read_panel_one_row_per_tag_sample(tmp_path): + path = _csv( + tmp_path, + [ + ["S1", "AgA", "AAAA", "Off-Target"], + ["S2", "AgB", "AAAA", "Off-Target"], + ["S1", "AgC", "CCCC", "Target"], + ], + ["Samples", "Name", "Sequence", "Type"], + ) + panel, dropped = read_panel(path, ROLES) + assert panel.height == 3 + assert set(panel.columns) >= {"tag", "sample", "Name", "Type"} + assert dropped == [] + + +def test_read_panel_without_sample_column_uses_star(tmp_path): + path = _csv(tmp_path, [["AgA", "AAAA"], ["AgB", "CCCC"]], ["Name", "Sequence"]) + panel, _ = read_panel(path, {"barcode": "Sequence", "feature": "Name", "sample": ""}) + assert panel["sample"].unique().to_list() == ["*"] + + +def test_consistent_properties_keeps_agreeing_values(): + panel = pl.DataFrame( + { + "tag": ["AAAA", "AAAA"], + "sample": ["S1", "S2"], + "Name": ["AgA", "AgA"], + "Channel": ["PE", "PE"], + } + ) + props, bad = consistent_properties(panel, ["Name", "Channel"]) + assert props["AAAA"] == {"Name": "AgA", "Channel": "PE"} + assert bad == [] + + +def test_consistent_properties_drops_disagreeing_and_reports_it(): + # Same barcode, different names across two samples' panels — the real shape + # this rule exists for. Names are synthetic: this repository is public. + panel = pl.DataFrame( + { + "tag": ["AAAA", "AAAA"], + "sample": ["S1", "S2"], + "Name": ["AgA", "AgB"], + "Channel": ["APC", "APC"], + } + ) + props, bad = consistent_properties(panel, ["Name", "Channel"]) + assert props["AAAA"] == {"Channel": "APC"} + assert bad == [("AAAA", "Name", ["AgA", "AgB"])] + + +def test_consistent_properties_ignores_blanks(): + panel = pl.DataFrame({"tag": ["AAAA", "AAAA"], "sample": ["S1", "S2"], "Name": ["AgA", ""]}) + props, bad = consistent_properties(panel, ["Name"]) + assert props["AAAA"] == {"Name": "AgA"} + assert bad == [] + + +def test_duplicate_tag_sample_pair_is_fatal(tmp_path): + path = _csv( + tmp_path, + [["S1", "AgA", "AAAA", "Target"], ["S1", "AgB", "AAAA", "Target"]], + ["Samples", "Name", "Sequence", "Type"], + ) + with pytest.raises(SystemExit) as e: + read_panel(path, ROLES) + assert "AAAA" in str(e.value) + + +def test_blank_barcode_row_is_reported_not_dropped(tmp_path): + path = _csv( + tmp_path, + [ + ["S1", "AgA", "AAAA", "Target"], + ["S1", "AgB", "", "Target"], + ], + ["Samples", "Name", "Sequence", "Type"], + ) + panel, dropped = read_panel(path, ROLES) + assert panel.height == 1 + assert dropped == [3] # 1-based line number in the CSV, header counted + + +def test_blank_sample_cell_is_fatal(tmp_path): + path = _csv( + tmp_path, + [ + ["S1", "AgA", "AAAA", "Target"], + ["", "AgB", "CCCC", "Target"], + ], + ["Samples", "Name", "Sequence", "Type"], + ) + with pytest.raises(SystemExit) as e: + read_panel(path, ROLES) + assert "3" in str(e.value) From bebb0bf37f1b60abe21794297edb3384ec032c16 Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 12:16:12 +0200 Subject: [PATCH 011/282] MILAB-6496: an empty panel line is a dropped row, not a blank sample --- software/per-cell-metrics/src/panel.py | 31 ++++++++++++++++---- software/per-cell-metrics/test/test_panel.py | 21 ++++++++++++- 2 files changed, 45 insertions(+), 7 deletions(-) diff --git a/software/per-cell-metrics/src/panel.py b/software/per-cell-metrics/src/panel.py index 256c770..c0dea7b 100644 --- a/software/per-cell-metrics/src/panel.py +++ b/software/per-cell-metrics/src/panel.py @@ -29,6 +29,18 @@ def read_panel(csv_path: str, roles: dict[str, str]) -> tuple[pl.DataFrame, list if sample_col and sample_col not in raw.columns: raise SystemExit(f"panel file has no sample column {sample_col!r}; columns are {raw.columns}") + # "_row" joins "tag" and "sample" as names this function owns. Colliding + # with one is a raw polars DuplicateError three lines later otherwise. + reserved = {"tag", "sample", "_row"} & set(raw.columns) + if reserved: + raise SystemExit( + f"panel file uses reserved column name(s) {sorted(reserved)}; rename them. " + "'tag' and 'sample' are what this reader produces." + ) + + # fill_null is load-bearing, not defensive: under infer_schema_length=0 a + # bare empty field parses to null while a quoted one parses to "", so the + # two spellings of blank would otherwise take different branches below. panel = raw.with_row_index("_row").with_columns(pl.col(barcode_col).str.strip_chars().fill_null("").alias("tag")) panel = panel.with_columns( pl.col(sample_col).str.strip_chars().fill_null("").alias("sample") @@ -36,7 +48,18 @@ def read_panel(csv_path: str, roles: dict[str, str]) -> tuple[pl.DataFrame, list else pl.lit(ANY_SAMPLE).alias("sample") ) - # A blank sample cell in a panel that HAS a sample column is fatal, never + # Blank barcodes are separated FIRST, and the order is the whole point. + # polars materializes a trailing blank line as a real all-null row, so a + # panel whose only flaw is a stray newline at EOF would otherwise die on + # the blank-sample check below — telling the user to remove a sample column + # that is not the problem. An empty line is a dropped row, not an ambiguous + # cell. Blank barcodes are returned rather than filtered away: dropping a + # malformed row silently is the failure the no-silent-drop rule exists to + # prevent, and worse, because nothing downstream can tell the panel was short. + dropped = [r + 2 for r in panel.filter(pl.col("tag") == "")["_row"]] + panel = panel.filter(pl.col("tag") != "") + + # A blank sample cell on a row that IS otherwise real is fatal, never # ANY_SAMPLE. "*" means the panel declares no sample dimension at all; # reading an empty cell that way would widen one malformed row into a claim # over every sample in the run. @@ -49,11 +72,7 @@ def read_panel(csv_path: str, roles: dict[str, str]) -> tuple[pl.DataFrame, list "entirely to declare one panel over every sample; a blank cell is ambiguous." ) - # Blank barcodes are returned, not filtered away. Dropping a malformed row - # silently is the same failure the property no-silent-drop rule exists to - # prevent, and worse: nothing downstream can tell the panel was short. - dropped = [r + 2 for r in panel.filter(pl.col("tag") == "")["_row"]] - panel = panel.filter(pl.col("tag") != "").drop("_row") + panel = panel.drop("_row") dupes = panel.group_by(["tag", "sample"]).len().filter(pl.col("len") > 1).sort(["tag", "sample"]) if dupes.height: diff --git a/software/per-cell-metrics/test/test_panel.py b/software/per-cell-metrics/test/test_panel.py index d8ed5c7..5a0c532 100644 --- a/software/per-cell-metrics/test/test_panel.py +++ b/software/per-cell-metrics/test/test_panel.py @@ -92,7 +92,8 @@ def test_blank_barcode_row_is_reported_not_dropped(tmp_path): ) panel, dropped = read_panel(path, ROLES) assert panel.height == 1 - assert dropped == [3] # 1-based line number in the CSV, header counted + assert dropped == [3] # CSV record ordinal, header counted (not the + # physical line, which differs if a quoted field contains a newline) def test_blank_sample_cell_is_fatal(tmp_path): @@ -107,3 +108,21 @@ def test_blank_sample_cell_is_fatal(tmp_path): with pytest.raises(SystemExit) as e: read_panel(path, ROLES) assert "3" in str(e.value) + + +def test_trailing_blank_line_is_not_a_blank_sample_cell(tmp_path): + # polars materializes a trailing newline as a real all-null row. A stray + # newline at EOF is the commonest shape a panel file arrives in; it must + # not read as an ambiguous sample cell. + p = tmp_path / "panel.csv" + p.write_text("Samples,Name,Sequence,Type\nS1,AgA,AAAA,Target\n\n") + panel, dropped = read_panel(str(p), ROLES) + assert panel.height == 1 + assert dropped == [3] + + +def test_reserved_column_name_is_fatal(tmp_path): + path = _csv(tmp_path, [["S1", "AgA", "AAAA", "x"]], ["Samples", "Name", "Sequence", "tag"]) + with pytest.raises(SystemExit) as e: + read_panel(path, ROLES) + assert "tag" in str(e.value) From 387990fb03244d619c1e311746b04f10bf5f2688 Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 12:21:13 +0200 Subject: [PATCH 012/282] MILAB-6496: a role column may be named tag --- software/per-cell-metrics/src/panel.py | 12 +++++++++--- software/per-cell-metrics/test/test_panel.py | 19 +++++++++++++++++++ 2 files changed, 28 insertions(+), 3 deletions(-) diff --git a/software/per-cell-metrics/src/panel.py b/software/per-cell-metrics/src/panel.py index c0dea7b..293ff84 100644 --- a/software/per-cell-metrics/src/panel.py +++ b/software/per-cell-metrics/src/panel.py @@ -29,9 +29,15 @@ def read_panel(csv_path: str, roles: dict[str, str]) -> tuple[pl.DataFrame, list if sample_col and sample_col not in raw.columns: raise SystemExit(f"panel file has no sample column {sample_col!r}; columns are {raw.columns}") - # "_row" joins "tag" and "sample" as names this function owns. Colliding - # with one is a raw polars DuplicateError three lines later otherwise. - reserved = {"tag", "sample", "_row"} & set(raw.columns) + # "_row" joins "tag" and "sample" as names this function owns; colliding + # with one is a raw polars DuplicateError ten lines later otherwise. But a + # ROLE column may legitimately be called "tag" or "sample" — emit_panel.py + # in this same package defaults --tag-col to "tag" — and it cannot collide, + # because alias() replaces a same-named source column and the role columns + # are excluded from the carry-through below. "_row" stays unconditional: it + # is injected, so any source column of that name really does collide. + role_cols = {barcode_col, sample_col} - {""} + reserved = ({"tag", "sample"} & (set(raw.columns) - role_cols)) | ({"_row"} & set(raw.columns)) if reserved: raise SystemExit( f"panel file uses reserved column name(s) {sorted(reserved)}; rename them. " diff --git a/software/per-cell-metrics/test/test_panel.py b/software/per-cell-metrics/test/test_panel.py index 5a0c532..0feae2e 100644 --- a/software/per-cell-metrics/test/test_panel.py +++ b/software/per-cell-metrics/test/test_panel.py @@ -122,7 +122,26 @@ def test_trailing_blank_line_is_not_a_blank_sample_cell(tmp_path): def test_reserved_column_name_is_fatal(tmp_path): + # A NON-role column named "tag" would be overwritten by the one this + # reader produces, so it is refused rather than silently shadowed. path = _csv(tmp_path, [["S1", "AgA", "AAAA", "x"]], ["Samples", "Name", "Sequence", "tag"]) with pytest.raises(SystemExit) as e: read_panel(path, ROLES) assert "tag" in str(e.value) + + path = _csv(tmp_path, [["S1", "AgA", "AAAA", "x"]], ["Samples", "Name", "Sequence", "sample"]) + with pytest.raises(SystemExit) as e: + read_panel(path, ROLES) + assert "sample" in str(e.value) + + +def test_role_column_may_be_named_tag(tmp_path): + # emit_panel.py in this package documents this very shape and defaults + # --tag-col to "tag". A role column cannot collide: alias() replaces the + # source column rather than duplicating it. + path = _csv(tmp_path, [["S1", "AgA", "AAAA"]], ["sample", "feature", "tag"]) + panel, dropped = read_panel(path, {"barcode": "tag", "feature": "feature", "sample": "sample"}) + assert panel.height == 1 + assert panel["tag"].to_list() == ["AAAA"] + assert panel["sample"].to_list() == ["S1"] + assert dropped == [] From a4eac4f623b6caadb6cf1525c9c52004b14c66d4 Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 12:28:05 +0200 Subject: [PATCH 013/282] MILAB-6496: refuse a role column named after a different produced column --- software/per-cell-metrics/src/panel.py | 29 ++++++++++++++------ software/per-cell-metrics/test/test_panel.py | 10 +++++++ 2 files changed, 30 insertions(+), 9 deletions(-) diff --git a/software/per-cell-metrics/src/panel.py b/software/per-cell-metrics/src/panel.py index 293ff84..c89260d 100644 --- a/software/per-cell-metrics/src/panel.py +++ b/software/per-cell-metrics/src/panel.py @@ -29,15 +29,26 @@ def read_panel(csv_path: str, roles: dict[str, str]) -> tuple[pl.DataFrame, list if sample_col and sample_col not in raw.columns: raise SystemExit(f"panel file has no sample column {sample_col!r}; columns are {raw.columns}") - # "_row" joins "tag" and "sample" as names this function owns; colliding - # with one is a raw polars DuplicateError ten lines later otherwise. But a - # ROLE column may legitimately be called "tag" or "sample" — emit_panel.py - # in this same package defaults --tag-col to "tag" — and it cannot collide, - # because alias() replaces a same-named source column and the role columns - # are excluded from the carry-through below. "_row" stays unconditional: it - # is injected, so any source column of that name really does collide. - role_cols = {barcode_col, sample_col} - {""} - reserved = ({"tag", "sample"} & (set(raw.columns) - role_cols)) | ({"_row"} & set(raw.columns)) + # A role column may be named after the column IT ITSELF produces, and after + # nothing else. emit_panel.py in this package defaults --tag-col to "tag", + # so a barcode column called "tag" must stay legal — alias() replaces the + # same-named source column rather than duplicating it. + # + # The exclusion cannot be widened to "bound to any role". A SAMPLE column + # named "tag" is fatal precisely because the barcode alias runs first and + # overwrites it, so the sample expression then reads barcodes and "sample" + # silently becomes a copy of "tag" — per-sample keying, the load-bearing + # property of this whole design, collapsing with no error and no duplicate + # to catch it. The mirror (a barcode column named "sample") happens to + # produce correct output today, but only because of the order of the two + # statements below; it is refused rather than left resting on that. + reserved = set() + if "tag" in raw.columns and barcode_col != "tag": + reserved.add("tag") + if "sample" in raw.columns and sample_col != "sample": + reserved.add("sample") + if "_row" in raw.columns: # injected, so any source column of that name collides + reserved.add("_row") if reserved: raise SystemExit( f"panel file uses reserved column name(s) {sorted(reserved)}; rename them. " diff --git a/software/per-cell-metrics/test/test_panel.py b/software/per-cell-metrics/test/test_panel.py index 0feae2e..c44050d 100644 --- a/software/per-cell-metrics/test/test_panel.py +++ b/software/per-cell-metrics/test/test_panel.py @@ -145,3 +145,13 @@ def test_role_column_may_be_named_tag(tmp_path): assert panel["tag"].to_list() == ["AAAA"] assert panel["sample"].to_list() == ["S1"] assert dropped == [] + + +def test_sample_role_named_tag_is_fatal(tmp_path): + # The barcode alias runs first and would overwrite this column, leaving + # "sample" a silent copy of the barcode — per-sample keying gone, and no + # duplicate raised because the pairs stay unique. Refused, not corrected. + path = _csv(tmp_path, [["S1", "AgA", "AAAA"]], ["tag", "Name", "Sequence"]) + with pytest.raises(SystemExit) as e: + read_panel(path, {"barcode": "Sequence", "feature": "Name", "sample": "tag"}) + assert "tag" in str(e.value) From 414838f9003af37200213695343c7565916b913b Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 12:35:02 +0200 Subject: [PATCH 014/282] MILAB-6496: refuse two roles bound to one column --- software/per-cell-metrics/src/panel.py | 19 ++++++++++++++ software/per-cell-metrics/test/test_panel.py | 27 +++++++++++++++++++- 2 files changed, 45 insertions(+), 1 deletion(-) diff --git a/software/per-cell-metrics/src/panel.py b/software/per-cell-metrics/src/panel.py index c89260d..d8e21d2 100644 --- a/software/per-cell-metrics/src/panel.py +++ b/software/per-cell-metrics/src/panel.py @@ -29,6 +29,25 @@ def read_panel(csv_path: str, roles: dict[str, str]) -> tuple[pl.DataFrame, list if sample_col and sample_col not in raw.columns: raise SystemExit(f"panel file has no sample column {sample_col!r}; columns are {raw.columns}") + # Two roles must not name the SAME column. This is reachable from the UI: + # the Sample-column dropdown in MainPage.vue offers every CSV column + # unfiltered, unlike its siblings for combine-mode and off-target, which + # exclude the barcode/feature roles. Picking the barcode column as the + # sample column returns "sample" as a copy of "tag" — per-sample keying + # gone, silently. The name-vs-role guard below does not catch it, because + # it is name-independent: it reproduces with any column name. + bound = [("barcode", barcode_col), ("feature", roles["feature"])] + if sample_col: + bound.append(("sample", sample_col)) + seen: dict[str, str] = {} + for role, col in bound: + if col in seen: + raise SystemExit( + f"panel file roles {seen[col]!r} and {role!r} both name column {col!r}; " + "each role needs a column of its own." + ) + seen[col] = role + # A role column may be named after the column IT ITSELF produces, and after # nothing else. emit_panel.py in this package defaults --tag-col to "tag", # so a barcode column called "tag" must stay legal — alias() replaces the diff --git a/software/per-cell-metrics/test/test_panel.py b/software/per-cell-metrics/test/test_panel.py index c44050d..94dc949 100644 --- a/software/per-cell-metrics/test/test_panel.py +++ b/software/per-cell-metrics/test/test_panel.py @@ -23,7 +23,7 @@ def test_read_panel_one_row_per_tag_sample(tmp_path): ) panel, dropped = read_panel(path, ROLES) assert panel.height == 3 - assert set(panel.columns) >= {"tag", "sample", "Name", "Type"} + assert set(panel.columns) == {"tag", "sample", "Name", "Type"} assert dropped == [] @@ -155,3 +155,28 @@ def test_sample_role_named_tag_is_fatal(tmp_path): with pytest.raises(SystemExit) as e: read_panel(path, {"barcode": "Sequence", "feature": "Name", "sample": "tag"}) assert "tag" in str(e.value) + + +def test_two_blank_barcode_rows_are_not_a_duplicate(tmp_path): + # Both rows have tag "", so they would collide as a duplicate (tag, sample) + # pair if the blank-barcode filter ran after the dupe check. + path = _csv( + tmp_path, + [ + ["S1", "AgA", "AAAA", "Target"], + ["S1", "AgB", "", "Target"], + ["S1", "AgC", "", "Target"], + ], + ["Samples", "Name", "Sequence", "Type"], + ) + panel, dropped = read_panel(path, ROLES) + assert panel.height == 1 + assert dropped == [3, 4] + + +def test_two_roles_on_one_column_is_fatal(tmp_path): + # Reachable from the UI: the Sample-column dropdown is unfiltered. + path = _csv(tmp_path, [["S1", "AgA", "AAAA"]], ["Samples", "Name", "Sequence"]) + with pytest.raises(SystemExit) as e: + read_panel(path, {"barcode": "Sequence", "feature": "Name", "sample": "Sequence"}) + assert "Sequence" in str(e.value) From a884473e88a2b57c40893d4890431cb6a9222ecc Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 12:57:47 +0200 Subject: [PATCH 015/282] MILAB-6496: document and pin the panel reader's contract --- software/per-cell-metrics/src/panel.py | 63 +++++++++--- software/per-cell-metrics/test/test_panel.py | 101 ++++++++++++------- 2 files changed, 114 insertions(+), 50 deletions(-) diff --git a/software/per-cell-metrics/src/panel.py b/software/per-cell-metrics/src/panel.py index d8e21d2..0d3bcfd 100644 --- a/software/per-cell-metrics/src/panel.py +++ b/software/per-cell-metrics/src/panel.py @@ -12,6 +12,8 @@ from __future__ import annotations +from typing import NamedTuple + import polars as pl # Stands for "every sample" when the panel carries no sample column. The unkeyed @@ -19,7 +21,35 @@ ANY_SAMPLE = "*" -def read_panel(csv_path: str, roles: dict[str, str]) -> tuple[pl.DataFrame, list[int]]: +class Panel(NamedTuple): + frame: pl.DataFrame + dropped_lines: list[int] + + +def _csv_line(row_index: int) -> int: + """1-based CSV record ordinal, header counted. Not the physical line number: + the two differ when a quoted field contains a newline.""" + return row_index + 2 + + +def read_panel(csv_path: str, roles: dict[str, str]) -> Panel: + """Read the panel CSV into a (tag, sample) table. + + Returns the table and the CSV lines dropped for having a blank barcode. + Those line numbers are 1-based record ordinals with the header counted; + they are not physical line numbers when a quoted field contains a newline. + + Normalisation is asymmetric on purpose: "tag" and "sample" are stripped, + because they are keys; property columns are carried through exactly as + written. consistent_properties() is the accessor that normalises them, so + reading a property column directly can yield " AgA " and "AgA" as two + distinct values. + + Compare emit_feature_properties.py, which consolidates the same file's + properties by feature NAME with first-non-empty-wins and no sample + dimension. That is the global-check failure this module exists to avoid; + the two rules coexist today and a caller must choose knowingly. + """ raw = pl.read_csv(csv_path, infer_schema_length=0) barcode_col, sample_col = roles["barcode"], roles.get("sample") or "" @@ -29,13 +59,13 @@ def read_panel(csv_path: str, roles: dict[str, str]) -> tuple[pl.DataFrame, list if sample_col and sample_col not in raw.columns: raise SystemExit(f"panel file has no sample column {sample_col!r}; columns are {raw.columns}") - # Two roles must not name the SAME column. This is reachable from the UI: - # the Sample-column dropdown in MainPage.vue offers every CSV column - # unfiltered, unlike its siblings for combine-mode and off-target, which - # exclude the barcode/feature roles. Picking the barcode column as the - # sample column returns "sample" as a copy of "tag" — per-sample keying - # gone, silently. The name-vs-role guard below does not catch it, because - # it is name-independent: it reproduces with any column name. + # Two roles on one column silently makes "sample" a copy of "tag" — the + # barcode alias below runs first and overwrites it, so the sample + # expression then reads barcodes: per-sample keying gone, with no error and + # no duplicate to catch it. The name-vs-role guard further down does not + # catch this either, because it is name-independent and reproduces with + # any column name. Reachable from the UI today — the Sample-column + # dropdown is unfiltered. bound = [("barcode", barcode_col), ("feature", roles["feature"])] if sample_col: bound.append(("sample", sample_col)) @@ -59,8 +89,9 @@ def read_panel(csv_path: str, roles: dict[str, str]) -> tuple[pl.DataFrame, list # silently becomes a copy of "tag" — per-sample keying, the load-bearing # property of this whole design, collapsing with no error and no duplicate # to catch it. The mirror (a barcode column named "sample") happens to - # produce correct output today, but only because of the order of the two - # statements below; it is refused rather than left resting on that. + # produce correct output today, but only because the barcode alias is + # applied before the sample alias; it is refused rather than left resting + # on that. reserved = set() if "tag" in raw.columns and barcode_col != "tag": reserved.add("tag") @@ -92,7 +123,7 @@ def read_panel(csv_path: str, roles: dict[str, str]) -> tuple[pl.DataFrame, list # cell. Blank barcodes are returned rather than filtered away: dropping a # malformed row silently is the failure the no-silent-drop rule exists to # prevent, and worse, because nothing downstream can tell the panel was short. - dropped = [r + 2 for r in panel.filter(pl.col("tag") == "")["_row"]] + dropped = [_csv_line(r) for r in panel.filter(pl.col("tag") == "")["_row"]] panel = panel.filter(pl.col("tag") != "") # A blank sample cell on a row that IS otherwise real is fatal, never @@ -102,7 +133,7 @@ def read_panel(csv_path: str, roles: dict[str, str]) -> tuple[pl.DataFrame, list if sample_col: blank_sample = panel.filter(pl.col("sample") == "") if blank_sample.height: - rows = ", ".join(str(r + 2) for r in blank_sample["_row"]) + rows = ", ".join(str(_csv_line(r)) for r in blank_sample["_row"]) raise SystemExit( f"panel file has a blank {sample_col!r} on line(s) {rows}. Leave the column out " "entirely to declare one panel over every sample; a blank cell is ambiguous." @@ -118,9 +149,9 @@ def read_panel(csv_path: str, roles: dict[str, str]) -> tuple[pl.DataFrame, list "Each (barcode, sample) pair must appear once." ) - drop = {barcode_col} | ({sample_col} if sample_col else set()) - kept = panel.select(["tag", "sample"] + [c for c in raw.columns if c not in drop]) - return kept, dropped + role_cols = {barcode_col} | ({sample_col} if sample_col else set()) + kept = panel.select(["tag", "sample"] + [c for c in raw.columns if c not in role_cols]) + return Panel(kept, dropped) def property_columns(panel: pl.DataFrame) -> list[str]: @@ -139,7 +170,7 @@ def consistent_properties( props: dict[str, dict[str, str]] = {} inconsistent: list[tuple[str, str, list[str]]] = [] for tag, rows in panel.group_by("tag", maintain_order=True): - name = tag[0] if isinstance(tag, tuple) else tag + (name,) = tag props[name] = {} for col in columns: values = sorted({v.strip() for v in rows[col].to_list() if v and v.strip()}) diff --git a/software/per-cell-metrics/test/test_panel.py b/software/per-cell-metrics/test/test_panel.py index 94dc949..0e6281b 100644 --- a/software/per-cell-metrics/test/test_panel.py +++ b/software/per-cell-metrics/test/test_panel.py @@ -1,11 +1,11 @@ import polars as pl import pytest -from panel import consistent_properties, read_panel +from panel import consistent_properties, property_columns, read_panel ROLES = {"barcode": "Sequence", "feature": "Name", "sample": "Samples"} -def _csv(tmp_path, rows, header): +def _csv(tmp_path, header=("Samples", "Name", "Sequence", "Type"), *, rows): p = tmp_path / "panel.csv" p.write_text("\n".join([",".join(header)] + [",".join(r) for r in rows]) + "\n") return str(p) @@ -14,12 +14,11 @@ def _csv(tmp_path, rows, header): def test_read_panel_one_row_per_tag_sample(tmp_path): path = _csv( tmp_path, - [ + rows=[ ["S1", "AgA", "AAAA", "Off-Target"], ["S2", "AgB", "AAAA", "Off-Target"], ["S1", "AgC", "CCCC", "Target"], ], - ["Samples", "Name", "Sequence", "Type"], ) panel, dropped = read_panel(path, ROLES) assert panel.height == 3 @@ -28,7 +27,7 @@ def test_read_panel_one_row_per_tag_sample(tmp_path): def test_read_panel_without_sample_column_uses_star(tmp_path): - path = _csv(tmp_path, [["AgA", "AAAA"], ["AgB", "CCCC"]], ["Name", "Sequence"]) + path = _csv(tmp_path, header=("Name", "Sequence"), rows=[["AgA", "AAAA"], ["AgB", "CCCC"]]) panel, _ = read_panel(path, {"barcode": "Sequence", "feature": "Name", "sample": ""}) assert panel["sample"].unique().to_list() == ["*"] @@ -71,24 +70,19 @@ def test_consistent_properties_ignores_blanks(): def test_duplicate_tag_sample_pair_is_fatal(tmp_path): - path = _csv( - tmp_path, - [["S1", "AgA", "AAAA", "Target"], ["S1", "AgB", "AAAA", "Target"]], - ["Samples", "Name", "Sequence", "Type"], - ) + path = _csv(tmp_path, rows=[["S1", "AgA", "AAAA", "Target"], ["S1", "AgB", "AAAA", "Target"]]) with pytest.raises(SystemExit) as e: read_panel(path, ROLES) - assert "AAAA" in str(e.value) + assert "AAAA/S1" in str(e.value) def test_blank_barcode_row_is_reported_not_dropped(tmp_path): path = _csv( tmp_path, - [ + rows=[ ["S1", "AgA", "AAAA", "Target"], ["S1", "AgB", "", "Target"], ], - ["Samples", "Name", "Sequence", "Type"], ) panel, dropped = read_panel(path, ROLES) assert panel.height == 1 @@ -99,15 +93,14 @@ def test_blank_barcode_row_is_reported_not_dropped(tmp_path): def test_blank_sample_cell_is_fatal(tmp_path): path = _csv( tmp_path, - [ + rows=[ ["S1", "AgA", "AAAA", "Target"], ["", "AgB", "CCCC", "Target"], ], - ["Samples", "Name", "Sequence", "Type"], ) with pytest.raises(SystemExit) as e: read_panel(path, ROLES) - assert "3" in str(e.value) + assert "line(s) 3." in str(e.value) def test_trailing_blank_line_is_not_a_blank_sample_cell(tmp_path): @@ -121,25 +114,21 @@ def test_trailing_blank_line_is_not_a_blank_sample_cell(tmp_path): assert dropped == [3] -def test_reserved_column_name_is_fatal(tmp_path): - # A NON-role column named "tag" would be overwritten by the one this - # reader produces, so it is refused rather than silently shadowed. - path = _csv(tmp_path, [["S1", "AgA", "AAAA", "x"]], ["Samples", "Name", "Sequence", "tag"]) - with pytest.raises(SystemExit) as e: - read_panel(path, ROLES) - assert "tag" in str(e.value) - - path = _csv(tmp_path, [["S1", "AgA", "AAAA", "x"]], ["Samples", "Name", "Sequence", "sample"]) +@pytest.mark.parametrize("bad_name", ["tag", "sample"]) +def test_reserved_column_name_is_fatal(tmp_path, bad_name): + # A NON-role column named "tag"/"sample" would be overwritten by the one + # this reader produces, so it is refused rather than silently shadowed. + path = _csv(tmp_path, header=("Samples", "Name", "Sequence", bad_name), rows=[["S1", "AgA", "AAAA", "x"]]) with pytest.raises(SystemExit) as e: read_panel(path, ROLES) - assert "sample" in str(e.value) + assert f"['{bad_name}']" in str(e.value) def test_role_column_may_be_named_tag(tmp_path): # emit_panel.py in this package documents this very shape and defaults # --tag-col to "tag". A role column cannot collide: alias() replaces the # source column rather than duplicating it. - path = _csv(tmp_path, [["S1", "AgA", "AAAA"]], ["sample", "feature", "tag"]) + path = _csv(tmp_path, header=("sample", "feature", "tag"), rows=[["S1", "AgA", "AAAA"]]) panel, dropped = read_panel(path, {"barcode": "tag", "feature": "feature", "sample": "sample"}) assert panel.height == 1 assert panel["tag"].to_list() == ["AAAA"] @@ -151,10 +140,10 @@ def test_sample_role_named_tag_is_fatal(tmp_path): # The barcode alias runs first and would overwrite this column, leaving # "sample" a silent copy of the barcode — per-sample keying gone, and no # duplicate raised because the pairs stay unique. Refused, not corrected. - path = _csv(tmp_path, [["S1", "AgA", "AAAA"]], ["tag", "Name", "Sequence"]) + path = _csv(tmp_path, header=("tag", "Name", "Sequence"), rows=[["S1", "AgA", "AAAA"]]) with pytest.raises(SystemExit) as e: read_panel(path, {"barcode": "Sequence", "feature": "Name", "sample": "tag"}) - assert "tag" in str(e.value) + assert "['tag']" in str(e.value) def test_two_blank_barcode_rows_are_not_a_duplicate(tmp_path): @@ -162,12 +151,11 @@ def test_two_blank_barcode_rows_are_not_a_duplicate(tmp_path): # pair if the blank-barcode filter ran after the dupe check. path = _csv( tmp_path, - [ + rows=[ ["S1", "AgA", "AAAA", "Target"], ["S1", "AgB", "", "Target"], ["S1", "AgC", "", "Target"], ], - ["Samples", "Name", "Sequence", "Type"], ) panel, dropped = read_panel(path, ROLES) assert panel.height == 1 @@ -175,8 +163,53 @@ def test_two_blank_barcode_rows_are_not_a_duplicate(tmp_path): def test_two_roles_on_one_column_is_fatal(tmp_path): - # Reachable from the UI: the Sample-column dropdown is unfiltered. - path = _csv(tmp_path, [["S1", "AgA", "AAAA"]], ["Samples", "Name", "Sequence"]) + # Two roles on one column silently makes "sample" a copy of "tag" — + # reachable from the UI today, since the Sample-column dropdown is + # unfiltered. + path = _csv(tmp_path, header=("Samples", "Name", "Sequence"), rows=[["S1", "AgA", "AAAA"]]) with pytest.raises(SystemExit) as e: read_panel(path, {"barcode": "Sequence", "feature": "Name", "sample": "Sequence"}) - assert "Sequence" in str(e.value) + assert "column 'Sequence'" in str(e.value) + + +def test_missing_barcode_column_is_fatal(tmp_path): + path = _csv(tmp_path, rows=[["S1", "AgA", "AAAA", "Target"]]) + with pytest.raises(SystemExit) as e: + read_panel(path, {"barcode": "NoSuchCol", "feature": "Name", "sample": "Samples"}) + assert "no barcode column 'NoSuchCol'" in str(e.value) + + +def test_missing_sample_column_is_fatal(tmp_path): + path = _csv(tmp_path, rows=[["S1", "AgA", "AAAA", "Target"]]) + with pytest.raises(SystemExit) as e: + read_panel(path, {"barcode": "Sequence", "feature": "Name", "sample": "NoSuchCol"}) + assert "no sample column 'NoSuchCol'" in str(e.value) + + +def test_literal_row_header_is_fatal(tmp_path): + path = _csv(tmp_path, header=("Samples", "Name", "Sequence", "_row"), rows=[["S1", "AgA", "AAAA", "x"]]) + with pytest.raises(SystemExit) as e: + read_panel(path, ROLES) + assert "['_row']" in str(e.value) + + +def test_feature_role_named_tag_is_fatal(tmp_path): + # Pins that the rev-8 barcode-only "tag" exemption stays narrow: a FEATURE + # role named "tag" is not covered by it. + path = _csv(tmp_path, header=("Samples", "Sequence", "tag"), rows=[["S1", "AAAA", "AgA"]]) + with pytest.raises(SystemExit) as e: + read_panel(path, {"barcode": "Sequence", "feature": "tag", "sample": "Samples"}) + assert "['tag']" in str(e.value) + + +def test_property_columns_excludes_tag_and_sample_and_preserves_order(): + # Downstream column layout depends on source order being preserved. + panel = pl.DataFrame({"Type": ["Target"], "tag": ["AAAA"], "Name": ["AgA"], "sample": ["S1"], "Channel": ["APC"]}) + assert property_columns(panel) == ["Type", "Name", "Channel"] + + +def test_barcode_is_stripped(tmp_path): + # tag equality is the join key for every later task. + path = _csv(tmp_path, rows=[["S1", "AgA", " AAAA ", "Target"]]) + panel, _ = read_panel(path, ROLES) + assert panel["tag"].to_list() == ["AAAA"] From 01afd0c0fb0b835a343ccfea4a8364b9e9fe7844 Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 13:01:37 +0200 Subject: [PATCH 016/282] MILAB-6496: floor per cell and tag, exempting the comparator --- software/per-cell-metrics/src/verdict.py | 73 +++++++++++++++++ .../per-cell-metrics/test/test_verdict.py | 81 +++++++++++++++++++ 2 files changed, 154 insertions(+) create mode 100644 software/per-cell-metrics/src/verdict.py create mode 100644 software/per-cell-metrics/test/test_verdict.py diff --git a/software/per-cell-metrics/src/verdict.py b/software/per-cell-metrics/src/verdict.py new file mode 100644 index 0000000..126f246 --- /dev/null +++ b/software/per-cell-metrics/src/verdict.py @@ -0,0 +1,73 @@ +"""Turning a cell's counts into states. + +Five steps, in this order, and the order is load-bearing: + + 1. the floor, on the raw count, per cell and per tag; + 2. densify — every cell against every identity its sample offered, so a cell + asked and silent is a real zero rather than a missing row; + 3. tags combine into an identity by the highest of their counts; + 4. the identity's count is read against that cell's own reference reading; + 5. the comparison becomes one of the four states. + +Step 2 exists because tag-stat emits only observed pairs. Without it an antigen +every cell failed to bind produces no rows at all, and the absence is +indistinguishable from a reagent nobody offered. + +The cell key is (sampleId, cellId) throughout: cell barcodes are bare 16-mers +shared across samples. + +This module implements step 1 only. +""" + +from __future__ import annotations + +import polars as pl + +CELL_KEY = ["sampleId", "cellId"] + +# The value the antibody-side lineage uses and later work inherited. Not a +# calibrated line, which is why it ships as a declared default the scientist +# can move rather than as a constant. +DEFAULT_FLOOR = 4 + + +def apply_floor(counts: pl.DataFrame, floor: int, reference_tags: set[str]) -> tuple[pl.DataFrame, dict[str, int]]: + """Zero every (cell, tag) count below `floor`, except the comparator's. + + A floored count contributes exactly as a count of zero does — the position + reads *not bound*, not *unreliable*. The floor is not a statement that the + reading could not be settled; it is that a count that small is not + distinguishable from none. + + Reference tags are exempt. The floor removes what is not evidence *of + binding*; the comparator is not evidence of binding, and flooring it lowers + every denominator and shifts the whole run toward *bound*. + + Scope limit: `reference_tags` is global, so a tag is a comparator in every + sample or in none. The panel is keyed (tag, sample) and can in principle + declare a barcode a control in one sample and a real antigen in another; + that case is NOT handled here, and Task 1's consistent_properties() would + already have dropped such a divergent role rather than honouring it per + sample. Revisit when the reference is resolved (Tasks 5 and 13), not here. + + Returns the floored counts and {"readingsFloored", "cellsEmptied"}, the two + quantities the quality measurement set asks of this step. + """ + if floor <= 0: + return counts, {"readingsFloored": 0, "cellsEmptied": 0} + + is_ref = pl.col("tag").is_in(list(reference_tags)) if reference_tags else pl.lit(False) + below = (pl.col("umiCount") < floor) & ~is_ref + + readings_floored = int(counts.select(below.sum()).item()) + out = counts.with_columns( + pl.when(below).then(pl.lit(0, dtype=pl.Int64)).otherwise(pl.col("umiCount")).alias("umiCount") + ) + + # "Emptied" is scoped to non-reference readings: a cell holding only the + # comparator never had evidence of binding for the floor to remove. + before = counts.filter(~is_ref).group_by(CELL_KEY).agg(pl.len().alias("n")) + after = out.filter(~is_ref).filter(pl.col("umiCount") > 0).group_by(CELL_KEY).agg(pl.len().alias("n")) + cells_emptied = before.join(after, on=CELL_KEY, how="anti").height + + return out, {"readingsFloored": readings_floored, "cellsEmptied": cells_emptied} diff --git a/software/per-cell-metrics/test/test_verdict.py b/software/per-cell-metrics/test/test_verdict.py new file mode 100644 index 0000000..fac14c1 --- /dev/null +++ b/software/per-cell-metrics/test/test_verdict.py @@ -0,0 +1,81 @@ +import polars as pl +from verdict import DEFAULT_FLOOR, apply_floor + + +def _counts(rows): + return pl.DataFrame( + rows, orient="row", schema={"sampleId": pl.String, "cellId": pl.String, "tag": pl.String, "umiCount": pl.Int64} + ) + + +def test_default_floor_is_four(): + assert DEFAULT_FLOOR == 4 + + +def test_counts_below_the_floor_become_zero(): + df = _counts([("S1", "c1", "AAAA", 3), ("S1", "c1", "CCCC", 4)]) + out, stats = apply_floor(df, floor=4, reference_tags=set()) + assert out.sort("tag")["umiCount"].to_list() == [0, 4] + assert stats["readingsFloored"] == 1 + + +def test_floor_is_per_cell_and_tag_not_per_cell_total(): + df = _counts([("S1", "c1", "AAAA", 3), ("S1", "c1", "CCCC", 3)]) + out, stats = apply_floor(df, floor=4, reference_tags=set()) + assert out["umiCount"].to_list() == [0, 0] + assert stats["readingsFloored"] == 2 + + +def test_reference_tags_are_never_floored(): + df = _counts([("S1", "c1", "CTRL", 1), ("S1", "c1", "AAAA", 1)]) + out, _ = apply_floor(df, floor=4, reference_tags={"CTRL"}) + got = dict(zip(out["tag"].to_list(), out["umiCount"].to_list(), strict=True)) + assert got["CTRL"] == 1 # the comparator is not evidence of binding + assert got["AAAA"] == 0 + + +def test_cells_left_with_nothing_are_counted(): + df = _counts([("S1", "c1", "AAAA", 1), ("S1", "c2", "AAAA", 9)]) + _, stats = apply_floor(df, floor=4, reference_tags=set()) + assert stats["cellsEmptied"] == 1 + + +def test_same_barcode_in_two_samples_stays_two_cells(): + df = _counts([("S1", "c1", "AAAA", 9), ("S2", "c1", "AAAA", 9)]) + out, _ = apply_floor(df, floor=4, reference_tags=set()) + assert out.height == 2 + + +def test_floor_of_zero_removes_nothing(): + df = _counts([("S1", "c1", "AAAA", 1)]) + out, stats = apply_floor(df, floor=0, reference_tags=set()) + assert out["umiCount"].to_list() == [1] and stats["readingsFloored"] == 0 + + +def test_count_exactly_at_the_floor_survives(): + df = _counts([("S1", "c1", "AAAA", 4)]) + out, _ = apply_floor(df, floor=4, reference_tags=set()) + assert out["umiCount"].to_list() == [4] + + +def test_a_cell_holding_only_the_reference_is_not_emptied(): + # Its non-reference readings are absent, not zeroed. "Emptied" means the + # floor took a cell's evidence away, not that it never had any. + df = _counts([("S1", "c1", "CTRL", 1)]) + _, stats = apply_floor(df, floor=4, reference_tags={"CTRL"}) + assert stats["cellsEmptied"] == 0 + + +def test_a_cell_keeping_one_reading_is_not_emptied(): + df = _counts([("S1", "c1", "AAAA", 1), ("S1", "c1", "CCCC", 9)]) + _, stats = apply_floor(df, floor=4, reference_tags=set()) + assert stats["cellsEmptied"] == 0 + assert stats["readingsFloored"] == 1 + + +def test_the_same_cell_id_in_two_samples_empties_independently(): + # (sampleId, cellId) is the key. Keying on cellId alone would let S2's + # surviving reading rescue S1's emptied cell. + df = _counts([("S1", "c1", "AAAA", 1), ("S2", "c1", "AAAA", 9)]) + _, stats = apply_floor(df, floor=4, reference_tags=set()) + assert stats["cellsEmptied"] == 1 From 909625ba49200d9c3b2005006b1eb085828959c6 Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 13:04:01 +0200 Subject: [PATCH 017/282] MILAB-6496: pin the disabled-floor short circuit --- software/per-cell-metrics/src/verdict.py | 2 ++ software/per-cell-metrics/test/test_verdict.py | 10 ++++++++++ 2 files changed, 12 insertions(+) diff --git a/software/per-cell-metrics/src/verdict.py b/software/per-cell-metrics/src/verdict.py index 126f246..292c023 100644 --- a/software/per-cell-metrics/src/verdict.py +++ b/software/per-cell-metrics/src/verdict.py @@ -53,6 +53,8 @@ def apply_floor(counts: pl.DataFrame, floor: int, reference_tags: set[str]) -> t Returns the floored counts and {"readingsFloored", "cellsEmptied"}, the two quantities the quality measurement set asks of this step. """ + # Not an optimisation: falling through would count a cell whose only + # reading is already zero as "emptied", when the floor removed nothing. if floor <= 0: return counts, {"readingsFloored": 0, "cellsEmptied": 0} diff --git a/software/per-cell-metrics/test/test_verdict.py b/software/per-cell-metrics/test/test_verdict.py index fac14c1..97a11e2 100644 --- a/software/per-cell-metrics/test/test_verdict.py +++ b/software/per-cell-metrics/test/test_verdict.py @@ -79,3 +79,13 @@ def test_the_same_cell_id_in_two_samples_empties_independently(): df = _counts([("S1", "c1", "AAAA", 1), ("S2", "c1", "AAAA", 9)]) _, stats = apply_floor(df, floor=4, reference_tags=set()) assert stats["cellsEmptied"] == 1 + + +def test_a_disabled_floor_is_a_no_op_even_for_a_zero_reading(): + # floor <= 0 returns early, and that early return is behavioural rather + # than an optimisation: falling through would count a cell whose only + # reading is already 0 as "emptied", when the floor removed nothing. + df = _counts([("S1", "c1", "AAAA", 0)]) + out, stats = apply_floor(df, floor=0, reference_tags=set()) + assert out["umiCount"].to_list() == [0] + assert stats == {"readingsFloored": 0, "cellsEmptied": 0} From d41e441e33c98256cba1286ecdc91378f25e4ca3 Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 13:08:42 +0200 Subject: [PATCH 018/282] MILAB-6496: state the floor's assumptions without citing the plan --- software/per-cell-metrics/src/verdict.py | 22 +++++++++++++++++----- 1 file changed, 17 insertions(+), 5 deletions(-) diff --git a/software/per-cell-metrics/src/verdict.py b/software/per-cell-metrics/src/verdict.py index 292c023..42feea4 100644 --- a/software/per-cell-metrics/src/verdict.py +++ b/software/per-cell-metrics/src/verdict.py @@ -44,20 +44,32 @@ def apply_floor(counts: pl.DataFrame, floor: int, reference_tags: set[str]) -> t every denominator and shifts the whole run toward *bound*. Scope limit: `reference_tags` is global, so a tag is a comparator in every - sample or in none. The panel is keyed (tag, sample) and can in principle - declare a barcode a control in one sample and a real antigen in another; - that case is NOT handled here, and Task 1's consistent_properties() would - already have dropped such a divergent role rather than honouring it per - sample. Revisit when the reference is resolved (Tasks 5 and 13), not here. + sample or in none. The panel is keyed (tag, sample) and could in principle + declare a barcode a control in one sample and a real antigen in another. + That case is not handled here — and the panel reader's consistent_properties() + drops any property whose value disagrees across a tag's rows, so a + per-sample control designation would already have been discarded rather + than honoured. Handling it belongs where the reference is selected and + where the CLI resolves it, not in the floor. Returns the floored counts and {"readingsFloored", "cellsEmptied"}, the two quantities the quality measurement set asks of this step. + + Both counters assume the sparse frame this step receives, where every row + is an observed reading and so a count is at least 1. Densification, which + manufactures genuine zeros, happens after this step: run it before, and + every manufactured row inflates readingsFloored while every unbound cell + counts as emptied though the floor removed nothing. """ # Not an optimisation: falling through would count a cell whose only # reading is already zero as "emptied", when the floor removed nothing. if floor <= 0: return counts, {"readingsFloored": 0, "cellsEmptied": 0} + # is_in yields null for a null tag, so a null-tag row would escape both the + # floor and the emptied populations here while flooring normally when no + # reference is declared. The panel reader never emits one; this is a note + # for anyone who feeds this an unvalidated frame. is_ref = pl.col("tag").is_in(list(reference_tags)) if reference_tags else pl.lit(False) below = (pl.col("umiCount") < floor) & ~is_ref From 0a9c6bc41da5ce61f2b67968f892da45167f60cb Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 13:17:09 +0200 Subject: [PATCH 019/282] MILAB-6496: shape the verdict module for the steps that follow --- software/per-cell-metrics/src/verdict.py | 42 +++++++++++++------ .../per-cell-metrics/test/test_verdict.py | 12 +++++- 2 files changed, 39 insertions(+), 15 deletions(-) diff --git a/software/per-cell-metrics/src/verdict.py b/software/per-cell-metrics/src/verdict.py index 42feea4..e30a0a5 100644 --- a/software/per-cell-metrics/src/verdict.py +++ b/software/per-cell-metrics/src/verdict.py @@ -16,22 +16,36 @@ The cell key is (sampleId, cellId) throughout: cell barcodes are bare 16-mers shared across samples. -This module implements step 1 only. +Compare `min_umi` in per_cell_metrics.py, which is also a UMI threshold on +this data but resolves the other way: a barcode below it makes the feature +absent for that cell, omitted rather than zeroed. Both keep a reading exactly +at the threshold. The difference is the whole point of the floor — a floored +reading is still a reading, and it answers "not bound"; an omitted one leaves +nothing to answer with. + +After step 3 this module holds two frame shapes: the sparse per-tag frame the +floor works on, and the per-identity frame combining produces from it — both +keyed by CELL_KEY, which is the column vocabulary spanning both. """ from __future__ import annotations +from typing import NamedTuple + import polars as pl -CELL_KEY = ["sampleId", "cellId"] +CELL_KEY = ("sampleId", "cellId") -# The value the antibody-side lineage uses and later work inherited. Not a -# calibrated line, which is why it ships as a declared default the scientist -# can move rather than as a constant. +# Uncalibrated: a declared default the scientist can move, not a fitted line. DEFAULT_FLOOR = 4 -def apply_floor(counts: pl.DataFrame, floor: int, reference_tags: set[str]) -> tuple[pl.DataFrame, dict[str, int]]: +class Floored(NamedTuple): + counts: pl.DataFrame + stats: dict[str, int] + + +def apply_floor(counts: pl.DataFrame, floor: int, reference_tags: set[str]) -> Floored: """Zero every (cell, tag) count below `floor`, except the comparator's. A floored count contributes exactly as a count of zero does — the position @@ -52,8 +66,8 @@ def apply_floor(counts: pl.DataFrame, floor: int, reference_tags: set[str]) -> t than honoured. Handling it belongs where the reference is selected and where the CLI resolves it, not in the floor. - Returns the floored counts and {"readingsFloored", "cellsEmptied"}, the two - quantities the quality measurement set asks of this step. + Returns the floored counts and {"readingsFloored", "cellsEmptied"}: the two + counters that land in this sample's row of the QC report. Both counters assume the sparse frame this step receives, where every row is an observed reading and so a count is at least 1. Densification, which @@ -64,7 +78,7 @@ def apply_floor(counts: pl.DataFrame, floor: int, reference_tags: set[str]) -> t # Not an optimisation: falling through would count a cell whose only # reading is already zero as "emptied", when the floor removed nothing. if floor <= 0: - return counts, {"readingsFloored": 0, "cellsEmptied": 0} + return Floored(counts, {"readingsFloored": 0, "cellsEmptied": 0}) # is_in yields null for a null tag, so a null-tag row would escape both the # floor and the emptied populations here while flooring normally when no @@ -80,8 +94,10 @@ def apply_floor(counts: pl.DataFrame, floor: int, reference_tags: set[str]) -> t # "Emptied" is scoped to non-reference readings: a cell holding only the # comparator never had evidence of binding for the floor to remove. - before = counts.filter(~is_ref).group_by(CELL_KEY).agg(pl.len().alias("n")) - after = out.filter(~is_ref).filter(pl.col("umiCount") > 0).group_by(CELL_KEY).agg(pl.len().alias("n")) - cells_emptied = before.join(after, on=CELL_KEY, how="anti").height + # had_evidence deliberately does not filter on umiCount > 0 — that absence + # is the sparse-frame assumption above, not an oversight to "symmetrise". + had_evidence = counts.filter(~is_ref).select(CELL_KEY).unique() + kept_evidence = out.filter(~is_ref & (pl.col("umiCount") > 0)).select(CELL_KEY).unique() + cells_emptied = had_evidence.join(kept_evidence, on=CELL_KEY, how="anti").height - return out, {"readingsFloored": readings_floored, "cellsEmptied": cells_emptied} + return Floored(out, {"readingsFloored": readings_floored, "cellsEmptied": cells_emptied}) diff --git a/software/per-cell-metrics/test/test_verdict.py b/software/per-cell-metrics/test/test_verdict.py index 97a11e2..3158843 100644 --- a/software/per-cell-metrics/test/test_verdict.py +++ b/software/per-cell-metrics/test/test_verdict.py @@ -40,7 +40,7 @@ def test_cells_left_with_nothing_are_counted(): assert stats["cellsEmptied"] == 1 -def test_same_barcode_in_two_samples_stays_two_cells(): +def test_the_floor_zeroes_readings_it_never_drops_rows(): df = _counts([("S1", "c1", "AAAA", 9), ("S2", "c1", "AAAA", 9)]) out, _ = apply_floor(df, floor=4, reference_tags=set()) assert out.height == 2 @@ -49,7 +49,8 @@ def test_same_barcode_in_two_samples_stays_two_cells(): def test_floor_of_zero_removes_nothing(): df = _counts([("S1", "c1", "AAAA", 1)]) out, stats = apply_floor(df, floor=0, reference_tags=set()) - assert out["umiCount"].to_list() == [1] and stats["readingsFloored"] == 0 + assert out["umiCount"].to_list() == [1] + assert stats["readingsFloored"] == 0 def test_count_exactly_at_the_floor_survives(): @@ -89,3 +90,10 @@ def test_a_disabled_floor_is_a_no_op_even_for_a_zero_reading(): out, stats = apply_floor(df, floor=0, reference_tags=set()) assert out["umiCount"].to_list() == [0] assert stats == {"readingsFloored": 0, "cellsEmptied": 0} + + +def test_an_empty_frame_floors_to_nothing(): + df = _counts([]) + out, stats = apply_floor(df, floor=4, reference_tags=set()) + assert out.height == 0 + assert stats == {"readingsFloored": 0, "cellsEmptied": 0} From 355a9e47d67140e419d89ce4439226b33d3b241d Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 13:19:36 +0200 Subject: [PATCH 020/282] MILAB-6496: pin the asymmetry the emptied count depends on --- software/per-cell-metrics/test/test_verdict.py | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/software/per-cell-metrics/test/test_verdict.py b/software/per-cell-metrics/test/test_verdict.py index 3158843..6356090 100644 --- a/software/per-cell-metrics/test/test_verdict.py +++ b/software/per-cell-metrics/test/test_verdict.py @@ -97,3 +97,15 @@ def test_an_empty_frame_floors_to_nothing(): out, stats = apply_floor(df, floor=4, reference_tags=set()) assert out.height == 0 assert stats == {"readingsFloored": 0, "cellsEmptied": 0} + + +def test_a_reading_that_was_already_zero_still_counts_as_evidence_lost(): + # Pins the deliberate asymmetry between had_evidence and kept_evidence. + # had_evidence must NOT filter on > 0: on the sparse frame this step is + # contracted to receive, every row is an observed reading, so a row's + # existence is what makes a cell one that had evidence. Adding "> 0" to + # had_evidence is a no-op on real input and silently changes this count + # once densified zeros exist — which is the reason densify runs after. + df = _counts([("S1", "c1", "AAAA", 0)]) + _, stats = apply_floor(df, floor=4, reference_tags=set()) + assert stats["cellsEmptied"] == 1 From 8c19610a1dffe7da13118b2c63607c1cf4c04417 Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 13:23:05 +0200 Subject: [PATCH 021/282] MILAB-6496: identity universe and per-sample offered derivation --- software/per-cell-metrics/src/panel.py | 38 +++++++++ software/per-cell-metrics/test/test_panel.py | 87 +++++++++++++++++++- 2 files changed, 124 insertions(+), 1 deletion(-) diff --git a/software/per-cell-metrics/src/panel.py b/software/per-cell-metrics/src/panel.py index 0d3bcfd..5ff0b03 100644 --- a/software/per-cell-metrics/src/panel.py +++ b/software/per-cell-metrics/src/panel.py @@ -179,3 +179,41 @@ def consistent_properties( elif len(values) > 1: inconsistent.append((name, col, values)) return props, inconsistent + + +def default_grouping(panel: pl.DataFrame, reference_tags: set[str]) -> dict[str, str]: + """One identity per tag, over non-reference tags. + + The feature name cannot key an identity: the same barcode carries a + different name in a different sample's panel, so name-keying splits one + reagent and can merge two. The reference is a comparator and never an + identity. + """ + return {t: t for t in panel["tag"].unique().to_list() if t not in reference_tags} + + +def identity_universe(panel: pl.DataFrame, grouping: dict[str, str]) -> set[str]: + """Every identity a question is asked at — the row set for every set's verdicts. + + A verdict exists at every identity, including ones a given set was never + offered: that is precisely where *never asked* lives. Using the offered set + as the row set instead makes an unoffered identity vanish from the answer. + """ + return {grouping[t] for t in panel["tag"].to_list() if t in grouping} + + +def offered_identities(panel: pl.DataFrame, grouping: dict[str, str], samples: list[str]) -> set[str]: + """Which identities a set was offered, given the samples its cells came from. + + An identity was offered when any one of its tags was on any of those + samples' panels. The `any` is deliberate: an identity is a group of tags, + and that group can span several panels. + + A sample the panel never mentions is offered nothing, so every identity + reads *never asked* for a set drawn from it. That is the honest reading of + a panel that does not cover the run, and the panel-versus-reads check is + what makes the gap visible rather than leaving it to be inferred. + """ + wanted = set(samples) + rows = panel.filter((pl.col("sample") == ANY_SAMPLE) | pl.col("sample").is_in(list(wanted))) + return {grouping[t] for t in rows["tag"].to_list() if t in grouping} diff --git a/software/per-cell-metrics/test/test_panel.py b/software/per-cell-metrics/test/test_panel.py index 0e6281b..a318c6f 100644 --- a/software/per-cell-metrics/test/test_panel.py +++ b/software/per-cell-metrics/test/test_panel.py @@ -1,6 +1,13 @@ import polars as pl import pytest -from panel import consistent_properties, property_columns, read_panel +from panel import ( + consistent_properties, + default_grouping, + identity_universe, + offered_identities, + property_columns, + read_panel, +) ROLES = {"barcode": "Sequence", "feature": "Name", "sample": "Samples"} @@ -213,3 +220,81 @@ def test_barcode_is_stripped(tmp_path): path = _csv(tmp_path, rows=[["S1", "AgA", " AAAA ", "Target"]]) panel, _ = read_panel(path, ROLES) assert panel["tag"].to_list() == ["AAAA"] + + +def test_universe_is_every_identity_not_a_per_set_subset(): + panel = pl.DataFrame({"tag": ["AAAA", "CCCC", "GGGG"], "sample": ["S1", "S2", "S2"], "Name": ["a", "c", "g"]}) + g = {"AAAA": "A", "CCCC": "C", "GGGG": "G"} + assert identity_universe(panel, g) == {"A", "C", "G"} + + +def test_reference_tags_never_enter_the_universe(): + panel = pl.DataFrame({"tag": ["AAAA", "CTRL"], "sample": ["S1", "S1"], "Name": ["a", "ctrl"]}) + g = default_grouping(panel, reference_tags={"CTRL"}) + assert g == {"AAAA": "AAAA"} + assert identity_universe(panel, g) == {"AAAA"} + + +def test_offered_is_the_union_over_the_sets_samples(): + panel = pl.DataFrame({"tag": ["AAAA", "CCCC", "GGGG"], "sample": ["S1", "S2", "S2"], "Name": ["a", "c", "g"]}) + g = {"AAAA": "A", "CCCC": "C", "GGGG": "G"} + assert offered_identities(panel, g, ["S1"]) == {"A"} + assert offered_identities(panel, g, ["S2"]) == {"C", "G"} + assert offered_identities(panel, g, ["S1", "S2"]) == {"A", "C", "G"} + + +def test_offered_needs_only_one_member_tag(): + panel = pl.DataFrame({"tag": ["AAAA", "CCCC"], "sample": ["S1", "S2"], "Name": ["a1", "a2"]}) + g = {"AAAA": "A", "CCCC": "A"} + assert offered_identities(panel, g, ["S1"]) == {"A"} + assert offered_identities(panel, g, ["S2"]) == {"A"} + + +def test_star_sample_offers_everything(): + panel = pl.DataFrame({"tag": ["AAAA", "CCCC"], "sample": ["*", "*"], "Name": ["a", "c"]}) + g = {"AAAA": "A", "CCCC": "C"} + assert offered_identities(panel, g, ["anything"]) == {"A", "C"} + + +def test_an_identity_not_offered_is_still_in_the_universe(): + # The universe is what makes a never-asked ROW exist. An earlier revision + # used `offered` as the row set, so a never-offered identity vanished + # instead of reading "never asked". + panel = pl.DataFrame({"tag": ["AAAA", "CCCC"], "sample": ["S1", "S2"], "Name": ["a", "c"]}) + g = {"AAAA": "A", "CCCC": "C"} + assert "C" in identity_universe(panel, g) + assert "C" not in offered_identities(panel, g, ["S1"]) + + +def test_a_sample_absent_from_the_panel_is_offered_nothing(): + # A set whose cells came from a sample the panel never mentions was + # offered nothing, so every identity reads "never asked" for it. That is + # the honest answer, not a bug — but it is a big claim from a silent + # lookup, so it is pinned here and reported by the panel/reads check. + panel = pl.DataFrame({"tag": ["AAAA"], "sample": ["S1"], "Name": ["a"]}) + g = {"AAAA": "A"} + assert offered_identities(panel, g, ["S9"]) == set() + + +def test_a_panel_of_only_references_has_an_empty_universe(): + panel = pl.DataFrame({"tag": ["CTRL"], "sample": ["S1"], "Name": ["ctrl"]}) + g = default_grouping(panel, reference_tags={"CTRL"}) + assert g == {} + assert identity_universe(panel, g) == set() + + +def test_no_samples_offers_nothing_but_the_star(): + # An empty sample list must not accidentally mean "all samples". + panel = pl.DataFrame({"tag": ["AAAA"], "sample": ["S1"], "Name": ["a"]}) + assert offered_identities(panel, {"AAAA": "A"}, []) == set() + star = pl.DataFrame({"tag": ["AAAA"], "sample": ["*"], "Name": ["a"]}) + assert offered_identities(star, {"AAAA": "A"}, []) == {"A"} + + +def test_a_tag_outside_the_grouping_is_skipped_not_an_error(): + # The reference is the ordinary case of this: it is on the panel and + # deliberately absent from the grouping. + panel = pl.DataFrame({"tag": ["AAAA", "ZZZZ"], "sample": ["S1", "S1"], "Name": ["a", "z"]}) + g = {"AAAA": "A"} + assert identity_universe(panel, g) == {"A"} + assert offered_identities(panel, g, ["S1"]) == {"A"} From 60563649ee49127a87cf5a8401089e36849915e3 Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 13:32:12 +0200 Subject: [PATCH 022/282] MILAB-6496: name the grouping type and state the identity contract --- software/per-cell-metrics/src/panel.py | 29 +++++++++++++++++++------- 1 file changed, 21 insertions(+), 8 deletions(-) diff --git a/software/per-cell-metrics/src/panel.py b/software/per-cell-metrics/src/panel.py index 5ff0b03..f786b0d 100644 --- a/software/per-cell-metrics/src/panel.py +++ b/software/per-cell-metrics/src/panel.py @@ -8,6 +8,11 @@ A tag is the barcode sequence. The feature name is a declared property, not an identity: a name only travels where every row for that tag agrees on it. + +An identity is a group of tags asked as one question. The universe is every +identity a verdict row exists at; offered is the subset a given set of cells +was actually presented. The universe always contains offered — and that gap is +where "never asked" lives. """ from __future__ import annotations @@ -181,28 +186,37 @@ def consistent_properties( return props, inconsistent -def default_grouping(panel: pl.DataFrame, reference_tags: set[str]) -> dict[str, str]: +# tag -> the identity that tag belongs to. Many tags may share one identity. +# A tag absent from the mapping gets no verdict row, which is why every builder +# must leave the reference tags out: the comparator has nothing to be compared +# against. identity_universe() takes no reference_tags of its own, deliberately — +# one place decides, so the two cannot drift. +Grouping = dict[str, str] + + +def default_grouping(panel: pl.DataFrame, reference_tags: set[str]) -> Grouping: """One identity per tag, over non-reference tags. The feature name cannot key an identity: the same barcode carries a different name in a different sample's panel, so name-keying splits one reagent and can merge two. The reference is a comparator and never an - identity. + identity — a verdict is a reading against the reference, so asking one of + the reference would compare it with itself. """ return {t: t for t in panel["tag"].unique().to_list() if t not in reference_tags} -def identity_universe(panel: pl.DataFrame, grouping: dict[str, str]) -> set[str]: +def identity_universe(panel: pl.DataFrame, grouping: Grouping) -> set[str]: """Every identity a question is asked at — the row set for every set's verdicts. A verdict exists at every identity, including ones a given set was never offered: that is precisely where *never asked* lives. Using the offered set as the row set instead makes an unoffered identity vanish from the answer. """ - return {grouping[t] for t in panel["tag"].to_list() if t in grouping} + return {grouping[t] for t in panel["tag"].unique().to_list() if t in grouping} -def offered_identities(panel: pl.DataFrame, grouping: dict[str, str], samples: list[str]) -> set[str]: +def offered_identities(panel: pl.DataFrame, grouping: Grouping, samples: list[str]) -> set[str]: """Which identities a set was offered, given the samples its cells came from. An identity was offered when any one of its tags was on any of those @@ -214,6 +228,5 @@ def offered_identities(panel: pl.DataFrame, grouping: dict[str, str], samples: l a panel that does not cover the run, and the panel-versus-reads check is what makes the gap visible rather than leaving it to be inferred. """ - wanted = set(samples) - rows = panel.filter((pl.col("sample") == ANY_SAMPLE) | pl.col("sample").is_in(list(wanted))) - return {grouping[t] for t in rows["tag"].to_list() if t in grouping} + rows = panel.filter((pl.col("sample") == ANY_SAMPLE) | pl.col("sample").is_in(samples)) + return {grouping[t] for t in rows["tag"].unique().to_list() if t in grouping} From 2e28a8d8f1fa190a7a73edc6020e7772199de9ae Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 13:41:59 +0200 Subject: [PATCH 023/282] MILAB-6496: select the comparator explicitly, and refuse a thin one --- software/per-cell-metrics/src/verdict.py | 94 ++++++++++++++++ .../per-cell-metrics/test/test_verdict.py | 105 +++++++++++++++++- 2 files changed, 198 insertions(+), 1 deletion(-) diff --git a/software/per-cell-metrics/src/verdict.py b/software/per-cell-metrics/src/verdict.py index e30a0a5..01a547a 100644 --- a/software/per-cell-metrics/src/verdict.py +++ b/software/per-cell-metrics/src/verdict.py @@ -30,6 +30,7 @@ from __future__ import annotations +from enum import Enum from typing import NamedTuple import polars as pl @@ -101,3 +102,96 @@ def apply_floor(counts: pl.DataFrame, floor: int, reference_tags: set[str]) -> F cells_emptied = had_evidence.join(kept_evidence, on=CELL_KEY, how="anti").height return Floored(out, {"readingsFloored": readings_floored, "cellsEmptied": cells_emptied}) + + +# Shipped defaults. Every one is a visible parameter rather than a constant, +# because nothing published sets any of them and a hard-coded line would pretend +# to a basis nobody has. +DEFAULT_PANEL_MIN_MEMBERS = 8 +DEFAULT_REFERENCE_THIN_LINE = 2 +DEFAULT_HIGH_REFERENCE_OBSERVATION_LINE = 100 + + +class ReferenceChoice(str, Enum): + """Which comparator served. Two runs served differently do not compare. + + EMPTY_DROPLETS is deliberately absent: it needs gene expression and an + empty-droplet population this block does not receive. Declaring a value the + software cannot serve would put a crashing option in the dropdown. + """ + + DECLARED = "declared reference tag" + PANEL = "the panel's own readings" + NONE = "no comparator available" + + +def resolve_default_source(reference_tags: set[str]) -> ReferenceChoice: + """The *default* source only. The scientist overrides it; this never does.""" + return ReferenceChoice.DECLARED if reference_tags else ReferenceChoice.NONE + + +def reference_by_cell( + counts: pl.DataFrame, + reference_tags: set[str], + source: ReferenceChoice, + cells: list[tuple[str, str]] | None = None, + panel_size: int = 0, + min_members: int = DEFAULT_PANEL_MIN_MEMBERS, +) -> tuple[dict[tuple[str, str], int], ReferenceChoice]: + """The reference reading per cell, and which source actually served. + + `source` is supplied, never inferred. The returned choice differs from the + requested one in exactly one direction — down to NONE where the requested + source cannot be served — because a comparison that cannot be made is + reported as absent rather than approximated. + """ + all_cells = cells or list(zip(counts["sampleId"].to_list(), counts["cellId"].to_list(), strict=True)) + + if source is ReferenceChoice.NONE: + return {}, ReferenceChoice.NONE + + if source is ReferenceChoice.DECLARED: + if not reference_tags: + return {}, ReferenceChoice.NONE + # Several reference tags combine as any identity's tags do: by the + # highest. Taking an arbitrary one would make the comparator depend on + # row order. + rows = ( + counts.filter(pl.col("tag").is_in(list(reference_tags))) + .group_by(CELL_KEY) + .agg(pl.col("umiCount").max().alias("ref")) + ) + elif source is ReferenceChoice.PANEL: + if panel_size < min_members: + # Comparing against two other antigens is not a background estimate. + # A scientist told plainly that a verdict could not be produced is + # better served than one handed a number resting on nothing. + return {}, ReferenceChoice.NONE + rows = counts.group_by(CELL_KEY).agg(pl.col("umiCount").median().alias("ref")) + else: + raise SystemExit(f"reference source {source.value!r} is not available in this run") + + ref = {(s, c): int(v) for s, c, v in zip(rows["sampleId"], rows["cellId"], rows["ref"], strict=True)} + # The tag was offered; a cell showing none of it read zero, not nothing. + for key in all_cells: + ref.setdefault(key, 0) + return ref, source + + +def gate_cells( + reference: dict[tuple[str, str], int], + threshold: int | None, + observation_line: int = DEFAULT_HIGH_REFERENCE_OBSERVATION_LINE, +) -> tuple[set[tuple[str, str]], int]: + """Which cells a declared gate sets aside, and how many read high regardless. + + The gate defaults off. The exposure count is returned either way, so a run's + exposure is visible to a scientist who has left it off — a sticky cell left + in returns as a confident *not bound*, which is the collapse the four-state + model exists to prevent. + """ + line = threshold if threshold is not None else observation_line + high = sum(1 for v in reference.values() if v >= line) + if threshold is None: + return set(), high + return {k for k, v in reference.items() if v >= threshold}, high diff --git a/software/per-cell-metrics/test/test_verdict.py b/software/per-cell-metrics/test/test_verdict.py index 6356090..f604fa1 100644 --- a/software/per-cell-metrics/test/test_verdict.py +++ b/software/per-cell-metrics/test/test_verdict.py @@ -1,5 +1,14 @@ import polars as pl -from verdict import DEFAULT_FLOOR, apply_floor +from verdict import ( + DEFAULT_FLOOR, + DEFAULT_PANEL_MIN_MEMBERS, + DEFAULT_REFERENCE_THIN_LINE, + ReferenceChoice, + apply_floor, + gate_cells, + reference_by_cell, + resolve_default_source, +) def _counts(rows): @@ -109,3 +118,97 @@ def test_a_reading_that_was_already_zero_still_counts_as_evidence_lost(): df = _counts([("S1", "c1", "AAAA", 0)]) _, stats = apply_floor(df, floor=4, reference_tags=set()) assert stats["cellsEmptied"] == 1 + + +def test_default_source_is_declared_where_a_reference_tag_exists(): + assert resolve_default_source({"CTRL"}) is ReferenceChoice.DECLARED + + +def test_default_source_never_upgrades_itself(): + assert resolve_default_source(set()) is ReferenceChoice.NONE + + +def test_empty_droplets_is_not_offered(): + assert not hasattr(ReferenceChoice, "EMPTY_DROPLETS") + + +def test_several_reference_tags_combine_by_the_highest(): + counts = _counts([("S1", "c1", "CTRL1", 3), ("S1", "c1", "CTRL2", 11)]) + ref, _ = reference_by_cell(counts, {"CTRL1", "CTRL2"}, ReferenceChoice.DECLARED) + assert ref[("S1", "c1")] == 11 # not 3, not arbitrary + + +def test_cell_missing_the_reference_tag_reads_zero(): + counts = _counts([("S1", "c1", "CTRL", 5), ("S1", "c2", "AAAA", 9)]) + ref, _ = reference_by_cell(counts, {"CTRL"}, ReferenceChoice.DECLARED) + assert ref[("S1", "c2")] == 0 + + +def test_panel_source_refuses_below_the_minimum(): + counts = _counts([("S1", "c1", "AAAA", 9)]) + ref, choice = reference_by_cell(counts, set(), ReferenceChoice.PANEL, panel_size=2, min_members=5) + assert choice is ReferenceChoice.NONE and ref == {} + + +def test_panel_source_serves_when_big_enough(): + counts = _counts([("S1", "c1", "AAAA", 9), ("S1", "c1", "CCCC", 1)]) + _, choice = reference_by_cell(counts, set(), ReferenceChoice.PANEL, panel_size=8, min_members=5) + assert choice is ReferenceChoice.PANEL + + +def test_source_none_yields_no_comparator(): + counts = _counts([("S1", "c1", "AAAA", 9)]) + ref, choice = reference_by_cell(counts, {"CTRL"}, ReferenceChoice.NONE) + assert choice is ReferenceChoice.NONE and ref == {} + + +def test_defaults_are_named_not_magic(): + assert DEFAULT_PANEL_MIN_MEMBERS > 0 + assert DEFAULT_REFERENCE_THIN_LINE >= 0 + + +def test_gate_defaults_off_but_still_measures_exposure(): + ref = {("S1", "c1"): 5000, ("S1", "c2"): 1} + aside, high = gate_cells(ref, threshold=None, observation_line=100) + assert aside == set() and high == 1 + + +def test_declared_gate_sets_aside_and_counts(): + ref = {("S1", "c1"): 900, ("S1", "c2"): 2} + aside, high = gate_cells(ref, threshold=100, observation_line=100) + assert aside == {("S1", "c1")} and high == 1 + + +def test_panel_source_serves_exactly_at_the_minimum(): + # The minimum is a floor, not a gap: a panel of exactly min_members is + # large enough. Nothing else in the suite distinguishes < from <=. + counts = _counts([("S1", "c1", "AAAA", 9), ("S1", "c1", "CCCC", 1)]) + _, choice = reference_by_cell(counts, set(), ReferenceChoice.PANEL, panel_size=5, min_members=5) + assert choice is ReferenceChoice.PANEL + + +def test_panel_source_refuses_one_below_the_minimum(): + counts = _counts([("S1", "c1", "AAAA", 9)]) + ref, choice = reference_by_cell(counts, set(), ReferenceChoice.PANEL, panel_size=4, min_members=5) + assert choice is ReferenceChoice.NONE + + +def test_the_gate_boundary_includes_the_line_itself(): + # A reading exactly at the threshold is high: the named value satisfies the + # condition it names, matching the floor (a count of exactly `floor` is + # evidence) and the panel minimum (exactly `min_members` is large enough). + # Both sides are pinned so that changing the comparison is a deliberate act. + at_line = {("S1", "c1"): 100} + just_below = {("S1", "c2"): 99} + aside_at, high_at = gate_cells(at_line, threshold=100, observation_line=100) + aside_below, high_below = gate_cells(just_below, threshold=100, observation_line=100) + assert aside_at == {("S1", "c1")} and high_at == 1 + assert aside_below == set() and high_below == 0 + + +def test_high_reference_counting_is_independent_of_the_gate_acting(): + # The exposure count must not quietly become "cells the gate removed". + ref = {("S1", "c1"): 500, ("S1", "c2"): 1} + _, high_off = gate_cells(ref, threshold=None, observation_line=100) + _, high_on = gate_cells(ref, threshold=100, observation_line=100) + assert high_off == high_on == 1 From 7248b9a8bc83ba24fc6601aba0fc64d225fb7a20 Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 13:46:22 +0200 Subject: [PATCH 024/282] MILAB-6496: pin the comparator's fallback rule, statistic, and defaults --- .../per-cell-metrics/test/test_verdict.py | 41 +++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/software/per-cell-metrics/test/test_verdict.py b/software/per-cell-metrics/test/test_verdict.py index f604fa1..ba65b91 100644 --- a/software/per-cell-metrics/test/test_verdict.py +++ b/software/per-cell-metrics/test/test_verdict.py @@ -1,6 +1,7 @@ import polars as pl from verdict import ( DEFAULT_FLOOR, + DEFAULT_HIGH_REFERENCE_OBSERVATION_LINE, DEFAULT_PANEL_MIN_MEMBERS, DEFAULT_REFERENCE_THIN_LINE, ReferenceChoice, @@ -163,8 +164,16 @@ def test_source_none_yields_no_comparator(): def test_defaults_are_named_not_magic(): + # Pins the actual shipped values, not just their sign. These are + # user-facing numbers that appear in a dropdown and change what the block + # produces, so an edit to any of them must be a deliberate, visible act — + # not a silent one that only this test would otherwise catch. The values + # themselves are not calibrated against real data. assert DEFAULT_PANEL_MIN_MEMBERS > 0 assert DEFAULT_REFERENCE_THIN_LINE >= 0 + assert DEFAULT_PANEL_MIN_MEMBERS == 8 + assert DEFAULT_REFERENCE_THIN_LINE == 2 + assert DEFAULT_HIGH_REFERENCE_OBSERVATION_LINE == 100 def test_gate_defaults_off_but_still_measures_exposure(): @@ -212,3 +221,35 @@ def test_high_reference_counting_is_independent_of_the_gate_acting(): _, high_off = gate_cells(ref, threshold=None, observation_line=100) _, high_on = gate_cells(ref, threshold=100, observation_line=100) assert high_off == high_on == 1 + + +def test_a_source_that_cannot_be_served_falls_to_none_and_never_sideways(): + # The served choice may only move down to NONE. Substituting a different + # comparator would silently answer a question the scientist did not ask, + # and two runs served by different comparators are not comparable. + counts = _counts([("S1", "c1", "AAAA", 9), ("S1", "c1", "CCCC", 3)]) + # DECLARED with nothing declared: not PANEL, even though a panel exists. + ref, choice = reference_by_cell(counts, set(), ReferenceChoice.DECLARED, panel_size=100, min_members=5) + assert choice is ReferenceChoice.NONE + assert ref == {} + # PANEL below the minimum: not DECLARED, even though a reference tag exists. + ref2, choice2 = reference_by_cell(counts, {"CTRL"}, ReferenceChoice.PANEL, panel_size=1, min_members=5) + assert choice2 is ReferenceChoice.NONE + assert ref2 == {} + + +def test_the_panel_comparator_is_the_median_not_the_mean(): + # A cell with one strong binder: the mean is dragged up by it, the median + # is not. The comparator is meant to stand for the cell's background, so a + # single high reading must not raise the bar it is measured against. + counts = _counts( + [ + ("S1", "c1", "AAAA", 1), + ("S1", "c1", "CCCC", 2), + ("S1", "c1", "GGGG", 3), + ("S1", "c1", "TTTT", 200), + ] + ) + ref, choice = reference_by_cell(counts, set(), ReferenceChoice.PANEL, panel_size=8, min_members=5) + assert choice is ReferenceChoice.PANEL + assert ref[("S1", "c1")] == 2 # median of 1,2,3,200 -> 2.5 -> int 2; mean would be 51 From f8abf8228b725588972f6d79f8e8e7e2b6b741bb Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 14:01:16 +0200 Subject: [PATCH 025/282] MILAB-6496: the cell list is authoritative, and exposure follows the observation line --- software/per-cell-metrics/src/verdict.py | 34 +++++++++-- .../per-cell-metrics/test/test_verdict.py | 58 +++++++++++++++++-- 2 files changed, 81 insertions(+), 11 deletions(-) diff --git a/software/per-cell-metrics/src/verdict.py b/software/per-cell-metrics/src/verdict.py index 01a547a..1997284 100644 --- a/software/per-cell-metrics/src/verdict.py +++ b/software/per-cell-metrics/src/verdict.py @@ -144,8 +144,29 @@ def reference_by_cell( requested one in exactly one direction — down to NONE where the requested source cannot be served — because a comparison that cannot be made is reported as absent rather than approximated. + + `cells`, when given, is authoritative in both directions: the result holds + exactly those cells, zero-filled where the comparator read nothing. Omit it + and the cell universe is taken from the counts frame instead, which covers + only cells with an observed reading — pass it explicitly wherever the + analysis has a cell list, or a cell that was asked and read nothing will be + missing rather than zero. """ - all_cells = cells or list(zip(counts["sampleId"].to_list(), counts["cellId"].to_list(), strict=True)) + all_cells = ( + list(zip(counts["sampleId"].to_list(), counts["cellId"].to_list(), strict=True)) if cells is None else cells + ) + # A semi join on the cell list, applied before either branch aggregates, so + # a cell outside the analysis is dropped before its rows are ever combined + # rather than combined and then discarded. + scoped = ( + counts + if cells is None + else counts.join( + pl.DataFrame(cells, orient="row", schema={"sampleId": pl.String, "cellId": pl.String}), + on=CELL_KEY, + how="semi", + ) + ) if source is ReferenceChoice.NONE: return {}, ReferenceChoice.NONE @@ -157,7 +178,7 @@ def reference_by_cell( # highest. Taking an arbitrary one would make the comparator depend on # row order. rows = ( - counts.filter(pl.col("tag").is_in(list(reference_tags))) + scoped.filter(pl.col("tag").is_in(list(reference_tags))) .group_by(CELL_KEY) .agg(pl.col("umiCount").max().alias("ref")) ) @@ -167,7 +188,7 @@ def reference_by_cell( # A scientist told plainly that a verdict could not be produced is # better served than one handed a number resting on nothing. return {}, ReferenceChoice.NONE - rows = counts.group_by(CELL_KEY).agg(pl.col("umiCount").median().alias("ref")) + rows = scoped.group_by(CELL_KEY).agg(pl.col("umiCount").median().alias("ref")) else: raise SystemExit(f"reference source {source.value!r} is not available in this run") @@ -190,8 +211,11 @@ def gate_cells( in returns as a confident *not bound*, which is the collapse the four-state model exists to prevent. """ - line = threshold if threshold is not None else observation_line - high = sum(1 for v in reference.values() if v >= line) + # The observation line is independent of the gate: it measures how many + # cells sat in high background, which is a fact about the run whether or + # not anything was set aside. Folding it into the threshold would make the + # count mean "cells the gate removed" and lose the measurement entirely. + high = sum(1 for v in reference.values() if v >= observation_line) if threshold is None: return set(), high return {k for k, v in reference.items() if v >= threshold}, high diff --git a/software/per-cell-metrics/test/test_verdict.py b/software/per-cell-metrics/test/test_verdict.py index ba65b91..ba93941 100644 --- a/software/per-cell-metrics/test/test_verdict.py +++ b/software/per-cell-metrics/test/test_verdict.py @@ -215,12 +215,24 @@ def test_the_gate_boundary_includes_the_line_itself(): assert aside_below == set() and high_below == 0 -def test_high_reference_counting_is_independent_of_the_gate_acting(): - # The exposure count must not quietly become "cells the gate removed". - ref = {("S1", "c1"): 500, ("S1", "c2"): 1} - _, high_off = gate_cells(ref, threshold=None, observation_line=100) - _, high_on = gate_cells(ref, threshold=100, observation_line=100) - assert high_off == high_on == 1 +def test_the_observation_line_is_independent_of_the_gate_threshold(): + # The two lines are separate parameters and must be given separate values + # here: with them equal, no assertion can tell whether the exposure count + # follows the observation line or the gate. It must follow the observation + # line, because it measures how many cells sat in high background — true + # whether or not the gate removed any of them. + ref = {("S1", "a"): 500, ("S1", "b"): 50, ("S1", "c"): 2000} + + aside_off, high_off = gate_cells(ref, threshold=None, observation_line=100) + assert aside_off == set() and high_off == 2 + + # Gate stricter than the observation line: fewer set aside, same exposure. + aside_hi, high_hi = gate_cells(ref, threshold=1000, observation_line=100) + assert aside_hi == {("S1", "c")} and high_hi == 2 + + # Gate looser than the observation line: more set aside, same exposure. + aside_lo, high_lo = gate_cells(ref, threshold=10, observation_line=100) + assert aside_lo == {("S1", "a"), ("S1", "b"), ("S1", "c")} and high_lo == 2 def test_a_source_that_cannot_be_served_falls_to_none_and_never_sideways(): @@ -253,3 +265,37 @@ def test_the_panel_comparator_is_the_median_not_the_mean(): ref, choice = reference_by_cell(counts, set(), ReferenceChoice.PANEL, panel_size=8, min_members=5) assert choice is ReferenceChoice.PANEL assert ref[("S1", "c1")] == 2 # median of 1,2,3,200 -> 2.5 -> int 2; mean would be 51 + + +def test_an_explicit_empty_cell_list_means_no_cells(): + # Not "derive them from the counts frame". An empty list is a statement. + counts = _counts([("S1", "c1", "CTRL", 7)]) + ref, choice = reference_by_cell(counts, {"CTRL"}, ReferenceChoice.DECLARED, cells=[]) + assert choice is ReferenceChoice.DECLARED + assert ref == {} + + +def test_cells_outside_the_given_list_are_excluded(): + # The cell list is the analysis. A cell with a real reference reading that + # is not in it has a comparator nobody will consult, and returning it would + # invite a reader to treat the result as the cell universe. + counts = _counts([("S1", "c1", "CTRL", 7), ("S1", "c2", "CTRL", 9)]) + ref, _ = reference_by_cell(counts, {"CTRL"}, ReferenceChoice.DECLARED, cells=[("S1", "c1")]) + assert ref == {("S1", "c1"): 7} + + +def test_a_named_cell_with_no_reference_reading_is_zero_not_missing(): + # Both directions in one assertion: c2 is added at 0, c3 is excluded. + counts = _counts([("S1", "c1", "CTRL", 7), ("S1", "c3", "CTRL", 4)]) + ref, _ = reference_by_cell(counts, {"CTRL"}, ReferenceChoice.DECLARED, cells=[("S1", "c1"), ("S1", "c2")]) + assert ref == {("S1", "c1"): 7, ("S1", "c2"): 0} + + +def test_the_panel_source_also_respects_the_given_cell_list(): + # Two branches now share the cell-list rule; only one is covered above. + counts = _counts([("S1", "c1", "AAAA", 9), ("S1", "c2", "AAAA", 3)]) + ref, choice = reference_by_cell( + counts, set(), ReferenceChoice.PANEL, cells=[("S1", "c1")], panel_size=8, min_members=5 + ) + assert choice is ReferenceChoice.PANEL + assert ref == {("S1", "c1"): 9} From b3f5842e065126a93286ee2d6e17ba3be6953f5b Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 14:07:10 +0200 Subject: [PATCH 026/282] MILAB-6496: check panel against reads in both directions, per sample --- software/per-cell-metrics/src/panel.py | 36 ++++++++ software/per-cell-metrics/test/test_panel.py | 94 ++++++++++++++++++++ 2 files changed, 130 insertions(+) diff --git a/software/per-cell-metrics/src/panel.py b/software/per-cell-metrics/src/panel.py index f786b0d..2cc46a3 100644 --- a/software/per-cell-metrics/src/panel.py +++ b/software/per-cell-metrics/src/panel.py @@ -230,3 +230,39 @@ def offered_identities(panel: pl.DataFrame, grouping: Grouping, samples: list[st """ rows = panel.filter((pl.col("sample") == ANY_SAMPLE) | pl.col("sample").is_in(samples)) return {grouping[t] for t in rows["tag"].unique().to_list() if t in grouping} + + +def panel_read_mismatch(panel: pl.DataFrame, seen: pl.DataFrame) -> pl.DataFrame: + """Both directions of the panel-versus-reads check, per sample. + + Neither direction can be known before the reads are processed, so by the + time either is known the reading exists; withholding it then would turn a + partial answer into none. This reports and never raises. + + Per sample, because the same barcode can carry a different antigen in a + different sample's panel: a global check lets a barcode undeclared in one + sample pass on another sample's declaration. + """ + rows = [] + global_panel = panel.filter(pl.col("sample") == ANY_SAMPLE) + + if global_panel.height: + declared = set(global_panel["tag"].to_list()) + observed = set(seen["tag"].to_list()) + for tag in sorted(declared - observed): + rows.append({"sample": ANY_SAMPLE, "tag": tag, "direction": "declared-never-seen"}) + for tag in sorted(observed - declared): + rows.append({"sample": ANY_SAMPLE, "tag": tag, "direction": "undeclared-in-panel"}) + else: + samples = sorted(set(panel["sample"].to_list()) | set(seen["sampleId"].to_list())) + for sample in samples: + declared = set(panel.filter(pl.col("sample") == sample)["tag"].to_list()) + observed = set(seen.filter(pl.col("sampleId") == sample)["tag"].to_list()) + for tag in sorted(declared - observed): + rows.append({"sample": sample, "tag": tag, "direction": "declared-never-seen"}) + for tag in sorted(observed - declared): + rows.append({"sample": sample, "tag": tag, "direction": "undeclared-in-panel"}) + + return pl.DataFrame(rows, schema={"sample": pl.String, "tag": pl.String, "direction": pl.String}).sort( + ["sample", "direction", "tag"] + ) diff --git a/software/per-cell-metrics/test/test_panel.py b/software/per-cell-metrics/test/test_panel.py index a318c6f..d8e7cb4 100644 --- a/software/per-cell-metrics/test/test_panel.py +++ b/software/per-cell-metrics/test/test_panel.py @@ -5,6 +5,7 @@ default_grouping, identity_universe, offered_identities, + panel_read_mismatch, property_columns, read_panel, ) @@ -298,3 +299,96 @@ def test_a_tag_outside_the_grouping_is_skipped_not_an_error(): g = {"AAAA": "A"} assert identity_universe(panel, g) == {"A"} assert offered_identities(panel, g, ["S1"]) == {"A"} + + +def _counts(rows): + # House shape for a reads/counts table, matching test_verdict.py: one row + # per (sample, cell, tag) with the UMI count. panel_read_mismatch only + # needs sampleId and tag, but the table it is handed always carries all + # four columns, so the fixture does too. + return pl.DataFrame( + rows, orient="row", schema={"sampleId": pl.String, "cellId": pl.String, "tag": pl.String, "umiCount": pl.Int64} + ) + + +def test_declared_tag_never_seen_is_reported_per_sample(): + panel = pl.DataFrame({"tag": ["AAAA", "GGGG"], "sample": ["S1", "S1"], "Name": ["a", "g"]}) + seen = _counts([("S1", "c1", "AAAA", 1)]) + out = panel_read_mismatch(panel, seen) + row = out.filter(pl.col("direction") == "declared-never-seen").row(0, named=True) + assert row["tag"] == "GGGG" and row["sample"] == "S1" + + +def test_undeclared_barcode_is_reported_per_sample(): + panel = pl.DataFrame({"tag": ["AAAA"], "sample": ["S1"], "Name": ["a"]}) + seen = _counts([("S1", "c1", "AAAA", 1), ("S1", "c1", "TTTT", 1)]) + out = panel_read_mismatch(panel, seen) + row = out.filter(pl.col("direction") == "undeclared-in-panel").row(0, named=True) + assert row["tag"] == "TTTT" and row["sample"] == "S1" + + +def test_a_barcode_declared_in_another_sample_does_not_pass_silently(): + # The failure this whole check exists to prevent: AAAA is declared for S3 + # only; it is read in S1, where nothing declares it. A global check would + # let S3's declaration excuse it there too. + panel = pl.DataFrame({"tag": ["CCCC", "AAAA"], "sample": ["S1", "S3"], "Name": ["c", "a"]}) + seen = _counts([("S1", "c1", "CCCC", 1), ("S1", "c1", "AAAA", 1)]) + out = panel_read_mismatch(panel, seen) + undeclared = out.filter(pl.col("direction") == "undeclared-in-panel") + assert ("S1", "AAAA") in list(zip(undeclared["sample"], undeclared["tag"], strict=True)) + + +def test_star_panel_checks_globally(): + panel = pl.DataFrame({"tag": ["AAAA"], "sample": ["*"], "Name": ["a"]}) + seen = _counts([("S1", "c1", "AAAA", 1), ("S2", "c1", "AAAA", 1)]) + assert panel_read_mismatch(panel, seen).height == 0 + + +def test_mismatch_never_raises(): + panel = pl.DataFrame({"tag": ["AAAA"], "sample": ["S1"], "Name": ["a"]}) + seen = _counts([("S9", "c1", "ZZZZ", 1)]) + out = panel_read_mismatch(panel, seen) # must not raise + assert out.height >= 1 + + +def test_a_sample_with_reads_but_no_panel_rows_reports_every_barcode(): + # The panel does not cover this sample at all. Every barcode it read is + # undeclared — a large claim, so it must be stated rather than inferred + # from an empty result. + panel = pl.DataFrame({"tag": ["AAAA"], "sample": ["S1"], "Name": ["a"]}) + seen = _counts([("S9", "c1", "CCCC", 5)]) + out = panel_read_mismatch(panel, seen) + rows = {(r["sample"], r["tag"], r["direction"]) for r in out.to_dicts()} + assert ("S9", "CCCC", "undeclared-in-panel") in rows + + +def test_a_sample_in_the_panel_with_no_reads_reports_every_tag(): + panel = pl.DataFrame({"tag": ["AAAA"], "sample": ["S1"], "Name": ["a"]}) + seen = _counts([("S2", "c1", "AAAA", 5)]) + out = panel_read_mismatch(panel, seen) + rows = {(r["sample"], r["tag"], r["direction"]) for r in out.to_dicts()} + assert ("S1", "AAAA", "declared-never-seen") in rows + + +def test_full_agreement_reports_nothing(): + # The empty result must mean agreement, not a check that failed to run. + panel = pl.DataFrame({"tag": ["AAAA", "CCCC"], "sample": ["S1", "S1"], "Name": ["a", "c"]}) + seen = _counts([("S1", "c1", "AAAA", 5), ("S1", "c1", "CCCC", 3)]) + assert panel_read_mismatch(panel, seen).height == 0 + + +def test_empty_inputs_do_not_raise(): + panel = pl.DataFrame( + {"tag": [], "sample": [], "Name": []}, schema={"tag": pl.String, "sample": pl.String, "Name": pl.String} + ) + assert panel_read_mismatch(panel, _counts([])).height == 0 + + +def test_both_directions_can_fire_for_one_sample_at_once(): + # A sample can simultaneously declare a tag it never read and read a + # barcode it never declared. Neither direction may mask the other. + panel = pl.DataFrame({"tag": ["AAAA"], "sample": ["S1"], "Name": ["a"]}) + seen = _counts([("S1", "c1", "CCCC", 5)]) + rows = {(r["sample"], r["tag"], r["direction"]) for r in panel_read_mismatch(panel, seen).to_dicts()} + assert ("S1", "AAAA", "declared-never-seen") in rows + assert ("S1", "CCCC", "undeclared-in-panel") in rows From 50b3600d0482cb6e6c6f76a2461231eaf054c4ee Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 14:14:02 +0200 Subject: [PATCH 027/282] MILAB-6496: state the comparator's contract and make its fallback rule one thing --- software/per-cell-metrics/src/verdict.py | 94 ++++++++++++++----- .../per-cell-metrics/test/test_verdict.py | 30 ++---- 2 files changed, 81 insertions(+), 43 deletions(-) diff --git a/software/per-cell-metrics/src/verdict.py b/software/per-cell-metrics/src/verdict.py index 1997284..38a9faa 100644 --- a/software/per-cell-metrics/src/verdict.py +++ b/software/per-cell-metrics/src/verdict.py @@ -108,6 +108,9 @@ def apply_floor(counts: pl.DataFrame, floor: int, reference_tags: set[str]) -> F # because nothing published sets any of them and a hard-coded line would pretend # to a basis nobody has. DEFAULT_PANEL_MIN_MEMBERS = 8 +# A reference reading below this is too thin to compare against: the position +# reads *unreliable*, not *not bound* — the comparison could not be made, not +# that it was made and failed. DEFAULT_REFERENCE_THIN_LINE = 2 DEFAULT_HIGH_REFERENCE_OBSERVATION_LINE = 100 @@ -130,6 +133,30 @@ def resolve_default_source(reference_tags: set[str]) -> ReferenceChoice: return ReferenceChoice.DECLARED if reference_tags else ReferenceChoice.NONE +def served_source( + source: ReferenceChoice, + reference_tags: set[str], + panel_size: int, + min_members: int, +) -> ReferenceChoice: + """The source that can actually be served. Only ever the one asked for, or NONE. + + A comparison that cannot be made is reported as absent rather than + approximated: a caller who asked for a comparator this run cannot produce + gets told plainly, never handed a different one it did not ask for. + """ + if source is ReferenceChoice.DECLARED and not reference_tags: + return ReferenceChoice.NONE + if source is ReferenceChoice.PANEL and panel_size < min_members: + return ReferenceChoice.NONE + return source + + +class Reference(NamedTuple): + by_cell: dict[tuple[str, str], int] + served: ReferenceChoice + + def reference_by_cell( counts: pl.DataFrame, reference_tags: set[str], @@ -137,13 +164,19 @@ def reference_by_cell( cells: list[tuple[str, str]] | None = None, panel_size: int = 0, min_members: int = DEFAULT_PANEL_MIN_MEMBERS, -) -> tuple[dict[tuple[str, str], int], ReferenceChoice]: +) -> Reference: """The reference reading per cell, and which source actually served. - `source` is supplied, never inferred. The returned choice differs from the - requested one in exactly one direction — down to NONE where the requested - source cannot be served — because a comparison that cannot be made is - reported as absent rather than approximated. + `source` is supplied, never inferred; `served_source` decides whether it + can actually be served, and the result differs from the request in exactly + one direction — down to NONE. + + `by_cell` is EMPTY when `served` is NONE — not a mapping of zeros. When a + comparator did serve, it holds a key for every analysed cell, zero where + that cell showed none of it. So a reader switches on `served`, never on key + presence: `by_cell.get(key, 0)` would read "no comparator was available" as + "the comparator read zero", which is the difference between a position that + could not be settled and one that was settled as not bound. `cells`, when given, is authoritative in both directions: the result holds exactly those cells, zero-filled where the comparator read nothing. Omit it @@ -151,9 +184,28 @@ def reference_by_cell( only cells with an observed reading — pass it explicitly wherever the analysis has a cell list, or a cell that was asked and read nothing will be missing rather than zero. + + Receives the floored, sparse per-tag frame — before densification. The + PANEL median is a median of observed readings; on a densified frame every + manufactured zero would drag it toward zero and change the comparator for + every cell, not just the ones that gained one. + + `reference_tags` is not excluded from the PANEL median: the panel + comparator is the cell's own readings, and a declared comparator, where + also present, is one of those readings rather than something held out of + them. """ + served = served_source(source, reference_tags, panel_size, min_members) + if served is ReferenceChoice.NONE: + return Reference({}, ReferenceChoice.NONE) + all_cells = ( - list(zip(counts["sampleId"].to_list(), counts["cellId"].to_list(), strict=True)) if cells is None else cells + # Deduplicated: a cell with several tag readings otherwise appears once + # per reading, and the zero-fill loop below would revisit it that many + # times for no effect but the extra work. + {(s, c) for s, c in zip(counts["sampleId"].to_list(), counts["cellId"].to_list(), strict=True)} + if cells is None + else cells ) # A semi join on the cell list, applied before either branch aggregates, so # a cell outside the analysis is dropped before its rows are ever combined @@ -168,12 +220,7 @@ def reference_by_cell( ) ) - if source is ReferenceChoice.NONE: - return {}, ReferenceChoice.NONE - - if source is ReferenceChoice.DECLARED: - if not reference_tags: - return {}, ReferenceChoice.NONE + if served is ReferenceChoice.DECLARED: # Several reference tags combine as any identity's tags do: by the # highest. Taking an arbitrary one would make the comparator depend on # row order. @@ -182,21 +229,24 @@ def reference_by_cell( .group_by(CELL_KEY) .agg(pl.col("umiCount").max().alias("ref")) ) - elif source is ReferenceChoice.PANEL: - if panel_size < min_members: - # Comparing against two other antigens is not a background estimate. - # A scientist told plainly that a verdict could not be produced is - # better served than one handed a number resting on nothing. - return {}, ReferenceChoice.NONE - rows = scoped.group_by(CELL_KEY).agg(pl.col("umiCount").median().alias("ref")) + elif served is ReferenceChoice.PANEL: + # cast(Int64) truncates rather than rounds: a panel split between 1 and + # 2 medians to 1.5, cast to 1 — one below the default thin line of 2 — + # so that cell reads unreliable rather than being compared against a + # number resting on half a UMI. + rows = scoped.group_by(CELL_KEY).agg(pl.col("umiCount").median().cast(pl.Int64).alias("ref")) else: - raise SystemExit(f"reference source {source.value!r} is not available in this run") + # Reachable only if ReferenceChoice gains a member with no aggregation + # branch here — most plausibly EMPTY_DROPLETS. That is a missing + # implementation, not a fact about this run, so it must not be + # reported as "unavailable this time". + raise SystemExit(f"no comparator implementation for reference source {served.value!r}") - ref = {(s, c): int(v) for s, c, v in zip(rows["sampleId"], rows["cellId"], rows["ref"], strict=True)} + ref = {(s, c): v for s, c, v in zip(rows["sampleId"], rows["cellId"], rows["ref"], strict=True)} # The tag was offered; a cell showing none of it read zero, not nothing. for key in all_cells: ref.setdefault(key, 0) - return ref, source + return Reference(ref, served) def gate_cells( diff --git a/software/per-cell-metrics/test/test_verdict.py b/software/per-cell-metrics/test/test_verdict.py index ba93941..52163eb 100644 --- a/software/per-cell-metrics/test/test_verdict.py +++ b/software/per-cell-metrics/test/test_verdict.py @@ -130,6 +130,9 @@ def test_default_source_never_upgrades_itself(): def test_empty_droplets_is_not_offered(): + # A tripwire, not a permanent ban: the day this block genuinely receives + # gene expression and an empty-droplet population, EMPTY_DROPLETS gets + # implemented and this test is deleted, not fixed. assert not hasattr(ReferenceChoice, "EMPTY_DROPLETS") @@ -145,32 +148,17 @@ def test_cell_missing_the_reference_tag_reads_zero(): assert ref[("S1", "c2")] == 0 -def test_panel_source_refuses_below_the_minimum(): - counts = _counts([("S1", "c1", "AAAA", 9)]) - ref, choice = reference_by_cell(counts, set(), ReferenceChoice.PANEL, panel_size=2, min_members=5) - assert choice is ReferenceChoice.NONE and ref == {} - - -def test_panel_source_serves_when_big_enough(): - counts = _counts([("S1", "c1", "AAAA", 9), ("S1", "c1", "CCCC", 1)]) - _, choice = reference_by_cell(counts, set(), ReferenceChoice.PANEL, panel_size=8, min_members=5) - assert choice is ReferenceChoice.PANEL - - def test_source_none_yields_no_comparator(): counts = _counts([("S1", "c1", "AAAA", 9)]) ref, choice = reference_by_cell(counts, {"CTRL"}, ReferenceChoice.NONE) assert choice is ReferenceChoice.NONE and ref == {} -def test_defaults_are_named_not_magic(): - # Pins the actual shipped values, not just their sign. These are - # user-facing numbers that appear in a dropdown and change what the block - # produces, so an edit to any of them must be a deliberate, visible act — - # not a silent one that only this test would otherwise catch. The values - # themselves are not calibrated against real data. - assert DEFAULT_PANEL_MIN_MEMBERS > 0 - assert DEFAULT_REFERENCE_THIN_LINE >= 0 +def test_shipped_defaults_are_pinned(): + # These are user-facing numbers that appear in a dropdown and change what + # the block produces, so an edit to any of them must be a deliberate, + # visible act — not a silent one that only this test would otherwise + # catch. The values themselves are not calibrated against real data. assert DEFAULT_PANEL_MIN_MEMBERS == 8 assert DEFAULT_REFERENCE_THIN_LINE == 2 assert DEFAULT_HIGH_REFERENCE_OBSERVATION_LINE == 100 @@ -199,7 +187,7 @@ def test_panel_source_serves_exactly_at_the_minimum(): def test_panel_source_refuses_one_below_the_minimum(): counts = _counts([("S1", "c1", "AAAA", 9)]) ref, choice = reference_by_cell(counts, set(), ReferenceChoice.PANEL, panel_size=4, min_members=5) - assert choice is ReferenceChoice.NONE + assert choice is ReferenceChoice.NONE and ref == {} def test_the_gate_boundary_includes_the_line_itself(): From 91f57f062ac1f9ec96b1205c225944fa686074ff Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 14:16:33 +0200 Subject: [PATCH 028/282] MILAB-6496: pin the panel median's truncation where rounding would differ --- .../per-cell-metrics/test/test_verdict.py | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/software/per-cell-metrics/test/test_verdict.py b/software/per-cell-metrics/test/test_verdict.py index 52163eb..d0c4c41 100644 --- a/software/per-cell-metrics/test/test_verdict.py +++ b/software/per-cell-metrics/test/test_verdict.py @@ -255,6 +255,25 @@ def test_the_panel_comparator_is_the_median_not_the_mean(): assert ref[("S1", "c1")] == 2 # median of 1,2,3,200 -> 2.5 -> int 2; mean would be 51 +def test_the_panel_median_truncates_rather_than_rounds(): + # A median of 1.5 is the value that separates the two: truncation gives 1, + # and polars' round-half-to-even gives 2. At a median of 2.5 both give 2, + # so a fixture there cannot tell them apart — and the difference matters, + # because 1 falls below the thin line of 2 and reads unreliable while 2 is + # compared normally. Truncation is the behaviour; this pins it. + counts = _counts( + [ + ("S1", "c1", "AAAA", 1), + ("S1", "c1", "CCCC", 1), + ("S1", "c1", "GGGG", 2), + ("S1", "c1", "TTTT", 2), + ] + ) + ref, choice = reference_by_cell(counts, set(), ReferenceChoice.PANEL, panel_size=8, min_members=5) + assert choice is ReferenceChoice.PANEL + assert ref[("S1", "c1")] == 1 + + def test_an_explicit_empty_cell_list_means_no_cells(): # Not "derive them from the counts frame". An empty list is a statement. counts = _counts([("S1", "c1", "CTRL", 7)]) From b8df8882bba53b2da205f51e2387cb67673efd2e Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 14:20:23 +0200 Subject: [PATCH 029/282] MILAB-6496: refuse a literal star sample, and never raise on null keys --- software/per-cell-metrics/src/panel.py | 26 +++++++++++++++- software/per-cell-metrics/test/test_panel.py | 32 ++++++++++++++++++++ 2 files changed, 57 insertions(+), 1 deletion(-) diff --git a/software/per-cell-metrics/src/panel.py b/software/per-cell-metrics/src/panel.py index 2cc46a3..61eff99 100644 --- a/software/per-cell-metrics/src/panel.py +++ b/software/per-cell-metrics/src/panel.py @@ -144,6 +144,21 @@ def read_panel(csv_path: str, roles: dict[str, str]) -> Panel: "entirely to declare one panel over every sample; a blank cell is ambiguous." ) + # ANY_SAMPLE is what this reader writes when there is no sample column, + # not a sample name a caller can declare. Accepting it in an explicit + # sample column would let one row claim every sample; downstream, a + # frame mixing "*" with real sample names is exactly what turns the + # panel-versus-reads check blind. + star = panel.filter(pl.col("sample") == ANY_SAMPLE) + if star.height: + rows = ", ".join(str(_csv_line(r)) for r in star["_row"]) + raise SystemExit( + f"panel file has the literal {ANY_SAMPLE!r} in column {sample_col!r} on line(s) " + f"{rows}. Leave the column out entirely to declare one panel over every sample; " + f"{ANY_SAMPLE!r} is what this reader writes when there is no sample column, not a " + "sample name you can use." + ) + panel = panel.drop("_row") dupes = panel.group_by(["tag", "sample"]).len().filter(pl.col("len") > 1).sort(["tag", "sample"]) @@ -243,10 +258,19 @@ def panel_read_mismatch(panel: pl.DataFrame, seen: pl.DataFrame) -> pl.DataFrame different sample's panel: a global check lets a barcode undeclared in one sample pass on another sample's declaration. """ + # A row with no sample or no barcode cannot be placed on either side of the + # comparison, and a null key is not a usable p-column key. Dropping them + # keeps the promise that this check never raises. + seen = seen.filter(pl.col("sampleId").is_not_null() & pl.col("tag").is_not_null()) + rows = [] global_panel = panel.filter(pl.col("sample") == ANY_SAMPLE) - if global_panel.height: + # All rows, not any: a frame mixing "*" with real sample names must not take + # the global branch, which would discard every named row and report a + # per-sample disagreement as agreement. The reader refuses such a frame, so + # this is the second line of defence for a caller that builds one directly. + if panel.height and global_panel.height == panel.height: declared = set(global_panel["tag"].to_list()) observed = set(seen["tag"].to_list()) for tag in sorted(declared - observed): diff --git a/software/per-cell-metrics/test/test_panel.py b/software/per-cell-metrics/test/test_panel.py index d8e7cb4..036aa7d 100644 --- a/software/per-cell-metrics/test/test_panel.py +++ b/software/per-cell-metrics/test/test_panel.py @@ -392,3 +392,35 @@ def test_both_directions_can_fire_for_one_sample_at_once(): rows = {(r["sample"], r["tag"], r["direction"]) for r in panel_read_mismatch(panel, seen).to_dicts()} assert ("S1", "AAAA", "declared-never-seen") in rows assert ("S1", "CCCC", "undeclared-in-panel") in rows + + +def test_a_literal_star_in_a_sample_column_is_fatal(tmp_path): + # "*" is what the reader writes when there is no sample column. Accepting it + # as a sample name lets one row claim every sample, and downstream the whole + # panel-versus-reads check goes blind. + path = _csv(tmp_path, rows=[["S1", "AgA", "AAAA", "T"], ["*", "AgB", "CCCC", "T"]]) + with pytest.raises(SystemExit) as e: + read_panel(path, ROLES) + assert "*" in str(e.value) + + +def test_a_mixed_star_and_named_panel_does_not_go_global(): + # Second line of defence: the reader refuses this frame, but a caller + # building one directly must not get an empty table for a real disagreement. + panel = pl.DataFrame({"tag": ["AAAA", "CCCC"], "sample": ["*", "S1"], "Name": ["a", "c"]}) + seen = _counts([("S1", "c1", "AAAA", 5)]) + rows = {(r["sample"], r["tag"], r["direction"]) for r in panel_read_mismatch(panel, seen).to_dicts()} + assert ("S1", "CCCC", "declared-never-seen") in rows + + +def test_null_keys_in_the_reads_do_not_raise(): + panel = pl.DataFrame({"tag": ["AAAA"], "sample": ["S1"], "Name": ["a"]}) + seen = pl.DataFrame( + [("S1", "c1", "AAAA", 5), (None, "c2", "CCCC", 3), ("S1", "c3", None, 1)], + orient="row", + schema={"sampleId": pl.String, "cellId": pl.String, "tag": pl.String, "umiCount": pl.Int64}, + ) + out = panel_read_mismatch(panel, seen) + assert out.height == 0 + assert None not in out["sample"].to_list() + assert None not in out["tag"].to_list() From 2d1224b3e85bc70ddfeae1c67b54e78daa8d044d Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 14:29:25 +0200 Subject: [PATCH 030/282] MILAB-6496: read a count against its reference and produce a state --- software/per-cell-metrics/src/verdict.py | 238 ++++++++++++++++++ .../per-cell-metrics/test/test_verdict.py | 233 +++++++++++++++++ 2 files changed, 471 insertions(+) diff --git a/software/per-cell-metrics/src/verdict.py b/software/per-cell-metrics/src/verdict.py index 38a9faa..88e439e 100644 --- a/software/per-cell-metrics/src/verdict.py +++ b/software/per-cell-metrics/src/verdict.py @@ -33,7 +33,9 @@ from enum import Enum from typing import NamedTuple +import numpy as np import polars as pl +from scipy.stats import beta CELL_KEY = ("sampleId", "cellId") @@ -269,3 +271,239 @@ def gate_cells( if threshold is None: return set(), high return {k for k, v in reference.items() if v >= threshold}, high + + +# The cutoff and the three beta constants are the dominant tool's, inherited +# rather than justified: nothing published argues any of the four over other +# values. They ship as the default so a run's numbers reconcile with what a +# scientist already has. +BETA_X, BETA_A_OFFSET, BETA_B_OFFSET = 0.925, 1, 3 +BOUND_CUTOFF = 75.0 + + +class State(str, Enum): + """The four states a verdict takes. There is no fifth. + + NEVER_ASKED means the experiment did not put the identity to those cells. + UNRELIABLE means it did and the data cannot settle it. Neither is a kind of + NOT_BOUND, and collapsing either into it makes a claim the data does not + support. + """ + + BOUND = "bound" + NOT_BOUND = "not bound" + NEVER_ASKED = "never asked" + UNRELIABLE = "unreliable" + + +def combine_tags_to_identities(counts: pl.DataFrame, grouping: dict[str, str]) -> pl.DataFrame: + """An identity's reading in a cell is the highest of its tags' counts. + + Counts are not added together and summing is not offered. Requiring every + tag to clear was measured and is the worst option available. Summing would + need the reference scaled to a summed identity, which assumes each tag picks + up background at the rate the reference does — and tags differ in how + readily they are taken up, by an amount nobody has measured. + """ + mapped = counts.with_columns(pl.col("tag").replace_strict(grouping, default=None).alias("identity")).filter( + pl.col("identity").is_not_null() + ) + return mapped.group_by([*CELL_KEY, "identity"]).agg(pl.col("umiCount").max().alias("umiCount")) + + +def densify(identities: pl.DataFrame, cells: pl.DataFrame, offered_by_sample: dict[str, set[str]]) -> pl.DataFrame: + """Every cell against every identity its sample offered, zeros filled in. + + tag-stat emits only observed pairs, so without this an antigen every cell + failed to bind produces no rows and its failure is indistinguishable from a + reagent nobody offered. The epitope-mapping case turns on exactly that + distinction: there, *not bound* is the finding. + + This is the reference implementation. Production uses `silent_tally` + instead: on a realistic run this grid is 11-20x the sparse input and does + not fit a large panel at all. This function exists so a test can hold it + up as the oracle `silent_tally` is checked against, never to run in the + block itself. + """ + grid = ( + pl.concat( + [ + cells.filter(pl.col("sampleId") == sample).join( + pl.DataFrame({"identity": sorted(offered)}), how="cross" + ) + for sample, offered in sorted(offered_by_sample.items()) + if offered + ], + how="vertical", + ) + if offered_by_sample + else cells.head(0).with_columns(pl.lit(None, pl.String).alias("identity")) + ) + + return grid.join(identities, on=[*CELL_KEY, "identity"], how="left").with_columns( + pl.col("umiCount").fill_null(0).cast(pl.Int64) + ) + + +def specificity_score(antigen_count, reference_count): + """How specifically the antigen count exceeds the reference: 0-100. + + At antigen_count = 0 this is 0.042 at reference_count = 0 and falls for + every larger reference_count. It cannot clear any cutoff this block offers + above that, which is what lets a silent cell's state be known without a row + ever being written for it. + """ + a = np.asarray(antigen_count, dtype=float) + BETA_A_OFFSET + b = np.asarray(reference_count, dtype=float) + BETA_B_OFFSET + return (1.0 - beta.cdf(BETA_X, a, b)) * 100.0 + + +def _cell_admissibility_reason( + key: tuple[str, str], reference: dict[tuple[str, str], int], thin_line: int, gated: set[tuple[str, str]] +) -> str | None: + """Why this cell's comparison cannot be made, or None if it can be. + + Identity-independent: a cell that cannot be compared cannot be compared + against any identity, so this takes no identity and answers the same way + for every one the cell was asked about. `read_states` and `silent_tally` + both call this rather than each carrying its own copy of the same three + checks. + + `key not in reference` is deliberate, not `reference.get(key, 0)`: a + missing key means no comparator existed for this cell, and defaulting it + to 0 would read as "the comparator served and found nothing" — a settled + comparison rather than the absence of one. + """ + if key in gated: + return "cell set aside by the admissibility gate" + if key not in reference: + return "no comparator for this cell" + if reference[key] < thin_line: + return "the comparator rests on too little to compare against" + return None + + +def read_states( + identities: pl.DataFrame, + reference: dict[tuple[str, str], int], + cutoff: float, + thin_line: int, + gated: set[tuple[str, str]], +) -> pl.DataFrame: + """Give every (cell, identity) row a state. + + Three routes to UNRELIABLE and they mean different things, all recorded in + `unreliableReason`: the cell has no comparator; the comparator rests on + almost nothing, which is the absence of a comparison rather than a poor one; + or an admissibility gate set the cell aside. Gated cells stay in the frame — + dropping them made a set whose every cell was set aside read *never asked* + instead of *unreliable*. + + Emits umiCount and referenceCount, never the score. Re-derivation under a + new grouping needs the counts, and no binding level may leave the block. + """ + keys = list(zip(identities["sampleId"].to_list(), identities["cellId"].to_list(), strict=True)) + reasons = [_cell_admissibility_reason(k, reference, thin_line, gated) for k in keys] + refs = [reference.get(k) for k in keys] + + df = identities.with_columns( + pl.Series("referenceCount", refs, dtype=pl.Int64), + pl.Series("unreliableReason", reasons, dtype=pl.String), + ) + + scored = specificity_score( + df["umiCount"].to_numpy(), + np.nan_to_num(df["referenceCount"].cast(pl.Float64).to_numpy(), nan=0.0), + ) + + df = df.with_columns(pl.Series("_score", scored, dtype=pl.Float64)).with_columns( + pl.when(pl.col("unreliableReason").is_not_null()) + .then(pl.lit(State.UNRELIABLE.value)) + .when(pl.col("_score") >= cutoff) + .then(pl.lit(State.BOUND.value)) + .otherwise(pl.lit(State.NOT_BOUND.value)) + .alias("state") + ) + + return df.select([*CELL_KEY, "identity", "umiCount", "referenceCount", "state", "unreliableReason"]) + + +def silent_tally( + observed: pl.DataFrame, + cells: pl.DataFrame, + offered_by_sample: dict[str, set[str]], + reference: dict[tuple[str, str], int], + thin_line: int, + gated: set[tuple[str, str]], +) -> pl.DataFrame: + """Per (sample, identity): how many asked cells were never observed, and how they resolve. + + The production path A1 describes. `densify` followed by `read_states` is + the reference this must agree with, kept only for tests: on a realistic + panel the dense grid is 11-20x the sparse input and does not fit at all. + + A silently admissible cell's count is 0, and specificity_score(0, r) is + 0.042 at r = 0 and smaller for every larger r — below every cutoff this + block offers. So a silent cell resolves to NOT_BOUND unless the cell itself + cannot be compared (gated, no comparator, or below the thin line), which is + a per-cell fact independent of which identity was silent. That is what lets + this be three cheap terms instead of a materialized row per silent cell: + + asked = cells of the sample, for every identity it offered + observed = the (cell, identity) rows read_states already produced + silentUnreliable = inadmissible cells of the sample − inadmissible cells among the observed + silentNotBound = asked − observed − silentUnreliable + + `observed` is `read_states`' output on the sparse frame — one row per + (cell, identity) pair `combine_tags_to_identities` actually produced, state + already resolved. Nothing here inspects an identity's silent reading; the + admissibility check that decides silentUnreliable never takes an identity. + """ + keys = list(zip(cells["sampleId"].to_list(), cells["cellId"].to_list(), strict=True)) + inadmissible = {k for k in keys if _cell_admissibility_reason(k, reference, thin_line, gated) is not None} + + sample_of = dict(zip(keys, cells["sampleId"].to_list(), strict=True)) + asked_count = {} + for sample in offered_by_sample: + asked_count[sample] = sum(1 for k in keys if sample_of[k] == sample) + inadmissible_count = {} + for sample in offered_by_sample: + inadmissible_count[sample] = sum(1 for k in keys if sample_of[k] == sample and k in inadmissible) + + obs_keys = list(zip(observed["sampleId"].to_list(), observed["cellId"].to_list(), strict=True)) + obs_identity = observed["identity"].to_list() + observed_count: dict[tuple[str, str], int] = {} + observed_inadmissible_count: dict[tuple[str, str], int] = {} + for k, ident in zip(obs_keys, obs_identity, strict=True): + sample = sample_of.get(k) + if sample is None: + continue + pair = (sample, ident) + observed_count[pair] = observed_count.get(pair, 0) + 1 + if k in inadmissible: + observed_inadmissible_count[pair] = observed_inadmissible_count.get(pair, 0) + 1 + + rows = [] + for sample, offered in sorted(offered_by_sample.items()): + asked = asked_count.get(sample, 0) + total_inadmissible = inadmissible_count.get(sample, 0) + for identity in sorted(offered): + pair = (sample, identity) + observed_n = observed_count.get(pair, 0) + observed_inadmissible_n = observed_inadmissible_count.get(pair, 0) + silent_unreliable = total_inadmissible - observed_inadmissible_n + silent_not_bound = asked - observed_n - silent_unreliable + rows.append((sample, identity, asked, observed_n, silent_unreliable, silent_not_bound)) + + return pl.DataFrame( + rows, + orient="row", + schema={ + "sampleId": pl.String, + "identity": pl.String, + "asked": pl.Int64, + "observed": pl.Int64, + "silentUnreliable": pl.Int64, + "silentNotBound": pl.Int64, + }, + ) diff --git a/software/per-cell-metrics/test/test_verdict.py b/software/per-cell-metrics/test/test_verdict.py index d0c4c41..d278920 100644 --- a/software/per-cell-metrics/test/test_verdict.py +++ b/software/per-cell-metrics/test/test_verdict.py @@ -1,14 +1,25 @@ +import math +import random + import polars as pl +from scipy.stats import beta from verdict import ( + BOUND_CUTOFF, DEFAULT_FLOOR, DEFAULT_HIGH_REFERENCE_OBSERVATION_LINE, DEFAULT_PANEL_MIN_MEMBERS, DEFAULT_REFERENCE_THIN_LINE, ReferenceChoice, + State, apply_floor, + combine_tags_to_identities, + densify, gate_cells, + read_states, reference_by_cell, resolve_default_source, + silent_tally, + specificity_score, ) @@ -306,3 +317,225 @@ def test_the_panel_source_also_respects_the_given_cell_list(): ) assert choice is ReferenceChoice.PANEL assert ref == {("S1", "c1"): 9} + + +def _ident(rows): + return pl.DataFrame( + rows, + orient="row", + schema={"sampleId": pl.String, "cellId": pl.String, "identity": pl.String, "umiCount": pl.Int64}, + ) + + +def _cells(pairs): + return pl.DataFrame(pairs, orient="row", schema={"sampleId": pl.String, "cellId": pl.String}) + + +def test_state_has_exactly_four_members(): + assert {s.value for s in State} == {"bound", "not bound", "never asked", "unreliable"} + + +def test_densify_gives_a_silent_cell_a_real_zero(): + counts = _ident([("S1", "c1", "A", 7)]) + cells = _cells([("S1", "c1")]) + out = densify(counts, cells, offered_by_sample={"S1": {"A", "B"}}).sort("identity") + assert out["identity"].to_list() == ["A", "B"] + assert out["umiCount"].to_list() == [7, 0] # B was asked and silent + + +def test_densify_does_not_invent_unoffered_identities(): + counts = _ident([("S1", "c1", "A", 7)]) + cells = _cells([("S1", "c1")]) + out = densify(counts, cells, offered_by_sample={"S1": {"A"}}) + assert out["identity"].to_list() == ["A"] + + +def test_identity_reading_is_the_highest_not_the_sum(): + df = _counts([("S1", "c1", "AAAA", 10), ("S1", "c1", "CCCC", 7)]) + out = combine_tags_to_identities(df, {"AAAA": "A", "CCCC": "A"}) + assert out["umiCount"].to_list() == [10] + + +def test_combine_keeps_sample_id(): + df = _counts([("S1", "c1", "AAAA", 5), ("S2", "c1", "AAAA", 9)]) + out = combine_tags_to_identities(df, {"AAAA": "A"}) + assert out.height == 2 and set(out["sampleId"].to_list()) == {"S1", "S2"} + + +def test_specificity_score_matches_the_published_formula(): + assert math.isclose(specificity_score(10, 2), (1.0 - beta.cdf(0.925, 11, 5)) * 100.0, rel_tol=1e-12) + + +def test_cutoff_is_seventy_five(): + assert BOUND_CUTOFF == 75.0 + + +def test_high_count_against_a_quiet_reference_is_bound(): + # thin_line=0 here, not the default 2: a reference reading of 0 is itself + # below the default thin line (Task 5's own precedent -- a median that + # truncates to 1 already reads unreliable at that default), so pinning the + # score computation in isolation needs the thin-line rule turned off. + out = read_states(_ident([("S1", "c1", "A", 200)]), {("S1", "c1"): 0}, 75.0, 0, set()) + assert out["state"].to_list() == [State.BOUND.value] + + +def test_zero_reads_not_bound_never_unreliable(): + out = read_states(_ident([("S1", "c1", "A", 0)]), {("S1", "c1"): 5}, 75.0, 2, set()) + assert out["state"].to_list() == [State.NOT_BOUND.value] + + +def test_no_reference_reading_is_unreliable(): + out = read_states(_ident([("S1", "c9", "A", 50)]), {}, 75.0, 2, set()) + assert out["state"].to_list() == [State.UNRELIABLE.value] + + +def test_reference_below_the_thin_line_is_unreliable_not_scored(): + # A comparison against almost nothing is not a comparison. + out = read_states(_ident([("S1", "c1", "A", 50)]), {("S1", "c1"): 1}, 75.0, thin_line=2, gated=set()) + assert out["state"].to_list() == [State.UNRELIABLE.value] + + +def test_gated_cell_is_unreliable_and_stays_in_the_frame(): + out = read_states(_ident([("S1", "c1", "A", 500)]), {("S1", "c1"): 900}, 75.0, 2, gated={("S1", "c1")}) + assert out.height == 1 + assert out["state"].to_list() == [State.UNRELIABLE.value] + + +def test_never_asked_is_not_produced_here(): + out = read_states(_ident([("S1", "c1", "A", 0)]), {("S1", "c1"): 5}, 75.0, 2, set()) + assert State.NEVER_ASKED.value not in out["state"].to_list() + + +def test_no_score_column_leaves_the_reading(): + out = read_states(_ident([("S1", "c1", "A", 50)]), {("S1", "c1"): 5}, 75.0, 2, set()) + assert "score" not in out.columns + assert {"umiCount", "referenceCount"} <= set(out.columns) + + +def test_score_bounded(): + for a, r in [(0, 0), (5, 5), (1000, 3)]: + assert 0.0 <= specificity_score(a, r) <= 100.0 + + +def test_a_score_exactly_at_the_cutoff_is_bound(): + # The named value satisfies the condition it names, as everywhere else here. + # Integer counts have no rational preimage of a fixed cutoff like 75.0 under + # the beta CDF, so the exact boundary is built the other way round: compute + # a reading's own score, then feed that exact value back in as the cutoff. + # The comparison then lands on the line with no floating-point drift, and + # ">=" must call it bound. + exact = specificity_score(10, 2) + out = read_states(_ident([("S1", "c1", "A", 10)]), {("S1", "c1"): 2}, cutoff=exact, thin_line=2, gated=set()) + assert out["state"].to_list() == [State.BOUND.value] + + +def test_a_reference_exactly_at_the_thin_line_is_scored_not_unreliable(): + # Below the line the comparison does not exist; AT the line it does. This + # is the pair that makes the thin line a floor rather than a gap. + at_line = read_states(_ident([("S1", "c1", "A", 0)]), {("S1", "c1"): 2}, 75.0, thin_line=2, gated=set()) + below_line = read_states(_ident([("S1", "c2", "A", 0)]), {("S1", "c2"): 1}, 75.0, thin_line=2, gated=set()) + assert at_line["unreliableReason"].to_list() == [None] + assert at_line["state"].to_list() == [State.NOT_BOUND.value] + assert below_line["state"].to_list() == [State.UNRELIABLE.value] + + +def test_no_comparator_is_unreliable_but_a_comparator_reading_zero_is_scored(): + # The two must not collapse. served=NONE (modelled here as an empty + # reference dict, per reference_by_cell's contract) means no comparison + # existed; a comparator present and reading 0 is a real comparison and + # scores normally -- a positive antigen count against a zero reference is + # the strongest evidence there is. thin_line=0 isolates that from the + # separate (and, at the default of 2, overlapping) thin-line rule. + no_comparator = read_states(_ident([("S1", "c1", "A", 200)]), {}, 75.0, 0, set()) + zero_comparator = read_states(_ident([("S1", "c1", "A", 200)]), {("S1", "c1"): 0}, 75.0, 0, set()) + assert no_comparator["state"].to_list() == [State.UNRELIABLE.value] + assert no_comparator["unreliableReason"].to_list() == ["no comparator for this cell"] + assert zero_comparator["state"].to_list() == [State.BOUND.value] + assert zero_comparator["unreliableReason"].to_list() == [None] + + +def test_silent_admissible_cell_can_never_score_bound(): + # The fact the analytic path rests on: specificity_score(0, r) is 0.042 at + # r = 0 and smaller for every larger r, below every cutoff this block + # offers above that. A silent admissible cell is therefore always + # *not bound*, which is what lets silent_tally skip materializing its row. + assert math.isclose(specificity_score(0, 0), 0.042, abs_tol=5e-4) + scores = [specificity_score(0, r) for r in range(0, 50)] + assert scores == sorted(scores, reverse=True) + assert all(s < 0.05 for s in scores) + + +def test_silent_tally_agrees_with_the_densify_oracle_on_small_random_inputs(): + # A property test: build a small, varied population by construction -- + # several samples, cells, identities, some cells gated, some below the + # thin line, some with a normal reference -- and check silent_tally's + # three cheap terms against the dense grid built by densify and read + # through read_states, which never skips a row. + rng = random.Random(20260817) + samples = ["S1", "S2", "S3"] + identities = ["A", "B", "C"] + thin_line = 2 + gated: set[tuple[str, str]] = set() + reference: dict[tuple[str, str], int] = {} + cell_rows = [] + tag_rows = [] + offered_by_sample: dict[str, set[str]] = {} + + for sample in samples: + offered_by_sample[sample] = set(rng.sample(identities, k=rng.randint(1, len(identities)))) + for i in range(6): + cell = f"c{i}" + cell_rows.append((sample, cell)) + key = (sample, cell) + # Reference reading: sometimes missing (no comparator), sometimes + # thin, sometimes ordinary. + roll = rng.random() + if roll < 0.2: + pass # no comparator for this cell + elif roll < 0.4: + reference[key] = 1 # below thin_line=2 + else: + reference[key] = rng.randint(2, 20) + if rng.random() < 0.15: + gated.add(key) + # Sparse observed readings: only some (cell, identity) pairs the + # sample offered actually got a tag-stat row. + for identity in offered_by_sample[sample]: + if rng.random() < 0.5: + tag_rows.append((sample, cell, identity, rng.randint(0, 30))) + + cells = _cells(cell_rows) + sparse_identities = _ident(tag_rows) + observed = read_states(sparse_identities, reference, BOUND_CUTOFF, thin_line, gated) + + dense = densify(sparse_identities, cells, offered_by_sample) + oracle = read_states(dense, reference, BOUND_CUTOFF, thin_line, gated) + + tally = silent_tally(observed, cells, offered_by_sample, reference, thin_line, gated) + + for sample in samples: + for identity in offered_by_sample[sample]: + oracle_group = oracle.filter((pl.col("sampleId") == sample) & (pl.col("identity") == identity)) + observed_group = observed.filter((pl.col("sampleId") == sample) & (pl.col("identity") == identity)) + tally_row = tally.filter((pl.col("sampleId") == sample) & (pl.col("identity") == identity)).row( + 0, named=True + ) + + oracle_states = oracle_group["state"].to_list() + observed_states = observed_group["state"].to_list() + + # A silent admissible cell can never be observed as bound, so the + # oracle and the sparse frame must agree exactly on bound counts. + assert oracle_states.count(State.BOUND.value) == observed_states.count(State.BOUND.value) + + expected_silent_unreliable = oracle_states.count(State.UNRELIABLE.value) - observed_states.count( + State.UNRELIABLE.value + ) + expected_silent_not_bound = oracle_states.count(State.NOT_BOUND.value) - observed_states.count( + State.NOT_BOUND.value + ) + + assert tally_row["asked"] == len(oracle_states) + assert tally_row["observed"] == len(observed_states) + assert tally_row["silentUnreliable"] == expected_silent_unreliable + assert tally_row["silentNotBound"] == expected_silent_not_bound From 10f3ae4384427e7618d0253bb9f52af205aa3f02 Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 14:30:28 +0200 Subject: [PATCH 031/282] MILAB-6496: state the mismatch table's contract and collapse its duplicated rule --- software/per-cell-metrics/src/panel.py | 46 +++++++++++++------- software/per-cell-metrics/test/test_panel.py | 32 +++++++++----- 2 files changed, 52 insertions(+), 26 deletions(-) diff --git a/software/per-cell-metrics/src/panel.py b/software/per-cell-metrics/src/panel.py index 61eff99..c6c3908 100644 --- a/software/per-cell-metrics/src/panel.py +++ b/software/per-cell-metrics/src/panel.py @@ -257,10 +257,19 @@ def panel_read_mismatch(panel: pl.DataFrame, seen: pl.DataFrame) -> pl.DataFrame Per sample, because the same barcode can carry a different antigen in a different sample's panel: a global check lets a barcode undeclared in one sample pass on another sample's declaration. + + In the global case every row is keyed ANY_SAMPLE, so "*" appears as a value + beside real sample ids — it is not a sampleId. For declared-never-seen that + key is honest, because the claim really is global. For undeclared-in-panel + it is lossy: a barcode read only in one sample reports under "*" and which + sample carried it is not recoverable. That is accepted, because a panel with + no sample dimension has no per-sample declaration to compare against. """ - # A row with no sample or no barcode cannot be placed on either side of the - # comparison, and a null key is not a usable p-column key. Dropping them - # keeps the promise that this check never raises. + # Neither side can place a row with no sample or no barcode, and a null is + # not a usable p-column key. The reader never emits one; this keeps the + # promise true for a caller that builds a frame directly, as the star + # branch below does for the same reason. + panel = panel.filter(pl.col("sample").is_not_null() & pl.col("tag").is_not_null()) seen = seen.filter(pl.col("sampleId").is_not_null() & pl.col("tag").is_not_null()) rows = [] @@ -270,22 +279,27 @@ def panel_read_mismatch(panel: pl.DataFrame, seen: pl.DataFrame) -> pl.DataFrame # the global branch, which would discard every named row and report a # per-sample disagreement as agreement. The reader refuses such a frame, so # this is the second line of defence for a caller that builds one directly. + # In such a frame "*" is then compared as a literal sample name, so a star + # row reports a disagreement against a sample called "*". Those extra rows + # are intended: a caller who builds a frame the reader refuses gets noise + # rather than a silent pass. if panel.height and global_panel.height == panel.height: - declared = set(global_panel["tag"].to_list()) - observed = set(seen["tag"].to_list()) + pairs = [(ANY_SAMPLE, set(global_panel["tag"].to_list()), set(seen["tag"].to_list()))] + else: + pairs = [ + ( + s, + set(panel.filter(pl.col("sample") == s)["tag"].to_list()), + set(seen.filter(pl.col("sampleId") == s)["tag"].to_list()), + ) + for s in sorted(set(panel["sample"].to_list()) | set(seen["sampleId"].to_list())) + ] + + for sample, declared, observed in pairs: for tag in sorted(declared - observed): - rows.append({"sample": ANY_SAMPLE, "tag": tag, "direction": "declared-never-seen"}) + rows.append({"sample": sample, "tag": tag, "direction": "declared-never-seen"}) for tag in sorted(observed - declared): - rows.append({"sample": ANY_SAMPLE, "tag": tag, "direction": "undeclared-in-panel"}) - else: - samples = sorted(set(panel["sample"].to_list()) | set(seen["sampleId"].to_list())) - for sample in samples: - declared = set(panel.filter(pl.col("sample") == sample)["tag"].to_list()) - observed = set(seen.filter(pl.col("sampleId") == sample)["tag"].to_list()) - for tag in sorted(declared - observed): - rows.append({"sample": sample, "tag": tag, "direction": "declared-never-seen"}) - for tag in sorted(observed - declared): - rows.append({"sample": sample, "tag": tag, "direction": "undeclared-in-panel"}) + rows.append({"sample": sample, "tag": tag, "direction": "undeclared-in-panel"}) return pl.DataFrame(rows, schema={"sample": pl.String, "tag": pl.String, "direction": pl.String}).sort( ["sample", "direction", "tag"] diff --git a/software/per-cell-metrics/test/test_panel.py b/software/per-cell-metrics/test/test_panel.py index 036aa7d..76541ef 100644 --- a/software/per-cell-metrics/test/test_panel.py +++ b/software/per-cell-metrics/test/test_panel.py @@ -338,7 +338,7 @@ def test_a_barcode_declared_in_another_sample_does_not_pass_silently(): assert ("S1", "AAAA") in list(zip(undeclared["sample"], undeclared["tag"], strict=True)) -def test_star_panel_checks_globally(): +def test_a_star_panel_is_satisfied_by_reads_in_any_sample(): panel = pl.DataFrame({"tag": ["AAAA"], "sample": ["*"], "Name": ["a"]}) seen = _counts([("S1", "c1", "AAAA", 1), ("S2", "c1", "AAAA", 1)]) assert panel_read_mismatch(panel, seen).height == 0 @@ -347,8 +347,11 @@ def test_star_panel_checks_globally(): def test_mismatch_never_raises(): panel = pl.DataFrame({"tag": ["AAAA"], "sample": ["S1"], "Name": ["a"]}) seen = _counts([("S9", "c1", "ZZZZ", 1)]) - out = panel_read_mismatch(panel, seen) # must not raise - assert out.height >= 1 + rows = {(r["sample"], r["tag"], r["direction"]) for r in panel_read_mismatch(panel, seen).to_dicts()} + assert rows == { + ("S1", "AAAA", "declared-never-seen"), + ("S9", "ZZZZ", "undeclared-in-panel"), + } def test_a_sample_with_reads_but_no_panel_rows_reports_every_barcode(): @@ -407,19 +410,28 @@ def test_a_literal_star_in_a_sample_column_is_fatal(tmp_path): def test_a_mixed_star_and_named_panel_does_not_go_global(): # Second line of defence: the reader refuses this frame, but a caller # building one directly must not get an empty table for a real disagreement. + # The output is asserted in full, not just membership: a mixed frame reports + # every row as noise, including the star row compared as a literal sample + # name, and a future "clean that spurious row up" edit must not pass here. panel = pl.DataFrame({"tag": ["AAAA", "CCCC"], "sample": ["*", "S1"], "Name": ["a", "c"]}) seen = _counts([("S1", "c1", "AAAA", 5)]) rows = {(r["sample"], r["tag"], r["direction"]) for r in panel_read_mismatch(panel, seen).to_dicts()} - assert ("S1", "CCCC", "declared-never-seen") in rows + assert rows == { + ("*", "AAAA", "declared-never-seen"), + ("S1", "AAAA", "undeclared-in-panel"), + ("S1", "CCCC", "declared-never-seen"), + } -def test_null_keys_in_the_reads_do_not_raise(): - panel = pl.DataFrame({"tag": ["AAAA"], "sample": ["S1"], "Name": ["a"]}) - seen = pl.DataFrame( - [("S1", "c1", "AAAA", 5), (None, "c2", "CCCC", 3), ("S1", "c3", None, 1)], - orient="row", - schema={"sampleId": pl.String, "cellId": pl.String, "tag": pl.String, "umiCount": pl.Int64}, +def test_null_keys_on_either_side_do_not_raise(): + # A null tag or a null sample cannot be placed on either side of the + # comparison, on either input — the panel side gets the same guard as the + # reads side, not a narrower one. + panel = pl.DataFrame( + {"tag": ["AAAA", "CCCC", None], "sample": ["S1", None, "S1"], "Name": ["a", "c", "z"]}, + schema={"tag": pl.String, "sample": pl.String, "Name": pl.String}, ) + seen = _counts([("S1", "c1", "AAAA", 5), (None, "c2", "CCCC", 3), ("S1", "c3", None, 1)]) out = panel_read_mismatch(panel, seen) assert out.height == 0 assert None not in out["sample"].to_list() From bb43618cbcdd682403805dc4d3e53f823d76ba3b Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 14:52:20 +0200 Subject: [PATCH 032/282] MILAB-6496: describe the sparse path the module actually takes The module header still listed densify as production step 2 while the code below it said densify never runs in the block. Recast as the four production steps, with the silent positions counted analytically and densify named as the test oracle. Pins the admissibility precedence, which is observable through the exported reason column, and stops densify raising on a sample stained with nothing. --- software/per-cell-metrics/src/verdict.py | 59 +++++++++++-------- .../per-cell-metrics/test/test_verdict.py | 30 +++++++++- 2 files changed, 60 insertions(+), 29 deletions(-) diff --git a/software/per-cell-metrics/src/verdict.py b/software/per-cell-metrics/src/verdict.py index 88e439e..76814b2 100644 --- a/software/per-cell-metrics/src/verdict.py +++ b/software/per-cell-metrics/src/verdict.py @@ -1,17 +1,21 @@ """Turning a cell's counts into states. -Five steps, in this order, and the order is load-bearing: +Four steps in production, in this order, and the order is load-bearing: 1. the floor, on the raw count, per cell and per tag; - 2. densify — every cell against every identity its sample offered, so a cell - asked and silent is a real zero rather than a missing row; - 3. tags combine into an identity by the highest of their counts; - 4. the identity's count is read against that cell's own reference reading; - 5. the comparison becomes one of the four states. - -Step 2 exists because tag-stat emits only observed pairs. Without it an antigen -every cell failed to bind produces no rows at all, and the absence is -indistinguishable from a reagent nobody offered. + 2. tags combine into an identity by the highest of their counts; + 3. the identity's count is read against that cell's own reference reading; + 4. the comparison becomes one of the four states. + +tag-stat emits only observed pairs, so a cell asked about an identity and +silent — no positive tag reading for it — produces no row at all, and that +absence is not evidence of nothing: an antigen every cell failed to bind must +still read *not bound*, not disappear as though nobody offered it. Rather than +materialize a row for every such silent position, production counts them +analytically, in `silent_tally`. `densify` — every cell against every +identity its sample offered, zeros filled in — builds that row-per-position +grid and survives only as the test oracle `silent_tally`'s counts are checked +against; it never runs in the block. The cell key is (sampleId, cellId) throughout: cell barcodes are bare 16-mers shared across samples. @@ -23,7 +27,7 @@ reading is still a reading, and it answers "not bound"; an omitted one leaves nothing to answer with. -After step 3 this module holds two frame shapes: the sparse per-tag frame the +After step 2 this module holds two frame shapes: the sparse per-tag frame the floor works on, and the per-identity frame combining produces from it — both keyed by CELL_KEY, which is the column vocabulary spanning both. """ @@ -325,18 +329,18 @@ def densify(identities: pl.DataFrame, cells: pl.DataFrame, offered_by_sample: di up as the oracle `silent_tally` is checked against, never to run in the block itself. """ + # Guard on the assembled blocks, not on offered_by_sample: a map whose every + # value is empty — a sample stained with nothing — is non-empty itself but + # contributes no block, and concat of an empty list raises. That shape is + # exactly what a property test probing a stained-with-nothing sample builds. + blocks = [ + cells.filter(pl.col("sampleId") == sample).join(pl.DataFrame({"identity": sorted(offered)}), how="cross") + for sample, offered in sorted(offered_by_sample.items()) + if offered + ] grid = ( - pl.concat( - [ - cells.filter(pl.col("sampleId") == sample).join( - pl.DataFrame({"identity": sorted(offered)}), how="cross" - ) - for sample, offered in sorted(offered_by_sample.items()) - if offered - ], - how="vertical", - ) - if offered_by_sample + pl.concat(blocks, how="vertical") + if blocks else cells.head(0).with_columns(pl.lit(None, pl.String).alias("identity")) ) @@ -397,7 +401,9 @@ def read_states( almost nothing, which is the absence of a comparison rather than a poor one; or an admissibility gate set the cell aside. Gated cells stay in the frame — dropping them made a set whose every cell was set aside read *never asked* - instead of *unreliable*. + instead of *unreliable*. The gate is checked first: a cell the gate set + aside was not measured at all, so how thin its comparator is does not + matter and must not be the reason reported. Emits umiCount and referenceCount, never the score. Re-derivation under a new grouping needs the counts, and no binding level may leave the block. @@ -438,9 +444,10 @@ def silent_tally( ) -> pl.DataFrame: """Per (sample, identity): how many asked cells were never observed, and how they resolve. - The production path A1 describes. `densify` followed by `read_states` is - the reference this must agree with, kept only for tests: on a realistic - panel the dense grid is 11-20x the sparse input and does not fit at all. + The sparse path: silent positions are counted, never materialized. + `densify` followed by `read_states` is the reference this must agree + with, kept only for tests: on a realistic panel the dense grid is + 11-20x the sparse input and does not fit at all. A silently admissible cell's count is 0, and specificity_score(0, r) is 0.042 at r = 0 and smaller for every larger r — below every cutoff this diff --git a/software/per-cell-metrics/test/test_verdict.py b/software/per-cell-metrics/test/test_verdict.py index d278920..33f5fc2 100644 --- a/software/per-cell-metrics/test/test_verdict.py +++ b/software/per-cell-metrics/test/test_verdict.py @@ -372,9 +372,9 @@ def test_cutoff_is_seventy_five(): def test_high_count_against_a_quiet_reference_is_bound(): # thin_line=0 here, not the default 2: a reference reading of 0 is itself - # below the default thin line (Task 5's own precedent -- a median that - # truncates to 1 already reads unreliable at that default), so pinning the - # score computation in isolation needs the thin-line rule turned off. + # below the default thin line (a panel median that truncates to 1 already + # reads unreliable at that default), so pinning the score computation in + # isolation needs the thin-line rule turned off. out = read_states(_ident([("S1", "c1", "A", 200)]), {("S1", "c1"): 0}, 75.0, 0, set()) assert out["state"].to_list() == [State.BOUND.value] @@ -401,6 +401,30 @@ def test_gated_cell_is_unreliable_and_stays_in_the_frame(): assert out["state"].to_list() == [State.UNRELIABLE.value] +def test_a_gated_cell_reports_the_gate_even_when_its_reference_is_thin(): + # Both conditions hold at once: the gate set this cell aside AND its + # comparator is below the thin line. Reachable whenever the gate threshold + # sits at or below the thin line, and both are user-set. The state is + # unreliable either way, but the reason is an exported column that a later + # step reads to tell a panel problem from a re-run problem, so which + # condition wins is a fact someone acts on. The gate wins: a cell it set + # aside was not measured at all, so its comparator's thickness is moot. + out = read_states(_ident([("S1", "c1", "A", 500)]), {("S1", "c1"): 1}, 75.0, thin_line=2, gated={("S1", "c1")}) + assert out["state"].to_list() == [State.UNRELIABLE.value] + reason = out["unreliableReason"].to_list()[0] + assert "gate" in reason + assert "compare" not in reason # not the thin-comparator reason + + +def test_densify_handles_a_sample_stained_with_nothing(): + # A non-empty offered map whose every value is empty contributes no block. + # Guarding on the map rather than the assembled blocks raised here. + out = densify(_ident([]), _cells([("S1", "c1")]), offered_by_sample={"S1": set()}) + assert out.height == 0 + assert out.schema["identity"] == pl.String + assert out.schema["umiCount"] == pl.Int64 + + def test_never_asked_is_not_produced_here(): out = read_states(_ident([("S1", "c1", "A", 0)]), {("S1", "c1"): 5}, 75.0, 2, set()) assert State.NEVER_ASKED.value not in out["state"].to_list() From 51fce7e164f6683b9a99f1ee416b849a4ad736b5 Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 15:10:33 +0200 Subject: [PATCH 033/282] MILAB-6496: name the tally's preconditions and its reason codes --- software/per-cell-metrics/src/verdict.py | 189 +++++++++++++----- .../per-cell-metrics/test/test_verdict.py | 169 +++++++++++----- 2 files changed, 252 insertions(+), 106 deletions(-) diff --git a/software/per-cell-metrics/src/verdict.py b/software/per-cell-metrics/src/verdict.py index 76814b2..70152e2 100644 --- a/software/per-cell-metrics/src/verdict.py +++ b/software/per-cell-metrics/src/verdict.py @@ -27,9 +27,11 @@ reading is still a reading, and it answers "not bound"; an omitted one leaves nothing to answer with. -After step 2 this module holds two frame shapes: the sparse per-tag frame the -floor works on, and the per-identity frame combining produces from it — both -keyed by CELL_KEY, which is the column vocabulary spanning both. +After step 2 this module holds three frame shapes: the sparse per-tag frame +the floor works on and the per-identity frame combining produces from it, +both keyed by CELL_KEY, which is the column vocabulary spanning both; and the +per-(sampleId, identity) frame `silent_tally` returns, keyed one level +coarser than CELL_KEY, not by it. """ from __future__ import annotations @@ -77,10 +79,12 @@ def apply_floor(counts: pl.DataFrame, floor: int, reference_tags: set[str]) -> F counters that land in this sample's row of the QC report. Both counters assume the sparse frame this step receives, where every row - is an observed reading and so a count is at least 1. Densification, which - manufactures genuine zeros, happens after this step: run it before, and - every manufactured row inflates readingsFloored while every unbound cell - counts as emptied though the floor removed nothing. + is an observed reading and so a count is at least 1. If densification — + which manufactures genuine zeros — ever ran before this step, every + manufactured row would inflate readingsFloored while every unbound cell + would count as emptied though the floor removed nothing. In production it + never does: `densify` exists only as the test oracle `silent_tally` is + checked against, and never runs in the block. """ # Not an optimisation: falling through would count a cell whose only # reading is already zero as "emptied", when the floor removed nothing. @@ -300,6 +304,16 @@ class State(str, Enum): UNRELIABLE = "unreliable" +class UnreliableReason(str, Enum): + """Why a cell's comparison could not be made. The value is the prose that + reaches a reader; the member is what code compares against, so the wording + can change without breaking a caller.""" + + GATED = "cell set aside by the admissibility gate" + NO_COMPARATOR = "no comparator for this cell" + THIN_COMPARATOR = "the comparator rests on too little to compare against" + + def combine_tags_to_identities(counts: pl.DataFrame, grouping: dict[str, str]) -> pl.DataFrame: """An identity's reading in a cell is the highest of its tags' counts. @@ -352,19 +366,40 @@ def densify(identities: pl.DataFrame, cells: pl.DataFrame, offered_by_sample: di def specificity_score(antigen_count, reference_count): """How specifically the antigen count exceeds the reference: 0-100. - At antigen_count = 0 this is 0.042 at reference_count = 0 and falls for - every larger reference_count. It cannot clear any cutoff this block offers - above that, which is what lets a silent cell's state be known without a row - ever being written for it. + At antigen_count = 0 this is specificity_score(0, 0) ~= 0.0422 at + reference_count = 0 and falls for every larger reference_count. That is + the module's central claim: `silent_tally` relies on a silent admissible + cell never scoring BOUND, which is what lets its state be known without a + row ever being written for it. The claim holds only for `cutoff` strictly + above 0.0422 — a cutoff at or below that bound breaks the equivalence + between `silent_tally` and the `densify` oracle with no error raised + here. This module does not refuse such a cutoff; refusing one is the + CLI's job. """ a = np.asarray(antigen_count, dtype=float) + BETA_A_OFFSET b = np.asarray(reference_count, dtype=float) + BETA_B_OFFSET return (1.0 - beta.cdf(BETA_X, a, b)) * 100.0 -def _cell_admissibility_reason( - key: tuple[str, str], reference: dict[tuple[str, str], int], thin_line: int, gated: set[tuple[str, str]] -) -> str | None: +class Admissibility(NamedTuple): + """The triple `read_states` and `silent_tally` must agree on to agree on + what "cannot be compared" means for a cell. + + Sharing `_cell_admissibility_reason` makes both functions agree on the + *rule*; it does nothing to make them agree on the *arguments* the rule is + applied to, since each call site built its own triple. Bundling the + triple here and passing the same one to both makes disagreement — e.g. + `read_states` given a reference restricted to observed cells while + `silent_tally` gets the full one, which sends `silentUnreliable` wrong or + negative — impossible by construction rather than by discipline. + """ + + reference: dict[tuple[str, str], int] + thin_line: int + gated: set[tuple[str, str]] + + +def _cell_admissibility_reason(key: tuple[str, str], admissibility: Admissibility) -> UnreliableReason | None: """Why this cell's comparison cannot be made, or None if it can be. Identity-independent: a cell that cannot be compared cannot be compared @@ -373,27 +408,22 @@ def _cell_admissibility_reason( both call this rather than each carrying its own copy of the same three checks. - `key not in reference` is deliberate, not `reference.get(key, 0)`: a - missing key means no comparator existed for this cell, and defaulting it - to 0 would read as "the comparator served and found nothing" — a settled - comparison rather than the absence of one. + `key not in admissibility.reference` is deliberate, not + `reference.get(key, 0)`: a missing key means no comparator existed for + this cell, and defaulting it to 0 would read as "the comparator served + and found nothing" — a settled comparison rather than the absence of one. """ + reference, thin_line, gated = admissibility if key in gated: - return "cell set aside by the admissibility gate" + return UnreliableReason.GATED if key not in reference: - return "no comparator for this cell" + return UnreliableReason.NO_COMPARATOR if reference[key] < thin_line: - return "the comparator rests on too little to compare against" + return UnreliableReason.THIN_COMPARATOR return None -def read_states( - identities: pl.DataFrame, - reference: dict[tuple[str, str], int], - cutoff: float, - thin_line: int, - gated: set[tuple[str, str]], -) -> pl.DataFrame: +def read_states(identities: pl.DataFrame, admissibility: Admissibility, cutoff: float) -> pl.DataFrame: """Give every (cell, identity) row a state. Three routes to UNRELIABLE and they mean different things, all recorded in @@ -407,14 +437,26 @@ def read_states( Emits umiCount and referenceCount, never the score. Re-derivation under a new grouping needs the counts, and no binding level may leave the block. + + `referenceCount` is nullable, and null is not 0: null means no comparator + served this cell at all, 0 means a comparator served and read nothing. A + downstream `fill_null(0)` on this column destroys exactly the distinction + this module argues for everywhere else — collapsing "not measured" into + "measured as zero". + + A cell present in `identities` but absent from the cell list + `silent_tally` is given is emitted a row here — this function does not + take a cell list to check against — and is silently dropped by + `silent_tally`, which only counts cells it was told about. """ + reference, _, _ = admissibility keys = list(zip(identities["sampleId"].to_list(), identities["cellId"].to_list(), strict=True)) - reasons = [_cell_admissibility_reason(k, reference, thin_line, gated) for k in keys] + reasons = [_cell_admissibility_reason(k, admissibility) for k in keys] refs = [reference.get(k) for k in keys] df = identities.with_columns( pl.Series("referenceCount", refs, dtype=pl.Int64), - pl.Series("unreliableReason", reasons, dtype=pl.String), + pl.Series("unreliableReason", [r.value if r is not None else None for r in reasons], dtype=pl.String), ) scored = specificity_score( @@ -438,9 +480,7 @@ def silent_tally( observed: pl.DataFrame, cells: pl.DataFrame, offered_by_sample: dict[str, set[str]], - reference: dict[tuple[str, str], int], - thin_line: int, - gated: set[tuple[str, str]], + admissibility: Admissibility, ) -> pl.DataFrame: """Per (sample, identity): how many asked cells were never observed, and how they resolve. @@ -450,11 +490,15 @@ def silent_tally( 11-20x the sparse input and does not fit at all. A silently admissible cell's count is 0, and specificity_score(0, r) is - 0.042 at r = 0 and smaller for every larger r — below every cutoff this - block offers. So a silent cell resolves to NOT_BOUND unless the cell itself - cannot be compared (gated, no comparator, or below the thin line), which is - a per-cell fact independent of which identity was silent. That is what lets - this be three cheap terms instead of a materialized row per silent cell: + specificity_score(0, 0) ~= 0.0422 at r = 0 and smaller for every larger r. + So a silent cell resolves to NOT_BOUND unless the cell itself cannot be + compared (gated, no comparator, or below the thin line), which is a + per-cell fact independent of which identity was silent — but only when + `cutoff` is strictly above that ~0.0422 bound (see `specificity_score`). + At or below it the dense oracle can call a silent admissible cell BOUND + while this function still reports it NOT_BOUND, silently: refusing such a + cutoff is the CLI's job, not this function's. Above the bound, three + cheap terms replace a materialized row per silent cell: asked = cells of the sample, for every identity it offered observed = the (cell, identity) rows read_states already produced @@ -465,33 +509,71 @@ def silent_tally( (cell, identity) pair `combine_tags_to_identities` actually produced, state already resolved. Nothing here inspects an identity's silent reading; the admissibility check that decides silentUnreliable never takes an identity. - """ - keys = list(zip(cells["sampleId"].to_list(), cells["cellId"].to_list(), strict=True)) - inadmissible = {k for k in keys if _cell_admissibility_reason(k, reference, thin_line, gated) is not None} - sample_of = dict(zip(keys, cells["sampleId"].to_list(), strict=True)) - asked_count = {} - for sample in offered_by_sample: - asked_count[sample] = sum(1 for k in keys if sample_of[k] == sample) - inadmissible_count = {} - for sample in offered_by_sample: - inadmissible_count[sample] = sum(1 for k in keys if sample_of[k] == sample and k in inadmissible) + Precondition, unchecked by types: `cells` must be unique on the cell key, + and `observed` unique on (cell, identity). A duplicated `cells` row is + harmless — it is deduplicated below — but a duplicated `observed` row is + not: it is double-counted against totals that count the cell once, which + can drive `silentUnreliable` negative (verified: -1 for a single + duplicated observed row on an inadmissible cell). The assertion below + turns that into a loud failure rather than a silently wrong number. + + A cell present in `identities` but absent from `cells` is emitted a row + by `read_states`, which takes no cell list to check against, and is + silently dropped here, where `cells` is the cell universe. + """ + # A duplicated cells row must not count twice: unlike a duplicated + # observed row (see the precondition above), this one is a legitimate + # no-op to guard against, not a contract violation to surface. + keys = list(dict.fromkeys(zip(cells["sampleId"].to_list(), cells["cellId"].to_list(), strict=True))) + inadmissible = {k for k in keys if _cell_admissibility_reason(k, admissibility) is not None} + cell_keys = set(keys) + + # Single accumulating pass, not one loop per sample: the previous version + # scanned all of `keys` once per sample in offered_by_sample, which is + # O(groups x cells) and harmless at 24 samples but quadratic once a wider + # key groups thousands of sets. `k[0]` is the sample directly — no need + # for a cell->sample dict when every key already carries it. + asked_count: dict[str, int] = {} + inadmissible_count: dict[str, int] = {} + for k in keys: + sample = k[0] + asked_count[sample] = asked_count.get(sample, 0) + 1 + if k in inadmissible: + inadmissible_count[sample] = inadmissible_count.get(sample, 0) + 1 obs_keys = list(zip(observed["sampleId"].to_list(), observed["cellId"].to_list(), strict=True)) obs_identity = observed["identity"].to_list() observed_count: dict[tuple[str, str], int] = {} observed_inadmissible_count: dict[tuple[str, str], int] = {} for k, ident in zip(obs_keys, obs_identity, strict=True): - sample = sample_of.get(k) - if sample is None: + if k not in cell_keys: + # In `identities` but absent from `cells`: read_states emitted a + # row for it, and it is dropped here rather than counted against + # a cell universe that never named it. continue - pair = (sample, ident) + pair = (k[0], ident) observed_count[pair] = observed_count.get(pair, 0) + 1 if k in inadmissible: observed_inadmissible_count[pair] = observed_inadmissible_count.get(pair, 0) + 1 rows = [] for sample, offered in sorted(offered_by_sample.items()): + # asked and total_inadmissible are hoisted out of the identity loop + # below because a sample offers the same identities to every one of + # its cells, so neither term depends on which identity is being + # tallied. That precondition is what makes the hoist valid, not an + # incidental property of this key: a regrouping that does not nest + # inside a sample — a group spanning two panels, say — breaks it, + # because then both terms DO depend on which identity was silent. + # Generalising the key means moving both terms back inside this loop, + # not renaming a column and leaving them where they are. + # + # `offered_by_sample` itself does not generalise with the key: what + # was offered is a property of the staining, which is done per + # sample, so it stays keyed by sample no matter what the output key + # becomes. Generalising the key is two inputs changing, not one + # renamed one. asked = asked_count.get(sample, 0) total_inadmissible = inadmissible_count.get(sample, 0) for identity in sorted(offered): @@ -500,6 +582,11 @@ def silent_tally( observed_inadmissible_n = observed_inadmissible_count.get(pair, 0) silent_unreliable = total_inadmissible - observed_inadmissible_n silent_not_bound = asked - observed_n - silent_unreliable + assert asked >= 0 and silent_unreliable >= 0 and silent_not_bound >= 0, ( + f"negative silent term for {sample!r}/{identity!r} " + f"(asked={asked}, silentUnreliable={silent_unreliable}, silentNotBound={silent_not_bound}): " + "cells or observed violated the uniqueness precondition documented above" + ) rows.append((sample, identity, asked, observed_n, silent_unreliable, silent_not_bound)) return pl.DataFrame( diff --git a/software/per-cell-metrics/test/test_verdict.py b/software/per-cell-metrics/test/test_verdict.py index 33f5fc2..26e3531 100644 --- a/software/per-cell-metrics/test/test_verdict.py +++ b/software/per-cell-metrics/test/test_verdict.py @@ -2,6 +2,7 @@ import random import polars as pl +import pytest from scipy.stats import beta from verdict import ( BOUND_CUTOFF, @@ -9,8 +10,10 @@ DEFAULT_HIGH_REFERENCE_OBSERVATION_LINE, DEFAULT_PANEL_MIN_MEMBERS, DEFAULT_REFERENCE_THIN_LINE, + Admissibility, ReferenceChoice, State, + UnreliableReason, apply_floor, combine_tags_to_identities, densify, @@ -175,18 +178,6 @@ def test_shipped_defaults_are_pinned(): assert DEFAULT_HIGH_REFERENCE_OBSERVATION_LINE == 100 -def test_gate_defaults_off_but_still_measures_exposure(): - ref = {("S1", "c1"): 5000, ("S1", "c2"): 1} - aside, high = gate_cells(ref, threshold=None, observation_line=100) - assert aside == set() and high == 1 - - -def test_declared_gate_sets_aside_and_counts(): - ref = {("S1", "c1"): 900, ("S1", "c2"): 2} - aside, high = gate_cells(ref, threshold=100, observation_line=100) - assert aside == {("S1", "c1")} and high == 1 - - def test_panel_source_serves_exactly_at_the_minimum(): # The minimum is a floor, not a gap: a panel of exactly min_members is # large enough. Nothing else in the suite distinguishes < from <=. @@ -375,28 +366,23 @@ def test_high_count_against_a_quiet_reference_is_bound(): # below the default thin line (a panel median that truncates to 1 already # reads unreliable at that default), so pinning the score computation in # isolation needs the thin-line rule turned off. - out = read_states(_ident([("S1", "c1", "A", 200)]), {("S1", "c1"): 0}, 75.0, 0, set()) + out = read_states(_ident([("S1", "c1", "A", 200)]), Admissibility({("S1", "c1"): 0}, 0, set()), 75.0) assert out["state"].to_list() == [State.BOUND.value] def test_zero_reads_not_bound_never_unreliable(): - out = read_states(_ident([("S1", "c1", "A", 0)]), {("S1", "c1"): 5}, 75.0, 2, set()) + out = read_states(_ident([("S1", "c1", "A", 0)]), Admissibility({("S1", "c1"): 5}, 2, set()), 75.0) assert out["state"].to_list() == [State.NOT_BOUND.value] -def test_no_reference_reading_is_unreliable(): - out = read_states(_ident([("S1", "c9", "A", 50)]), {}, 75.0, 2, set()) - assert out["state"].to_list() == [State.UNRELIABLE.value] - - def test_reference_below_the_thin_line_is_unreliable_not_scored(): # A comparison against almost nothing is not a comparison. - out = read_states(_ident([("S1", "c1", "A", 50)]), {("S1", "c1"): 1}, 75.0, thin_line=2, gated=set()) + out = read_states(_ident([("S1", "c1", "A", 50)]), Admissibility({("S1", "c1"): 1}, 2, set()), 75.0) assert out["state"].to_list() == [State.UNRELIABLE.value] def test_gated_cell_is_unreliable_and_stays_in_the_frame(): - out = read_states(_ident([("S1", "c1", "A", 500)]), {("S1", "c1"): 900}, 75.0, 2, gated={("S1", "c1")}) + out = read_states(_ident([("S1", "c1", "A", 500)]), Admissibility({("S1", "c1"): 900}, 2, {("S1", "c1")}), 75.0) assert out.height == 1 assert out["state"].to_list() == [State.UNRELIABLE.value] @@ -409,11 +395,10 @@ def test_a_gated_cell_reports_the_gate_even_when_its_reference_is_thin(): # step reads to tell a panel problem from a re-run problem, so which # condition wins is a fact someone acts on. The gate wins: a cell it set # aside was not measured at all, so its comparator's thickness is moot. - out = read_states(_ident([("S1", "c1", "A", 500)]), {("S1", "c1"): 1}, 75.0, thin_line=2, gated={("S1", "c1")}) + out = read_states(_ident([("S1", "c1", "A", 500)]), Admissibility({("S1", "c1"): 1}, 2, {("S1", "c1")}), 75.0) assert out["state"].to_list() == [State.UNRELIABLE.value] reason = out["unreliableReason"].to_list()[0] - assert "gate" in reason - assert "compare" not in reason # not the thin-comparator reason + assert reason == UnreliableReason.GATED # not the thin-comparator reason def test_densify_handles_a_sample_stained_with_nothing(): @@ -426,17 +411,17 @@ def test_densify_handles_a_sample_stained_with_nothing(): def test_never_asked_is_not_produced_here(): - out = read_states(_ident([("S1", "c1", "A", 0)]), {("S1", "c1"): 5}, 75.0, 2, set()) + out = read_states(_ident([("S1", "c1", "A", 0)]), Admissibility({("S1", "c1"): 5}, 2, set()), 75.0) assert State.NEVER_ASKED.value not in out["state"].to_list() def test_no_score_column_leaves_the_reading(): - out = read_states(_ident([("S1", "c1", "A", 50)]), {("S1", "c1"): 5}, 75.0, 2, set()) + out = read_states(_ident([("S1", "c1", "A", 50)]), Admissibility({("S1", "c1"): 5}, 2, set()), 75.0) assert "score" not in out.columns assert {"umiCount", "referenceCount"} <= set(out.columns) -def test_score_bounded(): +def test_specificity_score_stays_within_zero_and_hundred_at_sample_points(): for a, r in [(0, 0), (5, 5), (1000, 3)]: assert 0.0 <= specificity_score(a, r) <= 100.0 @@ -449,15 +434,15 @@ def test_a_score_exactly_at_the_cutoff_is_bound(): # The comparison then lands on the line with no floating-point drift, and # ">=" must call it bound. exact = specificity_score(10, 2) - out = read_states(_ident([("S1", "c1", "A", 10)]), {("S1", "c1"): 2}, cutoff=exact, thin_line=2, gated=set()) + out = read_states(_ident([("S1", "c1", "A", 10)]), Admissibility({("S1", "c1"): 2}, 2, set()), cutoff=exact) assert out["state"].to_list() == [State.BOUND.value] def test_a_reference_exactly_at_the_thin_line_is_scored_not_unreliable(): # Below the line the comparison does not exist; AT the line it does. This # is the pair that makes the thin line a floor rather than a gap. - at_line = read_states(_ident([("S1", "c1", "A", 0)]), {("S1", "c1"): 2}, 75.0, thin_line=2, gated=set()) - below_line = read_states(_ident([("S1", "c2", "A", 0)]), {("S1", "c2"): 1}, 75.0, thin_line=2, gated=set()) + at_line = read_states(_ident([("S1", "c1", "A", 0)]), Admissibility({("S1", "c1"): 2}, 2, set()), 75.0) + below_line = read_states(_ident([("S1", "c2", "A", 0)]), Admissibility({("S1", "c2"): 1}, 2, set()), 75.0) assert at_line["unreliableReason"].to_list() == [None] assert at_line["state"].to_list() == [State.NOT_BOUND.value] assert below_line["state"].to_list() == [State.UNRELIABLE.value] @@ -470,32 +455,63 @@ def test_no_comparator_is_unreliable_but_a_comparator_reading_zero_is_scored(): # scores normally -- a positive antigen count against a zero reference is # the strongest evidence there is. thin_line=0 isolates that from the # separate (and, at the default of 2, overlapping) thin-line rule. - no_comparator = read_states(_ident([("S1", "c1", "A", 200)]), {}, 75.0, 0, set()) - zero_comparator = read_states(_ident([("S1", "c1", "A", 200)]), {("S1", "c1"): 0}, 75.0, 0, set()) + # + # This also subsumes the plain no-comparator-is-unreliable check: nothing + # else in the suite needs a weaker, reason-blind version of this. + no_comparator = read_states(_ident([("S1", "c1", "A", 200)]), Admissibility({}, 0, set()), 75.0) + zero_comparator = read_states(_ident([("S1", "c1", "A", 200)]), Admissibility({("S1", "c1"): 0}, 0, set()), 75.0) assert no_comparator["state"].to_list() == [State.UNRELIABLE.value] - assert no_comparator["unreliableReason"].to_list() == ["no comparator for this cell"] + assert no_comparator["unreliableReason"].to_list() == [UnreliableReason.NO_COMPARATOR] assert zero_comparator["state"].to_list() == [State.BOUND.value] assert zero_comparator["unreliableReason"].to_list() == [None] def test_silent_admissible_cell_can_never_score_bound(): - # The fact the analytic path rests on: specificity_score(0, r) is 0.042 at - # r = 0 and smaller for every larger r, below every cutoff this block - # offers above that. A silent admissible cell is therefore always - # *not bound*, which is what lets silent_tally skip materializing its row. - assert math.isclose(specificity_score(0, 0), 0.042, abs_tol=5e-4) + # The fact the analytic path rests on: specificity_score(0, r) is ~0.0422 + # at r = 0 and smaller for every larger r. A silent admissible cell is + # therefore always *not bound* for any cutoff above that bound, which is + # what lets silent_tally skip materializing its row. + assert math.isclose(specificity_score(0, 0), 0.0422, abs_tol=5e-4) scores = [specificity_score(0, r) for r in range(0, 50)] assert scores == sorted(scores, reverse=True) assert all(s < 0.05 for s in scores) -def test_silent_tally_agrees_with_the_densify_oracle_on_small_random_inputs(): - # A property test: build a small, varied population by construction -- - # several samples, cells, identities, some cells gated, some below the - # thin line, some with a normal reference -- and check silent_tally's - # three cheap terms against the dense grid built by densify and read - # through read_states, which never skips a row. - rng = random.Random(20260817) +def test_duplicated_cells_rows_give_the_deduped_answer(): + # A row-count bug this project shipped once already: keys built from + # `cells` without dedup counted the duplicated c2 row as if it were a + # second cell. asked must count distinct cells (2), not rows (3), and + # silentNotBound must follow from the deduped count. + cells = _cells([("S1", "c1"), ("S1", "c2"), ("S1", "c2")]) + admissibility = Admissibility({("S1", "c1"): 5, ("S1", "c2"): 5}, 2, set()) + observed = read_states(_ident([("S1", "c1", "A", 50)]), admissibility, 75.0) + tally = silent_tally(observed, cells, {"S1": {"A"}}, admissibility) + row = tally.row(0, named=True) + assert row["asked"] == 2 # not 3: the duplicated c2 row counts once + assert row["silentNotBound"] == 1 + + +def test_duplicated_observed_rows_are_rejected_not_silently_wrong(): + # Recorded rather than latent: without the assertion in silent_tally, this + # combination silently returned silentUnreliable == -1 (a duplicated + # observed row for an inadmissible cell is counted twice against a total + # that counts the cell once). `observed` must be unique on + # (cell, identity); this input violates that, so the function must now + # refuse it loudly instead of emitting a negative count. + cells = _cells([("S1", "c1")]) + admissibility = Admissibility({}, 2, set()) # no comparator for c1: inadmissible + observed = read_states(_ident([("S1", "c1", "A", 50), ("S1", "c1", "A", 50)]), admissibility, 75.0) + with pytest.raises(AssertionError): + silent_tally(observed, cells, {"S1": {"A"}}, admissibility) + + +def _check_silent_tally_matches_oracle(seed, cutoff=BOUND_CUTOFF, force_empty_sample=None): + # Shared by every arm below: build a small, varied population by + # construction -- several samples, cells, identities, some cells gated, + # some below the thin line, some with a normal reference -- and check + # silent_tally's three cheap terms against the dense grid built by + # densify and read through read_states, which never skips a row. + rng = random.Random(seed) samples = ["S1", "S2", "S3"] identities = ["A", "B", "C"] thin_line = 2 @@ -506,7 +522,10 @@ def test_silent_tally_agrees_with_the_densify_oracle_on_small_random_inputs(): offered_by_sample: dict[str, set[str]] = {} for sample in samples: - offered_by_sample[sample] = set(rng.sample(identities, k=rng.randint(1, len(identities)))) + if sample == force_empty_sample: + offered_by_sample[sample] = set() + else: + offered_by_sample[sample] = set(rng.sample(identities, k=rng.randint(1, len(identities)))) for i in range(6): cell = f"c{i}" cell_rows.append((sample, cell)) @@ -530,20 +549,31 @@ def test_silent_tally_agrees_with_the_densify_oracle_on_small_random_inputs(): cells = _cells(cell_rows) sparse_identities = _ident(tag_rows) - observed = read_states(sparse_identities, reference, BOUND_CUTOFF, thin_line, gated) + admissibility = Admissibility(reference, thin_line, gated) + observed = read_states(sparse_identities, admissibility, cutoff) dense = densify(sparse_identities, cells, offered_by_sample) - oracle = read_states(dense, reference, BOUND_CUTOFF, thin_line, gated) + oracle = read_states(dense, admissibility, cutoff) - tally = silent_tally(observed, cells, offered_by_sample, reference, thin_line, gated) + tally = silent_tally(observed, cells, offered_by_sample, admissibility) + + # A tally that emits extra rows -- one for an identity a sample never + # offered -- must fail here, not just disagree on counts. + expected_row_count = sum(len(offered) for offered in offered_by_sample.values()) + assert tally.height == expected_row_count for sample in samples: - for identity in offered_by_sample[sample]: - oracle_group = oracle.filter((pl.col("sampleId") == sample) & (pl.col("identity") == identity)) - observed_group = observed.filter((pl.col("sampleId") == sample) & (pl.col("identity") == identity)) - tally_row = tally.filter((pl.col("sampleId") == sample) & (pl.col("identity") == identity)).row( - 0, named=True - ) + offered = offered_by_sample[sample] + for identity in identities: + group_filter = (pl.col("sampleId") == sample) & (pl.col("identity") == identity) + if identity not in offered: + # Never asked of this sample: no row at all, not a zero row. + assert tally.filter(group_filter).height == 0 + continue + + oracle_group = oracle.filter(group_filter) + observed_group = observed.filter(group_filter) + tally_row = tally.filter(group_filter).row(0, named=True) oracle_states = oracle_group["state"].to_list() observed_states = observed_group["state"].to_list() @@ -563,3 +593,32 @@ def test_silent_tally_agrees_with_the_densify_oracle_on_small_random_inputs(): assert tally_row["observed"] == len(observed_states) assert tally_row["silentUnreliable"] == expected_silent_unreliable assert tally_row["silentNotBound"] == expected_silent_not_bound + + +@pytest.mark.parametrize( + "seed, force_empty_sample", + [ + (20260817, None), + (1, None), + (2, None), + (3, None), + (4, None), + (5, None), + (6, None), + # The generator above never draws an empty offered set on its own; + # force one so the empty-block path in densify and the zero-row case + # in silent_tally are both exercised against the oracle, not just + # against each other. + (7, "S2"), + ], +) +def test_silent_tally_agrees_with_the_densify_oracle_on_small_random_inputs(seed, force_empty_sample): + _check_silent_tally_matches_oracle(seed, force_empty_sample=force_empty_sample) + + +def test_silent_tally_agrees_with_the_oracle_at_a_low_valid_cutoff(): + # 0.5 is comfortably above specificity_score(0, 0) ~= 0.0422, the bound + # named in specificity_score's and silent_tally's docstrings. This guards + # the boundary itself rather than assuming BOUND_CUTOFF=75.0 is + # representative of every cutoff the equivalence must hold for. + _check_silent_tally_matches_oracle(seed=20260817, cutoff=0.5) From d7a1614a10568bf28b9d594e4f7baa3f982909d3 Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 15:30:51 +0200 Subject: [PATCH 034/282] MILAB-6496: reduce a set's cells to its verdict, counting the silent ones --- software/per-cell-metrics/src/combine.py | 253 ++++++++++++++++++ software/per-cell-metrics/src/verdict.py | 195 +++++++++----- .../per-cell-metrics/test/test_combine.py | 240 +++++++++++++++++ .../per-cell-metrics/test/test_verdict.py | 144 +++++++++- 4 files changed, 764 insertions(+), 68 deletions(-) create mode 100644 software/per-cell-metrics/src/combine.py create mode 100644 software/per-cell-metrics/test/test_combine.py diff --git a/software/per-cell-metrics/src/combine.py b/software/per-cell-metrics/src/combine.py new file mode 100644 index 0000000..b53acf9 --- /dev/null +++ b/software/per-cell-metrics/src/combine.py @@ -0,0 +1,253 @@ +"""Reducing a set's cells to the set's verdict, identity by identity. + +Cells of one set are replicates of one measurement, so where they differ at +an identity the difference is error and the modal answer is the best +available reading of what the receptor did. The vote is per identity: a +single winning antigen collapses a set that bound several, which the model +this reduces from -- four states per (cell, identity) -- exists to keep +distinct. + +The row set is the identity universe, never the offered subset: a set's +verdict at an identity the panel never offered is NEVER_ASKED, and that +comes only from the offered map, never from a row's absence. `offered` is +keyed by sample, matching `silent_tally`'s `offered_by_sample` -- what a +panel offered is a property of the staining, not of the clonotype grouping +built on top of it -- so a set's own offered set is the union, over its +member samples, of what each sample's panel offered. + +A cell asked about an identity and showing no reading in `states` is silent, +not absent from the vote: `silent_tally`, generalised to key its tally by an +arbitrary per-cell group rather than only by sample, supplies the silent +contribution here. A silent admissible cell always resolves NOT_BOUND (see +`specificity_score` in verdict.py), so silent cells vote not bound; silent +inadmissible cells vote nowhere, which is exactly what keeps a set every one +of whose cells failed to bind reading NOT_BOUND rather than UNRELIABLE or +NEVER_ASKED. +""" + +from __future__ import annotations + +from enum import Enum + +import polars as pl +from verdict import Admissibility, State, UnreliableReason, _cell_admissibility_reason, silent_tally + +# Both limits default permissively because the failure they would prevent is +# visible and the failure they would cause is not. Requiring two voting cells +# would silently discard every singleton, which many clonotypes in a run are. +DEFAULT_MIN_VOTERS = 1 +DEFAULT_MIN_AGREEMENT = None + +SETTLED = (State.BOUND.value, State.NOT_BOUND.value) + + +class SetUnreliableReason(str, Enum): + """Why a set's verdict at one identity could not be settled, or why it + was never asked. Cell-level admissibility (`verdict.UnreliableReason`) + answers "why can't this cell be compared"; this answers "why can't this + set's cells, taken together, produce a verdict" -- a different grain, + kept in its own enum rather than folded into the cell-level one, whose + own docstring scopes it to a single cell's comparison. + + NEVER_OFFERED is the reason recorded on a NEVER_ASKED row: not itself a + reliability problem, but the same column carries it, so a reader always + finds a reason there whenever the state is not BOUND or NOT_BOUND. + + NO_COMPARATOR and THIN_COMPARATOR reuse the cell-level vocabulary's + concepts because they describe the same underlying fact, just observed + for a whole set rather than one cell: no settled vote exists because + every one of the set's asked cells individually failed the same + comparator check. ALL_CELLS_GATED is reported only when every asked cell + was gated with no other reason mixed in; a set with a mix of gated and + comparator-failed cells is reported by whichever comparator failure is + present, since an admissibility gate excluding only part of a set is not + by itself why the rest failed to settle. + """ + + NEVER_OFFERED = "never-offered" + NO_COMPARATOR = "no-comparator" + THIN_COMPARATOR = "thin-comparator" + ALL_CELLS_GATED = "all-cells-gated" + TIE = "tie" + TOO_FEW_VOTERS = "too-few-voters" + + +def _dominant_reason(asked_keys: list[tuple[str, str]], admissibility: Admissibility) -> SetUnreliableReason: + """The one reason that explains why none of `asked_keys` settled. + + Called only when the set's tally has zero settled votes for the + identity, which happens only when every asked cell is individually + inadmissible: an admissible cell always settles, either directly (a + BOUND or NOT_BOUND row) or, if silent, through `silent_tally`'s proof + that a silent admissible cell always resolves NOT_BOUND. So every key + here has a real, non-None cell-level reason, and this only has to pick + among the three. + """ + reasons = {_cell_admissibility_reason(k, admissibility) for k in asked_keys} + if reasons == {UnreliableReason.GATED}: + return SetUnreliableReason.ALL_CELLS_GATED + if UnreliableReason.NO_COMPARATOR in reasons: + return SetUnreliableReason.NO_COMPARATOR + return SetUnreliableReason.THIN_COMPARATOR + + +def _majority(counts: dict[str, int]) -> tuple[str, int, bool]: + """The leading state, its count, and whether more than one state is tied for it.""" + top = max(counts.values()) + leaders = sorted(state for state, n in counts.items() if n == top) + return leaders[0], top, len(leaders) > 1 + + +def combine_cells( + states: pl.DataFrame, + universe: set[str], + offered: dict[str, set[str]], + cells_by_set: dict[str, list[tuple[str, str]]], + admissibility: Admissibility, + min_voters: int = DEFAULT_MIN_VOTERS, + min_agreement: float | None = DEFAULT_MIN_AGREEMENT, +) -> pl.DataFrame: + """One row per (set, identity) over the whole universe. + + `states` is per-cell output shaped like `read_states`' -- columns + setId, sampleId, cellId, identity, state -- one row per (cell, identity) + that got an explicit reading; a cell asked about an identity and absent + here is silent for it, not unasked. + + `offered` is keyed by sample: for a given set, the identities it was + offered are the union, over its member samples, of what each sample's + panel offered -- so a set spanning two samples with different panels + reads as offered whatever either panel offered, while `cellsCouldAnswer` + below still counts only the members whose OWN sample offered that + specific identity. + + `cells_by_set` gives each set's full cell membership, including cells + with no row in `states` at all -- the set's silent cells, which vote + through `silent_tally` rather than through a row that was never written. + """ + group_by_cell: dict[tuple[str, str], str] = {} + for set_id, members in cells_by_set.items(): + for key in members: + group_by_cell[key] = set_id + + cells_frame = pl.DataFrame(list(group_by_cell), orient="row", schema={"sampleId": pl.String, "cellId": pl.String}) + tally = silent_tally(states, cells_frame, offered, admissibility, group_by_cell=group_by_cell, group_column="setId") + silent_by_pair = {(row["setId"], row["identity"]): row for row in tally.iter_rows(named=True)} + + settled = states.filter(pl.col("state").is_in(SETTLED)) + explicit_counts: dict[tuple[str, str], dict[str, int]] = {} + for set_id, identity, state in zip( + settled["setId"].to_list(), settled["identity"].to_list(), settled["state"].to_list(), strict=True + ): + bucket = explicit_counts.setdefault((set_id, identity), {}) + bucket[state] = bucket.get(state, 0) + 1 + + rows = [] + for set_id in sorted(cells_by_set): + members = cells_by_set[set_id] + offered_for_set = set().union(*(offered.get(sample, set()) for sample, _ in members)) if members else set() + + for identity in sorted(universe): + if identity not in offered_for_set: + rows.append( + { + "setId": set_id, + "identity": identity, + "state": State.NEVER_ASKED.value, + "cellsCouldAnswer": 0, + "cellsAnswered": 0, + "agreement": None, + "unreliableReason": SetUnreliableReason.NEVER_OFFERED.value, + } + ) + continue + + # Guaranteed present: `identity` is in `offered_for_set` only + # because at least one member's own sample offers it, which is + # exactly the condition under which silent_tally emits a row for + # (set_id, identity). + silent_row = silent_by_pair[(set_id, identity)] + could = silent_row["asked"] + + counts = dict(explicit_counts.get((set_id, identity), {})) + counts[State.NOT_BOUND.value] = counts.get(State.NOT_BOUND.value, 0) + silent_row["silentNotBound"] + answered = sum(counts.values()) + + if answered == 0: + asked_keys = [key for key in members if identity in offered.get(key[0], set())] + reason = _dominant_reason(asked_keys, admissibility) + rows.append( + { + "setId": set_id, + "identity": identity, + "state": State.UNRELIABLE.value, + "cellsCouldAnswer": could, + "cellsAnswered": 0, + "agreement": None, + "unreliableReason": reason.value, + } + ) + continue + + if answered < min_voters: + rows.append( + { + "setId": set_id, + "identity": identity, + "state": State.UNRELIABLE.value, + "cellsCouldAnswer": could, + "cellsAnswered": answered, + "agreement": None, + "unreliableReason": SetUnreliableReason.TOO_FEW_VOTERS.value, + } + ) + continue + + top_state, top_count, tied = _majority(counts) + agreement = top_count / answered + + # A tie is not a thin majority but the absence of one: half (or a + # third, or a quarter) of the settled votes contradict the rest, + # and nothing in the reading says which side to believe. The + # reason vocabulary has no separate label for "settled, but below + # the agreement floor" -- that case is reported as TIE too, since + # both mean the same thing to a reader: the majority that formed + # was not decisive enough to stand. + if tied or (min_agreement is not None and agreement < min_agreement): + rows.append( + { + "setId": set_id, + "identity": identity, + "state": State.UNRELIABLE.value, + "cellsCouldAnswer": could, + "cellsAnswered": answered, + "agreement": agreement, + "unreliableReason": SetUnreliableReason.TIE.value, + } + ) + continue + + rows.append( + { + "setId": set_id, + "identity": identity, + "state": top_state, + "cellsCouldAnswer": could, + "cellsAnswered": answered, + "agreement": agreement, + "unreliableReason": None, + } + ) + + return pl.DataFrame( + rows, + schema={ + "setId": pl.String, + "identity": pl.String, + "state": pl.String, + "cellsCouldAnswer": pl.Int64, + "cellsAnswered": pl.Int64, + "agreement": pl.Float64, + "unreliableReason": pl.String, + }, + ).sort(["setId", "identity"]) diff --git a/software/per-cell-metrics/src/verdict.py b/software/per-cell-metrics/src/verdict.py index 70152e2..44b4372 100644 --- a/software/per-cell-metrics/src/verdict.py +++ b/software/per-cell-metrics/src/verdict.py @@ -30,8 +30,11 @@ After step 2 this module holds three frame shapes: the sparse per-tag frame the floor works on and the per-identity frame combining produces from it, both keyed by CELL_KEY, which is the column vocabulary spanning both; and the -per-(sampleId, identity) frame `silent_tally` returns, keyed one level -coarser than CELL_KEY, not by it. +per-(group, identity) frame `silent_tally` returns, keyed one level coarser +than CELL_KEY, not by it. The group defaults to sampleId -- a sample offers +the same identities to every one of its cells -- but callers reducing cells +to a coarser unit than a sample (a clonotype set spanning several samples, +say) pass their own per-cell grouping instead. """ from __future__ import annotations @@ -481,8 +484,10 @@ def silent_tally( cells: pl.DataFrame, offered_by_sample: dict[str, set[str]], admissibility: Admissibility, + group_by_cell: dict[tuple[str, str], str] | None = None, + group_column: str = "sampleId", ) -> pl.DataFrame: - """Per (sample, identity): how many asked cells were never observed, and how they resolve. + """Per (group, identity): how many asked cells were never observed, and how they resolve. The sparse path: silent positions are counted, never materialized. `densify` followed by `read_states` is the reference this must agree @@ -500,9 +505,10 @@ def silent_tally( cutoff is the CLI's job, not this function's. Above the bound, three cheap terms replace a materialized row per silent cell: - asked = cells of the sample, for every identity it offered + asked = cells of the group, for every identity offered to one of its members observed = the (cell, identity) rows read_states already produced - silentUnreliable = inadmissible cells of the sample − inadmissible cells among the observed + silentUnreliable = inadmissible cells the group counts toward that identity − + inadmissible cells among the observed silentNotBound = asked − observed − silentUnreliable `observed` is `read_states`' output on the sparse frame — one row per @@ -510,6 +516,28 @@ def silent_tally( already resolved. Nothing here inspects an identity's silent reading; the admissibility check that decides silentUnreliable never takes an identity. + `group_by_cell` maps a cell key to the unit the tally reports per. It + defaults to None, which groups by the cell's own sampleId — the original + behaviour, and the only grouping under which every member of a group is + guaranteed to share one offered set. That guarantee is what lets `asked` + and `total_inadmissible` be computed once per group below rather than + once per (group, identity): a sample offers the same identities to every + one of its cells. A group that can span samples with different offered + sets — a clonotype set spanning several samples, say — does not have + that guarantee: whether a member counts toward `identity` depends on + whether THAT MEMBER'S OWN SAMPLE offered `identity`, which can differ + member to member, so this function computes both terms inside the + identity loop for that case instead of hoisting them above it. + `offered_by_sample` itself is never regrouped: what a panel offered is a + property of the staining, done per sample, so it stays keyed by sample + regardless of what `group_by_cell` reports. Every cell key present in + `cells` must have an entry in `group_by_cell` when one is given. + + `group_column` names the key column in the returned frame — "sampleId" + by default, matching the default grouping; a caller passing a custom + `group_by_cell` should also pass a `group_column` that names what the + values in it actually are. + Precondition, unchecked by types: `cells` must be unique on the cell key, and `observed` unique on (cell, identity). A duplicated `cells` row is harmless — it is deduplicated below — but a duplicated `observed` row is @@ -529,71 +557,112 @@ def silent_tally( inadmissible = {k for k in keys if _cell_admissibility_reason(k, admissibility) is not None} cell_keys = set(keys) - # Single accumulating pass, not one loop per sample: the previous version - # scanned all of `keys` once per sample in offered_by_sample, which is - # O(groups x cells) and harmless at 24 samples but quadratic once a wider - # key groups thousands of sets. `k[0]` is the sample directly — no need - # for a cell->sample dict when every key already carries it. - asked_count: dict[str, int] = {} - inadmissible_count: dict[str, int] = {} - for k in keys: - sample = k[0] - asked_count[sample] = asked_count.get(sample, 0) + 1 - if k in inadmissible: - inadmissible_count[sample] = inadmissible_count.get(sample, 0) + 1 - obs_keys = list(zip(observed["sampleId"].to_list(), observed["cellId"].to_list(), strict=True)) obs_identity = observed["identity"].to_list() - observed_count: dict[tuple[str, str], int] = {} - observed_inadmissible_count: dict[tuple[str, str], int] = {} - for k, ident in zip(obs_keys, obs_identity, strict=True): - if k not in cell_keys: - # In `identities` but absent from `cells`: read_states emitted a - # row for it, and it is dropped here rather than counted against - # a cell universe that never named it. - continue - pair = (k[0], ident) - observed_count[pair] = observed_count.get(pair, 0) + 1 - if k in inadmissible: - observed_inadmissible_count[pair] = observed_inadmissible_count.get(pair, 0) + 1 - - rows = [] - for sample, offered in sorted(offered_by_sample.items()): - # asked and total_inadmissible are hoisted out of the identity loop - # below because a sample offers the same identities to every one of - # its cells, so neither term depends on which identity is being - # tallied. That precondition is what makes the hoist valid, not an - # incidental property of this key: a regrouping that does not nest - # inside a sample — a group spanning two panels, say — breaks it, - # because then both terms DO depend on which identity was silent. - # Generalising the key means moving both terms back inside this loop, - # not renaming a column and leaving them where they are. - # - # `offered_by_sample` itself does not generalise with the key: what - # was offered is a property of the staining, which is done per - # sample, so it stays keyed by sample no matter what the output key - # becomes. Generalising the key is two inputs changing, not one - # renamed one. - asked = asked_count.get(sample, 0) - total_inadmissible = inadmissible_count.get(sample, 0) - for identity in sorted(offered): - pair = (sample, identity) - observed_n = observed_count.get(pair, 0) - observed_inadmissible_n = observed_inadmissible_count.get(pair, 0) - silent_unreliable = total_inadmissible - observed_inadmissible_n - silent_not_bound = asked - observed_n - silent_unreliable - assert asked >= 0 and silent_unreliable >= 0 and silent_not_bound >= 0, ( - f"negative silent term for {sample!r}/{identity!r} " - f"(asked={asked}, silentUnreliable={silent_unreliable}, silentNotBound={silent_not_bound}): " - "cells or observed violated the uniqueness precondition documented above" - ) - rows.append((sample, identity, asked, observed_n, silent_unreliable, silent_not_bound)) + + rows: list[tuple[str, str, int, int, int, int]] = [] + + if group_by_cell is None: + # Sample-keyed path: unchanged from before generalisation. Single + # accumulating pass, not one loop per sample: scanning all of `keys` + # once per sample in offered_by_sample is O(groups x cells) and + # harmless at 24 samples but quadratic once a wider key groups + # thousands of sets. `k[0]` is the sample directly — no need for a + # cell->sample dict when every key already carries it. + asked_count: dict[str, int] = {} + inadmissible_count: dict[str, int] = {} + for k in keys: + sample = k[0] + asked_count[sample] = asked_count.get(sample, 0) + 1 + if k in inadmissible: + inadmissible_count[sample] = inadmissible_count.get(sample, 0) + 1 + + observed_count: dict[tuple[str, str], int] = {} + observed_inadmissible_count: dict[tuple[str, str], int] = {} + for k, ident in zip(obs_keys, obs_identity, strict=True): + if k not in cell_keys: + # In `identities` but absent from `cells`: read_states emitted + # a row for it, and it is dropped here rather than counted + # against a cell universe that never named it. + continue + pair = (k[0], ident) + observed_count[pair] = observed_count.get(pair, 0) + 1 + if k in inadmissible: + observed_inadmissible_count[pair] = observed_inadmissible_count.get(pair, 0) + 1 + + for sample, offered in sorted(offered_by_sample.items()): + # asked and total_inadmissible are hoisted out of the identity + # loop below because a sample offers the same identities to every + # one of its cells, so neither term depends on which identity is + # being tallied. That precondition is what makes the hoist valid. + asked = asked_count.get(sample, 0) + total_inadmissible = inadmissible_count.get(sample, 0) + for identity in sorted(offered): + pair = (sample, identity) + observed_n = observed_count.get(pair, 0) + observed_inadmissible_n = observed_inadmissible_count.get(pair, 0) + silent_unreliable = total_inadmissible - observed_inadmissible_n + silent_not_bound = asked - observed_n - silent_unreliable + assert asked >= 0 and silent_unreliable >= 0 and silent_not_bound >= 0, ( + f"negative silent term for {sample!r}/{identity!r} " + f"(asked={asked}, silentUnreliable={silent_unreliable}, silentNotBound={silent_not_bound}): " + "cells or observed violated the uniqueness precondition documented above" + ) + rows.append((sample, identity, asked, observed_n, silent_unreliable, silent_not_bound)) + else: + # Group-keyed path: a group can mix members from samples with + # different offered sets, so neither term can be hoisted above the + # identity loop the way the sample-keyed path hoists them. Instead, + # each group is walked once, member by member, checking that + # member's OWN sample's offered set and accumulating a per-identity + # count as it goes — one pass per group, producing every identity's + # `asked`/`total_inadmissible` together, rather than one count + # computed before the identity loop that would silently apply to an + # identity some members' samples never offered. + keys_by_group: dict[str, list[tuple[str, str]]] = {} + for k in keys: + keys_by_group.setdefault(group_by_cell[k], []).append(k) + + observed_count = {} + observed_inadmissible_count = {} + for k, ident in zip(obs_keys, obs_identity, strict=True): + if k not in cell_keys: + continue + pair = (group_by_cell[k], ident) + observed_count[pair] = observed_count.get(pair, 0) + 1 + if k in inadmissible: + observed_inadmissible_count[pair] = observed_inadmissible_count.get(pair, 0) + 1 + + for group in sorted(keys_by_group): + asked_by_identity: dict[str, int] = {} + inadmissible_by_identity: dict[str, int] = {} + for k in keys_by_group[group]: + member_is_inadmissible = k in inadmissible + for identity in offered_by_sample.get(k[0], set()): + asked_by_identity[identity] = asked_by_identity.get(identity, 0) + 1 + if member_is_inadmissible: + inadmissible_by_identity[identity] = inadmissible_by_identity.get(identity, 0) + 1 + + for identity in sorted(asked_by_identity): + asked = asked_by_identity[identity] + total_inadmissible = inadmissible_by_identity.get(identity, 0) + pair = (group, identity) + observed_n = observed_count.get(pair, 0) + observed_inadmissible_n = observed_inadmissible_count.get(pair, 0) + silent_unreliable = total_inadmissible - observed_inadmissible_n + silent_not_bound = asked - observed_n - silent_unreliable + assert asked >= 0 and silent_unreliable >= 0 and silent_not_bound >= 0, ( + f"negative silent term for {group!r}/{identity!r} " + f"(asked={asked}, silentUnreliable={silent_unreliable}, silentNotBound={silent_not_bound}): " + "cells or observed violated the uniqueness precondition documented above" + ) + rows.append((group, identity, asked, observed_n, silent_unreliable, silent_not_bound)) return pl.DataFrame( rows, orient="row", schema={ - "sampleId": pl.String, + group_column: pl.String, "identity": pl.String, "asked": pl.Int64, "observed": pl.Int64, diff --git a/software/per-cell-metrics/test/test_combine.py b/software/per-cell-metrics/test/test_combine.py new file mode 100644 index 0000000..7b02564 --- /dev/null +++ b/software/per-cell-metrics/test/test_combine.py @@ -0,0 +1,240 @@ +import polars as pl +from combine import DEFAULT_MIN_VOTERS, SetUnreliableReason, combine_cells +from verdict import Admissibility, State, combine_tags_to_identities, gate_cells, read_states + +B, N, U, NA = (State.BOUND.value, State.NOT_BOUND.value, State.UNRELIABLE.value, State.NEVER_ASKED.value) + + +_STATES_SCHEMA = { + "setId": pl.String, + "sampleId": pl.String, + "cellId": pl.String, + "identity": pl.String, + "state": pl.String, +} + + +def _states(rows): + return pl.DataFrame(rows, orient="row", schema=_STATES_SCHEMA) + + +def _row(out, identity): + return out.filter(pl.col("identity") == identity).row(0, named=True) + + +# A permissive admissibility used by every test whose cells all have an +# explicit row in `states` -- no cell is silent, so asked == observed for +# every identity and the silent terms are 0 regardless of what this holds. +_NEUTRAL = Admissibility({}, 0, set()) + + +def test_majority_wins(): + df = _states([("s1", "S1", "c1", "A", B), ("s1", "S1", "c2", "A", B), ("s1", "S1", "c3", "A", N)]) + cells_by_set = {"s1": [("S1", "c1"), ("S1", "c2"), ("S1", "c3")]} + out = combine_cells(df, universe={"A"}, offered={"S1": {"A"}}, cells_by_set=cells_by_set, admissibility=_NEUTRAL) + r = _row(out, "A") + assert r["state"] == B and r["cellsAnswered"] == 3 and r["agreement"] == 2 / 3 + + +def test_vote_is_per_identity_so_a_set_can_bind_several(): + df = _states([("s1", "S1", "c1", i, B) for i in ("A", "C")]) + cells_by_set = {"s1": [("S1", "c1")]} + out = combine_cells( + df, universe={"A", "C"}, offered={"S1": {"A", "C"}}, cells_by_set=cells_by_set, admissibility=_NEUTRAL + ).sort("identity") + assert out["state"].to_list() == [B, B] + + +def test_a_tie_cannot_be_settled(): + df = _states([("s1", "S1", "c1", "A", B), ("s1", "S1", "c2", "A", N)]) + cells_by_set = {"s1": [("S1", "c1"), ("S1", "c2")]} + out = combine_cells(df, {"A"}, {"S1": {"A"}}, cells_by_set, _NEUTRAL) + r = _row(out, "A") + assert r["state"] == U + assert r["unreliableReason"] == SetUnreliableReason.TIE.value + + +def test_a_three_way_split_that_ties_at_the_top_is_also_unreliable(): + # Not just the minimal 1-vs-1 tie: three cells settle bound, three settle + # not bound. The tie check must compare the leading counts, not special- + # case a count of one. + df = _states([("s1", "S1", f"b{i}", "A", B) for i in range(3)] + [("s1", "S1", f"n{i}", "A", N) for i in range(3)]) + cells_by_set = {"s1": [("S1", f"b{i}") for i in range(3)] + [("S1", f"n{i}") for i in range(3)]} + r = _row(combine_cells(df, {"A"}, {"S1": {"A"}}, cells_by_set, _NEUTRAL), "A") + assert r["state"] == U and r["cellsAnswered"] == 6 + assert r["unreliableReason"] == SetUnreliableReason.TIE.value + + +def test_never_asked_comes_only_from_not_being_offered(): + # Z is in the universe and NOT offered -> never asked. + df = _states([("s1", "S1", "c1", "A", B)]) + cells_by_set = {"s1": [("S1", "c1")]} + out = combine_cells( + df, universe={"A", "Z"}, offered={"S1": {"A"}}, cells_by_set=cells_by_set, admissibility=_NEUTRAL + ) + r = _row(out, "Z") + assert r["state"] == NA + assert r["cellsCouldAnswer"] == 0 + assert r["unreliableReason"] == SetUnreliableReason.NEVER_OFFERED.value + + +def test_an_offered_identity_nobody_bound_is_not_bound_not_never_asked(): + # Explicit rows, every one not-bound: offered, everybody read zero, so + # the verdict is not bound, never never-asked. + df = _states([("s1", "S1", "c1", "A", N), ("s1", "S1", "c2", "A", N)]) + cells_by_set = {"s1": [("S1", "c1"), ("S1", "c2")]} + r = _row(combine_cells(df, {"A"}, {"S1": {"A"}}, cells_by_set, _NEUTRAL), "A") + assert r["state"] == N and r["state"] != NA + + +def test_silent_cells_vote_an_antigen_every_cell_failed_still_reads_not_bound(): + # The defect this reduction exists to avoid: five cells asked about A, + # none has a row in `states` at all (tag-stat never observed a reading + # for any of them), and all five are admissible. Silent admissible cells + # resolve not bound, so the set must read not bound with all five voting + # -- never unreliable (which is what happens if silent cells are simply + # excluded from the tally) and never never-asked. + df = _states([]) + members = [("S1", f"c{i}") for i in range(5)] + cells_by_set = {"s1": members} + admissibility = Admissibility({k: 5 for k in members}, 2, set()) + r = _row(combine_cells(df, {"A"}, {"S1": {"A"}}, cells_by_set, admissibility), "A") + assert r["state"] == N + assert r["cellsAnswered"] == 5 + assert r["cellsCouldAnswer"] == 5 + assert r["agreement"] == 1.0 + + +def test_unsettled_cells_do_not_vote_but_do_count_as_could_answer(): + df = _states([("s1", "S1", "c1", "A", B), ("s1", "S1", "c2", "A", U), ("s1", "S1", "c3", "A", U)]) + cells_by_set = {"s1": [("S1", "c1"), ("S1", "c2"), ("S1", "c3")]} + r = _row(combine_cells(df, {"A"}, {"S1": {"A"}}, cells_by_set, _NEUTRAL), "A") + assert r["state"] == B and r["cellsAnswered"] == 1 and r["cellsCouldAnswer"] == 3 + + +def test_a_verdict_may_rest_on_one_cell_and_says_so(): + assert DEFAULT_MIN_VOTERS == 1 + df = _states([("s1", "S1", "c1", "A", B)]) + cells_by_set = {"s1": [("S1", "c1")]} + r = _row(combine_cells(df, {"A"}, {"S1": {"A"}}, cells_by_set, _NEUTRAL), "A") + assert r["state"] == B and r["cellsAnswered"] == 1 + + +def test_below_min_voters_is_unreliable_when_raised(): + df = _states([("s1", "S1", "c1", "A", B)]) + cells_by_set = {"s1": [("S1", "c1")]} + r = _row(combine_cells(df, {"A"}, {"S1": {"A"}}, cells_by_set, _NEUTRAL, min_voters=2), "A") + assert r["state"] == U + assert r["unreliableReason"] == SetUnreliableReason.TOO_FEW_VOTERS.value + + +def test_exactly_min_voters_settles(): + # The named value satisfies the condition it names, as elsewhere in this + # project: two settled votes with min_voters=2 must settle, not fail. + df = _states([("s1", "S1", "c1", "A", B), ("s1", "S1", "c2", "A", B)]) + cells_by_set = {"s1": [("S1", "c1"), ("S1", "c2")]} + r = _row(combine_cells(df, {"A"}, {"S1": {"A"}}, cells_by_set, _NEUTRAL, min_voters=2), "A") + assert r["state"] == B and r["cellsAnswered"] == 2 + + +def test_narrow_majority_stands_and_reports_how_narrow(): + df = _states([("s1", "S1", f"c{i}", "A", B) for i in range(6)] + [("s1", "S1", f"d{i}", "A", N) for i in range(5)]) + cells_by_set = {"s1": [("S1", f"c{i}") for i in range(6)] + [("S1", f"d{i}") for i in range(5)]} + r = _row(combine_cells(df, {"A"}, {"S1": {"A"}}, cells_by_set, _NEUTRAL), "A") + assert r["state"] == B and r["agreement"] == 6 / 11 + + +def test_exactly_min_agreement_settles_when_raised(): + # 3 bound, 1 not bound -> agreement 0.75. Raising min_agreement to + # exactly 0.75 must still settle: the boundary belongs to the pass side. + df = _states([("s1", "S1", f"b{i}", "A", B) for i in range(3)] + [("s1", "S1", "n0", "A", N)]) + cells_by_set = {"s1": [("S1", f"b{i}") for i in range(3)] + [("S1", "n0")]} + r = _row(combine_cells(df, {"A"}, {"S1": {"A"}}, cells_by_set, _NEUTRAL, min_agreement=0.75), "A") + assert r["state"] == B and r["agreement"] == 0.75 + + +def test_just_below_min_agreement_is_unreliable(): + df = _states([("s1", "S1", f"b{i}", "A", B) for i in range(3)] + [("s1", "S1", "n0", "A", N)]) + cells_by_set = {"s1": [("S1", f"b{i}") for i in range(3)] + [("S1", "n0")]} + r = _row(combine_cells(df, {"A"}, {"S1": {"A"}}, cells_by_set, _NEUTRAL, min_agreement=0.76), "A") + assert r["state"] == U + assert r["unreliableReason"] == SetUnreliableReason.TIE.value + + +def test_set_with_every_cell_set_aside_is_unreliable_through_the_real_pipeline(): + # Driven through read_states, not fed a synthetic UNRELIABLE row: a gate + # excludes both of this set's cells, read_states produces the real + # UNRELIABLE rows from that, and combine_cells must still resolve the + # set to unreliable, reason all-cells-gated -- derived from the cells' + # own UnreliableReason.GATED, not hard-coded. + counts = pl.DataFrame( + [("S1", "c1", "TAG", 500), ("S1", "c2", "TAG", 500)], + orient="row", + schema={"sampleId": pl.String, "cellId": pl.String, "tag": pl.String, "umiCount": pl.Int64}, + ) + identities = combine_tags_to_identities(counts, {"TAG": "A"}) + reference = {("S1", "c1"): 900, ("S1", "c2"): 900} + gated, _ = gate_cells(reference, threshold=800) + admissibility = Admissibility(reference, 2, gated) + per_cell = read_states(identities, admissibility, cutoff=75.0) + + states = per_cell.with_columns(pl.lit("s1").alias("setId")).select( + "setId", "sampleId", "cellId", "identity", "state" + ) + cells_by_set = {"s1": [("S1", "c1"), ("S1", "c2")]} + r = _row(combine_cells(states, {"A"}, {"S1": {"A"}}, cells_by_set, admissibility), "A") + assert r["state"] == U and r["cellsCouldAnswer"] == 2 and r["cellsAnswered"] == 0 + assert r["unreliableReason"] == SetUnreliableReason.ALL_CELLS_GATED.value + + +def test_all_cells_gated_is_not_reported_when_the_reason_mix_is_not_unanimous(): + # One cell gated, one with no comparator at all: the set-wide reason is + # not "all cells gated" (it is not true) but the comparator failure that + # is present, per _dominant_reason's documented priority. + df = _states([]) + members = [("S1", "c1"), ("S1", "c2")] + cells_by_set = {"s1": members} + admissibility = Admissibility({("S1", "c1"): 900}, 2, {("S1", "c1")}) # c2 has no comparator entry + r = _row(combine_cells(df, {"A"}, {"S1": {"A"}}, cells_by_set, admissibility), "A") + assert r["state"] == U + assert r["unreliableReason"] == SetUnreliableReason.NO_COMPARATOR.value + + +def test_cellscouldanswer_is_not_a_row_count(): + # THE defect this reduction exists to fix. 40 cells; only 3 have a row in + # `states`, the other 37 are silent and admissible. cellsCouldAnswer must + # reflect all 40 cells asked (their sample offered A), never the 3 rows. + explicit = [("s1", "S1", "c0", "A", B), ("s1", "S1", "c1", "A", B), ("s1", "S1", "c2", "A", N)] + df = _states(explicit) + members = [("S1", "c0"), ("S1", "c1"), ("S1", "c2")] + [("S1", f"s{i}") for i in range(37)] + cells_by_set = {"s1": members} + admissibility = Admissibility({k: 5 for k in members}, 2, set()) + r = _row(combine_cells(df, {"A"}, {"S1": {"A"}}, cells_by_set, admissibility), "A") + assert r["cellsCouldAnswer"] == 40 # not 3 + assert r["cellsAnswered"] == 40 # 2 explicit bound + 1 explicit not-bound + 37 silent not-bound + assert r["state"] == N # 38 not-bound votes beat 2 bound + + +def test_a_set_spanning_two_panels_counts_only_the_asked_cells_and_does_not_inflate_silent_unreliable(): + # S1 offers A, S2 offers B (not A). The set holds cells from both. For + # identity A: cellsCouldAnswer must count only S1's cells, and S2's gated + # cell -- which never offered A -- must not inflate silentUnreliable at A. + df = _states([]) + members = [("S1", "c1"), ("S1", "c2"), ("S2", "c3"), ("S2", "c4")] + cells_by_set = {"s1": members} + # S1's cells are admissible; S2's c3 is gated, c4 has a normal reference. + reference = {("S1", "c1"): 5, ("S1", "c2"): 5, ("S2", "c3"): 900, ("S2", "c4"): 5} + admissibility = Admissibility(reference, 2, {("S2", "c3")}) + out = combine_cells(df, {"A", "B"}, {"S1": {"A"}, "S2": {"B"}}, cells_by_set, admissibility) + + row_a = _row(out, "A") + assert row_a["cellsCouldAnswer"] == 2 # only S1's two cells, not all four + assert row_a["state"] == N # both S1 cells silent and admissible -> not bound + assert row_a["cellsAnswered"] == 2 + + row_b = _row(out, "B") + assert row_b["cellsCouldAnswer"] == 2 # only S2's two cells + # S2's gated cell counts against B (which S2 offers), and its silent + # not-bound cell (c4) settles: one voter, one vote, not bound. + assert row_b["cellsAnswered"] == 1 + assert row_b["state"] == N diff --git a/software/per-cell-metrics/test/test_verdict.py b/software/per-cell-metrics/test/test_verdict.py index 26e3531..6328a50 100644 --- a/software/per-cell-metrics/test/test_verdict.py +++ b/software/per-cell-metrics/test/test_verdict.py @@ -505,12 +505,10 @@ def test_duplicated_observed_rows_are_rejected_not_silently_wrong(): silent_tally(observed, cells, {"S1": {"A"}}, admissibility) -def _check_silent_tally_matches_oracle(seed, cutoff=BOUND_CUTOFF, force_empty_sample=None): - # Shared by every arm below: build a small, varied population by +def _build_silent_tally_population(seed, force_empty_sample=None): + # Shared by every check below: build a small, varied population by # construction -- several samples, cells, identities, some cells gated, - # some below the thin line, some with a normal reference -- and check - # silent_tally's three cheap terms against the dense grid built by - # densify and read through read_states, which never skips a row. + # some below the thin line, some with a normal reference. rng = random.Random(seed) samples = ["S1", "S2", "S3"] identities = ["A", "B", "C"] @@ -547,6 +545,17 @@ def _check_silent_tally_matches_oracle(seed, cutoff=BOUND_CUTOFF, force_empty_sa if rng.random() < 0.5: tag_rows.append((sample, cell, identity, rng.randint(0, 30))) + return samples, identities, thin_line, gated, reference, cell_rows, tag_rows, offered_by_sample + + +def _check_silent_tally_matches_oracle(seed, cutoff=BOUND_CUTOFF, force_empty_sample=None): + # Checks silent_tally's three cheap terms, sample-keyed (the default), + # against the dense grid built by densify and read through read_states, + # which never skips a row. + samples, identities, thin_line, gated, reference, cell_rows, tag_rows, offered_by_sample = ( + _build_silent_tally_population(seed, force_empty_sample) + ) + cells = _cells(cell_rows) sparse_identities = _ident(tag_rows) admissibility = Admissibility(reference, thin_line, gated) @@ -622,3 +631,128 @@ def test_silent_tally_agrees_with_the_oracle_at_a_low_valid_cutoff(): # the boundary itself rather than assuming BOUND_CUTOFF=75.0 is # representative of every cutoff the equivalence must hold for. _check_silent_tally_matches_oracle(seed=20260817, cutoff=0.5) + + +def _check_silent_tally_matches_oracle_grouped(seed, cutoff=BOUND_CUTOFF, force_empty_sample=None): + # Same population and same dense oracle as the sample-keyed check above, + # but the cells are regrouped into sets that mix samples with different + # offered identities, and silent_tally is called with that grouping. A + # set's cell index (0..5) becomes its group, independent of sample, so + # every group is guaranteed to contain a member from all three samples + # -- exactly the shape a hoisted asked/total_inadmissible would get + # wrong, since S1, S2, S3 are built with independently random offered + # sets and need not agree on what a given group's identity was offered. + samples, identities, thin_line, gated, reference, cell_rows, tag_rows, offered_by_sample = ( + _build_silent_tally_population(seed, force_empty_sample) + ) + + cells = _cells(cell_rows) + sparse_identities = _ident(tag_rows) + admissibility = Admissibility(reference, thin_line, gated) + observed = read_states(sparse_identities, admissibility, cutoff) + dense = densify(sparse_identities, cells, offered_by_sample) + oracle = read_states(dense, admissibility, cutoff) + + group_by_cell = {(sample, cell): cell for sample, cell in cell_rows} + groups = sorted({cell for _, cell in cell_rows}) + + tally = silent_tally( + observed, cells, offered_by_sample, admissibility, group_by_cell=group_by_cell, group_column="setId" + ) + + for group in groups: + members = {k for k, g in group_by_cell.items() if g == group} + offered_here = set().union(*(offered_by_sample[k[0]] for k in members)) + for identity in identities: + group_filter = pl.col("setId") == group + if identity not in offered_here: + # None of this group's members' own samples offered it: no + # row at all, not a zero row. + assert tally.filter(group_filter & (pl.col("identity") == identity)).height == 0 + continue + + def _states_for(frame): + # A plain Python filter, not a polars struct comparison: this + # only needs to run over a handful of rows in a test, and it + # sidesteps any doubt about how polars compares struct columns. + return [ + state + for sample_id, cell_id, ident, state in zip( + frame["sampleId"].to_list(), + frame["cellId"].to_list(), + frame["identity"].to_list(), + frame["state"].to_list(), + strict=True, + ) + if (sample_id, cell_id) in members and ident == identity + ] + + oracle_states = _states_for(oracle) + observed_states = _states_for(observed) + + tally_row = tally.filter(group_filter & (pl.col("identity") == identity)).row(0, named=True) + + expected_silent_unreliable = oracle_states.count(State.UNRELIABLE.value) - observed_states.count( + State.UNRELIABLE.value + ) + expected_silent_not_bound = oracle_states.count(State.NOT_BOUND.value) - observed_states.count( + State.NOT_BOUND.value + ) + + assert tally_row["asked"] == len(oracle_states) + assert tally_row["observed"] == len(observed_states) + assert tally_row["silentUnreliable"] == expected_silent_unreliable + assert tally_row["silentNotBound"] == expected_silent_not_bound + + +@pytest.mark.parametrize( + "seed, force_empty_sample", + [ + (20260817, None), + (1, None), + (2, None), + (7, "S2"), + ], +) +def test_silent_tally_agrees_with_the_oracle_when_groups_span_differing_panels(seed, force_empty_sample): + _check_silent_tally_matches_oracle_grouped(seed, force_empty_sample=force_empty_sample) + + +def test_silent_tally_group_column_is_named_by_the_caller(): + cells = _cells([("S1", "c1"), ("S2", "c1")]) + observed = _ident([]) + admissibility = Admissibility({("S1", "c1"): 5, ("S2", "c1"): 5}, 2, set()) + tally = silent_tally( + observed, + cells, + {"S1": {"A"}, "S2": {"A"}}, + admissibility, + group_by_cell={("S1", "c1"): "G1", ("S2", "c1"): "G1"}, + group_column="setId", + ) + assert tally.columns[0] == "setId" + row = tally.row(0, named=True) + assert row["setId"] == "G1" and row["asked"] == 2 # both samples' c1 land in one group + + +def test_a_group_spanning_two_panels_does_not_inflate_silent_unreliable(): + # THE hoist bug, pinned directly: S1 offers A, S2 does not. A group holds + # one cell from each, and S2's cell is gated. A hoisted total_inadmissible + # would count S2's gated cell against identity A too, even though S2 + # never offered A -- inflating silentUnreliable for an identity that + # cell was never asked about. + cells = _cells([("S1", "c1"), ("S2", "c2")]) + observed = _ident([]) # both cells silent + admissibility = Admissibility({("S1", "c1"): 5}, 2, {("S2", "c2")}) + tally = silent_tally( + observed, + cells, + offered_by_sample={"S1": {"A"}, "S2": {"B"}}, + admissibility=admissibility, + group_by_cell={("S1", "c1"): "G1", ("S2", "c2"): "G1"}, + group_column="setId", + ) + row_a = tally.filter(pl.col("identity") == "A").row(0, named=True) + assert row_a["asked"] == 1 # only S1's cell, not S2's + assert row_a["silentUnreliable"] == 0 # S2's gated cell must not count against A + assert row_a["silentNotBound"] == 1 # S1's silent, admissible cell votes not bound From 51170009295f39ab500d4cd8e6cf252be91c9409 Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 15:35:21 +0200 Subject: [PATCH 035/282] MILAB-6496: tell a narrow majority apart from a tie --- software/per-cell-metrics/src/combine.py | 40 +++++++++++++++---- .../per-cell-metrics/test/test_combine.py | 19 ++++++++- 2 files changed, 50 insertions(+), 9 deletions(-) diff --git a/software/per-cell-metrics/src/combine.py b/software/per-cell-metrics/src/combine.py index b53acf9..e5462c4 100644 --- a/software/per-cell-metrics/src/combine.py +++ b/software/per-cell-metrics/src/combine.py @@ -62,6 +62,16 @@ class SetUnreliableReason(str, Enum): comparator-failed cells is reported by whichever comparator failure is present, since an admissibility gate excluding only part of a set is not by itself why the rest failed to settle. + + TIE and BELOW_AGREEMENT_FLOOR both leave the identity UNRELIABLE, and + look alike from the counts alone, but they call for different action and + so are kept apart. A TIE has no majority to trust: the settled cells + split evenly, which may be real heterogeneity in the clone, and no + parameter moves it. A BELOW_AGREEMENT_FLOOR set has one: a majority + formed, and it was refused only because `min_agreement` was raised above + it -- the fix is to lower that floor or gather more cells, not to + suspect the biology. Since `min_agreement` defaults to off, this reason + can only appear because someone raised it. """ NEVER_OFFERED = "never-offered" @@ -69,6 +79,7 @@ class SetUnreliableReason(str, Enum): THIN_COMPARATOR = "thin-comparator" ALL_CELLS_GATED = "all-cells-gated" TIE = "tie" + BELOW_AGREEMENT_FLOOR = "below-agreement-floor" TOO_FEW_VOTERS = "too-few-voters" @@ -206,14 +217,13 @@ def combine_cells( top_state, top_count, tied = _majority(counts) agreement = top_count / answered - # A tie is not a thin majority but the absence of one: half (or a - # third, or a quarter) of the settled votes contradict the rest, - # and nothing in the reading says which side to believe. The - # reason vocabulary has no separate label for "settled, but below - # the agreement floor" -- that case is reported as TIE too, since - # both mean the same thing to a reader: the majority that formed - # was not decisive enough to stand. - if tied or (min_agreement is not None and agreement < min_agreement): + # A tie has no majority to trust: the settled cells split evenly, + # and nothing in the reading says which side to believe. A narrow + # majority below the agreement floor has one -- it was refused + # only because the operator raised that floor. The two states + # this identity could still not settle in call for different + # action, so they get different reasons. + if tied: rows.append( { "setId": set_id, @@ -227,6 +237,20 @@ def combine_cells( ) continue + if min_agreement is not None and agreement < min_agreement: + rows.append( + { + "setId": set_id, + "identity": identity, + "state": State.UNRELIABLE.value, + "cellsCouldAnswer": could, + "cellsAnswered": answered, + "agreement": agreement, + "unreliableReason": SetUnreliableReason.BELOW_AGREEMENT_FLOOR.value, + } + ) + continue + rows.append( { "setId": set_id, diff --git a/software/per-cell-metrics/test/test_combine.py b/software/per-cell-metrics/test/test_combine.py index 7b02564..3618938 100644 --- a/software/per-cell-metrics/test/test_combine.py +++ b/software/per-cell-metrics/test/test_combine.py @@ -153,11 +153,28 @@ def test_exactly_min_agreement_settles_when_raised(): assert r["state"] == B and r["agreement"] == 0.75 -def test_just_below_min_agreement_is_unreliable(): +def test_just_below_min_agreement_is_below_agreement_floor_not_tie(): + # A real majority exists here (3 of 4) -- it is refused only because the + # operator raised min_agreement above it. That is a different reason + # than a tie, which has no majority to refuse. df = _states([("s1", "S1", f"b{i}", "A", B) for i in range(3)] + [("s1", "S1", "n0", "A", N)]) cells_by_set = {"s1": [("S1", f"b{i}") for i in range(3)] + [("S1", "n0")]} r = _row(combine_cells(df, {"A"}, {"S1": {"A"}}, cells_by_set, _NEUTRAL, min_agreement=0.76), "A") assert r["state"] == U + assert r["unreliableReason"] == SetUnreliableReason.BELOW_AGREEMENT_FLOOR.value + + +def test_a_genuine_tie_still_reads_tie_even_when_min_agreement_would_also_fail_it(): + # A fixture-coincidence trap: a tie's agreement is exactly 0.5, so any + # min_agreement above 0.5 would ALSO fail it, and a fixture where both + # conditions hold cannot tell which branch produced the answer. Raise + # min_agreement to 0.6 on the same 1-vs-1 tie from test_a_tie_cannot_be_settled + # and confirm the reason is still TIE, not BELOW_AGREEMENT_FLOOR -- the + # tie check must run and win regardless of where the floor sits. + df = _states([("s1", "S1", "c1", "A", B), ("s1", "S1", "c2", "A", N)]) + cells_by_set = {"s1": [("S1", "c1"), ("S1", "c2")]} + r = _row(combine_cells(df, {"A"}, {"S1": {"A"}}, cells_by_set, _NEUTRAL, min_agreement=0.6), "A") + assert r["state"] == U assert r["unreliableReason"] == SetUnreliableReason.TIE.value From b1f3e56c398c22042c8fb6ea0471536d68cb82d5 Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 15:49:38 +0200 Subject: [PATCH 036/282] MILAB-6496: one source of truth for which set a cell belongs to --- software/per-cell-metrics/src/combine.py | 53 +++++++++++-- .../per-cell-metrics/test/test_combine.py | 78 ++++++++++++++----- 2 files changed, 106 insertions(+), 25 deletions(-) diff --git a/software/per-cell-metrics/src/combine.py b/software/per-cell-metrics/src/combine.py index e5462c4..204bd53 100644 --- a/software/per-cell-metrics/src/combine.py +++ b/software/per-cell-metrics/src/combine.py @@ -93,8 +93,22 @@ def _dominant_reason(asked_keys: list[tuple[str, str]], admissibility: Admissibi that a silent admissible cell always resolves NOT_BOUND. So every key here has a real, non-None cell-level reason, and this only has to pick among the three. + + The assertion is not a formality: it is the difference between that + claim failing loudly and failing as a wrong answer. Without it, an + admissible key slipping in here (the caller's vote-counting and its + admissibility disagreeing about which cells were asked) reads a `None` + reason, matches neither of the two checks below, and falls through to + THIN_COMPARATOR -- reporting a comparator problem for a cell whose + comparator is fine. """ reasons = {_cell_admissibility_reason(k, admissibility) for k in asked_keys} + assert None not in reasons, ( + f"an admissible cell reached _dominant_reason among {asked_keys!r}: this is only called " + "when cellsAnswered is 0, which should be possible only when every asked cell is " + "individually inadmissible -- a None reason here means the caller's vote count and " + "admissibility disagree about which cells were actually asked" + ) if reasons == {UnreliableReason.GATED}: return SetUnreliableReason.ALL_CELLS_GATED if UnreliableReason.NO_COMPARATOR in reasons: @@ -120,10 +134,17 @@ def combine_cells( ) -> pl.DataFrame: """One row per (set, identity) over the whole universe. - `states` is per-cell output shaped like `read_states`' -- columns - setId, sampleId, cellId, identity, state -- one row per (cell, identity) - that got an explicit reading; a cell asked about an identity and absent - here is silent for it, not unasked. + `states` is `read_states`' output directly -- columns sampleId, cellId, + identity, state, plus whatever else `read_states` emits, which this + ignores -- one row per (cell, identity) that got an explicit reading; a + cell asked about an identity and absent here is silent for it, not + unasked. There is deliberately no setId column in that shape: which set + a row belongs to is decided once, below, by looking its cell up in + `cells_by_set` -- never by trusting a column, which would be a second, + independently-suppliable source of truth for the same fact. A row for a + cell no set in `cells_by_set` lists is dropped, exactly as `silent_tally` + already drops such cells from its own counts; a vote is never counted + for a cell that was not asked. `offered` is keyed by sample: for a given set, the identities it was offered are the union, over its member samples, of what each sample's @@ -135,10 +156,19 @@ def combine_cells( `cells_by_set` gives each set's full cell membership, including cells with no row in `states` at all -- the set's silent cells, which vote through `silent_tally` rather than through a row that was never written. + It must be disjoint: a cell key may repeat within one set's own list + with no effect, but must not appear under two different set ids, which + is asserted below rather than left to surface later as a `silent_tally` + precondition failure whose message points at the wrong function. """ group_by_cell: dict[tuple[str, str], str] = {} for set_id, members in cells_by_set.items(): for key in members: + owner = group_by_cell.get(key) + assert owner is None or owner == set_id, ( + f"cell {key!r} appears in both set {owner!r} and set {set_id!r} in cells_by_set: " + "a cell must belong to exactly one set" + ) group_by_cell[key] = set_id cells_frame = pl.DataFrame(list(group_by_cell), orient="row", schema={"sampleId": pl.String, "cellId": pl.String}) @@ -147,9 +177,20 @@ def combine_cells( settled = states.filter(pl.col("state").is_in(SETTLED)) explicit_counts: dict[tuple[str, str], dict[str, int]] = {} - for set_id, identity, state in zip( - settled["setId"].to_list(), settled["identity"].to_list(), settled["state"].to_list(), strict=True + for sample_id, cell_id, identity, state in zip( + settled["sampleId"].to_list(), + settled["cellId"].to_list(), + settled["identity"].to_list(), + settled["state"].to_list(), + strict=True, ): + set_id = group_by_cell.get((sample_id, cell_id)) + if set_id is None: + # This cell is not in any set's membership list: the same drop + # `silent_tally` applies to a cell absent from its `cells` frame, + # kept here so a vote can never be counted for a cell nobody + # asked to vote. + continue bucket = explicit_counts.setdefault((set_id, identity), {}) bucket[state] = bucket.get(state, 0) + 1 diff --git a/software/per-cell-metrics/test/test_combine.py b/software/per-cell-metrics/test/test_combine.py index 3618938..c828113 100644 --- a/software/per-cell-metrics/test/test_combine.py +++ b/software/per-cell-metrics/test/test_combine.py @@ -1,12 +1,14 @@ import polars as pl +import pytest from combine import DEFAULT_MIN_VOTERS, SetUnreliableReason, combine_cells from verdict import Admissibility, State, combine_tags_to_identities, gate_cells, read_states B, N, U, NA = (State.BOUND.value, State.NOT_BOUND.value, State.UNRELIABLE.value, State.NEVER_ASKED.value) +# No setId column: `combine_cells` derives which set a row belongs to from +# `cells_by_set` alone, matching `read_states`' actual output shape. _STATES_SCHEMA = { - "setId": pl.String, "sampleId": pl.String, "cellId": pl.String, "identity": pl.String, @@ -29,7 +31,7 @@ def _row(out, identity): def test_majority_wins(): - df = _states([("s1", "S1", "c1", "A", B), ("s1", "S1", "c2", "A", B), ("s1", "S1", "c3", "A", N)]) + df = _states([("S1", "c1", "A", B), ("S1", "c2", "A", B), ("S1", "c3", "A", N)]) cells_by_set = {"s1": [("S1", "c1"), ("S1", "c2"), ("S1", "c3")]} out = combine_cells(df, universe={"A"}, offered={"S1": {"A"}}, cells_by_set=cells_by_set, admissibility=_NEUTRAL) r = _row(out, "A") @@ -37,7 +39,7 @@ def test_majority_wins(): def test_vote_is_per_identity_so_a_set_can_bind_several(): - df = _states([("s1", "S1", "c1", i, B) for i in ("A", "C")]) + df = _states([("S1", "c1", i, B) for i in ("A", "C")]) cells_by_set = {"s1": [("S1", "c1")]} out = combine_cells( df, universe={"A", "C"}, offered={"S1": {"A", "C"}}, cells_by_set=cells_by_set, admissibility=_NEUTRAL @@ -46,7 +48,7 @@ def test_vote_is_per_identity_so_a_set_can_bind_several(): def test_a_tie_cannot_be_settled(): - df = _states([("s1", "S1", "c1", "A", B), ("s1", "S1", "c2", "A", N)]) + df = _states([("S1", "c1", "A", B), ("S1", "c2", "A", N)]) cells_by_set = {"s1": [("S1", "c1"), ("S1", "c2")]} out = combine_cells(df, {"A"}, {"S1": {"A"}}, cells_by_set, _NEUTRAL) r = _row(out, "A") @@ -58,7 +60,7 @@ def test_a_three_way_split_that_ties_at_the_top_is_also_unreliable(): # Not just the minimal 1-vs-1 tie: three cells settle bound, three settle # not bound. The tie check must compare the leading counts, not special- # case a count of one. - df = _states([("s1", "S1", f"b{i}", "A", B) for i in range(3)] + [("s1", "S1", f"n{i}", "A", N) for i in range(3)]) + df = _states([("S1", f"b{i}", "A", B) for i in range(3)] + [("S1", f"n{i}", "A", N) for i in range(3)]) cells_by_set = {"s1": [("S1", f"b{i}") for i in range(3)] + [("S1", f"n{i}") for i in range(3)]} r = _row(combine_cells(df, {"A"}, {"S1": {"A"}}, cells_by_set, _NEUTRAL), "A") assert r["state"] == U and r["cellsAnswered"] == 6 @@ -67,7 +69,7 @@ def test_a_three_way_split_that_ties_at_the_top_is_also_unreliable(): def test_never_asked_comes_only_from_not_being_offered(): # Z is in the universe and NOT offered -> never asked. - df = _states([("s1", "S1", "c1", "A", B)]) + df = _states([("S1", "c1", "A", B)]) cells_by_set = {"s1": [("S1", "c1")]} out = combine_cells( df, universe={"A", "Z"}, offered={"S1": {"A"}}, cells_by_set=cells_by_set, admissibility=_NEUTRAL @@ -81,7 +83,7 @@ def test_never_asked_comes_only_from_not_being_offered(): def test_an_offered_identity_nobody_bound_is_not_bound_not_never_asked(): # Explicit rows, every one not-bound: offered, everybody read zero, so # the verdict is not bound, never never-asked. - df = _states([("s1", "S1", "c1", "A", N), ("s1", "S1", "c2", "A", N)]) + df = _states([("S1", "c1", "A", N), ("S1", "c2", "A", N)]) cells_by_set = {"s1": [("S1", "c1"), ("S1", "c2")]} r = _row(combine_cells(df, {"A"}, {"S1": {"A"}}, cells_by_set, _NEUTRAL), "A") assert r["state"] == N and r["state"] != NA @@ -106,7 +108,7 @@ def test_silent_cells_vote_an_antigen_every_cell_failed_still_reads_not_bound(): def test_unsettled_cells_do_not_vote_but_do_count_as_could_answer(): - df = _states([("s1", "S1", "c1", "A", B), ("s1", "S1", "c2", "A", U), ("s1", "S1", "c3", "A", U)]) + df = _states([("S1", "c1", "A", B), ("S1", "c2", "A", U), ("S1", "c3", "A", U)]) cells_by_set = {"s1": [("S1", "c1"), ("S1", "c2"), ("S1", "c3")]} r = _row(combine_cells(df, {"A"}, {"S1": {"A"}}, cells_by_set, _NEUTRAL), "A") assert r["state"] == B and r["cellsAnswered"] == 1 and r["cellsCouldAnswer"] == 3 @@ -114,14 +116,14 @@ def test_unsettled_cells_do_not_vote_but_do_count_as_could_answer(): def test_a_verdict_may_rest_on_one_cell_and_says_so(): assert DEFAULT_MIN_VOTERS == 1 - df = _states([("s1", "S1", "c1", "A", B)]) + df = _states([("S1", "c1", "A", B)]) cells_by_set = {"s1": [("S1", "c1")]} r = _row(combine_cells(df, {"A"}, {"S1": {"A"}}, cells_by_set, _NEUTRAL), "A") assert r["state"] == B and r["cellsAnswered"] == 1 def test_below_min_voters_is_unreliable_when_raised(): - df = _states([("s1", "S1", "c1", "A", B)]) + df = _states([("S1", "c1", "A", B)]) cells_by_set = {"s1": [("S1", "c1")]} r = _row(combine_cells(df, {"A"}, {"S1": {"A"}}, cells_by_set, _NEUTRAL, min_voters=2), "A") assert r["state"] == U @@ -131,14 +133,14 @@ def test_below_min_voters_is_unreliable_when_raised(): def test_exactly_min_voters_settles(): # The named value satisfies the condition it names, as elsewhere in this # project: two settled votes with min_voters=2 must settle, not fail. - df = _states([("s1", "S1", "c1", "A", B), ("s1", "S1", "c2", "A", B)]) + df = _states([("S1", "c1", "A", B), ("S1", "c2", "A", B)]) cells_by_set = {"s1": [("S1", "c1"), ("S1", "c2")]} r = _row(combine_cells(df, {"A"}, {"S1": {"A"}}, cells_by_set, _NEUTRAL, min_voters=2), "A") assert r["state"] == B and r["cellsAnswered"] == 2 def test_narrow_majority_stands_and_reports_how_narrow(): - df = _states([("s1", "S1", f"c{i}", "A", B) for i in range(6)] + [("s1", "S1", f"d{i}", "A", N) for i in range(5)]) + df = _states([("S1", f"c{i}", "A", B) for i in range(6)] + [("S1", f"d{i}", "A", N) for i in range(5)]) cells_by_set = {"s1": [("S1", f"c{i}") for i in range(6)] + [("S1", f"d{i}") for i in range(5)]} r = _row(combine_cells(df, {"A"}, {"S1": {"A"}}, cells_by_set, _NEUTRAL), "A") assert r["state"] == B and r["agreement"] == 6 / 11 @@ -147,7 +149,7 @@ def test_narrow_majority_stands_and_reports_how_narrow(): def test_exactly_min_agreement_settles_when_raised(): # 3 bound, 1 not bound -> agreement 0.75. Raising min_agreement to # exactly 0.75 must still settle: the boundary belongs to the pass side. - df = _states([("s1", "S1", f"b{i}", "A", B) for i in range(3)] + [("s1", "S1", "n0", "A", N)]) + df = _states([("S1", f"b{i}", "A", B) for i in range(3)] + [("S1", "n0", "A", N)]) cells_by_set = {"s1": [("S1", f"b{i}") for i in range(3)] + [("S1", "n0")]} r = _row(combine_cells(df, {"A"}, {"S1": {"A"}}, cells_by_set, _NEUTRAL, min_agreement=0.75), "A") assert r["state"] == B and r["agreement"] == 0.75 @@ -157,7 +159,7 @@ def test_just_below_min_agreement_is_below_agreement_floor_not_tie(): # A real majority exists here (3 of 4) -- it is refused only because the # operator raised min_agreement above it. That is a different reason # than a tie, which has no majority to refuse. - df = _states([("s1", "S1", f"b{i}", "A", B) for i in range(3)] + [("s1", "S1", "n0", "A", N)]) + df = _states([("S1", f"b{i}", "A", B) for i in range(3)] + [("S1", "n0", "A", N)]) cells_by_set = {"s1": [("S1", f"b{i}") for i in range(3)] + [("S1", "n0")]} r = _row(combine_cells(df, {"A"}, {"S1": {"A"}}, cells_by_set, _NEUTRAL, min_agreement=0.76), "A") assert r["state"] == U @@ -171,7 +173,7 @@ def test_a_genuine_tie_still_reads_tie_even_when_min_agreement_would_also_fail_i # min_agreement to 0.6 on the same 1-vs-1 tie from test_a_tie_cannot_be_settled # and confirm the reason is still TIE, not BELOW_AGREEMENT_FLOOR -- the # tie check must run and win regardless of where the floor sits. - df = _states([("s1", "S1", "c1", "A", B), ("s1", "S1", "c2", "A", N)]) + df = _states([("S1", "c1", "A", B), ("S1", "c2", "A", N)]) cells_by_set = {"s1": [("S1", "c1"), ("S1", "c2")]} r = _row(combine_cells(df, {"A"}, {"S1": {"A"}}, cells_by_set, _NEUTRAL, min_agreement=0.6), "A") assert r["state"] == U @@ -195,9 +197,9 @@ def test_set_with_every_cell_set_aside_is_unreliable_through_the_real_pipeline() admissibility = Admissibility(reference, 2, gated) per_cell = read_states(identities, admissibility, cutoff=75.0) - states = per_cell.with_columns(pl.lit("s1").alias("setId")).select( - "setId", "sampleId", "cellId", "identity", "state" - ) + # No setId to attach: which set these rows belong to comes from + # cells_by_set below, not from a column on states. + states = per_cell.select("sampleId", "cellId", "identity", "state") cells_by_set = {"s1": [("S1", "c1"), ("S1", "c2")]} r = _row(combine_cells(states, {"A"}, {"S1": {"A"}}, cells_by_set, admissibility), "A") assert r["state"] == U and r["cellsCouldAnswer"] == 2 and r["cellsAnswered"] == 0 @@ -221,7 +223,7 @@ def test_cellscouldanswer_is_not_a_row_count(): # THE defect this reduction exists to fix. 40 cells; only 3 have a row in # `states`, the other 37 are silent and admissible. cellsCouldAnswer must # reflect all 40 cells asked (their sample offered A), never the 3 rows. - explicit = [("s1", "S1", "c0", "A", B), ("s1", "S1", "c1", "A", B), ("s1", "S1", "c2", "A", N)] + explicit = [("S1", "c0", "A", B), ("S1", "c1", "A", B), ("S1", "c2", "A", N)] df = _states(explicit) members = [("S1", "c0"), ("S1", "c1"), ("S1", "c2")] + [("S1", f"s{i}") for i in range(37)] cells_by_set = {"s1": members} @@ -255,3 +257,41 @@ def test_a_set_spanning_two_panels_counts_only_the_asked_cells_and_does_not_infl # not-bound cell (c4) settles: one voter, one vote, not bound. assert row_b["cellsAnswered"] == 1 assert row_b["state"] == N + + +def test_a_row_for_a_cell_no_set_lists_is_ignored(): + # A stray row for a cell absent from every set's membership must not + # vote: cellsAnswered must never exceed cellsCouldAnswer. Before the fix, + # a stray row like this counted toward the set it happened to name in a + # setId column; there is no such column now, only cells_by_set, and this + # cell is not in it. + df = _states([("S1", "c1", "A", B), ("S1", "stray", "A", B)]) + cells_by_set = {"s1": [("S1", "c1")]} + r = _row(combine_cells(df, {"A"}, {"S1": {"A"}}, cells_by_set, _NEUTRAL), "A") + assert r["cellsCouldAnswer"] == 1 + assert r["cellsAnswered"] == 1 + assert r["cellsAnswered"] <= r["cellsCouldAnswer"] + + +def test_a_cell_in_two_sets_fails_naming_cells_by_set(): + # A cell listed under two different set ids is a malformed cells_by_set, + # not a silent_tally precondition violation: the failure must name the + # thing that is actually wrong. + cells_by_set = {"s1": [("S1", "c1")], "s2": [("S1", "c1")]} + df = _states([]) + with pytest.raises(AssertionError, match="cells_by_set"): + combine_cells(df, {"A"}, {"S1": {"A"}}, cells_by_set, _NEUTRAL) + + +def test_dominant_reason_raises_rather_than_falling_through_to_thin_comparator(): + # A malformed but constructible input: `states` claims this cell is + # UNRELIABLE, while `admissibility` says it is perfectly fine -- a real + # comparator, not gated, not thin. That contradiction is exactly what + # used to let an admissible key reach _dominant_reason and fall through + # to THIN_COMPARATOR; it must now raise instead of reporting a + # comparator problem for a cell whose comparator is fine. + df = _states([("S1", "c1", "A", U)]) + cells_by_set = {"s1": [("S1", "c1")]} + admissibility = Admissibility({("S1", "c1"): 10}, 2, set()) + with pytest.raises(AssertionError): + combine_cells(df, {"A"}, {"S1": {"A"}}, cells_by_set, admissibility) From 9ab626e8fcbcc3cd33427cde87426bf45bb3579b Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 15:57:18 +0200 Subject: [PATCH 037/282] MILAB-6496: name a bound competitor beside a negative --- software/per-cell-metrics/src/combine.py | 63 +++++++++++ .../per-cell-metrics/test/test_combine.py | 105 +++++++++++++++++- 2 files changed, 167 insertions(+), 1 deletion(-) diff --git a/software/per-cell-metrics/src/combine.py b/software/per-cell-metrics/src/combine.py index 204bd53..9b81f6e 100644 --- a/software/per-cell-metrics/src/combine.py +++ b/software/per-cell-metrics/src/combine.py @@ -316,3 +316,66 @@ def combine_cells( "unreliableReason": pl.String, }, ).sort(["setId", "identity"]) + + +def attach_competitor_notes(verdicts: pl.DataFrame, contending: list[set[str]]) -> pl.DataFrame: + """Name the bound competitor beside a not-bound reading; change nothing else. + + A negative beside a bound competitor and one beside nothing are different + evidence, and the counts cannot tell them apart. The verdict reports what + could have caused the reading and leaves the call to the reader; the state + stays at *not bound* because the doubt travels beside it. Only a settled + NOT_BOUND row is eligible: an UNRELIABLE or NEVER_ASKED row made no + comparison to begin with, so it has no negative for a competitor to sit + beside. + + `wasCompeted` exists so a statement can test the note. Without a predicate + a condition naming the off-target passes on the state alone, the doubt is + lost, and *not bound* becomes a claim the run never earned. It is emitted + as an explicit "true"/"false" string on every row, matching the convention + this project already uses for a boolean that becomes a p-column value + (see `emit_feature_properties.py`'s control-feature marker) -- never null, + because a downstream filter for the absence of contention must be able to + match on the flag alone. + + Which identities contend is chosen when the repertoire is annotated and is + never inferred from the counts: contention is a property of the design, so + no arithmetic over the readings recovers it. An identity may sit in more + than one declared group; the note then names the union of bound + competitors across every group that contains it, since each group is an + independent claim of contention and a reader has no reason to see only + one of them. The names are joined in sorted order so the same data always + produces the same string, which matters once this column is + content-addressed as a p-column. + """ + if not contending: + return verdicts.with_columns( + pl.lit(None, dtype=pl.String).alias("competedWith"), + pl.lit("false", dtype=pl.String).alias("wasCompeted"), + ) + + bound_by_set: dict[str, set[str]] = {} + for row in verdicts.filter(pl.col("state") == State.BOUND.value).iter_rows(named=True): + bound_by_set.setdefault(row["setId"], set()).add(row["identity"]) + + notes, flags = [], [] + for row in verdicts.iter_rows(named=True): + note = None + if row["state"] == State.NOT_BOUND.value: + bound_here = bound_by_set.get(row["setId"], set()) + rivals = { + other + for group in contending + if row["identity"] in group + for other in group & bound_here + if other != row["identity"] + } + if rivals: + note = ", ".join(sorted(rivals)) + notes.append(note) + flags.append("true" if note else "false") + + return verdicts.with_columns( + pl.Series("competedWith", notes, dtype=pl.String), + pl.Series("wasCompeted", flags, dtype=pl.String), + ) diff --git a/software/per-cell-metrics/test/test_combine.py b/software/per-cell-metrics/test/test_combine.py index c828113..3dcd64e 100644 --- a/software/per-cell-metrics/test/test_combine.py +++ b/software/per-cell-metrics/test/test_combine.py @@ -1,6 +1,6 @@ import polars as pl import pytest -from combine import DEFAULT_MIN_VOTERS, SetUnreliableReason, combine_cells +from combine import DEFAULT_MIN_VOTERS, SetUnreliableReason, attach_competitor_notes, combine_cells from verdict import Admissibility, State, combine_tags_to_identities, gate_cells, read_states B, N, U, NA = (State.BOUND.value, State.NOT_BOUND.value, State.UNRELIABLE.value, State.NEVER_ASKED.value) @@ -295,3 +295,106 @@ def test_dominant_reason_raises_rather_than_falling_through_to_thin_comparator() admissibility = Admissibility({("S1", "c1"): 10}, 2, set()) with pytest.raises(AssertionError): combine_cells(df, {"A"}, {"S1": {"A"}}, cells_by_set, admissibility) + + +def _verdicts(rows): + return pl.DataFrame(rows, orient="row", schema={"setId": pl.String, "identity": pl.String, "state": pl.String}) + + +def _competitor_row(out, identity): + return out.filter(pl.col("identity") == identity).row(0, named=True) + + +def test_negative_beside_a_bound_competitor_names_it(): + out = attach_competitor_notes(_verdicts([("s1", "A", B), ("s1", "C", N)]), [{"A", "C"}]) + r = _competitor_row(out, "C") + assert r["competedWith"] == "A" and r["state"] == N + + +def test_a_statement_can_test_the_note(): + out = attach_competitor_notes(_verdicts([("s1", "A", B), ("s1", "C", N)]), [{"A", "C"}]) + assert _competitor_row(out, "C")["wasCompeted"] == "true" + assert _competitor_row(out, "A")["wasCompeted"] == "false" + + +def test_no_note_where_no_competitor_was_bound(): + out = attach_competitor_notes(_verdicts([("s1", "A", N), ("s1", "C", N)]), [{"A", "C"}]) + assert _competitor_row(out, "C")["competedWith"] is None + + +def test_no_note_on_a_bound_identity(): + out = attach_competitor_notes(_verdicts([("s1", "A", B), ("s1", "C", B)]), [{"A", "C"}]) + assert out["competedWith"].to_list() == [None, None] + + +def test_no_note_without_a_declared_group(): + out = attach_competitor_notes(_verdicts([("s1", "A", B), ("s1", "C", N)]), []) + assert out["competedWith"].to_list() == [None, None] + + +def test_notes_do_not_leak_across_sets(): + out = attach_competitor_notes(_verdicts([("s1", "A", B), ("s2", "C", N)]), [{"A", "C"}]) + assert _competitor_row(out.filter(pl.col("setId") == "s2"), "C")["competedWith"] is None + + +def test_several_bound_competitors_are_all_named(): + out = attach_competitor_notes(_verdicts([("s1", "A", B), ("s1", "B", B), ("s1", "C", N)]), [{"A", "B", "C"}]) + assert _competitor_row(out, "C")["competedWith"] == "A, B" + + +def test_was_competed_is_the_string_false_never_null_with_no_declared_groups(): + # wasCompeted is the predicate a downstream statement filters on. With no + # contending groups at all, every row's flag must still be the literal + # string "false" -- a null here would make "wasCompeted == false" fail to + # match the exact rows the flag exists to describe. + out = attach_competitor_notes(_verdicts([("s1", "A", B), ("s1", "C", N)]), []) + assert out["wasCompeted"].to_list() == ["false", "false"] + assert out["wasCompeted"].dtype == pl.String + + +def test_was_competed_is_the_string_false_never_null_with_declared_groups_present(): + # Same requirement, but with a declared group in play and a row that + # simply has no bound rival: the flag column must not switch to null just + # because contention was possible elsewhere in the frame. + out = attach_competitor_notes(_verdicts([("s1", "A", N), ("s1", "C", N)]), [{"A", "C"}]) + assert out["wasCompeted"].to_list() == ["false", "false"] + + +def test_no_note_on_an_unreliable_reading(): + # An UNRELIABLE identity made no settled comparison, so it has no + # negative for a competitor to sit beside -- naming one would assert a + # comparison this run never made. + out = attach_competitor_notes(_verdicts([("s1", "A", B), ("s1", "C", U)]), [{"A", "C"}]) + r = _competitor_row(out, "C") + assert r["competedWith"] is None + assert r["wasCompeted"] == "false" + + +def test_no_note_on_a_never_asked_reading(): + out = attach_competitor_notes(_verdicts([("s1", "A", B), ("s1", "C", NA)]), [{"A", "C"}]) + r = _competitor_row(out, "C") + assert r["competedWith"] is None + assert r["wasCompeted"] == "false" + + +def test_overlapping_declared_groups_union_their_bound_competitors(): + # C sits in two declared groups, {A, C} and {C, D}, with A and D each + # bound in only one of them. The note names both: the union of bound + # competitors across every group that contains the identity, not just + # the first matching group. + out = attach_competitor_notes( + _verdicts([("s1", "A", B), ("s1", "D", B), ("s1", "C", N)]), + [{"A", "C"}, {"C", "D"}], + ) + assert _competitor_row(out, "C")["competedWith"] == "A, D" + + +def test_competitor_names_are_joined_in_sorted_order(): + # Three bound rivals whose declared-group and bound-set iteration order + # is not alphabetical; only a sorted join reliably reads "Bee, Mango, + # Zebra" run after run. A byte-stable column depends on this. + out = attach_competitor_notes( + _verdicts([("s1", "Zebra", B), ("s1", "Mango", B), ("s1", "Bee", B), ("s1", "C", N)]), + [{"Zebra", "Mango", "Bee", "C"}], + ) + assert _competitor_row(out, "C")["competedWith"] == "Bee, Mango, Zebra" From 04de5453f16d0eae381cf897f339f7ca71e6adfc Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 16:03:43 +0200 Subject: [PATCH 038/282] MILAB-6496: count a set against what it was offered and settled --- software/per-cell-metrics/src/combine.py | 40 +++++++ .../per-cell-metrics/test/test_combine.py | 105 +++++++++++++++++- 2 files changed, 144 insertions(+), 1 deletion(-) diff --git a/software/per-cell-metrics/src/combine.py b/software/per-cell-metrics/src/combine.py index 9b81f6e..5c494d5 100644 --- a/software/per-cell-metrics/src/combine.py +++ b/software/per-cell-metrics/src/combine.py @@ -379,3 +379,43 @@ def attach_competitor_notes(verdicts: pl.DataFrame, contending: list[set[str]]) pl.Series("competedWith", notes, dtype=pl.String), pl.Series("wasCompeted", flags, dtype=pl.String), ) + + +def set_counts(verdicts: pl.DataFrame) -> pl.DataFrame: + """Per set: bound, offered, settled, unsettled -- in identities. + + The denominator is the identities the set was offered and whose reading + settled, not the size of the panel. A clonotype whose cells came from a + sample carrying only eight of ten, and which bound all eight, covered + everything it was ever asked; reported as eight of ten it looks like a + clone with two failures. + + An offered position that did not settle leaves the count and is reported + beside it. Voiding the count instead is what the four-state model + literally implies, but on a large panel a single bad reading would then + destroy every count in the run. + + `offeredCount` always equals `settledCount + unsettledCount`, since + UNRELIABLE is the only offered-but-unsettled state. A set asked nothing + (every identity NEVER_ASKED) reports all four counts as zero; a consumer + computing a rate from `boundCount` and `offeredCount` must guard the + division themselves, since this function cannot produce a rate for a set + that was asked nothing. A set that is entirely UNRELIABLE reports + `boundCount=0, settledCount=0, unsettledCount=N` -- read that as nothing + settled, never as a failure to bind N identities, since none of them + were ever compared. + + `verdicts` is read at its existing (setId, identity) row grain, one row + per identity regardless of how many tags fed it, so counting rows counts + identities, never tags. + """ + return ( + verdicts.group_by("setId") + .agg( + (pl.col("state") == State.BOUND.value).sum().alias("boundCount"), + (pl.col("state") != State.NEVER_ASKED.value).sum().alias("offeredCount"), + pl.col("state").is_in(SETTLED).sum().alias("settledCount"), + (pl.col("state") == State.UNRELIABLE.value).sum().alias("unsettledCount"), + ) + .sort("setId") + ) diff --git a/software/per-cell-metrics/test/test_combine.py b/software/per-cell-metrics/test/test_combine.py index 3dcd64e..07ab4d9 100644 --- a/software/per-cell-metrics/test/test_combine.py +++ b/software/per-cell-metrics/test/test_combine.py @@ -1,6 +1,8 @@ +import random + import polars as pl import pytest -from combine import DEFAULT_MIN_VOTERS, SetUnreliableReason, attach_competitor_notes, combine_cells +from combine import DEFAULT_MIN_VOTERS, SetUnreliableReason, attach_competitor_notes, combine_cells, set_counts from verdict import Admissibility, State, combine_tags_to_identities, gate_cells, read_states B, N, U, NA = (State.BOUND.value, State.NOT_BOUND.value, State.UNRELIABLE.value, State.NEVER_ASKED.value) @@ -398,3 +400,104 @@ def test_competitor_names_are_joined_in_sorted_order(): [{"Zebra", "Mango", "Bee", "C"}], ) assert _competitor_row(out, "C")["competedWith"] == "Bee, Mango, Zebra" + + +def _v(rows): + return pl.DataFrame(rows, orient="row", schema={"setId": pl.String, "identity": pl.String, "state": pl.String}) + + +def test_denominator_is_offered_and_settled(): + v = _v([("s1", f"i{i}", B) for i in range(8)] + [("s1", "i8", U), ("s1", "i9", NA)]) + r = set_counts(v).row(0, named=True) + assert r["boundCount"] == 8 + assert r["settledCount"] == 8 # i8 unsettled, i9 never asked + assert r["offeredCount"] == 9 # never-asked is not offered + assert r["unsettledCount"] == 1 + + +def test_not_bound_is_settled_and_in_the_denominator(): + v = _v([("s1", "a", B), ("s1", "b", N)]) + r = set_counts(v).row(0, named=True) + assert r["boundCount"] == 1 and r["settledCount"] == 2 and r["unsettledCount"] == 0 + + +def test_never_asked_is_outside_the_denominator(): + v = _v([("s1", "a", B), ("s1", "b", NA)]) + r = set_counts(v).row(0, named=True) + assert r["offeredCount"] == 1 and r["settledCount"] == 1 + + +def test_counts_are_in_identities_not_tags(): + # One identity carried on two tags is one row here, so it counts once. + v = _v([("s1", "family", B)]) + assert set_counts(v).row(0, named=True)["boundCount"] == 1 + + +def test_each_set_counted_separately(): + v = _v([("s1", "a", B), ("s2", "a", N)]) + out = set_counts(v).sort("setId") + assert out["boundCount"].to_list() == [1, 0] + + +def test_offered_equals_settled_plus_unsettled_with_all_four_states_present(): + # A fixture carrying BOUND, NOT_BOUND, UNRELIABLE, and NEVER_ASKED at + # once, so the arithmetic relationship is pinned rather than incidentally + # true because some state never appeared. A predicate that counts the + # wrong states (say offeredCount including NEVER_ASKED, or settledCount + # including UNRELIABLE) passes every test above that uses only two or + # three states; this one does not let that slip through. + v = _v([("s1", "a", B), ("s1", "b", N), ("s1", "c", U), ("s1", "d", NA)]) + r = set_counts(v).row(0, named=True) + assert r["offeredCount"] == r["settledCount"] + r["unsettledCount"] + assert r["boundCount"] <= r["settledCount"] + assert r["boundCount"] == 1 + assert r["settledCount"] == 2 + assert r["unsettledCount"] == 1 + assert r["offeredCount"] == 3 + + +def test_a_set_asked_nothing_reports_all_zero_and_a_reader_must_guard_the_divide(): + # Every position NEVER_ASKED: offeredCount is 0, so a downstream reader + # computing boundCount / offeredCount would divide by zero. This pins + # what the row emits -- all zeros -- rather than leaving the shape + # undocumented; the guard against the zero is the caller's job, since + # this function cannot produce a rate for a set that was asked nothing. + v = _v([("s1", "a", NA), ("s1", "b", NA)]) + r = set_counts(v).row(0, named=True) + assert r["boundCount"] == 0 + assert r["offeredCount"] == 0 + assert r["settledCount"] == 0 + assert r["unsettledCount"] == 0 + + +def test_a_set_entirely_unreliable_reads_as_nothing_settled_not_as_a_bind_failure(): + # All positions UNRELIABLE: boundCount=0, settledCount=0, unsettledCount=N. + # This is the shape a fully-gated or comparator-less set produces, and it + # is the one most likely to be misread downstream as "bound none of N" -- + # the honest reading is that nothing settled, since no comparison was + # ever made. + v = _v([("s1", "a", U), ("s1", "b", U), ("s1", "c", U)]) + r = set_counts(v).row(0, named=True) + assert r["boundCount"] == 0 + assert r["settledCount"] == 0 + assert r["unsettledCount"] == 3 + assert r["offeredCount"] == 3 + + +def test_output_row_order_is_deterministic_regardless_of_input_row_order(): + # This becomes a p-column, so it must be byte-stable: the same verdicts + # fed in several shuffled row orders must produce one identical output, + # including row order, not merely equal counts. + rows = ( + [("s3", "a", B), ("s3", "b", N)] + + [("s1", "a", B), ("s1", "b", U), ("s1", "c", NA)] + + [("s2", "a", N), ("s2", "b", N)] + ) + baseline = set_counts(_v(rows)) + + rng = random.Random(1234) + for _ in range(5): + shuffled = list(rows) + rng.shuffle(shuffled) + out = set_counts(_v(shuffled)) + assert out.equals(baseline) From c9846931fb4e198e39a5eabe508ec14c6ea175f0 Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 16:09:28 +0200 Subject: [PATCH 039/282] MILAB-6496: strip the single-winner dominance readout MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The block computed a dominant feature per cell — one antigen named as the winner. The verdict model asks a different question of every antigen independently, and a single winner is not a simplification of it: a cell that bound three antigens has three verdicts, not one dominant one and two absences. Removes the dominance call, the consensus and specificity outputs, and the four CLI flags that drove them. The workflow, model and UI still reference those flags and outputs; they are rewritten for verdicts in later work, and the block does not run end to end until they are. --- .../per-cell-metrics/src/per_cell_metrics.py | 281 +-------- .../test/test_per_cell_metrics.py | 569 +----------------- 2 files changed, 26 insertions(+), 824 deletions(-) diff --git a/software/per-cell-metrics/src/per_cell_metrics.py b/software/per-cell-metrics/src/per_cell_metrics.py index f8251be..6c72e38 100644 --- a/software/per-cell-metrics/src/per_cell_metrics.py +++ b/software/per-cell-metrics/src/per_cell_metrics.py @@ -1,8 +1,7 @@ """Per-cell feature metrics for the Feature Integration block. Collapses mitool tag-stat output into a (cell x feature) UMI matrix, then computes within-cell -fractions, the consensus feature (dominant-category rule), and an optional Cell Ranger -specificity score. +fractions and a per-cell summary of that matrix. The math functions are pure and unit-tested; the CLI wires them to CSV I/O. Every output is sorted before writing: stable row order makes the CLI deterministic and keeps the workflow's pure-template @@ -14,122 +13,6 @@ import sys import polars as pl -from scipy.stats import beta - -DOMINANCE_FLOOR = 0.5 # threshold is user-adjustable down to 0.5, never lower - -# Schema for the no-control specificity output only. When no negative control is set we still emit a -# header-only specificity CSV (the workflow's output set is fixed), and that frame has no source rows, -# so it needs an explicit schema. Every other output is a pure-polars transform of `counts`, which -# carries its schema through the empty case natively (an empty join writes a header-only CSV, not a -# crash). -_SPECIFICITY_SCHEMA = { - "sampleId": pl.Utf8, - "cellId": pl.Utf8, - "feature": pl.Utf8, - "specificityScore": pl.Float64, -} - - -CROSS_REACTIVE = "Target cross-reactive" - - -def consensus_category( - counts: dict[str, float], - threshold: float, - control: str | None = None, - offtargets: frozenset[str] = frozenset(), - label_crossreactive: bool = False, -) -> str | None: - """Dominant-category rule. - - Returns the single dominant category when it is the unique maximum AND its share of the total is - >= threshold; "ambiguous" when signal exists but no unique category passes (a spread distribution, - or an exact split at the 0.5 floor); None when there is no signal at all. ``threshold`` is clamped - up to the 0.5 floor. - - The negative ``control`` and the ``offtargets`` set are references, not callable antigens: they are - excluded from the winner candidates, so a cell dominated by control/off-target signal is "ambiguous", - never the control or an off-target. Their UMIs are still counted in ``total`` (the denominator), so - control/off-target signal SUPPRESSES antigen dominance rather than being renormalised away — a cell - swamped by them correctly fails the threshold instead of having its top on-target inflated to 100%. - - ``offtargets`` designate features whose property (e.g. Type = Off-Target) marks them as - binders the user does not want to call. When they are supplied and ``label_crossreactive`` is set, - the overloaded "ambiguous" bucket is split: a cell whose on-target (non-excluded) signal collectively - passes the threshold but is spread across >= 2 on-target features is called "cross-reactive" (a - genuine multi-/cross-reactive binder — e.g. the same target's human + cyno variants) rather than - lumped with true noise. A cell whose on-target signal fails the threshold (off-target/control-swamped, - or a flat spread) stays "ambiguous". With no off-targets designated the rule is unchanged. - """ - threshold = max(threshold, DOMINANCE_FLOOR) - excluded = set(offtargets) - if control is not None: - excluded.add(control) - positive = {k: v for k, v in counts.items() if v > 0} - total = sum(positive.values()) - if total <= 0: - return None - candidates = {k: v for k, v in positive.items() if k not in excluded} - if not candidates: - return "ambiguous" # only control/off-target (or no) signal — no on-target to call - max_val = max(candidates.values()) - winners = [k for k, v in candidates.items() if v == max_val] - if len(winners) == 1 and (max_val / total) >= threshold: - return winners[0] - # cross-reactive: on-target signal collectively dominates but is split across >= 2 on-targets. - if label_crossreactive and len(candidates) >= 2 and (sum(candidates.values()) / total) >= threshold: - return CROSS_REACTIVE - return "ambiguous" - - -def offtarget_features( - tag_feature_csv: str, - csv_feature_col: str, - offtarget_col: str, - offtarget_values: frozenset[str], -) -> frozenset[str]: - """Feature names whose designated property (``offtarget_col``) value is in ``offtarget_values``. - - The off-target designation is property-driven: the user picks one imported per-feature property - column (e.g. ``antigen_class``) and the set of its values that mark a feature as off-target (e.g. - {"Off-Target", "Off-target"}). This reads the tag->feature CSV — which carries those property columns — - and returns the resolved set of off-target FEATURE names, so the dominant call can exclude them. - - Values are matched exactly, whitespace-trimmed but CASE-SENSITIVE (``strip()`` on both sides, no - case folding): a feature is off-target only if its ``offtarget_col`` value is byte-identical (after - trimming) to one the user selected. Real panels (e.g. B043) may carry mixed casing of one designation - — ``Off-Target`` and ``Off-target`` in a single column — so the user selects every casing they mean; - each distinct value is offered separately in the block's dropdown. Whitespace is trimmed because - leading/trailing spaces are invisible in the picker; casing is left intact because it is visible and - the user's to choose (the block never silently broadens a selection to unselected values). The - returned FEATURE names are verbatim (trimmed) from the CSV. - """ - mapping = pl.read_csv(tag_feature_csv) - if offtarget_col not in mapping.columns: - raise SystemExit( - f"--offtarget-col={offtarget_col!r} is not a column of the tag->feature CSV ({mapping.columns})" - ) - wanted_trimmed = {v.strip() for v in offtarget_values} - resolved = { - (feat or "").strip() - for feat, val in mapping.select( - pl.col(csv_feature_col).cast(pl.Utf8), - pl.col(offtarget_col).cast(pl.Utf8), - ).iter_rows() - if val is not None and val.strip() in wanted_trimmed - } - return frozenset(resolved) - - -def specificity_score(antigen_umi, control_umi): - """Cell Ranger BEAM specificity score, constants are Cell Ranger's: - (1 - betaCDF(0.925, antigenUMI + 1, controlUMI + 3)) * 100. - - Accepts scalars or numpy arrays. scipy's beta.cdf is vectorized, so the CLI passes whole columns - (the array path avoids a per-row Python loop); returns a numpy float or float array accordingly. - """ - return (1.0 - beta.cdf(0.925, antigen_umi + 1, control_umi + 3)) * 100.0 def combine_barcode_counts( @@ -150,8 +33,8 @@ def combine_barcode_counts( - ``"all"`` (AND): the feature is called ONLY when EVERY member barcode fired — each is present with ``umi >= min_umi`` in this cell — and its UMI is then the sum of the members. If any member is missing or below ``min_umi`` the feature is absent for this cell (omitted, not zero), so it does not - compete for dominance, take a fraction, or get a specificity score. This expresses the LIBRA-seq / - dual-probe design where a cell is antigen-specific only when both probe barcodes fire. + take a fraction of that cell's signal. This expresses the LIBRA-seq / dual-probe design where a cell + is antigen-specific only when both probe barcodes fire. ``barcode_umi`` holds only the barcodes with signal in this cell (mitool tag-stat emits count>0 rows). Off-panel barcodes (absent from ``barcode_to_feature``) are ignored, mirroring the inner join. @@ -189,48 +72,20 @@ def with_fraction(counts: pl.DataFrame) -> pl.DataFrame: ) -def with_specificity(frame: pl.DataFrame, control: str) -> pl.DataFrame: - """Add the per-(cell, feature) Cell Ranger specificity score vs the cell's control UMIs (0 when the - cell has no control reads). scipy beta.cdf is evaluated once over the whole column (no per-row loop). - An empty join carries the schema through. Computed once in main() and reused for both the exported - specificity CSV and the per-cell summary's max, so the two never diverge or recompute the betaCDF. - - The control itself is the reference, not a scored antigen: its own row's score is nulled, so the - control never appears as a scored feature in the exported specificity CSV (main() drops null scores) - and never drives the per-cell maxSpecificityScore (a max skips nulls).""" - ctrl = frame.filter(pl.col("feature") == control).select(["cellId", pl.col("umiCount").alias("_controlUmi")]) - joined = frame.join(ctrl, on="cellId", how="left").with_columns(pl.col("_controlUmi").fill_null(0)) - scores = specificity_score(joined["umiCount"].to_numpy(), joined["_controlUmi"].to_numpy()) - return ( - joined.with_columns(pl.Series("specificityScore", scores, dtype=pl.Float64)) - .with_columns( - pl.when(pl.col("feature") == control) - .then(pl.lit(None, dtype=pl.Float64)) - .otherwise(pl.col("specificityScore")) - .alias("specificityScore") - ) - .drop("_controlUmi") - ) - - def per_cell_summary(per_cell: pl.DataFrame) -> pl.DataFrame: - """One row per (sampleId, cellId): the cell's max feature UMI count and max feature fraction - (and, when a ``specificityScore`` column is present, the max specificity score), plus a + """One row per (sampleId, cellId): the cell's max feature UMI count and max feature fraction, plus a ``featureSummary`` string that lists every feature the cell has signal for as - ``feature (fraction%, umiCount UMI)``, bullet-separated and sorted by descending fraction (dominant - feature first, feature name as tie-break). Fractions display as whole percents, with "<1%" for a + ``feature (fraction%, umiCount UMI)``, bullet-separated and sorted by descending fraction (largest + share first, feature name as tie-break). Fractions display as whole percents, with "<1%" for a nonzero feature that rounds below 1%. - This is a TABLE-ONLY collapse of the (cell x feature) matrix -- the per-feature abundance, - fractions, consensus, and specificity outputs (the per-cell export contract) are unaffected. - ``per_cell`` is the (sampleId, cellId, feature, umiCount) long frame ALREADY carrying the - ``fraction`` column (and ``specificityScore`` when a negative control is set) that main() computed - once for the exported CSVs -- so the per-cell maxima can never diverge from the exported columns, - and the fraction window / betaCDF are not recomputed here. An empty frame carries its schema through - to a header-only summary. + This is a TABLE-ONLY collapse of the (cell x feature) matrix -- the per-feature abundance and + fractions outputs (the per-cell export contract) are unaffected. ``per_cell`` is the (sampleId, + cellId, feature, umiCount) long frame ALREADY carrying the ``fraction`` column that main() computed + once for the exported CSVs -- so the per-cell maxima can never diverge from the exported columns, and + the fraction window is not recomputed here. An empty frame carries its schema through to a + header-only summary. """ - has_control = "specificityScore" in per_cell.columns - # Whole-percent display of the fraction, with "<1%" for a nonzero feature that rounds below 1% (so a # real-but-tiny signal never reads as "0%"). Full-precision fractions stay in the exported columns. pct = (pl.col("fraction") * 100).round(0) @@ -252,15 +107,11 @@ def per_cell_summary(per_cell: pl.DataFrame) -> pl.DataFrame: pl.col("fraction").max().alias("maxFraction"), pl.col("_entry") .sort_by(["fraction", "feature"], descending=[True, False]) - # comma-separated, dominant feature first. + # comma-separated, largest share first. .str.join(", ") .alias("featureSummary"), ] - out_cols = ["sampleId", "cellId", "maxUmiCount", "maxFraction"] - if has_control: - aggs.append(pl.col("specificityScore").max().alias("maxSpecificityScore")) - out_cols.append("maxSpecificityScore") - out_cols.append("featureSummary") + out_cols = ["sampleId", "cellId", "maxUmiCount", "maxFraction", "featureSummary"] return per_cell.group_by(["sampleId", "cellId"]).agg(aggs).select(out_cols).sort(["sampleId", "cellId"]) @@ -430,35 +281,9 @@ def main() -> None: help="minimum per-barcode distinct-UMI count for a barcode to count as 'fired' under the 'all' " "(AND) combine mode (default 1)", ) - p.add_argument("--dominance-threshold", type=float, default=0.6) - p.add_argument("--control", default=None, help="negative-control feature name") - p.add_argument( - "--offtarget-col", - default=None, - help="CSV property column (e.g. antigen_class) designating on/off-target; features whose value " - "is in --offtarget-values are excluded from the dominant call (like the control) and enable the " - "cross-reactive label", - ) - p.add_argument( - "--offtarget-values", - default=None, - help="comma-separated values of --offtarget-col that mark a feature as off-target " - "(e.g. 'Off-Target,Off-target')", - ) p.add_argument("--output-prefix", default="result") args = p.parse_args() - # Resolve the off-target feature set from the designated property column + values. Both flags must be - # given together; features carrying an off-target value are excluded from the dominant call (as the - # control is) and turn on the cross-reactive label. Absent -> unchanged behaviour (empty set). - offtargets: frozenset[str] = frozenset() - if (args.offtarget_col is None) != (args.offtarget_values is None): - raise SystemExit("--offtarget-col and --offtarget-values must be given together") - if args.offtarget_col is not None: - wanted = frozenset(v.strip() for v in args.offtarget_values.split(",") if v.strip()) - offtargets = offtarget_features(args.tag_feature_csv, args.csv_feature_col, args.offtarget_col, wanted) - label_crossreactive = len(offtargets) > 0 - # Guard the user-mapped CSV column names: the two roles must be distinct, and neither may # collide with a tag-stat column. On the inner join, every tag-stat column is carried into the # joined frame -- so a --csv-feature-col that names ANY tag-stat column (e.g. `count`, @@ -514,82 +339,10 @@ def main() -> None: f"{args.output_prefix}_fractions.csv" ) - # consensus feature per cell (dominant-category rule), vectorized in polars: the - # dominant feature is the unique per-cell max whose share of the cell's total is >= the threshold - # (clamped to the 0.5 floor); otherwise "ambiguous". No-signal cells never occur here (tag-stat - # counts are all > 0), so None is never produced. Mirrors consensus_category, which the tests pin - # (and an oracle test cross-checks this vectorized path against it). - threshold = max(args.dominance_threshold, DOMINANCE_FLOOR) - # The negative control and the off-target features are references, not callable antigens: exclude - # them from the winner candidates so a control/off-target-dominated cell is "ambiguous", never the - # control or an off-target. Their UMIs stay in `_total` (the denominator, computed from the full - # `counts`), so their signal suppresses dominance rather than being renormalised away. When off- - # targets are designated, a cell whose on-target signal collectively passes the threshold but is - # spread across >= 2 on-targets is "cross-reactive". Mirrors consensus_category(control=..., off - # targets=..., label_crossreactive=...), which the oracle test pins the vectorized path against. - excluded = list(offtargets) + ([args.control] if args.control is not None else []) - antigens = counts if not excluded else counts.filter(~pl.col("feature").is_in(excluded)) - totals = counts.group_by(["sampleId", "cellId"]).agg(pl.col("umiCount").sum().alias("_total")) - tops = antigens.group_by(["sampleId", "cellId"]).agg( - pl.col("umiCount").max().alias("_max"), - (pl.col("umiCount") == pl.col("umiCount").max()).sum().alias("_nAtMax"), - pl.col("feature").sort_by("umiCount", descending=True).first().alias("_top"), - # on-target signal: sum + distinct on-target features present (for the cross-reactive branch) - pl.col("umiCount").sum().alias("_onTotal"), - pl.col("feature").n_unique().alias("_nOn"), - ) - ( - totals.join(tops, on=["sampleId", "cellId"], how="left") - .with_columns( - # _top is null for a cell whose only signal is control/off-target -> ambiguous. - pl.when( - pl.col("_top").is_not_null() - & (pl.col("_nAtMax") == 1) - & (pl.col("_max") / pl.col("_total") >= threshold) - ) - .then(pl.col("_top")) - .when( - # cross-reactive: on-target signal collectively dominates but is split across >= 2 on-targets - pl.lit(label_crossreactive) - & (pl.col("_nOn") >= 2) - & (pl.col("_onTotal") / pl.col("_total") >= threshold) - ) - .then(pl.lit(CROSS_REACTIVE)) - .otherwise(pl.lit("ambiguous")) - .alias("consensusFeature") - ) - .select(["sampleId", "cellId", "consensusFeature"]) - .sort(["sampleId", "cellId"]) - .write_csv(f"{args.output_prefix}_consensus.csv") - ) - - # optional specificity score per (cell, feature) vs the negative control. Computed - # once (with_specificity: scipy beta.cdf vectorized over the whole column) and reused for the - # per-cell summary's max. An empty join carries the schema through natively -> header-only CSV. - if args.control is not None: - summary_frame = with_specificity(cf, args.control) - ( - # The control's own row carries a null score (it is the reference, not a scored antigen) -- - # drop those so the exported specificity is antigen-only. summary_frame KEEPS the control row - # (with a null score) so the per-cell summary's umi/fraction breakdown still shows it. - summary_frame.filter(pl.col("specificityScore").is_not_null()) - .select(["sampleId", "cellId", "feature", "specificityScore"]) - .sort(["sampleId", "cellId", "feature"]) - .write_csv(f"{args.output_prefix}_specificity.csv") - ) - else: - # No negative control: still emit an (empty, header-only) specificity CSV so the workflow's - # fixed output set is satisfied. It is not imported when no control is set (main.tpl and the - # model gate the specificity column on hasControl). - pl.DataFrame(schema=_SPECIFICITY_SCHEMA).write_csv(f"{args.output_prefix}_specificity.csv") - summary_frame = cf - # per-cell summary (table-only collapse): one row per (sampleId, cellId) with the max feature UMI - # count / fraction (/ specificity, with a control) and the "feature (fraction%, umi) | ..." string. - # summary_frame already carries fraction (+ specificityScore with a control), so nothing is - # recomputed. The maxSpecificityScore column is present only with a control, matching how main.tpl / - # the model gate the specificity import on hasControl. - per_cell_summary(summary_frame).write_csv(f"{args.output_prefix}_per_cell_summary.csv") + # count / fraction and the "feature (fraction%, umi) | ..." string. cf already carries fraction, so + # nothing is recomputed. + per_cell_summary(cf).write_csv(f"{args.output_prefix}_per_cell_summary.csv") if __name__ == "__main__": diff --git a/software/per-cell-metrics/test/test_per_cell_metrics.py b/software/per-cell-metrics/test/test_per_cell_metrics.py index 201d6e6..104dc31 100644 --- a/software/per-cell-metrics/test/test_per_cell_metrics.py +++ b/software/per-cell-metrics/test/test_per_cell_metrics.py @@ -15,254 +15,13 @@ from hypothesis import given from hypothesis import strategies as st from per_cell_metrics import ( - CROSS_REACTIVE, _load, combine_barcode_counts, - consensus_category, - offtarget_features, - specificity_score, ) SRC = pathlib.Path(__file__).parents[1] / "src" / "per_cell_metrics.py" -# --- dominant-category rule (spec A-0012) --- - - -def test_consensus_single_winner_above_threshold(): - # 7 of 10 -> 0.7 >= 0.6 default -> that feature - assert consensus_category({"A": 7, "B": 2, "C": 1}, 0.6) == "A" - - -def test_consensus_winner_exactly_at_threshold(): - assert consensus_category({"A": 6, "B": 4}, 0.6) == "A" - - -def test_consensus_no_winner_is_ambiguous(): - # max share 0.4 < 0.6 -> ambiguous (signal present, none passes) - assert consensus_category({"A": 4, "B": 3, "C": 3}, 0.6) == "ambiguous" - - -def test_consensus_exact_half_split_at_floor_is_ambiguous(): - # 50/50 at the 0.5 floor -> tie -> ambiguous (A-0012: "an exact split at the 0.5 floor") - assert consensus_category({"A": 5, "B": 5}, 0.5) == "ambiguous" - - -def test_consensus_threshold_clamped_to_floor(): - # request 0.4 but floor is 0.5; 0.55 share passes 0.5, unique -> winner - assert consensus_category({"A": 11, "B": 9}, 0.4) == "A" - - -def test_consensus_single_category(): - assert consensus_category({"A": 3}, 0.6) == "A" - - -def test_consensus_no_signal_is_none(): - assert consensus_category({"A": 0, "B": 0}, 0.6) is None - assert consensus_category({}, 0.6) is None - - -# --- negative control is a reference, not a callable antigen (spec A-0014) --- - - -def test_consensus_excludes_control_from_candidates(): - # The control must not win consensus even when it has the most UMIs: AGX 3 / CTRL 5 -> the top - # antigen (AGX) share is 3/8 = 0.375 < 0.6 -> ambiguous, NOT "CTRL". - assert consensus_category({"AGX": 3, "CTRL": 5}, 0.6, control="CTRL") == "ambiguous" - - -def test_consensus_control_stays_in_denominator(): - # Control UMIs remain in the denominator, so control signal suppresses (not inflates) dominance. - # AGX 7 / CTRL 2 -> 7/9 = 0.78 >= 0.6 -> AGX (control did not spuriously push it under threshold). - assert consensus_category({"AGX": 7, "CTRL": 2}, 0.6, control="CTRL") == "AGX" - # Were the control dropped from the denominator, AGX 3 / CTRL 5 would renormalise to 1.0 and wrongly - # win; keeping it in the denominator makes the control-swamped cell correctly ambiguous. - assert consensus_category({"AGX": 3, "CTRL": 5}, 0.6, control="CTRL") == "ambiguous" - - -def test_consensus_control_only_is_ambiguous(): - # A cell whose only signal is the control has no antigen candidate -> ambiguous, never the control. - assert consensus_category({"CTRL": 5}, 0.6, control="CTRL") == "ambiguous" - - -def test_consensus_no_control_arg_is_unchanged(): - # control=None (no negative control set) keeps the original rule: every feature is a candidate. - assert consensus_category({"AGX": 3, "OTHER": 5}, 0.6) == "OTHER" - - -# --- off-target-aware dominant call + cross-reactive label (spec A-0014 Type-aware direction, F2) --- - - -def test_consensus_excludes_offtargets_like_control(): - # An off-target feature is excluded from the winners exactly as the control is: OT swamps the cell, - # the single on-target's share of the total is 3/8 = 0.375 < 0.6 -> ambiguous, never "OT". - assert consensus_category({"AGX": 3, "OT": 5}, 0.6, offtargets=frozenset({"OT"})) == "ambiguous" - - -def test_consensus_offtargets_stay_in_denominator(): - # Off-target UMIs remain in the denominator (suppress, not inflate): AGX 7 / OT 2 -> 7/9 >= 0.6 -> AGX. - assert consensus_category({"AGX": 7, "OT": 2}, 0.6, offtargets=frozenset({"OT"})) == "AGX" - - -def test_consensus_crossreactive_two_ontargets_pass_together(): - # Two on-targets (same target's human+cyno) split ~50/50 with only minor off-target signal: neither - # passes alone, but the on-target set is 90% of the total across 2 features -> cross-reactive, not - # ambiguous. This is the binder F2 rescues from the overloaded "ambiguous" bucket. - assert ( - consensus_category( - {"TgtA_human": 45, "TgtA_cyno": 45, "OT": 10}, - 0.6, - offtargets=frozenset({"OT"}), - label_crossreactive=True, - ) - == CROSS_REACTIVE - ) - - -def test_consensus_crossreactive_needs_label_flag(): - # Without the label flag the same split stays "ambiguous" (backward-compatible when the feature is off). - assert ( - consensus_category({"TgtA_human": 45, "TgtA_cyno": 45, "OT": 10}, 0.6, offtargets=frozenset({"OT"})) - == "ambiguous" - ) - - -def test_consensus_offtarget_swamped_is_ambiguous_not_crossreactive(): - # On-target set collectively below threshold (off-target-dominated) -> ambiguous, never cross-reactive: - # TgtA 20 + TgtB 20 = 40 of 100 (0.4 < 0.6); OT 60 swamps. - assert ( - consensus_category( - {"TgtA": 20, "TgtB": 20, "OT": 60}, - 0.6, - offtargets=frozenset({"OT"}), - label_crossreactive=True, - ) - == "ambiguous" - ) - - -def test_consensus_crossreactive_single_ontarget_still_calls_feature(): - # A single dominant on-target still wins outright (not cross-reactive): AGX 80 / OT 20 -> AGX. - assert ( - consensus_category({"AGX": 80, "OT": 20}, 0.6, offtargets=frozenset({"OT"}), label_crossreactive=True) == "AGX" - ) - - -def test_consensus_only_offtarget_signal_is_ambiguous(): - # A cell whose only signal is off-target has no on-target candidate -> ambiguous. - assert consensus_category({"OT": 5}, 0.6, offtargets=frozenset({"OT"}), label_crossreactive=True) == "ambiguous" - - -def test_offtarget_features_resolves_from_property_column(tmp_path): - # The off-target feature set is resolved from a designated property column + its off-target values. - csv = tmp_path / "tags.csv" - csv.write_text( - "tag,feature,antigen_class\n" - "b1,TgtA,Target\n" - "b2,TgtB,Target\n" - "b3,DecoyX,Decoy\n" - "b4,OTx, Off-Target \n" # whitespace tolerated (stripped) - ) - got = offtarget_features(str(csv), "feature", "antigen_class", frozenset({"Off-Target", "Decoy"})) - assert got == frozenset({"DecoyX", "OTx"}) - - -def test_offtarget_features_bad_column_exits(tmp_path): - csv = tmp_path / "tags.csv" - csv.write_text("tag,feature\nb1,TgtA\n") - with pytest.raises(SystemExit): - offtarget_features(str(csv), "feature", "nope", frozenset({"Off-Target"})) - - -def test_offtarget_features_matching_is_case_sensitive(tmp_path): - # Matching is whitespace-trimmed but CASE-SENSITIVE: selecting "Off-Target" catches only that exact - # value (surrounding whitespace tolerated), NOT "Off-target". Real B043 panels carry both casings in - # one Type column; the user selects every casing they mean (each is offered separately in the - # dropdown). The block never silently broadens a selection to unselected casings. Names stay verbatim. - csv = tmp_path / "tags.csv" - csv.write_text( - "tag,feature,Type\n" - "b1,AgExact,Off-Target\n" # exact match - "b2,AgSpaced, Off-Target \n" # surrounding whitespace -> trimmed, still matches - "b3,AgLower,Off-target\n" # lower 't' — a DIFFERENT value, not selected - "b4,AgOn,Target\n" - ) - # Selecting only "Off-Target": the exact and whitespace-padded rows match; the lowercase one does not. - assert offtarget_features(str(csv), "feature", "Type", frozenset({"Off-Target"})) == frozenset( - {"AgExact", "AgSpaced"} - ) - # Selecting BOTH casings explicitly catches the lowercase feature too — the user opts in. - assert offtarget_features(str(csv), "feature", "Type", frozenset({"Off-Target", "Off-target"})) == frozenset( - {"AgExact", "AgSpaced", "AgLower"} - ) - - -# --- specificity score (spec A-0014, Cell Ranger betaCDF) --- - - -def test_specificity_strong_signal_high_score(): - # many antigen UMIs, no control -> high confidence (the betaCDF formula gives ~98.6 here) - s = specificity_score(antigen_umi=100, control_umi=0) - assert s > 95.0 - - -def test_specificity_no_signal_low_score(): - # no antigen reads, control present -> low confidence - s = specificity_score(antigen_umi=0, control_umi=20) - assert 0.0 <= s < 5.0 - - -def test_specificity_formula_exact(): - # Reference-oracle guard against a constant typo. Weak on its own (mirrors the impl via the same - # scipy call); the bounds + monotonicity PROPERTIES below are the real behavioral guards. - from scipy.stats import beta - - a, c = 7, 3 - expected = (1.0 - float(beta.cdf(0.925, a + 1, c + 3))) * 100.0 - assert specificity_score(a, c) == pytest.approx(expected) - - -# --- properties (invariants that hold for ALL valid inputs) --- - - -@given( - st.dictionaries(st.text(min_size=1), st.integers(min_value=0, max_value=1000), max_size=8), - st.floats(min_value=0.5, max_value=1.0), -) -def test_consensus_result_in_domain(counts, threshold): - # The result is always a key present in counts, "ambiguous", or None -- never an arbitrary string. - r = consensus_category(counts, threshold) - assert r is None or r == "ambiguous" or r in counts - - -@given( - st.dictionaries(st.text(min_size=1), st.integers(min_value=0, max_value=1000), max_size=8), - st.floats(min_value=0.5, max_value=1.0), - st.sets(st.text(min_size=1), max_size=4), -) -def test_consensus_offtarget_result_in_domain(counts, threshold, offtargets): - # With off-targets + the label on, the result is an on-target key, "cross-reactive", "ambiguous", or - # None -- and never an off-target/control key (they can never win). - r = consensus_category(counts, threshold, offtargets=frozenset(offtargets), label_crossreactive=True) - assert r is None or r in ("ambiguous", CROSS_REACTIVE) or (r in counts and r not in offtargets) - - -@given(st.integers(min_value=0, max_value=10_000), st.integers(min_value=0, max_value=10_000)) -def test_specificity_bounded_0_100(antigen, control): - # It is a confidence percentage: always within [0, 100]. - assert 0.0 <= specificity_score(antigen, control) <= 100.0 - - -@given( - st.integers(min_value=0, max_value=500), # control - st.integers(min_value=0, max_value=500), # base antigen - st.integers(min_value=1, max_value=500), # delta -) -def test_specificity_monotonic_in_antigen(control, base, delta): - # More antigen UMIs (same control) never lowers confidence. - assert specificity_score(base + delta, control) >= specificity_score(base, control) - - # --- multi-barcode antigen combine modes: sum (OR) / all (AND) --- # A dual-barcode antigen (BG505 read out by b1 + b2) alongside a single-barcode antigen (OTHER = cx). @@ -281,7 +40,7 @@ def test_combine_all_both_fire_emits_summed(): def test_combine_all_one_missing_omits_feature(): # Only one BG505 barcode fired -> under AND the antigen is NOT called; the cell has no BG505 entry - # at all (omitted, not zero), so it never competes for dominance or takes a fraction. + # at all (omitted, not zero), so it never takes a fraction of that cell's signal. assert combine_barcode_counts({"b1": 5}, _B2F, _FB, {"BG505": "all"}) == {} @@ -398,34 +157,10 @@ def test_cli_writes_outputs(tagstat_tsv, tags_csv, tmp_path): check=True, cwd=tmp_path, ) - for name in ["result_abundance.csv", "result_fractions.csv", "result_consensus.csv"]: + for name in ["result_abundance.csv", "result_fractions.csv", "result_per_cell_summary.csv"]: assert (tmp_path / name).exists(), f"missing {name}" -@pytest.mark.slow -def test_cli_consensus_golden(tagstat_tsv, tags_csv, tmp_path): - # End-to-end over the committed bed: cell1 dominant on AGX, cell2 ambiguous, cell3 single-feature. - subprocess.run( - [ - sys.executable, - str(SRC), - str(tagstat_tsv), - str(tags_csv), - "--sample-id", - "s1", - "--output-prefix", - str(tmp_path / "result"), - ], - check=True, - cwd=tmp_path, - ) - with open(tmp_path / "result_consensus.csv", newline="") as f: - by_cell = {row["cellId"]: row["consensusFeature"] for row in csv.DictReader(f)} - assert by_cell["cell1"] == "AGX" - assert by_cell["cell2"] == "ambiguous" - assert by_cell["cell3"] == "AGX" - - @pytest.mark.slow def test_cli_abundance_uses_unique_umi(tagstat_tsv, tags_csv, tmp_path): # DP-2: the matrix must use mitool's deduplicated `unique_UMI` (cell1/AGX = 3 distinct UMIs), @@ -512,27 +247,6 @@ def test_cli_rejects_colliding_feature_col(tagstat_tsv, tmp_path): assert r.returncode != 0 -@pytest.mark.slow -def test_cli_with_control_writes_specificity(tagstat_tsv, tags_csv, tmp_path): - subprocess.run( - [ - sys.executable, - str(SRC), - str(tagstat_tsv), - str(tags_csv), - "--sample-id", - "s1", - "--control", - "CTRL", - "--output-prefix", - str(tmp_path / "result"), - ], - check=True, - cwd=tmp_path, - ) - assert (tmp_path / "result_specificity.csv").exists() - - @pytest.mark.slow @pytest.mark.parametrize( "tagstat_body", @@ -544,9 +258,9 @@ def test_cli_with_control_writes_specificity(tagstat_tsv, tags_csv, tmp_path): ) def test_cli_empty_join_writes_header_only_not_crash(tags_csv, tmp_path, tagstat_body): # Regression: when no (cell, feature) pair survives the tag->feature join -- a wrong read geometry, - # or a sample with no on-panel reads -- the run must still emit all four CSVs header-only, never - # crash. --control exercises the specificity write too. (consensus and specificity are pure-polars - # transforms that carry their schema through the empty case; this guards that they stay header-only.) + # or a sample with no on-panel reads -- the run must still emit both CSVs header-only, never crash. + # (abundance and fractions are pure-polars transforms that carry their schema through the empty case; + # this guards that they stay header-only.) tagstat = tmp_path / "tagstat.tsv" tagstat.write_text("CELL\tFEATURE\tcount\ttotalWeight\tunique_UMI\n" + tagstat_body) @@ -558,8 +272,6 @@ def test_cli_empty_join_writes_header_only_not_crash(tags_csv, tmp_path, tagstat str(tags_csv), "--sample-id", "s1", - "--control", - "CTRL", "--output-prefix", str(tmp_path / "result"), ], @@ -569,8 +281,6 @@ def test_cli_empty_join_writes_header_only_not_crash(tags_csv, tmp_path, tagstat for name, header in [ ("result_abundance.csv", ["sampleId", "cellId", "feature", "umiCount"]), ("result_fractions.csv", ["sampleId", "cellId", "feature", "fraction"]), - ("result_consensus.csv", ["sampleId", "cellId", "consensusFeature"]), - ("result_specificity.csv", ["sampleId", "cellId", "feature", "specificityScore"]), ]: p = tmp_path / name assert p.exists(), f"missing {name}" @@ -582,9 +292,9 @@ def test_cli_empty_join_writes_header_only_not_crash(tags_csv, tmp_path, tagstat @pytest.mark.slow def test_cli_per_cell_summary_maxima_match_exported_columns(tagstat_tsv, tags_csv, tmp_path): - # The per-cell summary's maxUmiCount / maxFraction / maxSpecificityScore are a collapse of the - # exported (cell x feature) columns -- they must equal the per-cell max of those exported CSVs, not a - # separately-recomputed value (guards the with_fraction / with_specificity single-compute refactor). + # The per-cell summary's maxUmiCount / maxFraction are a collapse of the exported (cell x feature) + # columns -- they must equal the per-cell max of those exported CSVs, not a separately-recomputed + # value (guards the with_fraction single-compute reuse). subprocess.run( [ sys.executable, @@ -593,8 +303,6 @@ def test_cli_per_cell_summary_maxima_match_exported_columns(tagstat_tsv, tags_cs str(tags_csv), "--sample-id", "s1", - "--control", - "CTRL", "--output-prefix", str(tmp_path / "result"), ], @@ -612,7 +320,6 @@ def _max_by_cell(path, value_col, cast): exp_umi = _max_by_cell(tmp_path / "result_abundance.csv", "umiCount", int) exp_frac = _max_by_cell(tmp_path / "result_fractions.csv", "fraction", float) - exp_spec = _max_by_cell(tmp_path / "result_specificity.csv", "specificityScore", float) with open(tmp_path / "result_per_cell_summary.csv", newline="") as f: summary = {r["cellId"]: r for r in csv.DictReader(f)} @@ -621,259 +328,6 @@ def _max_by_cell(path, value_col, cast): for cell, row in summary.items(): assert int(row["maxUmiCount"]) == exp_umi[cell] assert float(row["maxFraction"]) == pytest.approx(exp_frac[cell]) - assert float(row["maxSpecificityScore"]) == pytest.approx(exp_spec[cell]) - - -@pytest.mark.slow -def test_cli_consensus_matches_pure_rule(tmp_path): - # Oracle: the vectorized CLI consensus must equal the pure consensus_category rule across cases the - # committed golden bed doesn't cover (unique winner, exact tie, sub-threshold spread, single feature). - tags = tmp_path / "tags.csv" - tags.write_text("tag,feature\nAAAA,AGX\nCCCC,BGX\nGGGG,CGX\n") - tagstat = tmp_path / "tagstat.tsv" - tagstat.write_text( - "CELL\tFEATURE\tcount\ttotalWeight\tunique_UMI\n" - "cellW\tAAAA\t8\t8\t8\n" - "cellW\tCCCC\t1\t1\t1\n" - "cellW\tGGGG\t1\t1\t1\n" # AGX 8 / BGX 1 / CGX 1 -> unique winner AGX (0.8 >= 0.6) - "cellX\tAAAA\t5\t5\t5\n" - "cellX\tCCCC\t5\t5\t5\n" # AGX 5 / BGX 5 -> tie, 0.5 < 0.6 -> ambiguous - "cellY\tAAAA\t4\t4\t4\n" - "cellY\tCCCC\t3\t3\t3\n" - "cellY\tGGGG\t3\t3\t3\n" # max share 0.4 < 0.6 -> ambiguous - "cellZ\tAAAA\t6\t6\t6\n" # single feature -> AGX - ) - subprocess.run( - [ - sys.executable, - str(SRC), - str(tagstat), - str(tags), - "--sample-id", - "s1", - "--dominance-threshold", - "0.6", - "--output-prefix", - str(tmp_path / "result"), - ], - check=True, - cwd=tmp_path, - ) - with open(tmp_path / "result_consensus.csv", newline="") as f: - got = {r["cellId"]: r["consensusFeature"] for r in csv.DictReader(f)} - expected = { - "cellW": consensus_category({"AGX": 8, "BGX": 1, "CGX": 1}, 0.6), - "cellX": consensus_category({"AGX": 5, "BGX": 5}, 0.6), - "cellY": consensus_category({"AGX": 4, "BGX": 3, "CGX": 3}, 0.6), - "cellZ": consensus_category({"AGX": 6}, 0.6), - } - assert got == expected # vectorized CLI == the pure rule - # ...and the pure rule is what we think (guards against a vacuous match to a wrong rule) - assert expected == {"cellW": "AGX", "cellX": "ambiguous", "cellY": "ambiguous", "cellZ": "AGX"} - - -@pytest.mark.slow -def test_cli_specificity_matches_pure_score(tagstat_tsv, tags_csv, tmp_path): - # Oracle: the vectorized specificity column must equal the pure specificity_score per (cell, feature) - # vs the cell's control (CTRL) UMIs -- 0 when the cell has no control reads. Guards the array path - # (scipy beta.cdf over whole columns) against the scalar formula, including the fill_null(0) case. - subprocess.run( - [ - sys.executable, - str(SRC), - str(tagstat_tsv), - str(tags_csv), - "--sample-id", - "s1", - "--control", - "CTRL", - "--output-prefix", - str(tmp_path / "result"), - ], - check=True, - cwd=tmp_path, - ) - with open(tmp_path / "result_abundance.csv", newline="") as f: - umi = {(r["cellId"], r["feature"]): int(r["umiCount"]) for r in csv.DictReader(f)} - control_umi = {cell: umi.get((cell, "CTRL"), 0) for (cell, _feat) in umi} - with open(tmp_path / "result_specificity.csv", newline="") as f: - rows = list(csv.DictReader(f)) - assert rows # non-empty: the committed bed has cells and features - assert "CTRL" not in {r["feature"] for r in rows} # control is the reference, not a scored feature - for r in rows: - expected = specificity_score(umi[(r["cellId"], r["feature"])], control_umi[r["cellId"]]) - assert float(r["specificityScore"]) == pytest.approx(float(expected)) - - -@pytest.mark.slow -def test_cli_consensus_excludes_control(tmp_path): - # With --control set, the control is a reference and never a called antigen (spec A-0014): a - # control-dominated cell must be "ambiguous", not the control. Control UMIs stay in the denominator, - # so they suppress dominance rather than being renormalised away. - tags = tmp_path / "tags.csv" - tags.write_text("tag,feature\nAAAA,AGX\nGGGG,CTRL\n") - tagstat = tmp_path / "tagstat.tsv" - tagstat.write_text( - "CELL\tFEATURE\tcount\ttotalWeight\tunique_UMI\n" - "cellP\tAAAA\t3\t3\t3\n" - "cellP\tGGGG\t5\t5\t5\n" # AGX 3 / CTRL 5 -> top antigen 3/8 < 0.6 -> ambiguous (NOT CTRL) - "cellQ\tAAAA\t7\t7\t7\n" - "cellQ\tGGGG\t2\t2\t2\n" # AGX 7 / CTRL 2 -> 7/9 = 0.78 >= 0.6 -> AGX - "cellR\tGGGG\t5\t5\t5\n" # only control signal -> ambiguous - ) - subprocess.run( - [ - sys.executable, - str(SRC), - str(tagstat), - str(tags), - "--sample-id", - "s1", - "--control", - "CTRL", - "--dominance-threshold", - "0.6", - "--output-prefix", - str(tmp_path / "result"), - ], - check=True, - cwd=tmp_path, - ) - with open(tmp_path / "result_consensus.csv", newline="") as f: - got = {r["cellId"]: r["consensusFeature"] for r in csv.DictReader(f)} - assert got == {"cellP": "ambiguous", "cellQ": "AGX", "cellR": "ambiguous"} - # ...and the vectorized CLI agrees with the pure rule (guards against a vacuous match). - assert got == { - "cellP": consensus_category({"AGX": 3, "CTRL": 5}, 0.6, control="CTRL"), - "cellQ": consensus_category({"AGX": 7, "CTRL": 2}, 0.6, control="CTRL"), - "cellR": consensus_category({"CTRL": 5}, 0.6, control="CTRL"), - } - - -@pytest.mark.slow -def test_cli_consensus_offtarget_and_crossreactive(tmp_path): - # End-to-end: with an --offtarget-col/--offtarget-values designation the vectorized consensus must - # match the pure rule -- off-targets excluded from winners, and an on-target-split cell called - # cross-reactive. antigen_class is a per-feature property column of the tag CSV (A-0026 pass-through). - tags = tmp_path / "tags.csv" - tags.write_text("tag,feature,antigen_class\nAAAA,TgtA_human,Target\nCCCC,TgtA_cyno,Target\nGGGG,OTx,Off-Target\n") - tagstat = tmp_path / "tagstat.tsv" - tagstat.write_text( - "CELL\tFEATURE\tcount\ttotalWeight\tunique_UMI\n" - "cellX\tAAAA\t45\t45\t45\n" - "cellX\tCCCC\t45\t45\t45\n" - "cellX\tGGGG\t10\t10\t10\n" # human+cyno split 45/45, OT 10 -> cross-reactive - "cellY\tAAAA\t80\t80\t80\n" - "cellY\tGGGG\t20\t20\t20\n" # single on-target 80/100 -> TgtA_human - "cellZ\tAAAA\t20\t20\t20\n" - "cellZ\tCCCC\t20\t20\t20\n" - "cellZ\tGGGG\t60\t60\t60\n" # OT-swamped (on-target 40/100 < 0.6) -> ambiguous - "cellW\tGGGG\t7\t7\t7\n" # only off-target signal -> ambiguous - ) - subprocess.run( - [ - sys.executable, - str(SRC), - str(tagstat), - str(tags), - "--sample-id", - "s1", - "--dominance-threshold", - "0.6", - "--offtarget-col", - "antigen_class", - "--offtarget-values", - "Off-Target,Decoy", - "--output-prefix", - str(tmp_path / "result"), - ], - check=True, - cwd=tmp_path, - ) - with open(tmp_path / "result_consensus.csv", newline="") as f: - got = {r["cellId"]: r["consensusFeature"] for r in csv.DictReader(f)} - ot = frozenset({"OTx"}) - expected = { - "cellX": consensus_category( - {"TgtA_human": 45, "TgtA_cyno": 45, "OTx": 10}, 0.6, offtargets=ot, label_crossreactive=True - ), - "cellY": consensus_category({"TgtA_human": 80, "OTx": 20}, 0.6, offtargets=ot, label_crossreactive=True), - "cellZ": consensus_category( - {"TgtA_human": 20, "TgtA_cyno": 20, "OTx": 60}, 0.6, offtargets=ot, label_crossreactive=True - ), - "cellW": consensus_category({"OTx": 7}, 0.6, offtargets=ot, label_crossreactive=True), - } - assert got == expected # vectorized CLI == pure rule - # ...and the pure rule is what we intend (guards against a vacuous match). - assert expected == { - "cellX": CROSS_REACTIVE, - "cellY": "TgtA_human", - "cellZ": "ambiguous", - "cellW": "ambiguous", - } - - -@pytest.mark.slow -def test_cli_offtarget_flags_require_each_other(tmp_path): - # --offtarget-col without --offtarget-values (or vice versa) is a user error -> exit non-zero. - tags = tmp_path / "tags.csv" - tags.write_text("tag,feature,antigen_class\nAAAA,TgtA,Target\n") - tagstat = tmp_path / "tagstat.tsv" - tagstat.write_text("CELL\tFEATURE\tcount\ttotalWeight\tunique_UMI\ncX\tAAAA\t3\t3\t3\n") - r = subprocess.run( - [ - sys.executable, - str(SRC), - str(tagstat), - str(tags), - "--sample-id", - "s1", - "--offtarget-col", - "antigen_class", - "--output-prefix", - str(tmp_path / "result"), - ], - cwd=tmp_path, - ) - assert r.returncode != 0 - - -@pytest.mark.slow -def test_cli_control_not_scored_as_feature(tmp_path): - # The control is the specificity reference, not a scored antigen: it must not appear as a feature in - # the specificity output, and a control-heavy cell's maxSpecificityScore must be the real antigen's - # score vs the control, never the control's self-score. - tags = tmp_path / "tags.csv" - tags.write_text("tag,feature\nAAAA,AGX\nGGGG,CTRL\n") - tagstat = tmp_path / "tagstat.tsv" - tagstat.write_text( - "CELL\tFEATURE\tcount\ttotalWeight\tunique_UMI\n" - "cellP\tAAAA\t3\t3\t3\n" - "cellP\tGGGG\t9\t9\t9\n" # control-heavy cell - ) - subprocess.run( - [ - sys.executable, - str(SRC), - str(tagstat), - str(tags), - "--sample-id", - "s1", - "--control", - "CTRL", - "--output-prefix", - str(tmp_path / "result"), - ], - check=True, - cwd=tmp_path, - ) - with open(tmp_path / "result_specificity.csv", newline="") as f: - spec_features = {r["feature"] for r in csv.DictReader(f)} - assert spec_features == {"AGX"} # CTRL is not emitted as a scored feature - with open(tmp_path / "result_per_cell_summary.csv", newline="") as f: - summary = {r["cellId"]: r for r in csv.DictReader(f)} - # AGX 3 UMIs vs CTRL 9 UMIs — the antigen's score, not specificity_score(9, 9) (the control self-score) - assert float(summary["cellP"]["maxSpecificityScore"]) == pytest.approx(specificity_score(3, 9)) @pytest.mark.slow @@ -887,7 +341,7 @@ def test_cli_combine_all_gates_dual_barcode_antigen(tmp_path): tagstat.write_text( "CELL\tFEATURE\tcount\ttotalWeight\tunique_UMI\n" "cellBoth\tb1\t6\t6\t6\n" - "cellBoth\tb2\t6\t6\t6\n" # both BG505 barcodes fire -> BG505 called (12), dominant + "cellBoth\tb2\t6\t6\t6\n" # both BG505 barcodes fire -> BG505 called (12) "cellOne\tb1\t9\t9\t9\n" # only b1 fired -> BG505 NOT called "cellOne\tcx\t1\t1\t1\n" # OTHER present ) @@ -912,11 +366,6 @@ def test_cli_combine_all_gates_dual_barcode_antigen(tmp_path): assert umi[("cellBoth", "BG505")] == 12 # AND: both fired -> summed assert ("cellOne", "BG505") not in umi # AND: only one fired -> omitted entirely assert umi[("cellOne", "OTHER")] == 1 - # consensus follows: cellBoth is BG505; cellOne has only OTHER present -> OTHER - with open(tmp_path / "result_consensus.csv", newline="") as f: - cons = {r["cellId"]: r["consensusFeature"] for r in csv.DictReader(f)} - assert cons["cellBoth"] == "BG505" - assert cons["cellOne"] == "OTHER" @pytest.mark.slow From bfaccbff187b16b37146dc84e6bc65c9dbdb6580 Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 16:14:28 +0200 Subject: [PATCH 040/282] MILAB-6496: count a set's silent cells when it contradicts itself --- software/per-cell-metrics/src/combine.py | 115 +++++++++++++ .../per-cell-metrics/test/test_combine.py | 159 +++++++++++++++++- 2 files changed, 273 insertions(+), 1 deletion(-) diff --git a/software/per-cell-metrics/src/combine.py b/software/per-cell-metrics/src/combine.py index 5c494d5..578cde6 100644 --- a/software/per-cell-metrics/src/combine.py +++ b/software/per-cell-metrics/src/combine.py @@ -419,3 +419,118 @@ def set_counts(verdicts: pl.DataFrame) -> pl.DataFrame: ) .sort("setId") ) + + +def self_disagreement( + states: pl.DataFrame, + universe: set[str], + offered: dict[str, set[str]], + cells_by_set: dict[str, list[tuple[str, str]]], + admissibility: Admissibility, + level: str, +) -> pl.DataFrame: + """How often sets contradict themselves, at an identity or at a tag. + + A clonotype is one receptor and one receptor has one specificity, so + where two of a set's evaluable cells read differently at one position, at + least one reading is wrong. That makes this the cheapest quality signal + available: no threshold and no external reference, because the + contradiction comes from the data disagreeing with itself. + + A set's evaluable cells at a position are every admissible cell that + settled there, whether the reading is an explicit row in `states` or + silent. A silent admissible cell always resolves not bound (see + `specificity_score` in verdict.py), so it is as evaluable as an explicit + row and votes not bound; an inadmissible cell, silent or explicit, never + votes. The silent count comes from `silent_tally`, generalised to key its + tally by set rather than only by sample -- the same source + `combine_cells` draws on for the same fact, never recomputed here. + + Both levels are always carried by calling this twice, once per level. The + two answer different questions: a tag with a high rate is a reagent + misbehaving for everyone; an identity with a high rate is the answer a + scientist acts on being unstable. Where an identity carries one tag the + two coincide, and saying so beats dropping a row a reader would then hunt + for. + + The tag figure is diagnostic only. It rests on comparing each tag against + the reference separately, which no verdict is built from, so it is never + read as evidence about an answer. + + `states` carries a `key` column holding the identity or the tag according + to `level`, plus sampleId, cellId and state -- the same sparse shape + `combine_cells` takes as `states`, with `identity` renamed to `key` and + with no setId column: which set a cell belongs to comes only from + `cells_by_set`, never from a second column that could disagree with it. + `offered` and `universe` are at that same grain: the identities or the + tags each sample offers, and the full set of keys to report on, since a + key with every one of its cells silent has no explicit row anywhere in + `states` and cannot be recovered from it. + + Only a set's position with two or more evaluable cells contributes: a + singleton cannot disagree with itself. The rate is over sets evaluated, + not every set that exists, so a key nobody could evaluate reports a null + rate rather than a rate of zero, which would read as agreement. + """ + group_by_cell: dict[tuple[str, str], str] = {} + for set_id, members in cells_by_set.items(): + for cell_key in members: + owner = group_by_cell.get(cell_key) + assert owner is None or owner == set_id, ( + f"cell {cell_key!r} appears in both set {owner!r} and set {set_id!r} in cells_by_set: " + "a cell must belong to exactly one set" + ) + group_by_cell[cell_key] = set_id + + cells_frame = pl.DataFrame(list(group_by_cell), orient="row", schema={"sampleId": pl.String, "cellId": pl.String}) + observed_for_tally = states.select("sampleId", "cellId", pl.col("key").alias("identity")) + tally = silent_tally( + observed_for_tally, cells_frame, offered, admissibility, group_by_cell=group_by_cell, group_column="setId" + ) + + settled = states.filter(pl.col("state").is_in(SETTLED)) + explicit_counts: dict[tuple[str, str], dict[str, int]] = {} + for sample_id, cell_id, key, state in zip( + settled["sampleId"].to_list(), + settled["cellId"].to_list(), + settled["key"].to_list(), + settled["state"].to_list(), + strict=True, + ): + set_id = group_by_cell.get((sample_id, cell_id)) + if set_id is None: + # Same drop `combine_cells` applies: a vote is never counted for + # a cell that no set's membership list names. + continue + bucket = explicit_counts.setdefault((set_id, key), {}) + bucket[state] = bucket.get(state, 0) + 1 + + sets_evaluated: dict[str, int] = {} + sets_disagreeing: dict[str, int] = {} + for row in tally.iter_rows(named=True): + set_id, key = row["setId"], row["identity"] + counts = dict(explicit_counts.get((set_id, key), {})) + counts[State.NOT_BOUND.value] = counts.get(State.NOT_BOUND.value, 0) + row["silentNotBound"] + evaluable = sum(counts.values()) + if evaluable < 2: + continue + sets_evaluated[key] = sets_evaluated.get(key, 0) + 1 + if sum(1 for n in counts.values() if n > 0) > 1: + sets_disagreeing[key] = sets_disagreeing.get(key, 0) + 1 + + return ( + pl.DataFrame({"key": sorted(universe)}) + .with_columns( + pl.col("key").replace_strict(sets_evaluated, default=0, return_dtype=pl.Int64).alias("setsEvaluated"), + pl.col("key").replace_strict(sets_disagreeing, default=0, return_dtype=pl.Int64).alias("setsDisagreeing"), + ) + .with_columns( + pl.when(pl.col("setsEvaluated") > 0) + .then(pl.col("setsDisagreeing") / pl.col("setsEvaluated")) + .otherwise(None) + .alias("disagreementRate"), + pl.lit(level).alias("level"), + pl.lit("true" if level == "tag" else "false").alias("diagnosticOnly"), + ) + .sort("key") + ) diff --git a/software/per-cell-metrics/test/test_combine.py b/software/per-cell-metrics/test/test_combine.py index 07ab4d9..6195b08 100644 --- a/software/per-cell-metrics/test/test_combine.py +++ b/software/per-cell-metrics/test/test_combine.py @@ -2,7 +2,14 @@ import polars as pl import pytest -from combine import DEFAULT_MIN_VOTERS, SetUnreliableReason, attach_competitor_notes, combine_cells, set_counts +from combine import ( + DEFAULT_MIN_VOTERS, + SetUnreliableReason, + attach_competitor_notes, + combine_cells, + self_disagreement, + set_counts, +) from verdict import Admissibility, State, combine_tags_to_identities, gate_cells, read_states B, N, U, NA = (State.BOUND.value, State.NOT_BOUND.value, State.UNRELIABLE.value, State.NEVER_ASKED.value) @@ -501,3 +508,153 @@ def test_output_row_order_is_deterministic_regardless_of_input_row_order(): rng.shuffle(shuffled) out = set_counts(_v(shuffled)) assert out.equals(baseline) + + +# self_disagreement's states frame is keyed by `key` (an identity or a tag, +# according to `level`), never by `identity`: the same sparse per-cell shape +# `combine_cells` reads, minus a setId column -- set membership comes only +# from `cells_by_set`, matching that function's own rule. +_KEY_STATES_SCHEMA = {"sampleId": pl.String, "cellId": pl.String, "key": pl.String, "state": pl.String} + + +def _key_states(rows): + return pl.DataFrame(rows, orient="row", schema=_KEY_STATES_SCHEMA) + + +def _row_for_key(out, key): + return out.filter(pl.col("key") == key).row(0, named=True) + + +def test_a_set_whose_cells_agree_does_not_disagree(): + states = _key_states([("S1", "c1", "A", B), ("S1", "c2", "A", B)]) + cells_by_set = {"s1": [("S1", "c1"), ("S1", "c2")]} + out = self_disagreement(states, {"A"}, {"S1": {"A"}}, cells_by_set, _NEUTRAL, level="identity") + r = _row_for_key(out, "A") + assert r["setsEvaluated"] == 1 + assert r["disagreementRate"] == 0.0 + + +def test_a_set_whose_cells_differ_disagrees(): + states = _key_states([("S1", "c1", "A", B), ("S1", "c2", "A", N)]) + cells_by_set = {"s1": [("S1", "c1"), ("S1", "c2")]} + out = self_disagreement(states, {"A"}, {"S1": {"A"}}, cells_by_set, _NEUTRAL, level="identity") + assert _row_for_key(out, "A")["disagreementRate"] == 1.0 + + +def test_singletons_do_not_contribute(): + states = _key_states([("S1", "c1", "A", B)]) + cells_by_set = {"s1": [("S1", "c1")]} + out = self_disagreement(states, {"A"}, {"S1": {"A"}}, cells_by_set, _NEUTRAL, level="identity") + assert _row_for_key(out, "A")["setsEvaluated"] == 0 + + +def test_unsettled_cells_are_not_evaluable(): + # c2's row is UNRELIABLE, not silent: it has an explicit row and so is + # not asked through `silent_tally`, but UNRELIABLE never counts as a + # settled vote either. One evaluable cell remains -- a singleton -- so + # the position does not contribute. + states = _key_states([("S1", "c1", "A", B), ("S1", "c2", "A", U)]) + cells_by_set = {"s1": [("S1", "c1"), ("S1", "c2")]} + out = self_disagreement(states, {"A"}, {"S1": {"A"}}, cells_by_set, _NEUTRAL, level="identity") + assert _row_for_key(out, "A")["setsEvaluated"] == 0 + + +def test_both_levels_are_carried_even_when_they_coincide(): + states = _key_states([("S1", "c1", "AAAA", B), ("S1", "c2", "AAAA", N)]) + cells_by_set = {"s1": [("S1", "c1"), ("S1", "c2")]} + universe, offered = {"AAAA"}, {"S1": {"AAAA"}} + ident = self_disagreement(states, universe, offered, cells_by_set, _NEUTRAL, level="identity") + tag = self_disagreement(states, universe, offered, cells_by_set, _NEUTRAL, level="tag") + assert ident.height == 1 and tag.height == 1 + assert tag.row(0, named=True)["level"] == "tag" + assert ident.row(0, named=True)["level"] == "identity" + + +def test_tag_level_is_marked_diagnostic_only(): + states = _key_states([("S1", "c1", "AAAA", B)]) + cells_by_set = {"s1": [("S1", "c1")]} + universe, offered = {"AAAA"}, {"S1": {"AAAA"}} + tag = self_disagreement(states, universe, offered, cells_by_set, _NEUTRAL, level="tag") + ident = self_disagreement(states, universe, offered, cells_by_set, _NEUTRAL, level="identity") + assert tag.row(0, named=True)["diagnosticOnly"] == "true" + assert ident.row(0, named=True)["diagnosticOnly"] == "false" + + +def test_rate_is_over_sets_evaluated_not_all_sets(): + # s2 is a singleton at A and is not evaluable. + states = _key_states([("S1", "c1", "A", B), ("S1", "c2", "A", N), ("S1", "c3", "A", B)]) + cells_by_set = {"s1": [("S1", "c1"), ("S1", "c2")], "s2": [("S1", "c3")]} + out = self_disagreement(states, {"A"}, {"S1": {"A"}}, cells_by_set, _NEUTRAL, level="identity") + r = _row_for_key(out, "A") + assert r["setsEvaluated"] == 1 and r["disagreementRate"] == 1.0 + + +def test_silent_cells_flip_agreement_into_disagreement(): + # THE defect this generalisation exists to fix: a set with 2 observed + # bound cells and 38 silent, admissible not-bound cells. Counting rows on + # the sparse frame sees only the 2 bound rows and calls this agreement; + # the 38 silent cells are settled not-bound votes and the set actually + # disagrees as badly as it is possible to. + members = [("S1", "c0"), ("S1", "c1")] + [("S1", f"s{i}") for i in range(38)] + states = _key_states([("S1", "c0", "A", B), ("S1", "c1", "A", B)]) + cells_by_set = {"s1": members} + admissibility = Admissibility({k: 5 for k in members}, 2, set()) + out = self_disagreement(states, {"A"}, {"S1": {"A"}}, cells_by_set, admissibility, level="identity") + r = _row_for_key(out, "A") + assert r["setsEvaluated"] == 1 + assert r["disagreementRate"] == 1.0 + + +def test_one_observed_positive_among_many_silent_negatives_is_evaluable(): + # A single explicit row is a singleton by row count alone, but 19 silent, + # admissible cells settle not-bound alongside it: 20 evaluable cells, not + # a discarded singleton. + members = [("S1", "c0")] + [("S1", f"s{i}") for i in range(19)] + states = _key_states([("S1", "c0", "A", B)]) + cells_by_set = {"s1": members} + admissibility = Admissibility({k: 5 for k in members}, 2, set()) + out = self_disagreement(states, {"A"}, {"S1": {"A"}}, cells_by_set, admissibility, level="identity") + r = _row_for_key(out, "A") + assert r["setsEvaluated"] == 1 + assert r["disagreementRate"] == 1.0 + + +def test_all_silent_not_bound_cells_agree(): + # The mirror of the defect test: every cell of the set is silent and + # admissible, so every one settles not-bound. All evaluable cells give + # the same settled state, so the set agrees with itself -- this must not + # be over-corrected into calling every silent set a disagreement. + members = [("S1", f"s{i}") for i in range(5)] + states = _key_states([]) + cells_by_set = {"s1": members} + admissibility = Admissibility({k: 5 for k in members}, 2, set()) + out = self_disagreement(states, {"A"}, {"S1": {"A"}}, cells_by_set, admissibility, level="identity") + r = _row_for_key(out, "A") + assert r["setsEvaluated"] == 1 + assert r["disagreementRate"] == 0.0 + + +def test_self_disagreement_output_is_deterministic_regardless_of_input_row_order(): + # This becomes a p-column, so it must be byte-stable across row orders. + rows = [ + ("S1", "c0", "A", B), + ("S1", "c1", "A", N), + ("S1", "d0", "B", B), + ("S1", "d1", "B", B), + ("S2", "e0", "A", N), + ] + cells_by_set = { + "s1": [("S1", "c0"), ("S1", "c1")], + "s2": [("S1", "d0"), ("S1", "d1")], + "s3": [("S2", "e0")], + } + universe = {"A", "B"} + offered = {"S1": {"A", "B"}, "S2": {"A", "B"}} + baseline = self_disagreement(_key_states(rows), universe, offered, cells_by_set, _NEUTRAL, level="identity") + + rng = random.Random(2026) + for _ in range(5): + shuffled = list(rows) + rng.shuffle(shuffled) + out = self_disagreement(_key_states(shuffled), universe, offered, cells_by_set, _NEUTRAL, level="identity") + assert out.equals(baseline) From 22a3709bb9de1004199019ade0f2eaf8652dff04 Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 16:31:57 +0200 Subject: [PATCH 041/282] MILAB-6496: declare the run's quality measurements and compute what nothing else does --- software/per-cell-metrics/src/qc_measures.py | 307 +++++++++++++++++ .../per-cell-metrics/test/test_qc_measures.py | 320 ++++++++++++++++++ 2 files changed, 627 insertions(+) create mode 100644 software/per-cell-metrics/src/qc_measures.py create mode 100644 software/per-cell-metrics/test/test_qc_measures.py diff --git a/software/per-cell-metrics/src/qc_measures.py b/software/per-cell-metrics/src/qc_measures.py new file mode 100644 index 0000000..9908af5 --- /dev/null +++ b/software/per-cell-metrics/src/qc_measures.py @@ -0,0 +1,307 @@ +"""The quality measurements a run carries. + +Every measurement carries what it counts, because the reader who meets it is not +the person who chose it: a fraction with a name and no statement of what went +into the numerator gets read as whatever the name suggests, and several of these +names suggest more than they carry. + +Where a line can be defended, it also carries what a bad value implies. Where +none can, it carries nothing about what a bad value would mean -- nothing is +known, so the number and its distribution are shown and the reader judges. + +None carries what to do about it. Advice depends on the run, the study and what +else is available, none of which this readout knows. + +A measurement this module cannot compute is declared anyway, with the reason it +cannot, so a reader never mistakes "nothing computed this yet" for "this was +checked and found fine." Most of the set is computed elsewhere in this package +and only declared here -- undeclared barcodes and declared-but-unseen tags in +``panel.py``, the floor's counts and the high-reference-cell count in +``verdict.py``, both levels of self-disagreement in ``combine.py``, and the read +and per-cell totals in ``qc_report.py``. This module declares the full set and +computes only what none of those already do. +""" + +from __future__ import annotations + +from dataclasses import dataclass + +import numpy as np +import polars as pl + + +@dataclass(frozen=True) +class Measurement: + id: str + label: str + level: str # "sample" | "tag" | "identity" + counts: str # what went into it + implies: str | None = None # what a bad value means, where a line exists + line: str | None = None # which defence route backs `implies`, if any + deferred_reason: str | None = None # set only when nothing computes this yet + + +MEASUREMENTS: tuple[Measurement, ...] = ( + Measurement( + "readsTotal", + "Reads total and fraction matched", + "sample", + "Every read the parser saw, and the share matching the tag pattern.", + "A low matched share means the pattern does not fit the library's geometry.", + "inherited", + ), + Measurement( + "panelAssignedFraction", + "Fraction of antigen reads usable", + "sample", + "Reads whose corrected barcode is on the panel, over reads matched.", + "A low share means most reads carry barcodes the panel never declared.", + "inherited", + ), + # The spec's one row for saturation and reads-per-barcode covers two figures + # with different fates in this build: reads-per-barcode can be derived from + # counts this package already has, saturation cannot. One Measurement could + # not carry "computed" and "deferred" at once, so the row becomes two ids + # here, both at the row's declared level. + Measurement( + "sequencingSaturation", + "Sequencing saturation", + "sample", + "Duplicate reads over total reads.", + deferred_reason="needs read-level data the per-sample fan-out discards", + ), + Measurement( + "readsPerBarcode", + "Reads per barcode", + "sample", + "Reads matched, over barcodes observed.", + "Below the vendor's recommended minimum the library is undersequenced.", + "recommended-and-observed", + ), + Measurement( + "antigenCountDistribution", + "Distribution of antigen count per barcode", + "sample", + "Deciles of the total antigen count per cell barcode.", + ), + Measurement( + "aggregateBarcodeFraction", + "Fraction of reads in aggregate barcodes", + "sample", + "Reads in barcodes flagged as aggregates, over reads matched.", + deferred_reason="no aggregate-barcode detection exists in this block", + ), + Measurement( + "undeclaredBarcodes", + "Undeclared barcodes, and which sequences", + "tag", + "Barcodes the reads carry that the sample's panel does not declare, and which sequences they are.", + "A declared-nothing barcode carrying reads means the panel file is incomplete.", + "inherited", + ), + Measurement( + "declaredNeverSeen", + "Declared tags the reads never show", + "tag", + "Tags on the sample's panel with no reads at all.", + "A reagent that produced nothing did not work in this run.", + "categorical", + ), + Measurement( + "floorRemoved", + "Counts the floor removed, and cells left with none", + "sample", + "Readings the floor zeroed, and cells whose every non-reference reading was floored.", + ), + Measurement( + "uniqueCountsPerCell", + "Reads and unique counts per cell", + "sample", + "Reads and distinct UMIs per cell barcode.", + ), + Measurement( + "highReferenceCells", + "Cells carrying a high reference reading", + "sample", + "Cells whose reference reading is at or above the observation line.", + "A high share means ambient material or sticky cells are widespread.", + "observation-line", + ), + Measurement( + "perAntigen", + "Per antigen: signal, above the line, and the median", + "tag", + "Per tag: cells with any reading, cells whose reading was bound, and the median count among those.", + ), + Measurement( + "identityDisagreement", + "Clonotype self-disagreement at an identity", + "identity", + "Clonotypes whose evaluable cells did not all agree, over clonotypes with two or more evaluable " + "cells, at an identity.", + "A high rate means the answer a scientist acts on is unstable.", + "against-the-run", + ), + Measurement( + "tagDisagreement", + "Clonotype self-disagreement at a single tag", + "tag", + "The same, computed at a single tag rather than an identity. Diagnostic only: it rests on comparing " + "each tag against the reference separately, which no verdict is built from.", + "A tag standing clear of the others in its panel is misbehaving, whether or not the identities it feeds are.", + "against-the-run", + ), + Measurement( + "knownAnswerRecovered", + "Whether a declared known answer came back", + "sample", + "The quantity recovered for a clonotype declared in advance, against what was intended.", + deferred_reason="no input declares a known answer", + ), +) + +NOT_EVALUATED = "not evaluated" + + +def measurement_row(m: Measurement) -> dict: + """One declared measurement, rendered for a reader who never opens this module. + + A deferred measurement renders with its own reason attached and keeps its + place in the set -- the difference between "checked and fine" and "never + checked" is lost the moment a deferred id simply has no row. + """ + return { + "id": m.id, + "label": m.label, + "level": m.level, + "counts": m.counts, + "implies": m.implies, + "status": NOT_EVALUATED if m.deferred_reason else None, + "reason": m.deferred_reason, + } + + +def measurement_rows() -> list[dict]: + """Every declared measurement, deferred ones included, in declaration order.""" + return [measurement_row(m) for m in MEASUREMENTS] + + +def per_antigen_measures(states: pl.DataFrame) -> pl.DataFrame: + """Per tag: cells with any reading, cells whose reading was bound, and the median count among those. + + Grouped by tag, not by identity: a tag's own reagent behaviour is the + question this answers, and an identity built from several tags would let + one weak tag hide behind a stronger one in the same combined figure. + + `states` is the tag-grain shape -- one row per (cell, tag) with an + explicit reading, columns `tag`, `umiCount` and `state` -- the same sparse + frame a tag-level self-disagreement count is taken from, not the frame + tags have already been combined into an identity on. + + The sparse frame is the right input here: "cells with any reading" means + cells with an observed reading, and a cell silent for this tag has no + count to contribute. There is no asked population to complete this + against, unlike a per-cell total or a reads-per-barcode rate, both of + which are asked-cell questions and use a densified or a whole-run count + instead. + """ + return ( + states.group_by("tag") + .agg( + (pl.col("umiCount") > 0).sum().alias("cellsWithSignal"), + (pl.col("state") == "bound").sum().alias("cellsAboveTheLine"), + pl.col("umiCount").filter(pl.col("state") == "bound").median().alias("medianAboveTheLine"), + ) + .sort("tag") + ) + + +def reads_per_barcode(reads_matched: int, barcodes_observed: int) -> float | None: + """Reads matched, over barcodes observed. + + Both counts already exist in the per-sample QC row -- `readsMatched` and + `cellsDetected` -- so this only divides them; neither is recounted here. + + None when no barcode was observed. A rate over zero barcodes is not a + small number, it is no number, and returning None keeps that distinct + from a rate that was computed and happens to be zero. + """ + if barcodes_observed <= 0: + return None + return reads_matched / barcodes_observed + + +# The extremes are included alongside the interior deciles so the distribution's +# edges are visible, not only its middle: eleven points, 0 through 100 by 10. +DECILE_POINTS: tuple[int, ...] = tuple(range(0, 101, 10)) + + +def antigen_count_deciles(counts: pl.DataFrame) -> pl.DataFrame: + """Deciles of the total antigen count per cell barcode. + + `counts` is the sparse per-(cell, tag) frame -- one row per observed + reading, columns sampleId, cellId, umiCount -- taken before flooring or + identity-combining, the same shape the floor itself works on. A cell's + total sums every tag it shows any reading for; a cell with no row at all + contributes no total, since crediting it a total of zero would read as a + reading rather than as the absence it is. + + Returns one row per decile point, columns `decile` and `value`. An empty + input still returns all eleven decile rows, `value` null throughout: no + cells observed is eleven declared, unanswered points, never an empty + frame -- the same "declared, not absent" rule the deferred measurements + follow. + """ + if counts.height == 0: + return pl.DataFrame( + {"decile": list(DECILE_POINTS), "value": [None] * len(DECILE_POINTS)}, + schema={"decile": pl.Int64, "value": pl.Float64}, + ) + + totals = counts.group_by(["sampleId", "cellId"]).agg(pl.col("umiCount").sum().alias("total"))["total"].to_numpy() + values = [float(np.quantile(totals, p / 100)) for p in DECILE_POINTS] + return pl.DataFrame({"decile": list(DECILE_POINTS), "value": values}) + + +def attach_alerting_identities( + identity_measures: pl.DataFrame, + grouping: dict[str, set[str]], + alerting: set[str], +) -> pl.DataFrame: + """Beside each alerting tag, the identity figures for the identities it feeds. + + Deciding which tags alert is a threshold call made elsewhere, not here; + `alerting` names the tags a caller has already flagged. `grouping` maps a + tag to every identity it feeds -- ordinarily one, since one tag combines + into one identity, but kept as a set rather than a single value so a tag + feeding more than one identity attaches beside all of them rather than + arbitrarily one. + + A noisy reagent whose identities read steady is a reagent to replace, not + a run to distrust; this attachment is what lets a reader tell the two + apart, by showing both figures rather than only the tag's own. + + `identity_measures` is self-disagreement's own identity-level output: a + `key` column holding the identity, plus its measures. Returns one row per + (alerting tag, identity it feeds) -- so a tag feeding two identities + produces two rows, neither dropped -- with columns `tag`, `identity`, and + every column `identity_measures` carries besides `key`. A tag in + `alerting` that feeds no known identity contributes no row, since there is + no identity figure to attach beside it. + """ + identity_columns = [c for c in identity_measures.columns if c != "key"] + pairs = [(tag, identity) for tag in sorted(alerting) for identity in sorted(grouping.get(tag, ()))] + + if not pairs: + return pl.DataFrame( + schema={ + "tag": pl.String, + "identity": pl.String, + **{c: identity_measures.schema[c] for c in identity_columns}, + } + ) + + pair_frame = pl.DataFrame(pairs, orient="row", schema={"tag": pl.String, "identity": pl.String}) + return pair_frame.join(identity_measures.rename({"key": "identity"}), on="identity", how="left").sort( + ["tag", "identity"] + ) diff --git a/software/per-cell-metrics/test/test_qc_measures.py b/software/per-cell-metrics/test/test_qc_measures.py new file mode 100644 index 0000000..8d8ec9f --- /dev/null +++ b/software/per-cell-metrics/test/test_qc_measures.py @@ -0,0 +1,320 @@ +import dataclasses + +import polars as pl +import pytest +from qc_measures import ( + MEASUREMENTS, + Measurement, + antigen_count_deciles, + attach_alerting_identities, + measurement_rows, + per_antigen_measures, + reads_per_barcode, +) + +# The spec's row for sequencing saturation and reads per barcode covers two +# figures with different fates in this build -- one derivable from counts the +# package already has, the other not -- so it becomes two declared ids here, +# both at the row's stated level. Every other row maps one to one. This is the +# expected per-id level, built from the spec's own table rather than copied +# from this module, so a level typo on any id changes the multiset below. +EXPECTED_LEVEL_BY_ID = { + "readsTotal": "sample", + "panelAssignedFraction": "sample", + "sequencingSaturation": "sample", + "readsPerBarcode": "sample", + "antigenCountDistribution": "sample", + "aggregateBarcodeFraction": "sample", + "undeclaredBarcodes": "tag", + "declaredNeverSeen": "tag", + "floorRemoved": "sample", + "uniqueCountsPerCell": "sample", + "highReferenceCells": "sample", + "perAntigen": "tag", + "identityDisagreement": "identity", + "tagDisagreement": "tag", + "knownAnswerRecovered": "sample", +} + +DEFERRED_IDS = {"sequencingSaturation", "aggregateBarcodeFraction", "knownAnswerRecovered"} + + +def test_every_declared_id_is_expected_and_every_expected_id_is_declared(): + assert {m.id for m in MEASUREMENTS} == set(EXPECTED_LEVEL_BY_ID) + + +def test_declared_levels_match_the_spec_as_a_multiset(): + # A multiset comparison, not a per-id comparison: swapping any one + # measurement's level changes how many times that level appears overall, + # so a typo trips this even without knowing which id was mistyped. + declared = sorted(m.level for m in MEASUREMENTS) + expected = sorted(EXPECTED_LEVEL_BY_ID.values()) + assert declared == expected + + +def test_every_measurement_declares_a_known_level(): + assert {m.level for m in MEASUREMENTS} <= {"sample", "tag", "identity"} + + +def test_every_measurement_says_what_it_counts(): + assert all(m.counts for m in MEASUREMENTS) + + +def test_measurement_has_no_produced_today_field(): + # produced_today would answer whether the superseded tool produced this + # measurement, which reads backwards from what a reader of this block + # needs: deferred_reason is None already answers whether THIS build does. + field_names = {f.name for f in dataclasses.fields(Measurement)} + assert "produced_today" not in field_names + + +def test_an_unjudged_measurement_says_nothing_about_a_bad_value(): + for m in MEASUREMENTS: + if m.line is None: + assert m.implies is None, m.id + + +BANNED_ADVICE_PHRASES = ( + "should", + "must", + "need to", + "needs to", + "ought", + "advise", + "advice", + "we suggest", + "try ", + "consider ", + "re-run", + "rerun", + "replace", + "avoid", + "ensure", + "make sure", + "flag for", + "recommend that", + "recommend you", +) + +# A sentence opening with one of these reads as an instruction regardless of +# what follows -- "Replace the reagent." vs "A reagent that produced nothing +# did not work" -- so this catches advice phrased as an imperative, which the +# substring list above does not, since none of these words are banned outright +# (several appear as ordinary nouns/adjectives elsewhere in the set, e.g. "the +# vendor's recommended minimum"). +IMPERATIVE_OPENERS = { + "check", + "verify", + "replace", + "remove", + "increase", + "decrease", + "use", + "try", + "consider", + "avoid", + "ensure", + "fix", + "rerun", + "lower", + "raise", + "discard", + "exclude", + "flag", + "recheck", + "investigate", + "review", +} + + +def test_no_measurement_carries_advice(): + for m in MEASUREMENTS: + text = f"{m.counts} {m.implies or ''}" + lowered = text.lower() + assert not any(phrase in lowered for phrase in BANNED_ADVICE_PHRASES), m.id + + for sentence in text.split("."): + first_word = sentence.strip().split(" ", 1)[0].strip(",:;").lower() + assert first_word not in IMPERATIVE_OPENERS, (m.id, sentence) + + +def test_deferred_measurements_are_declared_not_omitted(): + deferred = {m.id for m in MEASUREMENTS if m.deferred_reason} + assert deferred == DEFERRED_IDS + + +def test_deferred_measurement_reasons_are_stated(): + for m in MEASUREMENTS: + if m.id in DEFERRED_IDS: + assert m.deferred_reason, m.id + assert m.implies is None, m.id + + +def test_deferred_measurement_produces_a_not_evaluated_row_with_its_reason(): + rows = measurement_rows() + # Never absent: every declared id, deferred or not, has a row. + assert {r["id"] for r in rows} == {m.id for m in MEASUREMENTS} + + by_id = {r["id"]: r for r in rows} + for deferred_id in DEFERRED_IDS: + row = by_id[deferred_id] + assert row["status"] == "not evaluated" + assert row["reason"] + + +def test_a_computed_measurement_carries_no_status(): + rows = measurement_rows() + by_id = {r["id"]: r for r in rows} + for m in MEASUREMENTS: + if m.id not in DEFERRED_IDS: + assert by_id[m.id]["status"] is None, m.id + assert by_id[m.id]["reason"] is None, m.id + + +# --- per_antigen_measures: tag grain ----------------------------------------- + + +def test_per_antigen_measures_reports_signal_above_and_median(): + states = pl.DataFrame( + { + "tag": ["T1", "T1", "T1"], + "umiCount": [0, 10, 40], + "state": ["not bound", "bound", "bound"], + } + ) + out = per_antigen_measures(states).row(0, named=True) + assert out["cellsWithSignal"] == 2 + assert out["cellsAboveTheLine"] == 2 + assert out["medianAboveTheLine"] == 25.0 + + +def test_per_antigen_measures_differs_between_tag_and_identity_grain(): + # T1 and T2 both feed one identity. As tags, T1 shows one weak cell (one + # bound of two); as the combined identity, the same cells collapse to one + # row and T1's weak showing is no longer visible on its own. + tag_grain = pl.DataFrame( + { + "tag": ["T1", "T1", "T2", "T2"], + "umiCount": [8, 1, 20, 15], + "state": ["bound", "not bound", "bound", "bound"], + } + ) + identity_grain = pl.DataFrame( + { + "tag": ["ID1", "ID1", "ID1"], # the identity each cell's highest tag reading combined into + "umiCount": [20, 1, 15], + "state": ["bound", "not bound", "bound"], + } + ) + + by_tag = per_antigen_measures(tag_grain) + by_identity = per_antigen_measures(identity_grain) + + assert by_tag.height == 2 + assert by_identity.height == 1 + assert dict(zip(by_tag["tag"], by_tag["cellsAboveTheLine"], strict=True)) == {"T1": 1, "T2": 2} + assert by_identity.row(0, named=True)["cellsAboveTheLine"] == 2 + + +# --- reads_per_barcode -------------------------------------------------------- + + +def test_reads_per_barcode_computes_the_rate(): + assert reads_per_barcode(1000, 200) == 5.0 + + +def test_reads_per_barcode_zero_barcodes_observed_does_not_divide_by_zero(): + assert reads_per_barcode(1000, 0) is None + + +# --- antigen_count_deciles ----------------------------------------------------- + + +def _cell_counts(totals: dict[str, int], sample_id: str = "S1") -> pl.DataFrame: + return pl.DataFrame( + { + "sampleId": [sample_id] * len(totals), + "cellId": list(totals.keys()), + "umiCount": list(totals.values()), + }, + schema={"sampleId": pl.String, "cellId": pl.String, "umiCount": pl.Int64}, + ) + + +def test_antigen_count_deciles_on_a_known_distribution(): + # 11 cells with totals 0, 10, ..., 100: with linear interpolation over 11 + # sorted points, the p-th percentile lands exactly on index p/10, so every + # decile equals its own cell's total -- a fixture an off-by-one position + # error cannot pass unnoticed on. + counts = _cell_counts({f"c{i}": i * 10 for i in range(11)}) + out = antigen_count_deciles(counts) + assert out["decile"].to_list() == list(range(0, 101, 10)) + assert out["value"].to_list() == [float(i * 10) for i in range(11)] + + +def test_antigen_count_deciles_single_cell_sample(): + counts = _cell_counts({"c0": 42}) + out = antigen_count_deciles(counts) + assert out.height == 11 + assert all(v == 42.0 for v in out["value"].to_list()) + + +def test_antigen_count_deciles_empty_sample(): + counts = _cell_counts({}) + out = antigen_count_deciles(counts) + assert out.height == 11 + assert out["decile"].to_list() == list(range(0, 101, 10)) + assert all(v is None for v in out["value"].to_list()) + + +# --- attach_alerting_identities ------------------------------------------------ + + +def _identity_measures(rows: dict[str, tuple[int, int, float]]) -> pl.DataFrame: + keys = list(rows) + return pl.DataFrame( + { + "key": keys, + "setsEvaluated": [rows[k][0] for k in keys], + "setsDisagreeing": [rows[k][1] for k in keys], + "disagreementRate": [rows[k][2] for k in keys], + "level": ["identity"] * len(keys), + "diagnosticOnly": ["false"] * len(keys), + } + ) + + +def test_attach_alerting_identities_feeding_two_identities_attaches_both(): + identity_measures = _identity_measures({"ID1": (10, 1, 0.1), "ID2": (8, 0, 0.0), "ID3": (5, 2, 0.4)}) + grouping = {"T1": {"ID1", "ID2"}, "T2": {"ID3"}} + + out = attach_alerting_identities(identity_measures, grouping, alerting={"T1"}) + + assert set(out["identity"].to_list()) == {"ID1", "ID2"} + assert set(out["tag"].to_list()) == {"T1"} + assert "ID3" not in out["identity"].to_list() + # The identity's own figures travel with it, not just its name. + row = out.filter(pl.col("identity") == "ID1").row(0, named=True) + assert row["setsEvaluated"] == 10 + assert row["disagreementRate"] == pytest.approx(0.1) + + +def test_attach_alerting_identities_no_alerting_tags_returns_no_rows(): + identity_measures = _identity_measures({"ID1": (10, 1, 0.1)}) + out = attach_alerting_identities(identity_measures, {"T1": {"ID1"}}, alerting=set()) + assert out.height == 0 + assert set(out.columns) == { + "tag", + "identity", + "setsEvaluated", + "setsDisagreeing", + "disagreementRate", + "level", + "diagnosticOnly", + } + + +def test_attach_alerting_identities_tag_with_no_grouping_entry_contributes_no_row(): + identity_measures = _identity_measures({"ID1": (10, 1, 0.1)}) + out = attach_alerting_identities(identity_measures, {"T1": {"ID1"}}, alerting={"CONTROL"}) + assert out.height == 0 From c0d50fd0c8889d53982e67698d116a82cef545a3 Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 17:24:30 +0200 Subject: [PATCH 042/282] MILAB-6496: QC statuses, parameterised lines, three-level rollup with coverage --- software/per-cell-metrics/src/qc_measures.py | 167 +++++++++++++++- .../per-cell-metrics/test/test_qc_measures.py | 183 ++++++++++++++++++ 2 files changed, 344 insertions(+), 6 deletions(-) diff --git a/software/per-cell-metrics/src/qc_measures.py b/software/per-cell-metrics/src/qc_measures.py index 9908af5..4ad97cb 100644 --- a/software/per-cell-metrics/src/qc_measures.py +++ b/software/per-cell-metrics/src/qc_measures.py @@ -25,11 +25,29 @@ from __future__ import annotations from dataclasses import dataclass +from enum import Enum import numpy as np import polars as pl +class Status(str, Enum): + ACCEPTABLE = "acceptable" + ALERTING = "alerting" + UNJUDGED = "unjudged" + NOT_EVALUATED = "not evaluated" + + +@dataclass(frozen=True) +class Coverage: + """A level's status, and how much of it was actually checked.""" + + status: Status + judged: int + unjudged: int + not_evaluated: int + + @dataclass(frozen=True) class Measurement: id: str @@ -46,9 +64,11 @@ class Measurement: "readsTotal", "Reads total and fraction matched", "sample", + # No line: the four inherited numbers atom 315 names are the usable + # antigen-read fraction, the undeclared-barcode fraction, the aggregate- + # barcode read fraction and barcode validity. The matched share is on + # none of them, so nothing here says what a low one would mean. "Every read the parser saw, and the share matching the tag pattern.", - "A low matched share means the pattern does not fit the library's geometry.", - "inherited", ), Measurement( "panelAssignedFraction", @@ -123,9 +143,11 @@ class Measurement: "highReferenceCells", "Cells carrying a high reference reading", "sample", + # No line: the observation line is the per-cell threshold this + # measurement uses to decide which cells to count, not a defended line on + # the share it reports. Nothing in the field publishes a share of cells + # that is too high, so the share is shown and the reader judges. "Cells whose reference reading is at or above the observation line.", - "A high share means ambient material or sticky cells are widespread.", - "observation-line", ), Measurement( "perAntigen", @@ -160,7 +182,140 @@ class Measurement: ), ) -NOT_EVALUATED = "not evaluated" +# The four routes a line can be defended by, and no others. `Measurement.line` +# names one of these or None, and it is the *only* declaration of which +# measurements carry a line -- the tables below are derived facts about the +# route, never a second opinion on whether a line exists. A test asserts the +# correspondence in both directions. +LINE_ROUTES: frozenset[str] = frozenset({"inherited", "categorical", "recommended-and-observed", "against-the-run"}) + +# Three of the four routes put an absolute number on the measurement. The +# fourth compares the run against itself and carries no number at all; see +# `outlier_status`. +NUMERIC_LINE_ROUTES: frozenset[str] = frozenset({"inherited", "categorical", "recommended-and-observed"}) + +# Every line is a parameter with a shipped default, and the operator may +# override any of them. No line is invented -- where none of the four routes +# applies the measurement stays unjudged rather than being given a number with +# nothing behind it. +DEFAULT_LINES: dict[str, float] = { + "panelAssignedFraction": 0.5, # inherited + "undeclaredBarcodes": 0.1, # inherited + "declaredNeverSeen": 0, # categorical: alerting at zero reads + "readsPerBarcode": 5_000, # recommended-and-observed +} + +# How each line is read. Deliberately *not* overridable: an operator moves a +# number, never a direction. Three comparisons rather than one flag, because a +# floor and a categorical fact disagree at the boundary -- `readsPerBarcode` +# alerts strictly *below* the recommendation, while `declaredNeverSeen` alerts +# *at* zero. One `<=` cannot serve both. +# +# at-least acceptable at or above the line, alerting strictly below +# at-most acceptable at or below the line, alerting strictly above +# alerting-at alerting where the value equals the line +# +# In every case the named value satisfies the condition it names. +_COMPARISON: dict[str, str] = { + "panelAssignedFraction": "at-least", + "undeclaredBarcodes": "at-most", + "declaredNeverSeen": "alerting-at", + "readsPerBarcode": "at-least", +} + +_ORDINAL = {Status.ACCEPTABLE: 0, Status.ALERTING: 1} + +_DEFERRED: frozenset[str] = frozenset(m.id for m in MEASUREMENTS if m.deferred_reason) + + +def status_for(measurement: str, value: float | None, lines: dict[str, float]) -> Status: + """How one measurement reads, given the lines in force. + + A deferred measurement is not evaluated whatever it is handed: nothing + computes it, so a value reaching here is a caller's mistake and must not be + laundered into a judgement about the run. + """ + if measurement in _DEFERRED or value is None: + return Status.NOT_EVALUATED + if measurement not in lines: + return Status.UNJUDGED + line = lines[measurement] + comparison = _COMPARISON[measurement] + if comparison == "at-least": + bad = value < line + elif comparison == "at-most": + bad = value > line + else: + bad = value == line + return Status.ALERTING if bad else Status.ACCEPTABLE + + +# The interquartile fence a value must clear to count as standing apart from its +# peers. A parameter like every other line, and visible for the same reason. +DEFAULT_OUTLIER_FENCE: float = 3.0 + +MIN_PEERS_TO_COMPARE = 3 + + +def outlier_status( + value: float | None, + peers: list[float], + fence: float = DEFAULT_OUTLIER_FENCE, +) -> Status: + """The fourth route: a value standing clear of its peers in the same panel. + + Needs no published number to be valid, which is the same ground the + self-disagreement measure stands on in the first place. + + `peers` **excludes** `value` -- the other tags in the same panel, not all of + them. Including it would let a single extreme reading inflate the upper + quartile it is then measured against, so the one case the measure exists to + catch is the one it would miss. + + Only high values are flagged. A disagreement rate below its peers is a tag + behaving better than the panel, which is not a finding. + + Unjudged below `MIN_PEERS_TO_COMPARE` peers, where a quartile is not a + distribution but an arithmetic accident of two or three numbers. + """ + if value is None: + return Status.NOT_EVALUATED + if len(peers) < MIN_PEERS_TO_COMPARE: + return Status.UNJUDGED + q1, q3 = (float(q) for q in np.quantile(peers, [0.25, 0.75])) + return Status.ALERTING if value > q3 + (q3 - q1) * fence else Status.ACCEPTABLE + + +def roll_up(statuses: list[Status]) -> Coverage: + """The worst status among those that carry one, plus coverage. + + Coverage stays out of the ordinal because acceptable/alerting and + not-evaluated answer different questions. The first says whether something + is wrong; the second says whether anybody looked. Ranked on one scale, an + unchecked run becomes indistinguishable from a checked one. + """ + judged = [s for s in statuses if s in _ORDINAL] + unjudged = sum(1 for s in statuses if s is Status.UNJUDGED) + not_evaluated = sum(1 for s in statuses if s is Status.NOT_EVALUATED) + status = max(judged, key=lambda s: _ORDINAL[s]) if judged else Status.NOT_EVALUATED + return Coverage(status, len(judged), unjudged, not_evaluated) + + +def roll_up_panel(tag_statuses: list[Status], identity_statuses: list[Status]) -> Coverage: + """A panel carries the worst status among its per-tag and per-identity measurements.""" + return roll_up([*tag_statuses, *identity_statuses]) + + +def roll_up_capture(sample_statuses: list[Status], panel_statuses: list[Status]) -> Coverage: + """A capture carries the worst status among every sample and every panel within it. + + Sample and panel are separate axes rather than nested: a per-tag failure is + usually a property of the reagent across the whole run rather than of any one + sample, and a dead reagent would otherwise mark every sample alerting. The + two call for different actions, and nothing hides because the capture rolls + up both. + """ + return roll_up([*sample_statuses, *panel_statuses]) def measurement_row(m: Measurement) -> dict: @@ -176,7 +331,7 @@ def measurement_row(m: Measurement) -> dict: "level": m.level, "counts": m.counts, "implies": m.implies, - "status": NOT_EVALUATED if m.deferred_reason else None, + "status": Status.NOT_EVALUATED if m.deferred_reason else None, "reason": m.deferred_reason, } diff --git a/software/per-cell-metrics/test/test_qc_measures.py b/software/per-cell-metrics/test/test_qc_measures.py index 8d8ec9f..730dc5c 100644 --- a/software/per-cell-metrics/test/test_qc_measures.py +++ b/software/per-cell-metrics/test/test_qc_measures.py @@ -3,13 +3,25 @@ import polars as pl import pytest from qc_measures import ( + _COMPARISON, + DEFAULT_LINES, + DEFAULT_OUTLIER_FENCE, + LINE_ROUTES, MEASUREMENTS, + NUMERIC_LINE_ROUTES, + Coverage, Measurement, + Status, antigen_count_deciles, attach_alerting_identities, measurement_rows, + outlier_status, per_antigen_measures, reads_per_barcode, + roll_up, + roll_up_capture, + roll_up_panel, + status_for, ) # The spec's row for sequencing saturation and reads per barcode covers two @@ -318,3 +330,174 @@ def test_attach_alerting_identities_tag_with_no_grouping_entry_contributes_no_ro identity_measures = _identity_measures({"ID1": (10, 1, 0.1)}) out = attach_alerting_identities(identity_measures, {"T1": {"ID1"}}, alerting={"CONTROL"}) assert out.height == 0 + + +def test_four_readings_only_two_are_statuses(): + assert {s.value for s in Status} == {"acceptable", "alerting", "unjudged", "not evaluated"} + + +# --- the route is the single authority ------------------------------------- +# Both directions, so neither table can grow an entry the other does not know +# about. This is what stops `DEFAULT_LINES` becoming a second declaration of +# which measurements carry a line. + + +def test_every_declared_route_is_one_of_the_atoms_four(): + assert {m.line for m in MEASUREMENTS if m.line} <= LINE_ROUTES + assert LINE_ROUTES == {"inherited", "categorical", "recommended-and-observed", "against-the-run"} + + +def test_a_numeric_route_has_a_line_and_a_comparison_and_nothing_else_does(): + numeric = {m.id for m in MEASUREMENTS if m.line in NUMERIC_LINE_ROUTES} + assert set(DEFAULT_LINES) == numeric + assert set(_COMPARISON) == numeric + + +def test_an_unjudged_measurement_claims_nothing_about_a_bad_value(): + # Atom 315: where no line can be defended, nothing is said about what a bad + # value would mean, because nothing is known. + for m in MEASUREMENTS: + if m.line is None and m.deferred_reason is None: + assert m.implies is None, m.id + + +def test_reads_total_and_high_reference_cells_are_unjudged(): + by_id = {m.id: m for m in MEASUREMENTS} + assert by_id["readsTotal"].line is None + assert by_id["highReferenceCells"].line is None + assert status_for("readsTotal", 0.5, DEFAULT_LINES) is Status.UNJUDGED + + +def test_the_invented_matched_fraction_line_is_gone(): + assert "matchedFraction" not in DEFAULT_LINES + assert "readsTotal" not in DEFAULT_LINES + + +# --- lines are parameters, and every boundary is pinned -------------------- + + +def test_depth_line_is_a_parameter_not_a_literal(): + assert DEFAULT_LINES["readsPerBarcode"] == 5_000 + assert status_for("readsPerBarcode", 4_000, {"readsPerBarcode": 5_000}) is Status.ALERTING + assert status_for("readsPerBarcode", 4_000, {"readsPerBarcode": 1_000}) is Status.ACCEPTABLE + + +def test_at_least_is_acceptable_exactly_at_the_line(): + # Atom 315 alerts *below* the recommendation, so the recommendation itself + # is acceptable. The named value satisfies the condition it names. + assert status_for("readsPerBarcode", 5_000, DEFAULT_LINES) is Status.ACCEPTABLE + assert status_for("readsPerBarcode", 4_999, DEFAULT_LINES) is Status.ALERTING + assert status_for("panelAssignedFraction", 0.5, DEFAULT_LINES) is Status.ACCEPTABLE + assert status_for("panelAssignedFraction", 0.49, DEFAULT_LINES) is Status.ALERTING + + +def test_at_most_is_acceptable_exactly_at_the_line(): + assert status_for("undeclaredBarcodes", 0.1, DEFAULT_LINES) is Status.ACCEPTABLE + assert status_for("undeclaredBarcodes", 0.11, DEFAULT_LINES) is Status.ALERTING + + +def test_categorical_alerts_only_on_the_named_fact(): + # Alerting *at* zero -- a different predicate from "at or below a floor", + # which is why one direction flag cannot serve both. + assert status_for("declaredNeverSeen", 0, DEFAULT_LINES) is Status.ALERTING + assert status_for("declaredNeverSeen", 1, DEFAULT_LINES) is Status.ACCEPTABLE + + +def test_no_defensible_line_means_unjudged(): + assert status_for("antigenCountDistribution", 12, DEFAULT_LINES) is Status.UNJUDGED + + +def test_a_deferred_measurement_is_never_unjudged_even_holding_a_value(): + assert status_for("aggregateBarcodeFraction", 0.9, DEFAULT_LINES) is Status.NOT_EVALUATED + + +def test_a_missing_value_is_not_evaluated(): + assert status_for("readsPerBarcode", None, DEFAULT_LINES) is Status.NOT_EVALUATED + + +# --- the against-the-run route --------------------------------------------- + + +def test_a_lone_outlier_is_flagged_because_peers_exclude_the_value(): + # If `peers` included the value, one extreme reading would inflate q3 and + # could never be flagged -- the measure would defeat itself, and no fixture + # carrying a second outlier would reveal it. + assert outlier_status(0.9, [0.01, 0.02, 0.03, 0.02, 0.01]) is Status.ALERTING + + +def test_a_value_inside_its_peers_is_acceptable(): + assert outlier_status(0.02, [0.01, 0.02, 0.03, 0.02, 0.01]) is Status.ACCEPTABLE + + +def test_the_fence_multiplier_is_a_parameter(): + # q1 0.02, q3 0.04, so the default fence sits at 0.10 and a fence of 0.5 + # at 0.05. 0.08 falls between them, which is the only way the parameter is + # observable at all -- a value outside both brackets proves nothing. + peers = [0.01, 0.02, 0.03, 0.04, 0.05] + assert outlier_status(0.08, peers, fence=DEFAULT_OUTLIER_FENCE) is Status.ACCEPTABLE + assert outlier_status(0.08, peers, fence=0.5) is Status.ALERTING + + +def test_a_value_exactly_at_the_fence_is_acceptable(): + peers = [0.0, 1.0, 2.0, 3.0, 4.0] + q1, q3 = 1.0, 3.0 + fence = q3 + (q3 - q1) * DEFAULT_OUTLIER_FENCE + assert outlier_status(fence, peers) is Status.ACCEPTABLE + assert outlier_status(fence + 0.1, peers) is Status.ALERTING + + +def test_too_few_peers_is_unjudged_but_a_missing_value_is_not_evaluated(): + # Two different absences: nothing to compare against, versus nothing to + # compare. Collapsing them would make an uncomparable tag look unchecked. + assert outlier_status(0.9, [0.01, 0.02]) is Status.UNJUDGED + assert outlier_status(None, [0.01, 0.02, 0.03]) is Status.NOT_EVALUATED + + +# --- the three-level rollup ----------------------------------------------- + + +def test_rollup_takes_the_worst_status(): + assert roll_up([Status.ACCEPTABLE, Status.ALERTING]).status is Status.ALERTING + + +def test_a_rollup_returns_a_coverage(): + assert isinstance(roll_up([Status.ACCEPTABLE]), Coverage) + + +def test_coverage_never_enters_the_ordinal(): + r = roll_up([Status.ACCEPTABLE, Status.UNJUDGED, Status.NOT_EVALUATED]) + assert r.status is Status.ACCEPTABLE + + +def test_coverage_is_reported_beside_the_status(): + r = roll_up([Status.ACCEPTABLE, Status.ALERTING, Status.UNJUDGED, Status.NOT_EVALUATED]) + assert (r.judged, r.unjudged, r.not_evaluated) == (2, 1, 1) + + +def test_a_level_with_nothing_judgeable_is_not_evaluated(): + assert roll_up([Status.UNJUDGED, Status.NOT_EVALUATED]).status is Status.NOT_EVALUATED + + +def test_a_level_with_no_measurements_at_all_is_not_evaluated(): + r = roll_up([]) + assert r.status is Status.NOT_EVALUATED + assert (r.judged, r.unjudged, r.not_evaluated) == (0, 0, 0) + + +def test_panel_rolls_up_tag_and_identity_measurements(): + r = roll_up_panel(tag_statuses=[Status.ACCEPTABLE], identity_statuses=[Status.ALERTING]) + assert r.status is Status.ALERTING + + +def test_capture_rolls_up_samples_and_panels(): + r = roll_up_capture(sample_statuses=[Status.ACCEPTABLE], panel_statuses=[Status.ALERTING]) + assert r.status is Status.ALERTING + + +def test_a_dead_reagent_does_not_mark_every_sample_alerting(): + # Sample and panel are separate axes rather than nested: the panel alerts, + # the samples stay clean, and the capture still shows the problem. + samples = [roll_up([Status.ACCEPTABLE]).status for _ in range(3)] + panel = roll_up_panel(tag_statuses=[Status.ALERTING], identity_statuses=[Status.ACCEPTABLE]) + assert samples == [Status.ACCEPTABLE] * 3 + assert roll_up_capture(sample_statuses=samples, panel_statuses=[panel.status]).status is Status.ALERTING From 18794d27dd531b64b6b3470ac16be9e8fa043454 Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 17:27:51 +0200 Subject: [PATCH 043/282] MILAB-6496: close two mutation-surviving gaps in the QC status tests The coverage fixture used one unjudged and one not-evaluated measurement, so swapping the two counters left every assertion passing -- the questions "was a line defensible" and "did anybody look" were silently interchangeable. The counts now differ. DEFAULT_OUTLIER_FENCE was unpinned while every fence assertion derived its expected value from it, so the constant was free to move unnoticed. It is now pinned like every other line. --- .../per-cell-metrics/test/test_qc_measures.py | 23 +++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/software/per-cell-metrics/test/test_qc_measures.py b/software/per-cell-metrics/test/test_qc_measures.py index 730dc5c..6ceb565 100644 --- a/software/per-cell-metrics/test/test_qc_measures.py +++ b/software/per-cell-metrics/test/test_qc_measures.py @@ -429,6 +429,13 @@ def test_a_value_inside_its_peers_is_acceptable(): assert outlier_status(0.02, [0.01, 0.02, 0.03, 0.02, 0.01]) is Status.ACCEPTABLE +def test_the_shipped_fence_is_the_far_out_fence(): + # Pinned like every other line. Without this, every fence assertion below + # derives its expected value from the constant, so the constant itself is + # free to move and no test notices. + assert DEFAULT_OUTLIER_FENCE == 3.0 + + def test_the_fence_multiplier_is_a_parameter(): # q1 0.02, q3 0.04, so the default fence sits at 0.10 and a fence of 0.5 # at 0.05. 0.08 falls between them, which is the only way the parameter is @@ -470,8 +477,20 @@ def test_coverage_never_enters_the_ordinal(): def test_coverage_is_reported_beside_the_status(): - r = roll_up([Status.ACCEPTABLE, Status.ALERTING, Status.UNJUDGED, Status.NOT_EVALUATED]) - assert (r.judged, r.unjudged, r.not_evaluated) == (2, 1, 1) + # Two unjudged against one not-evaluated, deliberately unequal: with one of + # each, a counter that reported the other's total would read correctly and + # the two questions "was a line defensible" and "did anybody look" would be + # silently interchangeable. + r = roll_up( + [ + Status.ACCEPTABLE, + Status.ALERTING, + Status.UNJUDGED, + Status.UNJUDGED, + Status.NOT_EVALUATED, + ] + ) + assert (r.judged, r.unjudged, r.not_evaluated) == (2, 2, 1) def test_a_level_with_nothing_judgeable_is_not_evaluated(): From 9717f6cf9d7cf4283bbe8debe2e02bed2134b1dc Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 18:23:57 +0200 Subject: [PATCH 044/282] MILAB-6496: refuse an against-the-run measurement instead of calling it unjudged identityDisagreement and tagDisagreement declare the against-the-run route, so the spec says they carry a status. status_for resolved a status only from the line table, and neither has a line there -- correctly, since that route carries no absolute number -- so both returned unjudged. Unjudged is the one wrong answer that hides: it never enters a rollup, so an outlying reagent would leave its whole panel reading clean, which is the failure the rollup exists to invert. status_for now refuses these outright and names outlier_status, turning a silent wrong reading into a loud one. --- software/per-cell-metrics/src/qc_measures.py | 15 +++++++++++++ .../per-cell-metrics/test/test_qc_measures.py | 21 +++++++++++++++++++ 2 files changed, 36 insertions(+) diff --git a/software/per-cell-metrics/src/qc_measures.py b/software/per-cell-metrics/src/qc_measures.py index 4ad97cb..57f3559 100644 --- a/software/per-cell-metrics/src/qc_measures.py +++ b/software/per-cell-metrics/src/qc_measures.py @@ -227,6 +227,8 @@ class Measurement: _DEFERRED: frozenset[str] = frozenset(m.id for m in MEASUREMENTS if m.deferred_reason) +_AGAINST_THE_RUN: frozenset[str] = frozenset(m.id for m in MEASUREMENTS if m.line == "against-the-run") + def status_for(measurement: str, value: float | None, lines: dict[str, float]) -> Status: """How one measurement reads, given the lines in force. @@ -234,7 +236,20 @@ def status_for(measurement: str, value: float | None, lines: dict[str, float]) - A deferred measurement is not evaluated whatever it is handed: nothing computes it, so a value reaching here is a caller's mistake and must not be laundered into a judgement about the run. + + A measurement on the against-the-run route is refused outright rather than + answered. It does carry a status -- one this function cannot compute, since + the comparison is against the measurement's peers in the same panel and no + peers are passed here. Returning `unjudged` instead would be the worst + available answer: unjudged never enters a rollup, so an outlying reagent + would leave its panel reading clean, which is the exact failure the rollup + exists to invert. Call `outlier_status` for these. """ + if measurement in _AGAINST_THE_RUN: + raise ValueError( + f"{measurement!r} is judged against the run itself, not against a line: " + "call outlier_status(value, peers) with the measurement's peers in the same panel" + ) if measurement in _DEFERRED or value is None: return Status.NOT_EVALUATED if measurement not in lines: diff --git a/software/per-cell-metrics/test/test_qc_measures.py b/software/per-cell-metrics/test/test_qc_measures.py index 6ceb565..1fef3be 100644 --- a/software/per-cell-metrics/test/test_qc_measures.py +++ b/software/per-cell-metrics/test/test_qc_measures.py @@ -418,6 +418,27 @@ def test_a_missing_value_is_not_evaluated(): # --- the against-the-run route --------------------------------------------- +@pytest.mark.parametrize("measurement", ["identityDisagreement", "tagDisagreement"]) +def test_an_against_the_run_measurement_is_refused_not_called_unjudged(measurement): + # These two do carry a status; `status_for` just cannot compute it, having + # no peers. Answering `unjudged` would be worse than refusing: unjudged + # never enters a rollup, so an outlying reagent would leave its panel + # reading clean -- the failure the rollup exists to invert. + with pytest.raises(ValueError, match="outlier_status"): + status_for(measurement, 0.4, DEFAULT_LINES) + + +def test_every_measurement_either_answers_or_refuses_but_never_lies(): + # Walks the whole declared set so a route added later cannot quietly fall + # through to `unjudged`, which is the one wrong answer that hides. + for m in MEASUREMENTS: + if m.line == "against-the-run": + with pytest.raises(ValueError): + status_for(m.id, 0.4, DEFAULT_LINES) + else: + assert status_for(m.id, 0.4, DEFAULT_LINES) in set(Status) + + def test_a_lone_outlier_is_flagged_because_peers_exclude_the_value(): # If `peers` included the value, one extreme reading would inflate q3 and # could never be flagged -- the measure would defeat itself, and no fixture From 4a66204ce991a78e7827583b460ce1617f5cdabb Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 18:31:18 +0200 Subject: [PATCH 045/282] MILAB-6496: correct three QC lines against the field's published numbers A spec review checked the shipped lines against the evidence file behind the spec's "lines come from four routes" decision. Three of the four did not survive it. Operator ruled on each; decisions recorded in .meta/milab-6496-decisions.md. The undeclared-barcode line of 0.1 was invented. The field publishes 0.50, and for one aggregate library fraction -- while this measurement is per sequence at tag level, where a fraction's line does not transfer. Given a count instead, any upper bound collapses into "alerting if a single undeclared barcode exists". It now ships unjudged, with its sequences and no claim about what a bad value means. panelAssignedFraction was labelled "Fraction of antigen reads usable", whose published line is 0.20, while shipping 0.50 -- the complement of the unrecognized fraction, which is what the code actually computes. The label was the part that had drifted: the shipped p-column's own description already named the recognized fraction. Relabelled, line unchanged. The id is a p-column name and is untouched. readsPerBarcode applied a per-cell recommendation to a per-observed-barcode rate. In droplet data that denominator runs one to two orders of magnitude above the called-cell count, so a healthy library would alert. Renamed readsPerCell and now divides by the cell list, which the entrypoint supplies; no p-column existed to break. at-most keeps its place in the comparison vocabulary but now has no member, so it is exercised against a registered stand-in rather than left an untested branch. --- software/per-cell-metrics/src/qc_measures.py | 99 +++++++++++++------ .../per-cell-metrics/test/test_qc_measures.py | 54 ++++++---- 2 files changed, 105 insertions(+), 48 deletions(-) diff --git a/software/per-cell-metrics/src/qc_measures.py b/software/per-cell-metrics/src/qc_measures.py index 57f3559..05cbf6b 100644 --- a/software/per-cell-metrics/src/qc_measures.py +++ b/software/per-cell-metrics/src/qc_measures.py @@ -70,19 +70,29 @@ class Measurement: # none of them, so nothing here says what a low one would mean. "Every read the parser saw, and the share matching the tag pattern.", ), + # The label names the recognized fraction rather than the spec row's + # "usable" fraction, because that is the quantity this block computes and + # has always computed: `qc_report._refine_assigned_fraction` returns the + # refine-tags step's outputCount/inputCount, the share of reads kept after + # correcting the barcode against the panel. The field's "usable" fraction + # additionally requires a cell-associated barcode and a valid UMI, and + # carries a different published line (0.20 against this one's 0.50). The + # shipped p-column's own description already says the recognized fraction; + # only this label had drifted. The id is a p-column name and must not be + # renamed -- a p-column's identity is its name, domain and axes. Measurement( "panelAssignedFraction", - "Fraction of antigen reads usable", + "Fraction of antigen reads matching the panel", "sample", "Reads whose corrected barcode is on the panel, over reads matched.", "A low share means most reads carry barcodes the panel never declared.", "inherited", ), - # The spec's one row for saturation and reads-per-barcode covers two figures - # with different fates in this build: reads-per-barcode can be derived from - # counts this package already has, saturation cannot. One Measurement could - # not carry "computed" and "deferred" at once, so the row becomes two ids - # here, both at the row's declared level. + # The spec's one row for saturation and sequencing depth covers two figures + # with different fates in this build: depth can be derived from counts this + # package already has, saturation cannot. One Measurement could not carry + # "computed" and "deferred" at once, so the row becomes two ids here, both + # at the row's declared level. Measurement( "sequencingSaturation", "Sequencing saturation", @@ -90,11 +100,17 @@ class Measurement: "Duplicate reads over total reads.", deferred_reason="needs read-level data the per-sample fan-out discards", ), + # Per *cell*, not per observed barcode. The vendor's five thousand is a + # per-cell recommendation, and in droplet data the observed-barcode count + # exceeds the called-cell count by one to two orders of magnitude, because + # ambient antigen reads land on most barcodes -- so dividing by it would + # alert on a healthy library. The cell list is an input that arrives later + # than this module, which is why the division happens in the entrypoint. Measurement( - "readsPerBarcode", - "Reads per barcode", + "readsPerCell", + "Reads per cell", "sample", - "Reads matched, over barcodes observed.", + "Reads matched, over cells in the cell list.", "Below the vendor's recommended minimum the library is undersequenced.", "recommended-and-observed", ), @@ -111,13 +127,21 @@ class Measurement: "Reads in barcodes flagged as aggregates, over reads matched.", deferred_reason="no aggregate-barcode detection exists in this block", ), + # No line, and this one is worth spelling out because the spec looks like it + # supplies one. Atom 315 lists "the fraction of undeclared barcodes" among + # its four inherited numbers, and the field does publish 0.50 -- but for one + # aggregate library fraction. This measurement is per sequence at tag level, + # which is the improvement the spec set asks for, and a fraction's line does + # not transfer to a list of sequences. Given a count instead, any at-most + # line collapses into "alerting if a single undeclared barcode exists" -- a + # categorical predicate wearing an inherited number. So it ships unjudged, + # with its sequences and their counts, and says nothing about what a bad + # value would mean. Measurement( "undeclaredBarcodes", "Undeclared barcodes, and which sequences", "tag", "Barcodes the reads carry that the sample's panel does not declare, and which sequences they are.", - "A declared-nothing barcode carrying reads means the panel file is incomplete.", - "inherited", ), Measurement( "declaredNeverSeen", @@ -199,28 +223,31 @@ class Measurement: # applies the measurement stays unjudged rather than being given a number with # nothing behind it. DEFAULT_LINES: dict[str, float] = { - "panelAssignedFraction": 0.5, # inherited - "undeclaredBarcodes": 0.1, # inherited + "panelAssignedFraction": 0.5, # inherited: complement of the field's 0.50 unrecognized line "declaredNeverSeen": 0, # categorical: alerting at zero reads - "readsPerBarcode": 5_000, # recommended-and-observed + "readsPerCell": 5_000, # recommended-and-observed: the vendor's per-cell depth } # How each line is read. Deliberately *not* overridable: an operator moves a # number, never a direction. Three comparisons rather than one flag, because a -# floor and a categorical fact disagree at the boundary -- `readsPerBarcode` -# alerts strictly *below* the recommendation, while `declaredNeverSeen` alerts -# *at* zero. One `<=` cannot serve both. +# floor and a categorical fact disagree at the boundary -- `readsPerCell` alerts +# strictly *below* the recommendation, while `declaredNeverSeen` alerts *at* +# zero. One `<=` cannot serve both. # # at-least acceptable at or above the line, alerting strictly below # at-most acceptable at or below the line, alerting strictly above # alerting-at alerting where the value equals the line # # In every case the named value satisfies the condition it names. +# +# `at-most` currently has no member. It is kept because it is one of the three +# readings a line can have, not because something uses it: the only candidate +# was the undeclared-barcode fraction, which ships unjudged for want of a +# defensible line rather than for want of a direction. _COMPARISON: dict[str, str] = { "panelAssignedFraction": "at-least", - "undeclaredBarcodes": "at-most", "declaredNeverSeen": "alerting-at", - "readsPerBarcode": "at-least", + "readsPerCell": "at-least", } _ORDINAL = {Status.ACCEPTABLE: 0, Status.ALERTING: 1} @@ -371,9 +398,8 @@ def per_antigen_measures(states: pl.DataFrame) -> pl.DataFrame: The sparse frame is the right input here: "cells with any reading" means cells with an observed reading, and a cell silent for this tag has no count to contribute. There is no asked population to complete this - against, unlike a per-cell total or a reads-per-barcode rate, both of - which are asked-cell questions and use a densified or a whole-run count - instead. + against, unlike a per-cell total or a reads-per-cell rate, both of which + are asked-cell questions and use a densified or a whole-run count instead. """ return ( states.group_by("tag") @@ -386,19 +412,30 @@ def per_antigen_measures(states: pl.DataFrame) -> pl.DataFrame: ) -def reads_per_barcode(reads_matched: int, barcodes_observed: int) -> float | None: - """Reads matched, over barcodes observed. +def reads_per_cell(reads_matched: int, cells_in_list: int) -> float | None: + """Reads matched, over cells in the cell list. + + The denominator is the **cell list**, not the barcodes the reads happened to + touch. The vendor's five-thousand recommendation this is judged against is + per called cell, and in droplet data the observed-barcode count runs one to + two orders of magnitude higher, because ambient antigen reads land on most + barcodes. Dividing by observed barcodes would make a healthy library alert, + which is worse than not judging depth at all: a status that fires on good + runs teaches a reader to ignore it. - Both counts already exist in the per-sample QC row -- `readsMatched` and - `cellsDetected` -- so this only divides them; neither is recounted here. + `reads_matched` already exists in the per-sample QC row; the cell list is a + separate input that arrives with gene expression or with the receptors, so + the caller supplies its size. Deliberately not `cellsDetected` from that + same row -- that is the observed-barcode count this docstring exists to + warn against. - None when no barcode was observed. A rate over zero barcodes is not a - small number, it is no number, and returning None keeps that distinct - from a rate that was computed and happens to be zero. + None when the cell list is empty. A rate over no cells is not a small + number, it is no number, and returning None keeps that distinct from a rate + that was computed and happens to be zero. """ - if barcodes_observed <= 0: + if cells_in_list <= 0: return None - return reads_matched / barcodes_observed + return reads_matched / cells_in_list # The extremes are included alongside the interior deciles so the distribution's diff --git a/software/per-cell-metrics/test/test_qc_measures.py b/software/per-cell-metrics/test/test_qc_measures.py index 1fef3be..6c1bc78 100644 --- a/software/per-cell-metrics/test/test_qc_measures.py +++ b/software/per-cell-metrics/test/test_qc_measures.py @@ -17,14 +17,14 @@ measurement_rows, outlier_status, per_antigen_measures, - reads_per_barcode, + reads_per_cell, roll_up, roll_up_capture, roll_up_panel, status_for, ) -# The spec's row for sequencing saturation and reads per barcode covers two +# The spec's row for sequencing saturation and sequencing depth covers two # figures with different fates in this build -- one derivable from counts the # package already has, the other not -- so it becomes two declared ids here, # both at the row's stated level. Every other row maps one to one. This is the @@ -34,7 +34,7 @@ "readsTotal": "sample", "panelAssignedFraction": "sample", "sequencingSaturation": "sample", - "readsPerBarcode": "sample", + "readsPerCell": "sample", "antigenCountDistribution": "sample", "aggregateBarcodeFraction": "sample", "undeclaredBarcodes": "tag", @@ -228,15 +228,15 @@ def test_per_antigen_measures_differs_between_tag_and_identity_grain(): assert by_identity.row(0, named=True)["cellsAboveTheLine"] == 2 -# --- reads_per_barcode -------------------------------------------------------- +# --- reads_per_cell -------------------------------------------------------- -def test_reads_per_barcode_computes_the_rate(): - assert reads_per_barcode(1000, 200) == 5.0 +def test_reads_per_cell_computes_the_rate(): + assert reads_per_cell(1000, 200) == 5.0 -def test_reads_per_barcode_zero_barcodes_observed_does_not_divide_by_zero(): - assert reads_per_barcode(1000, 0) is None +def test_reads_per_cell_empty_cell_list_does_not_divide_by_zero(): + assert reads_per_cell(1000, 0) is None # --- antigen_count_deciles ----------------------------------------------------- @@ -377,23 +377,43 @@ def test_the_invented_matched_fraction_line_is_gone(): def test_depth_line_is_a_parameter_not_a_literal(): - assert DEFAULT_LINES["readsPerBarcode"] == 5_000 - assert status_for("readsPerBarcode", 4_000, {"readsPerBarcode": 5_000}) is Status.ALERTING - assert status_for("readsPerBarcode", 4_000, {"readsPerBarcode": 1_000}) is Status.ACCEPTABLE + assert DEFAULT_LINES["readsPerCell"] == 5_000 + assert status_for("readsPerCell", 4_000, {"readsPerCell": 5_000}) is Status.ALERTING + assert status_for("readsPerCell", 4_000, {"readsPerCell": 1_000}) is Status.ACCEPTABLE def test_at_least_is_acceptable_exactly_at_the_line(): # Atom 315 alerts *below* the recommendation, so the recommendation itself # is acceptable. The named value satisfies the condition it names. - assert status_for("readsPerBarcode", 5_000, DEFAULT_LINES) is Status.ACCEPTABLE - assert status_for("readsPerBarcode", 4_999, DEFAULT_LINES) is Status.ALERTING + assert status_for("readsPerCell", 5_000, DEFAULT_LINES) is Status.ACCEPTABLE + assert status_for("readsPerCell", 4_999, DEFAULT_LINES) is Status.ALERTING assert status_for("panelAssignedFraction", 0.5, DEFAULT_LINES) is Status.ACCEPTABLE assert status_for("panelAssignedFraction", 0.49, DEFAULT_LINES) is Status.ALERTING -def test_at_most_is_acceptable_exactly_at_the_line(): - assert status_for("undeclaredBarcodes", 0.1, DEFAULT_LINES) is Status.ACCEPTABLE - assert status_for("undeclaredBarcodes", 0.11, DEFAULT_LINES) is Status.ALERTING +def test_at_most_is_acceptable_exactly_at_the_line(monkeypatch): + # No shipped measurement reads `at-most`: the only candidate was the + # undeclared-barcode fraction, which now ships unjudged. The reading stays in + # the vocabulary because a line can be an upper bound as easily as a lower + # one, so it is exercised against a registered stand-in rather than left as + # an untested branch. + monkeypatch.setitem(_COMPARISON, "syntheticUpperBound", "at-most") + lines = {"syntheticUpperBound": 0.1} + assert status_for("syntheticUpperBound", 0.1, lines) is Status.ACCEPTABLE + assert status_for("syntheticUpperBound", 0.11, lines) is Status.ALERTING + + +def test_the_undeclared_barcode_fraction_ships_unjudged(): + # Atom 315 lists it among the four inherited numbers and the field does + # publish 0.50 -- but for one aggregate library fraction, while this + # measurement is per sequence at tag level. A fraction's line does not + # transfer to a list of sequences, and given a count any upper bound + # collapses into "alerting if a single undeclared barcode exists". + by_id = {m.id: m for m in MEASUREMENTS} + assert by_id["undeclaredBarcodes"].line is None + assert by_id["undeclaredBarcodes"].implies is None + assert "undeclaredBarcodes" not in DEFAULT_LINES + assert status_for("undeclaredBarcodes", 0.4, DEFAULT_LINES) is Status.UNJUDGED def test_categorical_alerts_only_on_the_named_fact(): @@ -412,7 +432,7 @@ def test_a_deferred_measurement_is_never_unjudged_even_holding_a_value(): def test_a_missing_value_is_not_evaluated(): - assert status_for("readsPerBarcode", None, DEFAULT_LINES) is Status.NOT_EVALUATED + assert status_for("readsPerCell", None, DEFAULT_LINES) is Status.NOT_EVALUATED # --- the against-the-run route --------------------------------------------- From 7147d8d5b1c1cc229f858b0618528c958b1a89f8 Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 19:03:13 +0200 Subject: [PATCH 046/282] MILAB-6496: add the emit-verdicts entrypoint Composes the twelve modules into one CLI: the panel reader, the identity universe, the count floor, the reference and its gate, the per-cell state reader, the cell-to-set majority vote, the competitor notes, both levels of self-disagreement, and the quality measurements with their statuses and rollups. The grid of every cell against every identity is never built. A cutoff at or below specificity_score(0, 0) is refused, because below that bound the analytic silent count and the row-per-position reference part company with no error raised. One Admissibility bundle reaches read_states, combine_cells and self_disagreement, so none of them can disagree about which cells cannot be compared. What each sample was offered stays keyed by sample throughout. Barcodes outside the cell list are labelled rather than dropped, and the sparse per-tag counts plus the per-cell scalars are exported as the material a reader needs to regroup the panel without a re-run. --- software/per-cell-metrics/package.json | 18 + .../per-cell-metrics/src/emit_verdicts.py | 924 ++++++++++++++++++ .../test/test_emit_verdicts.py | 324 ++++++ 3 files changed, 1266 insertions(+) create mode 100644 software/per-cell-metrics/src/emit_verdicts.py create mode 100644 software/per-cell-metrics/test/test_emit_verdicts.py diff --git a/software/per-cell-metrics/package.json b/software/per-cell-metrics/package.json index 23efb8f..1ba6419 100644 --- a/software/per-cell-metrics/package.json +++ b/software/per-cell-metrics/package.json @@ -108,6 +108,24 @@ ] } }, + "emit-verdicts": { + "binary": { + "artifact": { + "type": "python", + "registry": "platforma-open", + "environment": "@platforma-open/milaboratories.runenv-python-3:3.12.10", + "dependencies": { + "toolset": "pip", + "requirements": "requirements.txt" + }, + "root": "./src" + }, + "cmd": [ + "python", + "{pkg}/emit_verdicts.py" + ] + } + }, "parse-gate": { "binary": { "artifact": { diff --git a/software/per-cell-metrics/src/emit_verdicts.py b/software/per-cell-metrics/src/emit_verdicts.py new file mode 100644 index 0000000..68147a5 --- /dev/null +++ b/software/per-cell-metrics/src/emit_verdicts.py @@ -0,0 +1,924 @@ +"""The entrypoint: counts, a panel and a cell list become a four-state verdict. + +Composes the reading in one order, and the order is load-bearing at every +step: the floor works on the raw per-(cell, tag) counts; a cell's reference +reading is taken from the floored frame; tags combine into an identity by the +highest of their counts; the identity's count is read against that cell's own +reference; and a set's cells combine by majority. Reversing any pair changes +the answer -- flooring after combining would floor one reading where two were +taken, and taking the reference before the floor would compare against a +number the floor has already been applied to elsewhere. + +**The grid of every cell against every identity is never built.** A silent +cell -- one asked about an identity and showing no reading for it -- scores +`specificity_score(0, r)`, which is at most ~0.0422 and falls as the +reference rises, so it settles *not bound* unless the cell itself cannot be +compared. `silent_tally` counts those positions analytically, and this +entrypoint never materializes them: on a realistic panel the grid is 11-20x +the sparse input and a pMHC panel does not fit at all. Two consequences are +enforced here rather than downstream. A `--cutoff` at or below that ~0.0422 +bound is refused, because below it the analytic count and the row-per-position +reference disagree with no error raised. And the row-per-position reference +implementation in verdict.py is never called from production; the test suite +asserts this file does not name it. + +`offered` is keyed by SAMPLE throughout and is never regrouped by set. What a +panel offered is a property of the staining, which is done per sample; a set +spanning two samples was offered whatever either sample's panel offered, and +`combine_cells` takes that union itself. Keying the map by set instead makes +every lookup miss, reads every offered set as empty, and raises nothing. + +One `Admissibility` bundle is built and handed to `read_states`, +`combine_cells` and `self_disagreement` alike. The bundle exists so those +cannot be given different reference dicts and then disagree about which cells +"cannot be compared", which shows up as a silent-position count that is wrong +or negative rather than as an error. + +Every frame is sorted before it is written. `combine_tags_to_identities` +groups without maintaining order, so an unsorted frame varies run to run, and +a p-column's identity is its content -- an unstable byte order costs every +downstream node its dedup with nothing to show for it. +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import sys +from collections import Counter +from typing import NamedTuple + +import polars as pl +from combine import ( + DEFAULT_MIN_AGREEMENT, + DEFAULT_MIN_VOTERS, + attach_competitor_notes, + combine_cells, + self_disagreement, + set_counts, +) +from panel import ( + ANY_SAMPLE, + Grouping, + consistent_properties, + default_grouping, + identity_universe, + offered_identities, + panel_read_mismatch, + property_columns, + read_panel, +) +from qc_measures import ( + DEFAULT_LINES, + MEASUREMENTS, + Coverage, + Status, + antigen_count_deciles, + outlier_status, + per_antigen_measures, + reads_per_cell, + roll_up, + roll_up_capture, + roll_up_panel, + status_for, +) +from verdict import ( + BOUND_CUTOFF, + DEFAULT_FLOOR, + DEFAULT_HIGH_REFERENCE_OBSERVATION_LINE, + DEFAULT_PANEL_MIN_MEMBERS, + DEFAULT_REFERENCE_THIN_LINE, + Admissibility, + ReferenceChoice, + _cell_admissibility_reason, + apply_floor, + combine_tags_to_identities, + gate_cells, + read_states, + reference_by_cell, + resolve_default_source, + specificity_score, +) + +CellKey = tuple[str, str] + +# A silent cell's count is zero, and a zero count's best possible score is +# specificity_score(0, 0). At or below it the analytic silent count and the +# row-per-position reference part company over a silent admissible cell, +# quietly: one calls it bound, the other not bound, and nothing raises. +# `silent_tally` states that refusing such a cutoff belongs to the CLI. +ANALYTIC_CUTOFF_BOUND = float(specificity_score(0, 0)) + +# The pivoted per-identity summary costs one column per identity, so it is +# emitted only for panels small enough that a wide frame is still a table a +# reader can open. Declared rather than derived -- nothing published says +# where a table stops being readable -- and deliberately well under the +# thousand-plus identities a pMHC panel carries. +IDENTITY_SUMMARY_MAX_IDENTITIES = 100 + +# A rollup is reported in the same frame as the measurements it aggregates, +# as a row whose measurement is the rollup itself. A measurement is an axis +# value here, so a level's summary costs a row rather than a column. +ROLLUP = "rollup" +ROLLUP_COUNTS = "The worst status among this level's measurements, and how much of it was checked." + +MEASUREMENT_BY_ID = {m.id: m for m in MEASUREMENTS} + + +class QcRow(NamedTuple): + """One measurement at one level entity, before its declaration is attached. + + `status` and `coverage` are both carried because a measurement's own + status is not recoverable from a coverage triple: `roll_up` reports + *not evaluated* for a level with nothing judgeable in it, so a row that + was computed and left unjudged would come back saying nobody looked. The + triple says how much of the level was checked; the status says whether + what was checked is wrong. + + `panel_id` is set on tag-level and identity-level rows and left empty on + the rest: a panel carries the worst status among its per-tag and + per-identity measurements, so those rows have to say which panel they + belong to or the panel rollup has nothing to gather. + """ + + level: str + entity: str + measurement: str + value: float | None + detail: str + panel_id: str + status: Status + coverage: Coverage + + +def _write_sorted(frame: pl.DataFrame, path: str, by: list[str]) -> None: + """Write a frame in a fixed row order, header-only when it has no rows. + + Every frame reaching here is built with an explicit schema, so an empty + one still carries its columns and writes a header rather than an empty + file. A consumer meeting a header-only frame knows the step ran and found + nothing; one meeting an empty file cannot tell that from a step that + never ran. + """ + frame.sort(by).write_csv(path) + + +def _read_columns(path: str, columns: tuple[str, ...], what: str) -> pl.DataFrame: + """Read a CSV as strings, keeping the named columns and stripping them. + + Read as strings and stripped because these columns are join keys against + the panel, whose reader strips `tag` and `sample` for the same reason. A + tag written " AAAA " on one side and "AAAA" on the other joins to nothing + and reports the barcode as both undeclared and never seen. + """ + frame = pl.read_csv(path, infer_schema_length=0) + missing = [c for c in columns if c not in frame.columns] + if missing: + raise SystemExit(f"{what} {path!r} has no column(s) {missing}; columns are {frame.columns}") + return frame.select([pl.col(c).str.strip_chars().fill_null("") for c in columns]) + + +def _read_counts(path: str) -> pl.DataFrame: + counts = _read_columns(path, ("sampleId", "cellId", "tag", "umiCount"), "counts file") + return counts.with_columns(pl.col("umiCount").cast(pl.Int64)) + + +def _json_arg(raw: str | None, flag: str): + if raw is None or not raw.strip(): + return None + try: + return json.loads(raw) + except json.JSONDecodeError as exc: + raise SystemExit(f"{flag} is not valid JSON: {exc}") from exc + + +def _build_grouping( + rule: dict | None, + panel: pl.DataFrame, + properties: dict[str, dict[str, str]], + reference_tags: set[str], +) -> tuple[Grouping, str]: + """The tag -> identity map the run reads at, and the id of the rule behind it. + + A property grouping is built from `consistent_properties`, never from the + panel column: the panel reader strips `tag` and `sample` and carries + property values through exactly as written, so reading the column + directly makes " Spike " and "Spike" two identities that no fixture + without stray whitespace would ever reveal. + + Reference tags are excluded here rather than by `identity_universe`, + which takes no reference tags and never will -- one place decides, so the + two cannot drift. Leaving them in would give the comparator an identity + of its own, with a verdict read by comparing it against itself. + + A tag the grouping column says nothing about keeps its own identity + instead of vanishing. Dropping it would remove a declared reagent from + the answer with nothing downstream able to tell the panel was short. + """ + by_tag = default_grouping(panel, reference_tags) + if rule is None or rule.get("by") == "tag": + return by_tag, "per-tag" + if rule.get("by") != "property": + raise SystemExit(f"--grouping must be {{'by':'tag'}} or {{'by':'property','column':...}}; got {rule!r}") + + column = rule.get("column") or "" + declared = property_columns(panel) + if column not in declared: + raise SystemExit(f"--grouping names property column {column!r}, which the panel does not declare: {declared}") + + grouping: Grouping = {} + ungrouped: list[str] = [] + for tag in sorted(by_tag): + value = properties.get(tag, {}).get(column) + if value: + grouping[tag] = value + else: + grouping[tag] = tag + ungrouped.append(tag) + if ungrouped: + print( + f"[emit-verdicts] {len(ungrouped)} tag(s) carry no agreed {column!r} value and stand as their own " + f"identity: {ungrouped[:8]}", + file=sys.stderr, + ) + return grouping, f"property:{column}" + + +def _identity_labels( + grouping: Grouping, properties: dict[str, dict[str, str]], feature_col: str, rule_id: str +) -> dict[str, str]: + """A readable name per identity, never two identities under one name. + + Under a property grouping the identity is the property value, which is + already the name a reader recognises. Under the per-tag grouping the + identity is a barcode, so the panel's feature name stands in -- and where + two barcodes carry the same name the tag is appended, because two + identities sharing a label are two rows a reader cannot tell apart. + """ + if rule_id != "per-tag": + return {identity: identity for identity in set(grouping.values())} + names = {tag: (properties.get(tag, {}).get(feature_col) or tag) for tag in grouping} + collisions = Counter(names.values()) + return {tag: (f"{name} ({tag})" if collisions[name] > 1 else name) for tag, name in names.items()} + + +def _panel_id(tags: frozenset[str]) -> str: + """A stable id for a declared tag set. + + No panel file names its panel, so the id is derived from the sorted tag + list and is the same in every re-run of the same declaration. Where one + panel covers every sample the axis takes a single value and drops out. + """ + return hashlib.sha256("\t".join(sorted(tags)).encode()).hexdigest()[:12] + + +def _declared_by_sample(panel: pl.DataFrame, samples: list[str]) -> dict[str, frozenset[str]]: + """Each sample's declared tag set, with the unkeyed panel applying to all.""" + everywhere = set(panel.filter(pl.col("sample") == ANY_SAMPLE)["tag"].to_list()) + return { + sample: frozenset(everywhere | set(panel.filter(pl.col("sample") == sample)["tag"].to_list())) + for sample in samples + } + + +def _cells_by_set(linker: pl.DataFrame) -> dict[str, list[CellKey]]: + """Set membership from the linker, each cell listed once under its set. + + `combine_cells` asserts the map is disjoint, so a cell listed under two + sets fails loudly there rather than being counted twice into a tally that + counts every cell once. + """ + members: dict[str, list[CellKey]] = {} + seen: set[tuple[str, CellKey]] = set() + for sample_id, cell_id, set_id in linker.iter_rows(): + key = (sample_id, cell_id) + if (set_id, key) in seen: + continue + seen.add((set_id, key)) + members.setdefault(set_id, []).append(key) + return {set_id: sorted(keys) for set_id, keys in sorted(members.items())} + + +def _pivot_identity_summary(verdicts: pl.DataFrame, universe: set[str]) -> tuple[pl.DataFrame, bool]: + """The per-set verdict row, one column per identity. + + Pivoted onto the set axis alone because a column carrying an axis the + clonotype anchor does not have is dropped with no error by the block that + consumes this, so a `(set, identity)` column is invisible there. Gated on + identity count: the pivot costs a column per identity and a large panel + would turn one artifact into a thousand. + """ + if len(universe) > IDENTITY_SUMMARY_MAX_IDENTITIES or verdicts.height == 0: + sets = verdicts.select("setId").unique() if verdicts.height else pl.DataFrame(schema={"setId": pl.String}) + return sets, False + wide = verdicts.pivot(on="identity", index="setId", values="state") + return wide.select(["setId", *sorted(universe)]), True + + +def _leaf(level, entity, measurement, value, detail, panel_id, status: Status) -> QcRow: + """One measurement's row: its own status, and the coverage of that one status. + + The triple comes from `roll_up` so a leaf and a rollup are counted by one + rule, but the row keeps the status `roll_up` would have flattened. + """ + return QcRow(level, entity, measurement, value, detail, panel_id, status, roll_up([status])) + + +def _sum_coverage(status: Status, parts: list[Coverage]) -> Coverage: + """A rollup over rollups: the status from the rollup rule, the counts summed. + + `roll_up_capture` takes statuses, so handing it the statuses of levels + that were themselves rolled up gives the right status and the wrong + counts -- a sample that was fully computed but had nothing judgeable + arrives as *not evaluated* and increments the capture's not-evaluated + count, collapsing "nothing was wrong" into "nobody looked". Summing the + constituent coverages keeps the two apart. + """ + return Coverage( + status, + sum(c.judged for c in parts), + sum(c.unjudged for c in parts), + sum(c.not_evaluated for c in parts), + ) + + +def _qc_frame(rows: list[QcRow]) -> pl.DataFrame: + """The measurement set as a frame keyed (level, entity, measurement). + + Every declared measurement keeps its place whether or not this run could + compute it, and a measurement nothing computed reads *not evaluated* with + its reason rather than being absent: a reader must never mistake "nothing + computed this yet" for "this was checked and found fine". + + A field with nothing in it is written null rather than as an empty string. + polars quotes an empty string to keep it apart from a null, and a quoted + empty cell is a value a downstream import would carry as one. + """ + built = [] + for row in rows: + declared = MEASUREMENT_BY_ID.get(row.measurement) + built.append( + { + "level": row.level, + "entity": row.entity, + "panelId": row.panel_id or None, + "measurement": row.measurement, + "value": row.value, + "detail": row.detail or None, + "status": row.status.value, + "judged": row.coverage.judged, + "unjudged": row.coverage.unjudged, + "notEvaluated": row.coverage.not_evaluated, + "counts": ROLLUP_COUNTS if declared is None else declared.counts, + "implies": None if declared is None else declared.implies, + "reason": None if declared is None else declared.deferred_reason, + } + ) + return pl.DataFrame( + built, + schema={ + "level": pl.String, + "entity": pl.String, + "panelId": pl.String, + "measurement": pl.String, + "value": pl.Float64, + "detail": pl.String, + "status": pl.String, + "judged": pl.Int64, + "unjudged": pl.Int64, + "notEvaluated": pl.Int64, + "counts": pl.String, + "implies": pl.String, + "reason": pl.String, + }, + ) + + +def _add(rows: list[QcRow], level: str, entity: str, measurement: str, value, detail: str = "", panel_id: str = ""): + """Append one measurement row, taking its status from the lines in force. + + `status_for` refuses the two measurements judged against the run itself; + those are added through `outlier_status` at their own call sites and + never reach here. + """ + rows.append( + _leaf(level, entity, measurement, value, detail, panel_id, status_for(measurement, value, DEFAULT_LINES)) + ) + + +def _median(values: list[float]) -> float | None: + return float(pl.Series(values).median()) if values else None + + +# Long on purpose, and not decomposed: this is one composition taken in the +# one order the reading has, and splitting it into stages would put the order +# in the call sites rather than in the code a reader follows top to bottom. +def main() -> None: + p = argparse.ArgumentParser(description="Read antigen counts into a four-state binding verdict per set.") + p.add_argument("counts_csv", help="sparse per-(sampleId, cellId, tag) UMI counts") + p.add_argument("panel_csv", help="the panel file: which tags each sample was stained with") + p.add_argument("--linker", default=None, help="cell -> clonotype set CSV (sampleId, cellId, setId)") + p.add_argument("--cells", default=None, help="the cell list (sampleId, cellId); overrides the linker's cells") + p.add_argument("--barcode-col", default="tag", help="panel column holding the barcode sequence") + p.add_argument("--feature-col", default="feature", help="panel column holding the antigen name") + p.add_argument("--sample-col", default="", help="panel column holding the sample; empty declares one panel for all") + p.add_argument("--role-column", default="", help="panel column declaring each tag's role") + p.add_argument("--reference-values", default="", help="comma-separated role values marking a comparator tag") + p.add_argument( + "--reference-source", + default=None, + choices=["declared", "panel", "none"], + help="which comparator to ask for; the run may serve 'none' instead, never a different one", + ) + p.add_argument("--panel-min-members", type=int, default=DEFAULT_PANEL_MIN_MEMBERS) + p.add_argument("--reference-thin-line", type=int, default=DEFAULT_REFERENCE_THIN_LINE) + p.add_argument("--floor", type=int, default=DEFAULT_FLOOR, help="zero every non-comparator reading below this") + p.add_argument( + "--cutoff", type=float, default=BOUND_CUTOFF, help="specificity score at or above which a cell binds" + ) + p.add_argument("--min-voters", type=int, default=DEFAULT_MIN_VOTERS) + p.add_argument("--min-agreement", type=float, default=DEFAULT_MIN_AGREEMENT) + p.add_argument("--gate-threshold", type=int, default=None, help="set aside cells whose comparator reads this high") + p.add_argument("--high-reference-line", type=int, default=DEFAULT_HIGH_REFERENCE_OBSERVATION_LINE) + p.add_argument("--grouping", default=None, help="JSON: {'by':'tag'} or {'by':'property','column':...}") + p.add_argument("--contending", default=None, help="JSON: groups of identities that contend, as a list of lists") + p.add_argument("--capture-map", default=None, help="JSON: sampleId -> captureId") + p.add_argument( + "--qc-summary", default=None, help="per-sample read QC CSV (sampleId, readsTotal, readsMatched, ...)" + ) + p.add_argument("--output-prefix", default="result") + args = p.parse_args() + + if args.cutoff <= ANALYTIC_CUTOFF_BOUND: + raise SystemExit( + f"--cutoff must be strictly above {ANALYTIC_CUTOFF_BOUND:.4f}, the best score a zero count can reach. " + f"At or below it a cell that was asked and read nothing settles one way when counted and the other " + f"when written out, with no error raised. Got {args.cutoff}." + ) + + prefix = args.output_prefix + roles = {"barcode": args.barcode_col, "feature": args.feature_col} + if args.sample_col: + roles["sample"] = args.sample_col + panel, dropped_lines = read_panel(args.panel_csv, roles) + + prop_cols = property_columns(panel) + properties, inconsistent = consistent_properties(panel, prop_cols) + for tag, column, values in inconsistent: + print( + f"[emit-verdicts] tag {tag!r} declares {column!r} as {values}; it carries no agreed value", file=sys.stderr + ) + + # The reference designation is read through `consistent_properties`, which + # strips the value and drops any property a tag's rows disagree about. A + # per-sample comparator designation is therefore discarded rather than + # honoured, which is what `apply_floor` documents: a tag is a comparator in + # every sample or in none. + reference_values = {v.strip() for v in args.reference_values.split(",") if v.strip()} + reference_tags: set[str] = set() + if args.role_column and reference_values: + if args.role_column not in prop_cols: + raise SystemExit(f"--role-column {args.role_column!r} is not a panel column; columns are {prop_cols}") + reference_tags = {t for t, props in properties.items() if props.get(args.role_column) in reference_values} + + grouping_rule = _json_arg(args.grouping, "--grouping") + grouping, grouping_id = _build_grouping(grouping_rule, panel, properties, reference_tags) + universe = identity_universe(panel, grouping) + by_tag_grouping = default_grouping(panel, reference_tags) + tag_universe = identity_universe(panel, by_tag_grouping) + + contending_raw = _json_arg(args.contending, "--contending") or [] + contending = [set(group) for group in contending_raw] + capture_of_sample: dict[str, str] = _json_arg(args.capture_map, "--capture-map") or {} + + counts = _read_counts(args.counts_csv) + + # The cell list is an input and never derived from the antigen readings: + # nothing in the counts separates a cell from a droplet that held none. + # `--cells` wins over the linker where both arrive, because a list from + # gene expression covers cells whose receptor never assembled and the + # linker cannot. + linker = ( + _read_columns(args.linker, ("sampleId", "cellId", "setId"), "linker file") + if args.linker + else pl.DataFrame(schema={"sampleId": pl.String, "cellId": pl.String, "setId": pl.String}) + ) + cells_by_set = _cells_by_set(linker) + linker_cells = {key for keys in cells_by_set.values() for key in keys} + if args.cells: + listed = _read_columns(args.cells, ("sampleId", "cellId"), "cell list") + cell_list = set(listed.iter_rows()) + cell_list_source = "cell list" + elif args.linker: + cell_list = linker_cells + cell_list_source = "clonotype linker" + else: + cell_list = set(counts.select("sampleId", "cellId").unique().iter_rows()) + cell_list_source = "observed barcodes" + + observed_cells = set(counts.select("sampleId", "cellId").unique().iter_rows()) + # Barcodes outside the cell list stay in the frame, labelled: one dropped + # here is indistinguishable afterwards from one that never existed, and + # its antigen counts are real whatever the list says about it. + analysed_cells = sorted(cell_list | observed_cells | linker_cells) + + panel_samples = {s for s in panel["sample"].to_list() if s != ANY_SAMPLE} + samples = sorted( + {s for s, _ in observed_cells} | {s for s, _ in cell_list} | {s for s, _ in linker_cells} | panel_samples + ) + + # The floor is applied per sample so the counters it returns land in each + # sample's own QC row. A cell key carries its sample, so partitioning is + # exact on both counters and the run totals are their sums -- there is no + # second implementation of the rule to drift from this one. + floor_stats: dict[str, dict[str, int]] = {} + parts = [] + for sample in samples: + floored_part = apply_floor(counts.filter(pl.col("sampleId") == sample), args.floor, reference_tags) + parts.append(floored_part.counts) + floor_stats[sample] = floored_part.stats + floored = pl.concat(parts) if parts else counts + readings_floored = sum(s["readingsFloored"] for s in floor_stats.values()) + cells_emptied = sum(s["cellsEmptied"] for s in floor_stats.values()) + + source = ReferenceChoice[args.reference_source.upper()] if args.reference_source else None + if source is None: + source = resolve_default_source(reference_tags) + reference = reference_by_cell( + floored, + reference_tags, + source, + cells=analysed_cells, + panel_size=int(panel["tag"].n_unique()), + min_members=args.panel_min_members, + ) + gated, cells_high_reference = gate_cells(reference.by_cell, args.gate_threshold, args.high_reference_line) + + # Built once and handed to every consumer. Two bundles built from two + # reference dicts do not raise; they disagree about which cells cannot be + # compared, and the silent-position count comes out wrong or negative. + admissibility = Admissibility(reference.by_cell, args.reference_thin_line, gated) + + non_reference = floored.filter(~pl.col("tag").is_in(list(reference_tags))) if reference_tags else floored + identities = combine_tags_to_identities(non_reference, grouping) + states = read_states(identities, admissibility, args.cutoff) + + # The per-tag reading is diagnostic only -- it compares each tag against + # the reference separately, which no verdict is built from -- but the + # measurement set carries it at both levels always, so where the chosen + # grouping is not the per-tag one it is read a second time. + if grouping == by_tag_grouping: + tag_states = states + else: + tag_states = read_states(combine_tags_to_identities(non_reference, by_tag_grouping), admissibility, args.cutoff) + + offered_by_sample = {s: offered_identities(panel, grouping, [s]) for s in samples} + tag_offered_by_sample = {s: offered_identities(panel, by_tag_grouping, [s]) for s in samples} + + verdicts = attach_competitor_notes( + combine_cells( + states, + universe, + offered_by_sample, + cells_by_set, + admissibility, + args.min_voters, + args.min_agreement, + ), + contending, + ) + _write_sorted(verdicts, f"{prefix}_verdicts.csv", ["setId", "identity"]) + _write_sorted(set_counts(verdicts), f"{prefix}_set_counts.csv", ["setId"]) + + summary, summary_emitted = _pivot_identity_summary(verdicts, universe) + _write_sorted(summary, f"{prefix}_identity_summary.csv", ["setId"]) + + # The re-derivation material: the sparse per-tag counts and the per-cell + # scalars together reproduce every per-cell state exactly, at a small + # fraction of the size a per-cell-per-identity table would take. A reader + # regrouping the panel re-takes the highest member, re-scores against the + # same reference, and re-votes, without a re-run. + in_list = pl.DataFrame( + [(s, c, "true") for s, c in sorted(cell_list)], + orient="row", + schema={"sampleId": pl.String, "cellId": pl.String, "inCellList": pl.String}, + ) + reference_frame = pl.DataFrame( + [(s, c, reference.by_cell.get((s, c))) for s, c in analysed_cells], + orient="row", + schema={"sampleId": pl.String, "cellId": pl.String, "referenceCount": pl.Int64}, + ) + cell_counts = ( + non_reference.join(reference_frame, on=["sampleId", "cellId"], how="left") + .join(in_list, on=["sampleId", "cellId"], how="left") + .with_columns(pl.col("inCellList").fill_null("false")) + .select(["sampleId", "cellId", "tag", "umiCount", "referenceCount", "inCellList"]) + ) + _write_sorted(cell_counts, f"{prefix}_cell_counts.csv", ["sampleId", "cellId", "tag"]) + + cell_scalars = ( + reference_frame.join(in_list, on=["sampleId", "cellId"], how="left") + .with_columns(pl.col("inCellList").fill_null("false")) + .with_columns( + pl.Series( + "admissibility", + [ + (lambda reason: "admissible" if reason is None else reason.value)( + _cell_admissibility_reason(key, admissibility) + ) + for key in analysed_cells + ], + dtype=pl.String, + ) + ) + .select(["sampleId", "cellId", "referenceCount", "admissibility", "inCellList"]) + ) + _write_sorted(cell_scalars, f"{prefix}_cell_scalars.csv", ["sampleId", "cellId"]) + + offered_frame = pl.DataFrame( + [(sample, identity) for sample in samples for identity in sorted(offered_by_sample[sample])], + orient="row", + schema={"sampleId": pl.String, "identity": pl.String}, + ) + _write_sorted(offered_frame, f"{prefix}_offered.csv", ["sampleId", "identity"]) + + linker_frame = pl.DataFrame( + sorted(grouping.items()), orient="row", schema={"tag": pl.String, "identity": pl.String} + ) + _write_sorted(linker_frame, f"{prefix}_tag_identity.csv", ["tag", "identity"]) + + labels = _identity_labels(grouping, properties, args.feature_col, grouping_id) + identity_labels = pl.DataFrame( + [(identity, labels.get(identity, identity)) for identity in sorted(universe)], + orient="row", + schema={"identity": pl.String, "label": pl.String}, + ) + _write_sorted(identity_labels, f"{prefix}_identity_labels.csv", ["identity"]) + + declared = _declared_by_sample(panel, samples) + panel_of_sample = {sample: _panel_id(tags) for sample, tags in declared.items()} + tags_of_panel: dict[str, frozenset[str]] = {panel_of_sample[s]: declared[s] for s in samples} + samples_of_panel: dict[str, list[str]] = {} + for sample in samples: + samples_of_panel.setdefault(panel_of_sample[sample], []).append(sample) + + panel_labels = pl.DataFrame( + [ + (panel_id, f"{len(tags_of_panel[panel_id])} tags: {', '.join(samples_of_panel[panel_id])}") + for panel_id in sorted(tags_of_panel) + ], + orient="row", + schema={"panelId": pl.String, "label": pl.String}, + ) + _write_sorted(panel_labels, f"{prefix}_panel_labels.csv", ["panelId"]) + + sample_panel = pl.DataFrame( + [(sample, panel_of_sample[sample]) for sample in samples], + orient="row", + schema={"sampleId": pl.String, "panelId": pl.String}, + ) + _write_sorted(sample_panel, f"{prefix}_sample_panel.csv", ["sampleId"]) + + # Both directions of the panel-versus-reads check, re-keyed onto the + # panel: a per-tag failure is a property of the declared tag set rather + # than of any one sample that carries it. The samples reporting it travel + # in the row so nothing about where it was seen is lost. + seen = counts.select("sampleId", "tag").unique() + unknown_panel = _panel_id(frozenset()) + mismatch_rows: dict[tuple[str, str, str], set[str]] = {} + for row in panel_read_mismatch(panel, seen).iter_rows(named=True): + # In the unkeyed case every row comes back under "*", which is not a + # sample id: the declaration really is global, so it reports against + # every sample in the run. + affected = samples if row["sample"] == ANY_SAMPLE else [row["sample"]] + for sample in affected: + key = (panel_of_sample.get(sample, unknown_panel), row["tag"], row["direction"]) + mismatch_rows.setdefault(key, set()).add(sample) + mismatch = pl.DataFrame( + [(panel_id, tag, direction, ", ".join(sorted(s))) for (panel_id, tag, direction), s in mismatch_rows.items()], + orient="row", + schema={"panelId": pl.String, "tag": pl.String, "direction": pl.String, "samples": pl.String}, + ) + _write_sorted(mismatch, f"{prefix}_panel_mismatch.csv", ["panelId", "direction", "tag"]) + + # ---- the quality measurements ------------------------------------------------- + + identity_dis = self_disagreement( + states.select("sampleId", "cellId", pl.col("identity").alias("key"), "state"), + universe, + offered_by_sample, + cells_by_set, + admissibility, + "identity", + ) + tag_dis = self_disagreement( + tag_states.select("sampleId", "cellId", pl.col("identity").alias("key"), "state"), + tag_universe, + tag_offered_by_sample, + cells_by_set, + admissibility, + "tag", + ) + + read_qc: dict[str, dict] = {} + if args.qc_summary: + for row in pl.read_csv(args.qc_summary, infer_schema_length=0).iter_rows(named=True): + read_qc[str(row.get("sampleId", "")).strip()] = row + + def _number(row: dict, column: str) -> float | None: + raw = row.get(column) + if raw is None or str(raw).strip() == "": + return None + return float(raw) + + rows: list[QcRow] = [] + sample_coverage: dict[str, Coverage] = {} + for sample in samples: + first = len(rows) + sample_counts = counts.filter(pl.col("sampleId") == sample) + listed_here = [key for key in sorted(cell_list) if key[0] == sample] + qc = read_qc.get(sample, {}) + + reads_matched = _number(qc, "readsMatched") + matched_detail = "" if reads_matched is None else f"readsMatched={int(reads_matched)}" + _add(rows, "sample", sample, "readsTotal", _number(qc, "readsTotal"), matched_detail) + _add(rows, "sample", sample, "panelAssignedFraction", _number(qc, "panelAssignedFraction")) + _add(rows, "sample", sample, "sequencingSaturation", None) + # The denominator is the cell list, never the barcodes the reads + # happened to touch: the five-thousand recommendation is per called + # cell, and in droplet data observed barcodes run one to two orders of + # magnitude higher, so dividing by them would alert on a healthy run. + depth = reads_per_cell(int(reads_matched), len(listed_here)) if reads_matched is not None else None + _add(rows, "sample", sample, "readsPerCell", depth, f"cellsInList={len(listed_here)}") + + deciles = antigen_count_deciles(sample_counts) + decile_detail = "|".join( + f"{d}:{'' if v is None else round(v, 3)}" for d, v in zip(deciles["decile"], deciles["value"], strict=True) + ) + middle = deciles.filter(pl.col("decile") == 50)["value"].to_list() + _add(rows, "sample", sample, "antigenCountDistribution", middle[0] if middle else None, decile_detail) + _add(rows, "sample", sample, "aggregateBarcodeFraction", None) + + stats = floor_stats.get(sample, {"readingsFloored": 0, "cellsEmptied": 0}) + _add( + rows, + "sample", + sample, + "floorRemoved", + float(stats["readingsFloored"]), + f"cellsEmptied={stats['cellsEmptied']}", + ) + + listed_totals = ( + sample_counts.join(in_list, on=["sampleId", "cellId"], how="semi") + .group_by("cellId") + .agg(pl.col("umiCount").sum().alias("total"))["total"] + .to_list() + ) + _add( + rows, + "sample", + sample, + "uniqueCountsPerCell", + _median([float(v) for v in listed_totals]), + f"cellsWithAReading={len(listed_totals)}", + ) + + here = {key: value for key, value in reference.by_cell.items() if key[0] == sample} + _, high_here = gate_cells(here, None, args.high_reference_line) + _add(rows, "sample", sample, "highReferenceCells", float(high_here), f"cellsWithAComparator={len(here)}") + _add(rows, "sample", sample, "knownAnswerRecovered", None) + + sample_coverage[sample] = roll_up([r.status for r in rows[first:]]) + + tag_rate = dict(zip(tag_dis["key"].to_list(), tag_dis["disagreementRate"].to_list(), strict=True)) + identity_rate = dict(zip(identity_dis["key"].to_list(), identity_dis["disagreementRate"].to_list(), strict=True)) + identities_of_panel: dict[str, set[str]] = { + panel_id: {grouping[t] for t in tags if t in grouping} for panel_id, tags in tags_of_panel.items() + } + per_sample_tag_total = { + (row["sampleId"], row["tag"]): row["total"] + for row in counts.group_by(["sampleId", "tag"]) + .agg(pl.col("umiCount").sum().alias("total")) + .iter_rows(named=True) + } + + panel_coverage: dict[str, Coverage] = {} + for panel_id in sorted(tags_of_panel): + first = len(rows) + panel_samples_here = samples_of_panel[panel_id] + panel_tags = tags_of_panel[panel_id] + here_total = { + tag: float(sum(per_sample_tag_total.get((s, tag), 0) for s in panel_samples_here)) + for tag in {t for (s, t) in per_sample_tag_total if s in panel_samples_here} | set(panel_tags) + } + observed_here = {tag for tag, total in here_total.items() if total > 0} + + # A declared tag is alerting at zero reads, so every declared tag gets + # a row rather than only the ones that produced nothing: reporting + # only the failures leaves a reader unable to tell a clean panel from + # an unchecked one. + for tag in sorted(panel_tags): + _add(rows, "tag", tag, "declaredNeverSeen", here_total[tag], "", panel_id) + for tag in sorted(observed_here - panel_tags): + _add(rows, "tag", tag, "undeclaredBarcodes", here_total[tag], "", panel_id) + + panel_states = tag_states.filter(pl.col("sampleId").is_in(panel_samples_here)).rename({"identity": "tag"}) + for row in per_antigen_measures(panel_states).iter_rows(named=True): + _add( + rows, + "tag", + row["tag"], + "perAntigen", + float(row["cellsAboveTheLine"]), + f"cellsWithSignal={row['cellsWithSignal']}|medianAboveTheLine={row['medianAboveTheLine']}", + panel_id, + ) + + # Judged against the run rather than against a line, so `status_for` + # refuses these and `outlier_status` answers instead. The peers are + # the other members of the same panel and never include the value + # being judged: including it would inflate the upper quartile it is + # then measured against, so the one reading the measure exists to + # catch is the one it would miss. + for tag in sorted(panel_tags & set(tag_rate)): + peers = [tag_rate[o] for o in panel_tags if o != tag and tag_rate.get(o) is not None] + status = outlier_status(tag_rate[tag], peers) + rows.append(_leaf("tag", tag, "tagDisagreement", tag_rate[tag], "", panel_id, status)) + tag_statuses = [r.status for r in rows[first:]] + + identity_first = len(rows) + panel_identities = identities_of_panel[panel_id] + for identity in sorted(panel_identities & set(identity_rate)): + peers = [identity_rate[o] for o in panel_identities if o != identity and identity_rate.get(o) is not None] + status = outlier_status(identity_rate[identity], peers) + rows.append( + _leaf("identity", identity, "identityDisagreement", identity_rate[identity], "", panel_id, status) + ) + identity_statuses = [r.status for r in rows[identity_first:]] + panel_coverage[panel_id] = roll_up_panel(tag_statuses, identity_statuses) + + for sample in samples: + coverage = sample_coverage[sample] + rows.append(QcRow("sample", sample, ROLLUP, None, "", "", coverage.status, coverage)) + for panel_id in sorted(panel_coverage): + coverage = panel_coverage[panel_id] + rows.append(QcRow("panel", panel_id, ROLLUP, None, "", panel_id, coverage.status, coverage)) + + # The capture axis ships whether or not a capture assignment reached the + # block: adding an axis to a released column changes its identity, adding + # a value does not. With no assignment the single row reads *not + # evaluated*, which is what it is -- nobody looked -- and never an absence. + captures: dict[str, list[str]] = {} + for sample in samples: + captures.setdefault(capture_of_sample.get(sample, "unassigned"), []).append(sample) + if not capture_of_sample: + captures = {"unassigned": []} + for capture, its_samples in sorted(captures.items()): + its_panels = sorted({panel_of_sample[s] for s in its_samples}) + from_samples = [sample_coverage[s] for s in its_samples] + from_panels = [panel_coverage[p] for p in its_panels] + worst = roll_up_capture([c.status for c in from_samples], [c.status for c in from_panels]).status + coverage = _sum_coverage(worst, from_samples + from_panels) + rows.append(QcRow("capture", capture, ROLLUP, None, "", "", worst, coverage)) + + _write_sorted(_qc_frame(rows), f"{prefix}_qc.csv", ["level", "entity", "panelId", "measurement"]) + + meta = { + "referenceChoice": reference.served.value, + "referenceSourceRequested": source.value, + "cellListSource": cell_list_source, + "cellsInList": len(cell_list), + "cellsAnalysed": len(analysed_cells), + "floor": args.floor, + "cutoff": args.cutoff, + "minVoters": args.min_voters, + "minAgreement": args.min_agreement, + "gateThreshold": args.gate_threshold, + "highReferenceLine": args.high_reference_line, + "panelMinMembers": args.panel_min_members, + "referenceThinLine": args.reference_thin_line, + "roleColumn": args.role_column, + "referenceValues": sorted(reference_values), + "referenceTags": sorted(reference_tags), + "grouping": grouping_rule or {"by": "tag"}, + "groupingId": grouping_id, + "contending": [sorted(group) for group in contending], + "identityCount": len(universe), + "identitySummaryEmitted": summary_emitted, + "identitySummaryLimit": IDENTITY_SUMMARY_MAX_IDENTITIES, + "readingsFloored": readings_floored, + "cellsEmptied": cells_emptied, + "cellsHighReference": cells_high_reference, + "cellsSetAside": len(gated), + "panelLinesDropped": dropped_lines, + "samples": samples, + "setCount": len(cells_by_set), + } + with open(f"{prefix}_run_meta.json", "w") as out: + json.dump(meta, out, indent=2, sort_keys=True) + + +if __name__ == "__main__": + main() diff --git a/software/per-cell-metrics/test/test_emit_verdicts.py b/software/per-cell-metrics/test/test_emit_verdicts.py new file mode 100644 index 0000000..5562799 --- /dev/null +++ b/software/per-cell-metrics/test/test_emit_verdicts.py @@ -0,0 +1,324 @@ +import json +import subprocess +import sys +from pathlib import Path + +import polars as pl +import pytest + +SRC = Path(__file__).resolve().parents[1] / "src" + + +def _run(cwd, *args): + return subprocess.run( + [sys.executable, str(SRC / "emit_verdicts.py"), *map(str, args)], cwd=cwd, capture_output=True, text=True + ) + + +BASE = [ + "counts.csv", + "panel.csv", + "--linker", + "linker.csv", + "--barcode-col", + "Sequence", + "--feature-col", + "Name", + "--sample-col", + "Samples", + "--role-column", + "Type", + "--reference-values", + "Control", + "--output-prefix", + "result", +] + + +@pytest.fixture +def bed(tmp_path): + # The antigen counts clear the shipped cutoff of 75 against a reference of + # 6: specificity_score(500, 6) and specificity_score(600, 6) are both 100, + # while a silent cell scores specificity_score(0, 6), which is ~7.5e-09. + # Counts of 50 and 60 score 3.1 and 7.2 and would read *not bound*, which + # is a fact about the beta score rather than about this pipeline. + (tmp_path / "counts.csv").write_text( + "sampleId,cellId,tag,umiCount\nS1,c1,AAAA,500\nS1,c1,CTRL,6\nS1,c2,AAAA,600\nS1,c2,CTRL,6\nS1,c3,CTRL,6\n" + ) # c3 was asked about AAAA and read nothing + (tmp_path / "panel.csv").write_text("Samples,Name,Sequence,Type\nS1,AgA,AAAA,Target\nS1,Ctrl,CTRL,Control\n") + (tmp_path / "linker.csv").write_text("sampleId,cellId,setId\nS1,c1,K1\nS1,c2,K1\nS1,c3,K1\n") + return tmp_path + + +def test_writes_every_artifact(bed): + r = _run(bed, *BASE) + assert r.returncode == 0, r.stderr + for name in ( + "result_verdicts.csv", + "result_set_counts.csv", + "result_cell_counts.csv", + "result_cell_scalars.csv", + "result_offered.csv", + "result_panel_mismatch.csv", + "result_run_meta.json", + ): + assert (bed / name).exists(), name + + +def test_a_silent_cell_votes_not_bound(bed): + # c3 has no AAAA row in the counts. It was offered AAAA, so it must vote. + _run(bed, *BASE) + v = pl.read_csv(bed / "result_verdicts.csv") + r = v.filter(pl.col("identity") == "AAAA").row(0, named=True) + assert r["cellsAnswered"] == 3 # not 2 + assert r["state"] == "bound" # 2 of 3 + + +def test_the_reference_tag_gets_no_verdict(bed): + _run(bed, *BASE) + v = pl.read_csv(bed / "result_verdicts.csv") + assert "CTRL" not in v["identity"].to_list() + + +def test_cell_counts_carry_the_re_derivation_material(bed): + _run(bed, *BASE) + c = pl.read_csv(bed / "result_cell_counts.csv") + assert {"sampleId", "cellId", "tag", "umiCount", "referenceCount", "inCellList"} <= set(c.columns) + + +def test_no_score_leaves_the_block(bed): + _run(bed, *BASE) + for f in ("result_cell_scalars.csv", "result_verdicts.csv", "result_cell_counts.csv"): + assert "score" not in pl.read_csv(bed / f).columns + + +def test_run_meta_records_every_choice(bed): + _run(bed, *BASE) + m = json.loads((bed / "result_run_meta.json").read_text()) + for key in ( + "referenceChoice", + "cellListSource", + "floor", + "cutoff", + "minVoters", + "gateThreshold", + "panelMinMembers", + "referenceThinLine", + "grouping", + "contending", + "readingsFloored", + "cellsEmptied", + "cellsHighReference", + ): + assert key in m, key + + +def test_reference_source_none_produces_unreliable_not_a_crash(bed): + r = _run(bed, *BASE, "--reference-source", "none") + assert r.returncode == 0, r.stderr + v = pl.read_csv(bed / "result_verdicts.csv") + assert v.filter(pl.col("identity") == "AAAA").row(0, named=True)["state"] == "unreliable" + + +def test_contending_groups_reach_the_note(bed): + (bed / "panel.csv").write_text((bed / "panel.csv").read_text() + "S1,AgB,CCCC,Target\n") + (bed / "counts.csv").write_text((bed / "counts.csv").read_text() + "S1,c1,CCCC,1\nS1,c2,CCCC,1\nS1,c3,CCCC,1\n") + _run(bed, *BASE, "--contending", json.dumps([["AAAA", "CCCC"]])) + # Read without schema inference: the flag is a literal "true"/"false" + # string, which is what a boolean p-column value has to be here, and + # polars would otherwise infer the column back into a Boolean and hide + # whether the file carries the string at all. + v = pl.read_csv(bed / "result_verdicts.csv", infer_schema_length=0) + r = v.filter(pl.col("identity") == "CCCC").row(0, named=True) + assert r["competedWith"] == "AAAA" and r["wasCompeted"] == "true" + + +def test_barcode_outside_the_cell_list_is_labelled_not_dropped(bed): + (bed / "counts.csv").write_text((bed / "counts.csv").read_text() + "S1,zzz,AAAA,99\n") + _run(bed, *BASE) + # Without schema inference, for the reason given in the contending test. + c = pl.read_csv(bed / "result_cell_counts.csv", infer_schema_length=0) + assert "zzz" in c["cellId"].to_list() + assert c.filter(pl.col("cellId") == "zzz").row(0, named=True)["inCellList"] == "false" + + +def test_undeclared_barcode_is_reported_and_does_not_stop_the_reading(bed): + (bed / "counts.csv").write_text((bed / "counts.csv").read_text() + "S1,c1,TTTT,99\n") + r = _run(bed, *BASE) + assert r.returncode == 0 + m = pl.read_csv(bed / "result_panel_mismatch.csv") + assert "TTTT" in m.filter(pl.col("direction") == "undeclared-in-panel")["tag"].to_list() + assert pl.read_csv(bed / "result_verdicts.csv").height > 0 + + +def test_empty_join_writes_headers(bed): + (bed / "linker.csv").write_text("sampleId,cellId,setId\nS1,zzz,K9\n") + r = _run(bed, *BASE) + assert r.returncode == 0, r.stderr + assert {"setId", "identity", "state"} <= set(pl.read_csv(bed / "result_verdicts.csv").columns) + + +def test_a_cutoff_at_the_analytic_floor_is_refused(bed): + # At or below specificity_score(0, 0) the analytic tally and the dense + # oracle disagree about a silent admissible cell with no error raised. + # `silent_tally` states that refusing such a cutoff is this CLI's job. + r = _run(bed, *BASE, "--cutoff", "0.04") + assert r.returncode != 0 + assert "0.042" in (r.stderr + r.stdout) + assert _run(bed, *BASE, "--cutoff", "0.05").returncode == 0 + + +def test_the_dense_oracle_is_not_reachable_from_the_entrypoint(): + # The dense oracle exists to check the analytic tally in tests. On a + # realistic panel the grid it builds is 11-20x the sparse input, so a + # production caller is a memory failure waiting for a big panel. + assert "densify" not in (SRC / "emit_verdicts.py").read_text() + + +def test_property_grouping_normalises_and_excludes_the_reference(bed): + # The stray whitespace is the point: `read_panel` normalises tag and + # sample and leaves properties alone, so a builder reading the column + # directly makes " Spike " and "Spike" two identities. Built on + # `consistent_properties` it makes one. + (bed / "panel.csv").write_text( + "Samples,Name,Sequence,Type,Family\n" + "S1,AgA,AAAA,Target, Spike \n" + "S1,AgB,CCCC,Target,Spike\n" + "S1,Ctrl,CTRL,Control,Reference\n" + ) + (bed / "counts.csv").write_text((bed / "counts.csv").read_text() + "S1,c1,CCCC,40\nS1,c2,CCCC,40\nS1,c3,CCCC,1\n") + r = _run(bed, *BASE, "--grouping", json.dumps({"by": "property", "column": "Family"})) + assert r.returncode == 0, r.stderr + identities = set(pl.read_csv(bed / "result_verdicts.csv")["identity"].to_list()) + assert identities == {"Spike"} # one identity, not " Spike " and "Spike" + assert "Reference" not in identities # the comparator is never a candidate + + +DECLARED_FLAGS = ( + "--linker", + "--cells", + "--barcode-col", + "--feature-col", + "--sample-col", + "--role-column", + "--reference-values", + "--reference-source", + "--panel-min-members", + "--reference-thin-line", + "--floor", + "--cutoff", + "--min-voters", + "--min-agreement", + "--gate-threshold", + "--high-reference-line", + "--grouping", + "--contending", + "--capture-map", + "--output-prefix", +) + + +def test_every_declared_flag_is_reachable_from_the_command_line(bed): + # Every parameter of the reading is threaded from the workflow, so a + # parameter that exists only as a module default is one a scientist + # cannot move. The help text is the cheapest place the whole set is + # visible at once. + help_text = _run(bed, "--help").stdout + for flag in DECLARED_FLAGS: + assert flag in help_text, flag + + +def test_output_is_byte_stable_across_runs(bed): + # `combine_tags_to_identities` groups without maintaining order, so an + # unsorted frame varies run to run. A p-column's identity is content + # addressed, so an unstable byte order silently costs every downstream + # node its dedup. + _run(bed, *BASE) + first = {p.name: p.read_bytes() for p in bed.glob("result_*")} + _run(bed, *BASE) + second = {p.name: p.read_bytes() for p in bed.glob("result_*")} + assert first == second + + +def test_a_computed_but_unjudged_measurement_is_not_reported_as_unchecked(bed): + # `roll_up` answers *not evaluated* for a level with nothing judgeable in + # it, which is right for a level and wrong for the measurement itself: a + # measurement that WAS computed and carries no defensible line is + # unjudged, and reporting it as not evaluated collapses "nothing was + # wrong" into "nobody looked" -- the one distinction the status set + # exists to keep apart. The row keeps its own status; the triple beside + # it says how much was checked. + _run(bed, *BASE) + qc = pl.read_csv(bed / "result_qc.csv", infer_schema_length=0) + floor_row = qc.filter(pl.col("measurement") == "floorRemoved").row(0, named=True) + assert floor_row["status"] == "unjudged" + assert (floor_row["judged"], floor_row["unjudged"], floor_row["notEvaluated"]) == ("0", "1", "0") + + deferred = qc.filter(pl.col("measurement") == "sequencingSaturation").row(0, named=True) + assert deferred["status"] == "not evaluated" + assert deferred["reason"] # a deferred measurement says why nothing computed it + + +def test_a_capture_rollup_sums_the_coverage_it_aggregates(bed): + # `roll_up_capture` takes statuses, so a sample that was fully computed + # but had nothing judgeable arrives as *not evaluated* and would + # increment the capture's not-evaluated count by one, losing every + # measurement behind it. The counts are summed from the constituent + # coverages instead. + _run(bed, *BASE, "--capture-map", json.dumps({"S1": "C1"})) + qc = pl.read_csv(bed / "result_qc.csv", infer_schema_length=0) + rollups = qc.filter(pl.col("measurement") == "rollup") + capture = rollups.filter(pl.col("level") == "capture").row(0, named=True) + assert capture["entity"] == "C1" + + def _triple(level): + r = rollups.filter(pl.col("level") == level) + return [int(r[c].cast(pl.Int64).sum()) for c in ("judged", "unjudged", "notEvaluated")] + + assert [int(capture[c]) for c in ("judged", "unjudged", "notEvaluated")] == [ + s + p for s, p in zip(_triple("sample"), _triple("panel"), strict=True) + ] + + +def test_a_cell_list_of_its_own_overrides_the_linker_and_is_recorded(bed): + # The cell list is an input; the linker only says which set a cell + # belongs to. A list from gene expression covers cells whose receptor + # never assembled, which the linker structurally cannot, so which list a + # figure was computed against has to travel with the run. + (bed / "cells.csv").write_text("sampleId,cellId\nS1,c1\nS1,c2\n") + r = _run(bed, *BASE, "--cells", "cells.csv") + assert r.returncode == 0, r.stderr + meta = json.loads((bed / "result_run_meta.json").read_text()) + assert meta["cellListSource"] == "cell list" and meta["cellsInList"] == 2 + scalars = pl.read_csv(bed / "result_cell_scalars.csv", infer_schema_length=0) + assert scalars.filter(pl.col("cellId") == "c3").row(0, named=True)["inCellList"] == "false" + + +def test_a_gate_sets_cells_aside_and_says_how_many(bed): + r = _run(bed, *BASE, "--gate-threshold", "5") + assert r.returncode == 0, r.stderr + meta = json.loads((bed / "result_run_meta.json").read_text()) + assert meta["cellsSetAside"] == 3 # every one of c1, c2 and c3 reads the comparator at 6 + v = pl.read_csv(bed / "result_verdicts.csv") + assert v.filter(pl.col("identity") == "AAAA").row(0, named=True)["state"] == "unreliable" + + +def test_the_floor_runs_before_tags_combine(bed): + # Order is visible in the count: the floor works on the sparse per-tag + # frame, so two readings of one identity in one cell are two floored + # readings. Combining first would take the highest and floor one. + (bed / "panel.csv").write_text( + "Samples,Name,Sequence,Type,Family\n" + "S1,AgA,AAAA,Target,Spike\n" + "S1,AgB,CCCC,Target,Spike\n" + "S1,Ctrl,CTRL,Control,Reference\n" + ) + (bed / "counts.csv").write_text("sampleId,cellId,tag,umiCount\nS1,c1,AAAA,1\nS1,c1,CCCC,1\nS1,c1,CTRL,6\n") + (bed / "linker.csv").write_text("sampleId,cellId,setId\nS1,c1,K1\n") + r = _run(bed, *BASE, "--grouping", json.dumps({"by": "property", "column": "Family"})) + assert r.returncode == 0, r.stderr + meta = json.loads((bed / "result_run_meta.json").read_text()) + assert meta["readingsFloored"] == 2 + assert meta["cellsEmptied"] == 1 + v = pl.read_csv(bed / "result_verdicts.csv") + assert v.filter(pl.col("identity") == "Spike").row(0, named=True)["state"] == "not bound" From 08f7410acbb6927c8cdb8f6f00d2656e7d126401 Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 19:12:02 +0200 Subject: [PATCH 047/282] MILAB-6496: close four mutation-surviving gaps in the emit-verdicts tests The cutoff refusal is "at or below the analytic bound", but the test used 0.04 and 0.05 and never the bound itself, so it could not tell that from "below". Now asserted at specificity_score(0, 0) exactly. Nothing distinguished the comparator served from the one requested. A run asking for a panel comparator with too small a panel now asserts the record says none. Nothing distinguished the depth denominator. The bed now carries four observed barcodes against three listed cells, where dividing by the wrong one crosses the 5000 line: 18000/3 clears it, 18000/4 does not. The sortedness checks ran on a bed of one set and one identity, where sorted and unsorted are the same frame. A wider bed -- three identities declared in descending order across two sets -- makes the order observable. Two frames keep a redundant sort: `offered` and `tag_identity` are already built from sorted iterations, as the verdicts frame is by combine_cells. Left in place as cheap insurance against a future construction change. --- .../test/test_emit_verdicts.py | 88 +++++++++++++++++++ 1 file changed, 88 insertions(+) diff --git a/software/per-cell-metrics/test/test_emit_verdicts.py b/software/per-cell-metrics/test/test_emit_verdicts.py index 5562799..8c31040 100644 --- a/software/per-cell-metrics/test/test_emit_verdicts.py +++ b/software/per-cell-metrics/test/test_emit_verdicts.py @@ -162,12 +162,94 @@ def test_a_cutoff_at_the_analytic_floor_is_refused(bed): # At or below specificity_score(0, 0) the analytic tally and the dense # oracle disagree about a silent admissible cell with no error raised. # `silent_tally` states that refusing such a cutoff is this CLI's job. + # Tested *at* the bound, not merely either side of it. The refusal is + # "at or below", and 0.04/0.05 alone cannot tell that from "below". + from verdict import specificity_score + + bound = float(specificity_score(0, 0)) + r = _run(bed, *BASE, "--cutoff", "0.04") assert r.returncode != 0 assert "0.042" in (r.stderr + r.stdout) + assert _run(bed, *BASE, "--cutoff", repr(bound)).returncode != 0, "the bound itself must be refused" + assert _run(bed, *BASE, "--cutoff", repr(bound * 1.001)).returncode == 0 assert _run(bed, *BASE, "--cutoff", "0.05").returncode == 0 +def test_rows_are_sorted_on_a_bed_wide_enough_for_order_to_show(bed): + # The default bed has one set and one identity, where sorted and unsorted + # are the same frame and a missing sort is invisible. Three identities + # declared in descending order across two sets make the two differ. + (bed / "panel.csv").write_text( + "Samples,Name,Sequence,Type\nS1,AgZ,ZZZZ,Target\nS1,AgM,MMMM,Target\nS1,AgA,AAAA,Target\nS1,Ctrl,CTRL,Control\n" + ) + rows = ["sampleId,cellId,tag,umiCount"] + for cell in ("c1", "c2", "c3"): + rows.append(f"S1,{cell},CTRL,6") + for tag in ("ZZZZ", "MMMM", "AAAA"): + rows.append(f"S1,{cell},{tag},500") + (bed / "counts.csv").write_text("\n".join(rows) + "\n") + (bed / "linker.csv").write_text("sampleId,cellId,setId\nS1,c1,K2\nS1,c2,K1\nS1,c3,K1\n") + + r = _run(bed, *BASE) + assert r.returncode == 0, r.stderr + verdicts = pl.read_csv(bed / "result_verdicts.csv", infer_schema_length=0) + assert verdicts.height == 6, "two sets by three identities" + + for name, keys in ( + ("result_verdicts.csv", ["setId", "identity"]), + ("result_cell_counts.csv", ["sampleId", "cellId", "tag"]), + ("result_offered.csv", ["sampleId", "identity"]), + ("result_tag_identity.csv", ["tag", "identity"]), + ): + frame = pl.read_csv(bed / name, infer_schema_length=0) + assert frame.height > 1, f"{name} is too small for order to mean anything" + assert frame.equals(frame.sort(keys)), name + + +def test_run_meta_records_the_comparator_served_not_the_one_requested(bed): + # `served_source` degrades to `none` where it cannot honour a request -- + # here a panel comparator is asked for and the panel is far too small to + # stand in as one. Recording the request instead would claim a comparator + # the run never had, and two runs compared against different things would + # look like two runs compared against the same thing. + r = _run(bed, *BASE, "--reference-source", "panel", "--panel-min-members", "50") + assert r.returncode == 0, r.stderr + # The flag spelling and the recorded value differ on purpose: "none" is what + # a caller asks for, `ReferenceChoice.NONE` is what the record says happened. + from verdict import ReferenceChoice + + meta = json.loads((bed / "result_run_meta.json").read_text()) + assert meta["referenceChoice"] == ReferenceChoice.NONE.value + assert meta["referenceChoice"] != "panel", "the request must not be reported as though it were served" + + verdicts = pl.read_csv(bed / "result_verdicts.csv", infer_schema_length=0) + assert verdicts.filter(pl.col("identity") == "AAAA").row(0, named=True)["state"] == "unreliable" + + +def test_sequencing_depth_divides_by_the_cell_list_not_by_observed_barcodes(bed): + # The vendor's five thousand is per called cell. Observed barcodes exceed + # called cells by one to two orders of magnitude in droplet data, because + # ambient reads land on most barcodes, so dividing by them would let a + # badly undersequenced run read acceptable. The bed makes the two differ: + # four barcodes carry counts, three are in the cell list. + (bed / "counts.csv").write_text((bed / "counts.csv").read_text() + "S1,zzz,AAAA,7\n") + (bed / "qc.csv").write_text( + "sampleId,readsTotal,readsMatched,matchedFraction,cellsDetected," + "featuresDetected,totalUniqueUmis,medianUmisPerCell,panelAssignedFraction\n" + "S1,20000,18000,0.9,4,2,1200,300,0.82\n" + ) + r = _run(bed, *BASE, "--qc-summary", "qc.csv") + assert r.returncode == 0, r.stderr + + qc = pl.read_csv(bed / "result_qc.csv", infer_schema_length=0) + depth = qc.filter(pl.col("measurement") == "readsPerCell").row(0, named=True) + # 18000 / 3 listed cells = 6000, which clears the 5000 line. + # 18000 / 4 observed barcodes = 4500, which would not. + assert float(depth["value"]) == pytest.approx(6000.0) + assert depth["status"] == "acceptable" + + def test_the_dense_oracle_is_not_reachable_from_the_entrypoint(): # The dense oracle exists to check the analytic tally in tests. On a # realistic panel the grid it builds is 11-20x the sparse input, so a @@ -239,6 +321,12 @@ def test_output_is_byte_stable_across_runs(bed): second = {p.name: p.read_bytes() for p in bed.glob("result_*")} assert first == second + # Repeating the run is not enough on its own: polars groups deterministically + # for one input, so an unsorted frame reproduces itself byte for byte and + # this passes while the sort is missing. Sortedness itself is asserted in + # `test_rows_are_sorted_on_a_bed_wide_enough_for_order_to_show`, which needs + # a bed this one is too narrow to provide. + def test_a_computed_but_unjudged_measurement_is_not_reported_as_unchecked(bed): # `roll_up` answers *not evaluated* for a level with nothing judgeable in From 75f4b364507d3a201a40467217b185076945c388 Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 19:25:58 +0200 Subject: [PATCH 048/282] MILAB-6496: three spec divergences in the verdict entrypoint Found by reviewing the entrypoint against the spec atoms rather than the plan. All three changed what a default run returns, and none was covered by a test. The comparator skipped a rung. The order is a declared reagent, else the panel's own readings where the panel carries enough members, else nothing -- and only the empty-droplet comparator is conditional on a scientist asking for it. The default resolver answered declared-or-nothing, so a twenty-antigen panel with no declared control read unreliable throughout when it could have been read against itself. A panel below the minimum still falls to no comparator, so the founding three-antigen case is unchanged. The cell list was derived from the antigen counts when neither list input arrived. Which barcodes held a cell is an input, and nothing in the antigen readings separates a cell from an empty droplet. Labelling the derivation in the run record did not make it allowed, and it was not inert: reads-per-cell divides by it, and observed barcodes outnumber cells by one to two orders of magnitude, so a healthy library read undersequenced. With no list, membership is now unknown rather than false and the measurements needing one read not evaluated. The capture rollup discarded the membership it had just computed whenever no capture map was given, which is the ordinary case. The level whose stated job is that nothing hides aggregated nothing, reporting not evaluated over runs whose samples and panels were measured perfectly well. --- .../per-cell-metrics/src/emit_verdicts.py | 70 +++++++++++---- software/per-cell-metrics/src/verdict.py | 31 ++++++- .../test/test_emit_verdicts.py | 89 +++++++++++++++++++ 3 files changed, 172 insertions(+), 18 deletions(-) diff --git a/software/per-cell-metrics/src/emit_verdicts.py b/software/per-cell-metrics/src/emit_verdicts.py index 68147a5..b5b8962 100644 --- a/software/per-cell-metrics/src/emit_verdicts.py +++ b/software/per-cell-metrics/src/emit_verdicts.py @@ -514,18 +514,37 @@ def main() -> None: cell_list = linker_cells cell_list_source = "clonotype linker" else: - cell_list = set(counts.select("sampleId", "cellId").unique().iter_rows()) - cell_list_source = "observed barcodes" + # No list arrived, and one is NOT derived from the counts. Nothing in + # the antigen readings separates a cell from a droplet that held none, + # so the observed barcodes are not a cell list -- in droplet data they + # outnumber the cells by one to two orders of magnitude, because ambient + # antigen material lands on most barcodes. Standing them in would not + # merely be approximate: `readsPerCell` divides by this, so a healthy + # library would read undersequenced and alert. + # + # Every barcode is still analysed and every count still emitted. What is + # withheld is the claim that these barcodes are cells: `inCellList` is + # unknown rather than true, and the measurements that need a cell list + # read *not evaluated*, which is exactly the reading for "the run could + # not supply what this needed". + cell_list = None + cell_list_source = "none" + + # `cell_list is None` means no list arrived, which is different from a list + # that arrived empty: the first cannot answer "is this barcode a cell", the + # second answers "no". `listed` collapses both for the set arithmetic below, + # where either way there are no barcodes to add. + listed = cell_list if cell_list is not None else set() observed_cells = set(counts.select("sampleId", "cellId").unique().iter_rows()) # Barcodes outside the cell list stay in the frame, labelled: one dropped # here is indistinguishable afterwards from one that never existed, and # its antigen counts are real whatever the list says about it. - analysed_cells = sorted(cell_list | observed_cells | linker_cells) + analysed_cells = sorted(listed | observed_cells | linker_cells) panel_samples = {s for s in panel["sample"].to_list() if s != ANY_SAMPLE} samples = sorted( - {s for s, _ in observed_cells} | {s for s, _ in cell_list} | {s for s, _ in linker_cells} | panel_samples + {s for s, _ in observed_cells} | {s for s, _ in listed} | {s for s, _ in linker_cells} | panel_samples ) # The floor is applied per sample so the counters it returns land in each @@ -542,15 +561,20 @@ def main() -> None: readings_floored = sum(s["readingsFloored"] for s in floor_stats.values()) cells_emptied = sum(s["cellsEmptied"] for s in floor_stats.values()) + # One panel size, read once and passed to both. Deriving it separately for + # the default choice and for the resolution would let the two disagree about + # whether the panel is large enough to serve as its own comparator. + panel_size = int(panel["tag"].n_unique()) + source = ReferenceChoice[args.reference_source.upper()] if args.reference_source else None if source is None: - source = resolve_default_source(reference_tags) + source = resolve_default_source(reference_tags, panel_size, args.panel_min_members) reference = reference_by_cell( floored, reference_tags, source, cells=analysed_cells, - panel_size=int(panel["tag"].n_unique()), + panel_size=panel_size, min_members=args.panel_min_members, ) gated, cells_high_reference = gate_cells(reference.by_cell, args.gate_threshold, args.high_reference_line) @@ -599,8 +623,12 @@ def main() -> None: # fraction of the size a per-cell-per-identity table would take. A reader # regrouping the panel re-takes the highest member, re-scores against the # same reference, and re-votes, without a re-run. + # With no list, membership is unknown rather than false: a barcode nobody + # classified is not a barcode classified as "not a cell". "false" would be + # a claim the run cannot support. + unlisted_reads = "false" if cell_list is not None else "unknown" in_list = pl.DataFrame( - [(s, c, "true") for s, c in sorted(cell_list)], + [(s, c, "true") for s, c in sorted(listed)], orient="row", schema={"sampleId": pl.String, "cellId": pl.String, "inCellList": pl.String}, ) @@ -612,14 +640,14 @@ def main() -> None: cell_counts = ( non_reference.join(reference_frame, on=["sampleId", "cellId"], how="left") .join(in_list, on=["sampleId", "cellId"], how="left") - .with_columns(pl.col("inCellList").fill_null("false")) + .with_columns(pl.col("inCellList").fill_null(unlisted_reads)) .select(["sampleId", "cellId", "tag", "umiCount", "referenceCount", "inCellList"]) ) _write_sorted(cell_counts, f"{prefix}_cell_counts.csv", ["sampleId", "cellId", "tag"]) cell_scalars = ( reference_frame.join(in_list, on=["sampleId", "cellId"], how="left") - .with_columns(pl.col("inCellList").fill_null("false")) + .with_columns(pl.col("inCellList").fill_null(unlisted_reads)) .with_columns( pl.Series( "admissibility", @@ -737,7 +765,7 @@ def _number(row: dict, column: str) -> float | None: for sample in samples: first = len(rows) sample_counts = counts.filter(pl.col("sampleId") == sample) - listed_here = [key for key in sorted(cell_list) if key[0] == sample] + listed_here = [key for key in sorted(listed) if key[0] == sample] if cell_list is not None else None qc = read_qc.get(sample, {}) reads_matched = _number(qc, "readsMatched") @@ -749,8 +777,17 @@ def _number(row: dict, column: str) -> float | None: # happened to touch: the five-thousand recommendation is per called # cell, and in droplet data observed barcodes run one to two orders of # magnitude higher, so dividing by them would alert on a healthy run. - depth = reads_per_cell(int(reads_matched), len(listed_here)) if reads_matched is not None else None - _add(rows, "sample", sample, "readsPerCell", depth, f"cellsInList={len(listed_here)}") + # No cell list means no denominator, so depth is *not evaluated* -- + # the run could not supply what the measurement needed. Substituting + # the observed barcodes would answer a different question and, being + # one to two orders of magnitude larger, would alert on a fine library. + depth = ( + reads_per_cell(int(reads_matched), len(listed_here)) + if reads_matched is not None and listed_here is not None + else None + ) + detail = f"cellsInList={len(listed_here)}" if listed_here is not None else "no cell list supplied" + _add(rows, "sample", sample, "readsPerCell", depth, detail) deciles = antigen_count_deciles(sample_counts) decile_detail = "|".join( @@ -870,11 +907,14 @@ def _number(row: dict, column: str) -> float | None: # block: adding an axis to a released column changes its identity, adding # a value does not. With no assignment the single row reads *not # evaluated*, which is what it is -- nobody looked -- and never an absence. + # With no assignment every sample belongs to one unnamed capture, rather + # than to a capture with no members. Emptying the membership would make the + # one level whose whole job is that nothing hides aggregate nothing: it + # would read *not evaluated* over a run whose samples and panels were + # measured perfectly well. captures: dict[str, list[str]] = {} for sample in samples: captures.setdefault(capture_of_sample.get(sample, "unassigned"), []).append(sample) - if not capture_of_sample: - captures = {"unassigned": []} for capture, its_samples in sorted(captures.items()): its_panels = sorted({panel_of_sample[s] for s in its_samples}) from_samples = [sample_coverage[s] for s in its_samples] @@ -889,7 +929,7 @@ def _number(row: dict, column: str) -> float | None: "referenceChoice": reference.served.value, "referenceSourceRequested": source.value, "cellListSource": cell_list_source, - "cellsInList": len(cell_list), + "cellsInList": len(cell_list) if cell_list is not None else None, "cellsAnalysed": len(analysed_cells), "floor": args.floor, "cutoff": args.cutoff, diff --git a/software/per-cell-metrics/src/verdict.py b/software/per-cell-metrics/src/verdict.py index 44b4372..0f81574 100644 --- a/software/per-cell-metrics/src/verdict.py +++ b/software/per-cell-metrics/src/verdict.py @@ -141,9 +141,34 @@ class ReferenceChoice(str, Enum): NONE = "no comparator available" -def resolve_default_source(reference_tags: set[str]) -> ReferenceChoice: - """The *default* source only. The scientist overrides it; this never does.""" - return ReferenceChoice.DECLARED if reference_tags else ReferenceChoice.NONE +def resolve_default_source( + reference_tags: set[str], + panel_size: int = 0, + min_members: int = DEFAULT_PANEL_MIN_MEMBERS, +) -> ReferenceChoice: + """The *default* source only. The scientist overrides it; this never does. + + Three rungs, in order: a declared reagent, else the panel's own readings + where the panel carries enough members, else nothing. + + The middle rung is reached automatically, unlike the empty-droplet + comparator, which is offered only where a scientist asks for it. The + difference is what each one costs to be wrong about: an empty-droplet + population is a different experiment's data and switching to it silently + would change what a verdict means, while the panel's own readings are the + same cells already being read. A panel of twenty antigens with no declared + control is the configuration this ordering exists for -- falling straight + to *no comparator* there would make every identity unreliable in a run that + could be read perfectly well. + + A panel too small to stand in as its own comparator still falls to NONE, so + the founding three-antigen case is unaffected. + """ + if reference_tags: + return ReferenceChoice.DECLARED + if panel_size >= min_members: + return ReferenceChoice.PANEL + return ReferenceChoice.NONE def served_source( diff --git a/software/per-cell-metrics/test/test_emit_verdicts.py b/software/per-cell-metrics/test/test_emit_verdicts.py index 8c31040..7666156 100644 --- a/software/per-cell-metrics/test/test_emit_verdicts.py +++ b/software/per-cell-metrics/test/test_emit_verdicts.py @@ -5,6 +5,7 @@ import polars as pl import pytest +from verdict import ReferenceChoice SRC = Path(__file__).resolve().parents[1] / "src" @@ -120,6 +121,94 @@ def test_reference_source_none_produces_unreliable_not_a_crash(bed): assert v.filter(pl.col("identity") == "AAAA").row(0, named=True)["state"] == "unreliable" +def test_a_panel_with_no_declared_reference_falls_to_the_panel_not_to_nothing(bed): + # Three rungs in order: a declared reagent, else the panel's own readings + # where the panel carries enough members, else nothing. Skipping the middle + # rung makes every identity unreliable in a twenty-antigen run that could + # be read perfectly well -- which is the configuration the ordering exists + # for. Ten tags here, against a shipped minimum of eight. + tags = [f"T{i:02d}" for i in range(10)] + (bed / "panel.csv").write_text( + "Samples,Name,Sequence,Type\n" + "".join(f"S1,Ag{i},{t},Target\n" for i, t in enumerate(tags)) + ) + # Background counts sit *above* the shipped floor of 4. At 3 they would be + # floored to zero, the panel median would be 0, every cell would fall below + # the thin line, and the run would read unreliable for a reason that has + # nothing to do with which comparator was chosen -- hiding the very thing + # this test exists to check. + rows = ["sampleId,cellId,tag,umiCount"] + for cell in ("c1", "c2", "c3"): + rows.append(f"S1,{cell},{tags[0]},900") + rows.extend(f"S1,{cell},{t},10" for t in tags[1:]) + (bed / "counts.csv").write_text("\n".join(rows) + "\n") + + r = _run(bed, *BASE) + assert r.returncode == 0, r.stderr + meta = json.loads((bed / "result_run_meta.json").read_text()) + assert meta["referenceChoice"] == ReferenceChoice.PANEL.value + + states = set(pl.read_csv(bed / "result_verdicts.csv", infer_schema_length=0)["state"].to_list()) + assert states != {"unreliable"}, "the panel could serve as its own comparator and was not asked to" + + +def test_a_panel_too_small_to_serve_still_falls_to_no_comparator(bed): + # The founding three-antigen case: too small to stand in as its own + # comparator, so the third rung is right there and must not be skipped. + (bed / "panel.csv").write_text( + "Samples,Name,Sequence,Type\nS1,AgA,AAAA,Target\nS1,AgB,BBBB,Target\nS1,AgC,CCCC,Target\n" + ) + r = _run(bed, *BASE) + assert r.returncode == 0, r.stderr + meta = json.loads((bed / "result_run_meta.json").read_text()) + assert meta["referenceChoice"] == ReferenceChoice.NONE.value + + +def test_no_cell_list_leaves_membership_unknown_and_depth_unevaluated(bed): + # Which barcodes held a cell is an input. Nothing in the antigen readings + # separates a cell from an empty droplet, so with neither list input the + # observed barcodes must NOT stand in: they outnumber cells by one to two + # orders of magnitude, and `readsPerCell` divides by this, so a healthy + # library would read undersequenced. + no_linker = [a for a in BASE if a not in ("--linker", "linker.csv")] + (bed / "qc.csv").write_text( + "sampleId,readsTotal,readsMatched,matchedFraction,cellsDetected," + "featuresDetected,totalUniqueUmis,medianUmisPerCell,panelAssignedFraction\n" + "S1,20000,18000,0.9,3,2,1200,300,0.82\n" + ) + r = _run(bed, *no_linker, "--qc-summary", "qc.csv") + assert r.returncode == 0, r.stderr + + meta = json.loads((bed / "result_run_meta.json").read_text()) + assert meta["cellListSource"] == "none" + assert meta["cellsInList"] is None + + qc = pl.read_csv(bed / "result_qc.csv", infer_schema_length=0) + depth = qc.filter(pl.col("measurement") == "readsPerCell").row(0, named=True) + assert depth["status"] == "not evaluated" + + counts = pl.read_csv(bed / "result_cell_counts.csv", infer_schema_length=0) + assert set(counts["inCellList"].to_list()) == {"unknown"}, "unclassified is not the same as classified 'no'" + + +def test_the_capture_rollup_gathers_every_sample_when_no_capture_map_is_given(bed): + # The capture level exists so that nothing hides. Rolling up an empty + # membership makes it report *not evaluated* over a run whose samples and + # panels were measured perfectly well, which is the opposite of its job. + _run(bed, *BASE) + qc = pl.read_csv(bed / "result_qc.csv", infer_schema_length=0) + rollups = qc.filter(pl.col("measurement") == "rollup") + capture = rollups.filter(pl.col("level") == "capture").row(0, named=True) + + def _triple(level): + r = rollups.filter(pl.col("level") == level) + return [int(r[c].cast(pl.Int64).sum()) for c in ("judged", "unjudged", "notEvaluated")] + + assert sum(_triple("sample")) > 0, "the bed must have something for the capture to gather" + assert [int(capture[c]) for c in ("judged", "unjudged", "notEvaluated")] == [ + s + p for s, p in zip(_triple("sample"), _triple("panel"), strict=True) + ] + + def test_contending_groups_reach_the_note(bed): (bed / "panel.csv").write_text((bed / "panel.csv").read_text() + "S1,AgB,CCCC,Target\n") (bed / "counts.csv").write_text((bed / "counts.csv").read_text() + "S1,c1,CCCC,1\nS1,c2,CCCC,1\nS1,c3,CCCC,1\n") From f5160aefdef140f1242694e13eb0dd833034a958 Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 19:27:55 +0200 Subject: [PATCH 049/282] MILAB-6496: column specs for verdicts, support, set counts and QC --- workflow/src/column-specs.lib.tengo | 787 +++++++++++++++++++++++++--- 1 file changed, 703 insertions(+), 84 deletions(-) diff --git a/workflow/src/column-specs.lib.tengo b/workflow/src/column-specs.lib.tengo index 16bbbdc..3a8d5b1 100644 --- a/workflow/src/column-specs.lib.tengo +++ b/workflow/src/column-specs.lib.tengo @@ -1,20 +1,50 @@ // Export column + axis specs for the Feature Integration block. // +// Two layers live here. The per-cell/per-feature contract (pl7.app/feature/*) is the block's original +// surface and is unchanged. The verdict stage, in the second half of the file, describes what +// emit_verdicts.py writes and keys on its own axes; the two never share an axis. +// // Centralised here (mirrors blocks/peptide-extraction column-specs.lib.tengo and // blocks/mixcr-clonotyping calculate-export-specs.lib.tengo) so main.tpl.tengo stays readable and the // per-cell contract columns carry the full standard annotation set (abundance flags, order priority, // table visibility). Annotations do NOT affect p-column identity (name + domain + axes do), so the // downstream discovery contract is unchanged by anything in this file. +ll := import("@platforma-sdk/workflow-tengo:ll") maps := import("@platforma-sdk/workflow-tengo:maps") strings := import("@platforma-sdk/workflow-tengo:strings") json := import("json") +text := import("text") + +// No column this block emits may be orderable. A binding verdict is a statement about what the +// experiment could establish, and a count of verdicts is derived from that statement — neither is a +// magnitude, so neither may be ranked. `pl7.app/isScore` and the `pl7.app/score/*` family are what +// make a column rankable downstream (lead selection discovers its rank options by exactly those +// annotations), so they are refused here rather than reviewed for later. Ordinary filter annotations +// — `pl7.app/isDiscreteFilter`, `pl7.app/discreteValues` — are untouched by this rule. +// +// The guard runs over every annotation map this module builds, so a reintroduction fails the workflow +// render instead of shipping. It catches a computed key as well as a literal one; what it cannot see +// is a column built outside this module, which is why the same assertion is also made against the +// block's emitted columns in the block test. +SCORE_ANNOTATION := "pl7.app/isScore" +SCORE_FAMILY_PREFIX := "pl7.app/score/" + +guardNoScore := func(annotations) { + for key, _ in annotations { + ll.assert( + key != SCORE_ANNOTATION && !text.has_prefix(key, SCORE_FAMILY_PREFIX), + "column-specs: annotation %v makes a column orderable; verdicts and their counts are filterable, never orderable", + key) + } + return annotations +} // Standard table annotations: order priority + default visibility. // visibility: undefined -> hidden, true -> default, false -> optional. a := func(order, defaultVisibility, spec) { - return maps.merge(spec, { + return maps.merge(guardNoScore(spec), { "pl7.app/table/orderPriority": string(order), "pl7.app/table/visibility": is_undefined(defaultVisibility) ? "hidden" : defaultVisibility ? "default" : "optional" }) @@ -48,7 +78,12 @@ featureAxis := func(blockId) { } // valueOutputs: the per-cell contract value columns, as processColumn `Xsv` output declarations. -// hasControl gates the specificity score — only meaningful with a negative control. +// +// `hasControl` no longer gates anything here and is kept only so main.tpl.tengo's call site is +// unchanged. The consensus/dominant-feature column and the Cell Ranger specificity score are both +// gone: the dominance rule they reported was removed from per_cell_metrics.py, which no longer writes +// either CSV, and a specificity score is a binding magnitude — the thing a four-state verdict +// replaces. Declaring an output whose CSV no longer exists costs a failed import, not a missing column. // // umiCount is the primary abundance (abundance/isPrimary). It is deliberately NOT isAnchor: this // column is meant to be DISCOVERED under the downstream VDJ single-cell anchor (via the cellLinker), @@ -113,72 +148,18 @@ valueOutputs := func(blockId, sampleAxisName, hasControl) { path: ["fractions"] } - consensus := { - type: "Xsv", - xsvType: "csv", - settings: { - axes: [{ column: "cellId", spec: cell }], - columns: [{ - column: "consensusFeature", id: "consensusFeature", - spec: { - name: "pl7.app/feature/consensusFeature", - valueType: "String", - annotations: a(88000, true, { - "pl7.app/label": "Consensus feature", - "pl7.app/description": "The cell's dominant feature/antigen — assigned only when one feature's share reaches the dominance threshold (default 0.6); otherwise 'ambiguous'.", - "pl7.app/isDiscreteFilter": "true" - }) - } - }], - storageFormat: "Parquet", - partitionKeyLength: 0 - }, - name: "consensus", - path: ["consensus"] - } - - outputs := [abundance, fractions, consensus] - - if hasControl { - outputs = append(outputs, { - type: "Xsv", - xsvType: "csv", - settings: { - axes: [{ column: "cellId", spec: cell }, { column: "feature", spec: feat }], - columns: [{ - column: "specificityScore", id: "specificityScore", - spec: { - name: "pl7.app/feature/specificityScore", - valueType: "Double", - annotations: a(87000, true, { - "pl7.app/label": "Specificity score", - "pl7.app/description": "Cell Ranger's BEAM specificity score (0–100): confidence that this antigen's binding exceeds the negative control, from their UMI counts — not binding strength or affinity. Needs a designated negative control.", - "pl7.app/min": "0", - "pl7.app/max": "100", - "pl7.app/isScore": "true", - "pl7.app/format": ".1f" - }) - } - }], - storageFormat: "Parquet", - partitionKeyLength: 0 - }, - name: "specificity", - path: ["specificity"] - }) - } - - return outputs + return [abundance, fractions] } // perCellSummaryOutput: the TABLE-ONLY per-cell collapse (one row per [sampleId, cellId]) produced by // per_cell_metrics.py's result_per_cell_summary.csv. Its columns are the cell's max feature UMI count -// / max feature fraction (/ max specificity score, with a control) plus a "feature : umi : fraction | -// ..." summary string sorted by descending fraction. This drives the Main results table INSTEAD of the -// per-(cell x feature) matrix; the per-cell export contract (abundance/fractions/consensus/specificity) -// is unchanged. The "Max ..." labels distinguish these aggregates from the exported per-feature -// columns. maxSpecificityScore is emitted (and imported) only with a negative control — same gating as -// the per-feature specificity column. Keyed [cellId]; the sample axis is prepended by processColumn. +// and max feature fraction plus a "feature (fraction%, umi), ..." summary string sorted by descending +// fraction. This drives the Main results table INSTEAD of the per-(cell x feature) matrix; the per-cell +// export contract (abundance/fractions) is unchanged. The "Max ..." labels distinguish these aggregates +// from the exported per-feature columns. The max-specificity aggregate went with the per-feature +// specificity column: per_cell_metrics.py stopped writing it, and it read a binding magnitude. +// `hasControl` is kept only so main.tpl.tengo's call site is unchanged. +// Keyed [cellId]; the sample axis is prepended by processColumn. perCellSummaryOutput := func(blockId, sampleAxisName, hasControl) { cell := cellAxis(sampleAxisName) @@ -211,24 +192,6 @@ perCellSummaryOutput := func(blockId, sampleAxisName, hasControl) { } ] - if hasControl { - cols = append(cols, { - column: "maxSpecificityScore", id: "maxSpecificityScore", - spec: { - name: "pl7.app/feature/maxSpecificityScore", - valueType: "Double", - annotations: a(85000, true, { - "pl7.app/label": "Max Specificity score", - "pl7.app/description": "The cell's highest per-feature Cell Ranger BEAM specificity score (0–100): confidence that the best-supported feature's binding exceeds the negative control. Needs a designated negative control.", - "pl7.app/min": "0", - "pl7.app/max": "100", - "pl7.app/isScore": "true", - "pl7.app/format": ".1f" - }) - } - }) - } - // The "all features" summary string (mirrors antibody-sequence-liabilities' pl7.app/isSummary // column). Listed last (lowest orderPriority) so the headline aggregates read first. cols = append(cols, { @@ -368,12 +331,647 @@ negativeControlColumn := func() { spec: { name: "pl7.app/feature/negativeControl", valueType: "String", - annotations: { + annotations: guardNoScore({ "pl7.app/label": "Negative control", "pl7.app/table/visibility": "hidden" + }) + } + } +} + +// ===================================================================================== +// The verdict stage +// +// Everything below describes what emit_verdicts.py writes: twelve CSVs plus a run-meta JSON. +// One function per CSV, each returning a complete xsv.importFile spec, so the file that names a +// column is also the file that says which CSV column it comes from. +// ===================================================================================== + +// --- Axes ----------------------------------------------------------------------------- +// +// Four axes are minted rather than reused. The tempting shortcut — putting tag keys on the block's +// existing pl7.app/feature/featureId axis — would ship an axis whose identity is unchanged while its +// value space inverts: feature names before, barcode sequences here. No downstream query would fail +// and no join would error; joins would simply return rows for the wrong thing. The legacy +// pl7.app/feature/* columns above are therefore left exactly as they are — not re-keyed, not +// re-pointed — and the verdict stage keys on its own vocabulary. + +// tagAxis: one declared barcode sequence. +// +// `tagType` sits in the domain rather than in the name because a second kind of tag (surface markers) +// is then a branch inside one vocabulary instead of a second vocabulary. blockId keeps two blocks' +// tags from colliding, exactly as it does for featureAxis. +tagAxis := func(blockId) { + return { + name: "pl7.app/antigen/tagId", + type: "String", + domain: { + "pl7.app/blockId": blockId, + "pl7.app/antigen/tagType": "antigen" + } + } +} + +// identityAxis: the thing a verdict is about — one tag, or a group of tags read as one antigen. +// +// `groupingId` is in the DOMAIN, and ships even though v1 only ever sets "per-tag". Domain is part of +// axis identity and annotations are not, so without it the axis would keep one identity while its +// values changed meaning under a regrouping: a saved downstream filter would silently match nothing, +// or match something else. With it, one run can carry several groupings side by side as distinct +// axes. `segmentedBy` tells a reader which domain key separates them. +identityAxis := func(blockId, groupingId) { + return { + name: "pl7.app/antigen/identityId", + type: "String", + domain: { + "pl7.app/blockId": blockId, + "pl7.app/antigen/tagType": "antigen", + "pl7.app/antigen/groupingId": groupingId + }, + annotations: { + "pl7.app/label": "Antigen identity", + "pl7.app/segmentedBy": string(json.encode(["pl7.app/antigen/groupingId"])) + } + } +} + +// panelAxis: one distinct declared tag set. No panel file names its panel, so the id is a hash of the +// sorted tag list (emit_verdicts.py `_panel_id`), stable across re-runs of the same declaration. +// Where one panel covers every sample the axis takes a single value and drops out of view. +panelAxis := func(blockId) { + return { + name: "pl7.app/antigen/panelId", + type: "String", + domain: { "pl7.app/blockId": blockId } + } +} + +// captureAxis: the physical capture a sample came off. Minted now, before any capture assignment +// reaches the block, because adding an axis to a released QC column changes that column's identity +// while adding a value to an existing axis does not. Every capture presently rolls up as "not +// evaluated", which is the honest reading — a non-evaluation is never rendered as an absence. +// No emitted column is keyed on it yet: the capture rollup travels as a row in the QC frame, and the +// axis is exported so the sample->capture map can be added without re-identifying anything. +captureAxis := func(blockId) { + return { + name: "pl7.app/antigen/captureId", + type: "String", + domain: { "pl7.app/blockId": blockId } + } +} + +// The QC key is three axes, not two. emit_verdicts.py writes result_qc.csv keyed +// (level, entity, measurement), and `entity` is load-bearing rather than incidental: under the +// default per-tag grouping a tag id and an identity id are the SAME string, so without a part of the +// key naming which kind of thing the row is about, a tag row and an identity row for the same barcode +// collide. A measurement is an axis VALUE, so the fifteenth measurement costs a row, not a column. +qcLevelAxis := func(blockId) { + return { + name: "pl7.app/antigen/qcLevel", + type: "String", + domain: { "pl7.app/blockId": blockId }, + annotations: { "pl7.app/label": "Level" } + } +} + +qcEntityAxis := func(blockId) { + return { + name: "pl7.app/antigen/qcEntity", + type: "String", + domain: { "pl7.app/blockId": blockId }, + annotations: { "pl7.app/label": "Measured thing" } + } +} + +qcMeasurementAxis := func(blockId) { + return { + name: "pl7.app/antigen/qcMeasurement", + type: "String", + domain: { "pl7.app/blockId": blockId }, + annotations: { "pl7.app/label": "Measurement" } + } +} + +// --- Value vocabularies --------------------------------------------------------------- +// +// Each mirrors a closed enum in the software, so a filter offers every value the data can take. +// There is deliberately no default cutoff: the whole score family is refused, and a default cutoff +// is part of it. All four states stay reachable through the ordinary discrete filter. +VERDICT_STATES := string(json.encode(["bound", "not bound", "never asked", "unreliable"])) +BOOL_VALUES := string(json.encode(["true", "false"])) +UNRELIABLE_REASONS := string(json.encode([ + "never-offered", "no-comparator", "thin-comparator", "all-cells-gated", + "tie", "below-agreement-floor", "too-few-voters"])) +ADMISSIBILITY_VALUES := string(json.encode([ + "admissible", + "cell set aside by the admissibility gate", + "no comparator for this cell", + "the comparator rests on too little to compare against"])) +QC_STATUSES := string(json.encode(["acceptable", "alerting", "unjudged", "not evaluated"])) +MISMATCH_DIRECTIONS := string(json.encode(["declared-never-seen", "undeclared-in-panel"])) + +// servedDomain: which comparator and which cell list actually served this run. +// +// In the DOMAIN, not the annotations. Annotations are excluded from column identity, so two runs +// served by different comparators would otherwise emit columns of identical identity and be silently +// unioned in a pool holding both — and a verdict compared against a declared reference tag is not the +// same reading as one compared against the panel's own signal. `served` is what emit_verdicts.py +// RESOLVED, not what was requested: it degrades to "no comparator available" where it cannot serve, +// which is why it reaches this module from the run-meta JSON as a template input rather than from the +// block's arguments. +servedDomain := func(served) { + return { + "pl7.app/antigen/referenceChoice": served.referenceChoice, + "pl7.app/antigen/cellListSource": served.cellListSource + } +} + +// servedNotes: the floor and the cutoff, which stay ANNOTATIONS. +// They are informational — they move where the line falls, not what a reading means — so a change to +// them must not fork column identity the way a change of comparator does. +servedNotes := func(served) { + return { + "pl7.app/antigen/countFloor": string(served.floor), + "pl7.app/antigen/boundCutoff": string(served.cutoff) + } +} + +// --- Verdicts: result_verdicts.csv, keyed (setId, identity) ----------------------------- +// +// The set axis is taken VERBATIM from the cell linker's third axis (linkerCol.spec.axesSpec[2]) so +// the verdicts land on the very clonotype key the clonotyping run produced; rebuilding it here would +// produce a lookalike axis that joins to nothing. +verdictsImportSpec := func(setAxisSpec, identityAxisSpec, served) { + // The importer builds each column's axesSpec from the `axes` list below, so only the domain — which + // every column in this family shares — is added here. + verdictCol := func(colName, id, spec) { + spec.domain = servedDomain(served) + return { column: colName, id: id, spec: spec } + } + return { + axes: [{ column: "setId", spec: setAxisSpec }, { column: "identity", spec: identityAxisSpec }], + columns: [ + verdictCol("state", "verdict", { + name: "pl7.app/antigen/verdict", + valueType: "String", + annotations: a(100000, true, maps.merge(servedNotes(served), { + "pl7.app/label": "Binding verdict", + "pl7.app/isDiscreteFilter": "true", + "pl7.app/discreteValues": VERDICT_STATES, + "pl7.app/description": "One of four states. 'Never asked' means the experiment did not put this antigen to these cells; 'unreliable' means it did and the data cannot settle it. Neither is a kind of 'not bound'." + })) + }), + verdictCol("unreliableReason", "unreliableReason", { + name: "pl7.app/antigen/unreliableReason", + valueType: "String", + annotations: a(96800, false, { + "pl7.app/label": "Why unsettled", + "pl7.app/isDiscreteFilter": "true", + "pl7.app/discreteValues": UNRELIABLE_REASONS, + "pl7.app/description": "A statement left unsettled by a position the experiment never asked calls for a panel change; one left unsettled by a reading that did not survive calls for a re-run. A bare 'unreliable' cannot tell them apart." + }) + }), + verdictCol("cellsCouldAnswer", "cellsCouldAnswer", { + name: "pl7.app/antigen/cellsCouldAnswer", + valueType: "Int", + annotations: a(99000, true, { + "pl7.app/label": "Cells that could answer", + "pl7.app/min": "0" + }) + }), + verdictCol("cellsAnswered", "cellsAnswered", { + name: "pl7.app/antigen/cellsAnswered", + valueType: "Int", + annotations: a(98000, true, { + "pl7.app/label": "Cells that answered", + "pl7.app/min": "0" + }) + }), + verdictCol("agreement", "agreement", { + name: "pl7.app/antigen/agreement", + valueType: "Double", + annotations: a(97000, false, { + "pl7.app/label": "Cell agreement", + "pl7.app/format": ".2p", + "pl7.app/min": "0", + "pl7.app/max": "1" + }) + }), + verdictCol("wasCompeted", "wasCompeted", { + name: "pl7.app/antigen/wasCompeted", + valueType: "String", + annotations: a(96500, true, { + "pl7.app/label": "Reading was competed", + "pl7.app/isDiscreteFilter": "true", + "pl7.app/discreteValues": BOOL_VALUES, + "pl7.app/description": "True where this antigen read 'not bound' and something it was declared to compete with read 'bound' for the same clonotype. A statement can test this; the state itself is unchanged." + }) + }), + verdictCol("competedWith", "competedWith", { + name: "pl7.app/antigen/competedWith", + valueType: "String", + annotations: a(96000, false, { "pl7.app/label": "Competed with" }) + }) + ], + storageFormat: "Parquet", + partitionKeyLength: 0 + } +} + +// --- Set counts: result_set_counts.csv, keyed (setId) ----------------------------------- +// +// These are the family that actually reaches lead selection: a column carrying an axis the clonotype +// anchor does not have is dropped there with no error, so nothing keyed (set, identity) is visible. +// They count identities, never tags, and they are still not orderable — a count of verdicts is +// derived from verdicts, and ordering by it is ordering by binding breadth. A downstream block +// assembles a ranked list from these plus measurements from other assays. +// +// `settledCount` is emitted because set_counts() writes it; it always equals +// offeredCount - unsettledCount, and it is cheaper to import than to make every reader re-derive. +setCountsImportSpec := func(setAxisSpec, served) { + countCol := func(colName, name, label, order, description) { + annotations := { "pl7.app/label": label, "pl7.app/min": "0" } + if description != "" { + annotations["pl7.app/description"] = description + } + return { + column: colName, + id: colName, + spec: { + name: name, + valueType: "Int", + domain: servedDomain(served), + annotations: a(order, true, annotations) } } } + return { + axes: [{ column: "setId", spec: setAxisSpec }], + columns: [ + countCol("boundCount", "pl7.app/antigen/boundCount", "Antigens bound", 95000, + "How many distinct antigen identities this clonotype bound. Breadth, never strength: nothing here says how well it bound any of them."), + countCol("offeredCount", "pl7.app/antigen/offeredCount", "Antigens offered", 94000, + "How many identities this clonotype's cells were actually stained with — the denominator a rate must use, since a clone offered eight of ten and binding all eight failed nothing."), + countCol("settledCount", "pl7.app/antigen/settledCount", "Antigens settled", 93500, + "Of the identities offered, how many the data could settle either way."), + countCol("unsettledCount", "pl7.app/antigen/unsettledCount", "Antigens unsettled", 93000, + "Offered identities the data could not settle. They stay in the offered count and are reported here beside it, so one bad reading does not void every count in the run.") + ], + storageFormat: "Parquet", + partitionKeyLength: 0 + } +} + +// --- Per-identity summary: result_identity_summary.csv, keyed (setId) -------------------- +// +// The same verdicts pivoted onto the set axis alone, one column per identity, so a per-antigen state +// sits on the clonotype trunk where lead selection can see it. The identity travels in the DOMAIN — +// the pattern this block already uses for pl7.app/feature/property — so two identities are two +// distinct p-columns rather than one column with a colliding name. +// +// Size-gated upstream: emit_verdicts.py writes only setId when the identity count passes its limit, +// and records both the limit and whether it emitted in the run meta. Callers pass the identities the +// CSV actually carries, so an empty list here yields no columns rather than a failed import. +// +// The label is the identity string itself. The readable name lives in result_identity_labels.csv, +// which is imported as a column rather than read as a value, so it is not available while these +// specs are built. +identitySummaryImportSpec := func(setAxisSpec, identities, groupingId, served) { + cols := [] + for i, identity in identities { + cols = append(cols, { + column: identity, + id: "identity_" + strings.substituteSpecialCharacters(identity), + spec: { + name: "pl7.app/antigen/identityVerdict", + valueType: "String", + domain: maps.merge(servedDomain(served), { + "pl7.app/antigen/identityId": identity, + "pl7.app/antigen/groupingId": groupingId + }), + annotations: a(92000 - i, false, { + "pl7.app/label": identity, + "pl7.app/isDiscreteFilter": "true", + "pl7.app/discreteValues": VERDICT_STATES + }) + } + }) + } + return { + axes: [{ column: "setId", spec: setAxisSpec }], + columns: cols, + storageFormat: "Parquet", + partitionKeyLength: 0 + } +} + +// --- Re-derivation material ------------------------------------------------------------- +// +// The block emits no dense per-cell-per-identity table: on a realistic run it is the largest artifact +// the block would produce, and a pMHC panel does not fit at all. The sparse per-tag counts plus the +// per-cell scalars reproduce every per-cell state exactly at a small fraction of the size — a reader +// regrouping the panel re-takes the highest member, re-scores against the same reference and re-votes +// without a re-run. Both are EXPORTS: outputs are visible only to this block's own model, so +// re-derivation material returned as an output reaches nobody. + +// result_cell_counts.csv, keyed (sampleId, cellId, tag). +// The CSV also repeats referenceCount and inCellList on every tag row. They are per-CELL facts and +// are imported once, at their own grain, from result_cell_scalars.csv; importing them here as well +// would put one fact in two columns at two different keys. +cellTagCountsImportSpec := func(sampleAxisSpec, cellAxisSpec, tagAxisSpec) { + return { + axes: [ + { column: "sampleId", spec: sampleAxisSpec }, + { column: "cellId", spec: cellAxisSpec }, + { column: "tag", spec: tagAxisSpec } + ], + columns: [{ + column: "umiCount", + id: "tagUmiCount", + spec: { + name: "pl7.app/antigen/umiCount", + valueType: "Int", + annotations: a(80000, true, { + "pl7.app/label": "Tag UMI count", + "pl7.app/description": "Molecules seen for this tag in this cell, after the count floor. Sparse: a cell with no row for a tag saw nothing for it, which is a reading, not a gap.", + "pl7.app/min": "0", + "pl7.app/isAbundance": "true", + "pl7.app/abundance/unit": "molecules", + "pl7.app/abundance/normalized": "false" + }) + } + }], + storageFormat: "Parquet", + partitionKeyLength: 0 + } +} + +// result_cell_scalars.csv, keyed (sampleId, cellId). +// The comparator is in the domain for the same reason it is on the verdicts: referenceCount and +// admissibility ARE the comparator's own output, so two runs served differently do not carry the +// same reading under one identity. +cellScalarsImportSpec := func(sampleAxisSpec, cellAxisSpec, served) { + scalarCol := func(colName, name, valueType, annotations) { + return { + column: colName, + id: colName, + spec: { + name: name, + valueType: valueType, + domain: servedDomain(served), + annotations: annotations + } + } + } + return { + axes: [ + { column: "sampleId", spec: sampleAxisSpec }, + { column: "cellId", spec: cellAxisSpec } + ], + columns: [ + scalarCol("referenceCount", "pl7.app/antigen/referenceCount", "Int", + a(79000, true, { + "pl7.app/label": "Reference count", + "pl7.app/description": "What this cell's reading was compared against. Empty where the run had no comparator for the cell.", + "pl7.app/min": "0" + })), + scalarCol("admissibility", "pl7.app/antigen/admissibility", "String", + a(78000, true, { + "pl7.app/label": "Admissibility", + "pl7.app/isDiscreteFilter": "true", + "pl7.app/discreteValues": ADMISSIBILITY_VALUES, + "pl7.app/description": "Whether this cell's readings could be compared at all, and if not, why. An inadmissible cell casts no vote; it does not vote 'not bound'." + })), + scalarCol("inCellList", "pl7.app/antigen/inCellList", "String", + a(77000, false, { + "pl7.app/label": "In cell list", + "pl7.app/isDiscreteFilter": "true", + "pl7.app/discreteValues": BOOL_VALUES + })) + ], + storageFormat: "Parquet", + partitionKeyLength: 0 + } +} + +// --- Scope: which identities each sample was stained with -------------------------------- +// +// result_offered.csv, keyed (sampleId, identity). The one piece of per-sample scope the block +// computes and would otherwise throw away: without it "never asked" is a claim a reader cannot check. +// Sparse — a row exists only where the sample's panel offered the identity. +// +// NOTE: the CSV as emit_verdicts.py writes it carries only the two key columns and no value column, +// so the constant "offered" column named here does not exist yet. See the task report. +offeredImportSpec := func(sampleAxisSpec, identityAxisSpec) { + return { + axes: [ + { column: "sampleId", spec: sampleAxisSpec }, + { column: "identity", spec: identityAxisSpec } + ], + columns: [{ + column: "offered", + id: "offered", + spec: { + name: "pl7.app/antigen/offered", + valueType: "String", + annotations: a(76000, false, { + "pl7.app/label": "Offered", + "pl7.app/isDiscreteFilter": "true", + "pl7.app/discreteValues": BOOL_VALUES, + "pl7.app/description": "Present where this sample's panel declared this antigen. Absent means the sample was never stained with it." + }) + } + }], + storageFormat: "Parquet", + partitionKeyLength: 0 + } +} + +// --- tag -> identity linker: result_tag_identity.csv -------------------------------------- +// +// One row per tag. This is what lets a reader put a tag's count next to its verdict without either +// layer knowing about the other: the linker carries both axes and its value is irrelevant, so it is +// a constant column, matching pl7.app/sc/cellLinker's shape. Order priority 0 and no default +// visibility: it is infrastructure and is hidden in tables. +// +// NOTE: as with the offered frame, emit_verdicts.py writes only the two key columns, so the constant +// column named here does not exist yet. See the task report. +tagIdentityLinkerImportSpec := func(tagAxisSpec, identityAxisSpec) { + return { + axes: [ + { column: "tag", spec: tagAxisSpec }, + { column: "identity", spec: identityAxisSpec } + ], + columns: [{ + column: "1", + id: "tagIdentityLinker", + spec: { + name: "pl7.app/antigen/tagIdentityLinker", + valueType: "Int", + annotations: a(0, undefined, { + "pl7.app/label": "Tag / antigen linker", + "pl7.app/isLinkerColumn": "true" + }) + } + }], + storageFormat: "Parquet", + partitionKeyLength: 0 + } +} + +// --- Labels ------------------------------------------------------------------------------- +// +// A label column is satisfied only by a column NAMED pl7.app/label carrying exactly one axis. The +// pl7.app/isLabel annotation is not read by the consumers that matter, so it cannot stand in for the +// name. Two identities must never share a label: where two tags carry the same consistent name, +// emit_verdicts.py appends the tag, which is where the name map lives and the only place the +// collision is visible. +identityLabelsImportSpec := func(identityAxisSpec) { + return { + axes: [{ column: "identity", spec: identityAxisSpec }], + columns: [{ + column: "label", + id: "identityLabel", + spec: { + name: "pl7.app/label", + valueType: "String", + annotations: a(0, undefined, { "pl7.app/label": "Antigen" }) + } + }], + storageFormat: "Parquet", + partitionKeyLength: 0 + } +} + +// A panel has no name in any real panel file, so its label is built from what a reader can recognise +// it by: how many tags it declares and which samples came off it. +panelLabelsImportSpec := func(panelAxisSpec) { + return { + axes: [{ column: "panelId", spec: panelAxisSpec }], + columns: [{ + column: "label", + id: "panelLabel", + spec: { + name: "pl7.app/label", + valueType: "String", + annotations: a(0, undefined, { "pl7.app/label": "Panel" }) + } + }], + storageFormat: "Parquet", + partitionKeyLength: 0 + } +} + +// --- sample -> panel: result_sample_panel.csv ---------------------------------------------- +// +// Keyed [sampleId] with the panel as the value, so per-tag QC keyed (panel, tag) can be read back to +// the samples it covers. Where one panel covers every sample this column is constant and drops out. +samplePanelImportSpec := func(sampleAxisSpec) { + return { + axes: [{ column: "sampleId", spec: sampleAxisSpec }], + columns: [{ + column: "panelId", + id: "panelOfSample", + spec: { + name: "pl7.app/antigen/panelOfSample", + valueType: "String", + annotations: a(75000, false, { + "pl7.app/label": "Panel", + "pl7.app/isDiscreteFilter": "true" + }) + } + }], + storageFormat: "Parquet", + partitionKeyLength: 0 + } +} + +// --- Panel versus reads: result_panel_mismatch.csv ------------------------------------------ +// +// Keyed (panelId, tag), because a per-tag failure is a property of the declared tag set rather than +// of any one sample carrying it; the samples that reported it travel in the row so nothing about +// where it was seen is lost. Emitted as a p-column rather than a raw file: a mismatch report the user +// cannot see defeats its purpose. +panelMismatchImportSpec := func(panelAxisSpec, tagAxisSpec) { + return { + axes: [ + { column: "panelId", spec: panelAxisSpec }, + { column: "tag", spec: tagAxisSpec } + ], + columns: [ + { + column: "direction", + id: "mismatchDirection", + spec: { + name: "pl7.app/antigen/panelMismatchDirection", + valueType: "String", + annotations: a(74000, true, { + "pl7.app/label": "Mismatch", + "pl7.app/isDiscreteFilter": "true", + "pl7.app/discreteValues": MISMATCH_DIRECTIONS, + "pl7.app/description": "'declared-never-seen': the panel declares this tag and no read carried it. 'undeclared-in-panel': reads carried a tag the panel never declared. Both are checked because either alone hides half the mismatch." + }) + } + }, + { + column: "samples", + id: "mismatchSamples", + spec: { + name: "pl7.app/antigen/panelMismatchSamples", + valueType: "String", + annotations: a(73000, true, { "pl7.app/label": "Samples affected" }) + } + } + ], + storageFormat: "Parquet", + partitionKeyLength: 0 + } +} + +// --- Quality measurements: result_qc.csv, keyed (level, entity, measurement) ------------------ +// +// Every declared measurement keeps its place whether or not this run could compute it: a measurement +// nothing computed reads "not evaluated" with its reason rather than being absent, so a reader can +// never mistake "nothing computed this yet" for "this was checked and found fine". `panelId` travels +// as an ordinary column rather than a fourth axis — it is how a panel rollup finds its constituents, +// and a sample-level row simply leaves it empty. +qcImportSpec := func(levelAxisSpec, entityAxisSpec, measurementAxisSpec) { + col := func(colName, name, valueType, label, order, visible, annotations) { + return { + column: colName, + id: colName, + spec: { + name: name, + valueType: valueType, + annotations: a(order, visible, maps.merge(annotations, { "pl7.app/label": label })) + } + } + } + return { + axes: [ + { column: "level", spec: levelAxisSpec }, + { column: "entity", spec: entityAxisSpec }, + { column: "measurement", spec: measurementAxisSpec } + ], + columns: [ + col("value", "pl7.app/antigen/qcValue", "Double", "Value", 72000, true, {}), + col("detail", "pl7.app/antigen/qcDetail", "String", "Detail", 71000, true, {}), + col("status", "pl7.app/antigen/qcStatus", "String", "Status", 70000, true, { + "pl7.app/isDiscreteFilter": "true", + "pl7.app/discreteValues": QC_STATUSES, + "pl7.app/description": "'unjudged' means no line exists to judge this against; 'not evaluated' means nothing computed it. Neither is a pass." + }), + col("judged", "pl7.app/antigen/qcJudged", "Int", "Judged", 69000, false, { "pl7.app/min": "0" }), + col("unjudged", "pl7.app/antigen/qcUnjudged", "Int", "Unjudged", 68000, false, { "pl7.app/min": "0" }), + col("notEvaluated", "pl7.app/antigen/qcNotEvaluated", "Int", "Not evaluated", 67000, false, { "pl7.app/min": "0" }), + col("counts", "pl7.app/antigen/qcCounts", "String", "What it counts", 66000, false, {}), + col("implies", "pl7.app/antigen/qcImplies", "String", "What a bad value means", 65000, false, {}), + col("reason", "pl7.app/antigen/qcReason", "String", "Why deferred", 64000, false, {}), + col("panelId", "pl7.app/antigen/qcPanelId", "String", "Panel", 63000, false, {}) + ], + storageFormat: "Parquet", + partitionKeyLength: 0 + } } export { @@ -384,5 +982,26 @@ export { qcFileMapOutput: qcFileMapOutput, qcSummaryColumnsSpec: qcSummaryColumnsSpec, featurePropertyImportColumns: featurePropertyImportColumns, - negativeControlColumn: negativeControlColumn + negativeControlColumn: negativeControlColumn, + + tagAxis: tagAxis, + identityAxis: identityAxis, + panelAxis: panelAxis, + captureAxis: captureAxis, + qcLevelAxis: qcLevelAxis, + qcEntityAxis: qcEntityAxis, + qcMeasurementAxis: qcMeasurementAxis, + + verdictsImportSpec: verdictsImportSpec, + setCountsImportSpec: setCountsImportSpec, + identitySummaryImportSpec: identitySummaryImportSpec, + cellTagCountsImportSpec: cellTagCountsImportSpec, + cellScalarsImportSpec: cellScalarsImportSpec, + offeredImportSpec: offeredImportSpec, + tagIdentityLinkerImportSpec: tagIdentityLinkerImportSpec, + identityLabelsImportSpec: identityLabelsImportSpec, + panelLabelsImportSpec: panelLabelsImportSpec, + samplePanelImportSpec: samplePanelImportSpec, + panelMismatchImportSpec: panelMismatchImportSpec, + qcImportSpec: qcImportSpec } From 2249cf8f2305b3d8ec5a34872239c1bcaf5183b4 Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 19:29:50 +0200 Subject: [PATCH 050/282] MILAB-6496: give the two key-only frames a value column result_offered.csv and result_tag_identity.csv carried key columns only. A p-column is built from a CSV's value columns, so a key-only file imports as nothing -- silently, since the file exists and is well formed. What each sample was offered, and which identity a tag feeds, would never have left the block, and the tag-to-identity linker the whole cross-layer join depends on could not have been declared at all. Offered carries offered="true"; the linker carries a column named "1" holding 1, matching the cell-linker convention used elsewhere in the platform. --- software/per-cell-metrics/src/emit_verdicts.py | 15 ++++++++++++--- .../per-cell-metrics/test/test_emit_verdicts.py | 15 +++++++++++++++ 2 files changed, 27 insertions(+), 3 deletions(-) diff --git a/software/per-cell-metrics/src/emit_verdicts.py b/software/per-cell-metrics/src/emit_verdicts.py index b5b8962..b4addfa 100644 --- a/software/per-cell-metrics/src/emit_verdicts.py +++ b/software/per-cell-metrics/src/emit_verdicts.py @@ -664,15 +664,24 @@ def main() -> None: ) _write_sorted(cell_scalars, f"{prefix}_cell_scalars.csv", ["sampleId", "cellId"]) + # Both of these frames are pure key sets -- what a sample was offered, and + # which identity a tag feeds -- and each carries a constant value column so + # it can become a p-column at all. A frame of key columns alone imports as + # nothing: columns are built from value columns, so a key-only file yields + # no column and the fact it records never leaves the block. offered_frame = pl.DataFrame( - [(sample, identity) for sample in samples for identity in sorted(offered_by_sample[sample])], + [(sample, identity, "true") for sample in samples for identity in sorted(offered_by_sample[sample])], orient="row", - schema={"sampleId": pl.String, "identity": pl.String}, + schema={"sampleId": pl.String, "identity": pl.String, "offered": pl.String}, ) _write_sorted(offered_frame, f"{prefix}_offered.csv", ["sampleId", "identity"]) + # The value column is named "1" and holds 1, matching the cell-linker + # convention already used for linker columns elsewhere in the platform. linker_frame = pl.DataFrame( - sorted(grouping.items()), orient="row", schema={"tag": pl.String, "identity": pl.String} + [(tag, identity, 1) for tag, identity in sorted(grouping.items())], + orient="row", + schema={"tag": pl.String, "identity": pl.String, "1": pl.Int64}, ) _write_sorted(linker_frame, f"{prefix}_tag_identity.csv", ["tag", "identity"]) diff --git a/software/per-cell-metrics/test/test_emit_verdicts.py b/software/per-cell-metrics/test/test_emit_verdicts.py index 7666156..b174160 100644 --- a/software/per-cell-metrics/test/test_emit_verdicts.py +++ b/software/per-cell-metrics/test/test_emit_verdicts.py @@ -121,6 +121,21 @@ def test_reference_source_none_produces_unreliable_not_a_crash(bed): assert v.filter(pl.col("identity") == "AAAA").row(0, named=True)["state"] == "unreliable" +def test_the_key_only_frames_carry_a_value_column_so_they_can_become_columns(bed): + # A p-column is built from a CSV's *value* columns, so a file of key columns + # alone imports as nothing at all -- silently, since the file exists and is + # well formed. What a sample was offered, and which identity a tag feeds, + # would simply never leave the block. + _run(bed, *BASE) + offered = pl.read_csv(bed / "result_offered.csv", infer_schema_length=0) + assert offered.columns == ["sampleId", "identity", "offered"] + assert set(offered["offered"].to_list()) == {"true"} + + linker = pl.read_csv(bed / "result_tag_identity.csv", infer_schema_length=0) + assert linker.columns == ["tag", "identity", "1"] + assert set(linker["1"].to_list()) == {"1"} + + def test_a_panel_with_no_declared_reference_falls_to_the_panel_not_to_nothing(bed): # Three rungs in order: a declared reagent, else the panel's own readings # where the panel carries enough members, else nothing. Skipping the middle From 44c229981067b1a8f2096f65c9873460e49c5813 Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 19:50:35 +0200 Subject: [PATCH 051/282] MILAB-6496: make two spec-required facts visible in the output Where a per-tag figure alerts, the identity figures for the identities that tag feeds are now shown beside it. attach_alerting_identities already implemented this and was imported nowhere, so nothing computed which tags alert. A noisy reagent whose identities read steady is a reagent to replace, not a run to distrust, and only the two numbers together say which. Neither is suppressed: the identity rows are still emitted in full, and the attachment is a copy on the row that raised the question. A tag the grouping property could not place is now named in the run record. It keeps its own identity rather than vanishing, so a bare barcode sits among the family identities; previously the only trace was a stderr line truncated to the first eight tags. A property the panel file does not carry narrows what can be answered, and the narrowing belongs where the answers are. --- .../per-cell-metrics/src/emit_verdicts.py | 49 +++++++++++- .../test/test_emit_verdicts.py | 75 +++++++++++++++++++ 2 files changed, 121 insertions(+), 3 deletions(-) diff --git a/software/per-cell-metrics/src/emit_verdicts.py b/software/per-cell-metrics/src/emit_verdicts.py index b4addfa..ee63be8 100644 --- a/software/per-cell-metrics/src/emit_verdicts.py +++ b/software/per-cell-metrics/src/emit_verdicts.py @@ -75,6 +75,7 @@ Coverage, Status, antigen_count_deciles, + attach_alerting_identities, outlier_status, per_antigen_measures, reads_per_cell, @@ -218,7 +219,7 @@ def _build_grouping( """ by_tag = default_grouping(panel, reference_tags) if rule is None or rule.get("by") == "tag": - return by_tag, "per-tag" + return by_tag, "per-tag", [] if rule.get("by") != "property": raise SystemExit(f"--grouping must be {{'by':'tag'}} or {{'by':'property','column':...}}; got {rule!r}") @@ -237,12 +238,17 @@ def _build_grouping( grouping[tag] = tag ungrouped.append(tag) if ungrouped: + # Also returned, not only logged. A property the file does not carry + # narrows what can be answered, and that narrowing has to be visible in + # the output rather than in a log line nobody reads afterwards: these + # tags are answered under a grouping that could not place them, so a + # bare barcode sits among the family identities and only this says why. print( f"[emit-verdicts] {len(ungrouped)} tag(s) carry no agreed {column!r} value and stand as their own " f"identity: {ungrouped[:8]}", file=sys.stderr, ) - return grouping, f"property:{column}" + return grouping, f"property:{column}", ungrouped def _identity_labels( @@ -483,7 +489,7 @@ def main() -> None: reference_tags = {t for t, props in properties.items() if props.get(args.role_column) in reference_values} grouping_rule = _json_arg(args.grouping, "--grouping") - grouping, grouping_id = _build_grouping(grouping_rule, panel, properties, reference_tags) + grouping, grouping_id, ungrouped_tags = _build_grouping(grouping_rule, panel, properties, reference_tags) universe = identity_universe(panel, grouping) by_tag_grouping = default_grouping(panel, reference_tags) tag_universe = identity_universe(panel, by_tag_grouping) @@ -888,10 +894,43 @@ def _number(row: dict, column: str) -> float | None: # being judged: including it would inflate the upper quartile it is # then measured against, so the one reading the measure exists to # catch is the one it would miss. + disagreement_at = len(rows) for tag in sorted(panel_tags & set(tag_rate)): peers = [tag_rate[o] for o in panel_tags if o != tag and tag_rate.get(o) is not None] status = outlier_status(tag_rate[tag], peers) rows.append(_leaf("tag", tag, "tagDisagreement", tag_rate[tag], "", panel_id, status)) + + # Beside an alerting tag, the figures for the identities it feeds. A + # noisy reagent whose identities read steady is a reagent to replace, + # not a run to distrust, and only the two numbers together say which. + # Neither is suppressed: the identity rows are emitted in full below, + # and this attaches a copy to the tag that raised the question so a + # reader meeting the alert does not have to go looking. + alerting_tags = {r.entity for r in rows[disagreement_at:] if r.status is Status.ALERTING} + if alerting_tags: + beside = attach_alerting_identities( + pl.DataFrame( + [ + (identity, identity_rate[identity]) + for identity in sorted(identities_of_panel[panel_id] & set(identity_rate)) + ], + orient="row", + schema={"key": pl.String, "identityDisagreement": pl.Float64}, + ), + {tag: {grouping[tag]} for tag in panel_tags if tag in grouping}, + alerting_tags, + ) + attached: dict[str, list[str]] = {} + for row in beside.iter_rows(named=True): + rate = row["identityDisagreement"] + attached.setdefault(row["tag"], []).append( + f"{row['identity']}={'' if rate is None else round(float(rate), 4)}" + ) + for i in range(disagreement_at, len(rows)): + feeds = attached.get(rows[i].entity) + if feeds: + rows[i] = rows[i]._replace(detail=f"identitiesFed={'|'.join(feeds)}") + tag_statuses = [r.status for r in rows[first:]] identity_first = len(rows) @@ -953,6 +992,10 @@ def _number(row: dict, column: str) -> float | None: "referenceTags": sorted(reference_tags), "grouping": grouping_rule or {"by": "tag"}, "groupingId": grouping_id, + # The narrowing a short panel file costs, carried in the output + # rather than only in a log line: these tags were answered under a + # grouping that could not place them. + "tagsWithoutGroupingValue": sorted(ungrouped_tags), "contending": [sorted(group) for group in contending], "identityCount": len(universe), "identitySummaryEmitted": summary_emitted, diff --git a/software/per-cell-metrics/test/test_emit_verdicts.py b/software/per-cell-metrics/test/test_emit_verdicts.py index b174160..b4f4817 100644 --- a/software/per-cell-metrics/test/test_emit_verdicts.py +++ b/software/per-cell-metrics/test/test_emit_verdicts.py @@ -121,6 +121,81 @@ def test_reference_source_none_produces_unreliable_not_a_crash(bed): assert v.filter(pl.col("identity") == "AAAA").row(0, named=True)["state"] == "unreliable" +def test_a_tag_the_grouping_could_not_place_is_named_in_the_output(bed): + # A property the panel file does not carry narrows what can be answered, + # and the narrowing has to be visible where the answers are. Such a tag + # keeps its own identity rather than vanishing, so a bare barcode sits + # among the family identities -- inferable from the labels, but only this + # says why it is there. + (bed / "panel.csv").write_text( + "Samples,Name,Sequence,Type,Family\n" + "S1,AgA,AAAA,Target,Spike\n" + "S1,AgB,CCCC,Target,\n" + "S1,Ctrl,CTRL,Control,Reference\n" + ) + (bed / "counts.csv").write_text((bed / "counts.csv").read_text() + "S1,c1,CCCC,40\nS1,c2,CCCC,40\nS1,c3,CCCC,40\n") + r = _run(bed, *BASE, "--grouping", json.dumps({"by": "property", "column": "Family"})) + assert r.returncode == 0, r.stderr + + meta = json.loads((bed / "result_run_meta.json").read_text()) + assert meta["tagsWithoutGroupingValue"] == ["CCCC"] + + identities = set(pl.read_csv(bed / "result_verdicts.csv", infer_schema_length=0)["identity"].to_list()) + assert identities == {"Spike", "CCCC"}, "the unplaceable tag keeps its own identity rather than vanishing" + + +def test_a_tag_grouping_reports_no_unplaceable_tags(bed): + # The default grouping places every tag by construction, so the field is + # present and empty rather than absent -- a reader must be able to tell + # "none" from "not checked". + _run(bed, *BASE) + meta = json.loads((bed / "result_run_meta.json").read_text()) + assert meta["tagsWithoutGroupingValue"] == [] + + +def test_an_alerting_tag_carries_the_figures_for_the_identities_it_feeds(bed): + # A noisy reagent whose identities read steady is a reagent to replace, not + # a run to distrust, and only the two numbers together say which. One tag + # whose clonotype disagrees with itself, against four that do not: its rate + # stands clear of its peers, so it alerts and must carry the identity + # figures beside it. + tags = [f"T{i:02d}" for i in range(5)] + (bed / "panel.csv").write_text( + "Samples,Name,Sequence,Type\n" + + "".join(f"S1,Ag{i},{t},Target\n" for i, t in enumerate(tags)) + + "S1,Ctrl,CTRL,Control\n" + ) + rows = ["sampleId,cellId,tag,umiCount"] + for cell in ("c1", "c2", "c3", "c4"): + rows.append(f"S1,{cell},CTRL,6") + # T00 splits the clonotype: two cells bind it, two do not. + rows.append(f"S1,{cell},{tags[0]},{500 if cell in ('c1', 'c2') else 5}") + rows.extend(f"S1,{cell},{t},5" for t in tags[1:]) + (bed / "counts.csv").write_text("\n".join(rows) + "\n") + (bed / "linker.csv").write_text("sampleId,cellId,setId\nS1,c1,K1\nS1,c2,K1\nS1,c3,K1\nS1,c4,K1\n") + + r = _run(bed, *BASE) + assert r.returncode == 0, r.stderr + qc = pl.read_csv(bed / "result_qc.csv", infer_schema_length=0) + tag_rows = qc.filter(pl.col("measurement") == "tagDisagreement") + + alerting = tag_rows.filter(pl.col("status") == "alerting") + assert alerting.height == 1, "exactly the split tag should stand clear of its peers" + row = alerting.row(0, named=True) + assert row["entity"] == tags[0] + assert row["detail"].startswith("identitiesFed="), row["detail"] + assert tags[0] in row["detail"] + + # Neither figure is suppressed: the identity rows are still emitted in full. + assert qc.filter(pl.col("measurement") == "identityDisagreement").height > 0 + + # And a tag that did not alert carries no attachment -- the pairing is the + # answer to a question the alert raised, not decoration on every row. + quiet = tag_rows.filter(pl.col("status") != "alerting") + assert quiet.height > 0 + assert not any((d or "").startswith("identitiesFed=") for d in quiet["detail"].to_list()) + + def test_the_key_only_frames_carry_a_value_column_so_they_can_become_columns(bed): # A p-column is built from a CSV's *value* columns, so a file of key columns # alone imports as nothing at all -- silently, since the file exists and is From 190816f8ba16540c755df57f53f963d6cbb0c7fa Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 20:00:04 +0200 Subject: [PATCH 052/282] MILAB-6496: acceptance scenarios driven through the real pipeline Each scenario writes a counts CSV, a panel CSV and a linker CSV and runs emit_verdicts.py as a subprocess. No test constructs a per-cell state frame, which is how an earlier revision of these checks passed while the reading was turning an antigen every cell failed to bind into 'never asked'. Covers epitope loss produced from zero count rows, an unasked off-target that leaves one clonotype unsettled and a bound off-target that disqualifies another, support of 40 cells against 3 inside one clonotype's row set, and a set whose every cell the gate set aside. --- .../test/test_acceptance_scenarios.py | 384 ++++++++++++++++++ 1 file changed, 384 insertions(+) create mode 100644 software/per-cell-metrics/test/test_acceptance_scenarios.py diff --git a/software/per-cell-metrics/test/test_acceptance_scenarios.py b/software/per-cell-metrics/test/test_acceptance_scenarios.py new file mode 100644 index 0000000..bfaf7c3 --- /dev/null +++ b/software/per-cell-metrics/test/test_acceptance_scenarios.py @@ -0,0 +1,384 @@ +"""The spec's acceptance scenarios, each driven from files through the CLI. + +Every scenario writes a counts CSV, a panel CSV and a linker CSV, runs +`emit_verdicts.py` as a subprocess, and asserts on the CSVs it wrote. Nothing +here builds a per-cell state frame, calls `read_states`, or reaches into a +module: an earlier revision of these scenarios did exactly that and passed +while the pipeline was turning an antigen every cell failed to bind into +*never asked*. A scenario that constructs the states it then reads tests its +own assertion, not the reading. + +Three numbers are load-bearing in every bed below, so they are stated once +here rather than rediscovered by whoever next changes a count. + +*The cutoff is 75 and the score is a beta function, not a ratio.* Against a +reference of 6, a count of 500 scores 100 and binds, while counts of 50 and 60 +score 3.1 and 7.2 and read *not bound* -- large-looking counts that cannot +reach the cutoff. Against a reference of 20 a count of 500 still scores 99.85. +Check the score before asserting a state; `specificity_score(count, reference)` +in verdict.py answers directly. + +*The floor is 4.* Any antigen reading of 1-3 is zeroed before anything else +runs, so background counts here sit at 5 or above. A floored reading can also +drag a panel-derived comparator to zero, put every cell below the reference +thin line of 2, and turn a whole run *unreliable* for a reason that has +nothing to do with the scenario. + +*A cell with no comparator reading is inadmissible and votes nowhere.* Every +bed declares a comparator tag whose role value matches `--reference-values`, +and gives every one of its cells a count for it. + +Tags are the identities under the default per-tag grouping, so they are named +for the part they play (`TARGET`, `OFF1`) rather than written as barcode +sequences. The pipeline treats a tag as an opaque string. +""" + +import json +import subprocess +import sys +from pathlib import Path + +import polars as pl +import pytest + +SRC = Path(__file__).resolve().parents[1] / "src" + + +def _run(cwd, *args): + return subprocess.run( + [sys.executable, str(SRC / "emit_verdicts.py"), *map(str, args)], cwd=cwd, capture_output=True, text=True + ) + + +BASE = [ + "counts.csv", + "panel.csv", + "--linker", + "linker.csv", + "--barcode-col", + "Sequence", + "--feature-col", + "Name", + "--sample-col", + "Samples", + "--role-column", + "Type", + "--reference-values", + "Control", + "--output-prefix", + "result", +] + +# A comparator reading every cell shares. Above the floor of 4 so it survives +# it, above the reference thin line of 2 so the cell is admissible, and well +# below the high-reference observation line of 100 so nothing in these beds is +# flagged for background it does not have. +COMPARATOR = 6 + +# Clears the cutoff of 75 against a comparator of 6: the score is 100. +BINDING = 500 + +# Survives the floor of 4 and scores 0.0 against a comparator of 6. A reading +# that is present and settles *not bound*, as distinct from a cell that was +# asked and produced no row at all. +BACKGROUND = 5 + + +def _verdicts(bed): + # Read without schema inference throughout: `unreliableReason` is null on a + # settled row, and polars would otherwise infer the counts back into + # integers and the reason column's nulls into something a test cannot tell + # from an empty string. + return pl.read_csv(bed / "result_verdicts.csv", infer_schema_length=0) + + +def _row(bed, set_id, identity): + got = _verdicts(bed).filter((pl.col("setId") == set_id) & (pl.col("identity") == identity)) + assert got.height == 1, f"expected exactly one ({set_id}, {identity}) row, got {got.height}" + return got.row(0, named=True) + + +# --------------------------------------------------------------------------- +# Epitope loss: the finding is a failure, and the failure comes from silence. +# --------------------------------------------------------------------------- + + +@pytest.fixture +def epitope_bed(tmp_path): + """One clonotype against an unmutated antigen and four point mutants. + + The fourth mutant has **no rows at all** in the counts file. That is the + whole point of the bed: tag-stat emits only observed (cell, tag) pairs, so + an antigen every cell failed to bind arrives as nothing, and the reading + has to recover *not bound* from the panel saying those cells were offered + it. Writing zero-count rows instead would hand the pipeline the answer. + """ + (tmp_path / "panel.csv").write_text( + "Samples,Name,Sequence,Type\n" + "S1,AgWT,WT,Target\n" + "S1,AgM1,M1,Target\n" + "S1,AgM2,M2,Target\n" + "S1,AgM3,M3,Target\n" + "S1,AgM4,M4,Target\n" + "S1,Ctrl,CTRL,Control\n" + ) + cells = ("c1", "c2", "c3", "c4") + rows = ["sampleId,cellId,tag,umiCount"] + for cell in cells: + rows.append(f"S1,{cell},CTRL,{COMPARATOR}") + # The clone binds the unmutated antigen and the first three mutants; + # the epitope it grabs survives those substitutions. + for tag in ("WT", "M1", "M2", "M3"): + rows.append(f"S1,{cell},{tag},{BINDING}") + # M4 deliberately absent -- not zero, absent. + (tmp_path / "counts.csv").write_text("\n".join(rows) + "\n") + (tmp_path / "linker.csv").write_text("sampleId,cellId,setId\n" + "".join(f"S1,{cell},K1\n" for cell in cells)) + return tmp_path + + +def test_the_mutant_no_cell_bound_reads_not_bound_not_never_asked(epitope_bed): + # The scientist's statement is "binds the unmutated antigen and fails on + # the fourth mutant", so *not bound* is the finding and the run must + # produce it from silence. Reading M4 as *never asked* -- the failure this + # scenario exists to catch -- turns the finding into a gap, and the + # clonotype whose whole value is that failure goes back as unsettled. + r = _run(epitope_bed, *BASE) + assert r.returncode == 0, r.stderr + + wt = _row(epitope_bed, "K1", "WT") + assert wt["state"] == "bound" + + m4 = _row(epitope_bed, "K1", "M4") + assert m4["state"] == "not bound", "the cells were offered M4 and were silent; silence is a failure to bind" + assert m4["unreliableReason"] is None, "a settled reading carries no reason for not settling" + + # Every one of the four cells was offered M4 and every one of them voted. + # A reading resting on the four silences is what makes the failure a + # finding rather than an absence of data. + assert (int(m4["cellsCouldAnswer"]), int(m4["cellsAnswered"])) == (4, 4) + + +def test_the_silent_mutant_is_reported_as_a_reagent_that_produced_nothing(epitope_bed): + # Two different statements, both true and neither substituting for the + # other: the verdict says the clone failed to bind M4, and the quality + # measurement says M4 returned no reads in this run. A reader deciding + # whether the failure is biology or a dead reagent needs both, and the + # verdict must not be suppressed to make the second point. + assert _run(epitope_bed, *BASE).returncode == 0 + qc = pl.read_csv(epitope_bed / "result_qc.csv", infer_schema_length=0) + never_seen = qc.filter((pl.col("measurement") == "declaredNeverSeen") & (pl.col("entity") == "M4")) + assert never_seen.height == 1 + assert float(never_seen.row(0, named=True)["value"]) == 0.0 + assert _row(epitope_bed, "K1", "M4")["state"] == "not bound" + + +# --------------------------------------------------------------------------- +# Unasked off-target: one clonotype left unsettled, one positively disqualified. +# --------------------------------------------------------------------------- + + +@pytest.fixture +def off_target_bed(tmp_path): + """Two clonotypes under "binds the target and nothing on the off-target list". + + The list is OFF1, OFF2, OFF3. Neither sample's panel carries all three, and + the two samples omit different ones, which is what puts the two clonotypes + on opposite sides of the statement. + + KA comes from S1, whose panel omits OFF3: KA has a clean reading on the + off-targets it was asked about and an unsettled position on the one it was + not. KB comes from S2, whose panel omits OFF2 -- but KB **binds** OFF1, + which S2 did ask. The run disqualified KB on a position it settled, and the + unasked one changes nothing about that. + """ + (tmp_path / "panel.csv").write_text( + "Samples,Name,Sequence,Type\n" + "S1,AgTarget,TARGET,Target\n" + "S1,AgOff1,OFF1,Target\n" + "S1,AgOff2,OFF2,Target\n" + "S1,Ctrl,CTRL,Control\n" + "S2,AgTarget,TARGET,Target\n" + "S2,AgOff1,OFF1,Target\n" + "S2,AgOff3,OFF3,Target\n" + "S2,Ctrl,CTRL,Control\n" + ) + rows = ["sampleId,cellId,tag,umiCount"] + for cell in ("a1", "a2", "a3"): + rows.append(f"S1,{cell},CTRL,{COMPARATOR}") + rows.append(f"S1,{cell},TARGET,{BINDING}") + # OFF1 read low and OFF2 read nothing at all: both routes to *not + # bound* in one clonotype, so the clean off-target list does not rest + # on either route alone. + rows.append(f"S1,{cell},OFF1,{BACKGROUND}") + for cell in ("b1", "b2", "b3"): + rows.append(f"S2,{cell},CTRL,{COMPARATOR}") + rows.append(f"S2,{cell},TARGET,{BINDING}") + rows.append(f"S2,{cell},OFF1,{BINDING}") + rows.append(f"S2,{cell},OFF3,{BACKGROUND}") + (tmp_path / "counts.csv").write_text("\n".join(rows) + "\n") + (tmp_path / "linker.csv").write_text( + "sampleId,cellId,setId\n" + + "".join(f"S1,{cell},KA\n" for cell in ("a1", "a2", "a3")) + + "".join(f"S2,{cell},KB\n" for cell in ("b1", "b2", "b3")) + ) + return tmp_path + + +def test_an_off_target_the_panel_omitted_is_present_and_reads_never_asked(off_target_bed): + # The row has to exist. Dropping it discards a lead for a question nobody + # asked; keeping it as anything settled asserts a clean off-target the run + # never produced. Present, in a state that says the statement could not be + # settled, naming the position responsible. + r = _run(off_target_bed, *BASE) + assert r.returncode == 0, r.stderr + + unasked = _row(off_target_bed, "KA", "OFF3") + assert unasked["state"] == "never asked" + assert unasked["unreliableReason"] == "never-offered" + assert int(unasked["cellsCouldAnswer"]) == 0 # no cell of KA was ever offered OFF3 + + # And the positions S1 did ask are settled, so the clonotype is unsettled + # by exactly one position rather than by a bed that says nothing. + assert _row(off_target_bed, "KA", "TARGET")["state"] == "bound" + assert _row(off_target_bed, "KA", "OFF1")["state"] == "not bound" + assert _row(off_target_bed, "KA", "OFF2")["state"] == "not bound" + + +def test_a_bound_off_target_survives_beside_an_unasked_one(off_target_bed): + # The other half of the check. The obvious way to satisfy the scenario + # above -- let any unsettled position make the whole statement unsettled -- + # sends a demonstrated off-target binder back as a maybe, silently, which + # is the direction that costs money. KB's bound OFF1 must reach the output + # so a downstream statement can fail KB on it. + r = _run(off_target_bed, *BASE) + assert r.returncode == 0, r.stderr + + assert _row(off_target_bed, "KB", "OFF2")["state"] == "never asked" + + bound_off_target = _row(off_target_bed, "KB", "OFF1") + assert bound_off_target["state"] == "bound" + assert int(bound_off_target["cellsAnswered"]) == 3 + + # The same identity settled in opposite directions for the two clonotypes, + # each from its own cells. A pipeline that folded the unasked position into + # the whole statement would make these two look alike. + assert _row(off_target_bed, "KA", "OFF1")["state"] == "not bound" + + # `set_counts` is what a ranked list is built from, so KB's disqualifying + # bind has to be countable there too: target plus off-target, two bound. + counts = pl.read_csv(off_target_bed / "result_set_counts.csv", infer_schema_length=0) + assert int(counts.filter(pl.col("setId") == "KB").row(0, named=True)["boundCount"]) == 2 + + +# --------------------------------------------------------------------------- +# Support travels with the reading. +# --------------------------------------------------------------------------- + + +@pytest.fixture +def support_bed(tmp_path): + """One clonotype spanning two samples whose panels share nothing but the comparator. + + Forty of the clone's cells sit in S1, which offered AGA and not AGB; three + sit in S2, which offered AGB and not AGA. Both positions bind, so the + states are identical and the only thing separating a reading resting on + forty cells from one resting on three is the support carried beside it. + """ + (tmp_path / "panel.csv").write_text( + "Samples,Name,Sequence,Type\nS1,AgA,AGA,Target\nS1,Ctrl,CTRL,Control\nS2,AgB,AGB,Target\nS2,Ctrl,CTRL,Control\n" + ) + deep = [f"d{i:02d}" for i in range(40)] + thin = ["t1", "t2", "t3"] + rows = ["sampleId,cellId,tag,umiCount"] + for cell in deep: + rows.append(f"S1,{cell},CTRL,{COMPARATOR}") + rows.append(f"S1,{cell},AGA,{BINDING}") + for cell in thin: + rows.append(f"S2,{cell},CTRL,{COMPARATOR}") + rows.append(f"S2,{cell},AGB,{BINDING}") + (tmp_path / "counts.csv").write_text("\n".join(rows) + "\n") + (tmp_path / "linker.csv").write_text( + "sampleId,cellId,setId\n" + + "".join(f"S1,{cell},K1\n" for cell in deep) + + "".join(f"S2,{cell},K1\n" for cell in thin) + ) + return tmp_path + + +def test_a_reading_on_forty_cells_and_one_on_three_are_distinguishable(support_bed): + # Cells of one clonotype are replicates of one measurement, so how many + # could answer is how much confidence the reading deserves. Both positions + # here read *bound*, so a row carrying only the state makes a decision + # taken on three cells indistinguishable from one taken on forty -- inside + # a single clonotype's row set, which is where the two really do differ. + r = _run(support_bed, *BASE) + assert r.returncode == 0, r.stderr + + deep = _row(support_bed, "K1", "AGA") + thin = _row(support_bed, "K1", "AGB") + assert deep["state"] == thin["state"] == "bound" + + assert (int(deep["cellsCouldAnswer"]), int(deep["cellsAnswered"])) == (40, 40) + assert (int(thin["cellsCouldAnswer"]), int(thin["cellsAnswered"])) == (3, 3) + assert int(deep["cellsAnswered"]) != int(thin["cellsAnswered"]) + + # Agreement travels the same way: both readings are unanimous, and a + # reader has the figure rather than having to infer it from the states. + assert float(deep["agreement"]) == 1.0 and float(thin["agreement"]) == 1.0 + + +# --------------------------------------------------------------------------- +# Every cell set aside: the question was put and the data cannot settle it. +# --------------------------------------------------------------------------- + + +@pytest.fixture +def gated_bed(tmp_path): + """A clonotype whose every cell sits in high comparator background. + + The comparator reads 20 in every cell, so a gate at 10 sets all three + aside. The antigen count of 500 scores 99.85 against a comparator of 20 and + binds outright with the gate off -- which is what makes the *unreliable* + reading the gate's doing rather than an absence of signal. Nothing injects + an `unreliable` row; the state is reached by running the same bed twice, + once through the gate and once past it. + """ + (tmp_path / "panel.csv").write_text("Samples,Name,Sequence,Type\nS1,AgA,AGA,Target\nS1,Ctrl,CTRL,Control\n") + cells = ("c1", "c2", "c3") + rows = ["sampleId,cellId,tag,umiCount"] + for cell in cells: + # 20 is above the reference thin line of 2, so these cells are + # admissible until the gate is what sets them aside -- and below the + # high-reference observation line of 100, so the bed is not also + # exercising that measurement. + rows.append(f"S1,{cell},CTRL,20") + rows.append(f"S1,{cell},AGA,{BINDING}") + (tmp_path / "counts.csv").write_text("\n".join(rows) + "\n") + (tmp_path / "linker.csv").write_text("sampleId,cellId,setId\n" + "".join(f"S1,{cell},K1\n" for cell in cells)) + return tmp_path + + +def test_a_set_whose_every_cell_was_gated_reads_unreliable_and_never_not_bound(gated_bed): + # The cells were dropped because their readings could not be trusted, so + # nothing about the receptor was established. *Not bound* would assert a + # clean reading the run never produced, in the direction that costs money; + # *never asked* would claim the experiment did not put the question, which + # it did. + r = _run(gated_bed, *BASE, "--gate-threshold", "10") + assert r.returncode == 0, r.stderr + + meta = json.loads((gated_bed / "result_run_meta.json").read_text()) + assert meta["cellsSetAside"] == 3, "the gate has to be what removed them" + + gated = _row(gated_bed, "K1", "AGA") + assert gated["state"] == "unreliable" + assert gated["unreliableReason"] == "all-cells-gated" + # The question was put to three cells and none of them could answer it. + # Reporting zero on both would lose the distinction the state carries. + assert (int(gated["cellsCouldAnswer"]), int(gated["cellsAnswered"])) == (3, 0) + + # Same files, gate off: the identity binds outright. Without this the test + # would pass just as well over a bed with no signal in it, where the + # *unreliable* reading says nothing about the gate. + assert _run(gated_bed, *BASE).returncode == 0 + assert _row(gated_bed, "K1", "AGA")["state"] == "bound" From 8dabb45a039e72dd6f330c6a555a421751f62247 Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 20:16:10 +0200 Subject: [PATCH 053/282] MILAB-6496: a synthetic fixture bed for the awkward panel shapes Every verdict test so far writes its own three-line bed, which keeps each one readable and none of them realistic: a run whose panel is one size, whose comparator is one tag and whose cells all come from one sample cannot show what happens when four samples were stained differently. The bed at software/test-data/fixtures/verdicts/ carries those shapes at once -- four samples with panels of 3, 4, 4 and 5 tags, four barcodes recurring under two antigen names each, one antigen on two barcodes, no comparator / one / two, and a barcode declared on one sample and read on another. Three panel files share one counts file, so the same readings can be put against a comparator of 6, of 60 and of none. Every count is chosen against a named threshold and the choice is written down: 8 clears the floor of 4 and still scores 0.0001, so it is compared and fails; 500 scores 100 against a comparator of 6 and 0.1 against 60; 5000 survives either; the one comparator reading of 1 sits below the thin line of 2 and is the bed's only source of *unreliable*. Seven tests read the bed through the CLI: identity keyed by barcode where the names would split, all four states in one run, *never asked* on the short panel, both directions of the per-sample panel-versus-reads check, one antigen read by its highest member, the higher of two comparators serving, and a panel with no comparator standing in as its own. The handles they need are recovered from the files by the role each barcode plays rather than written down, so a bed regenerated under a different seed still exercises the same shapes. The generator is stdlib only and takes one fixed seed. Barcodes are random ACGT, antigens are AgNN, samples are SNN: the repository is public and nothing here comes from a real panel. --- .../test/test_emit_verdicts.py | 261 +++++++++++++++++- .../test-data/fixtures/verdicts/README.md | 77 ++++++ .../test-data/fixtures/verdicts/counts.csv | 64 +++++ .../test-data/fixtures/verdicts/generate.py | 232 ++++++++++++++++ .../test-data/fixtures/verdicts/linker.csv | 12 + .../test-data/fixtures/verdicts/panel.csv | 17 ++ .../verdicts/panel_multi_reference.csv | 25 ++ .../verdicts/panel_with_reference.csv | 21 ++ 8 files changed, 708 insertions(+), 1 deletion(-) create mode 100644 software/test-data/fixtures/verdicts/README.md create mode 100644 software/test-data/fixtures/verdicts/counts.csv create mode 100644 software/test-data/fixtures/verdicts/generate.py create mode 100644 software/test-data/fixtures/verdicts/linker.csv create mode 100644 software/test-data/fixtures/verdicts/panel.csv create mode 100644 software/test-data/fixtures/verdicts/panel_multi_reference.csv create mode 100644 software/test-data/fixtures/verdicts/panel_with_reference.csv diff --git a/software/per-cell-metrics/test/test_emit_verdicts.py b/software/per-cell-metrics/test/test_emit_verdicts.py index b4f4817..de5dcfa 100644 --- a/software/per-cell-metrics/test/test_emit_verdicts.py +++ b/software/per-cell-metrics/test/test_emit_verdicts.py @@ -1,11 +1,12 @@ import json +import shutil import subprocess import sys from pathlib import Path import polars as pl import pytest -from verdict import ReferenceChoice +from verdict import DEFAULT_PANEL_MIN_MEMBERS, ReferenceChoice SRC = Path(__file__).resolve().parents[1] / "src" @@ -589,3 +590,261 @@ def test_the_floor_runs_before_tags_combine(bed): assert meta["cellsEmptied"] == 1 v = pl.read_csv(bed / "result_verdicts.csv") assert v.filter(pl.col("identity") == "Spike").row(0, named=True)["state"] == "not bound" + + +# ---- the committed fixture bed --------------------------------------------------------------- +# +# Every test above writes its own three-line bed, which keeps each one readable and keeps none of +# them realistic: a run whose panel is one size, whose comparator is one tag and whose cells all +# come from one sample cannot show what happens when four samples were stained differently. The +# committed bed at software/test-data/fixtures/verdicts/ carries the awkward panel shapes at once -- +# panels of differing size, barcodes recurring under different names, one antigen on two barcodes, +# one comparator and two, and a barcode declared on one sample and read on another. + +VERDICT_BED = Path(__file__).resolve().parents[2] / "test-data" / "fixtures" / "verdicts" +VERDICT_BED_FILES = ("counts.csv", "linker.csv", "panel.csv", "panel_with_reference.csv", "panel_multi_reference.csv") + +NAME_GROUPING = ("--grouping", json.dumps({"by": "property", "column": "Name"})) + + +@pytest.fixture +def wide_bed(tmp_path): + """The committed bed, copied so a run's output files never land in the repository.""" + for name in VERDICT_BED_FILES: + source = VERDICT_BED / name + if not source.exists(): + pytest.fail(f"committed bed missing at {source}; regenerate it with generate.py", pytrace=False) + shutil.copy(source, tmp_path / name) + return tmp_path + + +def _bed_args(panel_csv, *extra): + # The bed's column names are the ones BASE already names, so only the panel file varies across + # the three shapes: no comparator, one comparator, two. + return ["counts.csv", panel_csv, *BASE[2:], *extra] + + +def _bed_shape(bed): + """The handles these tests need, recovered from the bed by the role each barcode plays. + + Derived rather than written down because the sequences come from a seeded RNG. A bed regenerated + under a different seed still has four barcodes carrying two antigen names, one antigen carried on + two barcodes and one barcode declared by a single sample and read only in another; spelling the + sequences out here would tie every assertion below to the seed instead of to the shape. + """ + panel = pl.read_csv(bed / "panel_multi_reference.csv", infer_schema_length=0) + counts = pl.read_csv(bed / "counts.csv", infer_schema_length=0) + linker = pl.read_csv(bed / "linker.csv", infer_schema_length=0) + + names: dict[str, set[str]] = {} + declared_in: dict[str, set[str]] = {} + offered: dict[str, set[str]] = {} + for row in panel.filter(pl.col("Type") != "Control").iter_rows(named=True): + names.setdefault(row["Sequence"], set()).add(row["Name"]) + declared_in.setdefault(row["Sequence"], set()).add(row["Samples"]) + offered.setdefault(row["Samples"], set()).add(row["Sequence"]) + + tags_of_name: dict[str, set[str]] = {} + for tag, tag_names in names.items(): + for name in tag_names: + tags_of_name.setdefault(name, set()).add(tag) + + read_in: dict[str, set[str]] = {} + for sample, tag in counts.select("sampleId", "tag").iter_rows(): + read_in.setdefault(tag, set()).add(sample) + + sets_of_sample: dict[str, set[str]] = {} + samples_of_set: dict[str, set[str]] = {} + for sample, set_id in linker.select("sampleId", "setId").iter_rows(): + sets_of_sample.setdefault(sample, set()).add(set_id) + samples_of_set.setdefault(set_id, set()).add(sample) + + # Declared by exactly one sample and read in none of that sample's cells: the only arrangement in + # which both directions of the panel-versus-reads check fire on the same barcode at once. + cross = [t for t, samples in declared_in.items() if len(samples) == 1 and not samples & read_in.get(t, set())] + shared = [(name, sorted(tags)) for name, tags in tags_of_name.items() if len(tags) > 1] + short_sample = min(offered, key=lambda s: (len(offered[s]), s)) + spanning = sorted(s for s, samples in samples_of_set.items() if len(samples) > 1) + + return { + "antigens": set(names), + "names": set(tags_of_name), + "renamed": {t for t, tag_names in names.items() if len(tag_names) > 1}, + "shared": shared, + "cross": cross, + "read_in": read_in, + "declared_in": declared_in, + "offered": offered, + "short_sample": short_sample, + "sets_of_sample": sets_of_sample, + "spanning": spanning, + } + + +def _states(bed): + v = pl.read_csv(bed / "result_verdicts.csv", infer_schema_length=0) + return {(r["setId"], r["identity"]): r["state"] for r in v.iter_rows(named=True)} + + +def _only_set(shape, sample): + sets = shape["sets_of_sample"][sample] + assert len(sets) == 1, f"the bed must draw one set from {sample} for this assertion to be about that set" + return next(iter(sets)) + + +def _samples_of(shape, set_id): + return {s for s, sets in shape["sets_of_sample"].items() if set_id in sets} + + +def test_the_bed_keys_identity_by_barcode_where_the_names_would_split(wide_bed): + shape = _bed_shape(wide_bed) + assert len(shape["renamed"]) >= 4, "the bed must carry the case that makes name keying wrong" + + r = _run(wide_bed, *_bed_args("panel_with_reference.csv")) + assert r.returncode == 0, r.stderr + identities = set(pl.read_csv(wide_bed / "result_verdicts.csv", infer_schema_length=0)["identity"].to_list()) + assert identities == shape["antigens"], "one identity per declared barcode, whatever it was named" + + # The two keyings do not even agree on how many questions the run asks: keying on the name would + # split each renamed barcode in two and fuse the antigen carried on two barcodes into one. + assert len(shape["names"]) > len(shape["antigens"]) + + # A label is not an identity, and two identities under one label are two rows a reader cannot + # tell apart -- so where two barcodes share a name the label has to carry the barcode as well. + labels = dict(pl.read_csv(wide_bed / "result_identity_labels.csv", infer_schema_length=0).iter_rows()) + assert set(labels) == shape["antigens"] + assert len(set(labels.values())) == len(labels) + + # And from the other side: asked to group by the name, the run cannot place exactly the renamed + # barcodes, because no one name holds for them. It says so rather than dropping them. + r = _run(wide_bed, *_bed_args("panel_with_reference.csv", *NAME_GROUPING)) + assert r.returncode == 0, r.stderr + meta = json.loads((wide_bed / "result_run_meta.json").read_text()) + assert set(meta["tagsWithoutGroupingValue"]) == shape["renamed"] + + +def test_the_bed_reaches_all_four_states_in_one_run(wide_bed): + # A bed that cannot reach a state tests nothing about it. All four come from one run here: bound + # from counts of 500 and 5000 against a comparator of 6, not bound from counts of 8, never asked + # from the three-tag panel, and unreliable from the one cell whose comparator reads 1 -- below + # the thin line of 2, so that cell cannot be compared at all. + r = _run(wide_bed, *_bed_args("panel_with_reference.csv")) + assert r.returncode == 0, r.stderr + v = pl.read_csv(wide_bed / "result_verdicts.csv", infer_schema_length=0) + assert set(v["state"].to_list()) == {"bound", "not bound", "never asked", "unreliable"} + + +def test_the_short_panel_is_where_never_asked_appears(wide_bed): + shape = _bed_shape(wide_bed) + short = shape["short_sample"] + assert len(shape["offered"][short]) < len(shape["antigens"]), "the bed needs panels of differing size" + + r = _run(wide_bed, *_bed_args("panel_with_reference.csv")) + assert r.returncode == 0, r.stderr + states = _states(wide_bed) + + unasked = {i for (s, i), state in states.items() if s == _only_set(shape, short) and state == "never asked"} + assert unasked == shape["antigens"] - shape["offered"][short] + assert unasked + + # A set spanning two samples was offered whatever either panel offered, so the gap closes where + # the two panels together cover the run. Nothing in it reads never asked. + spanning = shape["spanning"] + assert spanning, "the bed needs one set drawn from two samples" + covered = set().union(*(shape["offered"][s] for s in _samples_of(shape, spanning[0]))) + assert covered == shape["antigens"], "the spanning set's panels must together cover the universe" + assert not [i for (s, i), state in states.items() if s == spanning[0] and state == "never asked"] + + +def test_the_panel_mismatch_fires_per_sample_in_both_directions(wide_bed): + shape = _bed_shape(wide_bed) + assert len(shape["cross"]) == 1, "the bed carries exactly one barcode declared here and read there" + tag = shape["cross"][0] + declaring = next(iter(shape["declared_in"][tag])) + reading = sorted(shape["read_in"][tag]) + + # Read against the two-comparator panel, the only one here declaring every barcode the counts + # carry: on the others the undeclared comparator adds rows and the table is no longer a clean + # statement about this one barcode. + r = _run(wide_bed, *_bed_args("panel_multi_reference.csv")) + assert r.returncode == 0, r.stderr + + m = pl.read_csv(wide_bed / "result_panel_mismatch.csv", infer_schema_length=0) + rows = {(row["tag"], row["direction"]): row["samples"] for row in m.iter_rows(named=True)} + assert m.height == 2, f"only the cross declaration should mismatch; got {m.to_dicts()}" + assert rows[(tag, "declared-never-seen")] == declaring + assert rows[(tag, "undeclared-in-panel")] == ", ".join(reading) + + # A global check would have cancelled these two against each other. The verdicts show why that + # matters: the sample that read the barcode never declared it, so its set reads never asked + # while a real count of 500 sits in the counts file -- the verdict follows the panel. + states = _states(wide_bed) + assert states[(_only_set(shape, reading[0]), tag)] == "never asked" + + # And the sample that declared it read nothing: its cells were offered the identity and could be + # compared, so they answer not bound. A silent cell that can be compared is a negative answer, + # not an absent one, and reading it as never asked was the earlier revision's bug. + assert states[(_only_set(shape, declaring), tag)] == "not bound" + + +def test_one_antigen_on_two_barcodes_is_read_by_its_highest_member(wide_bed): + shape = _bed_shape(wide_bed) + assert len(shape["shared"]) == 1, "the bed carries exactly one antigen on two barcodes" + name, (first, second) = shape["shared"][0] + spanning = shape["spanning"][0] + + # Per barcode the two cells that carry them bind opposite ones, so each barcode splits its set + # one to one and reads unreliable on the tie. + r = _run(wide_bed, *_bed_args("panel_with_reference.csv")) + assert r.returncode == 0, r.stderr + per_tag = _states(wide_bed) + assert per_tag[(spanning, first)] == "unreliable" + assert per_tag[(spanning, second)] == "unreliable" + + # Read as one antigen the two barcodes combine by the highest member, never by the sum and never + # by an arbitrary one: each cell's reading becomes 500, both cells bind, and the set is bound. + # Summing would reach the same verdict here by accident; what the highest rule buys is that a + # cell's answer does not depend on how many barcodes happened to carry the antigen. + r = _run(wide_bed, *_bed_args("panel_with_reference.csv", *NAME_GROUPING)) + assert r.returncode == 0, r.stderr + assert _states(wide_bed)[(spanning, name)] == "bound" + + +def test_the_higher_of_two_declared_comparators_serves(wide_bed): + # Several comparator tags combine as any identity's tags do: by the highest. The bed's second + # comparator reads 60 against the first's 6, and specificity_score(500, 6) is 100 while + # specificity_score(500, 60) is 0.1 -- so a count of 500 binds against the lower comparator and + # fails against the higher. Taking the lower, or an arbitrary one, would make the two runs + # identical; taking the higher can only ever withdraw a binding. + assert _run(wide_bed, *_bed_args("panel_with_reference.csv")).returncode == 0 + one = _states(wide_bed) + assert _run(wide_bed, *_bed_args("panel_multi_reference.csv")).returncode == 0 + two = _states(wide_bed) + + assert set(one) == set(two), "the two panels declare the same identities" + bound_one = {key for key, state in one.items() if state == "bound"} + bound_two = {key for key, state in two.items() if state == "bound"} + assert bound_two < bound_one, "the higher comparator must withdraw at least one binding" + assert bound_two, "and must not withdraw them all, or the bed says nothing about which one served" + # Withdrawn, not made unanswerable: the comparison was made against a bigger number and failed. + assert {two[key] for key in bound_one - bound_two} == {"not bound"} + + +def test_the_bed_panel_without_a_declared_comparator_serves_as_its_own(wide_bed): + shape = _bed_shape(wide_bed) + assert len(shape["antigens"]) >= DEFAULT_PANEL_MIN_MEMBERS, "a panel this small cannot stand in" + + r = _run(wide_bed, *_bed_args("panel.csv")) + assert r.returncode == 0, r.stderr + meta = json.loads((wide_bed / "result_run_meta.json").read_text()) + assert meta["referenceChoice"] == ReferenceChoice.PANEL.value + states = set(_states(wide_bed).values()) + assert states != {"unreliable"}, "the panel could serve as its own comparator and was not asked to" + without = meta["readingsFloored"] + + # The floor spares a comparator's reading, and only a declared comparator has one to spare. With + # no declaration the thin comparator reading of 1 is floored like any other count, so this run + # floors strictly more than the same counts read against a declared comparator. + assert _run(wide_bed, *_bed_args("panel_with_reference.csv")).returncode == 0 + with_declared = json.loads((wide_bed / "result_run_meta.json").read_text())["readingsFloored"] + assert without > with_declared > 0 diff --git a/software/test-data/fixtures/verdicts/README.md b/software/test-data/fixtures/verdicts/README.md new file mode 100644 index 0000000..070aac7 --- /dev/null +++ b/software/test-data/fixtures/verdicts/README.md @@ -0,0 +1,77 @@ +# Verdict Fixture Bed + +The bed the binding-verdict tests run the whole CLI against. Committed rather than generated at test +time, so a test never has to run the generator, and excluded from ruff. Regenerate with +`python generate.py` — it is stdlib only and takes one fixed seed, so the files come back byte +identical. + +Entirely synthetic. This repository is public: barcodes are random ACGT strings, antigens are `AgNN`, +samples are `SNN`, and no real sequence, antigen or sample identifier appears anywhere. + +## Files + +| File | What it is | +|---|---| +| `panel.csv` | Four samples, panels of 3, 4, 4 and 5 tags. **No** comparator tag. | +| `panel_with_reference.csv` | The same panels plus **one** comparator tag (`Ctrl1`) on every sample. | +| `panel_multi_reference.csv` | The same panels plus **two** comparator tags (`Ctrl1`, `Ctrl2`) on every sample. | +| `counts.csv` | Sparse per-(sample, cell, barcode) UMI counts for all eleven cells. | +| `linker.csv` | Cell to clonotype set: `K01`, `K02`, `K03` (spanning two samples), `K04` (a singleton). | + +Columns are `Samples,Name,Sequence,Type` in the panels, so a run reads the bed with +`--barcode-col Sequence --feature-col Name --sample-col Samples --role-column Type +--reference-values Control`. + +One `counts.csv` serves all three panels, so the two comparator barcodes are read in every run +including those whose panel does not declare them. Those readings surface as `undeclared-in-panel` +rows and are expected: `panel_multi_reference.csv` is the only panel here that declares every barcode +the counts carry, so it is the bed to use when the mismatch table itself is under test. + +## Panel shapes covered + +| Shape | How the bed carries it | +|---|---| +| Per-sample panels of differing size | 3, 4, 4 and 5 tags. `K01` is drawn from the three-tag sample alone, so five of the eight identities read *never asked*. | +| Same barcode, different names across samples | Four barcodes carry two `AgNN` names each. A fifth recurs under one name, so a test can tell "recurs" from "recurs inconsistently". | +| One panel over every sample | Not a separate file: the comparator rows are declared on all four samples, which is the unkeyed case within a keyed panel. | +| A designated negative control | `panel_with_reference.csv`. | +| **No** negative control | `panel.csv`. Eight distinct barcodes, exactly the shipped minimum of eight, so the panel serves as its own comparator rather than falling silent. | +| **Several** designated controls | `panel_multi_reference.csv`. `Ctrl2` reads above `Ctrl1` in every cell, so the served comparator is 60 and not 6 — and bindings that hold at 6 fall away at 60. | +| One antigen on several barcodes | `Ag07` is carried on two barcodes, both on the fourth sample. | +| A barcode declared in one sample, read in another | `Ag06`'s barcode is declared by the third sample only and read in the second only, so both directions of the check fire on different samples at once. | +| Free-text properties, inconsistently spelled | Not carried here. The panel has no free-text property column beyond `Name`; `test_panel.py` covers the hygiene measurement. | + +## The counts, and which threshold each one is for + +Shipped defaults in `verdict.py`: floor **4**, comparator thin line **2**, bound cutoff **75** on +`specificity_score`, high-reference observation line **100**. The score is a beta function and not a +ratio, so the useful values are not where intuition puts them — against a comparator of 6 a count of +8 scores 0.0001, 50 scores 3.1, 60 scores 7.2 and 500 scores 100. + +| Count | Chosen against | +|---|---| +| `8` | The *not bound* reading. Above the floor of 4, so it survives to be compared, and 0.0001 against a comparator of 6, so it is compared and fails. A count of 2 would be zeroed by the floor and read *not bound* for a different reason. | +| `500` | The *bound* reading while the comparator is 6 (score 100) — and a *not bound* reading against 60 (score 0.1). That difference is what the two-comparator panel measures. | +| `5000` | Bound against either comparator, so one binding survives on the two-comparator panel and the bed does not degenerate into all *not bound*. | +| `2` (one reading only) | Below the floor of 4, so it is zeroed and counted in `readingsFloored`. | +| `6` (`Ctrl1`) | Above the thin line of 2 so cells can be compared, and far below 500 so a real binding clears the cutoff. | +| `60` (`Ctrl2`) | Above `Ctrl1` so the highest-member rule is observable, and below the high-reference line of 100 so that measurement stays quiet. | +| `1` (`Ctrl1` in one cell) | Below the thin line of 2. This is the bed's only source of *unreliable*: raise it and the fourth state disappears. | + +## What each set reads, on `panel_with_reference.csv` + +| Set | Cells | Reads | +|---|---|---| +| `K01` | three cells of the three-tag sample | *bound* twice, *not bound* once, *never asked* five times. One of its cells is silent on a bound identity and votes *not bound* against two that bind it. | +| `K02` | three cells of a four-tag sample | *bound* twice, *not bound* twice, *never asked* four times. One of its readings is floored. | +| `K03` | four cells across two samples | Offered the union of two panels, so nothing in it reads *never asked*. Two identities read *unreliable* on a tie. | +| `K04` | one cell whose comparator reads 1 | *unreliable* everywhere it was offered, *never asked* elsewhere. | + +Two readings are worth naming, because both are states an earlier revision got wrong: + +- `Ag06`'s barcode is declared by the third sample and read in none of its cells. `K03` draws from + that sample, so it was offered `Ag06`, its cells could be compared, and they read nothing — which + is *not bound*, not *never asked*. A silent cell that can be compared is a negative answer. +- The same barcode is read in the second sample, which never declared it. `K02` therefore reads + *never asked* at that identity while a real count of 500 sits in `counts.csv`. The verdict follows + the panel; the mismatch table is what makes the reading visible. diff --git a/software/test-data/fixtures/verdicts/counts.csv b/software/test-data/fixtures/verdicts/counts.csv new file mode 100644 index 0000000..9b28b23 --- /dev/null +++ b/software/test-data/fixtures/verdicts/counts.csv @@ -0,0 +1,64 @@ +sampleId,cellId,tag,umiCount +S01,c01,TGTAGACGCATA,6 +S01,c01,GGGGAATTCAAT,60 +S01,c01,AGAACCCCCCTT,5000 +S01,c01,AGTTAAGAACAA,8 +S01,c01,AAGCAACAATCT,5000 +S01,c02,TGTAGACGCATA,6 +S01,c02,GGGGAATTCAAT,60 +S01,c02,AGAACCCCCCTT,5000 +S01,c02,AGTTAAGAACAA,8 +S01,c02,AAGCAACAATCT,5000 +S01,c03,TGTAGACGCATA,6 +S01,c03,GGGGAATTCAAT,60 +S01,c03,AGAACCCCCCTT,5000 +S01,c03,AGTTAAGAACAA,8 +S02,c04,TGTAGACGCATA,6 +S02,c04,GGGGAATTCAAT,60 +S02,c04,AGAACCCCCCTT,500 +S02,c04,AGTTAAGAACAA,8 +S02,c04,TCGTGGTCCTGG,500 +S02,c04,TCCGTGACTTTG,8 +S02,c04,ACCTTACGGGCT,500 +S02,c05,TGTAGACGCATA,6 +S02,c05,GGGGAATTCAAT,60 +S02,c05,AGAACCCCCCTT,500 +S02,c05,AGTTAAGAACAA,8 +S02,c05,TCGTGGTCCTGG,500 +S02,c05,TCCGTGACTTTG,8 +S02,c06,TGTAGACGCATA,6 +S02,c06,GGGGAATTCAAT,60 +S02,c06,AGAACCCCCCTT,500 +S02,c06,AGTTAAGAACAA,8 +S02,c06,TCGTGGTCCTGG,2 +S02,c06,TCCGTGACTTTG,8 +S03,c07,TGTAGACGCATA,6 +S03,c07,GGGGAATTCAAT,60 +S03,c07,AGAACCCCCCTT,500 +S03,c07,AAGCAACAATCT,500 +S03,c07,TCGTGGTCCTGG,8 +S03,c08,TGTAGACGCATA,6 +S03,c08,GGGGAATTCAAT,60 +S03,c08,AGAACCCCCCTT,500 +S03,c08,AAGCAACAATCT,500 +S03,c08,TCGTGGTCCTGG,8 +S04,c09,TGTAGACGCATA,6 +S04,c09,GGGGAATTCAAT,60 +S04,c09,AGAACCCCCCTT,500 +S04,c09,AGTTAAGAACAA,8 +S04,c09,TCCGTGACTTTG,500 +S04,c09,CTTTTTGCCGTT,500 +S04,c09,CATCTCTAGTCT,8 +S04,c10,TGTAGACGCATA,6 +S04,c10,GGGGAATTCAAT,60 +S04,c10,AGAACCCCCCTT,500 +S04,c10,AGTTAAGAACAA,8 +S04,c10,TCCGTGACTTTG,500 +S04,c10,CTTTTTGCCGTT,8 +S04,c10,CATCTCTAGTCT,500 +S04,c11,TGTAGACGCATA,1 +S04,c11,AGAACCCCCCTT,500 +S04,c11,AGTTAAGAACAA,8 +S04,c11,TCCGTGACTTTG,500 +S04,c11,CTTTTTGCCGTT,8 +S04,c11,CATCTCTAGTCT,8 diff --git a/software/test-data/fixtures/verdicts/generate.py b/software/test-data/fixtures/verdicts/generate.py new file mode 100644 index 0000000..b4b7789 --- /dev/null +++ b/software/test-data/fixtures/verdicts/generate.py @@ -0,0 +1,232 @@ +"""Regenerate the synthetic verdict fixture bed. + +Run from this directory: python generate.py + +Stdlib only, and the only random thing is the barcode alphabet soup: every count, name, sample and +set membership below is written out by hand, because each one is load-bearing against a threshold and +a generated number would be load-bearing against nothing. The seed is passed to `random.Random` +rather than seeding the module, so the bed regenerates byte-identically and a second generator +running in the same process cannot disturb this one. + +Everything here is invented. The repository is public: no real barcode sequence, antigen name or +sample identifier may appear. Barcodes are drawn from ACGT, antigens are `AgNN`, samples are `SNN`. + +The thresholds the counts are chosen against, all shipped defaults in `verdict.py`: + + floor 4 a reading below this is zeroed before anything else runs + reference thin line 2 a comparator below this leaves the cell impossible to compare + bound cutoff 75 on `specificity_score`, which is a beta function and not a ratio + high-reference line 100 a comparator at or above this is flagged as an observation + +Against a comparator of 6 the score is 0.0001 at a count of 8, 3.1 at 50, 7.2 at 60 and 100 at 500. +Against a comparator of 60 it is 0.1 at 500 and 100 at 5000. That is why 8 means *not bound*, 500 +means *bound* only while the comparator stays at 6, and 5000 is the count that survives the higher +comparator of the two-control panel. +""" + +import random + +SEED = 20260817 +BARCODE_LENGTH = 12 + +# Slot names, not sequences. The tests never hard-code a sequence; they recover each barcode by the +# role it plays in the panel (two names, two barcodes under one name, declared here and read there), +# so a regenerated bed with different sequences still exercises the same shapes. +ANTIGEN_SLOTS = ["A0", "A1", "A2", "A3", "A4", "A5", "A6", "A7"] +CONTROL_SLOTS = ["R0", "R1"] + +# (sample, antigen name, barcode slot). Panels of 3, 4, 4 and 5 tags across four samples. +# +# A0, A1, A3, A4 each recur under two different names -- the case that makes name-keyed identity +# wrong, and the reason the pipeline keys on the barcode. +# A2 recurs under ONE name, so a test can tell "recurs" from "recurs inconsistently". +# A6 and A7 both carry Ag07: one antigen on two barcodes, read by the highest member. +# A5 is declared by S03 alone and (see COUNTS) read in S02 alone, which is the only way both +# directions of the panel-versus-reads check can be seen to run per sample rather than +# globally: a global check would let S03's declaration excuse the reading in S02. +PANEL = [ + ("S01", "Ag01", "A0"), + ("S01", "Ag02", "A1"), + ("S01", "Ag03", "A2"), + ("S02", "Ag11", "A0"), + ("S02", "Ag02", "A1"), + ("S02", "Ag04", "A3"), + ("S02", "Ag05", "A4"), + ("S03", "Ag01", "A0"), + ("S03", "Ag03", "A2"), + ("S03", "Ag14", "A3"), + ("S03", "Ag06", "A5"), + ("S04", "Ag11", "A0"), + ("S04", "Ag12", "A1"), + ("S04", "Ag15", "A4"), + ("S04", "Ag07", "A6"), + ("S04", "Ag07", "A7"), +] + +SAMPLES = ["S01", "S02", "S03", "S04"] + +# The comparator rows the two reference beds add, on every sample: a comparator declared in one +# sample and not another is discarded by `consistent_properties` rather than honoured, so a tag is a +# comparator everywhere or nowhere. +CONTROL_NAMES = {"R0": "Ctrl1", "R1": "Ctrl2"} + +# (sample, cell, barcode slot, umiCount). +# +# R0 reads 6 in every cell but c11, where it reads 1 -- below the thin line of 2, so c11 cannot be +# compared at all and every identity its set was offered reads *unreliable*. That is the only source +# of that state in the bed, so lifting c11's 1 costs the fourth state. +# +# R1 reads 60, above R0 everywhere it appears, so the two-control panel's comparator is 60 and not 6. +# 60 also sits below the high-reference observation line of 100, keeping that measurement quiet. +# c11 has no R1 row, so it stays impossible to compare on the two-control panel too. +# +# 8 is the *not bound* count: above the floor of 4, so the reading survives to be compared, and +# 0.0001 against a comparator of 6, so it is compared and fails. A count of 2 would be zeroed by the +# floor and would read *not bound* for the wrong reason. +# 500 is the *bound* count against a comparator of 6 (score 100) and a *not bound* count against 60 +# (score 0.1) -- that difference is what the two-control bed measures. +# 5000 stays bound against either comparator. +COUNTS = [ + # -- S01, set K01: the three-tag panel, so five of the eight identities were never asked. + ("S01", "c01", "R0", 6), + ("S01", "c01", "R1", 60), + ("S01", "c01", "A0", 5000), + ("S01", "c01", "A1", 8), + ("S01", "c01", "A2", 5000), + ("S01", "c02", "R0", 6), + ("S01", "c02", "R1", 60), + ("S01", "c02", "A0", 5000), + ("S01", "c02", "A1", 8), + ("S01", "c02", "A2", 5000), + ("S01", "c03", "R0", 6), + ("S01", "c03", "R1", 60), + ("S01", "c03", "A0", 5000), + ("S01", "c03", "A1", 8), + # c03 has no A2 row: a cell that was asked and read nothing. It is comparable, so it votes + # *not bound* against the two cells that bound A2, and the majority still says bound. + # -- S02, set K02. + ("S02", "c04", "R0", 6), + ("S02", "c04", "R1", 60), + ("S02", "c04", "A0", 500), + ("S02", "c04", "A1", 8), + ("S02", "c04", "A3", 500), + ("S02", "c04", "A4", 8), + # A5 is read here and declared only by S03. The reading is real and the panel still says S02 was + # never asked, which is the whole reason the mismatch table has to travel with the answer. + ("S02", "c04", "A5", 500), + ("S02", "c05", "R0", 6), + ("S02", "c05", "R1", 60), + ("S02", "c05", "A0", 500), + ("S02", "c05", "A1", 8), + ("S02", "c05", "A3", 500), + ("S02", "c05", "A4", 8), + ("S02", "c06", "R0", 6), + ("S02", "c06", "R1", 60), + ("S02", "c06", "A0", 500), + ("S02", "c06", "A1", 8), + # 2 is below the floor of 4 and is zeroed, so this is the bed's floored reading and c06 votes + # *not bound* on A3 while c04 and c05 bind it. + ("S02", "c06", "A3", 2), + ("S02", "c06", "A4", 8), + # -- S03 and S04 together form set K03, so its offered set is the union of two panels and + # nothing in it reads *never asked*. + ("S03", "c07", "R0", 6), + ("S03", "c07", "R1", 60), + ("S03", "c07", "A0", 500), + ("S03", "c07", "A2", 500), + ("S03", "c07", "A3", 8), + # No A5 row in S03 at all, though S03 is the only sample that declares it: the other direction + # of the same check. Both S03 cells were offered A5 and read nothing, so K03 reads *not bound* + # at A5 -- not *never asked*, which is the regression this shape exists to catch. + ("S03", "c08", "R0", 6), + ("S03", "c08", "R1", 60), + ("S03", "c08", "A0", 500), + ("S03", "c08", "A2", 500), + ("S03", "c08", "A3", 8), + ("S04", "c09", "R0", 6), + ("S04", "c09", "R1", 60), + ("S04", "c09", "A0", 500), + ("S04", "c09", "A1", 8), + ("S04", "c09", "A4", 500), + # A6 and A7 are the two barcodes of Ag07, and the two S04 cells of K03 bind opposite ones. Read + # per barcode each splits its set one to one and reads *unreliable* on the tie; read as one + # antigen by the highest member, both cells bind Ag07 and the set reads *bound*. + ("S04", "c09", "A6", 500), + ("S04", "c09", "A7", 8), + ("S04", "c10", "R0", 6), + ("S04", "c10", "R1", 60), + ("S04", "c10", "A0", 500), + ("S04", "c10", "A1", 8), + ("S04", "c10", "A4", 500), + ("S04", "c10", "A6", 8), + ("S04", "c10", "A7", 500), + # -- S04, set K04: one cell, comparator below the thin line. + ("S04", "c11", "R0", 1), + ("S04", "c11", "A0", 500), + ("S04", "c11", "A1", 8), + ("S04", "c11", "A4", 500), + ("S04", "c11", "A6", 8), + ("S04", "c11", "A7", 8), +] + +# (sample, cell, set). K03 spans two samples on purpose; K04 is a singleton, which many real +# clonotypes are. +LINKER = [ + ("S01", "c01", "K01"), + ("S01", "c02", "K01"), + ("S01", "c03", "K01"), + ("S02", "c04", "K02"), + ("S02", "c05", "K02"), + ("S02", "c06", "K02"), + ("S03", "c07", "K03"), + ("S03", "c08", "K03"), + ("S04", "c09", "K03"), + ("S04", "c10", "K03"), + ("S04", "c11", "K04"), +] + + +def barcodes() -> dict[str, str]: + """A distinct ACGT sequence per slot, in slot order, from the fixed seed.""" + rng = random.Random(SEED) + assigned: dict[str, str] = {} + used: set[str] = set() + for slot in ANTIGEN_SLOTS + CONTROL_SLOTS: + while True: + seq = "".join(rng.choice("ACGT") for _ in range(BARCODE_LENGTH)) + if seq not in used: + break + used.add(seq) + assigned[slot] = seq + return assigned + + +def write_panel(path: str, seq: dict[str, str], controls: list[str]) -> None: + with open(path, "w") as f: + f.write("Samples,Name,Sequence,Type\n") + for sample, name, slot in PANEL: + f.write(f"{sample},{name},{seq[slot]},Target\n") + for sample in SAMPLES: + for slot in controls: + f.write(f"{sample},{CONTROL_NAMES[slot]},{seq[slot]},Control\n") + + +def main() -> None: + seq = barcodes() + write_panel("panel.csv", seq, []) + write_panel("panel_with_reference.csv", seq, ["R0"]) + write_panel("panel_multi_reference.csv", seq, ["R0", "R1"]) + + with open("counts.csv", "w") as f: + f.write("sampleId,cellId,tag,umiCount\n") + for sample, cell, slot, umi in COUNTS: + f.write(f"{sample},{cell},{seq[slot]},{umi}\n") + + with open("linker.csv", "w") as f: + f.write("sampleId,cellId,setId\n") + for sample, cell, set_id in LINKER: + f.write(f"{sample},{cell},{set_id}\n") + + +if __name__ == "__main__": + main() diff --git a/software/test-data/fixtures/verdicts/linker.csv b/software/test-data/fixtures/verdicts/linker.csv new file mode 100644 index 0000000..5e7dafa --- /dev/null +++ b/software/test-data/fixtures/verdicts/linker.csv @@ -0,0 +1,12 @@ +sampleId,cellId,setId +S01,c01,K01 +S01,c02,K01 +S01,c03,K01 +S02,c04,K02 +S02,c05,K02 +S02,c06,K02 +S03,c07,K03 +S03,c08,K03 +S04,c09,K03 +S04,c10,K03 +S04,c11,K04 diff --git a/software/test-data/fixtures/verdicts/panel.csv b/software/test-data/fixtures/verdicts/panel.csv new file mode 100644 index 0000000..7df09f6 --- /dev/null +++ b/software/test-data/fixtures/verdicts/panel.csv @@ -0,0 +1,17 @@ +Samples,Name,Sequence,Type +S01,Ag01,AGAACCCCCCTT,Target +S01,Ag02,AGTTAAGAACAA,Target +S01,Ag03,AAGCAACAATCT,Target +S02,Ag11,AGAACCCCCCTT,Target +S02,Ag02,AGTTAAGAACAA,Target +S02,Ag04,TCGTGGTCCTGG,Target +S02,Ag05,TCCGTGACTTTG,Target +S03,Ag01,AGAACCCCCCTT,Target +S03,Ag03,AAGCAACAATCT,Target +S03,Ag14,TCGTGGTCCTGG,Target +S03,Ag06,ACCTTACGGGCT,Target +S04,Ag11,AGAACCCCCCTT,Target +S04,Ag12,AGTTAAGAACAA,Target +S04,Ag15,TCCGTGACTTTG,Target +S04,Ag07,CTTTTTGCCGTT,Target +S04,Ag07,CATCTCTAGTCT,Target diff --git a/software/test-data/fixtures/verdicts/panel_multi_reference.csv b/software/test-data/fixtures/verdicts/panel_multi_reference.csv new file mode 100644 index 0000000..f8a5dc4 --- /dev/null +++ b/software/test-data/fixtures/verdicts/panel_multi_reference.csv @@ -0,0 +1,25 @@ +Samples,Name,Sequence,Type +S01,Ag01,AGAACCCCCCTT,Target +S01,Ag02,AGTTAAGAACAA,Target +S01,Ag03,AAGCAACAATCT,Target +S02,Ag11,AGAACCCCCCTT,Target +S02,Ag02,AGTTAAGAACAA,Target +S02,Ag04,TCGTGGTCCTGG,Target +S02,Ag05,TCCGTGACTTTG,Target +S03,Ag01,AGAACCCCCCTT,Target +S03,Ag03,AAGCAACAATCT,Target +S03,Ag14,TCGTGGTCCTGG,Target +S03,Ag06,ACCTTACGGGCT,Target +S04,Ag11,AGAACCCCCCTT,Target +S04,Ag12,AGTTAAGAACAA,Target +S04,Ag15,TCCGTGACTTTG,Target +S04,Ag07,CTTTTTGCCGTT,Target +S04,Ag07,CATCTCTAGTCT,Target +S01,Ctrl1,TGTAGACGCATA,Control +S01,Ctrl2,GGGGAATTCAAT,Control +S02,Ctrl1,TGTAGACGCATA,Control +S02,Ctrl2,GGGGAATTCAAT,Control +S03,Ctrl1,TGTAGACGCATA,Control +S03,Ctrl2,GGGGAATTCAAT,Control +S04,Ctrl1,TGTAGACGCATA,Control +S04,Ctrl2,GGGGAATTCAAT,Control diff --git a/software/test-data/fixtures/verdicts/panel_with_reference.csv b/software/test-data/fixtures/verdicts/panel_with_reference.csv new file mode 100644 index 0000000..e77f2df --- /dev/null +++ b/software/test-data/fixtures/verdicts/panel_with_reference.csv @@ -0,0 +1,21 @@ +Samples,Name,Sequence,Type +S01,Ag01,AGAACCCCCCTT,Target +S01,Ag02,AGTTAAGAACAA,Target +S01,Ag03,AAGCAACAATCT,Target +S02,Ag11,AGAACCCCCCTT,Target +S02,Ag02,AGTTAAGAACAA,Target +S02,Ag04,TCGTGGTCCTGG,Target +S02,Ag05,TCCGTGACTTTG,Target +S03,Ag01,AGAACCCCCCTT,Target +S03,Ag03,AAGCAACAATCT,Target +S03,Ag14,TCGTGGTCCTGG,Target +S03,Ag06,ACCTTACGGGCT,Target +S04,Ag11,AGAACCCCCCTT,Target +S04,Ag12,AGTTAAGAACAA,Target +S04,Ag15,TCCGTGACTTTG,Target +S04,Ag07,CTTTTTGCCGTT,Target +S04,Ag07,CATCTCTAGTCT,Target +S01,Ctrl1,TGTAGACGCATA,Control +S02,Ctrl1,TGTAGACGCATA,Control +S03,Ctrl1,TGTAGACGCATA,Control +S04,Ctrl1,TGTAGACGCATA,Control From de4b5a46b5640bd000493b6fd58ffa54cfae7309 Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 20:17:44 +0200 Subject: [PATCH 054/282] MILAB-6496: the verdict stage, with every parameter threaded MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The post-fan-out stage that runs emit-verdicts and imports its thirteen outputs, plus the cleanup that made the block runnable again. The reading is answered across the whole run rather than per sample, so the per-sample tag-stat tables are gathered into one sparse counts CSV first (gather-counts.tpl.tengo), keyed with the real sampleId taken from the resource-map key. Keys come through maps.getKeys, which sorts them: a bare range over a map has no defined order, which would vary the concatenated file's handle run to run and cost every downstream node its deduplication with nothing reported. The stage is two templates. verdict-run resolves the bundle, exports the linker as flat rows and runs the exec; verdict-import takes the run record as an input and builds the specs. The comparator and the cell list that SERVED are resolved by the software, not requested — a declared comparator degrades to none where the panel carries no reference tag — and both sit in the emitted columns' domain, so specs built inside the exec's own template would have recorded the request instead. The single-cell V(D)J dataset is resolved in the body and deliberately not in prepare: prepare's bundle is awaited in full, spec and data, before the body runs, so a linker there would hold the whole mitool fan-out behind the clonotyping chain. A dataset may bring one linker per receptor; the one whose clonotype axis is the anchor's is the one the user chose, so a BCR + TCR run needs no panic. A sample-axis mismatch is named instead of joining to nothing. Without a dataset the block still runs and emits its per-cell contract columns, the per-sample QC and the per-feature properties. Threading is now assertable without a backend. verdict-args builds the whole command line, staged filenames included, and unit tests cover every flag — including --qc-summary, without which readsTotal, panelAssignedFraction and readsPerCell have no source at all and read "not evaluated" silently, taking the only sequencing-depth alert the block ships with them. fanout-inputs closes the per-sample body inputs and the resource meta to a checked list, so a verdict parameter cannot reach the fan-out and cost every user a full parse / refine-tags / tag-stat re-run. Cleanup: the consensus and specificity outputs, the four CLI flags that drove them, and the negative control that gated the specificity import are removed from main, fb-pipeline and fb-downstream. per_cell_metrics.py stopped writing either table and stopped accepting the flags, so the block could not complete a run until now. The run record gains the identity list. The pivoted per-identity summary builds one p-column per identity, and the identities are panel data unknown until the software runs; a count cannot name them, so without the list the only per-antigen state a clonotype-anchored reader can see imported as nothing. --- .../per-cell-metrics/src/emit_verdicts.py | 6 + workflow/package.json | 3 +- workflow/src/column-specs.lib.tengo | 25 +- workflow/src/fanout-inputs.lib.tengo | 68 ++++ workflow/src/fanout-inputs.test.tengo | 60 ++++ workflow/src/fb-downstream.tpl.tengo | 39 +-- workflow/src/fb-pipeline.tpl.tengo | 15 +- workflow/src/gather-counts.tpl.tengo | 60 ++++ workflow/src/main.tpl.tengo | 290 +++++++++++++----- workflow/src/qc-summary.tpl.tengo | 9 +- workflow/src/verdict-args.lib.tengo | 187 +++++++++++ workflow/src/verdict-args.test.tengo | 119 +++++++ workflow/src/verdict-import.tpl.tengo | 151 +++++++++ workflow/src/verdict-linker.lib.tengo | 91 ++++++ workflow/src/verdict-linker.test.tengo | 76 +++++ workflow/src/verdict-run.tpl.tengo | 123 ++++++++ 16 files changed, 1187 insertions(+), 135 deletions(-) create mode 100644 workflow/src/fanout-inputs.lib.tengo create mode 100644 workflow/src/fanout-inputs.test.tengo create mode 100644 workflow/src/gather-counts.tpl.tengo create mode 100644 workflow/src/verdict-args.lib.tengo create mode 100644 workflow/src/verdict-args.test.tengo create mode 100644 workflow/src/verdict-import.tpl.tengo create mode 100644 workflow/src/verdict-linker.lib.tengo create mode 100644 workflow/src/verdict-linker.test.tengo create mode 100644 workflow/src/verdict-run.tpl.tengo diff --git a/software/per-cell-metrics/src/emit_verdicts.py b/software/per-cell-metrics/src/emit_verdicts.py index ee63be8..ced20bd 100644 --- a/software/per-cell-metrics/src/emit_verdicts.py +++ b/software/per-cell-metrics/src/emit_verdicts.py @@ -998,6 +998,12 @@ def _number(row: dict, column: str) -> float | None: "tagsWithoutGroupingValue": sorted(ungrouped_tags), "contending": [sorted(group) for group in contending], "identityCount": len(universe), + # The identities themselves, in the order the pivot lays them out. The workflow builds one + # p-column per column of result_identity_summary.csv, and the column names are the identities -- + # which are panel data, unknown until this runs. A count cannot name them, so without this the + # pivoted summary imports as nothing and the only per-antigen state a clonotype-anchored reader + # can see disappears with no error. + "identities": sorted(universe), "identitySummaryEmitted": summary_emitted, "identitySummaryLimit": IDENTITY_SUMMARY_MAX_IDENTITIES, "readingsFloored": readings_floored, diff --git a/workflow/package.json b/workflow/package.json index fcfffcb..4b0a1da 100644 --- a/workflow/package.json +++ b/workflow/package.json @@ -7,7 +7,8 @@ "scripts": { "build": "shx rm -rf dist && pl-tengo build", "format": "/usr/bin/env emacs --script ./format.el || echo 'No emacs.'", - "check": "pl-tengo check" + "check": "pl-tengo check", + "test": "pl-tengo test" }, "dependencies": { "@platforma-open/milaboratories.feature-integration.per-cell-metrics": "workspace:*", diff --git a/workflow/src/column-specs.lib.tengo b/workflow/src/column-specs.lib.tengo index 3a8d5b1..e24fba3 100644 --- a/workflow/src/column-specs.lib.tengo +++ b/workflow/src/column-specs.lib.tengo @@ -79,16 +79,16 @@ featureAxis := func(blockId) { // valueOutputs: the per-cell contract value columns, as processColumn `Xsv` output declarations. // -// `hasControl` no longer gates anything here and is kept only so main.tpl.tengo's call site is -// unchanged. The consensus/dominant-feature column and the Cell Ranger specificity score are both -// gone: the dominance rule they reported was removed from per_cell_metrics.py, which no longer writes -// either CSV, and a specificity score is a binding magnitude — the thing a four-state verdict -// replaces. Declaring an output whose CSV no longer exists costs a failed import, not a missing column. +// The consensus/dominant-feature column and the Cell Ranger specificity score are both gone: the +// dominance rule they reported was removed from per_cell_metrics.py, which no longer writes either CSV, +// and a specificity score is a binding magnitude — the thing a four-state verdict replaces. The negative +// control that once gated the specificity output therefore gates nothing here, and this function no +// longer takes it. // // umiCount is the primary abundance (abundance/isPrimary). It is deliberately NOT isAnchor: this // column is meant to be DISCOVERED under the downstream VDJ single-cell anchor (via the cellLinker), // not to be an anchor itself, so marking it isAnchor would be semantically wrong here. -valueOutputs := func(blockId, sampleAxisName, hasControl) { +valueOutputs := func(blockId, sampleAxisName) { cell := cellAxis(sampleAxisName) feat := featureAxis(blockId) @@ -158,9 +158,9 @@ valueOutputs := func(blockId, sampleAxisName, hasControl) { // export contract (abundance/fractions) is unchanged. The "Max ..." labels distinguish these aggregates // from the exported per-feature columns. The max-specificity aggregate went with the per-feature // specificity column: per_cell_metrics.py stopped writing it, and it read a binding magnitude. -// `hasControl` is kept only so main.tpl.tengo's call site is unchanged. +// With it went the negative control this function used to take, which gated nothing else. // Keyed [cellId]; the sample axis is prepended by processColumn. -perCellSummaryOutput := func(blockId, sampleAxisName, hasControl) { +perCellSummaryOutput := func(blockId, sampleAxisName) { cell := cellAxis(sampleAxisName) cols := [ @@ -760,8 +760,8 @@ cellScalarsImportSpec := func(sampleAxisSpec, cellAxisSpec, served) { // computes and would otherwise throw away: without it "never asked" is a claim a reader cannot check. // Sparse — a row exists only where the sample's panel offered the identity. // -// NOTE: the CSV as emit_verdicts.py writes it carries only the two key columns and no value column, -// so the constant "offered" column named here does not exist yet. See the task report. +// The value is a constant "true": a file of key columns alone imports as nothing, because columns are +// built from value columns, so the fact the frame records would never leave the block. offeredImportSpec := func(sampleAxisSpec, identityAxisSpec) { return { axes: [ @@ -794,8 +794,9 @@ offeredImportSpec := func(sampleAxisSpec, identityAxisSpec) { // a constant column, matching pl7.app/sc/cellLinker's shape. Order priority 0 and no default // visibility: it is infrastructure and is hidden in tables. // -// NOTE: as with the offered frame, emit_verdicts.py writes only the two key columns, so the constant -// column named here does not exist yet. See the task report. +// The value column is named "1" and holds 1, matching the cell-linker convention used elsewhere in the +// platform, and it exists for the same reason the offered frame's does: a key-only file imports as no +// column at all. tagIdentityLinkerImportSpec := func(tagAxisSpec, identityAxisSpec) { return { axes: [ diff --git a/workflow/src/fanout-inputs.lib.tengo b/workflow/src/fanout-inputs.lib.tengo new file mode 100644 index 0000000..fb711b3 --- /dev/null +++ b/workflow/src/fanout-inputs.lib.tengo @@ -0,0 +1,68 @@ +// What may reach the per-sample mitool fan-out, as a closed list. +// +// The block runs in two stages, and the whole value of the split is that the expensive stage does not +// notice the cheap one. When only the reading's parameters change — a different grouping, a different +// floor — the body re-executes and processColumn is re-declared with identical inputs, so every +// per-sample mitool body is recovered from cache and only the verdict stage re-runs. That holds exactly +// as long as no verdict-stage parameter and no resolved clonotype linker enters `extra` or `metaExtra`, +// because both are part of each per-sample body's identity: one added key costs every user a full re-run +// of parse, refine-tags and tag-stat, and nothing in the render or the logs says why. +// +// The eight-field extra-input block in main.tpl.tengo is precisely where a convenience addition lands, so +// the list is closed here and checked at render time rather than reviewed later. + +maps := import("@platforma-sdk/workflow-tengo:maps") + +// Every key main.tpl.tengo may pass as a per-sample body input. All of these describe how one sample's +// reads are parsed and counted; none of them describes how the counts are read. +EXTRA_INPUT_KEYS := [ + "barcodeSeqColumn", + "cellWhitelist", + "combineColumn", + "featureNameColumn", + "fileExtension", + "limitInput", + "minUmi", + "pattern", + "sampleColumn", + "sampleLabels", + "tags", + "tagsCsv" +] + +// The per-sample resource meta. A distinct meta when the user overrides RAM is correct — an overridden +// run must not dedup with the formula-sized one — but nothing else belongs here. +META_EXTRA_KEYS := [ + "mitoolBaseMemGB", + "mitoolCPUs", + "mitoolMemOverrideGB" +] + +_contains := func(list, value) { + for _, item in list { + if item == value { + return true + } + } + return false +} + +// unlisted: the keys of `m` that the allowlist does not cover, sorted. +// +// Iterated with maps.forEach, whose key order is sorted, so the message a render fails with is the same +// on every run rather than depending on map layout. +unlisted := func(m, allowed) { + offenders := [] + maps.forEach(m, func(key, _) { + if !_contains(allowed, key) { + offenders = append(offenders, key) + } + }) + return offenders +} + +export { + EXTRA_INPUT_KEYS: EXTRA_INPUT_KEYS, + META_EXTRA_KEYS: META_EXTRA_KEYS, + unlisted: unlisted +} diff --git a/workflow/src/fanout-inputs.test.tengo b/workflow/src/fanout-inputs.test.tengo new file mode 100644 index 0000000..65bf9ee --- /dev/null +++ b/workflow/src/fanout-inputs.test.tengo @@ -0,0 +1,60 @@ +test := import("@platforma-sdk/workflow-tengo:test") +fanout := import(":fanout-inputs") +va := import(":verdict-args") + +_one := func(key) { + m := {} + m[key] = "x" + return m +} + +// The whole point of the two-stage split: changing the reading must not re-run the per-sample mitool +// chain. Every verdict-stage parameter is checked against both allowlists, so adding one to +// verdict-args.PARAMETER_NAMES and to the fan-out at the same time fails here rather than costing every +// user a full parse / refine-tags / tag-stat re-run with nothing in the logs to explain it. +Test_no_verdict_parameter_may_reach_the_fanout := func() { + for _, name in va.PARAMETER_NAMES { + test.isEqual([name], fanout.unlisted(_one(name), fanout.EXTRA_INPUT_KEYS), + "verdict-stage parameter " + name + " must not be a per-sample body input") + test.isEqual([name], fanout.unlisted(_one(name), fanout.META_EXTRA_KEYS), + "verdict-stage parameter " + name + " must not be part of the per-sample resource meta") + } +} + +// The resolved clonotype linker is a resource, not a parameter, and it is the other thing that must never +// enter: it would put every per-sample body behind the clonotyping chain as well as re-keying it. +Test_no_linker_resource_may_reach_the_fanout := func() { + for _, name in ["linker", "linkerColumn", "cellLinker", "columns", "setAxisSpec"] { + test.isEqual([name], fanout.unlisted(_one(name), fanout.EXTRA_INPUT_KEYS), + name + " must not be a per-sample body input") + test.isEqual([name], fanout.unlisted(_one(name), fanout.META_EXTRA_KEYS), + name + " must not be part of the per-sample resource meta") + } +} + +// The allowlist accepts exactly what main.tpl.tengo passes today, so the render-time check is not +// vacuously satisfied by an over-broad list. +Test_the_declared_fanout_inputs_are_accepted := func() { + extra := {} + for _, key in fanout.EXTRA_INPUT_KEYS { + extra[key] = "x" + } + test.isEqual([], fanout.unlisted(extra, fanout.EXTRA_INPUT_KEYS), + "every declared per-sample body input is accepted") + + meta := {} + for _, key in fanout.META_EXTRA_KEYS { + meta[key] = 1 + } + test.isEqual([], fanout.unlisted(meta, fanout.META_EXTRA_KEYS), + "every declared resource-meta key is accepted") +} + +// Offenders come back sorted, so a failing render reports the same message every time. +Test_offenders_are_reported_sorted := func() { + m := {} + m.zzz = 1 + m.aaa = 2 + test.isEqual(["aaa", "zzz"], fanout.unlisted(m, fanout.EXTRA_INPUT_KEYS), + "unlisted keys are reported in sorted order") +} diff --git a/workflow/src/fb-downstream.tpl.tengo b/workflow/src/fb-downstream.tpl.tengo index 42d7e69..cd09f5d 100644 --- a/workflow/src/fb-downstream.tpl.tengo +++ b/workflow/src/fb-downstream.tpl.tengo @@ -5,12 +5,18 @@ // outputs). They consume tagstat.tsv (fb-tagstat) plus the parse/refine JSON reports (fb-parse / // fb-refine). Running them inside this render.create boundary returns their per-sample outputs as render // outputs that flatten cleanly. Mirrors blocks/peptide-extraction downstream-pipeline.tpl.tengo. +// +// Nothing here knows about the binding verdicts. The verdict stage reads the run as a whole — a panel is a +// property of a declared tag set rather than of one sample, and the quality rollup spans samples — so it +// runs once, after this fan-out, on the gathered counts. Keeping every one of its parameters out of this +// template is what lets a change to the reading recover each per-sample body from cache instead of +// re-running parse, refine-tags and tag-stat. self := import("@platforma-sdk/workflow-tengo:tpl") exec := import("@platforma-sdk/workflow-tengo:exec") assets := import("@platforma-sdk/workflow-tengo:assets") -self.defineOutputs("abundance", "fractions", "consensus", "specificity", "perCellSummary", "qc", "qcJson", "metricsLogStream") +self.defineOutputs("abundance", "fractions", "perCellSummary", "qc", "qcJson", "metricsLogStream") metricsSw := assets.importSoftware("@platforma-open/milaboratories.feature-integration.per-cell-metrics:main") qcReportSw := assets.importSoftware("@platforma-open/milaboratories.feature-integration.per-cell-metrics:qc-report") @@ -21,8 +27,6 @@ self.body(func(inputs) { refineReport := inputs.refineReport tags := inputs.tags // { cell, umi, feature } tag names (from the model; see pattern.ts) tagsCsv := inputs.tagsCsv - control := inputs.control // undefined / "" -> no specificity score (render.create skips undefined) - dominanceThreshold := inputs.dominanceThreshold barcodeSeqColumn := inputs.barcodeSeqColumn featureNameColumn := inputs.featureNameColumn // Optional multi-barcode antigen combine mode. "" = off (per_cell_metrics defaults every feature to @@ -30,12 +34,6 @@ self.body(func(inputs) { // AND per-barcode "fired" floor. combineColumn := is_undefined(inputs.combineColumn) ? "" : inputs.combineColumn minUmi := is_undefined(inputs.minUmi) ? 1 : inputs.minUmi - // Optional off-target designation (F2). offtargetColumn names an imported per-feature property column - // (e.g. antigen_class); offtargetValues is the comma-separated set of its values marking a feature as - // off-target. Both empty = off (unchanged dominant call). Features so marked are excluded from the - // dominant call (like the control) and turn on the cross-reactive label. - offtargetColumn := is_undefined(inputs.offtargetColumn) ? "" : inputs.offtargetColumn - offtargetValues := is_undefined(inputs.offtargetValues) ? "" : inputs.offtargetValues // Per-sample QC summary: reads parsed/matched (parse JSON report) + cell/feature/UMI metrics // (tag-stat) + best-effort panel-assigned fraction (refine JSON report). @@ -70,7 +68,6 @@ self.body(func(inputs) { between(formula.gib(metricsBaseGB), formula.gib(256)). staticFallback(formula.gib(metricsBaseGB)) - hasControl := !is_undefined(control) && control != "" metrics := exec.builder(). printErrStreamToStdout(). saveStdoutStream(). @@ -87,37 +84,27 @@ self.body(func(inputs) { arg("--umi-count-col").arg("unique_" + tags.umi). arg("--csv-barcode-col").arg(barcodeSeqColumn). arg("--csv-feature-col").arg(featureNameColumn). - arg("--dominance-threshold").arg(string(dominanceThreshold)). arg("--output-prefix").arg("result"). saveFile("result_abundance.csv"). saveFile("result_fractions.csv"). - saveFile("result_consensus.csv"). - saveFile("result_specificity.csv"). saveFile("result_per_cell_summary.csv") - if hasControl { - metrics = metrics.arg("--control").arg(control) - } // Multi-barcode antigen combine mode (optional). Only pass --combine-col when a column is configured; // otherwise per_cell_metrics defaults every feature to OR/sum (unchanged behaviour). --min-umi is the // AND per-barcode "fired" floor and is harmless when no combine column is set. if combineColumn != "" { metrics = metrics.arg("--combine-col").arg(combineColumn).arg("--min-umi").arg(string(minUmi)) } - // Off-target designation (optional, F2). Only pass both flags when a column AND values are configured; - // per_cell_metrics requires them together and leaves the dominant call unchanged when absent. - if offtargetColumn != "" && offtargetValues != "" { - metrics = metrics.arg("--offtarget-col").arg(offtargetColumn).arg("--offtarget-values").arg(offtargetValues) - } metrics = metrics.run() - // specificity is meaningful only with a control feature, so the Python always writes - // result_specificity.csv (empty header-only when no --control) and we always return it. main.tpl and - // the model gate the specificity *import* on hasControl, so the empty file is simply never read. + // The consensus (dominant-feature) and Cell Ranger specificity tables are gone with the dominance rule + // they reported: per_cell_metrics.py no longer writes either, and no longer takes the four flags that + // drove them (--control, --dominance-threshold, --offtarget-col, --offtarget-values). The binding + // question is now asked of every antigen independently in the verdict stage, so a cell that bound three + // antigens carries three verdicts rather than one winner and two absences. The negative-control + // designation survives as a per-feature marker column, emitted outside this fan-out. return { abundance: metrics.getFile("result_abundance.csv"), fractions: metrics.getFile("result_fractions.csv"), - consensus: metrics.getFile("result_consensus.csv"), - specificity: metrics.getFile("result_specificity.csv"), perCellSummary: metrics.getFile("result_per_cell_summary.csv"), qc: qc.getFile("result_qc.csv"), qcJson: qc.getFileContent("result_qc.json"), // inline content (saveFileContent) so the model can read it diff --git a/workflow/src/fb-pipeline.tpl.tengo b/workflow/src/fb-pipeline.tpl.tengo index d323223..841fd20 100644 --- a/workflow/src/fb-pipeline.tpl.tengo +++ b/workflow/src/fb-pipeline.tpl.tengo @@ -31,7 +31,7 @@ fbParseTpl := assets.importTemplate(":fb-parse") fbRefineTagstatTpl := assets.importTemplate(":fb-refine-tagstat") fbDownstreamTpl := assets.importTemplate(":fb-downstream") -self.defineOutputs("abundance", "fractions", "consensus", "specificity", "perCellSummary", "qc", "qcJson", "stepReports", "stepLogs", "parseLogStream", "metricsLogStream", "parsedMic", "parseReport", "refineReport", "tagstatTsv") +self.defineOutputs("abundance", "fractions", "perCellSummary", "qc", "qcJson", "stepReports", "stepLogs", "parseLogStream", "metricsLogStream", "parsedMic", "parseReport", "refineReport", "tagstatTsv") self.body(func(inputs) { inputData := inputs[pConstants.VALUE_FIELD_NAME] @@ -40,17 +40,12 @@ self.body(func(inputs) { pattern := inputs.pattern tags := inputs.tags // { cell, umi, feature } tag names (from the model; see pattern.ts) tagsCsv := inputs.tagsCsv - dominanceThreshold := inputs.dominanceThreshold - control := inputs.control // may be undefined / "" -> no specificity score fileExtension := inputs.fileExtension mitoolCPUs := inputs.mitoolCPUs barcodeSeqColumn := inputs.barcodeSeqColumn featureNameColumn := inputs.featureNameColumn combineColumn := is_undefined(inputs.combineColumn) ? "" : inputs.combineColumn minUmi := is_undefined(inputs.minUmi) ? 1 : inputs.minUmi - // Off-target designation (optional, F2) — property column + comma-separated off-target values. - offtargetColumn := is_undefined(inputs.offtargetColumn) ? "" : inputs.offtargetColumn - offtargetValues := is_undefined(inputs.offtargetValues) ? "" : inputs.offtargetValues cellWhitelist := inputs.cellWhitelist // "" = de-novo CELL correction (default); else a 10x built-in memBaseGB := inputs.mitoolBaseMemGB @@ -199,22 +194,16 @@ self.body(func(inputs) { refineReport: refineReport, tags: tags, tagsCsv: tagsCsvForSample, - control: control, // undefined -> render.create skips it -> no specificity - dominanceThreshold: dominanceThreshold, barcodeSeqColumn: barcodeSeqColumn, featureNameColumn: featureNameColumn, combineColumn: combineColumn, - minUmi: minUmi, - offtargetColumn: offtargetColumn, - offtargetValues: offtargetValues + minUmi: minUmi }) // The body must return exactly the defineOutputs set and no output may be undefined. return { abundance: downstreamRun.output("abundance"), fractions: downstreamRun.output("fractions"), - consensus: downstreamRun.output("consensus"), - specificity: downstreamRun.output("specificity"), perCellSummary: downstreamRun.output("perCellSummary"), qc: downstreamRun.output("qc"), qcJson: downstreamRun.output("qcJson"), diff --git a/workflow/src/gather-counts.tpl.tengo b/workflow/src/gather-counts.tpl.tengo new file mode 100644 index 0000000..b223b25 --- /dev/null +++ b/workflow/src/gather-counts.tpl.tengo @@ -0,0 +1,60 @@ +// One sparse count table for the whole run, gathered from the per-sample tag-stat tables. +// +// emit_verdicts.py reads every sample at once: a panel is a property of the declared tag set rather than +// of one sample, the panel-versus-reads check runs in both directions across the run, and the QC rollup +// spans samples. So the per-sample [sampleId] -> tag-stat file map produced by the fan-out is concatenated +// here into a single (sampleId, cellId, tag, umiCount) CSV. +// +// The real sampleId is injected from the resource-map key, exactly as qc-summary.tpl.tengo does: the +// per-sample execs are handed a constant sample id, so the tables themselves cannot say which sample they +// came from. +// +// Rendered as a separate template because the resource map must be awaited before its per-sample keys can +// be iterated, which cannot be done in main's body. +// +// The keys are taken through maps.getKeys, which returns them sorted. A bare `for k, v in` over a map has +// no defined order in Tengo, so the concatenated file's resource handle would vary from run to run — and +// every node downstream of it silently loses deduplication, with nothing reported. + +self := import("@platforma-sdk/workflow-tengo:tpl") +ll := import("@platforma-sdk/workflow-tengo:ll") +maps := import("@platforma-sdk/workflow-tengo:maps") +pt := import("@platforma-sdk/workflow-tengo:pt") + +json := import("json") + +self.defineOutputs("countsFile") + +self.body(func(inputs) { + tagstatData := inputs.tagstatData // ResourceMap: [sampleId] -> per-sample tag-stat TSV + tags := inputs.tags // { cell, umi, feature } — the mitool tag names, so the headers are known + + inputsMap := tagstatData.inputs() + ll.assert(len(inputsMap) > 0, "no per-sample tag-stat tables to gather") + + // The gather reads every sample's distinct-UMI table, which scales with the run rather than with one + // sample, so it is not a light-queue job. + wf := pt.workflow().cpu(2).mem("8GiB") + + dfs := [] + for key in maps.getKeys(inputsMap) { + sampleId := json.decode(key)[0] + df := wf.frame(inputsMap[key], { xsvType: "tsv", inferSchema: false }) + // tag-stat -u writes one row per (cell, feature barcode) with the distinct-UMI count under + // "unique_". The FEATURE tag holds the barcode SEQUENCE, corrected against the panel by + // refine-tags, which is the key emit_verdicts.py joins the panel on. + dfs = append(dfs, df.select([ + pt.lit(sampleId).alias("sampleId"), + pt.col(tags.cell).alias("cellId"), + pt.col(tags.feature).alias("tag"), + pt.col("unique_" + tags.umi).alias("umiCount") + ]...)) + } + + combined := len(dfs) > 1 ? pt.concat(dfs) : dfs[0] + combined.save("counts.csv", { xsvType: "csv" }) + + return { + countsFile: wf.run().getFile("counts.csv") + } +}) diff --git a/workflow/src/main.tpl.tengo b/workflow/src/main.tpl.tengo index 710258a..5a59ccd 100644 --- a/workflow/src/main.tpl.tengo +++ b/workflow/src/main.tpl.tengo @@ -19,11 +19,15 @@ exec := import("@platforma-sdk/workflow-tengo:exec") ll := import("@platforma-sdk/workflow-tengo:ll") file := import("@platforma-sdk/workflow-tengo:file") columnSpecs := import(":column-specs") +fanout := import(":fanout-inputs") fbPipelineTpl := assets.importTemplate(":fb-pipeline") prerunTpl := assets.importTemplate(":prerun") qcSummaryTpl := assets.importTemplate(":qc-summary") featurePropsTpl := assets.importTemplate(":fb-feature-properties") +gatherCountsTpl := assets.importTemplate(":gather-counts") +verdictRunTpl := assets.importTemplate(":verdict-run") +verdictImportTpl := assets.importTemplate(":verdict-import") featurePropsSw := assets.importSoftware("@platforma-open/milaboratories.feature-integration.per-cell-metrics:feature-properties") @@ -43,6 +47,12 @@ defaultMitoolCPUs := 8 // feature-name list for the control dropdown (see prerun.tpl). wf.setPreRun(prerunTpl) +// Only the FASTQ dataset is resolved here, and the single-cell V(D)J dataset deliberately is NOT. +// Everything in prepare's bundle is awaited in full — spec AND data — before the body runs, so resolving +// the clonotype dataset here would mean the per-sample mitool fan-out is not even declared until the whole +// clonotyping chain has finished. Two chains that run concurrently today would run one after the other, +// roughly doubling wall clock, and nothing would surface it until a real dataset. The body needs only the +// FASTQ column; the clonotype linker is resolved inside the verdict stage's own child templates. wf.prepare(func(args) { return { resolvedFastq: wf.resolve(args.fbFastqRef) @@ -67,28 +77,24 @@ wf.body(func(args) { // refine-tags/tag-stat steps reference by name; any extra flanks/spacers/anchors reach mitool as-is. pattern := args.pattern + // The negative-control feature. It designates a per-feature marker column for downstream consumers and + // no longer reaches the per-sample metrics step: the dominance rule it once gated, and the specificity + // score beside it, are gone from per_cell_metrics.py. A four-state verdict asks the binding question of + // every antigen independently, so there is no winner for a control to be excluded from. control := args.controlFeature hasControl := !is_undefined(control) && control != "" - dominanceThreshold := is_undefined(args.dominanceThreshold) ? 0.6 : args.dominanceThreshold // Cell-barcode whitelist for refine-tags CELL correction. "" = de-novo (default). Always a defined // string so the extra-input field below resolves and never stalls the body. cellWhitelist := is_undefined(args.cellWhitelist) ? "" : args.cellWhitelist - // Off-target designation (optional, F2). The model gives offtargetProperty (an imported per-feature - // property column, e.g. antigen_class) and offtargetValues (the multi-selected values of that column - // marking a feature as off-target). Join the value list into the comma-separated string the Python - // --offtarget-values expects; "" for either = feature off (unchanged dominant call). - offtargetProperty := is_undefined(args.offtargetProperty) ? "" : args.offtargetProperty - offtargetValueList := is_undefined(args.offtargetValues) ? [] : args.offtargetValues - offtargetValues := "" - for i, v in offtargetValueList { - offtargetValues = i == 0 ? v : offtargetValues + "," + v - } - // Per-sample body inputs. processColumn binds EVERY extra key as a body input field and waits for // each to be set before firing the body (pframes/index.lib.tengo: `renderInputs["__extra_"+k]=v`), // so a key whose value is `undefined` creates a field that never resolves and stalls the body. - // Therefore `control` is added only when a control feature is configured. + // + // This map is a closed list, enforced below. Everything in it describes how one sample's reads are + // parsed and counted; nothing in it describes how the counts are read. That is what lets a change to + // the reading recover every per-sample mitool body from cache instead of re-running parse, refine-tags + // and tag-stat for every sample. extraInputs := { pattern: pattern, tags: args.tags, // mitool tag names (CELL/UMI/FEATURE) — single source is the model (pattern.ts) @@ -97,7 +103,6 @@ wf.body(func(args) { sampleColumn: is_undefined(args.sampleColumn) ? "" : args.sampleColumn, sampleLabels: is_undefined(args.sampleLabels) ? {} : args.sampleLabels, tagsCsv: csvFile, - dominanceThreshold: dominanceThreshold, fileExtension: fileExtension, barcodeSeqColumn: args.barcodeSeqColumn, featureNameColumn: args.featureNameColumn, @@ -105,16 +110,11 @@ wf.body(func(args) { // its barcodes = OR). minUmi is the AND-mode per-barcode "fired" floor; always defined (default 1). combineColumn: is_undefined(args.combineColumn) ? "" : args.combineColumn, minUmi: is_undefined(args.minUmi) ? 1 : args.minUmi, - // Off-target designation (F2) — property column + comma-joined off-target values ("" = off). - offtargetColumn: offtargetProperty, - offtargetValues: offtargetValues, cellWhitelist: cellWhitelist } - if hasControl { - extraInputs.control = control - } - // Preview (dry-run) read cap. Only add the field when set (an undefined extra-input field would stall - // the per-sample body, like `control`); absent -> full run (fb-parse applies no --limit). + // Preview (dry-run) read cap. Only add the field when set (an undefined extra-input field creates a + // body field that never resolves and stalls the sample); absent -> full run (fb-parse applies no + // --limit). if !is_undefined(args.limitInput) { extraInputs.limitInput = args.limitInput } @@ -123,8 +123,8 @@ wf.body(func(args) { // [sampleId] file map (assembled into a table by qc-summary.tpl, NOT imported here — a per-sample // scalar cannot be xsv-imported inside the shared processColumn) + the per-sample QC-JSON map (read // by the model to build the Analysis logs). Column/axis specs come from column-specs.lib.tengo. - outputs := columnSpecs.valueOutputs(blockId, sampleAxis.name, hasControl) - outputs = append(outputs, columnSpecs.perCellSummaryOutput(blockId, sampleAxis.name, hasControl)) + outputs := columnSpecs.valueOutputs(blockId, sampleAxis.name) + outputs = append(outputs, columnSpecs.perCellSummaryOutput(blockId, sampleAxis.name)) outputs = append(outputs, columnSpecs.qcFileMapOutput(blockId)) outputs = append(outputs, { type: "Resource", @@ -217,7 +217,7 @@ wf.body(func(args) { // CIDConflictError (poisoning the whole per-sample body). Pinning these files by content hash keeps // those resolutions stable — the metrics analogue of the mitool loop above. A `path` maps each pin to // the existing body output without colliding with the Xsv/File output of the same name. - for pn in ["abundance", "fractions", "consensus", "specificity", "perCellSummary"] { + for pn in ["abundance", "fractions", "perCellSummary"] { outputs = append(outputs, { type: "Resource", spec: { @@ -242,6 +242,18 @@ wf.body(func(args) { metaExtra.mitoolMemOverrideGB = args.perProcessMemGB } + // Both fan-out maps are closed lists, checked here rather than reviewed later. Each key is part of + // every per-sample body's identity, so one added key costs every user a full parse / refine-tags / + // tag-stat re-run and nothing in the render or the logs says why. A verdict-stage parameter or the + // resolved clonotype linker added for convenience is exactly the addition that would land here. + // fanout-inputs.test.tengo asserts the lists exclude both. + ll.assert(len(fanout.unlisted(extraInputs, fanout.EXTRA_INPUT_KEYS)) == 0, + "per-sample body inputs outside the allowlist: %v. Anything the verdict stage needs belongs in the verdict stage, not in the mitool fan-out.", + fanout.unlisted(extraInputs, fanout.EXTRA_INPUT_KEYS)) + ll.assert(len(fanout.unlisted(metaExtra, fanout.META_EXTRA_KEYS)) == 0, + "per-sample resource meta keys outside the allowlist: %v.", + fanout.unlisted(metaExtra, fanout.META_EXTRA_KEYS)) + perSampleResults := pframes.processColumn( { spec: inputSpec, data: inputData }, fbPipelineTpl, @@ -278,22 +290,17 @@ wf.body(func(args) { // Exported per-cell contract frame (goes to the result pool -> VDJ Multiomic Integration). This is the // full per-(cell x feature) matrix and is UNCHANGED by the table collapse below — downstream still - // consumes abundance/fractions/consensus/specificity at [sampleId, cellId, featureId]. + // consumes abundance and fractions at [sampleId, cellId, featureId]. fb := pframes.pFrameBuilder() addXsvTo(fb, "abundance") addXsvTo(fb, "fractions") - addXsvTo(fb, "consensus") - if hasControl { - addXsvTo(fb, "specificity") - } finalPf := fb.build() - // Main results table (table-only): one row per [sampleId, cellId] — the consensus feature plus the - // per-cell summary aggregates (max UMI count / fraction, max specificity with a control) and the - // "feature : umi : fraction | ..." string listing every feature the cell has signal for. This - // collapses the matrix above for display; the export contract is unaffected. + // Main results table (table-only): one row per [sampleId, cellId] — the per-cell summary aggregates + // (max UMI count, max fraction) and the "feature (fraction%, umi), ..." string listing every feature + // the cell has signal for. This collapses the matrix above for display; the export contract is + // unaffected. tableFb := pframes.pFrameBuilder() - addXsvTo(tableFb, "consensus") addXsvTo(tableFb, "perCellSummary") tablePf := tableFb.build() @@ -340,52 +347,173 @@ wf.body(func(args) { traceSeedSpec: inputSpec }) - return { - outputs: { - // Collapsed per-cell results table (one row per [sampleId, cellId]). - perCellTable: pframes.exportFrame(tablePf), - // Per-sample QC summary table (outputs-only): reads parsed/matched, cells/features, UMIs. - qcSummaryTable: pframes.exportFrame(qcSummaryResult.output("qcSummaryTable")), - // Per-sample QC metrics as JSON (fb-pipeline qcJson) -> model builds the live "Analysis logs" - // (completed-sample heartbeat + run-level summary) from it. - qcJson: perSampleResults.outputData("qcJson"), - // Per-sample x per-step report-file map -> model derives each sample's current step from - // which reports are present (deterministic per-step progress). - stepReports: perSampleResults.outputData("stepReports"), - // Per-sample x per-step LIVE stdout-stream map -> model per-step log handles + live progress. - stepLogs: perSampleResults.outputData("stepLogs"), - // Flat parse stdout stream -> model live parse % + early sample roster (registers first). - parseLogStream: perSampleResults.outputData("parseLogStream"), - // per-cell-metrics (Python) stdout -> model 4-metrics step log. - metricsLogStream: perSampleResults.outputData("metricsLogStream"), - // mitool intermediates kept reachable so they're persisted by content hash (dedup / CID - // stability — see the output-spec note above). No model/UI consumer; the persistence is the point. - parsedMic: perSampleResults.outputData("parsedMic"), - parseReport: perSampleResults.outputData("parseReport"), - refineReport: perSampleResults.outputData("refineReport"), - tagstatTsv: perSampleResults.outputData("tagstatTsv"), - // metrics-exec file outputs kept reachable so they're content-hash-persisted (see the persist - // loop above) — this is what makes capturing the metrics stdout CID-safe. No model consumer. - abundancePersist: perSampleResults.outputData("abundancePersist"), - fractionsPersist: perSampleResults.outputData("fractionsPersist"), - consensusPersist: perSampleResults.outputData("consensusPersist"), - specificityPersist: perSampleResults.outputData("specificityPersist"), - perCellSummaryPersist: perSampleResults.outputData("perCellSummaryPersist"), - // Expose the tag->feature CSV import HANDLE so the model can watch it via getImportProgress() - // (model/src/index.ts). That model output is the actual upload driver: it registers the import - // with the middle-layer so the CSV bytes are pushed. Without it the CSV blob never materialises - // and every per-sample body hangs on __extra_tagsCsv. Mirrors immune-assay-data - // (`dataImportHandle: importFile.handle` + model getImportProgress). - tagFeatureCsvImportHandle: csvImport.handle - }, - exports: { - perCellFeatures: finalPf, - // Per-feature properties (A-0026), keyed on the shared feature axis (pl7.app/feature/featureId). - // A separate frame keeps the per-cell contract above unchanged; the result pool is flat, so - // these are discoverable/joinable by the feature axis downstream (VDJ Multiomic Integration - // reuses this exact feature axis for its per-feature outputs). Empty when the CSV has no extra - // columns. - featureProperties: featurePropsResult.output("featureProperties") + // --- the verdict stage --- + // + // Conditional on a chosen single-cell V(D)J dataset. Without one there is no clonotype set for a verdict + // to be about, and — because the linker query is scoped by that dataset's anchor — no way to say which + // receptor's linker the reading would be answered against. The block still runs and still emits its + // per-cell contract columns, the per-sample QC and the per-feature properties: a missing input narrows + // what can be answered, and nothing fails. + verdictRun := undefined + verdictImport := undefined + if !is_undefined(args.datasetRef) { + // The reading is answered across the whole run, not per sample: a panel is a property of a declared + // tag set rather than of one sample, the panel-versus-reads check runs in both directions across the + // run, and the quality rollup spans samples and panels. So the per-sample counts are gathered into + // one table first, in a child template because the file map must be awaited before its keys can be + // iterated. + countsGather := render.create(gatherCountsTpl, { + tagstatData: perSampleResults.outputData("tagstatTsv"), + tags: args.tags + }) + + // Built HERE, in the body, and not in prepare — see the note on wf.prepare above. The bundle is + // passed to both child templates unresolved: verdict-run holds it resolved, as a map with methods, + // which cannot be serialized as a template input, so each template resolves it for itself. + bundleBuilder := wf.createPBundleBuilder() + bundleBuilder.ignoreMissingDomains() + bundleBuilder.addAnchor("main", args.datasetRef) + // Resolved by NAME as a MULTI query, because the linker is infrastructure: it carries + // pl7.app/isLinkerColumn and is hidden in tables, so it is not a column a user can pick. A dataset + // may bring one linker per receptor, and verdict-linker.lib.tengo selects the one whose clonotype + // axis IS the anchor's — choosing the dataset is choosing the receptor, so a legitimate BCR + TCR + // run needs no panic and gets none. + bundleBuilder.addMulti({ name: "pl7.app/sc/cellLinker" }, "linker") + verdictBundle := bundleBuilder.build() + + // Only the reading's own parameters, and nothing that touched the fan-out. + verdictParams := { + barcodeSeqColumn: args.barcodeSeqColumn, + featureNameColumn: args.featureNameColumn, + sampleColumn: is_undefined(args.sampleColumn) ? "" : args.sampleColumn + } + // Optional parameters are ADDED, never defaulted to undefined. verdict-args distinguishes an absent + // value — leave the CLI's own default, or leave the line off entirely — from a supplied one, and a + // key whose value crosses a template boundary as undefined arrives back as JSON null, which is a + // third thing neither side means. + setParam := func(key, value) { + if !is_undefined(value) { + verdictParams[key] = value + } } + setParam("roleColumn", args.roleColumn) + setParam("referenceValues", args.referenceValues) + setParam("referenceSource", args.referenceSource) + setParam("panelReferenceMinMembers", args.panelReferenceMinMembers) + setParam("referenceThinLine", args.referenceThinLine) + setParam("countFloor", args.countFloor) + setParam("boundCutoff", args.boundCutoff) + setParam("minVotingCells", args.minVotingCells) + setParam("minAgreement", args.minAgreement) + setParam("gateThreshold", args.gateThreshold) + setParam("highReferenceLine", args.highReferenceLine) + setParam("grouping", args.grouping) + setParam("contendingGroups", args.contendingGroups) + setParam("captureMap", args.captureMap) + + verdictRun = render.createEphemeral(verdictRunTpl, { + columns: verdictBundle, + datasetRef: args.datasetRef, + sampleAxisSpec: sampleAxis, + countsFile: countsGather.output("countsFile"), + panelFile: csvFile, + // The combined per-sample read QC, from the same child template that imports it as a table. + // Without it readsTotal, panelAssignedFraction and readsPerCell have no source and read "not + // evaluated" — silently, since a non-evaluation is a legitimate state rather than an error, and + // readsPerCell is the only sequencing-depth alert the block ships. + qcSummaryFile: qcSummaryResult.output("qcSummaryCsv"), + params: verdictParams + }) + + // The import takes the run record as an INPUT so the specs it builds carry the comparator that + // actually SERVED. The software degrades a request it cannot honour — a declared comparator becomes + // none where the panel has no reference tag — and that choice sits in the emitted columns' domain, + // so specs built inside the exec's own template would record the request instead. + verdictImport = render.createEphemeral(verdictImportTpl, { + columns: verdictBundle, + datasetRef: args.datasetRef, + blockId: blockId, + sampleAxisSpec: sampleAxis, + traceSeedSpec: inputSpec, + runMeta: verdictRun.output("runMeta"), + verdicts: verdictRun.output("verdicts"), + setCounts: verdictRun.output("setCounts"), + identitySummary: verdictRun.output("identitySummary"), + cellCounts: verdictRun.output("cellCounts"), + cellScalars: verdictRun.output("cellScalars"), + offered: verdictRun.output("offered"), + tagIdentity: verdictRun.output("tagIdentity"), + identityLabels: verdictRun.output("identityLabels"), + panelLabels: verdictRun.output("panelLabels"), + samplePanel: verdictRun.output("samplePanel"), + panelMismatch: verdictRun.output("panelMismatch"), + qc: verdictRun.output("qc") + }) + } + + blockOutputs := { + // Collapsed per-cell results table (one row per [sampleId, cellId]). + perCellTable: pframes.exportFrame(tablePf), + // Per-sample QC summary table (outputs-only): reads parsed/matched, cells/features, UMIs. + qcSummaryTable: pframes.exportFrame(qcSummaryResult.output("qcSummaryTable")), + // Per-sample QC metrics as JSON (fb-pipeline qcJson) -> model builds the live "Analysis logs" + // (completed-sample heartbeat + run-level summary) from it. + qcJson: perSampleResults.outputData("qcJson"), + // Per-sample x per-step report-file map -> model derives each sample's current step from + // which reports are present (deterministic per-step progress). + stepReports: perSampleResults.outputData("stepReports"), + // Per-sample x per-step LIVE stdout-stream map -> model per-step log handles + live progress. + stepLogs: perSampleResults.outputData("stepLogs"), + // Flat parse stdout stream -> model live parse % + early sample roster (registers first). + parseLogStream: perSampleResults.outputData("parseLogStream"), + // per-cell-metrics (Python) stdout -> model 4-metrics step log. + metricsLogStream: perSampleResults.outputData("metricsLogStream"), + // mitool intermediates kept reachable so they're persisted by content hash (dedup / CID + // stability — see the output-spec note above). No model/UI consumer; the persistence is the point. + parsedMic: perSampleResults.outputData("parsedMic"), + parseReport: perSampleResults.outputData("parseReport"), + refineReport: perSampleResults.outputData("refineReport"), + tagstatTsv: perSampleResults.outputData("tagstatTsv"), + // metrics-exec file outputs kept reachable so they're content-hash-persisted (see the persist + // loop above) — this is what makes capturing the metrics stdout CID-safe. No model consumer. + abundancePersist: perSampleResults.outputData("abundancePersist"), + fractionsPersist: perSampleResults.outputData("fractionsPersist"), + perCellSummaryPersist: perSampleResults.outputData("perCellSummaryPersist"), + // Expose the tag->feature CSV import HANDLE so the model can watch it via getImportProgress() + // (model/src/index.ts). That model output is the actual upload driver: it registers the import + // with the middle-layer so the CSV bytes are pushed. Without it the CSV blob never materialises + // and every per-sample body hangs on __extra_tagsCsv. Mirrors immune-assay-data + // (`dataImportHandle: importFile.handle` + model getImportProgress). + tagFeatureCsvImportHandle: csvImport.handle + } + + blockExports := { + perCellFeatures: finalPf, + // Per-feature properties (A-0026), keyed on the shared feature axis (pl7.app/feature/featureId). + // A separate frame keeps the per-cell contract above unchanged; the result pool is flat, so + // these are discoverable/joinable by the feature axis downstream (VDJ Multiomic Integration + // reuses this exact feature axis for its per-feature outputs). Empty when the CSV has no extra + // columns. + featureProperties: featurePropsResult.output("featureProperties") + } + + // Only present when a V(D)J dataset was chosen. A template output field cannot be assigned undefined, + // so the verdict results are added rather than defaulted. + if !is_undefined(verdictImport) { + // Everything a downstream block joins to: the verdicts, the set-keyed counts that are the only + // family lead selection can see, the pivoted per-identity summary, the offered scope, the + // re-derivation material, the tag -> identity linker and the label columns. + blockExports.antigenVerdicts = verdictImport.output("antigenVerdicts") + // The run's own report. Outputs rather than exports: these are read by this block's model and UI. + blockOutputs.antigenQcTable = pframes.exportFrame(verdictImport.output("qcTable")) + blockOutputs.antigenPanelMismatchTable = pframes.exportFrame(verdictImport.output("panelMismatchTable")) + // What the run was answered under, including the comparator and cell list that actually served and + // every parameter the reading used. Read as content by the model for the run summary. + blockOutputs.antigenRunMeta = verdictRun.output("runMeta") + } + + return { + outputs: blockOutputs, + exports: blockExports } }) diff --git a/workflow/src/qc-summary.tpl.tengo b/workflow/src/qc-summary.tpl.tengo index 4b91c8b..f0ce4ee 100644 --- a/workflow/src/qc-summary.tpl.tengo +++ b/workflow/src/qc-summary.tpl.tengo @@ -20,7 +20,7 @@ columnSpecs := import(":column-specs") json := import("json") -self.defineOutputs("qcSummaryTable") +self.defineOutputs("qcSummaryTable", "qcSummaryCsv") self.body(func(inputs) { qcData := inputs.qcData // ResourceMap: [sampleId] -> per-sample QC CSV file @@ -61,6 +61,11 @@ self.body(func(inputs) { qcSummaryTable := xsv.importFile(csvFile, "csv", columnsSpec, { cpu: 1, mem: "2GiB" }) return { - qcSummaryTable: qcSummaryTable + qcSummaryTable: qcSummaryTable, + // The same combined table as a FILE. The verdict stage reads its read counts through --qc-summary, + // and it is returned here rather than concatenated a second time because the two must agree: the + // per-sample sampleId is injected from the resource-map key, so a second gather could key the rows + // differently and the verdict stage would silently match no sample and evaluate nothing. + qcSummaryCsv: csvFile } }) diff --git a/workflow/src/verdict-args.lib.tengo b/workflow/src/verdict-args.lib.tengo new file mode 100644 index 0000000..efa10b6 --- /dev/null +++ b/workflow/src/verdict-args.lib.tengo @@ -0,0 +1,187 @@ +// The emit-verdicts command line, built in one place and unit-tested. +// +// Every flag this CLI takes has a default, so an unthreaded parameter is silent in both directions: the +// render succeeds, the exec succeeds, and the reading is answered under a value the user never chose. The +// per-sample QC file is the sharpest case — without `--qc-summary` the run has no read count at all, so +// `readsTotal`, `panelAssignedFraction` and `readsPerCell` render as declared-but-unchecked rather than +// failing, and `readsPerCell` is the only sequencing-depth alert the block ships. Building the whole +// argument list here, including the staged input filenames, is what makes the threading assertable +// without a backend. +// +// The staged filenames are exported beside the flags that name them: verdict-run.tpl.tengo stages each +// file under the same constant it is referenced by, so a rename cannot leave a flag pointing at a file +// the exec never received. + +ll := import("@platforma-sdk/workflow-tengo:ll") +canonical := import("@platforma-sdk/workflow-tengo:canonical") + +FILE_COUNTS := "counts.csv" +FILE_PANEL := "panel.csv" +FILE_LINKER := "linker.csv" +FILE_QC_SUMMARY := "qc_summary.csv" +OUTPUT_PREFIX := "result" + +// The block arguments this module reads. Exported so fanout-inputs.test.tengo can assert that none of +// them has leaked into the per-sample mitool fan-out, which is what keeps the fan-out cached when only +// the reading's parameters change. Extend this list whenever a parameter is added below. +PARAMETER_NAMES := [ + "boundCutoff", + "captureMap", + "contendingGroups", + "countFloor", + "datasetRef", + "gateThreshold", + "grouping", + "highReferenceLine", + "minAgreement", + "minVotingCells", + "panelReferenceMinMembers", + "referenceSource", + "referenceThinLine", + "referenceValues", + "roleColumn" +] + +// Defaults mirroring the CLI's own (verdict.py DEFAULT_FLOOR, BOUND_CUTOFF, combine.py +// DEFAULT_MIN_VOTERS, DEFAULT_PANEL_MIN_MEMBERS, DEFAULT_REFERENCE_THIN_LINE, +// DEFAULT_HIGH_REFERENCE_OBSERVATION_LINE). They are restated rather than left to argparse so the value +// that produced a run is always on the command line, where the run record and a re-run both see it; a +// parameter carried only as a CLI default drifts the moment one side changes. +DEFAULT_COUNT_FLOOR := 4 +DEFAULT_BOUND_CUTOFF := 75 +DEFAULT_MIN_VOTING_CELLS := 1 +DEFAULT_PANEL_MIN_MEMBERS := 8 +DEFAULT_REFERENCE_THIN_LINE := 2 +DEFAULT_HIGH_REFERENCE_LINE := 100 + +_num := func(value, fallback) { + return is_undefined(value) ? fallback : value +} + +// An integer flag. int() first because a model that sends 4.0 would otherwise reach argparse as "4.0", +// which `type=int` rejects — a whole-run failure from a value that was correct. +_intArg := func(value) { + return string(int(value)) +} + +// build: the full argument vector, positionals first. +// +// `params` is the block's args map plus `hasLinker` / `hasQcSummary`, which say whether the caller staged +// those two files. A flag naming a file the exec did not receive is a hard failure, so each is emitted +// only with its file. +build := func(params) { + args := [FILE_COUNTS, FILE_PANEL] + + add := func(flag, value) { + args = append(args, flag) + args = append(args, value) + } + + if !is_undefined(params.hasLinker) && params.hasLinker { + add("--linker", FILE_LINKER) + } + if !is_undefined(params.hasQcSummary) && params.hasQcSummary { + add("--qc-summary", FILE_QC_SUMMARY) + } + + // The panel-file column roles. Passed even when empty: an empty --sample-col declares one panel for + // every sample and an empty --role-column declares no comparator designation, both of which are + // readings rather than omissions. + add("--barcode-col", params.barcodeSeqColumn) + add("--feature-col", params.featureNameColumn) + add("--sample-col", is_undefined(params.sampleColumn) ? "" : params.sampleColumn) + add("--role-column", is_undefined(params.roleColumn) ? "" : params.roleColumn) + + // Comma-separated, because that is what the CLI splits on. The list is already de-duplicated and + // sorted by the args lambda, so the string is canonical and two equivalent selections produce one + // cache key. + referenceValues := "" + if !is_undefined(params.referenceValues) { + for i, value in params.referenceValues { + referenceValues = i == 0 ? value : referenceValues + "," + value + } + } + add("--reference-values", referenceValues) + + // Which comparator was ASKED for. The run may serve "none" instead — never a different one — and what + // served is read back from the run record, not from here. + if !is_undefined(params.referenceSource) && params.referenceSource != "" { + add("--reference-source", params.referenceSource) + } + + add("--panel-min-members", _intArg(_num(params.panelReferenceMinMembers, DEFAULT_PANEL_MIN_MEMBERS))) + add("--reference-thin-line", _intArg(_num(params.referenceThinLine, DEFAULT_REFERENCE_THIN_LINE))) + add("--floor", _intArg(_num(params.countFloor, DEFAULT_COUNT_FLOOR))) + add("--cutoff", string(_num(params.boundCutoff, DEFAULT_BOUND_CUTOFF))) + add("--min-voters", _intArg(_num(params.minVotingCells, DEFAULT_MIN_VOTING_CELLS))) + add("--high-reference-line", _intArg(_num(params.highReferenceLine, DEFAULT_HIGH_REFERENCE_LINE))) + + // Off by default, and off means absent rather than zero: a floor of 0 would make every majority pass + // the check instead of skipping the check, and the two are different claims about the reading. + if !is_undefined(params.minAgreement) { + add("--min-agreement", string(params.minAgreement)) + } + // The admissibility gate. Absent and zero both mean off, and zero would set aside every cell. + if !is_undefined(params.gateThreshold) && params.gateThreshold > 0 { + add("--gate-threshold", _intArg(params.gateThreshold)) + } + + // The three JSON-valued flags are encoded key-sorted rather than with the plain encoder. Tengo maps + // have no iteration order, so the plain encoder can emit the same rule as two different strings on two + // renders — which changes the exec's argument list, and with it the cache key, for a run that is + // identical in every way that matters. + // + // A rule over declared properties, not a tag map. Absent means one identity per tag, which is the + // CLI's own default, so the flag is omitted rather than carrying a hand-built {"by":"tag"}. + if !is_undefined(params.grouping) { + add("--grouping", canonical.encode(params.grouping)) + } + // Groups of identities declared to contend for one binding site. The args lambda drops any group of + // fewer than two members and sorts the rest, so an empty list here means nothing contends. + if !is_undefined(params.contendingGroups) && len(params.contendingGroups) > 0 { + add("--contending", canonical.encode(params.contendingGroups)) + } + // No capture assignment reaches the block yet, so this is normally absent and every capture rolls up + // as not evaluated. Threaded now so supplying the map is a model change alone. + if !is_undefined(params.captureMap) { + add("--capture-map", canonical.encode(params.captureMap)) + } + + add("--output-prefix", OUTPUT_PREFIX) + + ll.assert(len(args) % 2 == 0, "verdict-args: every flag must carry a value") + return args +} + +// has: whether the built vector contains a flag. Used by the threading test. +has := func(args, flag) { + for _, a in args { + if a == flag { + return true + } + } + return false +} + +// valueOf: the value following a flag, or undefined. Used by the threading test to check that a flag +// carries what the caller asked for rather than merely appearing. +valueOf := func(args, flag) { + for i, a in args { + if a == flag && i + 1 < len(args) { + return args[i + 1] + } + } + return undefined +} + +export { + FILE_COUNTS: FILE_COUNTS, + FILE_PANEL: FILE_PANEL, + FILE_LINKER: FILE_LINKER, + FILE_QC_SUMMARY: FILE_QC_SUMMARY, + OUTPUT_PREFIX: OUTPUT_PREFIX, + PARAMETER_NAMES: PARAMETER_NAMES, + build: build, + has: has, + valueOf: valueOf +} diff --git a/workflow/src/verdict-args.test.tengo b/workflow/src/verdict-args.test.tengo new file mode 100644 index 0000000..df3265c --- /dev/null +++ b/workflow/src/verdict-args.test.tengo @@ -0,0 +1,119 @@ +test := import("@platforma-sdk/workflow-tengo:test") +va := import(":verdict-args") + +// A fully configured reading. Every optional parameter is set, so a flag missing from the built vector is +// a threading defect rather than an intentional omission. +_full := { + hasLinker: true, + hasQcSummary: true, + barcodeSeqColumn: "Sequence", + featureNameColumn: "Name", + sampleColumn: "Samples", + roleColumn: "Type", + referenceValues: ["Control", "Isotype"], + referenceSource: "declared", + panelReferenceMinMembers: 8, + referenceThinLine: 2, + countFloor: 4, + boundCutoff: 75, + minVotingCells: 1, + minAgreement: 0.6, + gateThreshold: 40, + highReferenceLine: 100, + grouping: { by: "property", column: "family" }, + contendingGroups: [["AgA", "AgB"]], + captureMap: { S1: "lane1" } +} + +// Every flag emit_verdicts.py declares, checked against the built vector. An unthreaded parameter fails +// nothing at render or exec time — the CLI defaults it — so this list is the only place the omission +// shows up. +Test_every_flag_is_threaded := func() { + args := va.build(_full) + flags := [ + "--linker", "--qc-summary", + "--barcode-col", "--feature-col", "--sample-col", "--role-column", "--reference-values", + "--reference-source", "--panel-min-members", "--reference-thin-line", "--floor", "--cutoff", + "--min-voters", "--min-agreement", "--gate-threshold", "--high-reference-line", + "--grouping", "--contending", "--capture-map", "--output-prefix" + ] + for _, flag in flags { + test.isTrue(va.has(args, flag), "argument list is missing " + flag) + } +} + +// The per-sample read QC is the one input whose absence is invisible: three of the fifteen measurements +// have no other source, and they render as declared-but-unchecked rather than as an error. The flag must +// name the file the run template stages, or the exec fails on a path it never received. +Test_qc_summary_points_at_the_staged_file := func() { + args := va.build(_full) + test.isEqual(va.FILE_QC_SUMMARY, va.valueOf(args, "--qc-summary"), + "--qc-summary must name the staged per-sample QC CSV") +} + +// The two positionals lead, and each names a staged file. +Test_positionals_lead_and_name_staged_files := func() { + args := va.build(_full) + test.isEqual(va.FILE_COUNTS, args[0], "the counts table is the first positional") + test.isEqual(va.FILE_PANEL, args[1], "the panel file is the second positional") +} + +// A flag naming a file the exec never received is a hard failure, so neither file flag may appear on its +// own. Without a dataset there is no linker; the rest of the reading still runs. +Test_file_flags_follow_their_files := func() { + args := va.build({ barcodeSeqColumn: "Sequence", featureNameColumn: "Name" }) + test.isFalse(va.has(args, "--linker"), "the linker flag must be absent when no linker was staged") + test.isFalse(va.has(args, "--qc-summary"), "the QC flag must be absent when no QC table was staged") +} + +// Off means absent, not zero. A zero gate would set aside every cell and a zero agreement floor would +// pass every majority instead of skipping the check — both are readings, and neither is "off". +Test_optional_lines_are_absent_when_off := func() { + args := va.build({ barcodeSeqColumn: "Sequence", featureNameColumn: "Name", gateThreshold: 0 }) + test.isFalse(va.has(args, "--gate-threshold"), "the gate must be absent when off") + test.isFalse(va.has(args, "--min-agreement"), "the agreement floor must be absent when off") + test.isFalse(va.has(args, "--grouping"), "an absent grouping rule leaves the CLI's per-tag default") + test.isFalse(va.has(args, "--contending"), "no contending groups means no flag") + test.isFalse(va.has(args, "--reference-source"), "an unchosen comparator is resolved by the software") +} + +// The parameters that shape the reading are always on the command line, defaulted here rather than left +// to argparse, so the value a run was answered under is recorded with the run. +Test_shaping_parameters_are_always_present := func() { + args := va.build({ barcodeSeqColumn: "Sequence", featureNameColumn: "Name" }) + test.isEqual("4", va.valueOf(args, "--floor"), "the count floor is stated even when defaulted") + test.isEqual("75", va.valueOf(args, "--cutoff"), "the bound cutoff is stated even when defaulted") + test.isEqual("1", va.valueOf(args, "--min-voters"), "the voter minimum is stated even when defaulted") + test.isEqual("8", va.valueOf(args, "--panel-min-members"), "the panel minimum is stated") + test.isEqual("2", va.valueOf(args, "--reference-thin-line"), "the thin-comparator line is stated") + test.isEqual("100", va.valueOf(args, "--high-reference-line"), "the high-reference line is stated") +} + +// argparse declares --floor, --min-voters, --panel-min-members, --reference-thin-line, +// --high-reference-line and --gate-threshold as `type=int`, which rejects "4.0". A model that rounds to a +// whole number still hands it over as a JSON number, so the conversion happens here. +Test_integer_flags_carry_no_decimal_point := func() { + args := va.build({ + barcodeSeqColumn: "Sequence", featureNameColumn: "Name", + countFloor: 4.0, minVotingCells: 2.0, gateThreshold: 40.0, + panelReferenceMinMembers: 8.0, referenceThinLine: 2.0, highReferenceLine: 100.0 + }) + test.isEqual("4", va.valueOf(args, "--floor"), "the count floor reaches argparse as an integer") + test.isEqual("2", va.valueOf(args, "--min-voters"), "the voter minimum reaches argparse as an integer") + test.isEqual("40", va.valueOf(args, "--gate-threshold"), "the gate reaches argparse as an integer") +} + +// The role values are comma-separated because that is what the CLI splits on, and they arrive already +// sorted so two equivalent selections produce one cache key. +Test_reference_values_are_comma_separated := func() { + args := va.build(_full) + test.isEqual("Control,Isotype", va.valueOf(args, "--reference-values"), + "role values reach the CLI comma-separated") +} + +// The grouping rule travels as JSON, so a property grouping reaches the software intact. +Test_grouping_rule_travels_as_json := func() { + args := va.build(_full) + test.isEqual("{\"by\":\"property\",\"column\":\"family\"}", va.valueOf(args, "--grouping"), + "the grouping rule is passed as JSON") +} diff --git a/workflow/src/verdict-import.tpl.tengo b/workflow/src/verdict-import.tpl.tengo new file mode 100644 index 0000000..381abaf --- /dev/null +++ b/workflow/src/verdict-import.tpl.tengo @@ -0,0 +1,151 @@ +// The verdict import: every table emit_verdicts.py wrote, turned into p-columns. +// +// Separate from the exec (verdict-run.tpl.tengo) because the run record has to be read as a VALUE before +// any of these specs can be built. `referenceChoice` and `cellListSource` are resolved by the software — +// a declared comparator degrades to none where the panel carries no reference tag, a panel comparator +// degrades below the minimum membership — and both sit in the emitted columns' DOMAIN, because a verdict +// read against a declared reference is not the same reading as one read against the panel's own signal. +// Domain is part of column identity and annotations are not, so recording the choice as an annotation +// would let two incomparable runs emit columns of identical identity and be unioned in a pool holding +// both. Building the specs inside the exec's own template would record the choice that was REQUESTED. +// +// Ephemeral: it awaits the bundle (for the clonotype axis, taken verbatim from the linker) and the run +// record. It receives the bundle UNRESOLVED from main.tpl.tengo — verdict-run holds it resolved, as a map +// with methods, which cannot be serialized as a template input, so each template resolves it itself. + +self := import("@platforma-sdk/workflow-tengo:tpl") +xsv := import("@platforma-sdk/workflow-tengo:pframes.xsv") +pframes := import("@platforma-sdk/workflow-tengo:pframes") +pSpec := import("@platforma-sdk/workflow-tengo:pframes.spec") +maps := import("@platforma-sdk/workflow-tengo:maps") +ll := import("@platforma-sdk/workflow-tengo:ll") +columnSpecs := import(":column-specs") +verdictLinker := import(":verdict-linker") + +json := import("json") + +// Returns three frames: antigenVerdicts (exported), qcTable and panelMismatchTable (this block's own). +// defineOutputs is deliberately absent — an ephemeral template takes its output names from what the body +// returns and never checks a declared list, so declaring one here would be inert. +self.awaitState("columns", "PColumnBundle") +self.awaitState("runMeta", "ResourceReady") + +self.body(func(inputs) { + blockId := inputs.blockId + sampleAxis := inputs.sampleAxisSpec + + // What actually served, read from the run record rather than from the block's arguments. + served := json.decode(string(inputs.runMeta.getData())) + + // The clonotype axis is taken VERBATIM from the linker rather than rebuilt: an axis assembled here + // would be a lookalike carrying a different identity, and would join to nothing. + columns := inputs.columns + anchorSpec := columns.getSpec(inputs.datasetRef) + linkerCol := verdictLinker.pick(columns.getColumns("linker"), anchorSpec.axesSpec[1]) + setAxis := linkerCol.spec.axesSpec[2] + + cellAxis := columnSpecs.cellAxis(sampleAxis.name) + identityAxis := columnSpecs.identityAxis(blockId, served.groupingId) + tagAxis := columnSpecs.tagAxis(blockId) + panelAxis := columnSpecs.panelAxis(blockId) + + // Seeded from the FASTQ dataset spec: these columns' subject is the antigen readout that dataset + // produced, so seeding from the linker would label them after the clonotyping run instead. The + // clonotyping root is already recorded where it belongs — pl7.app/vdj/scClonotypeKey carries its + // clonotyping run id in its own domain — so it needs no trace step of its own. makeTrace is variadic + // and the linker is deliberately not one of its steps. + trace := pSpec.makeTrace(inputs.traceSeedSpec, { + type: "milaboratories.feature-integration", + id: blockId, + importance: 30, + label: "Feature Barcode Profiling" + }) + + // One import per table. Keys are prefixed with the table name because two tables can carry the same + // column id — both label tables call their column "label" in the CSV — and one p-frame key must mean + // one column. + // + // The importer's result map is walked with maps.forEach, whose key order is sorted. A bare + // `for k, v in` has no defined order in Tengo, which would make the built frame's resource handle vary + // run to run and silently cost every downstream node its deduplication, with nothing reported. + addTo := func(fb, name, file, spec) { + imported := xsv.importFile(file, "csv", spec, { splitDataAndSpec: true }) + maps.forEach(imported, func(key, column) { + fb.add(name + "/" + key, trace.inject(column.spec), column.data) + }) + } + + // --- exported: everything a downstream block joins to ------------------------------------------- + exportFb := pframes.pFrameBuilder() + + addTo(exportFb, "verdicts", inputs.verdicts, + columnSpecs.verdictsImportSpec(setAxis, identityAxis, served)) + addTo(exportFb, "setCounts", inputs.setCounts, + columnSpecs.setCountsImportSpec(setAxis, served)) + + // The pivoted per-identity summary is the only per-antigen state lead selection can see: a column + // carrying an axis the clonotype anchor does not have is dropped there with no error, so nothing keyed + // (set, identity) reaches it. The pivot costs one column per identity, so the software gates it on an + // identity count and records both the limit and whether it emitted. When it did not, the CSV carries + // only the key column, and an empty identity list yields no columns rather than a failed import. + summaryIdentities := [] + if !is_undefined(served.identitySummaryEmitted) && served.identitySummaryEmitted { + ll.assert(!is_undefined(served.identities), + "the run record reports an emitted identity summary but names no identities") + summaryIdentities = served.identities + } + if len(summaryIdentities) > 0 { + addTo(exportFb, "identitySummary", inputs.identitySummary, + columnSpecs.identitySummaryImportSpec(setAxis, summaryIdentities, served.groupingId, served)) + } + + // The re-derivation material. Both are EXPORTS rather than outputs: an output is visible only to this + // block's own model, so material a reader needs in order to regroup the panel without a re-run would + // reach nobody. + addTo(exportFb, "cellCounts", inputs.cellCounts, + columnSpecs.cellTagCountsImportSpec(sampleAxis, cellAxis, tagAxis)) + addTo(exportFb, "cellScalars", inputs.cellScalars, + columnSpecs.cellScalarsImportSpec(sampleAxis, cellAxis, served)) + + // Which identities each sample was actually stained with — without it "never asked" is a claim a + // reader cannot check. + addTo(exportFb, "offered", inputs.offered, + columnSpecs.offeredImportSpec(sampleAxis, identityAxis)) + + // The tag -> identity linker: what lets a reader put a tag's count beside its verdict without either + // layer knowing about the other. + addTo(exportFb, "tagIdentity", inputs.tagIdentity, + columnSpecs.tagIdentityLinkerImportSpec(tagAxis, identityAxis)) + + // The label columns. A label is satisfied only by a column NAMED pl7.app/label with exactly one axis, + // and the readable names are imported as columns rather than read as values — which is also why they + // cannot be used while these specs are built. + addTo(exportFb, "identityLabels", inputs.identityLabels, + columnSpecs.identityLabelsImportSpec(identityAxis)) + addTo(exportFb, "panelLabels", inputs.panelLabels, + columnSpecs.panelLabelsImportSpec(panelAxis)) + + // sample -> panel, so per-tag QC keyed (panel, tag) can be read back to the samples it covers. Where + // one panel covers every sample this column is constant and drops out of view. + addTo(exportFb, "samplePanel", inputs.samplePanel, + columnSpecs.samplePanelImportSpec(sampleAxis)) + + // --- block-local: the run's own report ---------------------------------------------------------- + qcFb := pframes.pFrameBuilder() + addTo(qcFb, "qc", inputs.qc, columnSpecs.qcImportSpec( + columnSpecs.qcLevelAxis(blockId), + columnSpecs.qcEntityAxis(blockId), + columnSpecs.qcMeasurementAxis(blockId))) + + // The panel-versus-reads check is emitted as a p-column rather than left as a file: a mismatch report + // the user cannot see defeats its purpose. + mismatchFb := pframes.pFrameBuilder() + addTo(mismatchFb, "panelMismatch", inputs.panelMismatch, + columnSpecs.panelMismatchImportSpec(panelAxis, tagAxis)) + + return { + antigenVerdicts: exportFb.build(), + qcTable: qcFb.build(), + panelMismatchTable: mismatchFb.build() + } +}) diff --git a/workflow/src/verdict-linker.lib.tengo b/workflow/src/verdict-linker.lib.tengo new file mode 100644 index 0000000..57979bd --- /dev/null +++ b/workflow/src/verdict-linker.lib.tengo @@ -0,0 +1,91 @@ +// Choosing the cell linker, and refusing the two ways it can be wrong. +// +// The linker is infrastructure: it carries pl7.app/isLinkerColumn, is hidden in tables, and is therefore +// not a column a user can pick. The model stores the single-cell VDJ dataset anchor instead, and the +// linker is resolved from the bundle by name. A dataset may nevertheless bring several linkers — mixcr +// emits one per receptor — and the anchor is receptor-scoped, so the one whose clonotype axis IS the +// anchor's clonotype axis is the one the user chose. Picking by that identity rather than panicking on a +// count keeps a legitimate BCR + TCR run working. +// +// Shared by verdict-run.tpl.tengo and verdict-import.tpl.tengo, which each resolve the bundle +// independently: the run template holds it in resolved form, which cannot be serialized as a template +// input, so the import template receives the unresolved reference and resolves it again. + +ll := import("@platforma-sdk/workflow-tengo:ll") +maps := import("@platforma-sdk/workflow-tengo:maps") +canonical := import("@platforma-sdk/workflow-tengo:canonical") + +LINKER_ANNOTATION := "pl7.app/isLinkerColumn" + +// Axis identity is name plus domain; annotations are excluded from it. Compared through the key-sorted +// encoder because two maps holding the same pairs are not equal to each other in Tengo. +_axisKey := func(axis) { + return canonical.encode({ + name: axis.name, + domain: is_undefined(axis.domain) ? {} : axis.domain + }) +} + +sameAxis := func(a, b) { + return _axisKey(a) == _axisKey(b) +} + +// pick: the linker belonging to the chosen dataset. +// +// `clonotypeAxis` is the anchor's own clonotype axis (its second axis). A cell linker is keyed +// [sampleId, cellId, scClonotypeKey], so its third axis is the one to match. +pick := func(linkerCols, clonotypeAxis) { + if len(linkerCols) == 0 { + ll.panic("Antigen binding: the selected dataset has no cell linker. Choose a single-cell V(D)J dataset produced by the Import V(D)J Data block.") + } + for _, col in linkerCols { + axes := col.spec.axesSpec + if len(axes) == 3 && sameAxis(axes[2], clonotypeAxis) { + return col + } + } + ll.panic("Antigen binding: none of the %d cell linkers found is keyed on the selected dataset's clonotype axis (%v). Re-select the single-cell V(D)J dataset.", + len(linkerCols), clonotypeAxis.name) +} + +// sampleAxisMismatch: "" when the linker's sample axis is the block's, otherwise a readable reason. +// +// The block's sample axis comes from the FASTQ dataset; the linker's comes from the clonotyping run, and +// on a multiplexed input that can be pl7.app/sampleGroupId instead. Joining across the two produces no +// rows and no error, so the verdicts would simply come out empty. Named here so the caller can panic. +sampleAxisMismatch := func(linkerSpec, sampleAxisSpec) { + linkerSampleAxis := linkerSpec.axesSpec[0] + if sameAxis(linkerSampleAxis, sampleAxisSpec) { + return "" + } + return "Antigen binding: the V(D)J dataset's sample axis (" + linkerSampleAxis.name + + ") is not the feature-barcode dataset's (" + sampleAxisSpec.name + + "). The two runs were keyed differently — commonly one was multiplexed and the other was not — so " + + "no cell would match and every verdict would come out empty. Re-run clonotyping on the same samples." +} + +// plainSpec: the linker's spec with the linker marker removed. +// +// The CSV export builds a linker index for a column flagged this way, and that index requires exactly two +// connected components, which a three-axis cell linker does not have. The export wants the axis tuple as +// flat rows, so the marker is dropped from a COPY — mutating the resolved spec would change what every +// other reader of the bundle sees. +plainSpec := func(spec) { + annotations := {} + maps.forEach(is_undefined(spec.annotations) ? {} : spec.annotations, func(key, value) { + if key != LINKER_ANNOTATION { + annotations[key] = value + } + }) + plain := maps.clone(spec) + plain.annotations = annotations + return plain +} + +export { + LINKER_ANNOTATION: LINKER_ANNOTATION, + sameAxis: sameAxis, + pick: pick, + sampleAxisMismatch: sampleAxisMismatch, + plainSpec: plainSpec +} diff --git a/workflow/src/verdict-linker.test.tengo b/workflow/src/verdict-linker.test.tengo new file mode 100644 index 0000000..252c407 --- /dev/null +++ b/workflow/src/verdict-linker.test.tengo @@ -0,0 +1,76 @@ +test := import("@platforma-sdk/workflow-tengo:test") +linker := import(":verdict-linker") + +_sampleAxis := { name: "pl7.app/sampleId", type: "String" } +_cellAxis := { name: "pl7.app/sc/cellId", type: "String" } + +_clonotypeAxis := func(runId) { + return { + name: "pl7.app/vdj/scClonotypeKey", + type: "String", + domain: { "pl7.app/vdj/clonotypingRunId": runId } + } +} + +_linkerCol := func(clonotypeAxis) { + return { + spec: { + name: "pl7.app/sc/cellLinker", + valueType: "Int", + axesSpec: [_sampleAxis, _cellAxis, clonotypeAxis], + annotations: { "pl7.app/isLinkerColumn": "true", "pl7.app/label": "Cell linker" } + }, + data: "data-placeholder" + } +} + +// A dataset carrying one linker per receptor is legitimate, and the anchor says which receptor the user +// chose. Picking by the clonotype axis rather than by count is what keeps a BCR + TCR run working. +Test_pick_selects_the_anchors_own_linker := func() { + tcr := _linkerCol(_clonotypeAxis("run-tcr")) + bcr := _linkerCol(_clonotypeAxis("run-bcr")) + test.isEqual(bcr.spec, linker.pick([tcr, bcr], _clonotypeAxis("run-bcr")).spec, + "the linker keyed on the chosen dataset's clonotype axis is selected") + test.isEqual(tcr.spec, linker.pick([bcr, tcr], _clonotypeAxis("run-tcr")).spec, + "selection does not depend on the order the linkers were resolved in") +} + +// Axis identity is name plus domain. Two clonotyping runs produce axes of the same name whose domains +// differ, and treating them as one would join the verdicts to the wrong clonotypes. +Test_axis_identity_includes_the_domain := func() { + test.isTrue(linker.sameAxis(_clonotypeAxis("r1"), _clonotypeAxis("r1")), "same name and domain match") + test.isFalse(linker.sameAxis(_clonotypeAxis("r1"), _clonotypeAxis("r2")), + "the same axis name under a different clonotyping run is a different axis") + test.isTrue(linker.sameAxis({ name: "pl7.app/sampleId" }, { name: "pl7.app/sampleId", domain: {} }), + "an absent domain and an empty one are the same identity") +} + +// A sample-axis mismatch produces no rows and no error, so it is named rather than left to surface as an +// empty block. +Test_sample_axis_mismatch_is_named := func() { + matched := _linkerCol(_clonotypeAxis("r1")) + test.isEqual("", linker.sampleAxisMismatch(matched.spec, _sampleAxis), + "a matching sample axis reports no mismatch") + + grouped := { + spec: { + axesSpec: [{ name: "pl7.app/sampleGroupId", type: "String" }, _cellAxis, _clonotypeAxis("r1")] + } + } + test.isFalse(linker.sampleAxisMismatch(grouped.spec, _sampleAxis) == "", + "a multiplexed sample axis is reported as a mismatch rather than joining to nothing") +} + +// The marker is dropped from a copy: mutating the resolved spec would change what every other reader of +// the bundle sees, and the CSV export is the only consumer that needs it gone. +Test_plainSpec_copies_and_keeps_everything_else := func() { + original := _linkerCol(_clonotypeAxis("r1")).spec + plain := linker.plainSpec(original) + test.isTrue(is_undefined(plain.annotations["pl7.app/isLinkerColumn"]), + "the linker marker is removed from the exported spec") + test.isEqual("Cell linker", plain.annotations["pl7.app/label"], + "every other annotation survives") + test.isEqual("true", original.annotations["pl7.app/isLinkerColumn"], + "the resolved spec itself is untouched") + test.isEqual(original.axesSpec, plain.axesSpec, "the axis tuple is unchanged") +} diff --git a/workflow/src/verdict-run.tpl.tengo b/workflow/src/verdict-run.tpl.tengo new file mode 100644 index 0000000..4669e02 --- /dev/null +++ b/workflow/src/verdict-run.tpl.tengo @@ -0,0 +1,123 @@ +// The verdict exec, and nothing else. +// +// Split from the import (verdict-import.tpl.tengo) because which comparator SERVED the run is decided by +// the software, not by the request: reference_by_cell degrades a declared choice to none when the panel +// carries no reference tag, and a panel choice to none below the minimum membership. That choice belongs in +// the emitted columns' domain, so the specs have to be built from the run record — and reading a value an +// exec has not yet produced needs a template boundary. Building the specs here would silently record what +// was asked for instead of what answered. +// +// Ephemeral rather than pure because it resolves a PColumnBundle, whose data this template awaits. That +// wait happens HERE and not in main's prepare: prepare's bundle is awaited in full, spec and data, before +// the body runs, so a linker there would hold the entire per-sample mitool fan-out behind the whole +// clonotyping chain — two chains that run concurrently today, serialized, with nothing surfacing it until +// a real dataset. + +self := import("@platforma-sdk/workflow-tengo:tpl") +exec := import("@platforma-sdk/workflow-tengo:exec") +assets := import("@platforma-sdk/workflow-tengo:assets") +pframes := import("@platforma-sdk/workflow-tengo:pframes") +maps := import("@platforma-sdk/workflow-tengo:maps") +ll := import("@platforma-sdk/workflow-tengo:ll") +va := import(":verdict-args") +verdictLinker := import(":verdict-linker") + +verdictsSw := assets.importSoftware("@platforma-open/milaboratories.feature-integration.per-cell-metrics:emit-verdicts") + +// Every table emit_verdicts.py writes, paired with the output field it is returned on. One list, used as +// the saveFile set and as the returned keys, so a table cannot be saved and then quietly dropped — or +// named one thing here and another in the import. +RESULT_TABLES := [ + { out: "verdicts", file: "result_verdicts.csv" }, + { out: "setCounts", file: "result_set_counts.csv" }, + { out: "identitySummary", file: "result_identity_summary.csv" }, + { out: "cellCounts", file: "result_cell_counts.csv" }, + { out: "cellScalars", file: "result_cell_scalars.csv" }, + { out: "offered", file: "result_offered.csv" }, + { out: "tagIdentity", file: "result_tag_identity.csv" }, + { out: "identityLabels", file: "result_identity_labels.csv" }, + { out: "panelLabels", file: "result_panel_labels.csv" }, + { out: "samplePanel", file: "result_sample_panel.csv" }, + { out: "panelMismatch", file: "result_panel_mismatch.csv" }, + { out: "qc", file: "result_qc.csv" } +] + +self.awaitState("columns", "PColumnBundle") + +self.body(func(inputs) { + columns := inputs.columns + + // The anchor is the single-cell V(D)J dataset: axes [sampleId, scClonotypeKey]. Its clonotype axis is + // what identifies the linker belonging to the receptor the user chose. + anchorSpec := columns.getSpec(inputs.datasetRef) + ll.assert(len(anchorSpec.axesSpec) == 2, + "Antigen binding: the selected dataset is keyed on %d axes, not the expected [sampleId, scClonotypeKey].", + len(anchorSpec.axesSpec)) + + linkerCol := verdictLinker.pick(columns.getColumns("linker"), anchorSpec.axesSpec[1]) + linkerAxes := linkerCol.spec.axesSpec + + mismatch := verdictLinker.sampleAxisMismatch(linkerCol.spec, inputs.sampleAxisSpec) + if mismatch != "" { + ll.panic("%v", mismatch) + } + + // The linker as flat rows: (sampleId, cellId, setId). Headers are bound to axes by NAME, not by + // position, so a linker whose axes were declared in another order still exports correctly. The value + // column is unused by the software — the axis tuple is the whole content. + linkerCsv := pframes.csvFileBuilder() + linkerCsv.add({ spec: verdictLinker.plainSpec(linkerCol.spec), data: linkerCol.data }, { header: "linker" }) + linkerCsv.setAxisHeader(linkerAxes[0].name, "sampleId") + linkerCsv.setAxisHeader(linkerAxes[1].name, "cellId") + linkerCsv.setAxisHeader(linkerAxes[2].name, "setId") + linkerCsv.cpu(1) + linkerCsv.mem("8GiB") + linkerCsvFile := linkerCsv.build() + + // The reading holds one entry per analysed cell and one per (cell, tag) count, so it scales with the + // gathered counts table rather than with any one sample — the same sizing shape the per-cell metrics + // step uses, on the same floor. + formula := exec.formula + memFormula := formula.gib(16). + plus(formula.size("counts").times(8)). + between(formula.gib(16), formula.gib(256)). + staticFallback(formula.gib(16)) + + params := maps.merge(inputs.params, { + hasLinker: true, + hasQcSummary: !is_undefined(inputs.qcSummaryFile) + }) + + b := exec.builder(). + software(verdictsSw). + resources({ onCPU: { cpu: 4, ram: memFormula } }). + // polars sizes its thread pool to every host core by default; cap it to the granted CPU. + envWithVar("POLARS_MAX_THREADS", "{system.cpu}"). + addFile(va.FILE_COUNTS, inputs.countsFile, { tag: "counts" }). + addFile(va.FILE_PANEL, inputs.panelFile). + addFile(va.FILE_LINKER, linkerCsvFile) + + // The per-sample read QC. Without it readsTotal, panelAssignedFraction and readsPerCell have no source + // at all and read "not evaluated" — which is silent, because a non-evaluation is a legitimate state + // rather than an error, and readsPerCell is the only sequencing-depth alert the block ships. + if params.hasQcSummary { + b = b.addFile(va.FILE_QC_SUMMARY, inputs.qcSummaryFile) + } + + for _, a in va.build(params) { + b = b.arg(a) + } + for _, table in RESULT_TABLES { + b = b.saveFile(table.file) + } + run := b.saveFileContent("result_run_meta.json").run() + + // The run record travels as content because the import template reads it as a VALUE; the tables travel + // as files because the import template hands them to xsv.importFile. An exec result object is not + // serializable as a template output, so each table is returned on its own field. + result := { runMeta: run.getFileContent("result_run_meta.json") } + for _, table in RESULT_TABLES { + result[table.out] = run.getFile(table.file) + } + return result +}) From 01cabdd4096add8bc145a56a7601c540ed8660be Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 20:21:25 +0200 Subject: [PATCH 055/282] MILAB-6496: make the panel part of the QC key, not a column beside it A barcode recurring across panels is the ordinary case -- the same reagent is stained into several samples -- and it writes one QC row per panel at the same (level, entity, measurement). With the panel as a value column those rows share an axis key, which is a duplicate-key import: it does not raise, it keeps one row and loses the rest, so a reagent misbehaving in one panel and reading clean in another reports as whichever row happened to survive. Measured on the committed fixture bed: 22 of 122 QC rows collided, one identity appearing under four different panels. Sample-level and capture-level rows carry an empty panel, which is honest -- a per-sample measurement belongs to no single panel. Keying per-tag QC by (panel, tag) is what the architecture already called for. --- workflow/src/column-specs.lib.tengo | 18 +++++++++++++++--- workflow/src/verdict-import.tpl.tengo | 3 ++- 2 files changed, 17 insertions(+), 4 deletions(-) diff --git a/workflow/src/column-specs.lib.tengo b/workflow/src/column-specs.lib.tengo index e24fba3..51723d2 100644 --- a/workflow/src/column-specs.lib.tengo +++ b/workflow/src/column-specs.lib.tengo @@ -936,7 +936,19 @@ panelMismatchImportSpec := func(panelAxisSpec, tagAxisSpec) { // never mistake "nothing computed this yet" for "this was checked and found fine". `panelId` travels // as an ordinary column rather than a fourth axis — it is how a panel rollup finds its constituents, // and a sample-level row simply leaves it empty. -qcImportSpec := func(levelAxisSpec, entityAxisSpec, measurementAxisSpec) { +/* The panel is part of the KEY, not a value column beside it. A barcode + recurring across panels -- the ordinary case, since the same reagent is + stained into several samples -- writes one row per panel at the same + (level, entity, measurement). With the panel as a value column those rows + share an axis key, which is a duplicate-key import: it does not raise, it + silently keeps one row and loses the rest, so a reagent misbehaving in one + panel and not another reads as whichever row survived. Measured on the + committed fixture bed, 22 of 122 QC rows collided this way. + + Sample-level and capture-level rows carry an empty panel, which is honest: + a per-sample measurement belongs to no single panel. Per-tag QC keyed by + (panel, tag) is also what the architecture already required. */ +qcImportSpec := func(levelAxisSpec, entityAxisSpec, measurementAxisSpec, panelAxisSpec) { col := func(colName, name, valueType, label, order, visible, annotations) { return { column: colName, @@ -951,6 +963,7 @@ qcImportSpec := func(levelAxisSpec, entityAxisSpec, measurementAxisSpec) { return { axes: [ { column: "level", spec: levelAxisSpec }, + { column: "panelId", spec: panelAxisSpec }, { column: "entity", spec: entityAxisSpec }, { column: "measurement", spec: measurementAxisSpec } ], @@ -967,8 +980,7 @@ qcImportSpec := func(levelAxisSpec, entityAxisSpec, measurementAxisSpec) { col("notEvaluated", "pl7.app/antigen/qcNotEvaluated", "Int", "Not evaluated", 67000, false, { "pl7.app/min": "0" }), col("counts", "pl7.app/antigen/qcCounts", "String", "What it counts", 66000, false, {}), col("implies", "pl7.app/antigen/qcImplies", "String", "What a bad value means", 65000, false, {}), - col("reason", "pl7.app/antigen/qcReason", "String", "Why deferred", 64000, false, {}), - col("panelId", "pl7.app/antigen/qcPanelId", "String", "Panel", 63000, false, {}) + col("reason", "pl7.app/antigen/qcReason", "String", "Why deferred", 64000, false, {}) ], storageFormat: "Parquet", partitionKeyLength: 0 diff --git a/workflow/src/verdict-import.tpl.tengo b/workflow/src/verdict-import.tpl.tengo index 381abaf..f194e19 100644 --- a/workflow/src/verdict-import.tpl.tengo +++ b/workflow/src/verdict-import.tpl.tengo @@ -135,7 +135,8 @@ self.body(func(inputs) { addTo(qcFb, "qc", inputs.qc, columnSpecs.qcImportSpec( columnSpecs.qcLevelAxis(blockId), columnSpecs.qcEntityAxis(blockId), - columnSpecs.qcMeasurementAxis(blockId))) + columnSpecs.qcMeasurementAxis(blockId), + columnSpecs.panelAxis(blockId))) // The panel-versus-reads check is emitted as a p-column rather than left as a file: a mismatch report // the user cannot see defeats its purpose. From 6f315f73fbc6a28d146fe861e6a918940b522736 Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 20:30:00 +0200 Subject: [PATCH 056/282] MILAB-6496: guard the legacy feature axis where the block tests cannot The block test is deliberately the last task now, so the assertion that no emitted column keys on pl7.app/feature/featureId does not exist until after live verification. That axis is keyed by feature NAME; a new antigen table reusing it while carrying barcode SEQUENCES would keep the axis identity and invert its value space, so no downstream query would fail -- joins would return wrong rows. Every antigen table imported by this block goes through one helper, which makes it the single place the check can run for all of them. The legacy per-cell contract columns are built elsewhere and keep the axis legitimately. The predicate is split from the assertion so it can be tested: tengo has no try/catch, so a panicking guard cannot be exercised from a test at all. Both directions are covered -- the legacy axis is detected, the minted antigen axes pass. --- workflow/src/column-guards.test.tengo | 32 ++++++++++++++++++++++++++ workflow/src/column-specs.lib.tengo | 33 ++++++++++++++++++++++++++- workflow/src/verdict-import.tpl.tengo | 5 ++++ 3 files changed, 69 insertions(+), 1 deletion(-) create mode 100644 workflow/src/column-guards.test.tengo diff --git a/workflow/src/column-guards.test.tengo b/workflow/src/column-guards.test.tengo new file mode 100644 index 0000000..3d9e54e --- /dev/null +++ b/workflow/src/column-guards.test.tengo @@ -0,0 +1,32 @@ +test := import("@platforma-sdk/workflow-tengo:test") +columnSpecs := import(":column-specs") + +// The legacy per-cell contract keys features by NAME on pl7.app/feature/featureId. A new antigen table +// reusing that axis while carrying barcode SEQUENCES would leave the axis identity untouched and invert +// its value space, so nothing downstream would error — joins would simply return wrong rows. That is the +// worst available failure, which is why it is asserted at import rather than reviewed for. +// +// The predicate is tested rather than the assertion: tengo has no try/catch, so a panicking guard cannot +// be exercised from a test at all. +Test_the_legacy_feature_axis_is_detected := func() { + legacy := [ + { column: "sampleId", spec: { name: "pl7.app/sampleId", type: "String" } }, + { column: "tag", spec: { name: "pl7.app/feature/featureId", type: "String" } } + ] + test.isTrue(columnSpecs.usesLegacyFeatureAxis(legacy), + "an antigen table keyed on the legacy feature axis must be detected") +} + +// The minted antigen axes pass, so this is a check on one name rather than a blanket refusal that every +// caller would have to work around. +Test_the_minted_antigen_axes_are_not_the_legacy_axis := func() { + blockId := "blk" + minted := [ + { column: "tag", spec: columnSpecs.tagAxis(blockId) }, + { column: "panelId", spec: columnSpecs.panelAxis(blockId) } + ] + test.isFalse(columnSpecs.usesLegacyFeatureAxis(minted), + "the minted antigen axes are not the legacy axis and must pass") + test.isEqual(len(columnSpecs.guardNoLegacyFeatureAxis(minted)), 2, + "the guard returns its input unchanged when nothing is wrong") +} diff --git a/workflow/src/column-specs.lib.tengo b/workflow/src/column-specs.lib.tengo index 51723d2..11a2fb4 100644 --- a/workflow/src/column-specs.lib.tengo +++ b/workflow/src/column-specs.lib.tengo @@ -30,6 +30,7 @@ text := import("text") // block's emitted columns in the block test. SCORE_ANNOTATION := "pl7.app/isScore" SCORE_FAMILY_PREFIX := "pl7.app/score/" +LEGACY_FEATURE_AXIS := "pl7.app/feature/featureId" guardNoScore := func(annotations) { for key, _ in annotations { @@ -41,6 +42,34 @@ guardNoScore := func(annotations) { return annotations } +/* The legacy per-cell contract keys features by name on pl7.app/feature/featureId, and those columns + stay exactly as they are. What must never happen is a NEW column keying on that same axis while + carrying barcode sequences as its values: the axis identity would be unchanged and its value space + inverted, so no downstream query would fail — joins would simply return wrong rows, which is the + worst way for this to break. Every antigen axis is therefore minted under its own name, and this + asserts it for the axis sets the verdict families are built from. + + A stronger check — that no EMITTED column does this — needs a running workflow, and lives in the + block test. That test is deliberately the last task, so this is the only guard until then. */ +// Split from the assertion below so it can be tested both ways: tengo has no try/catch, so a predicate +// is the only part of a guard a test can actually exercise. +usesLegacyFeatureAxis := func(axesSpec) { + for _, axis in axesSpec { + if axis.spec.name == LEGACY_FEATURE_AXIS { + return true + } + } + return false +} + +guardNoLegacyFeatureAxis := func(axesSpec) { + ll.assert( + !usesLegacyFeatureAxis(axesSpec), + "column-specs: an antigen table keys on %v; antigen axes are minted under their own names so a barcode value space cannot inherit a feature-name axis identity", + LEGACY_FEATURE_AXIS) + return axesSpec +} + // Standard table annotations: order priority + default visibility. // visibility: undefined -> hidden, true -> default, false -> optional. a := func(order, defaultVisibility, spec) { @@ -1016,5 +1045,7 @@ export { panelLabelsImportSpec: panelLabelsImportSpec, samplePanelImportSpec: samplePanelImportSpec, panelMismatchImportSpec: panelMismatchImportSpec, - qcImportSpec: qcImportSpec + qcImportSpec: qcImportSpec, + guardNoLegacyFeatureAxis: guardNoLegacyFeatureAxis, + usesLegacyFeatureAxis: usesLegacyFeatureAxis } diff --git a/workflow/src/verdict-import.tpl.tengo b/workflow/src/verdict-import.tpl.tengo index f194e19..c151a1c 100644 --- a/workflow/src/verdict-import.tpl.tengo +++ b/workflow/src/verdict-import.tpl.tengo @@ -68,7 +68,12 @@ self.body(func(inputs) { // The importer's result map is walked with maps.forEach, whose key order is sorted. A bare // `for k, v in` has no defined order in Tengo, which would make the built frame's resource handle vary // run to run and silently cost every downstream node its deduplication, with nothing reported. + // Every antigen table imported by this block goes through here, which makes it the one place that + // can check them all. The check is that none keys on the legacy feature axis: that axis is keyed by + // feature NAME, and a new column reusing it while carrying barcode sequences would keep the axis + // identity and invert its value space, so joins downstream would return wrong rows rather than fail. addTo := func(fb, name, file, spec) { + columnSpecs.guardNoLegacyFeatureAxis(spec.axes) imported := xsv.importFile(file, "csv", spec, { splitDataAndSpec: true }) maps.forEach(imported, func(key, column) { fb.add(name + "/" + key, trace.inject(column.spec), column.data) From eb285a9b882168030f5a43c85a0d4e9ca621462b Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 20:35:27 +0200 Subject: [PATCH 057/282] MILAB-6496: model parameters for the reading, replacing dominance BlockData and BlockArgs gain the binding reading's own parameters: the optional single-cell V(D)J dataset anchor, the role column and reference values, the reference source, the panel minimum and thin line, the count floor, the bound cutoff, minimum voting cells, minimum agreement, the admissibility gate, the high-reference line, the grouping rule and the contending groups, plus the verdict grid state. Field names are the ones verdict-args.lib.tengo already consumes. The dataset is the block's one optional input: the args lambda never throws on its absence, and the workflow skips the verdict stage alone. The dataset is offered as an ANCHOR, not a linker ref, by the same result-pool query VDJ Multiomic Integration uses; the linker itself is infrastructure and stays unpickable. Grouping is a rule over declared panel properties, never a tag->identity map. Contending groups are canonicalised in the args lambda - each group sorted, groups sorted, groups of fewer than two members dropped - so a reorder in the editor cannot stale the block. Their options come from the new identityOptions output; nothing writes that list back to data. A v2 -> v3 migration drops dominanceThreshold and the off-target designation, whose rules no longer exist, and seeds the new defaults. controlFeature, combineColumn and minUmi are KEPT: main.tpl.tengo still passes them to emit_feature_properties.py and per_cell_metrics.py, which still implement them. --- model/src/index.ts | 171 +++++++++++++++++++++++++++++++++----- model/src/types.ts | 98 ++++++++++++++++++---- test/src/wf.test.ts | 13 ++- ui/src/pages/MainPage.vue | 78 +---------------- 4 files changed, 248 insertions(+), 112 deletions(-) diff --git a/model/src/index.ts b/model/src/index.ts index 70bc7f9..5c07e4b 100644 --- a/model/src/index.ts +++ b/model/src/index.ts @@ -15,9 +15,19 @@ export { assemblePattern, parsePattern, validatePattern } from "./pattern"; export type { PatternParts } from "./pattern"; export { allPresets, getPreset } from "./presets"; export type { Preset } from "./presets"; -export type { BlockArgs, BlockData } from "./types"; +export type { BlockArgs, BlockData, GroupingRule, ReferenceSource } from "./types"; -const DOMINANCE_FLOOR = 0.5; // threshold is user-adjustable down to 0.5, never lower +// The reading's shipped defaults. They restate the Python's own (verdict.py DEFAULT_FLOOR, +// BOUND_CUTOFF, DEFAULT_PANEL_MIN_MEMBERS, DEFAULT_REFERENCE_THIN_LINE, +// DEFAULT_HIGH_REFERENCE_OBSERVATION_LINE, combine.py DEFAULT_MIN_VOTERS) so the value that produced a +// run is a value the user can see and change, not an argparse default nobody chose. Every one of them +// is a declared default rather than a calibrated line: nothing published sets any of them. +const DEFAULT_COUNT_FLOOR = 4; +const DEFAULT_BOUND_CUTOFF = 75; +const DEFAULT_MIN_VOTING_CELLS = 1; +const DEFAULT_PANEL_REFERENCE_MIN_MEMBERS = 8; +const DEFAULT_REFERENCE_THIN_LINE = 2; +const DEFAULT_HIGH_REFERENCE_LINE = 100; // Ordinal step key -> the step a sample is CURRENTLY on once that report has settled. A stepReports entry // appears when its step finishes, so the furthest-present report implies the next running step @@ -199,9 +209,35 @@ function suggestSampleColumn(ctx: BlockRenderCtx): string return best?.col; } +// v2 data shape: the preset selector + pattern string, with the dominance-era parameters still on it. +// The dominant-feature readout, the off-target designation and the specificity score they fed are gone +// from per_cell_metrics.py, so nothing consumes these three any more. +type BlockDataV2 = Omit< + BlockData, + | "datasetRef" + | "roleColumn" + | "referenceValues" + | "referenceSource" + | "panelReferenceMinMembers" + | "referenceThinLine" + | "countFloor" + | "boundCutoff" + | "minVotingCells" + | "minAgreement" + | "gateThreshold" + | "highReferenceLine" + | "grouping" + | "contendingGroups" + | "verdictTableState" +> & { + dominanceThreshold: number; + offtargetProperty?: string; + offtargetValues?: string[]; +}; + // v1 (pre-preset) data shape: read geometry was three explicit length fields. v2 replaces them with a // preset selector + a mitool tag-pattern string (see model/src/pattern.ts, model/src/presets). -type BlockDataV1 = Omit & { +type BlockDataV1 = Omit & { cellLen: number; umiLen: number; featureLen: number; @@ -209,7 +245,7 @@ type BlockDataV1 = Omit & { const dataModel = new DataModelBuilder() .from("v1") - .migrate("v2", ({ cellLen, umiLen, featureLen, ...rest }) => { + .migrate("v2", ({ cellLen, umiLen, featureLen, ...rest }) => { // The shipped default (16/10/15) maps to the fixed BEAM preset; any other geometry maps to the // generic preset carrying the assembled pattern (offset 0 — the only layout the v1 UI could express). const isBeamDefault = cellLen === 16 && umiLen === 10 && featureLen === 15; @@ -227,15 +263,42 @@ const dataModel = new DataModelBuilder() }), }; }) + // v2 -> v3: the dominance parameters go and the reading's own arrive. The three dropped fields are + // dropped rather than carried: the rule they parameterised no longer exists, and a field kept "just in + // case" would still travel in the args hash and stale the block on an edit that changes no computation. + // The new numeric parameters are seeded with the shipped defaults so a migrated project renders the + // same run a fresh one would — a parameter left undefined here would reach the CLI as its argparse + // default, which is the same number arrived at without anyone choosing it. + .migrate( + "v3", + ({ dominanceThreshold: _d, offtargetProperty: _p, offtargetValues: _v, ...rest }) => ({ + ...rest, + countFloor: DEFAULT_COUNT_FLOOR, + boundCutoff: DEFAULT_BOUND_CUTOFF, + minVotingCells: DEFAULT_MIN_VOTING_CELLS, + panelReferenceMinMembers: DEFAULT_PANEL_REFERENCE_MIN_MEMBERS, + referenceThinLine: DEFAULT_REFERENCE_THIN_LINE, + highReferenceLine: DEFAULT_HIGH_REFERENCE_LINE, + verdictTableState: createPlDataTableStateV2(), + }), + ) .init(() => ({ - dominanceThreshold: 0.6, runMode: "full" as const, // full run by default; "dry" = read-limited Preview // Default preset = the geometry the block shipped with: 10x 5' v2 BEAM (16 / 10 / 15). presetId: "tenx-beam", cellWhitelist: "", // de-novo CELL correction by default defaultBlockLabel: "", + // The reading's parameters. minAgreement and gateThreshold are deliberately absent: both are off by + // default, and off means absent rather than zero (see the args projection). + countFloor: DEFAULT_COUNT_FLOOR, + boundCutoff: DEFAULT_BOUND_CUTOFF, + minVotingCells: DEFAULT_MIN_VOTING_CELLS, + panelReferenceMinMembers: DEFAULT_PANEL_REFERENCE_MIN_MEMBERS, + referenceThinLine: DEFAULT_REFERENCE_THIN_LINE, + highReferenceLine: DEFAULT_HIGH_REFERENCE_LINE, tableState: createPlDataTableStateV2(), qcSummaryTableState: createPlDataTableStateV2(), + verdictTableState: createPlDataTableStateV2(), })); export const platforma = BlockModelV3.create(dataModel) @@ -312,14 +375,35 @@ export const platforma = BlockModelV3.create(dataModel) ); } + // The reading's own parameters. The single-cell V(D)J dataset is deliberately NOT required: without + // it the block still emits the tag counts, the per-cell scalars, the panel-versus-reads check and the + // per-sample QC, none of which need a clonotype set. A missing input narrows what can be answered + // and nothing more. + if (data.countFloor < 0) throw new Error("The count floor cannot be negative"); + if (data.boundCutoff < 0 || data.boundCutoff > 100) + throw new Error("The bound cutoff is a score between 0 and 100"); + if (data.minVotingCells < 1) throw new Error("At least one cell must vote"); + // "declared" reads counts against a tag the panel marks as the comparator, and nothing marks one + // without the role values. Asking for it anyway would degrade to no comparator inside the run, where + // the choice is recorded but the user never sees they lost it. + if (data.referenceSource === "declared" && !data.referenceValues?.length) + throw new Error("Choose which role values mark the reference, or pick another source"); + + // Contending groups, canonicalised here rather than in the editor: the args value is a cache key, so + // the same declaration written in a different order must produce the same string or the block goes + // stale and re-runs the whole reading for nothing. A group of fewer than two members is dropped — + // one identity contends with nothing, and an empty group is that same case. + const contendingGroups = (data.contendingGroups ?? []) + .map((group) => [...new Set(group)].sort()) + .filter((group) => group.length > 1) + .sort((a, b) => a.join(" ").localeCompare(b.join(" "))); + return { fbFastqRef: data.fbFastqRef, tagFeatureCsvHandle: data.tagFeatureCsvHandle, barcodeSeqColumn: data.barcodeSeqColumn, featureNameColumn: data.featureNameColumn, controlFeature: data.controlFeature, - // canonicalize + clamp to the 0.5 floor - dominanceThreshold: Math.max(DOMINANCE_FLOOR, data.dominanceThreshold ?? 0.6), // Optional multi-barcode antigen combine mode. combineColumn names a tag-CSV column giving each // feature's mode (sum = OR, the default; all = AND, feature called only when every member barcode // fires). Projected only when set so the workflow default (every feature OR) is untouched otherwise. @@ -330,19 +414,37 @@ export const platforma = BlockModelV3.create(dataModel) ...(data.combineColumn && typeof data.minUmi === "number" && data.minUmi >= 1 ? { minUmi: Math.round(data.minUmi) } : {}), - // Optional off-target designation (F2). offtargetProperty names an imported per-feature property - // column (e.g. antigen_class); offtargetValues are that column's values marking a feature as - // off-target. Such features are excluded from the dominant call (like the control) and turn on the - // cross-reactive label. Projected only when both are set, so the dominant call is unchanged - // otherwise (empty column / values → workflow leaves the rule untouched). - ...(data.offtargetProperty && data.offtargetValues && data.offtargetValues.length > 0 - ? { - offtargetProperty: data.offtargetProperty, - // Sort + dedup: the Python treats these as a set, so canonicalize here so re-selecting the - // same values in a different order yields the same args hash (no needless stale / re-run). - offtargetValues: [...new Set(data.offtargetValues)].sort(), - } - : {}), + // --- the binding reading --- + // The dataset anchor. Absent is a legitimate state, not a half-filled form, so it projects as + // absent and the workflow skips the verdict stage alone. + datasetRef: data.datasetRef, + // Empty and absent are the same claim for both of these, so an empty selection projects as absent + // rather than as "" / [] — two spellings of one request would otherwise be two cache keys. + roleColumn: data.roleColumn || undefined, + // Sorted + de-duplicated: the Python reads these as a set, so re-picking the same values in a + // different order must not re-run the reading. + referenceValues: data.referenceValues?.length + ? [...new Set(data.referenceValues)].sort() + : undefined, + referenceSource: data.referenceSource, + panelReferenceMinMembers: Math.round(data.panelReferenceMinMembers), + referenceThinLine: Math.round(data.referenceThinLine), + countFloor: Math.round(data.countFloor), + boundCutoff: data.boundCutoff, + minVotingCells: Math.round(data.minVotingCells), + // Off by default, and off means ABSENT: a minimum agreement of 0 passes every majority instead of + // skipping the check, and a gate of 0 sets aside every cell instead of gating none. Both are + // different claims from "off", so neither is projected as zero. + minAgreement: data.minAgreement, + gateThreshold: + typeof data.gateThreshold === "number" && data.gateThreshold > 0 + ? Math.round(data.gateThreshold) + : undefined, + highReferenceLine: Math.round(data.highReferenceLine), + // A rule over declared panel properties, never a tag→identity map. Absent means one identity per + // tag, which is the reading's own default, so no hand-built { by: "tag" } is sent in its place. + grouping: data.grouping, + contendingGroups: contendingGroups.length > 0 ? contendingGroups : undefined, // Preview: cap reads only in dry mode; a full run omits it (all reads). Projected only when dry, so // toggling back to full changes the args hash and re-runs on the complete input. ...(data.runMode === "dry" && data.limitInput @@ -395,6 +497,35 @@ export const platforma = BlockModelV3.create(dataModel) ); }), ) + // The single-cell V(D)J dataset the verdicts are keyed by: columns on [sampleId, scClonotypeKey] + // flagged as anchors — the same query VDJ Multiomic Integration uses, so the two blocks offer the user + // the same list. There is deliberately no linkerOptions beside it: the cell linker carries + // pl7.app/isLinkerColumn and is hidden in tables, so it is not a column a user can pick, and the + // workflow resolves it from this anchor by name. + .output("datasetOptions", (ctx) => + ctx.resultPool.getOptions([ + { + axes: [{ name: "pl7.app/sampleId" }, { name: "pl7.app/vdj/scClonotypeKey" }], + annotations: { "pl7.app/isAnchor": "true" }, + }, + ]), + ) + // The identities the contending-groups editor picks from, live from the uploaded panel. An identity is + // whatever the grouping rule groups tags by: the tag itself under the default per-tag rule, and the + // property's value under a property rule — so the option list is the distinct values of the barcode + // column or of the chosen property column. Under the per-tag rule the ids ARE the barcode sequences, + // and they are their own labels: the panel metadata is column-wise (each column's distinct values), so + // it carries no tag→name pairing to name them by. Retentive so the editor does not blank on a rerun. + // + // This output exists so that only the USER'S PICKS are ever written to data. A watcher copying this + // list into data would make the output depend on data derived from it, and two open clients would race + // to write it. + .retentiveOutput("identityOptions", (ctx): { value: string; label: string }[] => { + const grouping = ctx.data.grouping; + const column = grouping?.by === "property" ? grouping.column : ctx.data.barcodeSeqColumn; + if (!column) return []; + return (readCsvMeta(ctx)?.valuesByColumn?.[column] ?? []).map((v) => ({ value: v, label: v })); + }) // Suggested block label for the sidebar subtitle: " / - ", derived from // the current inputs. Computed here (not in .subtitle) because the subtitle context has no result // pool; a UI watchEffect copies this into data.defaultBlockLabel. Each part is dropped until set. diff --git a/model/src/types.ts b/model/src/types.ts index 3a01e09..e455ee8 100644 --- a/model/src/types.ts +++ b/model/src/types.ts @@ -1,13 +1,32 @@ import type { ImportFileHandle, PlDataTableStateV2, PlRef } from "@platforma-sdk/model"; +/** + * Which comparator a count is read against. Selected, never inferred: two runs answered by different + * rules produce numbers that do not compare, and a scientist who did not choose the rule cannot know + * that happened. Undefined means the default for this panel — a declared reference tag where one + * exists, and otherwise no comparator at all. + */ +export type ReferenceSource = "declared" | "panel" | "none"; + +/** + * How tags become identities. A RULE over declared properties, never a tag->identity map: a map is + * keyed by tags, which are known only after the block runs, so any editor for it writes an output + * back into data. A property column name is knowable at prerun, from the panel header the block + * already enumerates. Absent means one identity per tag. + */ +export type GroupingRule = { by: "tag" } | { by: "property"; column: string }; + /** Workflow inputs (projected from BlockData by the args lambda; validated there). */ export type BlockArgs = { fbFastqRef: PlRef; // feature-barcode FASTQ column (from samples-and-data, result pool) tagFeatureCsvHandle: ImportFileHandle; // tag->feature CSV, user-uploaded barcodeSeqColumn: string; // CSV column holding the feature barcode (whitelist/panel) featureNameColumn: string; // CSV column holding the feature/antigen name - controlFeature?: string; // negative-control feature name; omitted -> no score - dominanceThreshold: number; // default 0.6, floor 0.5 + // Negative-control feature name. It no longer gates any per-cell rule — the verdict asks the binding + // question of every antigen independently — but main.tpl.tengo still passes it to + // emit_feature_properties.py as --control-feature, which emits the pl7.app/feature/negativeControl + // marker column consumers read. Omitted -> that marker is header-only. + controlFeature?: string; pattern: string; // Mitool tag pattern // mitool tag names baked into `pattern` tags: { cell: string; umi: string; feature: string }; @@ -31,12 +50,39 @@ export type BlockArgs = { // fires). minUmi is the AND per-barcode "fired" floor (integer >= 1; workflow default 1). combineColumn?: string; minUmi?: number; - // Optional off-target designation (F2). offtargetProperty names an imported per-feature property column - // (e.g. antigen_class); offtargetValues are that column's values marking a feature as off-target. Such - // features are excluded from the dominant call (like the control) and enable the "cross-reactive" label. - // Both present -> off-target-aware; omitted -> unchanged dominant call. - offtargetProperty?: string; - offtargetValues?: string[]; + + // --- the binding reading ------------------------------------------------------------------------- + // Everything below reaches emit_verdicts.py through verdict-args.lib.tengo, and nothing below reaches + // the per-sample mitool fan-out: a change to how the counts are READ recovers every per-sample body + // from cache and re-runs the verdict stage alone. + + // The single-cell V(D)J dataset ANCHOR (axes [pl7.app/sampleId, pl7.app/vdj/scClonotypeKey], + // pl7.app/isAnchor). Not a linker ref: the cell linker carries pl7.app/isLinkerColumn and is hidden in + // tables, so it is not a column a user can pick, and the workflow resolves it from this anchor by name. + // Because the anchor is receptor-scoped, choosing the dataset is choosing the receptor — which is what + // lets a BCR + TCR run bring two linkers without a panic. Optional: without it the block still emits + // every column not keyed by a clonotype set, and only the verdict stage is skipped. + datasetRef?: PlRef; + // The panel column declaring each tag's role, and the values of it that mark a tag as the comparator. + roleColumn?: string; + referenceValues?: string[]; + referenceSource?: ReferenceSource; + panelReferenceMinMembers: number; // members the panel needs before its own readings can serve + referenceThinLine: number; // below this the comparator rests on too little to compare against + countFloor: number; // counts below this are not evidence of binding + boundCutoff: number; // specificity score (0-100) at or above which a cell binds + minVotingCells: number; // a verdict may rest on one cell and say so + // Share (0-1) of answering cells the majority must reach. Off by default, and off means ABSENT rather + // than zero: a floor of 0 makes every majority pass the check instead of skipping the check. + minAgreement?: number; + // The admissibility gate, in comparator UMIs. Undefined means off; zero would set aside every cell, + // so the args lambda projects it only when positive. + gateThreshold?: number; + highReferenceLine: number; // where a reference reading counts as high, with the gate off + grouping?: GroupingRule; + // Identities declared to contend for one binding site. Canonicalised by the args lambda (each group + // sorted, groups sorted, groups of fewer than two members dropped). + contendingGroups?: string[][]; }; /** Unified persisted UI state. */ @@ -51,7 +97,6 @@ export type BlockData = { // Distinct values of the chosen sample column at pick time — snapshotted alongside the label map so // args() can gate Run purely from data (block when a dataset sample has no rows in the CSV). sampleColumnValues?: string[]; - dominanceThreshold: number; // Preview (dry-run) mode. "full" (default) processes all reads; "dry" caps mitool parse to `limitInput` // reads per sample so the user can check settings first. Mirrors mixcr-clonotyping / demultiplex-fastq. runMode?: "dry" | "full"; @@ -61,11 +106,36 @@ export type BlockData = { // barcode fires). minUmi is the AND per-barcode "fired" floor (integer >= 1; workflow default 1). combineColumn?: string; minUmi?: number; - // Optional off-target designation (F2). offtargetProperty names an imported per-feature property column - // (e.g. antigen_class); offtargetValues are that column's values marking a feature as off-target. Both - // present -> the dominant call excludes those features and enables the "cross-reactive" label. - offtargetProperty?: string; - offtargetValues?: string[]; + + // --- the binding reading ------------------------------------------------------------------------- + // See BlockArgs for what each one means to the reading; the notes here are about the DATA layer only. + + /** + * The single-cell V(D)J dataset anchor, and the block's one optional input. A missing dataset narrows + * what the block can answer — no clonotype set means no verdict — and stops nothing: the args lambda + * never throws on its absence. + */ + datasetRef?: PlRef; + roleColumn?: string; + referenceValues?: string[]; + referenceSource?: ReferenceSource; + panelReferenceMinMembers: number; + referenceThinLine: number; + countFloor: number; + boundCutoff: number; + minVotingCells: number; + minAgreement?: number; + gateThreshold?: number; + highReferenceLine: number; + grouping?: GroupingRule; + /** + * Written on a user gesture only. The identities to choose from come from the identityOptions model + * output, and a watcher that copied that output into data would make the model output depend on the + * data it feeds — a write-on-read loop, and a write race between two open clients. + */ + contendingGroups?: string[][]; + verdictTableState: PlDataTableStateV2; // verdict grid state (UI-only, never projected to args) + presetId?: string; pattern?: string; cellWhitelist?: string; // optional (defaults to "" = de-novo); see BlockArgs.cellWhitelist diff --git a/test/src/wf.test.ts b/test/src/wf.test.ts index 55961b7..f68f207 100644 --- a/test/src/wf.test.ts +++ b/test/src/wf.test.ts @@ -119,16 +119,25 @@ blockTest.skip( const csvHandle = await helpers.getLocalFileHandle("./assets/tags.csv"); // Configure the block. update-block-data must carry EVERY BlockArgsValid field, else the backend - // reports "currentArgs not set". controlFeature is optional (no specificity score here). + // reports "currentArgs not set". controlFeature is optional (no negative-control marker here), and + // so is datasetRef — with no single-cell V(D)J dataset the block skips the verdict stage and still + // emits everything this test reads. The reading's numeric parameters are required and carry the + // shipped defaults, the same values a freshly created block starts with. await project.mutateBlockStorage(fiBlockId, { operation: "update-block-data", value: { fbFastqRef: fiOutputs1.fastqOptions[0].ref, tagFeatureCsvHandle: csvHandle, - dominanceThreshold: 0.6, presetId: "tenx-beam", // 10x 5' v2 BEAM geometry (16/10/15); pattern owned by the preset + countFloor: 4, + boundCutoff: 75, + minVotingCells: 1, + panelReferenceMinMembers: 8, + referenceThinLine: 2, + highReferenceLine: 100, tableState: createPlDataTableStateV2(), qcSummaryTableState: createPlDataTableStateV2(), + verdictTableState: createPlDataTableStateV2(), } satisfies BlockData, }); diff --git a/ui/src/pages/MainPage.vue b/ui/src/pages/MainPage.vue index 7a68cbb..5c483ef 100644 --- a/ui/src/pages/MainPage.vue +++ b/ui/src/pages/MainPage.vue @@ -12,7 +12,6 @@ import { PlBtnGhost, PlBtnGroup, PlDropdown, - PlDropdownMulti, PlDropdownRef, PlFileInput, PlLogView, @@ -126,31 +125,6 @@ const combineColumnError = computed(() => { return undefined; }); -// Off-target designation (F2). The property dropdown offers imported per-feature property columns — -// csvColumnOptions minus the columns already bound to the barcode/feature/sample roles (those aren't -// per-feature properties). The values multi-select lists the chosen property's distinct values (from -// csvValuesByColumn). Features whose property value is selected are excluded from the dominant call -// (like the control) and enable the "cross-reactive" label. -const offtargetPropertyOptions = computed(() => - (app.model.outputs.csvColumnOptions ?? []).filter( - (o) => - o.value !== app.model.data.barcodeSeqColumn && - o.value !== app.model.data.featureNameColumn && - o.value !== app.model.data.sampleColumn, - ), -); -const offtargetValueOptions = computed(() => { - const prop = app.model.data.offtargetProperty; - if (!prop) return []; - return (app.model.outputs.csvValuesByColumn?.[prop] ?? []).map((v) => ({ value: v, label: v })); -}); -// Changing the property invalidates the selected values (they belong to the previous column), so clear -// them on that explicit gesture — same data→data pattern as clearControlOnInputChange. -function setOfftargetProperty(prop: string | undefined) { - app.model.data.offtargetProperty = prop; - app.model.data.offtargetValues = undefined; -} - // Run mode: read-limited Preview (dry run) vs full run — same PlBtnGroup pattern as mixcr-clonotyping / // demultiplex-fastq (Preview first). Feature-barcode is single-cell + shallow per cell, so the dry-run // default matches mixcr's single-cell recommendation (500k reads/sample). @@ -201,16 +175,12 @@ function clearSampleAwareOnInputChange() { } // CSV swap invalidates every CSV-derived selection: the barcode / feature-name columns (the new file's -// headers differ), the negative control, the off-target designation, and the sample-aware selection -// (columns/values change). Clear them all so the user re-picks against the new CSV. +// headers differ), the negative control, and the sample-aware selection (columns/values change). Clear +// them all so the user re-picks against the new CSV. function clearOnCsvChange() { app.model.data.barcodeSeqColumn = undefined; app.model.data.featureNameColumn = undefined; app.model.data.combineColumn = undefined; - // Off-target property/values name columns + values of the OLD CSV; a new CSV may not have them, and - // args() projects them unconditionally once set (no column-existence guard), so clear them here. - app.model.data.offtargetProperty = undefined; - app.model.data.offtargetValues = undefined; clearControlOnInputChange(); clearSampleAwareOnInputChange(); } @@ -483,37 +453,6 @@ const gridOptions = { - - - - - - - @@ -544,19 +483,6 @@ const gridOptions = { - - - Date: Mon, 17 Aug 2026 20:54:55 +0200 Subject: [PATCH 058/282] MILAB-6496: verdict view and settings for the reading MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Binding verdicts page: one row per (clonotype set, antigen identity), carrying the four-state verdict, why an unsettled one is unsettled, and the cells behind it. Filtering and default visibility come from the column specs the workflow already builds — the states and the competed flag are discrete filters, and nothing in the family is orderable. Three states the page names rather than leaves the reader to infer: no V(D)J dataset (the run skipped the verdict stage entirely, so no antigen columns were produced at all), no comparator (every reading unreliable, with the setting that changes it), and a comparator request the panel could not serve. The verdicts frame is surfaced as a block output as well as an export, because a block's own exports are not in its own result pool: without it the block that produced the verdicts is the one place that cannot show them. The identity label column travels in the table's column list for the same reason, and is what puts the antigen's name in the row where the identity id is a bare barcode. The reading's settings live in one component mounted in both Settings drawers, so the rule that produced a table can be changed from the table. Picking the role or grouping column snapshots the panel's headers into data, so args can refuse a column the panel no longer carries — emit_verdicts.py ends the whole run over one, and the user meets that as a dead run. A panel swap now clears every verdict setting that names a panel column or value. Also: the sample-column dropdown no longer offers columns already bound to the barcode or feature roles, the dead v-if="false" combine-mode markup is gone (its validation alert stays, since a migrated project can still carry the value), and the two settings strings promising a specificity score are rewritten to say what the negative control actually does. --- model/src/index.ts | 153 ++++++++++++- model/src/types.ts | 8 + ui/src/app.ts | 2 + ui/src/components/VerdictSettings.vue | 314 ++++++++++++++++++++++++++ ui/src/pages/MainPage.vue | 77 ++++--- ui/src/pages/VerdictsPage.vue | 100 ++++++++ workflow/src/main.tpl.tengo | 4 + 7 files changed, 618 insertions(+), 40 deletions(-) create mode 100644 ui/src/components/VerdictSettings.vue create mode 100644 ui/src/pages/VerdictsPage.vue diff --git a/model/src/index.ts b/model/src/index.ts index 5c07e4b..e9ab26a 100644 --- a/model/src/index.ts +++ b/model/src/index.ts @@ -9,7 +9,7 @@ import { } from "@platforma-sdk/model"; import { assemblePattern, CELL_TAG, FEATURE_TAG, UMI_TAG, validatePattern } from "./pattern"; import { getPreset } from "./presets"; -import type { BlockArgs, BlockData } from "./types"; +import type { BlockArgs, BlockData, ReferenceSource } from "./types"; export { assemblePattern, parsePattern, validatePattern } from "./pattern"; export type { PatternParts } from "./pattern"; @@ -29,6 +29,41 @@ const DEFAULT_PANEL_REFERENCE_MIN_MEMBERS = 8; const DEFAULT_REFERENCE_THIN_LINE = 2; const DEFAULT_HIGH_REFERENCE_LINE = 100; +// The two axes the verdict view is assembled from. The exported verdict frame carries every table the +// reading emitted — per-cell counts keyed by cell, the offered scope keyed by sample, the tag→identity +// linker keyed by tag — and any of them joined into this view fans one verdict row into many. So the view +// is pinned to its own key: what a verdict is ABOUT (the identity) and who it is about it FOR (the +// clonotype set). The set axis name is the one the datasetOptions query requires of an anchor, so a +// dataset that could be picked here always carries it. +const IDENTITY_AXIS = "pl7.app/antigen/identityId"; +const CLONOTYPE_SET_AXIS = "pl7.app/vdj/scClonotypeKey"; + +// The run record emit_verdicts.py writes (result_run_meta.json), read as content. Only the fields the UI +// states back to the user are typed here; the file carries every parameter the reading used. +export type VerdictRunMeta = { + /** The comparator that actually SERVED — a request the panel cannot honour degrades to none. */ + referenceChoice: string; + /** The comparator that was ASKED for, so a degraded run can say what it lost. */ + referenceSourceRequested: string; + referenceTags: string[]; + identityCount: number; + setCount: number; + cellsAnalysed: number; + /** Tags the grouping column said nothing about; each stands as its own identity, under a bare barcode. */ + tagsWithoutGroupingValue: string[]; +}; + +// What the software resolves an unset reference source to, restated so the dropdown can say it. Mirrors +// verdict.py resolve_default_source: a declared reagent, else the panel's own readings where the panel is +// big enough, else nothing. +export type ReferenceSourceChoices = { + options: { value: ReferenceSource; label: string; description: string }[]; + /** One line per source this panel cannot serve, saying why. */ + unavailable: string[]; + /** What an unset source resolves to for this panel, as a sentence. */ + fallback: string; +}; + // Ordinal step key -> the step a sample is CURRENTLY on once that report has settled. A stepReports entry // appears when its step finishes, so the furthest-present report implies the next running step export type SampleStep = "parsing" | "refining" | "counting" | "metrics"; @@ -388,6 +423,23 @@ export const platforma = BlockModelV3.create(dataModel) // the choice is recorded but the user never sees they lost it. if (data.referenceSource === "declared" && !data.referenceValues?.length) throw new Error("Choose which role values mark the reference, or pick another source"); + // A role column or a grouping column the panel does not carry ends the whole run at the exec + // (emit_verdicts.py exits rather than degrading), and the user meets that as a dead run with no hint + // of which setting caused it. The check is against the headers snapshotted when the column was picked + // — args reads data only — so a panel swap that leaves the pick behind disables Run with a message + // naming the column instead. + const panelColumns = data.panelColumnSnapshot; + if (panelColumns?.length) { + for (const [role, column] of [ + ["Reference role", data.roleColumn], + ["Grouping", data.grouping?.by === "property" ? data.grouping.column : undefined], + ] as const) { + if (column && !panelColumns.includes(column)) + throw new Error( + `${role} column "${column}" is not a column of the uploaded panel file; re-select it`, + ); + } + } // Contending groups, canonicalised here rather than in the editor: the args value is a cache key, so // the same declaration written in a different order must produce the same string or the block goes @@ -877,6 +929,101 @@ export const platforma = BlockModelV3.create(dataModel) }, { retentive: true, withStatus: true }, ) + // The verdict view: one row per (clonotype set, antigen identity), carrying the four-state verdict and + // the support behind it. Assembled from the exported verdict frame, which the workflow also surfaces as + // an output because a block's own exports are not in its own result pool. + // + // The identity LABEL column travels in the columns list rather than being discovered: createPlDataTable + // looks for label columns in the result pool, and this block's are not there. It is what puts the + // antigen's readable name in the row — emit_verdicts.py writes the panel's feature name, the tag itself + // where the panel names none, and "name (tag)" where two tags would otherwise share one label. + // + // createPlDataTableV2 rather than V3 for the same reason as perCellTable above: V3's discovery walks the + // whole result pool and hangs on the upstream Samples&Data File dataset. V2 takes the columns as given. + // Filtering, ordering and default visibility all come from the specs the workflow built — the four + // states and wasCompeted carry pl7.app/isDiscreteFilter, and no column carries an orderable annotation, + // which column-specs.lib.tengo enforces rather than assumes. + .output( + "verdictTable", + (ctx) => { + const pCols = ctx.outputs + ?.resolve({ field: "antigenVerdictsTable", allowPermanentAbsence: true }) + ?.getPColumns(); + if (pCols === undefined) return undefined; + const cols = pCols.filter((c) => { + const axes = c.spec.axesSpec.map((a) => a.name); + if (c.spec.name === "pl7.app/label") return axes.length === 1 && axes[0] === IDENTITY_AXIS; + return axes.length === 2 && axes[0] === CLONOTYPE_SET_AXIS && axes[1] === IDENTITY_AXIS; + }); + if (cols.length === 0) return undefined; + return createPlDataTableV2(ctx, cols, ctx.data.verdictTableState); + }, + { retentive: true, withStatus: true }, + ) + // What the reading was actually answered under. The page states the comparator that SERVED rather than + // the one that was requested, because the software degrades a request it cannot honour and a reader + // meeting an all-unreliable table otherwise has no way to learn that happened. Absent until a run with a + // V(D)J dataset has produced it. + .output("verdictRunMeta", (ctx): VerdictRunMeta | undefined => + ctx.outputs + ?.resolve({ field: "antigenRunMeta", allowPermanentAbsence: true }) + ?.getDataAsJsonOrUndefined(), + ) + // The comparator sources this panel can serve, with a line for each it cannot. Both facts are knowable + // before a run, from the panel metadata staging already emits: the panel's size is the count of distinct + // barcodes, and a declared comparator needs a role column and values of it that the column actually + // carries. Offering a source the run would silently degrade would record a choice the user never gets. + .retentiveOutput("referenceSources", (ctx): ReferenceSourceChoices => { + const meta = readCsvMeta(ctx); + const barcodeColumn = ctx.data.barcodeSeqColumn; + const panelSize = barcodeColumn + ? (meta?.valuesByColumn?.[barcodeColumn] ?? []).filter((v) => v.trim() !== "").length + : 0; + const minMembers = Math.round(ctx.data.panelReferenceMinMembers); + const roleColumn = ctx.data.roleColumn; + const roleValues = new Set(roleColumn ? (meta?.valuesByColumn?.[roleColumn] ?? []) : []); + const declaredTags = (ctx.data.referenceValues ?? []).filter((v) => roleValues.has(v)); + + const options: ReferenceSourceChoices["options"] = []; + const unavailable: string[] = []; + if (declaredTags.length > 0) + options.push({ + value: "declared", + label: "Declared reference tag", + description: "Counts are read against the tags the panel marks as the comparator.", + }); + else + unavailable.push( + "Declared reference tag — no tag is marked as the comparator yet. Choose the panel column " + + "that declares each tag's role, then the values of it that mark the comparator.", + ); + if (panelSize >= minMembers) + options.push({ + value: "panel", + label: "The panel's own readings", + description: `Counts are read against the rest of the panel (${panelSize} tags).`, + }); + else + unavailable.push( + `The panel's own readings — the panel declares ${panelSize} tag(s) and this source needs at ` + + `least ${minMembers}. Lower the panel minimum in Advanced Settings, or use a declared reference tag.`, + ); + options.push({ + value: "none", + label: "No comparator", + description: + "Every reading is left unreliable rather than compared against something that cannot serve.", + }); + + // The three-rung default, restated from verdict.py resolve_default_source. + const fallback = + declaredTags.length > 0 + ? "the declared reference tags" + : panelSize >= minMembers + ? "the panel's own readings" + : "no comparator — every reading would be unreliable"; + return { options, unavailable, fallback }; + }) .title(() => "Feature Barcode Profiling") // Standard block-label subtitle. The subtitle render context is args-only (no result pool / outputs // — touching them renders "Invalid subtitle"), so the dynamic " / - " @@ -896,6 +1043,10 @@ export const platforma = BlockModelV3.create(dataModel) ? [ { type: "link" as const, href: "/qc" as const, label: "Per-sample QC" }, { type: "link" as const, href: "/results" as const, label: "Per-cell results" }, + // Shown for every run, including one with no V(D)J dataset. That run produces no antigen + // columns at all, and the page saying so is the only place a user learns why — hiding the + // tab would leave the absence unexplained. + { type: "link" as const, href: "/verdicts" as const, label: "Binding verdicts" }, ] : []), ]; diff --git a/model/src/types.ts b/model/src/types.ts index e455ee8..2695cc3 100644 --- a/model/src/types.ts +++ b/model/src/types.ts @@ -117,6 +117,14 @@ export type BlockData = { */ datasetRef?: PlRef; roleColumn?: string; + /** + * The panel's headers as they stood when the role column or the grouping column was picked. Both of + * those name a panel column, and emit_verdicts.py exits the whole run when the panel does not carry the + * one it was given — a failure the user meets as a dead run rather than as a message about the setting + * that caused it. args() validates from data alone, so the headers have to BE in data; they are + * snapshotted on the pick gesture, exactly as sampleColumnValues is. + */ + panelColumnSnapshot?: string[]; referenceValues?: string[]; referenceSource?: ReferenceSource; panelReferenceMinMembers: number; diff --git a/ui/src/app.ts b/ui/src/app.ts index b3b6a76..09577ac 100644 --- a/ui/src/app.ts +++ b/ui/src/app.ts @@ -4,6 +4,7 @@ import { watchEffect } from "vue"; import MainPage from "./pages/MainPage.vue"; import QcSummaryPage from "./pages/QcSummaryPage.vue"; import ResultsPage from "./pages/ResultsPage.vue"; +import VerdictsPage from "./pages/VerdictsPage.vue"; export const sdkPlugin = defineAppV3(platforma, (app) => { // Block-label pattern: mirror the model's suggestedBlockLabel (" / → ") @@ -30,6 +31,7 @@ export const sdkPlugin = defineAppV3(platforma, (app) => { "/": () => MainPage, "/qc": () => QcSummaryPage, "/results": () => ResultsPage, + "/verdicts": () => VerdictsPage, }, }; }); diff --git a/ui/src/components/VerdictSettings.vue b/ui/src/components/VerdictSettings.vue new file mode 100644 index 0000000..9b0260e --- /dev/null +++ b/ui/src/components/VerdictSettings.vue @@ -0,0 +1,314 @@ + + + diff --git a/ui/src/pages/MainPage.vue b/ui/src/pages/MainPage.vue index 5c483ef..90a7732 100644 --- a/ui/src/pages/MainPage.vue +++ b/ui/src/pages/MainPage.vue @@ -30,6 +30,7 @@ import { AgGridVue } from "ag-grid-vue3"; import { computed, ref, watch } from "vue"; import { useApp } from "../app"; import PatternEditor from "../components/PatternEditor.vue"; +import VerdictSettings from "../components/VerdictSettings.vue"; import SampleReportPanel from "./SampleReportPanel.vue"; import { sampleResults, @@ -97,11 +98,11 @@ const tagMappingDisabled = computed( () => !app.model.data.tagFeatureCsvHandle || csvProcessing.value, ); -// Combine-mode column options exclude the columns already bound to the barcode/feature roles: those hold -// DNA barcodes / feature names, not per-feature modes ("sum"/"all"), so offering them only invites a -// mis-pick. The model's args() also rejects such a collision (belt-and-suspenders), but filtering the -// dropdown prevents the mistake up front. -const combineColumnOptions = computed(() => +// CSV columns not already bound to the barcode-sequence or feature-name roles. A column holding DNA +// barcodes or antigen names is not a sample column, and offering it only invites a mis-pick: the data +// layer refuses two roles on one column, but it refuses it at the end of the run. The model's args() also +// rejects the collision; filtering here prevents the mistake up front. +const roleFreeColumnOptions = computed(() => (app.model.outputs.csvColumnOptions ?? []).filter( (o) => o.value !== app.model.data.barcodeSeqColumn && o.value !== app.model.data.featureNameColumn, @@ -110,8 +111,9 @@ const combineColumnOptions = computed(() => // Visible reason when the Combine-mode column is invalid, so a disabled Run button is explained rather // than mysterious. The model's args() is the authoritative gate (it throws and greys out Run); this -// mirrors the same condition into an inline alert the user actually sees. Fires when the chosen column -// collides with the barcode/feature roles — e.g. a value left stale after changing the feature column. +// mirrors the same condition into an inline alert the user actually sees. The selector itself is not +// offered today, but a project saved while it was — or migrated — can still carry a value that collides +// with the barcode/feature roles, and without this the Run button would simply be grey. const combineColumnError = computed(() => { const c = app.model.data.combineColumn; if (!c) return undefined; @@ -175,12 +177,19 @@ function clearSampleAwareOnInputChange() { } // CSV swap invalidates every CSV-derived selection: the barcode / feature-name columns (the new file's -// headers differ), the negative control, and the sample-aware selection (columns/values change). Clear -// them all so the user re-picks against the new CSV. +// headers differ), the negative control, the sample-aware selection (columns/values change), and every +// setting of the binding reading that names a panel column or a panel value. The last group matters most: +// emit_verdicts.py ends the whole run when the role column or the grouping column is not one the panel +// carries, so a stale pick left behind here costs a run and reports it where the user never looks. function clearOnCsvChange() { app.model.data.barcodeSeqColumn = undefined; app.model.data.featureNameColumn = undefined; app.model.data.combineColumn = undefined; + app.model.data.roleColumn = undefined; + app.model.data.referenceValues = undefined; + app.model.data.grouping = undefined; + app.model.data.contendingGroups = undefined; + app.model.data.panelColumnSnapshot = undefined; clearControlOnInputChange(); clearSampleAwareOnInputChange(); } @@ -431,14 +440,15 @@ const gridOptions = { > - - + + + {{ combineColumnError }} + - Specificity scores will not be computed without a negative control feature + No negative control is designated, so no feature will be marked as the background control in + the output. Nothing else changes.
{{ line }}
 + + +import { + PlAgDataTableV2, + PlAlert, + PlBlockPage, + PlBtnGhost, + PlMaskIcon24, + PlSlideModal, + usePlDataTableSettingsV2, +} from "@platforma-sdk/ui-vue"; +import { computed, ref } from "vue"; +import { useApp } from "../app"; +import VerdictSettings from "../components/VerdictSettings.vue"; + +const app = useApp(); + +// One row per (clonotype set, antigen identity): the four-state verdict, why an unsettled one is +// unsettled, and the cells behind it. Filtering comes from the column specs — the states and the competed +// flag are discrete filters, and nothing in the family is orderable. +const tableSettings = usePlDataTableSettingsV2({ + model: () => app.model.outputs.verdictTable, +}); + +// The reading's own settings, reachable from the page they explain. +const settingsOpen = ref(false); + +// A missing V(D)J dataset is a legitimate state, not a half-filled form: the block runs, and the verdict +// stage alone is skipped. Read from data rather than from an output because the point is what the user +// has chosen, including before the next run. +const noDataset = computed(() => app.model.data.datasetRef === undefined); + +// What the run was actually answered under. The software degrades a comparator it cannot serve, so the +// choice that SERVED is the only one worth stating: a reader meeting a table of unreliable rows otherwise +// has nothing telling them the comparator they asked for was never available. +const runMeta = computed(() => app.model.outputs.verdictRunMeta); +const noComparator = computed(() => runMeta.value?.referenceChoice === "no comparator available"); +const comparatorDegraded = computed( + () => + runMeta.value !== undefined && + runMeta.value.referenceSourceRequested !== runMeta.value.referenceChoice, +); +// Tags the grouping column said nothing about stand as their own identity, under a bare barcode. The +// software reports this to stderr; a row a reader cannot place needs saying in the page too. +const ungroupedTags = computed(() => runMeta.value?.tagsWithoutGroupingValue ?? []); + + + diff --git a/workflow/src/main.tpl.tengo b/workflow/src/main.tpl.tengo index 5a59ccd..3951427 100644 --- a/workflow/src/main.tpl.tengo +++ b/workflow/src/main.tpl.tengo @@ -504,6 +504,10 @@ wf.body(func(args) { // family lead selection can see, the pivoted per-identity summary, the offered scope, the // re-derivation material, the tag -> identity linker and the label columns. blockExports.antigenVerdicts = verdictImport.output("antigenVerdicts") + // The same frame as an OUTPUT, because a block's own exports are not in its own result pool: without + // this the block that produced the verdicts is the one place that cannot show them. The model reads + // it, keeps the columns keyed (clonotype set, identity) plus the identity label, and drops the rest. + blockOutputs.antigenVerdictsTable = pframes.exportFrame(verdictImport.output("antigenVerdicts")) // The run's own report. Outputs rather than exports: these are read by this block's model and UI. blockOutputs.antigenQcTable = pframes.exportFrame(verdictImport.output("qcTable")) blockOutputs.antigenPanelMismatchTable = pframes.exportFrame(verdictImport.output("panelMismatchTable")) From b3e3aead45c7d7a3236136239bddd7b497cbf7d7 Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 21:12:57 +0200 Subject: [PATCH 059/282] MILAB-6496: a page for the quality measurements and the panel check MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit blockOutputs.antigenQcTable and blockOutputs.antigenPanelMismatchTable were computed, exported and read by nobody: the whole measurement set, its statuses, the coverage triple and the three-level rollup were invisible, and so was the panel-versus-reads check. Adds a Quality checks page carrying both, switched by a button group so each table gets the page's full height. Two model outputs resolve the frames with createPlDataTableV2 for the reason already recorded on perCellTable: V3's discovery walks the entire result pool and hangs on the upstream Samples & Data FASTQ dataset. The panel axis is a twelve-character hash of the sorted tag list, so both tables carry the panel label column from the exported verdict frame — a block's own exports are not in its own pool, so it travels in the columns list the way the identity label already does on the verdict table. The coverage triple and the deferral reason become visible by default. A status says whether what was checked is wrong and the triple says how much of the level was checked at all; a deferral whose reason is one click away is a blank cell to everyone who does not click, which reads as "checked, and fine" — the one thing the four-status vocabulary exists to prevent. Two grid states join the block data, and the v3 migration and the block test's data literal seed them alongside the verdict grid state. --- model/src/index.ts | 79 +++++++++++++++++ model/src/types.ts | 8 ++ test/src/wf.test.ts | 2 + ui/src/app.ts | 2 + ui/src/pages/QualityChecksPage.vue | 127 ++++++++++++++++++++++++++++ workflow/src/column-specs.lib.tengo | 15 +++- 6 files changed, 229 insertions(+), 4 deletions(-) create mode 100644 ui/src/pages/QualityChecksPage.vue diff --git a/model/src/index.ts b/model/src/index.ts index e9ab26a..27bc195 100644 --- a/model/src/index.ts +++ b/model/src/index.ts @@ -37,6 +37,7 @@ const DEFAULT_HIGH_REFERENCE_LINE = 100; // dataset that could be picked here always carries it. const IDENTITY_AXIS = "pl7.app/antigen/identityId"; const CLONOTYPE_SET_AXIS = "pl7.app/vdj/scClonotypeKey"; +const PANEL_AXIS = "pl7.app/antigen/panelId"; // The run record emit_verdicts.py writes (result_run_meta.json), read as content. Only the fields the UI // states back to the user are typed here; the file carries every parameter the reading used. @@ -194,6 +195,25 @@ function parseQcRows(ctx: BlockRenderCtx) { return (qcMap?.data ?? []).filter((e) => e.value != null); } +// The readable name of each declared tag set, for the two report tables keyed on the panel axis. +// +// A panel id is a twelve-character hash of the sorted tag list — stable across re-runs of the same +// declaration, and unreadable on sight. The name emit_verdicts.py writes for it (" tags: ") is a label column in the EXPORTED verdict frame, so it has to travel in the columns list +// like the identity label does on the verdict table: createPlDataTable discovers label columns in the +// result pool, and a block's own exports are not in its own pool. +function panelLabelColumns(ctx: BlockRenderCtx) { + const pCols = ctx.outputs + ?.resolve({ field: "antigenVerdictsTable", allowPermanentAbsence: true }) + ?.getPColumns(); + return (pCols ?? []).filter( + (c) => + c.spec.name === "pl7.app/label" && + c.spec.axesSpec.length === 1 && + c.spec.axesSpec[0].name === PANEL_AXIS, + ); +} + // Tag→feature CSV metadata from the prerun (emit-csv-meta), or undefined until staging has produced it. // Shared by the two column dropdowns, the control dropdown, and the csvColumnsLoading signal. function readCsvMeta(ctx: BlockRenderCtx): CsvMeta | undefined { @@ -264,6 +284,8 @@ type BlockDataV2 = Omit< | "grouping" | "contendingGroups" | "verdictTableState" + | "antigenQcTableState" + | "panelMismatchTableState" > & { dominanceThreshold: number; offtargetProperty?: string; @@ -315,6 +337,8 @@ const dataModel = new DataModelBuilder() referenceThinLine: DEFAULT_REFERENCE_THIN_LINE, highReferenceLine: DEFAULT_HIGH_REFERENCE_LINE, verdictTableState: createPlDataTableStateV2(), + antigenQcTableState: createPlDataTableStateV2(), + panelMismatchTableState: createPlDataTableStateV2(), }), ) .init(() => ({ @@ -334,6 +358,8 @@ const dataModel = new DataModelBuilder() tableState: createPlDataTableStateV2(), qcSummaryTableState: createPlDataTableStateV2(), verdictTableState: createPlDataTableStateV2(), + antigenQcTableState: createPlDataTableStateV2(), + panelMismatchTableState: createPlDataTableStateV2(), })); export const platforma = BlockModelV3.create(dataModel) @@ -960,6 +986,55 @@ export const platforma = BlockModelV3.create(dataModel) }, { retentive: true, withStatus: true }, ) + // The run's own quality report: one row per (level, panel, measured thing, measurement), carrying the + // measurement's status, the coverage triple beside it and — where nothing computed the measurement — the + // reason it was deferred. Every declared measurement keeps its row whether or not this run could compute + // it, so the frame is complete by construction and the view needs no handling for a measurement that is + // simply absent. + // + // The panel is part of the KEY here, not a value beside it: the same reagent stained into several samples + // writes one row per panel at the same (level, measured thing, measurement), and rows sharing an axis key + // are silently collapsed on import to whichever one survives. + // + // createPlDataTableV2 rather than V3 for the reason recorded on perCellTable above: V3's discovery walks + // the entire result pool and hangs on the upstream Samples & Data FASTQ dataset. Every column is keyed on + // this frame's own four axes, so no filtering by axis shape is needed — unlike verdictTable, which reads a + // frame holding several tables at once. + .output( + "antigenQcTable", + (ctx) => { + const pCols = ctx.outputs + ?.resolve({ field: "antigenQcTable", allowPermanentAbsence: true }) + ?.getPColumns(); + if (pCols === undefined || pCols.length === 0) return undefined; + return createPlDataTableV2( + ctx, + [...pCols, ...panelLabelColumns(ctx)], + ctx.data.antigenQcTableState, + ); + }, + { retentive: true, withStatus: true }, + ) + // The panel-versus-reads check: one row per (panel, tag), with the direction of the mismatch and the + // samples that reported it. Keyed on the panel rather than on the sample because a declared tag the reads + // never carried is a property of the declared tag set, not of any one sample; the samples travel in the + // row so nothing about where it was seen is lost. Both directions live in the one frame under the + // direction column, which is what lets a single table show them both. + .output( + "antigenPanelMismatchTable", + (ctx) => { + const pCols = ctx.outputs + ?.resolve({ field: "antigenPanelMismatchTable", allowPermanentAbsence: true }) + ?.getPColumns(); + if (pCols === undefined || pCols.length === 0) return undefined; + return createPlDataTableV2( + ctx, + [...pCols, ...panelLabelColumns(ctx)], + ctx.data.panelMismatchTableState, + ); + }, + { retentive: true, withStatus: true }, + ) // What the reading was actually answered under. The page states the comparator that SERVED rather than // the one that was requested, because the software degrades a request it cannot honour and a reader // meeting an all-unreliable table otherwise has no way to learn that happened. Absent until a run with a @@ -1047,6 +1122,10 @@ export const platforma = BlockModelV3.create(dataModel) // columns at all, and the page saying so is the only place a user learns why — hiding the // tab would leave the absence unexplained. { type: "link" as const, href: "/verdicts" as const, label: "Binding verdicts" }, + // Shown on the same terms as the verdict tab, for the same reason: a run with no V(D)J + // dataset produced neither the measurements nor the panel check, and the page saying so is + // the only place a user learns why. + { type: "link" as const, href: "/checks" as const, label: "Quality checks" }, ] : []), ]; diff --git a/model/src/types.ts b/model/src/types.ts index 2695cc3..43e50ea 100644 --- a/model/src/types.ts +++ b/model/src/types.ts @@ -143,6 +143,14 @@ export type BlockData = { */ contendingGroups?: string[][]; verdictTableState: PlDataTableStateV2; // verdict grid state (UI-only, never projected to args) + /** + * Grid state for the two halves of the run's own report — the quality measurements and the + * panel-versus-reads check. Separate states because they are separate frames on separate keys: the + * measurements are keyed (level, panel, measured thing, measurement) and the check is keyed + * (panel, tag), so a column set or filter saved for one means nothing in the other. UI-only. + */ + antigenQcTableState: PlDataTableStateV2; + panelMismatchTableState: PlDataTableStateV2; presetId?: string; pattern?: string; diff --git a/test/src/wf.test.ts b/test/src/wf.test.ts index f68f207..67d9655 100644 --- a/test/src/wf.test.ts +++ b/test/src/wf.test.ts @@ -138,6 +138,8 @@ blockTest.skip( tableState: createPlDataTableStateV2(), qcSummaryTableState: createPlDataTableStateV2(), verdictTableState: createPlDataTableStateV2(), + antigenQcTableState: createPlDataTableStateV2(), + panelMismatchTableState: createPlDataTableStateV2(), } satisfies BlockData, }); diff --git a/ui/src/app.ts b/ui/src/app.ts index 09577ac..aab38de 100644 --- a/ui/src/app.ts +++ b/ui/src/app.ts @@ -3,6 +3,7 @@ import { defineAppV3 } from "@platforma-sdk/ui-vue"; import { watchEffect } from "vue"; import MainPage from "./pages/MainPage.vue"; import QcSummaryPage from "./pages/QcSummaryPage.vue"; +import QualityChecksPage from "./pages/QualityChecksPage.vue"; import ResultsPage from "./pages/ResultsPage.vue"; import VerdictsPage from "./pages/VerdictsPage.vue"; @@ -32,6 +33,7 @@ export const sdkPlugin = defineAppV3(platforma, (app) => { "/qc": () => QcSummaryPage, "/results": () => ResultsPage, "/verdicts": () => VerdictsPage, + "/checks": () => QualityChecksPage, }, }; }); diff --git a/ui/src/pages/QualityChecksPage.vue b/ui/src/pages/QualityChecksPage.vue new file mode 100644 index 0000000..9d4dfaf --- /dev/null +++ b/ui/src/pages/QualityChecksPage.vue @@ -0,0 +1,127 @@ + + + diff --git a/workflow/src/column-specs.lib.tengo b/workflow/src/column-specs.lib.tengo index 11a2fb4..feb3c2b 100644 --- a/workflow/src/column-specs.lib.tengo +++ b/workflow/src/column-specs.lib.tengo @@ -1004,12 +1004,19 @@ qcImportSpec := func(levelAxisSpec, entityAxisSpec, measurementAxisSpec, panelAx "pl7.app/discreteValues": QC_STATUSES, "pl7.app/description": "'unjudged' means no line exists to judge this against; 'not evaluated' means nothing computed it. Neither is a pass." }), - col("judged", "pl7.app/antigen/qcJudged", "Int", "Judged", 69000, false, { "pl7.app/min": "0" }), - col("unjudged", "pl7.app/antigen/qcUnjudged", "Int", "Unjudged", 68000, false, { "pl7.app/min": "0" }), - col("notEvaluated", "pl7.app/antigen/qcNotEvaluated", "Int", "Not evaluated", 67000, false, { "pl7.app/min": "0" }), + // The coverage triple is shown by default, beside the status rather than folded into it: a + // status says whether what was checked is wrong, and the triple says how much of the level was + // checked at all. Hiding the triple behind the column chooser would leave "nothing here is + // wrong" and "almost nothing here was checkable" looking identical on the page. + col("judged", "pl7.app/antigen/qcJudged", "Int", "Judged", 69000, true, { "pl7.app/min": "0" }), + col("unjudged", "pl7.app/antigen/qcUnjudged", "Int", "Unjudged", 68000, true, { "pl7.app/min": "0" }), + col("notEvaluated", "pl7.app/antigen/qcNotEvaluated", "Int", "Not evaluated", 67000, true, { "pl7.app/min": "0" }), col("counts", "pl7.app/antigen/qcCounts", "String", "What it counts", 66000, false, {}), col("implies", "pl7.app/antigen/qcImplies", "String", "What a bad value means", 65000, false, {}), - col("reason", "pl7.app/antigen/qcReason", "String", "Why deferred", 64000, false, {}) + // Also shown by default. A row reading "not evaluated" carries the reason nothing computed it, + // and a deferral whose reason is one click away is a blank cell to everyone who does not click — + // which reads as "checked, and fine", the one thing this vocabulary exists to prevent. + col("reason", "pl7.app/antigen/qcReason", "String", "Why deferred", 64000, true, {}) ], storageFormat: "Parquet", partitionKeyLength: 0 From 10a910b8d75943352840591fccd3caaabd77bf3f Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 21:32:33 +0200 Subject: [PATCH 060/282] MILAB-6496: count a reading only where the cell's own sample offered it The silent tally drew its numerator and denominator from two different populations. `asked` counts a member only where that member's OWN sample offered the identity; the observed count applied no such test, so a reading from a cell whose sample never offered the identity was counted against a denominator that excluded it. Reachable on ordinary data: a sample-keyed panel, a clonotype set spanning two samples, and a tag declared for one sample but read in the other -- which is the undeclared-in-panel case this block measures on purpose. Two regimes, both reproduced. Above the imbalance threshold the run dies on an assertion blaming a uniqueness violation that is not there. Below it, the verdict is silently wrong: a vote from a cell that was never asked displaces a silent cell's real one, and a tie reads as bound with full agreement. This is not new policy. Where a set sits in one sample the reading is already discarded as never-asked; the multi-sample case now behaves the same way. The same guard goes on the explicit votes in combine_cells and self_disagreement, whose counts must agree with the tally's. Also: panelId is an AXIS of the imported QC frame and was written null on every sample-level and capture-level row -- 45 of 122 on the fixture bed. A null is not a usable p-column key, as panel.py states of itself. Those rows belong to no panel and now carry an empty string, which is what the column spec already documented. --- software/per-cell-metrics/src/combine.py | 16 +++++ .../per-cell-metrics/src/emit_verdicts.py | 4 +- software/per-cell-metrics/src/qc_measures.py | 10 +-- software/per-cell-metrics/src/verdict.py | 18 +++++ .../test/test_emit_verdicts.py | 71 +++++++++++++++++++ 5 files changed, 113 insertions(+), 6 deletions(-) diff --git a/software/per-cell-metrics/src/combine.py b/software/per-cell-metrics/src/combine.py index 578cde6..163ea7a 100644 --- a/software/per-cell-metrics/src/combine.py +++ b/software/per-cell-metrics/src/combine.py @@ -191,6 +191,14 @@ def combine_cells( # kept here so a vote can never be counted for a cell nobody # asked to vote. continue + if identity not in offered.get(sample_id, frozenset()): + # This cell's own sample never offered the identity, so the cell + # was never asked about it and its reading is not a vote. The + # denominator below counts only members whose own sample offered + # it; counting the vote here would mix two populations, and where + # a set sits in one sample this reading is already discarded as + # never-asked. The multi-sample case now behaves the same way. + continue bucket = explicit_counts.setdefault((set_id, identity), {}) bucket[state] = bucket.get(state, 0) + 1 @@ -502,6 +510,14 @@ def self_disagreement( # Same drop `combine_cells` applies: a vote is never counted for # a cell that no set's membership list names. continue + if key not in offered.get(sample_id, frozenset()): + # This cell's own sample never offered the identity, so the cell + # was never asked about it and its reading is not a vote. The + # denominator below counts only members whose own sample offered + # it; counting the vote here would mix two populations, and where + # a set sits in one sample this reading is already discarded as + # never-asked. The multi-sample case now behaves the same way. + continue bucket = explicit_counts.setdefault((set_id, key), {}) bucket[state] = bucket.get(state, 0) + 1 diff --git a/software/per-cell-metrics/src/emit_verdicts.py b/software/per-cell-metrics/src/emit_verdicts.py index ced20bd..59efaa7 100644 --- a/software/per-cell-metrics/src/emit_verdicts.py +++ b/software/per-cell-metrics/src/emit_verdicts.py @@ -199,7 +199,7 @@ def _build_grouping( panel: pl.DataFrame, properties: dict[str, dict[str, str]], reference_tags: set[str], -) -> tuple[Grouping, str]: +) -> tuple[Grouping, str, list[str]]: """The tag -> identity map the run reads at, and the id of the rule behind it. A property grouping is built from `consistent_properties`, never from the @@ -368,7 +368,7 @@ def _qc_frame(rows: list[QcRow]) -> pl.DataFrame: { "level": row.level, "entity": row.entity, - "panelId": row.panel_id or None, + "panelId": row.panel_id, # "" not None: this is an AXIS key, and a null is not a usable one "measurement": row.measurement, "value": row.value, "detail": row.detail or None, diff --git a/software/per-cell-metrics/src/qc_measures.py b/software/per-cell-metrics/src/qc_measures.py index 05cbf6b..8972a4d 100644 --- a/software/per-cell-metrics/src/qc_measures.py +++ b/software/per-cell-metrics/src/qc_measures.py @@ -64,10 +64,12 @@ class Measurement: "readsTotal", "Reads total and fraction matched", "sample", - # No line: the four inherited numbers atom 315 names are the usable - # antigen-read fraction, the undeclared-barcode fraction, the aggregate- - # barcode read fraction and barcode validity. The matched share is on - # none of them, so nothing here says what a low one would mean. + # No line. Exactly four numbers are inherited from the field, and the + # matched share is not one of them: the usable antigen-read fraction + # (published warn below 0.20), the undeclared-barcode fraction (warn + # above 0.50), the aggregate-barcode read fraction (warn above 0.05) + # and barcode validity (warn below 0.75). Nothing published says what a + # low matched share means, so nothing here claims to. "Every read the parser saw, and the share matching the tag pattern.", ), # The label names the recognized fraction rather than the spec row's diff --git a/software/per-cell-metrics/src/verdict.py b/software/per-cell-metrics/src/verdict.py index 0f81574..dea7d97 100644 --- a/software/per-cell-metrics/src/verdict.py +++ b/software/per-cell-metrics/src/verdict.py @@ -610,6 +610,15 @@ def silent_tally( # a row for it, and it is dropped here rather than counted # against a cell universe that never named it. continue + if ident not in offered_by_sample.get(k[0], frozenset()): + # Read, but this cell's OWN sample never offered the identity, + # so the cell was never asked about it. `asked` below counts + # only members whose own sample offered it, so counting this + # reading would draw the numerator and the denominator from two + # different populations. Where enough silent cells absorb the + # imbalance it does not even raise: it displaces a silent + # cell's real vote with one from a cell that was never asked. + continue pair = (k[0], ident) observed_count[pair] = observed_count.get(pair, 0) + 1 if k in inadmissible: @@ -653,6 +662,15 @@ def silent_tally( for k, ident in zip(obs_keys, obs_identity, strict=True): if k not in cell_keys: continue + if ident not in offered_by_sample.get(k[0], frozenset()): + # Read, but this cell's OWN sample never offered the identity, + # so the cell was never asked about it. `asked` below counts + # only members whose own sample offered it, so counting this + # reading would draw the numerator and the denominator from two + # different populations. Where enough silent cells absorb the + # imbalance it does not even raise: it displaces a silent + # cell's real vote with one from a cell that was never asked. + continue pair = (group_by_cell[k], ident) observed_count[pair] = observed_count.get(pair, 0) + 1 if k in inadmissible: diff --git a/software/per-cell-metrics/test/test_emit_verdicts.py b/software/per-cell-metrics/test/test_emit_verdicts.py index de5dcfa..f3dc9a2 100644 --- a/software/per-cell-metrics/test/test_emit_verdicts.py +++ b/software/per-cell-metrics/test/test_emit_verdicts.py @@ -848,3 +848,74 @@ def test_the_bed_panel_without_a_declared_comparator_serves_as_its_own(wide_bed) assert _run(wide_bed, *_bed_args("panel_with_reference.csv")).returncode == 0 with_declared = json.loads((wide_bed / "result_run_meta.json").read_text())["readingsFloored"] assert without > with_declared > 0 + + +def test_a_reading_from_a_sample_that_never_offered_it_is_not_a_vote(bed): + # The denominator counts only members whose OWN sample offered the identity. + # If the numerator does not apply the same test, the two are drawn from + # different populations: a reading from a cell that was never asked + # displaces a silent cell's real vote. Reachable whenever a sample-keyed + # panel meets a set spanning two samples and a tag declared for one sample + # is read in the other -- which is the undeclared-in-panel case this block + # measures on purpose, not an exotic shape. + (bed / "panel.csv").write_text( + "Samples,Name,Sequence,Type\nS1,Ctrl,CTRL,Control\nS2,Ctrl,CTRL,Control\nS2,AgX,XXXX,Target\n" + ) + (bed / "counts.csv").write_text( + "sampleId,cellId,tag,umiCount\n" + "S1,c1,CTRL,6\nS1,c1,XXXX,500\n" # S1 never offered XXXX; this is not a vote + "S2,c2,CTRL,6\nS2,c2,XXXX,500\n" # offered and bound + "S2,c3,CTRL,6\n" # offered and silent -> not bound + ) + (bed / "linker.csv").write_text("sampleId,cellId,setId\nS1,c1,K1\nS2,c2,K1\nS2,c3,K1\n") + + r = _run(bed, *BASE) + assert r.returncode == 0, r.stderr + row = ( + pl.read_csv(bed / "result_verdicts.csv", infer_schema_length=0) + .filter(pl.col("identity") == "XXXX") + .row(0, named=True) + ) + # One bound and one not-bound among the two cells that were actually asked. + assert row["state"] == "unreliable" + assert row["unreliableReason"] == "tie" + assert (row["cellsCouldAnswer"], row["cellsAnswered"]) == ("2", "2") + + +def test_every_asked_cell_reading_still_counts_when_both_samples_offered_it(bed): + # The guard above must not throw away legitimate cross-sample votes: with + # both samples offering the identity, all three cells vote as before. + (bed / "panel.csv").write_text( + "Samples,Name,Sequence,Type\n" + "S1,Ctrl,CTRL,Control\nS1,AgX,XXXX,Target\nS2,Ctrl,CTRL,Control\nS2,AgX,XXXX,Target\n" + ) + (bed / "counts.csv").write_text( + "sampleId,cellId,tag,umiCount\nS1,c1,CTRL,6\nS1,c1,XXXX,500\nS2,c2,CTRL,6\nS2,c2,XXXX,500\nS2,c3,CTRL,6\n" + ) + (bed / "linker.csv").write_text("sampleId,cellId,setId\nS1,c1,K1\nS2,c2,K1\nS2,c3,K1\n") + + r = _run(bed, *BASE) + assert r.returncode == 0, r.stderr + row = ( + pl.read_csv(bed / "result_verdicts.csv", infer_schema_length=0) + .filter(pl.col("identity") == "XXXX") + .row(0, named=True) + ) + assert (row["cellsCouldAnswer"], row["cellsAnswered"]) == ("3", "3") + assert row["state"] == "bound" # two bound against one silent + + +def test_no_qc_row_carries_a_null_panel_key(bed): + # panelId is an AXIS of the imported QC frame, and a null is not a usable + # p-column key. Sample-level and capture-level rows belong to no panel, so + # they carry an empty string -- which is a key -- never a null. + _run(bed, *BASE, "--capture-map", json.dumps({"S1": "C1"})) + qc = pl.read_csv(bed / "result_qc.csv", infer_schema_length=0) + assert qc["panelId"].null_count() == 0 + + # Both kinds must be present, or the assertion above proves nothing: rows + # that belong to a panel carry its id, rows that belong to none carry an + # empty string. + panels = set(qc["panelId"].to_list()) + assert "" in panels, "sample and capture rows belong to no panel and must carry an empty key" + assert any(p for p in panels), "tag and identity rows must carry a real panel id" From 6c68b3d2e3dfbcf7401f94ba7654affbcb0beaf9 Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 21:33:11 +0200 Subject: [PATCH 061/282] MILAB-6496: prose and message fixes from the STE100 review Three error messages named no control the reader could act on, and one sent them to the wrong drawer entirely: the panel minimum lives under "Advanced verdict settings", not the "Advanced Settings" accordion the message named, which holds a different parameter on another page. The four inherited QC numbers were cited as "atom 315 names them" -- a doc reference this workspace forbids in code comments, since a reader cannot follow it from here. Expanded into the four numbers and their published thresholds. _build_grouping was annotated as returning a 2-tuple and has returned a 3-tuple since the unplaceable-tag list was added. The fixture README said a comparator of 6 is "above the thin line of 2", but the test is `reference < thin_line`, so exactly 2 is comparable. Someone calibrating a new fixture at 2 would have predicted the wrong state. --- model/src/index.ts | 10 +++++++--- software/test-data/fixtures/verdicts/README.md | 2 +- 2 files changed, 8 insertions(+), 4 deletions(-) diff --git a/model/src/index.ts b/model/src/index.ts index 27bc195..2337954 100644 --- a/model/src/index.ts +++ b/model/src/index.ts @@ -448,7 +448,10 @@ export const platforma = BlockModelV3.create(dataModel) // without the role values. Asking for it anyway would degrade to no comparator inside the run, where // the choice is recorded but the user never sees they lost it. if (data.referenceSource === "declared" && !data.referenceValues?.length) - throw new Error("Choose which role values mark the reference, or pick another source"); + throw new Error( + 'Under "Values marking the reference", choose at least one value, or choose a ' + + 'different option for "What counts are read against".', + ); // A role column or a grouping column the panel does not carry ends the whole run at the exec // (emit_verdicts.py exits rather than degrading), and the user meets that as a dead run with no hint // of which setting caused it. The check is against the headers snapshotted when the column was picked @@ -462,7 +465,7 @@ export const platforma = BlockModelV3.create(dataModel) ] as const) { if (column && !panelColumns.includes(column)) throw new Error( - `${role} column "${column}" is not a column of the uploaded panel file; re-select it`, + `The ${role} column "${column}" is not in the uploaded panel file. Select a column from the new panel.`, ); } } @@ -1081,7 +1084,8 @@ export const platforma = BlockModelV3.create(dataModel) else unavailable.push( `The panel's own readings — the panel declares ${panelSize} tag(s) and this source needs at ` + - `least ${minMembers}. Lower the panel minimum in Advanced Settings, or use a declared reference tag.`, + `least ${minMembers}. Lower "Panel minimum for self-comparison" under "Advanced verdict settings", ` + + `or declare a reference tag.`, ); options.push({ value: "none", diff --git a/software/test-data/fixtures/verdicts/README.md b/software/test-data/fixtures/verdicts/README.md index 070aac7..30fc315 100644 --- a/software/test-data/fixtures/verdicts/README.md +++ b/software/test-data/fixtures/verdicts/README.md @@ -54,7 +54,7 @@ ratio, so the useful values are not where intuition puts them — against a comp | `500` | The *bound* reading while the comparator is 6 (score 100) — and a *not bound* reading against 60 (score 0.1). That difference is what the two-comparator panel measures. | | `5000` | Bound against either comparator, so one binding survives on the two-comparator panel and the bed does not degenerate into all *not bound*. | | `2` (one reading only) | Below the floor of 4, so it is zeroed and counted in `readingsFloored`. | -| `6` (`Ctrl1`) | Above the thin line of 2 so cells can be compared, and far below 500 so a real binding clears the cutoff. | +| `6` (`Ctrl1`) | At or above the thin line of 2 so cells can be compared, and far below 500 so a real binding clears the cutoff. The test is `reference < thin_line`, so a comparator reading exactly 2 is still comparable — calibrate a new fixture against that, not against "above 2". | | `60` (`Ctrl2`) | Above `Ctrl1` so the highest-member rule is observable, and below the high-reference line of 100 so that measurement stays quiet. | | `1` (`Ctrl1` in one cell) | Below the thin line of 2. This is the bed's only source of *unreliable*: raise it and the fourth state disappears. | From e3416f557fb1c047a74d01c651de8c47f0967979 Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 21:43:56 +0200 Subject: [PATCH 062/282] MILAB-6496: make five fixtures discriminate what they claim to test Found by mutation testing the suites: 63 single-edit mutations, 48 caught. The survivors below were all the same shape -- a fixture whose values cannot tell the right behaviour from a wrong one. The parameter-threading suite set every number to the library's own default, so a builder that ignored the block's value and emitted its fallback produced an identical vector. Six threading defects survived the file whose stated purpose is to catch exactly that. The fixture now differs from every default and the values are asserted, not just the flags' presence. The silent-tally oracle drew counts from 0-30 against references of 2-20. The highest score reachable that way is about 11.8 against a cutoff of 75, so no cell in any seed was ever bound, and the assertion that a silent admissible cell is never observed as bound compared zero against zero in every run. Counts now sometimes clear the cutoff: the oracle produces 16 bound rows across the seeds that previously produced none. cellsWithSignal and cellsAboveTheLine landed on the same rows, so counting bound cells for both passed. A cell that reads but does not bind now separates them. Two stated contracts had no test at all: the outlier fence is one-sided, because a tag disagreeing less than its peers is behaving well rather than misbehaving; and three peers is enough to compare against while two is not. --- .../per-cell-metrics/test/test_qc_measures.py | 28 ++++++++++++-- .../per-cell-metrics/test/test_verdict.py | 9 ++++- workflow/src/verdict-args.test.tengo | 37 ++++++++++++++++--- 3 files changed, 63 insertions(+), 11 deletions(-) diff --git a/software/per-cell-metrics/test/test_qc_measures.py b/software/per-cell-metrics/test/test_qc_measures.py index 6c1bc78..ca0ff61 100644 --- a/software/per-cell-metrics/test/test_qc_measures.py +++ b/software/per-cell-metrics/test/test_qc_measures.py @@ -187,15 +187,19 @@ def test_a_computed_measurement_carries_no_status(): def test_per_antigen_measures_reports_signal_above_and_median(): + # The cell reading 5 and not binding is what separates the two counters. + # Without it both land on the same rows and each reads 2, so a version that + # counted bound cells for both would pass -- and "cells with signal" would + # silently become "cells above the line" wherever it is reported. states = pl.DataFrame( { - "tag": ["T1", "T1", "T1"], - "umiCount": [0, 10, 40], - "state": ["not bound", "bound", "bound"], + "tag": ["T1", "T1", "T1", "T1"], + "umiCount": [0, 5, 10, 40], + "state": ["not bound", "not bound", "bound", "bound"], } ) out = per_antigen_measures(states).row(0, named=True) - assert out["cellsWithSignal"] == 2 + assert out["cellsWithSignal"] == 3 assert out["cellsAboveTheLine"] == 2 assert out["medianAboveTheLine"] == 25.0 @@ -561,3 +565,19 @@ def test_a_dead_reagent_does_not_mark_every_sample_alerting(): panel = roll_up_panel(tag_statuses=[Status.ALERTING], identity_statuses=[Status.ACCEPTABLE]) assert samples == [Status.ACCEPTABLE] * 3 assert roll_up_capture(sample_statuses=samples, panel_statuses=[panel.status]).status is Status.ALERTING + + +def test_outlier_status_flags_only_high_values(): + # "A disagreement rate below its peers is a tag behaving better than the + # panel, which is not a finding." A two-sided fence would alert on the + # best-behaved reagent in every panel, which is the opposite of the point. + peers = [0.4, 0.5, 0.6, 0.5] + assert outlier_status(0.0, peers) is Status.ACCEPTABLE + assert outlier_status(9.9, peers) is Status.ALERTING + + +def test_the_minimum_peer_count_is_satisfied_at_the_named_value(): + # The named value satisfies the condition it names: three peers is enough + # to compare against, two is not. Nothing else pins this boundary. + assert outlier_status(0.9, [0.01, 0.02, 0.03]) is not Status.UNJUDGED + assert outlier_status(0.9, [0.01, 0.02]) is Status.UNJUDGED diff --git a/software/per-cell-metrics/test/test_verdict.py b/software/per-cell-metrics/test/test_verdict.py index 6328a50..caa6c50 100644 --- a/software/per-cell-metrics/test/test_verdict.py +++ b/software/per-cell-metrics/test/test_verdict.py @@ -543,7 +543,14 @@ def _build_silent_tally_population(seed, force_empty_sample=None): # sample offered actually got a tag-stat row. for identity in offered_by_sample[sample]: if rng.random() < 0.5: - tag_rows.append((sample, cell, identity, rng.randint(0, 30))) + # Some readings must actually clear the cutoff. Against + # references of 2-20 the largest score a count of 30 can + # reach is about 11.8, so a population drawn only from + # 0-30 contains no bound cell at all -- and the oracle + # comparison's bound assertion below then reads 0 == 0 in + # every run, proving nothing about the claim it names. + count = rng.randint(0, 30) if rng.random() < 0.7 else rng.randint(200, 900) + tag_rows.append((sample, cell, identity, count)) return samples, identities, thin_line, gated, reference, cell_rows, tag_rows, offered_by_sample diff --git a/workflow/src/verdict-args.test.tengo b/workflow/src/verdict-args.test.tengo index df3265c..d53f3cd 100644 --- a/workflow/src/verdict-args.test.tengo +++ b/workflow/src/verdict-args.test.tengo @@ -12,14 +12,18 @@ _full := { roleColumn: "Type", referenceValues: ["Control", "Isotype"], referenceSource: "declared", - panelReferenceMinMembers: 8, - referenceThinLine: 2, - countFloor: 4, - boundCutoff: 75, - minVotingCells: 1, + // Every number here differs from the library's own default for it. With the + // defaults, a builder that ignored the block's value and emitted its fallback + // would produce an identical vector, and this whole file would pass while the + // parameter never reached the exec. + panelReferenceMinMembers: 12, + referenceThinLine: 3, + countFloor: 7, + boundCutoff: 90, + minVotingCells: 3, minAgreement: 0.6, gateThreshold: 40, - highReferenceLine: 100, + highReferenceLine: 250, grouping: { by: "property", column: "family" }, contendingGroups: [["AgA", "AgB"]], captureMap: { S1: "lane1" } @@ -40,6 +44,27 @@ Test_every_flag_is_threaded := func() { for _, flag in flags { test.isTrue(va.has(args, flag), "argument list is missing " + flag) } + + // Presence is not threading. A flag can appear carrying the library's own + // fallback while the block's value is dropped on the floor, which is the + // same silence this file exists to break, one step further in. + expected := { + "--barcode-col": "Sequence", + "--feature-col": "Name", + "--sample-col": "Samples", + "--role-column": "Type", + "--panel-min-members": "12", + "--reference-thin-line": "3", + "--floor": "7", + "--cutoff": "90", + "--min-voters": "3", + "--min-agreement": "0.6", + "--gate-threshold": "40", + "--high-reference-line": "250" + } + for flag, want in expected { + test.isEqual(want, string(va.valueOf(args, flag)), "wrong value threaded for " + flag) + } } // The per-sample read QC is the one input whose absence is invisible: three of the fifteen measurements From 3caa750a53f6e70b9131bdfdfd4e187fc36cd94f Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 21:54:49 +0200 Subject: [PATCH 063/282] MILAB-6496: changeset and SDK bump for the antigen binding verdicts Both packages the require-latest CI gate checks were behind and would have rejected the build: block-tools 2.12.8 -> 2.13.0 and tengo-builder 4.0.19 -> 4.0.22. Lockfile committed with them. The changeset lists all four subpackages and the root block package explicitly, because a minor bump does not propagate past patch on its own. Verified with changeset status: five packages at minor, the private test package at patch. --- .changeset/antigen-binding-verdicts.md | 25 ++++ pnpm-lock.yaml | 191 +++++++++++++++++++++++-- pnpm-workspace.yaml | 4 +- 3 files changed, 205 insertions(+), 15 deletions(-) create mode 100644 .changeset/antigen-binding-verdicts.md diff --git a/.changeset/antigen-binding-verdicts.md b/.changeset/antigen-binding-verdicts.md new file mode 100644 index 0000000..e09e3d2 --- /dev/null +++ b/.changeset/antigen-binding-verdicts.md @@ -0,0 +1,25 @@ +--- +'@platforma-open/milaboratories.feature-integration.workflow': minor +'@platforma-open/milaboratories.feature-integration.model': minor +'@platforma-open/milaboratories.feature-integration.ui': minor +'@platforma-open/milaboratories.feature-integration.per-cell-metrics': minor +'@platforma-open/milaboratories.feature-integration': minor +--- + +Antigen binding is reported as a four-state verdict per clonotype set and antigen identity: **bound**, **not bound**, **never asked**, or **unreliable**. The last two are not kinds of "not bound" — *never asked* means the experiment did not put that antigen to those cells, and *unreliable* means it did and the data cannot settle the result. Both are emitted as rows rather than left absent, so a reader can tell an unanswered question from a negative answer. + +An identity is a group of tags, not a feature name: the same barcode carries different names in different samples' panels, so name-keying splits one reagent and can merge two. Tags combine into an identity by the highest of their counts, and the grouping is a rule over the panel's declared properties rather than a frozen map, so the same run can be read at more than one grouping. + +**What each verdict rests on travels with it.** Every row carries how many of the set's cells could have answered at that identity, how many did, and the agreement among them — so a verdict resting on three cells is distinguishable from one resting on forty. Where an antigen read *not bound* while something it was declared to compete with read *bound* for the same clonotype, the row says so, and a downstream filter can test it. + +**Nothing is orderable.** No score, rank or per-antigen magnitude leaves the block, and a build-time assertion refuses any score annotation on an emitted column. A verdict is a statement about what the experiment could establish; ranking clonotypes by it is a downstream block's job, from these outputs plus other assays. + +**Quality measurements ship with the reading.** Fifteen measurements across sample, tag, identity, panel and capture levels, each stating what it counts and — only where a line can be defended — what a bad value implies. A measurement with no defensible line reads *unjudged* rather than being given an invented threshold, and one the run could not supply inputs for reads *not evaluated* with its reason. Every level reports coverage beside its status, so a run states both what is wrong and how much of it was actually checked. Sample and panel roll up as separate axes: a bad sample is prepared again, a bad reagent is replaced. + +The panel-versus-reads check is emitted as a p-column in both directions — barcodes the reads carry that no panel declared, and tags a panel declared that the reads never showed. + +**Removed:** the dominant-feature ("consensus") call and the per-cell specificity score. A single dominant antigen per cell answers a different question from the one this block now answers, and a specificity magnitude is exactly the narrowing the four-state verdict replaces. The `pl7.app/feature/consensusFeature` and specificity p-columns are no longer emitted. No block in this workspace consumes them. + +**Unchanged:** the per-cell UMI count and fraction columns, the negative-control marker column, and the combine-mode settings. + +**A single-cell V(D)J dataset is required for the antigen stage.** Without one the block still runs and still emits its per-cell UMI counts and fractions, per-sample QC and per-feature properties exactly as before — but it produces no verdicts, no per-antigen columns and no panel check, rather than producing empty ones. The Verdicts page says so instead of showing an empty table. diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 1b1a6b4..23917ec 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -31,14 +31,14 @@ catalogs: specifier: 2.3.1-131-main version: 2.3.1-131-main '@platforma-sdk/block-tools': - specifier: 2.12.8 - version: 2.12.8 + specifier: 2.13.0 + version: 2.13.0 '@platforma-sdk/model': specifier: 1.80.8 version: 1.80.8 '@platforma-sdk/tengo-builder': - specifier: 4.0.19 - version: 4.0.19 + specifier: 4.0.22 + version: 4.0.22 '@platforma-sdk/test': specifier: 1.80.9 version: 1.80.9 @@ -89,7 +89,7 @@ importers: version: 1.6.1(@types/node@25.3.2)(rollup@4.53.3)(vue@3.5.25(typescript@5.9.3))(yaml@2.8.1) '@platforma-sdk/block-tools': specifier: 'catalog:' - version: 2.12.8(@types/node@25.3.2) + version: 2.13.0(@types/node@25.3.2) shx: specifier: 'catalog:' version: 0.4.0 @@ -116,7 +116,7 @@ importers: version: link:../workflow '@platforma-sdk/block-tools': specifier: 'catalog:' - version: 2.12.8(@types/node@25.3.2) + version: 2.13.0(@types/node@25.3.2) '@platforma-sdk/model': specifier: 'catalog:' version: 1.80.8 @@ -150,7 +150,7 @@ importers: version: 1.3.1 '@platforma-sdk/block-tools': specifier: 'catalog:' - version: 2.12.8(@types/node@25.3.2) + version: 2.13.0(@types/node@25.3.2) software/per-cell-metrics: devDependencies: @@ -159,7 +159,7 @@ importers: version: 1.7.8 '@platforma-sdk/block-tools': specifier: 'catalog:' - version: 2.12.8(@types/node@25.3.2) + version: 2.13.0(@types/node@25.3.2) test: dependencies: @@ -240,7 +240,7 @@ importers: devDependencies: '@platforma-sdk/tengo-builder': specifier: 'catalog:' - version: 4.0.19 + version: 4.0.22 '@platforma-sdk/test': specifier: 'catalog:' version: 1.80.9(@bytecodealliance/preview2-shim@0.17.8)(@types/node@25.3.2)(vite@8.0.8(@types/node@25.3.2)(yaml@2.8.1)) @@ -810,11 +810,20 @@ packages: resolution: {integrity: sha512-GsFaMXCkMqkKIvwCQjCrwH+GHbPKBjhwo/8ZuUkWHqbI73Kky9I+pQltrlT0+MWpedCoosda53lgjYfyEPgxBg==} engines: {node: '>=12.10.0'} + '@grpc/grpc-js@1.14.4': + resolution: {integrity: sha512-k9Dj3DV/itK9D06Y8f190Qgop7/Ui+D0njFV3LHMPwPT75DpXLQohE9Wmz0QElrJnzsjB7KPWiKJbOl7IPDArQ==} + engines: {node: '>=12.10.0'} + '@grpc/proto-loader@0.7.13': resolution: {integrity: sha512-AiXO/bfe9bmxBjxxtYxFAXGZvMaN5s8kO+jBHAJCON8rJoB5YS/D6X7ZNc6XQkuHNmyl4CYaMI1fJ/Gn27RGGw==} engines: {node: '>=6'} hasBin: true + '@grpc/proto-loader@0.8.1': + resolution: {integrity: sha512-wtF6h+DY6M3YaDBPAmvuuA6jV8Sif9MjtOI5euKFWRgCDl5PeDpPsHR9u2l6St5ceY8AZgoNDww5+HvEsXFsGg==} + engines: {node: '>=6'} + hasBin: true + '@inquirer/ansi@1.0.2': resolution: {integrity: sha512-S8qNSZiYzFd0wAcyG5AXCvUHC5Sr7xpZ9wZ2py9XR88jUz8wooStVx5M6dRzczbBWjic9NP7+rY0Xi7qqK/aMQ==} engines: {node: '>=18'} @@ -1062,6 +1071,10 @@ packages: resolution: {integrity: sha512-yyep+I0FozB1pbf22maRCVjgrO3AEB8f88PyTmJM0K7Y5bfq0T/MgprTyPwMB34u1XfKI9M5yqHWmyuYdzq08w==} engines: {node: '>=22.19.0'} + '@milaboratories/pl-client@3.14.6': + resolution: {integrity: sha512-IPzh5bSibr02DMKpwlCW6TuHVxaDe2o6tjaEJ4P2y5lshRbe5SBR/ne/G/N7E7KEKxmpWq7ZapOCLkLyvqFFDw==} + engines: {node: '>=22.19.0'} + '@milaboratories/pl-config@1.8.5': resolution: {integrity: sha512-XnfYXSSkRxeImQ21k6I8y5apisvagcSgMGfEeyRxNdSGwUVJbbI8TwJk+XBEDQ4lErW8oqtLxKjFQuHJMzRUoQ==} @@ -1096,6 +1109,9 @@ packages: '@milaboratories/pl-model-backend@1.4.17': resolution: {integrity: sha512-ZoxNLmWs+MQxCU/s4ml5DHV9htu0xk9NHYQ+07AtWNhLRCjSeFcVTVQlt0EvFMlzytOKGjmMbCMhaKYUuh/wrA==} + '@milaboratories/pl-model-backend@1.4.20': + resolution: {integrity: sha512-AkV+PhmQms6WPwR3E75THrkD9fFE7OpfsIUC297HvnFUoCFXR6GOMGPworA/qipXReTmv192AqBpuaLdgAngyg==} + '@milaboratories/pl-model-common@1.23.0': resolution: {integrity: sha512-1uHb2pS+hWJyBKvfOlMYmjbFuWn+tNOULWj84mWASdqSjdYyHfVYbLq5esawM+dnZ2GBQIzJRbXrZYIMqH4Peg==} @@ -1105,12 +1121,18 @@ packages: '@milaboratories/pl-model-common@1.47.2': resolution: {integrity: sha512-XCEcL+CHYxZ/S/y9zrpzzaeZO+ZKQZ3Y9/pEKEsgDZy4VLtXZGG0RHV9byNmMGz8Y+7BahQ9tAf1d26TdxRVZQ==} + '@milaboratories/pl-model-common@1.47.3': + resolution: {integrity: sha512-tXmKujm+6ru/Fh9hr9H3iRBWbS+JE0Q7FNualtzJpijaB3SNtScR4erLTamdANv5RYNn7kbYpDWr6wAjAOajrg==} + '@milaboratories/pl-model-middle-layer@1.30.14': resolution: {integrity: sha512-gZwF0ux28uOnKFfCwjcD8beJCfzdyq1mqLgd/+T5zYCFsomTvPCMp2HYwzXmUp7poO/HTIAxSA6qT8As2q2WWg==} '@milaboratories/pl-model-middle-layer@1.30.7': resolution: {integrity: sha512-rs9x3Ron4ujR/UOdEgB8WUB1SvZ8ZAScT1Av/e4or+iiQ/CzhmK9nqtYamHVwKV+JgwIFbXQwxGvIHDVz955dQ==} + '@milaboratories/pl-model-middle-layer@1.31.0': + resolution: {integrity: sha512-D1vyGABBtCYyp2IhKd9eMoZtKUj1wUYpIzz4Xz5k+bfJWGEF8GmZGiOvO4oOvrsEsXlG4NyO249MnBKmHFcppQ==} + '@milaboratories/pl-tree@1.13.2': resolution: {integrity: sha512-zFN+CGNyxSlv7cLcFBYMWreneXqrLNWqOkZlXQmHmqfocVrTZ5EYa9nE7L/6opwzF2VaYTYD/iaxNcUmpD68pA==} engines: {node: '>=22.19.0'} @@ -1685,6 +1707,10 @@ packages: resolution: {integrity: sha512-UI/gnQuRO+mt7Vy6ootQN1KP0H2GdxGQxtcppGQI6CEAEfTWYmCk8f1C7QoWTgS6tjQkkKmLE8iDzAdbZqzchg==} hasBin: true + '@platforma-sdk/block-tools@2.13.0': + resolution: {integrity: sha512-xKiTmNLjfxCnnHRr9Q4ct3HdjLfkbLztvxpU++/pek0R8Bj/Lezeoji+eLpxAAYYUxjMr8jLj935eLK5MqKrSg==} + hasBin: true + '@platforma-sdk/blocks-deps-updater@2.2.0': resolution: {integrity: sha512-p9lBxhFXM9WoRsrJO7dfkiXSK+1m63yIn1sKhBO71eMbhrLMyVYHEOeNf3w5OCdbRF5QsNhXzWuiTmFK3zHFsA==} hasBin: true @@ -1698,8 +1724,11 @@ packages: '@platforma-sdk/package-builder-lib@1.2.1': resolution: {integrity: sha512-H6weitj7JxbiJSlteEFLafTJ+tfty6iv/imf3ysy8oCS8AZIRJk2VMW3M/aAc+xVkQeX7oVIwMwFMYrJIoFsAg==} - '@platforma-sdk/tengo-builder@4.0.19': - resolution: {integrity: sha512-N4Koocvmfe2D4yOnNkCTiRiz++AyinA9kZrSQ1cmVD2uofwaJXRBEmVsc029VciWSCrn8FmFt3KjZ1JPE4IieA==} + '@platforma-sdk/package-builder-lib@1.3.0': + resolution: {integrity: sha512-CdBjmNo6E1fBxKYWaXa49L/L2WLURxs2f1TAqxLIZlHRE4DZ6E1TEj3jNNKESWp+/9rwtLkTAzmTzNPrDgz+2Q==} + + '@platforma-sdk/tengo-builder@4.0.22': + resolution: {integrity: sha512-8+zDYDFFI2tQS7dplTcxgQdUwzt8+IO++sJiVbwBgwSg1Giyc2qMThEYPLK3QNFArgCsIDR6wQuS8t66pJsRPQ==} engines: {node: '>=22'} hasBin: true @@ -1743,12 +1772,21 @@ packages: '@protobufjs/codegen@2.0.4': resolution: {integrity: sha512-YyFaikqM5sH0ziFZCN3xDC7zeGaB/d0IUb9CATugHWbd1FRFwWwt4ld4OYMPWu5a3Xe01mGAULCdqhMlPl29Jg==} + '@protobufjs/codegen@2.0.5': + resolution: {integrity: sha512-zgXFLzW3Ap33e6d0Wlj4MGIm6Ce8O89n/apUaGNB/jx+hw+ruWEp7EwGUshdLKVRCxZW12fp9r40E1mQrf/34g==} + '@protobufjs/eventemitter@1.1.0': resolution: {integrity: sha512-j9ednRT81vYJ9OfVuXG6ERSTdEL1xVsNgqpkxMsbIabzSo3goCjDIveeGv5d03om39ML71RdmrGNjG5SReBP/Q==} + '@protobufjs/eventemitter@1.1.1': + resolution: {integrity: sha512-vW1GmwMZNnL+gMRaovlh9yZX74kc+TTU3FObkkurpMaRtBfLP3ldjS9KQWlwZgraRE0+dheEEoAxdzcJQ8eXZg==} + '@protobufjs/fetch@1.1.0': resolution: {integrity: sha512-lljVXpqXebpsijW71PZaCYeIcE5on1w5DlQy5WH6GLbFryLUrBD4932W/E2BSpfRJWseIL4v/KPgBFxDOIdKpQ==} + '@protobufjs/fetch@1.1.1': + resolution: {integrity: sha512-GpptLrs57adMSuHi3VNj0mAF8dwh36LMaYF6XyJ6JMWlVsc+t42tm1HSEDmOs3A8fC9yyeisgLhsTVQokOZ0zw==} + '@protobufjs/float@1.0.2': resolution: {integrity: sha512-Ddb+kVXlXst9d+R9PfTIxh1EdNkgoRe5tOX6t01f1lYWOvJnSPDBlG241QLzcyPdoNTsblLUdujGSE4RzrTZGQ==} @@ -1764,6 +1802,9 @@ packages: '@protobufjs/utf8@1.1.0': resolution: {integrity: sha512-Vvn3zZrhQZkkBE8LSuW3em98c0FwgO4nxzv6OdSxPKJIEKY2bGbHn+mhGIPerzI4twdxaP8/0+06HBpwf345Lw==} + '@protobufjs/utf8@1.1.2': + resolution: {integrity: sha512-b1UQwcEZ4yCnMCD8DAL1VlbvBJE9/IX4FTIp7BG1xYpf29SLazLSrqUkj4w7Y5y7cCVP6E5tcqqcI0xemPkHug==} + '@rolldown/binding-android-arm64@1.0.0-rc.15': resolution: {integrity: sha512-YYe6aWruPZDtHNpwu7+qAHEMbQ/yRl6atqb/AhznLTnD3UY99Q1jE7ihLSahNWkF4EqRPVC4SiR4O0UkLK02tA==} engines: {node: ^20.19.0 || >=22.12.0} @@ -3926,6 +3967,10 @@ packages: resolution: {integrity: sha512-mRUWCc3KUU4w1jU8sGxICXH/gNS94DvI1gxqDvBzhj1JpcsimQkYiOJfwsPUykUI5ZaspFbSgmBLER8IrQ3tqw==} engines: {node: '>=12.0.0'} + protobufjs@7.6.5: + resolution: {integrity: sha512-/FPD0nUc9jH6rfFjji9IBqOz4pcSE3CsT1m7Ep6Mdb0LxSUMj8hgl6GomOvZzpNpAqqGaXA0P3VSrZLFzIhQrw==} + engines: {node: '>=12.0.0'} + pump@3.0.2: resolution: {integrity: sha512-tUPXtzlGM8FE3P0ZL6DVs/3P58k9nk8/jZeQCurTJylQA8qFYzHFfhBJkuqyE0FifOsQ0uKWekiZ5g8wtr28cw==} @@ -5720,6 +5765,11 @@ snapshots: '@grpc/proto-loader': 0.7.13 '@js-sdsl/ordered-map': 4.4.2 + '@grpc/grpc-js@1.14.4': + dependencies: + '@grpc/proto-loader': 0.8.1 + '@js-sdsl/ordered-map': 4.4.2 + '@grpc/proto-loader@0.7.13': dependencies: lodash.camelcase: 4.3.0 @@ -5727,6 +5777,13 @@ snapshots: protobufjs: 7.4.0 yargs: 17.7.2 + '@grpc/proto-loader@0.8.1': + dependencies: + lodash.camelcase: 4.3.0 + long: 5.3.2 + protobufjs: 7.6.5 + yargs: 17.7.2 + '@inquirer/ansi@1.0.2': {} '@inquirer/checkbox@4.3.2(@types/node@25.3.2)': @@ -6059,6 +6116,24 @@ snapshots: utility-types: 3.11.0 yaml: 2.8.1 + '@milaboratories/pl-client@3.14.6': + dependencies: + '@grpc/grpc-js': 1.14.4 + '@milaboratories/pl-http': 1.2.4 + '@milaboratories/pl-model-common': 1.47.3 + '@milaboratories/ts-helpers': 1.8.6 + '@protobuf-ts/grpc-transport': 2.11.1(@grpc/grpc-js@1.14.4) + '@protobuf-ts/runtime': 2.11.1 + '@protobuf-ts/runtime-rpc': 2.11.1 + canonicalize: 2.1.0 + denque: 2.1.0 + long: 5.3.2 + lru-cache: 11.2.4 + openapi-fetch: 0.15.0 + undici: 7.16.0 + utility-types: 3.11.0 + yaml: 2.8.1 + '@milaboratories/pl-config@1.8.5': dependencies: '@milaboratories/ts-helpers': 1.8.6 @@ -6178,6 +6253,12 @@ snapshots: canonicalize: 2.1.0 zod: 3.25.76 + '@milaboratories/pl-model-backend@1.4.20': + dependencies: + '@milaboratories/pl-client': 3.14.6 + canonicalize: 2.1.0 + zod: 3.25.76 + '@milaboratories/pl-model-common@1.23.0': dependencies: '@milaboratories/pl-error-like': 1.12.5 @@ -6199,6 +6280,14 @@ snapshots: es-toolkit: 1.42.0 zod: 3.25.76 + '@milaboratories/pl-model-common@1.47.3': + dependencies: + '@milaboratories/helpers': 1.14.5 + '@milaboratories/pl-error-like': 1.12.10 + canonicalize: 2.1.0 + es-toolkit: 1.42.0 + zod: 3.25.76 + '@milaboratories/pl-model-middle-layer@1.30.14': dependencies: '@milaboratories/helpers': 1.14.5 @@ -6215,6 +6304,14 @@ snapshots: utility-types: 3.11.0 zod: 3.25.76 + '@milaboratories/pl-model-middle-layer@1.31.0': + dependencies: + '@milaboratories/helpers': 1.14.5 + '@milaboratories/pl-model-common': 1.47.3 + es-toolkit: 1.42.0 + utility-types: 3.11.0 + zod: 3.25.76 + '@milaboratories/pl-tree@1.13.2': dependencies: '@milaboratories/computable': 2.9.8 @@ -6854,6 +6951,31 @@ snapshots: - '@types/node' - aws-crt + '@platforma-sdk/block-tools@2.13.0(@types/node@25.3.2)': + dependencies: + '@aws-sdk/client-ecr-public': 3.859.0 + '@aws-sdk/client-s3': 3.859.0 + '@inquirer/prompts': 7.10.1(@types/node@25.3.2) + '@milaboratories/pl-http': 1.2.4 + '@milaboratories/pl-model-backend': 1.4.20 + '@milaboratories/pl-model-common': 1.47.3 + '@milaboratories/pl-model-middle-layer': 1.31.0 + '@milaboratories/resolve-helper': 1.1.3 + '@milaboratories/ts-helpers': 1.8.6 + '@platforma-sdk/blocks-deps-updater': 2.2.0 + '@platforma-sdk/package-builder-lib': 1.3.0 + canonicalize: 2.1.0 + commander: 15.0.0 + lru-cache: 11.2.4 + mime-types: 2.1.35 + tar: 7.4.3 + undici: 7.16.0 + yaml: 2.8.1 + zod: 3.25.76 + transitivePeerDependencies: + - '@types/node' + - aws-crt + '@platforma-sdk/blocks-deps-updater@2.2.0': dependencies: yaml: 2.8.1 @@ -6895,9 +7017,22 @@ snapshots: transitivePeerDependencies: - aws-crt - '@platforma-sdk/tengo-builder@4.0.19': + '@platforma-sdk/package-builder-lib@1.3.0': dependencies: - '@milaboratories/pl-model-backend': 1.4.17 + '@aws-sdk/client-s3': 3.859.0 + '@aws-sdk/lib-storage': 3.859.0(@aws-sdk/client-s3@3.859.0) + '@milaboratories/resolve-helper': 1.1.3 + archiver: 7.0.1 + undici: 7.16.0 + winston: 3.17.0 + yaml: 2.8.1 + zod: 3.25.76 + transitivePeerDependencies: + - aws-crt + + '@platforma-sdk/tengo-builder@4.0.22': + dependencies: + '@milaboratories/pl-model-backend': 1.4.20 '@milaboratories/resolve-helper': 1.1.3 '@milaboratories/tengo-tester': 1.6.4 '@milaboratories/ts-helpers': 1.8.6 @@ -7016,6 +7151,12 @@ snapshots: '@protobuf-ts/runtime': 2.11.1 '@protobuf-ts/runtime-rpc': 2.11.1 + '@protobuf-ts/grpc-transport@2.11.1(@grpc/grpc-js@1.14.4)': + dependencies: + '@grpc/grpc-js': 1.14.4 + '@protobuf-ts/runtime': 2.11.1 + '@protobuf-ts/runtime-rpc': 2.11.1 + '@protobuf-ts/plugin@2.11.1': dependencies: '@bufbuild/protobuf': 2.7.0 @@ -7041,13 +7182,21 @@ snapshots: '@protobufjs/codegen@2.0.4': {} + '@protobufjs/codegen@2.0.5': {} + '@protobufjs/eventemitter@1.1.0': {} + '@protobufjs/eventemitter@1.1.1': {} + '@protobufjs/fetch@1.1.0': dependencies: '@protobufjs/aspromise': 1.1.2 '@protobufjs/inquire': 1.1.0 + '@protobufjs/fetch@1.1.1': + dependencies: + '@protobufjs/aspromise': 1.1.2 + '@protobufjs/float@1.0.2': {} '@protobufjs/inquire@1.1.0': {} @@ -7058,6 +7207,8 @@ snapshots: '@protobufjs/utf8@1.1.0': {} + '@protobufjs/utf8@1.1.2': {} + '@rolldown/binding-android-arm64@1.0.0-rc.15': optional: true @@ -9290,6 +9441,20 @@ snapshots: '@types/node': 25.3.2 long: 5.3.2 + protobufjs@7.6.5: + dependencies: + '@protobufjs/aspromise': 1.1.2 + '@protobufjs/base64': 1.1.2 + '@protobufjs/codegen': 2.0.5 + '@protobufjs/eventemitter': 1.1.1 + '@protobufjs/fetch': 1.1.1 + '@protobufjs/float': 1.0.2 + '@protobufjs/path': 1.1.2 + '@protobufjs/pool': 1.1.0 + '@protobufjs/utf8': 1.1.2 + '@types/node': 25.3.2 + long: 5.3.2 + pump@3.0.2: dependencies: end-of-stream: 1.4.4 diff --git a/pnpm-workspace.yaml b/pnpm-workspace.yaml index 3a8102f..bce81ef 100644 --- a/pnpm-workspace.yaml +++ b/pnpm-workspace.yaml @@ -12,11 +12,11 @@ catalog: "@milaboratories/ts-configs": 1.3.1 typescript: ~5.9.3 "@platforma-sdk/workflow-tengo": 6.8.2 - "@platforma-sdk/block-tools": 2.12.8 + "@platforma-sdk/block-tools": 2.13.0 "@platforma-sdk/model": 1.80.8 "@platforma-sdk/ui-vue": 1.80.9 "@platforma-sdk/test": 1.80.9 - "@platforma-sdk/tengo-builder": 4.0.19 + "@platforma-sdk/tengo-builder": 4.0.22 "@platforma-sdk/package-builder": 3.14.2 "@platforma-sdk/blocks-deps-updater": 2.2.0 From a6e5c4d1968cae07b0d90bea352d5388e7f84e11 Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Mon, 17 Aug 2026 22:10:44 +0200 Subject: [PATCH 064/282] MILAB-6496: bump every catalog dependency that can move Nine entries forward: ts-builder 1.6.2, ts-configs 1.4.0, SDK model 1.81.1, ui-vue 1.81.1, test 1.81.3, package-builder 3.15.0, runenv-python-3 1.11.6, turbo 2.10.10, vitest 4.1.10. Build 10/10, 320 python tests, 19 tengo tests. Four held, each for a stated reason rather than by omission: vue stays at 3.5.24. Bumping it to 3.5.41 broke six type checks across five pages, including two that predate this branch -- the ui-vue plugin type and the data-table settings prop. Reverting vue alone cleared every one of them, so the SDK bumps are sound and the vue minor has to match what ui-vue was built against. That pin is structurer-managed. samples-and-data stays pinned exact at 1.13.3 with its model at 1.11.2. The catalog comment states why: 1.14+ are V3 and reject setBlockArgs, so resolving forward breaks the integration tests. @changesets/cli stays at 2.29.8. Version 3.0.0 requires pnpm >= 10 and this workspace runs 9.15. It is also the tool that gates the release, so a major on it is not something to discover during one. software-mitool stays at its main build; it is not a released version to bump to. --- pnpm-lock.yaml | 1638 +++++++++++++------------------------------ pnpm-workspace.yaml | 18 +- 2 files changed, 497 insertions(+), 1159 deletions(-) diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 23917ec..15e2371 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -13,14 +13,14 @@ catalogs: specifier: 1.14.5 version: 1.14.5 '@milaboratories/ts-builder': - specifier: 1.6.1 - version: 1.6.1 + specifier: 1.6.2 + version: 1.6.2 '@milaboratories/ts-configs': - specifier: 1.3.1 - version: 1.3.1 + specifier: 1.4.0 + version: 1.4.0 '@platforma-open/milaboratories.runenv-python-3': - specifier: 1.7.8 - version: 1.7.8 + specifier: 1.11.6 + version: 1.11.6 '@platforma-open/milaboratories.samples-and-data': specifier: 1.13.3 version: 1.13.3 @@ -34,17 +34,17 @@ catalogs: specifier: 2.13.0 version: 2.13.0 '@platforma-sdk/model': - specifier: 1.80.8 - version: 1.80.8 + specifier: 1.81.1 + version: 1.81.1 '@platforma-sdk/tengo-builder': specifier: 4.0.22 version: 4.0.22 '@platforma-sdk/test': - specifier: 1.80.9 - version: 1.80.9 + specifier: 1.81.3 + version: 1.81.3 '@platforma-sdk/ui-vue': - specifier: 1.80.9 - version: 1.80.9 + specifier: 1.81.1 + version: 1.81.1 '@platforma-sdk/workflow-tengo': specifier: 6.8.2 version: 6.8.2 @@ -58,14 +58,14 @@ catalogs: specifier: 0.4.0 version: 0.4.0 turbo: - specifier: 2.8.11 - version: 2.8.11 + specifier: 2.10.10 + version: 2.10.10 typescript: specifier: ~5.9.3 version: 5.9.3 vitest: - specifier: ~4.0.18 - version: 4.0.18 + specifier: ~4.1.10 + version: 4.1.10 vue: specifier: 3.5.24 version: 3.5.24 @@ -86,7 +86,7 @@ importers: version: 2.29.8(@types/node@25.3.2) '@milaboratories/ts-builder': specifier: 'catalog:' - version: 1.6.1(@types/node@25.3.2)(rollup@4.53.3)(vue@3.5.25(typescript@5.9.3))(yaml@2.8.1) + version: 1.6.2(@types/node@25.3.2)(rollup@4.53.3)(vue@3.5.41(typescript@5.9.3))(yaml@2.8.1) '@platforma-sdk/block-tools': specifier: 'catalog:' version: 2.13.0(@types/node@25.3.2) @@ -95,16 +95,16 @@ importers: version: 0.4.0 turbo: specifier: 'catalog:' - version: 2.8.11 + version: 2.10.10 block: devDependencies: '@milaboratories/ts-builder': specifier: 'catalog:' - version: 1.6.1(@types/node@25.3.2)(rollup@4.53.3)(vue@3.5.25(typescript@5.9.3))(yaml@2.8.1) + version: 1.6.2(@types/node@25.3.2)(rollup@4.53.3)(vue@3.5.41(typescript@5.9.3))(yaml@2.8.1) '@milaboratories/ts-configs': specifier: 'catalog:' - version: 1.3.1 + version: 1.4.0 '@platforma-open/milaboratories.feature-integration.model': specifier: workspace:* version: link:../model @@ -119,7 +119,7 @@ importers: version: 2.13.0(@types/node@25.3.2) '@platforma-sdk/model': specifier: 'catalog:' - version: 1.80.8 + version: 1.81.1 shx: specifier: 'catalog:' version: 0.4.0 @@ -134,7 +134,7 @@ importers: version: 1.14.5 '@platforma-sdk/model': specifier: 'catalog:' - version: 1.80.8 + version: 1.81.1 '@types/node': specifier: '*' version: 25.3.2 @@ -144,10 +144,10 @@ importers: devDependencies: '@milaboratories/ts-builder': specifier: 'catalog:' - version: 1.6.1(@types/node@25.3.2)(rollup@4.53.3)(vue@3.5.25(typescript@5.6.3))(yaml@2.8.1) + version: 1.6.2(@types/node@25.3.2)(rollup@4.53.3)(vue@3.5.41(typescript@5.6.3))(yaml@2.8.1) '@milaboratories/ts-configs': specifier: 'catalog:' - version: 1.3.1 + version: 1.4.0 '@platforma-sdk/block-tools': specifier: 'catalog:' version: 2.13.0(@types/node@25.3.2) @@ -156,7 +156,7 @@ importers: devDependencies: '@platforma-open/milaboratories.runenv-python-3': specifier: 'catalog:' - version: 1.7.8 + version: 1.11.6 '@platforma-sdk/block-tools': specifier: 'catalog:' version: 2.13.0(@types/node@25.3.2) @@ -174,7 +174,7 @@ importers: version: 1.11.2 '@platforma-sdk/model': specifier: 'catalog:' - version: 1.80.8 + version: 1.81.1 this-block: specifier: workspace:@platforma-open/milaboratories.feature-integration@* version: link:../block @@ -184,16 +184,16 @@ importers: devDependencies: '@milaboratories/ts-builder': specifier: 'catalog:' - version: 1.6.1(@types/node@25.3.2)(rollup@4.53.3)(vue@3.5.25(typescript@5.6.3))(yaml@2.8.1) + version: 1.6.2(@types/node@25.3.2)(rollup@4.53.3)(vue@3.5.41(typescript@5.6.3))(yaml@2.8.1) '@milaboratories/ts-configs': specifier: 'catalog:' - version: 1.3.1 + version: 1.4.0 '@platforma-sdk/test': specifier: 'catalog:' - version: 1.80.9(@bytecodealliance/preview2-shim@0.17.8)(@types/node@25.3.2)(vite@7.2.7(@types/node@25.3.2)(lightningcss@1.32.0)(yaml@2.8.1)) + version: 1.81.3(@bytecodealliance/preview2-shim@0.17.8)(@types/node@25.3.2)(vite@8.0.8(@types/node@25.3.2)(yaml@2.8.1)) vitest: specifier: 'catalog:' - version: 4.0.18(@types/node@25.3.2)(lightningcss@1.32.0)(yaml@2.8.1) + version: 4.1.10(@types/node@25.3.2)(@vitest/coverage-istanbul@4.1.4)(vite@8.0.8(@types/node@25.3.2)(yaml@2.8.1)) ui: dependencies: @@ -202,7 +202,7 @@ importers: version: link:../model '@platforma-sdk/ui-vue': specifier: 'catalog:' - version: 1.80.9(@bytecodealliance/preview2-shim@0.17.8)(typescript@5.6.3) + version: 1.81.1(@bytecodealliance/preview2-shim@0.17.8)(typescript@5.6.3) '@types/node': specifier: '*' version: 25.3.2 @@ -221,10 +221,10 @@ importers: devDependencies: '@milaboratories/ts-builder': specifier: 'catalog:' - version: 1.6.1(@types/node@25.3.2)(rollup@4.53.3)(vue@3.5.24(typescript@5.6.3))(yaml@2.8.1) + version: 1.6.2(@types/node@25.3.2)(rollup@4.53.3)(vue@3.5.24(typescript@5.6.3))(yaml@2.8.1) '@milaboratories/ts-configs': specifier: 'catalog:' - version: 1.3.1 + version: 1.4.0 workflow: dependencies: @@ -243,7 +243,7 @@ importers: version: 4.0.22 '@platforma-sdk/test': specifier: 'catalog:' - version: 1.80.9(@bytecodealliance/preview2-shim@0.17.8)(@types/node@25.3.2)(vite@8.0.8(@types/node@25.3.2)(yaml@2.8.1)) + version: 1.81.3(@bytecodealliance/preview2-shim@0.17.8)(@types/node@25.3.2)(vite@8.0.8(@types/node@25.3.2)(yaml@2.8.1)) shx: specifier: 'catalog:' version: 0.4.0 @@ -511,6 +511,10 @@ packages: resolution: {integrity: sha512-qMlSxKbpRlAridDExk92nSobyDdpPijUq2DW6oDnUqd0iOGxmQjyqhMIihI9+zv4LPyZdRje2cavWPbCbWm3eA==} engines: {node: '>=6.9.0'} + '@babel/helper-string-parser@7.29.7': + resolution: {integrity: sha512-Pb5ijPrZ89GDH8223L4UP8i6QApWxs04RbPQJTeWDV0/keR2E36MeKnyr6LYmUUvqRRI+Iv87SuF1W6ErINzYw==} + engines: {node: '>=6.9.0'} + '@babel/helper-string-parser@8.0.0': resolution: {integrity: sha512-6mJgmFFFIIO82vvoLt9XtRC7/TkzXfts1t/SpRX4IHSzMgqoPYCWesVu1udUPUWioAE/2fcG6WuI8zrkE1gwrg==} engines: {node: ^22.18.0 || >=24.11.0} @@ -519,6 +523,10 @@ packages: resolution: {integrity: sha512-qSs4ifwzKJSV39ucNjsvc6WVHs6b7S03sOh2OcHF9UHfVPqWWALUsNUVzhSBiItjRZoLHx7nIarVjqKVusUZ1Q==} engines: {node: '>=6.9.0'} + '@babel/helper-validator-identifier@7.29.7': + resolution: {integrity: sha512-qehxGkRj55h/ff8EMaJ+cYhyaKlHIxqYDn682wQD7RNp9UujOQsHog2uS0r2vzr4pW+sXf90NeeayjcNaX3fFg==} + engines: {node: '>=6.9.0'} + '@babel/helper-validator-identifier@8.0.2': resolution: {integrity: sha512-9Fr9QeyCAyi1BR1jKZ6uYQ24EIhQUx5ReHfQU7drOE+TPOb+w11/dsqLkMOT2U29OdCT71XajrOT8xDc1C7orA==} engines: {node: ^22.18.0 || >=24.11.0} @@ -536,6 +544,11 @@ packages: engines: {node: '>=6.0.0'} hasBin: true + '@babel/parser@7.29.8': + resolution: {integrity: sha512-E8lTAYNB1KW+FH+VGJuZM1ioAx2E6oVlvQFRrf5P8ZZmsiJXYAD9vTFV7yyEURNzgh1dFqMZuO6tUwcARbqFCA==} + engines: {node: '>=6.0.0'} + hasBin: true + '@babel/parser@8.0.0': resolution: {integrity: sha512-aLxAE+imI9bCcyaPrUDjBv3uSkWieifjLe0kuFOZF0zli0L6GCsTmsePnTr55adbIAgYz2zhN1vnFimCBUYcRQ==} engines: {node: ^22.18.0 || >=24.11.0} @@ -557,6 +570,10 @@ packages: resolution: {integrity: sha512-LwdZHpScM4Qz8Xw2iKSzS+cfglZzJGvofQICy7W7v4caru4EaAmyUuO6BGrbyQ2mYV11W0U8j5mBhd14dd3B0A==} engines: {node: '>=6.9.0'} + '@babel/types@7.29.8': + resolution: {integrity: sha512-Vj1jF3cPfxg7OAfoI7QnVKLoILlm2JF9pnVHrX8qx7AHMiYWT+NDAA7jChlNgRS4WTLc/fD1lXLmPixluj+3Gg==} + engines: {node: '>=6.9.0'} + '@babel/types@8.0.0': resolution: {integrity: sha512-K8ponJDxBwDHigkeFqaqT5wLGl4bTlwMafR8k7b5CPxr6Ww+UG9ls8Yx6Tcpboxu97eeGVEEyKcHmEyOwN1vSw==} engines: {node: ^22.18.0 || >=24.11.0} @@ -650,175 +667,10 @@ packages: '@emnapi/wasi-threads@1.2.2': resolution: {integrity: sha512-c95qOXkHdydNKhscBTebqEC1CVAZpyqOfVfBzQ1qgzyl3gfeldUjIggDbIZgDKsHLgnsM+igH7TJ/eAasaVuMA==} - '@esbuild/aix-ppc64@0.25.12': - resolution: {integrity: sha512-Hhmwd6CInZ3dwpuGTF8fJG6yoWmsToE+vYgD4nytZVxcu1ulHpUQRAB1UJ8+N1Am3Mz4+xOByoQoSZf4D+CpkA==} - engines: {node: '>=18'} - cpu: [ppc64] - os: [aix] - - '@esbuild/android-arm64@0.25.12': - resolution: {integrity: sha512-6AAmLG7zwD1Z159jCKPvAxZd4y/VTO0VkprYy+3N2FtJ8+BQWFXU+OxARIwA46c5tdD9SsKGZ/1ocqBS/gAKHg==} - engines: {node: '>=18'} - cpu: [arm64] - os: [android] - - '@esbuild/android-arm@0.25.12': - resolution: {integrity: sha512-VJ+sKvNA/GE7Ccacc9Cha7bpS8nyzVv0jdVgwNDaR4gDMC/2TTRc33Ip8qrNYUcpkOHUT5OZ0bUcNNVZQ9RLlg==} - engines: {node: '>=18'} - cpu: [arm] - os: [android] - - '@esbuild/android-x64@0.25.12': - resolution: {integrity: sha512-5jbb+2hhDHx5phYR2By8GTWEzn6I9UqR11Kwf22iKbNpYrsmRB18aX/9ivc5cabcUiAT/wM+YIZ6SG9QO6a8kg==} - engines: {node: '>=18'} - cpu: [x64] - os: [android] - - '@esbuild/darwin-arm64@0.25.12': - resolution: {integrity: sha512-N3zl+lxHCifgIlcMUP5016ESkeQjLj/959RxxNYIthIg+CQHInujFuXeWbWMgnTo4cp5XVHqFPmpyu9J65C1Yg==} - engines: {node: '>=18'} - cpu: [arm64] - os: [darwin] - - '@esbuild/darwin-x64@0.25.12': - resolution: {integrity: sha512-HQ9ka4Kx21qHXwtlTUVbKJOAnmG1ipXhdWTmNXiPzPfWKpXqASVcWdnf2bnL73wgjNrFXAa3yYvBSd9pzfEIpA==} - engines: {node: '>=18'} - cpu: [x64] - os: [darwin] - - '@esbuild/freebsd-arm64@0.25.12': - resolution: {integrity: sha512-gA0Bx759+7Jve03K1S0vkOu5Lg/85dou3EseOGUes8flVOGxbhDDh/iZaoek11Y8mtyKPGF3vP8XhnkDEAmzeg==} - engines: {node: '>=18'} - cpu: [arm64] - os: [freebsd] - - '@esbuild/freebsd-x64@0.25.12': - resolution: {integrity: sha512-TGbO26Yw2xsHzxtbVFGEXBFH0FRAP7gtcPE7P5yP7wGy7cXK2oO7RyOhL5NLiqTlBh47XhmIUXuGciXEqYFfBQ==} - engines: {node: '>=18'} - cpu: [x64] - os: [freebsd] - - '@esbuild/linux-arm64@0.25.12': - resolution: {integrity: sha512-8bwX7a8FghIgrupcxb4aUmYDLp8pX06rGh5HqDT7bB+8Rdells6mHvrFHHW2JAOPZUbnjUpKTLg6ECyzvas2AQ==} - engines: {node: '>=18'} - cpu: [arm64] - os: [linux] - - '@esbuild/linux-arm@0.25.12': - resolution: {integrity: sha512-lPDGyC1JPDou8kGcywY0YILzWlhhnRjdof3UlcoqYmS9El818LLfJJc3PXXgZHrHCAKs/Z2SeZtDJr5MrkxtOw==} - engines: {node: '>=18'} - cpu: [arm] - os: [linux] - - '@esbuild/linux-ia32@0.25.12': - resolution: {integrity: sha512-0y9KrdVnbMM2/vG8KfU0byhUN+EFCny9+8g202gYqSSVMonbsCfLjUO+rCci7pM0WBEtz+oK/PIwHkzxkyharA==} - engines: {node: '>=18'} - cpu: [ia32] - os: [linux] - - '@esbuild/linux-loong64@0.25.12': - resolution: {integrity: sha512-h///Lr5a9rib/v1GGqXVGzjL4TMvVTv+s1DPoxQdz7l/AYv6LDSxdIwzxkrPW438oUXiDtwM10o9PmwS/6Z0Ng==} - engines: {node: '>=18'} - cpu: [loong64] - os: [linux] - - '@esbuild/linux-mips64el@0.25.12': - resolution: {integrity: sha512-iyRrM1Pzy9GFMDLsXn1iHUm18nhKnNMWscjmp4+hpafcZjrr2WbT//d20xaGljXDBYHqRcl8HnxbX6uaA/eGVw==} - engines: {node: '>=18'} - cpu: [mips64el] - os: [linux] - - '@esbuild/linux-ppc64@0.25.12': - resolution: {integrity: sha512-9meM/lRXxMi5PSUqEXRCtVjEZBGwB7P/D4yT8UG/mwIdze2aV4Vo6U5gD3+RsoHXKkHCfSxZKzmDssVlRj1QQA==} - engines: {node: '>=18'} - cpu: [ppc64] - os: [linux] - - '@esbuild/linux-riscv64@0.25.12': - resolution: {integrity: sha512-Zr7KR4hgKUpWAwb1f3o5ygT04MzqVrGEGXGLnj15YQDJErYu/BGg+wmFlIDOdJp0PmB0lLvxFIOXZgFRrdjR0w==} - engines: {node: '>=18'} - cpu: [riscv64] - os: [linux] - - '@esbuild/linux-s390x@0.25.12': - resolution: {integrity: sha512-MsKncOcgTNvdtiISc/jZs/Zf8d0cl/t3gYWX8J9ubBnVOwlk65UIEEvgBORTiljloIWnBzLs4qhzPkJcitIzIg==} - engines: {node: '>=18'} - cpu: [s390x] - os: [linux] - - '@esbuild/linux-x64@0.25.12': - resolution: {integrity: sha512-uqZMTLr/zR/ed4jIGnwSLkaHmPjOjJvnm6TVVitAa08SLS9Z0VM8wIRx7gWbJB5/J54YuIMInDquWyYvQLZkgw==} - engines: {node: '>=18'} - cpu: [x64] - os: [linux] - - '@esbuild/netbsd-arm64@0.25.12': - resolution: {integrity: sha512-xXwcTq4GhRM7J9A8Gv5boanHhRa/Q9KLVmcyXHCTaM4wKfIpWkdXiMog/KsnxzJ0A1+nD+zoecuzqPmCRyBGjg==} - engines: {node: '>=18'} - cpu: [arm64] - os: [netbsd] - - '@esbuild/netbsd-x64@0.25.12': - resolution: {integrity: sha512-Ld5pTlzPy3YwGec4OuHh1aCVCRvOXdH8DgRjfDy/oumVovmuSzWfnSJg+VtakB9Cm0gxNO9BzWkj6mtO1FMXkQ==} - engines: {node: '>=18'} - cpu: [x64] - os: [netbsd] - - '@esbuild/openbsd-arm64@0.25.12': - resolution: {integrity: sha512-fF96T6KsBo/pkQI950FARU9apGNTSlZGsv1jZBAlcLL1MLjLNIWPBkj5NlSz8aAzYKg+eNqknrUJ24QBybeR5A==} - engines: {node: '>=18'} - cpu: [arm64] - os: [openbsd] - - '@esbuild/openbsd-x64@0.25.12': - resolution: {integrity: sha512-MZyXUkZHjQxUvzK7rN8DJ3SRmrVrke8ZyRusHlP+kuwqTcfWLyqMOE3sScPPyeIXN/mDJIfGXvcMqCgYKekoQw==} - engines: {node: '>=18'} - cpu: [x64] - os: [openbsd] - - '@esbuild/openharmony-arm64@0.25.12': - resolution: {integrity: sha512-rm0YWsqUSRrjncSXGA7Zv78Nbnw4XL6/dzr20cyrQf7ZmRcsovpcRBdhD43Nuk3y7XIoW2OxMVvwuRvk9XdASg==} - engines: {node: '>=18'} - cpu: [arm64] - os: [openharmony] - - '@esbuild/sunos-x64@0.25.12': - resolution: {integrity: sha512-3wGSCDyuTHQUzt0nV7bocDy72r2lI33QL3gkDNGkod22EsYl04sMf0qLb8luNKTOmgF/eDEDP5BFNwoBKH441w==} - engines: {node: '>=18'} - cpu: [x64] - os: [sunos] - - '@esbuild/win32-arm64@0.25.12': - resolution: {integrity: sha512-rMmLrur64A7+DKlnSuwqUdRKyd3UE7oPJZmnljqEptesKM8wx9J8gx5u0+9Pq0fQQW8vqeKebwNXdfOyP+8Bsg==} - engines: {node: '>=18'} - cpu: [arm64] - os: [win32] - - '@esbuild/win32-ia32@0.25.12': - resolution: {integrity: sha512-HkqnmmBoCbCwxUKKNPBixiWDGCpQGVsrQfJoVGYLPT41XWF8lHuE5N6WhVia2n4o5QK5M4tYr21827fNhi4byQ==} - engines: {node: '>=18'} - cpu: [ia32] - os: [win32] - - '@esbuild/win32-x64@0.25.12': - resolution: {integrity: sha512-alJC0uCZpTFrSL0CCDjcgleBXPnCrEAhTBILpeAp7M/OFgoqtAetfBzX0xM00MUsVVPpVjlPuMbREqnZCXaTnA==} - engines: {node: '>=18'} - cpu: [x64] - os: [win32] - - '@grpc/grpc-js@1.13.4': - resolution: {integrity: sha512-GsFaMXCkMqkKIvwCQjCrwH+GHbPKBjhwo/8ZuUkWHqbI73Kky9I+pQltrlT0+MWpedCoosda53lgjYfyEPgxBg==} - engines: {node: '>=12.10.0'} - '@grpc/grpc-js@1.14.4': resolution: {integrity: sha512-k9Dj3DV/itK9D06Y8f190Qgop7/Ui+D0njFV3LHMPwPT75DpXLQohE9Wmz0QElrJnzsjB7KPWiKJbOl7IPDArQ==} engines: {node: '>=12.10.0'} - '@grpc/proto-loader@0.7.13': - resolution: {integrity: sha512-AiXO/bfe9bmxBjxxtYxFAXGZvMaN5s8kO+jBHAJCON8rJoB5YS/D6X7ZNc6XQkuHNmyl4CYaMI1fJ/Gn27RGGw==} - engines: {node: '>=6'} - hasBin: true - '@grpc/proto-loader@0.8.1': resolution: {integrity: sha512-wtF6h+DY6M3YaDBPAmvuuA6jV8Sif9MjtOI5euKFWRgCDl5PeDpPsHR9u2l6St5ceY8AZgoNDww5+HvEsXFsGg==} engines: {node: '>=6'} @@ -970,20 +822,20 @@ packages: resolution: {integrity: sha512-ZXRY4jNvVgSVQ8DL3LTcakaAtXwTVUxE81hslsyD2AtoXW/wVob10HkOJ1X/pAlcI7D+2YoZKg5do8G/w6RYgA==} engines: {node: '>=8'} - '@jitl/quickjs-ffi-types@0.31.0': - resolution: {integrity: sha512-1yrgvXlmXH2oNj3eFTrkwacGJbmM0crwipA3ohCrjv52gBeDaD7PsTvFYinlAnqU8iPME3LGP437yk05a2oejw==} + '@jitl/quickjs-ffi-types@0.32.0': + resolution: {integrity: sha512-v9T+GQpmk43VDJ7d72sf0Nexhk+ArvtUihW27dy7lqAl0zBObFKtSBBIm5RBjwIhE8VwsPPm9PNuvPvNqLWUEg==} - '@jitl/quickjs-wasmfile-debug-asyncify@0.31.0': - resolution: {integrity: sha512-YkdzQdr1uaftFhgEnTRjTTZHk2SFZdpWO7XhOmRVbi6CEVsH9g5oNF8Ta1q3OuSJHRwwT8YsuR1YzEiEIJEk6w==} + '@jitl/quickjs-wasmfile-debug-asyncify@0.32.0': + resolution: {integrity: sha512-EX8zbXwGqCgAE764M+qvkHtyXDi/FUoMBea0JnES7vCM3P7a2+EOZOjGv85wtZ2sJhI1oJ+nekmqpOODFDY+hw==} - '@jitl/quickjs-wasmfile-debug-sync@0.31.0': - resolution: {integrity: sha512-8XvloaaWBONqcHXYs5tWOjdhQVxzULilIfB2hvZfS6S+fI4m2+lFiwQy7xeP8ExHmiZ7D8gZGChNkdLgjGfknw==} + '@jitl/quickjs-wasmfile-debug-sync@0.32.0': + resolution: {integrity: sha512-LeYWrPGC1uNCTBWvibo3ZLJj0CSVNYUXvJpXMCmuQ5Sap2cCACc3uvGvYV4homHHBAzfw5akoTqMMS4YFRtw+Q==} - '@jitl/quickjs-wasmfile-release-asyncify@0.31.0': - resolution: {integrity: sha512-uz0BbQYTxNsFkvkurd7vk2dOg57ElTBLCuvNtRl4rgrtbC++NIndD5qv2+AXb6yXDD3Uy1O2PCwmoaH0eXgEOg==} + '@jitl/quickjs-wasmfile-release-asyncify@0.32.0': + resolution: {integrity: sha512-3oSwPfja12ICz4aIblB58cuY8JlEq5Txt8Cut4VLo+LH47QN+mzCnSgnbB03hWzg1LBcc+VyyI9UOag7a1NF+Q==} - '@jitl/quickjs-wasmfile-release-sync@0.31.0': - resolution: {integrity: sha512-hYduecOByj9AsAfsJhZh5nA6exokmuFC8cls39+lYmTCGY51bgjJJJwReEu7Ff7vBWaQCL6TeDdVlnp2WYz0jw==} + '@jitl/quickjs-wasmfile-release-sync@0.32.0': + resolution: {integrity: sha512-BKNDI/TPBfGlLNGYpLrhcDGXmIk4xHm4MRAisOBnOzpXVn9HZWsfmMAc9WMBrAHjvvds6HOikKeaOBKdPdpVrg==} '@jridgewell/gen-mapping@0.3.13': resolution: {integrity: sha512-2kkt/7niJ6MgEPxF0bYdQ6etZaA+fQvDcLKckhy1yIQOzaoKjBBjSj63/aLVjYE3qhRt5dvM+uUyfCg6UKCBbA==} @@ -1028,8 +880,8 @@ packages: '@microsoft/tsdoc@0.16.0': resolution: {integrity: sha512-xgAyonlVVS+q7Vc7qLW0UrJU7rSFcETRWsqdXZtjzRU8dF+6CkozTK4V4y1LwOX7j8r/vHphjDeMeGI4tNGeGA==} - '@milaboratories/columns-collection-driver@0.2.2': - resolution: {integrity: sha512-wezb0Fu7iqwykw8ZRwMge72qXVr+eH2UEYsYGm7X38KQ3DVim2eWw3bKW+IcooFt/X18egsN8j4MPxUxUuVxcw==} + '@milaboratories/columns-collection-driver@0.2.3': + resolution: {integrity: sha512-3rNWmuQGvEaBEzGMHIWeOi1p8j8sMDpnPKNUQvT0Ji7/ArvnfM5HK8dywxB+n3mLK8c11lW16+auM5xuydJXGw==} '@milaboratories/computable@2.9.8': resolution: {integrity: sha512-X0ZtxOnIJlAd9Y7CyBtWSKHgVuTKXbIryMwJaoYSEz0gY3JZVnsD0f59J/lwe3Key0/lSYh3EbL/pP5jACIzfQ==} @@ -1043,12 +895,17 @@ packages: resolution: {integrity: sha512-Kiy0g7sEmFQxDwtnmTrdJ8XdUK0IDAB5ZnPCNEbK7lPGHIa+xG3kzfeR4y44QBdrYaK0NwG63D8Fc7dlv9KItg==} engines: {node: '>=22'} - '@milaboratories/pf-driver@1.8.4': - resolution: {integrity: sha512-ouTnOC+QfAF5AHLw6vAOoxwl/Kr8a3YvvBAmo5FstmVTzL57laVIu/QGpPTNBwwW5adUT7cv7Uou09c3t7uayw==} + '@milaboratories/pf-driver@1.9.0': + resolution: {integrity: sha512-OhMsf1jNMp31wRCrXHMU75WQj48Nc/lDeYTo4/6R6+FLYBYkK9yWgsSH+02V4WCKbUqsyF1Xy0+O+qsYjpqUDA==} engines: {node: '>=22.19.0'} - '@milaboratories/pf-spec-driver@1.4.23': - resolution: {integrity: sha512-/madO+vexJH92Utbz5Nh7rxKz9SnYUPZlsLZInWrMAJaXrIqcYZnCS1h2kBcNWJ4/9/YO/si7XtM1EDAmvImWw==} + '@milaboratories/pf-spec-driver@1.5.0': + resolution: {integrity: sha512-XWXrKCyCwcqS8EcukuyDqvMq0D6qt61zrkb3qGpWKyy0SVQNzXC2JL0ThCM2uGI7QQelgV0aagcg8xU5pGKWPw==} + + '@milaboratories/pf-spec@1.0.0': + resolution: {integrity: sha512-JIOhJdDHcKDItjlxOT4wVmnjejoazM69qK5ylu+g5q65RIcH/R5MF2OGiu9nV7QbLEXszll+EVnvx6jYhuE44w==} + peerDependencies: + '@bytecodealliance/preview2-shim': ^0.20.1 '@milaboratories/pframes-rs-node@1.1.56': resolution: {integrity: sha512-H6qltcR+HHb2kAy0U0zP90rqmv/MZGGkdXIyf89FUZTpoHOlXG4Ahxq7wXp9vviUczci4cLl2L0Q+dL2XGcsUA==} @@ -1060,17 +917,6 @@ packages: '@milaboratories/pframes-rs-wasip2@1.1.56': resolution: {integrity: sha512-54bhC6XCAVO09J/sqVwEKA4hhTa27BDDNdH8BE+6+LvjBVkg/qq2/f0MzdrVijrmagbr97F1zgj5wIgUqwCSoA==} - '@milaboratories/pframes-rs-wasm@1.1.56': - resolution: {integrity: sha512-k/RQqiF+SwoOtFnYQ+i34Lzna8prOoFbLHaPY5oEUOrB/czehosFMzRQJAeFDKOxI/SbrH4D5jmWzTgUuhavLQ==} - peerDependencies: - '@bytecodealliance/preview2-shim': 0.17.9 - '@milaboratories/pl-model-common': 1.46.2 - '@milaboratories/pl-model-middle-layer': 1.30.7 - - '@milaboratories/pl-client@3.14.3': - resolution: {integrity: sha512-yyep+I0FozB1pbf22maRCVjgrO3AEB8f88PyTmJM0K7Y5bfq0T/MgprTyPwMB34u1XfKI9M5yqHWmyuYdzq08w==} - engines: {node: '>=22.19.0'} - '@milaboratories/pl-client@3.14.6': resolution: {integrity: sha512-IPzh5bSibr02DMKpwlCW6TuHVxaDe2o6tjaEJ4P2y5lshRbe5SBR/ne/G/N7E7KEKxmpWq7ZapOCLkLyvqFFDw==} engines: {node: '>=22.19.0'} @@ -1078,12 +924,12 @@ packages: '@milaboratories/pl-config@1.8.5': resolution: {integrity: sha512-XnfYXSSkRxeImQ21k6I8y5apisvagcSgMGfEeyRxNdSGwUVJbbI8TwJk+XBEDQ4lErW8oqtLxKjFQuHJMzRUoQ==} - '@milaboratories/pl-deployments@3.0.13': - resolution: {integrity: sha512-uoM1fX6D8/3y6DmHHCj5A3aZVrmir5bZk+AgfdMp/QgCTwKyhvvE4uGUBMn27/uEJk54kFJ15KathxBA2UlD9A==} + '@milaboratories/pl-deployments@3.0.15': + resolution: {integrity: sha512-+i3bdvfPCXLZ789f5fRdtrbil3/RFyzSBNMkULcnyKY5u6GRZLjWDBC5aSE75PRTWx3Fxf743Fj9L3B6loMSWw==} engines: {node: '>=22.19.0'} - '@milaboratories/pl-drivers@1.16.11': - resolution: {integrity: sha512-lCmCIAtjybdJHW3ZBJvoDzbY9jyZQZeL/OawM9MqKOXBwZQy+E2Bi7G5IHJqG13hy8i6FvJQs2/IaWxRpfuInQ==} + '@milaboratories/pl-drivers@1.16.15': + resolution: {integrity: sha512-mVKIEHVOeAPdwSZZk+URAFD6zoyPiAARinpDfihDvkFcrlTLIzeoOvnQNPG5a8jqc0GiW+a9mnTrfjG7+n82Ww==} engines: {node: '>=22'} '@milaboratories/pl-error-like@1.12.10': @@ -1092,23 +938,20 @@ packages: '@milaboratories/pl-error-like@1.12.5': resolution: {integrity: sha512-opYP4OrB6JBMsH9RMRmAH44+MG7PWiV08dHW9+RsXGOaqX+rYXs9TTBXYRhlVMDLwwefSKzelvDg8HL748aM+A==} - '@milaboratories/pl-errors@1.4.32': - resolution: {integrity: sha512-57bft0ieS2sYek6g+FiF5FMeBV1cyVgOchI8sfJAtc2gTlTH5Py8m/z13ZweV4+mGmIGDLNakzQvYGlgloCDmg==} + '@milaboratories/pl-errors@1.4.35': + resolution: {integrity: sha512-rq5Lg9G7ax+LtMPdrkZg+Pdqk28JViiS/KIcQfk0Cg1ttZoAnhDhFIh6rAFy2eGybPtlzXqM32/ODKUSMawc8Q==} - '@milaboratories/pl-healthcheck@1.0.4': - resolution: {integrity: sha512-VLoF7iW7px8BG+vTT/nQ+qkJDNSsXUivRsyZlbM7VCxKdVrZwPfn/rqQIJ4g6daBONVtCqUhAFi52IeXRg5mxg==} + '@milaboratories/pl-healthcheck@1.0.5': + resolution: {integrity: sha512-ZkQti4VU2FapJBFRtZGfR5NDPXEAEFgTf/NfemypgY0aA75fFPi4/c3BpXX5K7dht+JOgHqkMrBHxKIuE2T+fA==} engines: {node: '>=22.19.0'} '@milaboratories/pl-http@1.2.4': resolution: {integrity: sha512-QKmhx+WEvJCV9dUy/SBdQk/ApaJ5ewBFgm/b+XPlS10SusAdqUUTGvK5+hq8YSuUMXlHb/dk++UtI5YlDuDl2Q==} - '@milaboratories/pl-middle-layer@1.66.8': - resolution: {integrity: sha512-VJ0t6g7x917xGN7qKvG5KHcezmzp/6b/+/Wer+QLLJueEb5gmMpWvjtLyfjNbJG8v2X85IhUQKdXETVsR3l2nw==} + '@milaboratories/pl-middle-layer@1.66.19': + resolution: {integrity: sha512-rFBmJDdBcIrdDHz730p3sL8+7uUs7VimXYGQo9IbayxmAlp5mnft1XlPUTuVbwx5QeRNGv4Cwya1T+7WL1wSmw==} engines: {node: '>=22.19.0'} - '@milaboratories/pl-model-backend@1.4.17': - resolution: {integrity: sha512-ZoxNLmWs+MQxCU/s4ml5DHV9htu0xk9NHYQ+07AtWNhLRCjSeFcVTVQlt0EvFMlzytOKGjmMbCMhaKYUuh/wrA==} - '@milaboratories/pl-model-backend@1.4.20': resolution: {integrity: sha512-AkV+PhmQms6WPwR3E75THrkD9fFE7OpfsIUC297HvnFUoCFXR6GOMGPworA/qipXReTmv192AqBpuaLdgAngyg==} @@ -1118,30 +961,24 @@ packages: '@milaboratories/pl-model-common@1.46.2': resolution: {integrity: sha512-VEeauisApYScvCS8lnK3zpFJ520xuTAodKJmjR8ulHcMrWMyWMfHEdGb7j5OMD0mM/OwTgmQrrJ5eB7Xd+xoOQ==} - '@milaboratories/pl-model-common@1.47.2': - resolution: {integrity: sha512-XCEcL+CHYxZ/S/y9zrpzzaeZO+ZKQZ3Y9/pEKEsgDZy4VLtXZGG0RHV9byNmMGz8Y+7BahQ9tAf1d26TdxRVZQ==} - '@milaboratories/pl-model-common@1.47.3': resolution: {integrity: sha512-tXmKujm+6ru/Fh9hr9H3iRBWbS+JE0Q7FNualtzJpijaB3SNtScR4erLTamdANv5RYNn7kbYpDWr6wAjAOajrg==} - '@milaboratories/pl-model-middle-layer@1.30.14': - resolution: {integrity: sha512-gZwF0ux28uOnKFfCwjcD8beJCfzdyq1mqLgd/+T5zYCFsomTvPCMp2HYwzXmUp7poO/HTIAxSA6qT8As2q2WWg==} - '@milaboratories/pl-model-middle-layer@1.30.7': resolution: {integrity: sha512-rs9x3Ron4ujR/UOdEgB8WUB1SvZ8ZAScT1Av/e4or+iiQ/CzhmK9nqtYamHVwKV+JgwIFbXQwxGvIHDVz955dQ==} '@milaboratories/pl-model-middle-layer@1.31.0': resolution: {integrity: sha512-D1vyGABBtCYyp2IhKd9eMoZtKUj1wUYpIzz4Xz5k+bfJWGEF8GmZGiOvO4oOvrsEsXlG4NyO249MnBKmHFcppQ==} - '@milaboratories/pl-tree@1.13.2': - resolution: {integrity: sha512-zFN+CGNyxSlv7cLcFBYMWreneXqrLNWqOkZlXQmHmqfocVrTZ5EYa9nE7L/6opwzF2VaYTYD/iaxNcUmpD68pA==} + '@milaboratories/pl-tree@1.13.6': + resolution: {integrity: sha512-R3wnMjbCNfAsyGJe3FQOqEyrNePd9ZCwkjDj2ps7d9oeiJ7QX6hYkIVuMlq8IQNYOJybfHc/dFEvWeYE3uAAZg==} engines: {node: '>=22.19.0'} '@milaboratories/ptabler-expression-js@1.1.9': resolution: {integrity: sha512-fH0gix6ObRI9/TgPk1S40EkdFLskSdh6AloNBhkedH7WyTwOmM7zmtjmp+n7TiRHzRfufqzwLOZ9TFb7qAXQRA==} - '@milaboratories/ptabler-expression-js@1.2.36': - resolution: {integrity: sha512-LiqCbEMc8VVSXfoFsZSTpSQyoxb5L5FQAe46EcS2vRMrmMcTZwR2QGXH8nfr0oeS4MyjZepsh+Iv+H7VWK7bkA==} + '@milaboratories/ptabler-expression-js@1.2.37': + resolution: {integrity: sha512-urVRk4b5Jse555euNNITlOJ437McZVtBnC5h/E6O+iHm+hfNIi6OP1aAD5ai4jOpHllm85zW5Ne7hVyyd71bJw==} '@milaboratories/resolve-helper@1.1.3': resolution: {integrity: sha512-38/dW/XRZQREOxAOOKtO0lzEWPCP/DH0qhB3q1kYcGoN++5V92/zbVwbYrMDeDcjTyo+D62iIep+sKXeWHa7Uw==} @@ -1154,19 +991,19 @@ packages: os: [darwin, linux, win32] hasBin: true - '@milaboratories/ts-builder@1.6.1': - resolution: {integrity: sha512-0m+I8bdxw6mGTfPt+xW8OGvburrxLUpI/QZsyACHsXwyfVRfDKvWB3/9Hm053KzwHqrzpeuvidEBdKRVRAi9KQ==} + '@milaboratories/ts-builder@1.6.2': + resolution: {integrity: sha512-6pS36U5EL2IBLIpoP0/5oSFA9jzjcJmYhgEODzt5kglovTzJuh15nEmJMeChr0OpFtGFjQT6o37qHerDGETIpA==} hasBin: true - '@milaboratories/ts-configs@1.3.1': - resolution: {integrity: sha512-MfLF+qgDwnD2BuncGzFqQxKuqq/0KtXcXXftcvp8E08xY9cl5kkmBHX/H8RYAH4FvF6ghb56c3I6iaxIa5xIUw==} + '@milaboratories/ts-configs@1.4.0': + resolution: {integrity: sha512-VzU9D+RiggsG4VYteJSBN4o8IjYae9GbZ8MofikMJQLOI1Ir9/pVyPXXGnG/TAGvmcjbK/psAiZVGSvI5MBPGg==} '@milaboratories/ts-helpers@1.8.6': resolution: {integrity: sha512-ef01tARUl+0Urt3x8HHAByrhYHg4Rnn7WiFyJ+joZFZl1T1pUKTgfB7Zxc8Cn06HIl4i6H7s5OSymjZePIGqfQ==} engines: {node: '>=22.19.0'} - '@milaboratories/uikit@2.15.17': - resolution: {integrity: sha512-P0DRxZY0y2nYpoISTuKtsuTLPsetdPnS8Z0FWJglDqrh3JBkhdyFd2WbPMukhZXY+FldklgmXvEmNLUis7dnKA==} + '@milaboratories/uikit@2.15.23': + resolution: {integrity: sha512-FrZPJsa3jry4juVofR42IPCHv+oKPAcdBL+OidX0/S2lhIAsCIrEKBcAN3yzWd2yvyAPb3qzt0hbwpuTY0UVlw==} '@napi-rs/wasm-runtime@1.1.6': resolution: {integrity: sha512-ZLv/JdUfkvOy9eCnnBaGfiO+XimbjebAeO+MRQqD/B+FR1tnRN0tpKSJHRbE8sFfS6aqsXZ67TQjfwfsxULVbg==} @@ -1583,26 +1420,41 @@ packages: resolution: {integrity: sha512-+1VkjdD0QBLPodGrJUeqarH8VAIvQODIbwh9XpP5Syisf7YoQgsJKPNFoqqLQlu+VQ/tVSshMR6loPMn8U+dPg==} engines: {node: '>=14'} - '@platforma-open/milaboratories.runenv-python-3.12.10-atls@1.2.4': - resolution: {integrity: sha512-yeklY7ISewNOQ5TWlNLRi6VSOGfcdTJPoDWtf5Z8na3xPP56fWZFuckX/ixZpGvcoyBSQf1kFBFnHPginX7jCw==} + '@platforma-open/milaboratories.runenv-python-3.12.10-atls@1.2.7': + resolution: {integrity: sha512-vJiU68NKgyupW5meeWB/pZZ8y5wewPV6/EcRQ2BwUhSyikDQrn83XzBlc/nTu9UmbYmBqQTrt59x0qKtcC0ILw==} + + '@platforma-open/milaboratories.runenv-python-3.12.10-clustering@0.1.1': + resolution: {integrity: sha512-hnr1tf0dwoqTyjtUVZcwQm13O0V8kz4jCUz+yvOgdvWMrbg4o/h1KbghtM8HivI+tRoowc629QJWdy0bc7inQA==} + + '@platforma-open/milaboratories.runenv-python-3.12.10-h5ad@1.1.5': + resolution: {integrity: sha512-cuGk6NMBocvNePoMTnDW4z8eIF5RQdDcgwuPHg9atW9/veOOp02p5jk0EwxmnbMSVV0yDwRMd+0crQAQqW9bCA==} - '@platforma-open/milaboratories.runenv-python-3.12.10-h5ad@1.1.4': - resolution: {integrity: sha512-Yo47x/rm/FF8x/7q2o7TXdAS7pEWqBd/+aHnQzgNAF0DYljzln99mq1L4lMrQx08SJBGrh84ZtgQl/weTzWSIw==} + '@platforma-open/milaboratories.runenv-python-3.12.10-humanness@0.2.0': + resolution: {integrity: sha512-YjKFPXe+caABLre5oZTwAxFHvVCEb5K8rVTEIUBhFYMtfLipVIix0LXi6LxWoOYdhoEeKgIc4zlWFDKs1xaEoQ==} '@platforma-open/milaboratories.runenv-python-3.12.10-parapred@1.1.0': resolution: {integrity: sha512-nEM4eEj7pFT6yi+P5028qtgK5v9A9H0XdVHDtrKX7xW5Jipc1RIliOEU3gzaH0E7UAzPFQGSqShlwakRcGXwdQ==} - '@platforma-open/milaboratories.runenv-python-3.12.10-rapids@1.4.4': - resolution: {integrity: sha512-Ebg+bTaBwsG+WGUSKNgibhg+Cq3TqyrqPkWOkNMa9a2cMDCerYMpKQkRQ/uMWY/uO0jpuW75ck3Lpxv9mCiuXg==} + '@platforma-open/milaboratories.runenv-python-3.12.10-pgen@0.2.0': + resolution: {integrity: sha512-+4pEVwRZaBAm0716NoNdK9cfF4E7jtsRUpcAWYCuPbbZJ3Q7bgfgq8bEhcck8No97l9jWN2rAeJegA9UQy5FgQ==} - '@platforma-open/milaboratories.runenv-python-3.12.10-sccoda@1.3.5': - resolution: {integrity: sha512-oNkt0u8fgZACcL3XmAu4v0tDGnTSc3qRQs07U8OFX+paQk7qH4ckt0psLGN1iHDKZ3Gw3nlgel0D6Lst02WahQ==} + '@platforma-open/milaboratories.runenv-python-3.12.10-rapids@1.7.2': + resolution: {integrity: sha512-rCG+R9WjdopEtEgJcSWNRIWhpo1ABGnA7rPn0BF6ufvLOtKXOGtaGENI7BJlkEaLb2Cbh2XUUgS5l2r5WGhIRw==} - '@platforma-open/milaboratories.runenv-python-3.12.10@1.3.6': - resolution: {integrity: sha512-Bv4IF0PtbnyKJmlW/ygcGdhJS3cSCsOFZWR9buzr+VteDITKS1xyISW6tkO3glVBgInITGrtaDPKXTeiurhOHA==} + '@platforma-open/milaboratories.runenv-python-3.12.10-sccoda@1.3.6': + resolution: {integrity: sha512-NAacPe2uFxf5zqDAa70bblP4ZicvA52lVyzhV7IzlyeEIClAsGatLbitMnRghc2TepwYhQY4EbvElkKDFaSSGw==} - '@platforma-open/milaboratories.runenv-python-3@1.7.8': - resolution: {integrity: sha512-vEvyMG30Q94Kp/9vSxElaDvvzkTXwLT8KbavRP0UwAeU7vpXdAuFhDEigrXFE/uMPlQJDqOL4Vx2xG5Tsr2DWw==} + '@platforma-open/milaboratories.runenv-python-3.12.10-scientific-slim@1.1.0': + resolution: {integrity: sha512-CSLEjBYUdHDf66QAfgQ9jo+MoW63Wr0ygdIncJE2siO2jMUzYCqgCNd8bMjaMfZa3YBC9bHca3Cjxbp4VYc54A==} + + '@platforma-open/milaboratories.runenv-python-3.12.10-torch-cuda@0.2.0': + resolution: {integrity: sha512-PRGkgsVY08tQvoucU8fJmeCF+x9XjiHHjNTbxWqddqx2EZpiUvmylkxtU8i8xJW13cesAI+xHv3NsVUXQaRDqA==} + + '@platforma-open/milaboratories.runenv-python-3.12.10@1.3.24': + resolution: {integrity: sha512-5ne8Fmlhu1YKB5RYgq4c/AvYzb+sUY4hzv/bKIj0E3aDgJdMYO6M6lMicDzQTMagrGRu9FIhDewIgo++TX3+Yw==} + + '@platforma-open/milaboratories.runenv-python-3@1.11.6': + resolution: {integrity: sha512-H3IR86pq0T4UOR66BZEYIkMmlhuyAiOnAP3O8y5M6A2C2UH/oCLGVYqrFF7XHWgjmacvZ7Qcz4KDtIxgJpFCMw==} '@platforma-open/milaboratories.samples-and-data.model@1.11.2': resolution: {integrity: sha512-y9r9LcvBGFmg1LPo+6MP8AWVTigYqJVzKwj1K/z1eJBGDASdroIFTtGB+3CEltOEOt6oDGAzbJC2oRQKMJGTYQ==} @@ -1631,8 +1483,8 @@ packages: '@platforma-open/milaboratories.software-ptabler.schema@1.13.2': resolution: {integrity: sha512-xQ5eD6WNLL490bQY2kEH/Dz+0SE5uyLexwOQzoUpKhBqdYb4eIKGA2NnZ9SDOgfDoiNFzTeYGQhuI+R3W6P0Og==} - '@platforma-open/milaboratories.software-ptabler.schema@1.15.20': - resolution: {integrity: sha512-+7PcB53IdEGZ9wS88UtPIH3tHk4dIZbk4RdUSywXCdnjzTC2qipK92zPMsC/vZE4cUo8XnETL6Ia+nSXrxwy9A==} + '@platforma-open/milaboratories.software-ptabler.schema@1.15.21': + resolution: {integrity: sha512-3EWO64poe5VcBIgVIdIFHwN7nPi000kPuaRmlVt3xnnQZqXUYDFNZfZ2Mc9/mE1Eq3FLwQxcy/5mng7LvgWnXg==} '@platforma-open/milaboratories.software-ptabler@1.14.0': resolution: {integrity: sha512-bkQvykUBygav4y5/GCZbAbwoU4z29AJhVufhAvEYSEMQtozw++CPXcci2OQJaz6+N5OClmznHEwDOMcU4KfRFQ==} @@ -1703,10 +1555,6 @@ packages: '@platforma-open/milaboratories.software-small-binaries@2.1.1': resolution: {integrity: sha512-KN1PR7YgUUfx1dxh/TtcoWpSZbOXbRxXzQuJ505qHuXuE0spYwC7bXnvJsEYbEwRcPMa0foTrjBnPNORE4yr+Q==} - '@platforma-sdk/block-tools@2.12.8': - resolution: {integrity: sha512-UI/gnQuRO+mt7Vy6ootQN1KP0H2GdxGQxtcppGQI6CEAEfTWYmCk8f1C7QoWTgS6tjQkkKmLE8iDzAdbZqzchg==} - hasBin: true - '@platforma-sdk/block-tools@2.13.0': resolution: {integrity: sha512-xKiTmNLjfxCnnHRr9Q4ct3HdjLfkbLztvxpU++/pek0R8Bj/Lezeoji+eLpxAAYYUxjMr8jLj935eLK5MqKrSg==} hasBin: true @@ -1718,11 +1566,8 @@ packages: '@platforma-sdk/model@1.51.2': resolution: {integrity: sha512-AFCQus1HYOW/TwrYYSKetgqW39S2XUCGI856H0dxjArJiTHzroUpGtUv+8CkklBCTH5KhWjd7+8BMMRx/MOZpQ==} - '@platforma-sdk/model@1.80.8': - resolution: {integrity: sha512-/Vp9U1EvEa3g0geh29g+KKkJ0n5WKB48fB1tJ3XJtTOTUTpK7thu9GEgnbOARLJ95SXxTiZzUKIY9xYa7YtpTQ==} - - '@platforma-sdk/package-builder-lib@1.2.1': - resolution: {integrity: sha512-H6weitj7JxbiJSlteEFLafTJ+tfty6iv/imf3ysy8oCS8AZIRJk2VMW3M/aAc+xVkQeX7oVIwMwFMYrJIoFsAg==} + '@platforma-sdk/model@1.81.1': + resolution: {integrity: sha512-pyCcCzMu+L0ELnNQkpmYMNw+A70WWCXng29WxOVllMTvFvr6/BaH4OUOA2q8ocseUlKm6qFQiRxmZTe53AcvlQ==} '@platforma-sdk/package-builder-lib@1.3.0': resolution: {integrity: sha512-CdBjmNo6E1fBxKYWaXa49L/L2WLURxs2f1TAqxLIZlHRE4DZ6E1TEj3jNNKESWp+/9rwtLkTAzmTzNPrDgz+2Q==} @@ -1732,11 +1577,11 @@ packages: engines: {node: '>=22'} hasBin: true - '@platforma-sdk/test@1.80.9': - resolution: {integrity: sha512-MN31C6VnvIP/yXxeZhf1iyDAZK3V6NjxkbR3ToJnUJrBvCvXp33KY8IV8f6H7UOP72+cAfrwXM4vW2+rDyxXzg==} + '@platforma-sdk/test@1.81.3': + resolution: {integrity: sha512-LSDJoQqpxX2G8W7ZshnYWroqNQiFoxXQlka4pEJTnPEO8YISW3lWknBhqsxMp2GyepQNoyl2WO1bfiQz+ssT5A==} - '@platforma-sdk/ui-vue@1.80.9': - resolution: {integrity: sha512-dvLA4Fil37gU2XK5n0MTqzmAYoKR6TWG+t2RBFqnTJzyWl9ZZGS0wWSp3hbpkcUIjj2edM35OdFVhqmKWoE/tg==} + '@platforma-sdk/ui-vue@1.81.1': + resolution: {integrity: sha512-xFnehLOXDHYwDFrdtiU/NBjV0l6pEVLkthfnBjR+g2pSrHJGEL9JDrY3DdzoNUSxU4fal+Ak5aoAJR5v8mOeLg==} '@platforma-sdk/workflow-tengo@5.8.0': resolution: {integrity: sha512-OJnnjBXt1VcS1QNMC90PlkXrWJoKQ/Nl3/NeTqzTmq1j5gT3cZXoIQYu1c31KZXBungUo8TphymBcs+qDGpVgA==} @@ -1769,39 +1614,24 @@ packages: '@protobufjs/base64@1.1.2': resolution: {integrity: sha512-AZkcAA5vnN/v4PDqKyMR5lx7hZttPDgClv83E//FMNhR2TMcLUhfRUBHCmSl0oi9zMgDDqRUJkSxO3wm85+XLg==} - '@protobufjs/codegen@2.0.4': - resolution: {integrity: sha512-YyFaikqM5sH0ziFZCN3xDC7zeGaB/d0IUb9CATugHWbd1FRFwWwt4ld4OYMPWu5a3Xe01mGAULCdqhMlPl29Jg==} - '@protobufjs/codegen@2.0.5': resolution: {integrity: sha512-zgXFLzW3Ap33e6d0Wlj4MGIm6Ce8O89n/apUaGNB/jx+hw+ruWEp7EwGUshdLKVRCxZW12fp9r40E1mQrf/34g==} - '@protobufjs/eventemitter@1.1.0': - resolution: {integrity: sha512-j9ednRT81vYJ9OfVuXG6ERSTdEL1xVsNgqpkxMsbIabzSo3goCjDIveeGv5d03om39ML71RdmrGNjG5SReBP/Q==} - '@protobufjs/eventemitter@1.1.1': resolution: {integrity: sha512-vW1GmwMZNnL+gMRaovlh9yZX74kc+TTU3FObkkurpMaRtBfLP3ldjS9KQWlwZgraRE0+dheEEoAxdzcJQ8eXZg==} - '@protobufjs/fetch@1.1.0': - resolution: {integrity: sha512-lljVXpqXebpsijW71PZaCYeIcE5on1w5DlQy5WH6GLbFryLUrBD4932W/E2BSpfRJWseIL4v/KPgBFxDOIdKpQ==} - '@protobufjs/fetch@1.1.1': resolution: {integrity: sha512-GpptLrs57adMSuHi3VNj0mAF8dwh36LMaYF6XyJ6JMWlVsc+t42tm1HSEDmOs3A8fC9yyeisgLhsTVQokOZ0zw==} '@protobufjs/float@1.0.2': resolution: {integrity: sha512-Ddb+kVXlXst9d+R9PfTIxh1EdNkgoRe5tOX6t01f1lYWOvJnSPDBlG241QLzcyPdoNTsblLUdujGSE4RzrTZGQ==} - '@protobufjs/inquire@1.1.0': - resolution: {integrity: sha512-kdSefcPdruJiFMVSbn801t4vFK7KB/5gd2fYvrxhuJYg8ILrmn9SKSX2tZdV6V+ksulWqS7aXjBcRXl3wHoD9Q==} - '@protobufjs/path@1.1.2': resolution: {integrity: sha512-6JOcJ5Tm08dOHAbdR3GrvP+yUUfkjG5ePsHYczMFLq3ZmMkAD98cDgcT2iA1lJ9NVwFd4tH/iSSoe44YWkltEA==} '@protobufjs/pool@1.1.0': resolution: {integrity: sha512-0kELaGSIDBKvcgS4zkjz1PeddatrjYcmMWOlAuAPwAeccUrPHdUqo/J6LiymHHEiJT5NrF1UVwxY14f+fy4WQw==} - '@protobufjs/utf8@1.1.0': - resolution: {integrity: sha512-Vvn3zZrhQZkkBE8LSuW3em98c0FwgO4nxzv6OdSxPKJIEKY2bGbHn+mhGIPerzI4twdxaP8/0+06HBpwf345Lw==} - '@protobufjs/utf8@1.1.2': resolution: {integrity: sha512-b1UQwcEZ4yCnMCD8DAL1VlbvBJE9/IX4FTIp7BG1xYpf29SLazLSrqUkj4w7Y5y7cCVP6E5tcqqcI0xemPkHug==} @@ -2356,6 +2186,36 @@ packages: '@standard-schema/spec@1.1.0': resolution: {integrity: sha512-l2aFy5jALhniG5HgqrD6jXLi/rUWrKvqN/qJx6yoJsgKhblVd+iqqU4RCXavm/jPityDo5TCvKMnpjKnOriy0w==} + '@turbo/darwin-64@2.10.10': + resolution: {integrity: sha512-gFDD+wRP5hWxBRghGyEbjpbLOY7aIU/wvsnKdMM7odQcp/wHMrnI83p0FyxxMRZnFH9ZD+S59MvcpOC5b+nrCA==} + cpu: [x64] + os: [darwin] + + '@turbo/darwin-arm64@2.10.10': + resolution: {integrity: sha512-VZYsxZ6yjyDosUqtiroAVSXPLmx/qBxdHJgIxdMH9RyNmLdOLOWtJnYMnI4qckwCgQMK85G3fu94/xk5+iBCgw==} + cpu: [arm64] + os: [darwin] + + '@turbo/linux-64@2.10.10': + resolution: {integrity: sha512-lAvW+yEnmsCKMEIwNugjozawvYytHKPhU0kfLBizu83MIs8OUb9KobYvkZ56L5akSM6K7+gBFLEIfQkaceh90g==} + cpu: [x64] + os: [android, linux] + + '@turbo/linux-arm64@2.10.10': + resolution: {integrity: sha512-MSJ+NkRTd79Z9+YEZpUV9VOWVOOigFhE+v/ETNYJEuTJp3r00y9YgFvDXrmM+DP8Kal6tk3U6xSugD2/Ojh+Jg==} + cpu: [arm64] + os: [android, linux] + + '@turbo/windows-64@2.10.10': + resolution: {integrity: sha512-ycWpXDkUfnDFDY9d+4Qna/UZotDB0wj+s9agrlmNt0Q7a3XHORhK8GPKJdzgzeutXu9EW5P/jyabTEHlohuDXw==} + cpu: [x64] + os: [win32] + + '@turbo/windows-arm64@2.10.10': + resolution: {integrity: sha512-PMk6zQN0csUFklLe+1hz/5G9uU1YmV0cEIey2R/bSeA6o69qcBTlN4A3jOqkgenOO5dOpMHmq2sEUZo8r1+Ssg==} + cpu: [arm64] + os: [win32] + '@tybys/wasm-util@0.10.3': resolution: {integrity: sha512-F3fo1MYrRJYL3zER0OUOmkutjr1Vp23m7OsSgp7nq4SP6OqX6C/56XFIPAl5bt3zaBRjmW7SGz3u/6LwFpYcOg==} @@ -2443,25 +2303,11 @@ packages: peerDependencies: vitest: 4.1.4 - '@vitest/expect@4.0.18': - resolution: {integrity: sha512-8sCWUyckXXYvx4opfzVY03EOiYVxyNrHS5QxX3DAIi5dpJAAkyJezHCP77VMX4HKA2LDT/Jpfo8i2r5BE3GnQQ==} - - '@vitest/expect@4.1.4': - resolution: {integrity: sha512-iPBpra+VDuXmBFI3FMKHSFXp3Gx5HfmSCE8X67Dn+bwephCnQCaB7qWK2ldHa+8ncN8hJU8VTMcxjPpyMkUjww==} - - '@vitest/mocker@4.0.18': - resolution: {integrity: sha512-HhVd0MDnzzsgevnOWCBj5Otnzobjy5wLBe4EdeeFGv8luMsGcYqDuFRMcttKWZA5vVO8RFjexVovXvAM4JoJDQ==} - peerDependencies: - msw: ^2.4.9 - vite: ^6.0.0 || ^7.0.0-0 - peerDependenciesMeta: - msw: - optional: true - vite: - optional: true + '@vitest/expect@4.1.10': + resolution: {integrity: sha512-YsCn+qAk1GWjQOWFEsEcL2gNQ0zmVmQu3T03qP6UyjhtmdtwtbuI+DASn/7iQB3HGTXkdBwGddzxPlmiql5vlA==} - '@vitest/mocker@4.1.4': - resolution: {integrity: sha512-R9HTZBhW6yCSGbGQnDnH3QHfJxokKN4KB+Yvk9Q1le7eQNYwiCyKxmLmurSpFy6BzJanSLuEUDrD+j97Q+ZLPg==} + '@vitest/mocker@4.1.10': + resolution: {integrity: sha512-v0xaezt+DKEmKfaxg133ldzADrwLGd7Ze1MfQQTYfvs8OqZIwbxyxaYURivwV7sWy5fqn3rH5uOrSp07bp44Ow==} peerDependencies: msw: ^2.4.9 vite: ^6.0.0 || ^7.0.0 || ^8.0.0 @@ -2471,35 +2317,20 @@ packages: vite: optional: true - '@vitest/pretty-format@4.0.18': - resolution: {integrity: sha512-P24GK3GulZWC5tz87ux0m8OADrQIUVDPIjjj65vBXYG17ZeU3qD7r+MNZ1RNv4l8CGU2vtTRqixrOi9fYk/yKw==} + '@vitest/pretty-format@4.1.10': + resolution: {integrity: sha512-W1HsjSH4MXQ9YfmmhLAoIYf1HRfekQCGngeIgcei6MP5QQGWUe0gkopdZQaVCFO+JDJMrAJGwa5pRpNpvy4P8Q==} - '@vitest/pretty-format@4.1.4': - resolution: {integrity: sha512-ddmDHU0gjEUyEVLxtZa7xamrpIefdEETu3nZjWtHeZX4QxqJ7tRxSteHVXJOcr8jhiLoGAhkK4WJ3WqBpjx42A==} + '@vitest/runner@4.1.10': + resolution: {integrity: sha512-IKI6kpIH+LmpROplyLwBBaCfMgOZOMsygVa6BARD6ahA04VRuJSa6OaVG7kRvSEMD870Vd91rSSw0eegtWyLGg==} - '@vitest/runner@4.0.18': - resolution: {integrity: sha512-rpk9y12PGa22Jg6g5M3UVVnTS7+zycIGk9ZNGN+m6tZHKQb7jrP7/77WfZy13Y/EUDd52NDsLRQhYKtv7XfPQw==} + '@vitest/snapshot@4.1.10': + resolution: {integrity: sha512-xRkfOT1qpTAi/Ti4Y1LtfRc3kEuqxGw59eN2jN9pRWMtS/XDevekhcFSqvQqjUNGksfjMJu3Y+oJ+4Ypn2OaJw==} - '@vitest/runner@4.1.4': - resolution: {integrity: sha512-xTp7VZ5aXP5ZJrn15UtJUWlx6qXLnGtF6jNxHepdPHpMfz/aVPx+htHtgcAL2mDXJgKhpoo2e9/hVJsIeFbytQ==} + '@vitest/spy@4.1.10': + resolution: {integrity: sha512-PLf/Ugvoq5wO/b4rwYCR1h2PSIdXz7wnkQFMiUpLdtM7l6pqVFcQIBEHyT1+l+cj7mNwAfZHzqXqDyjvOuwbDw==} - '@vitest/snapshot@4.0.18': - resolution: {integrity: sha512-PCiV0rcl7jKQjbgYqjtakly6T1uwv/5BQ9SwBLekVg/EaYeQFPiXcgrC2Y7vDMA8dM1SUEAEV82kgSQIlXNMvA==} - - '@vitest/snapshot@4.1.4': - resolution: {integrity: sha512-MCjCFgaS8aZz+m5nTcEcgk/xhWv0rEH4Yl53PPlMXOZ1/Ka2VcZU6CJ+MgYCZbcJvzGhQRjVrGQNZqkGPttIKw==} - - '@vitest/spy@4.0.18': - resolution: {integrity: sha512-cbQt3PTSD7P2OARdVW3qWER5EGq7PHlvE+QfzSC0lbwO+xnt7+XH06ZzFjFRgzUX//JmpxrCu92VdwvEPlWSNw==} - - '@vitest/spy@4.1.4': - resolution: {integrity: sha512-XxNdAsKW7C+FLydqFJLb5KhJtl3PGCMmYwFRfhvIgxJvLSXhhVI1zM8f1qD3Zg7RCjTSzDVyct6sghs9UEgBEQ==} - - '@vitest/utils@4.0.18': - resolution: {integrity: sha512-msMRKLMVLWygpK3u2Hybgi4MNjcYJvwTb0Ru09+fOyCXIgT5raYP041DRRdiJiI3k/2U6SEbAETB3YtBrUkCFA==} - - '@vitest/utils@4.1.4': - resolution: {integrity: sha512-13QMT+eysM5uVGa1rG4kegGYNp6cnQcsTc67ELFbhNLQO+vgsygtYJx2khvdt4gVQqSSpC/KT5FZZxUpP3Oatw==} + '@vitest/utils@4.1.10': + resolution: {integrity: sha512-fy9am/HWxbaGt/Sawrp90vt6Y6jQwf1RX77cz3uwoJwJVMli/e1IEwRPnMNJ7vKfPTwo0diXifkpPvwH9v7nGA==} '@volar/language-core@2.4.28': resolution: {integrity: sha512-w4qhIJ8ZSitgLAkVay6AbcnC7gP3glYM3fYwKV3srj8m494E3xtrCv6E+bWviiK/8hs6e6t1ij1s2Endql7vzQ==} @@ -2513,26 +2344,26 @@ packages: '@vue/compiler-core@3.5.24': resolution: {integrity: sha512-eDl5H57AOpNakGNAkFDH+y7kTqrQpJkZFXhWZQGyx/5Wh7B1uQYvcWkvZi11BDhscPgj8N7XV3oRwiPnx1Vrig==} - '@vue/compiler-core@3.5.25': - resolution: {integrity: sha512-vay5/oQJdsNHmliWoZfHPoVZZRmnSWhug0BYT34njkYTPqClh3DNWLkZNJBVSjsNMrg0CCrBfoKkjZQPM/QVUw==} + '@vue/compiler-core@3.5.41': + resolution: {integrity: sha512-q0Xtv/F9w2YO/7htQhtiL+Ev2WCJbe5N2hc+XfgyKkEKqWpSxknmT8QOuGdEKNdjPq0c3F7rNpFkTo3Kfrm7pg==} '@vue/compiler-dom@3.5.24': resolution: {integrity: sha512-1QHGAvs53gXkWdd3ZMGYuvQFXHW4ksKWPG8HP8/2BscrbZ0brw183q2oNWjMrSWImYLHxHrx1ItBQr50I/q2zw==} - '@vue/compiler-dom@3.5.25': - resolution: {integrity: sha512-4We0OAcMZsKgYoGlMjzYvaoErltdFI2/25wqanuTu+S4gismOTRTBPi4IASOjxWdzIwrYSjnqONfKvuqkXzE2Q==} + '@vue/compiler-dom@3.5.41': + resolution: {integrity: sha512-oKacVfNglLvGjnS6BXOlGL7EyG2h8X03pqXCjzotRZUaXGjbrTJUnVAQjrCqUnS+lyu31nwQjZY/d817GmCnfw==} '@vue/compiler-sfc@3.5.24': resolution: {integrity: sha512-8EG5YPRgmTB+YxYBM3VXy8zHD9SWHUJLIGPhDovo3Z8VOgvP+O7UP5vl0J4BBPWYD9vxtBabzW1EuEZ+Cqs14g==} - '@vue/compiler-sfc@3.5.25': - resolution: {integrity: sha512-PUgKp2rn8fFsI++lF2sO7gwO2d9Yj57Utr5yEsDf3GNaQcowCLKL7sf+LvVFvtJDXUp/03+dC6f2+LCv5aK1ag==} + '@vue/compiler-sfc@3.5.41': + resolution: {integrity: sha512-XJhip7R2wy6vX3knCxdZN4KracFaZUef58s1KYewqluedHIJaPIVfXoYT7MF1F8nCvv6k8bWWxDC8opMkg1VTQ==} '@vue/compiler-ssr@3.5.24': resolution: {integrity: sha512-trOvMWNBMQ/odMRHW7Ae1CdfYx+7MuiQu62Jtu36gMLXcaoqKvAyh+P73sYG9ll+6jLB6QPovqoKGGZROzkFFg==} - '@vue/compiler-ssr@3.5.25': - resolution: {integrity: sha512-ritPSKLBcParnsKYi+GNtbdbrIE1mtuFEJ4U1sWeuOMlIziK5GtOL85t5RhsNy4uWIXPgk+OUdpnXiTdzn8o3A==} + '@vue/compiler-ssr@3.5.41': + resolution: {integrity: sha512-U3v5OejKEGqOI0Wy0+Sz7hGuIFZHA4LSXzrNM3IMIeDyJEBBfTpX26n3SDgToRpP2bLc9FfI2j/kSgcJ8Emq5A==} '@vue/compiler-vue2@2.7.16': resolution: {integrity: sha512-qYC3Psj9S/mfu9uVi5WvNZIzq+xnXMhOwbTFKKDD7b1lhpnn71jXSFdTQ+WsIEk0ONCd7VV2IMm7ONl6tbQ86A==} @@ -2551,36 +2382,34 @@ packages: '@vue/reactivity@3.5.24': resolution: {integrity: sha512-BM8kBhtlkkbnyl4q+HiF5R5BL0ycDPfihowulm02q3WYp2vxgPcJuZO866qa/0u3idbMntKEtVNuAUp5bw4teg==} - '@vue/reactivity@3.5.25': - resolution: {integrity: sha512-5xfAypCQepv4Jog1U4zn8cZIcbKKFka3AgWHEFQeK65OW+Ys4XybP6z2kKgws4YB43KGpqp5D/K3go2UPPunLA==} + '@vue/reactivity@3.5.41': + resolution: {integrity: sha512-rznsqKM0np0x18EjzF8x88MpEhdNsffbvFbckLL5+oUKz1BxAImEmO7J1ArRYSyo6aQaVoBDp7jEkT91OOxydA==} '@vue/runtime-core@3.5.24': resolution: {integrity: sha512-RYP/byyKDgNIqfX/gNb2PB55dJmM97jc9wyF3jK7QUInYKypK2exmZMNwnjueWwGceEkP6NChd3D2ZVEp9undQ==} - '@vue/runtime-core@3.5.25': - resolution: {integrity: sha512-Z751v203YWwYzy460bzsYQISDfPjHTl+6Zzwo/a3CsAf+0ccEjQ8c+0CdX1WsumRTHeywvyUFtW6KvNukT/smA==} + '@vue/runtime-core@3.5.41': + resolution: {integrity: sha512-Vcry58hiAKwGen9Z1jUZE0feFsNArPCMOImYI8el48A9Idf6DuQYD0U05zZIF2Iad1hGhPSvcbBbAOhNr55fhg==} '@vue/runtime-dom@3.5.24': resolution: {integrity: sha512-Z8ANhr/i0XIluonHVjbUkjvn+CyrxbXRIxR7wn7+X7xlcb7dJsfITZbkVOeJZdP8VZwfrWRsWdShH6pngMxRjw==} - '@vue/runtime-dom@3.5.25': - resolution: {integrity: sha512-a4WrkYFbb19i9pjkz38zJBg8wa/rboNERq3+hRRb0dHiJh13c+6kAbgqCPfMaJ2gg4weWD3APZswASOfmKwamA==} + '@vue/runtime-dom@3.5.41': + resolution: {integrity: sha512-3vVBahVBS9+U6cmXBLyb8nE6/yYo4J/CGI9eVFs3KiMc0YHuudwKyShTD65jtJy/L9PUUxNAFu4cj4LiJ0UFbw==} '@vue/server-renderer@3.5.24': resolution: {integrity: sha512-Yh2j2Y4G/0/4z/xJ1Bad4mxaAk++C2v4kaa8oSYTMJBJ00/ndPuxCnWeot0/7/qafQFLh5pr6xeV6SdMcE/G1w==} peerDependencies: vue: 3.5.24 - '@vue/server-renderer@3.5.25': - resolution: {integrity: sha512-UJaXR54vMG61i8XNIzTSf2Q7MOqZHpp8+x3XLGtE3+fL+nQd+k7O5+X3D/uWrnQXOdMw5VPih+Uremcw+u1woQ==} - peerDependencies: - vue: 3.5.25 + '@vue/server-renderer@3.5.41': + resolution: {integrity: sha512-n6hx/pNFfbD6SuyeuMVkvqox8bwf/ET9JlA/kAz/imw8sw++wkqKe2mHX5KutjPpbKE4Z56yTHszoOjGMI9igQ==} '@vue/shared@3.5.24': resolution: {integrity: sha512-9cwHL2EsJBdi8NY22pngYYWzkTDhld6fAD6jlaeloNGciNSJL6bLpbxVgXl96X00Jtc6YWQv96YA/0sxex/k1A==} - '@vue/shared@3.5.25': - resolution: {integrity: sha512-AbOPdQQnAnzs58H2FrrDxYj/TJfmeS2jdfEEhgiKINy+bnOANmVizIEgq1r+C5zsbs6l1CCQxtcj71rwNQ4jWg==} + '@vue/shared@3.5.41': + resolution: {integrity: sha512-IOnwSCma8j+9xJT6b8H0dEYidC80NsYmNMlZxRsukYcSoGaDBohog5hDxzeUXdFeGWFA++vWvxqOmrr96VlqMA==} '@vue/test-utils@2.4.6': resolution: {integrity: sha512-FMxEjOpYNYiFe0GkaHsnJPXFHxQ6m4t8vI/ElPGpMWxZKpmRvQ33OIrvRXemy6yha03RxhOlQuy+gZMC3CQSow==} @@ -3008,8 +2837,8 @@ packages: resolution: {integrity: sha512-uV2QOWP2nWzsy2aMp8aRibhi9dlzF5Hgh5SHaB9OiTGEyDTiJJyx0uy51QXdyWbtAHNua4XJzUKca3OzKUd3vA==} engines: {node: '>= 8'} - csstype@3.1.3: - resolution: {integrity: sha512-M1uQkMl8rQK/szD0LNhtqxIPLpimGm8sOBwU7lLnCpSbTyY3yeU1Vc7l4KT5zT4s/yOxHH5O7tIuuLOCnLADRw==} + csstype@3.2.3: + resolution: {integrity: sha512-z1HGKcYy2xA8AGQfwrn0PAy+PB7X/GSj3UVJW9qKyn43xWa+gl5nXmU4qqLMRzWVLFC8KusUX8T/0kCiOYpAIQ==} d3-array@3.2.4: resolution: {integrity: sha512-tdQAmyA18i4J7wprpYq8ClcxZy3SC31QMeByyCFyRt7BVHdREQZ5lpzoe5mFEYZUWe+oq8HBvk9JjpibyEV4Jg==} @@ -3147,6 +2976,10 @@ packages: resolution: {integrity: sha512-V0hjH4dGPh9Ao5p0MoRY6BVqtwCjhz6vI5LT8AJ55H+4g9/4vbHx1I54fS0XuclLhDHArPQCiMjDxjaL8fPxhw==} engines: {node: '>=0.12'} + entities@7.0.1: + resolution: {integrity: sha512-TWrgLOFUQTH994YUyl1yT4uyavY5nNB5muff+RtWaqNVCAK408b5ZnnbNAUEWLTCpum9w6arT70i1XdQ4UeOPA==} + engines: {node: '>=0.12'} + es-module-lexer@1.7.0: resolution: {integrity: sha512-jEQoCwk8hyb2AZziIOLhDqpm5+2ww5uIE6lkO/6jcOCusfk6LhMHpXXfBLXTZ7Ydyt0j4VoUQv6uGNYbdW+kBA==} @@ -3156,11 +2989,6 @@ packages: es-toolkit@1.42.0: resolution: {integrity: sha512-SLHIyY7VfDJBM8clz4+T2oquwTQxEzu263AyhVK4jREOAwJ+8eebaa4wM3nlvnAqhDrMm2EsA6hWHaQsMPQ1nA==} - esbuild@0.25.12: - resolution: {integrity: sha512-bbPBYYrtZbkt6Os6FiTLCTFxvq4tt3JKall1vRwshA3fdVztsLAatFaZobhkBC8/BrPetoa0oksYoKXoG4ryJg==} - engines: {node: '>=18'} - hasBin: true - escalade@3.2.0: resolution: {integrity: sha512-WUj2qlxaQtO4g6Pq5c29GTcWGDyd8itL8zTlipgECz3JesAiiOKotd8JU6otB3PACgG6xkJUyVhboMS+bje/jA==} engines: {node: '>=6'} @@ -3732,6 +3560,11 @@ packages: engines: {node: ^10 || ^12 || ^13.7 || ^14 || >=15.0.1} hasBin: true + nanoid@3.3.18: + resolution: {integrity: sha512-DTg4MJbGMWkfi6VZFdNt2/caMbQy4Ou+Op/hJQvGEWcnVfoA1QA+xzRKAzw9jD6+GVOOeYr/mIcuDSdug6F6+w==} + engines: {node: ^10 || ^12 || ^13.7 || ^14 || >=15.0.1} + hasBin: true + napi-build-utils@2.0.0: resolution: {integrity: sha512-GEbrYkbfF7MoNaoh2iGG84Mnf/WZfB0GdGEsM8wz7Expx/LlWf5U8t9nvJKXSp3qr5IsEbK04cBGhol/KwOsWA==} @@ -3776,9 +3609,6 @@ packages: resolution: {integrity: sha512-rJgTQnkUnH1sFw8yT6VSU3zD3sWmu6sZhIseY8VX+GRu3P6F7Fu+JNDoXfklElbLJSnc3FUQHVe4cU5hj+BcUg==} engines: {node: '>=0.10.0'} - obug@2.1.1: - resolution: {integrity: sha512-uTqF9MuPraAQ+IsnPf366RG4cP9RtUi7MLO1N3KEc+wb0a6yKpeL0lmk2IB1jY5KHPAlTc6T/JRdC/YqxHNwkQ==} - obug@2.1.3: resolution: {integrity: sha512-9miFgM2OFba7hB+pRgvtV84pYTBaoTHohvmIgiRt6dRIzbwEOIaNaP+dIlGs2fNFoB0SeISs0Jz5WFVRid6Xyg==} engines: {node: '>=12.20.0'} @@ -3900,10 +3730,6 @@ packages: resolution: {integrity: sha512-JU3teHTNjmE2VCGFzuY8EXzCDVwEqB2a8fsIvwaStHhAWJEeVd1o1QD80CU6+ZdEXXSLbSsuLwJjkCBWqRQUVA==} engines: {node: '>=8.6'} - picomatch@4.0.3: - resolution: {integrity: sha512-5gTmgEY/sqK6gFXLIsQNH19lWb4ebPDLA4SdLP7dsWkIXHWlG66oPuVvXSGFPppYZz8ZDZq0dYYrbHfBCVUb1Q==} - engines: {node: '>=12'} - picomatch@4.0.4: resolution: {integrity: sha512-QP88BAKvMam/3NxH6vj2o21R6MjxZUAd6nlwAS/pnGvN9IVLocLHxGYIzFhg6fUQ+5th6P4dv4eW9jX3DSIj7A==} engines: {node: '>=12'} @@ -3942,6 +3768,10 @@ packages: resolution: {integrity: sha512-pMMHxBOZKFU6HgAZ4eyGnwXF/EvPGGqUr0MnZ5+99485wwW41kW91A4LOGxSHhgugZmSChL5AlElNdwlNgcnLQ==} engines: {node: ^10 || ^12 || >=14} + postcss@8.5.26: + resolution: {integrity: sha512-u82N74LFzG8ca+dD8puPnplTXoGH4fTPpVGuIbt36G3qvNlkvfD0lEAZSxaly3KX8TS/L1A1gsCEmvKmBcVbkQ==} + engines: {node: ^10 || ^12 || >=14} + prebuild-install@7.1.3: resolution: {integrity: sha512-8Mf2cbV7x1cXPUILADGI3wuhfqWvtiLA1iclTDbFRZkgRQS0NqsPZphna9V+HyTEadheuPmjaJMsbzKQFOzLug==} engines: {node: '>=10'} @@ -3963,10 +3793,6 @@ packages: proto-list@1.2.4: resolution: {integrity: sha512-vtK/94akxsTMhe0/cbfpR+syPuszcuwhqVjJq26CuNDgFGj682oRBXOP5MJpv2r7JtE8MsiepGIqvvOTBwn2vA==} - protobufjs@7.4.0: - resolution: {integrity: sha512-mRUWCc3KUU4w1jU8sGxICXH/gNS94DvI1gxqDvBzhj1JpcsimQkYiOJfwsPUykUI5ZaspFbSgmBLER8IrQ3tqw==} - engines: {node: '>=12.0.0'} - protobufjs@7.6.5: resolution: {integrity: sha512-/FPD0nUc9jH6rfFjji9IBqOz4pcSE3CsT1m7Ep6Mdb0LxSUMj8hgl6GomOvZzpNpAqqGaXA0P3VSrZLFzIhQrw==} engines: {node: '>=12.0.0'} @@ -3990,11 +3816,11 @@ packages: queue-tick@1.0.1: resolution: {integrity: sha512-kJt5qhMxoszgU/62PLP1CJytzd2NKetjSRnyuj31fDd3Rlcz3fzlFdFLD1SItunPwyqEOkca6GbV612BWfaBag==} - quickjs-emscripten-core@0.31.0: - resolution: {integrity: sha512-oQz8p0SiKDBc1TC7ZBK2fr0GoSHZKA0jZIeXxsnCyCs4y32FStzCW4d1h6E1sE0uHDMbGITbk2zhNaytaoJwXQ==} + quickjs-emscripten-core@0.32.0: + resolution: {integrity: sha512-QFnPfjFey8EqknSrSxe1hZrf1/8z7/6s1QzGOmKo6++02r7QRRX7ZoyNaZh7JuVjWsVW87KnQrbZqnHkOAzUyg==} - quickjs-emscripten@0.31.0: - resolution: {integrity: sha512-K7Yt78aRPLjPcqv3fIuLW1jW3pvwO21B9pmFOolsjM/57ZhdVXBr51GqJpalgBlkPu9foAvhEAuuQPnvIGvLvQ==} + quickjs-emscripten@0.32.0: + resolution: {integrity: sha512-So0Sqw869y/S2oE3Nuc0uT3Dhqgvsj8FSrwBdsuTosVsG8ME5/OcudU1GxsrIFdFABgy17GHnTVO9TYV/bLQcA==} engines: {node: '>=16.0.0'} rc@1.2.8: @@ -4231,9 +4057,6 @@ packages: stackback@0.0.2: resolution: {integrity: sha512-1XMJE5fQo1jGH6Y/7ebnwPOBEkIEnT4QF32d5R1+VXdXveM0IBMJt8zfaxX1P3QhVwrYe+576+jkANtSS2mBbw==} - std-env@3.10.0: - resolution: {integrity: sha512-5GS12FdOZNliM5mAOxFRg7Ir0pWz8MdpYm6AY6VPkGpbA7ZzmbzNcBJQ0GPvvyWgcY7QAhCgf9Uy89I03faLkg==} - std-env@4.1.0: resolution: {integrity: sha512-Rq7ybcX2RuC55r9oaPVEW7/xu3tj8u4GeBYHBWCychFtzMIr86A7e3PPEBPT37sHStKX3+TiX/Fr/ACmJLVlLQ==} @@ -4384,38 +4207,8 @@ packages: tunnel-agent@0.6.0: resolution: {integrity: sha512-McnNiV1l8RYeY8tBgEpuodCC1mLUdbSN+CYBL7kJsJNInOP8UjDDEwdk6Mw60vdLLrr5NHKZhMAOSrR2NZuQ+w==} - turbo-darwin-64@2.8.11: - resolution: {integrity: sha512-XKaCWaz4OCt77oYYvGCIRpvYD4c/aNaKjRkUpv+e8rN3RZb+5Xsyew4yRO+gaHdMIUhQznXNXfHlhs+/p7lIhA==} - cpu: [x64] - os: [darwin] - - turbo-darwin-arm64@2.8.11: - resolution: {integrity: sha512-VvynLHGUNvQ9k7GZjRPSsRcK4VkioTfFb7O7liAk4nHKjEcMdls7GqxzjVWgJiKz3hWmQGaP9hRa9UUnhVWCxA==} - cpu: [arm64] - os: [darwin] - - turbo-linux-64@2.8.11: - resolution: {integrity: sha512-cbSn37dcm+EmkQ7DD0euy7xV7o2el4GAOr1XujvkAyKjjNvQ+6QIUeDgQcwAx3D17zPpDvfDMJY2dLQadWnkmQ==} - cpu: [x64] - os: [linux] - - turbo-linux-arm64@2.8.11: - resolution: {integrity: sha512-+trymp2s2aBrhS04l6qFxcExzZ8ffndevuUB9c5RCeqsVpZeiWuGQlWNm5XjOmzoMayxRARZ5ma7yiWbGMiLqQ==} - cpu: [arm64] - os: [linux] - - turbo-windows-64@2.8.11: - resolution: {integrity: sha512-3kJjFSM4yw1n9Uzmi+XkAUgCae19l/bH6RJ442xo7mnZm0tpOjo33F+FYHoSVpIWVMd0HG0LDccyafPSdylQbA==} - cpu: [x64] - os: [win32] - - turbo-windows-arm64@2.8.11: - resolution: {integrity: sha512-JOM4uF2vuLsJUvibdR6X9QqdZr6BhC6Nhlrw4LKFPsXZZI/9HHLoqAiYRpE4MuzIwldCH/jVySnWXrI1SKto0g==} - cpu: [arm64] - os: [win32] - - turbo@2.8.11: - resolution: {integrity: sha512-H+rwSHHPLoyPOSoHdmI1zY0zy0GGj1Dmr7SeJW+nZiWLz2nex8EJ+fkdVabxXFMNEux+aywI4Sae8EqhmnOv4A==} + turbo@2.10.10: + resolution: {integrity: sha512-/90KTW+USzvYOPmafRZHVKLBsHXQ5810Ao/HdtJYAqguIhZ+XruS6eIUjqJUDtrSxaZYynNFht68qckGKAOWTA==} hasBin: true tweetnacl@0.14.5: @@ -4521,46 +4314,6 @@ packages: peerDependencies: vite: '*' - vite@7.2.7: - resolution: {integrity: sha512-ITcnkFeR3+fI8P1wMgItjGrR10170d8auB4EpMLPqmx6uxElH3a/hHGQabSHKdqd4FXWO1nFIp9rRn7JQ34ACQ==} - engines: {node: ^20.19.0 || >=22.12.0} - hasBin: true - peerDependencies: - '@types/node': ^20.19.0 || >=22.12.0 - jiti: '>=1.21.0' - less: ^4.0.0 - lightningcss: ^1.21.0 - sass: ^1.70.0 - sass-embedded: ^1.70.0 - stylus: '>=0.54.8' - sugarss: ^5.0.0 - terser: ^5.16.0 - tsx: ^4.8.1 - yaml: ^2.4.2 - peerDependenciesMeta: - '@types/node': - optional: true - jiti: - optional: true - less: - optional: true - lightningcss: - optional: true - sass: - optional: true - sass-embedded: - optional: true - stylus: - optional: true - sugarss: - optional: true - terser: - optional: true - tsx: - optional: true - yaml: - optional: true - vite@8.0.8: resolution: {integrity: sha512-dbU7/iLVa8KZALJyLOBOQ88nOXtNG8vxKuOT4I2mD+Ya70KPceF4IAmDsmU0h1Qsn5bPrvsY9HJstCRh3hG6Uw==} engines: {node: ^20.19.0 || >=22.12.0} @@ -4604,20 +4357,23 @@ packages: yaml: optional: true - vitest@4.0.18: - resolution: {integrity: sha512-hOQuK7h0FGKgBAas7v0mSAsnvrIgAvWmRFjmzpJ7SwFHH3g1k2u37JtYwOwmEKhK6ZO3v9ggDBBm0La1LCK4uQ==} + vitest@4.1.10: + resolution: {integrity: sha512-R9jUTe5S4Qb0HCd4TNqpC7oGcrMssMRGXLW80ubjWsW9VH5GF8y1Y0SFLY9AbqSk6nt0PnOx4H4WNJYZ13GUPw==} engines: {node: ^20.0.0 || ^22.0.0 || >=24.0.0} hasBin: true peerDependencies: '@edge-runtime/vm': '*' '@opentelemetry/api': ^1.9.0 '@types/node': ^20.0.0 || ^22.0.0 || >=24.0.0 - '@vitest/browser-playwright': 4.0.18 - '@vitest/browser-preview': 4.0.18 - '@vitest/browser-webdriverio': 4.0.18 - '@vitest/ui': 4.0.18 + '@vitest/browser-playwright': 4.1.10 + '@vitest/browser-preview': 4.1.10 + '@vitest/browser-webdriverio': 4.1.10 + '@vitest/coverage-istanbul': 4.1.10 + '@vitest/coverage-v8': 4.1.10 + '@vitest/ui': 4.1.10 happy-dom: '*' jsdom: '*' + vite: ^6.0.0 || ^7.0.0 || ^8.0.0 peerDependenciesMeta: '@edge-runtime/vm': optional: true @@ -4631,6 +4387,10 @@ packages: optional: true '@vitest/browser-webdriverio': optional: true + '@vitest/coverage-istanbul': + optional: true + '@vitest/coverage-v8': + optional: true '@vitest/ui': optional: true happy-dom: @@ -4638,55 +4398,14 @@ packages: jsdom: optional: true - vitest@4.1.4: - resolution: {integrity: sha512-tFuJqTxKb8AvfyqMfnavXdzfy3h3sWZRWwfluGbkeR7n0HUev+FmNgZ8SDrRBTVrVCjgH5cA21qGbCffMNtWvg==} - engines: {node: ^20.0.0 || ^22.0.0 || >=24.0.0} - hasBin: true - peerDependencies: - '@edge-runtime/vm': '*' - '@opentelemetry/api': ^1.9.0 - '@types/node': ^20.0.0 || ^22.0.0 || >=24.0.0 - '@vitest/browser-playwright': 4.1.4 - '@vitest/browser-preview': 4.1.4 - '@vitest/browser-webdriverio': 4.1.4 - '@vitest/coverage-istanbul': 4.1.4 - '@vitest/coverage-v8': 4.1.4 - '@vitest/ui': 4.1.4 - happy-dom: '*' - jsdom: '*' - vite: ^6.0.0 || ^7.0.0 || ^8.0.0 - peerDependenciesMeta: - '@edge-runtime/vm': - optional: true - '@opentelemetry/api': - optional: true - '@types/node': - optional: true - '@vitest/browser-playwright': - optional: true - '@vitest/browser-preview': - optional: true - '@vitest/browser-webdriverio': - optional: true - '@vitest/coverage-istanbul': - optional: true - '@vitest/coverage-v8': - optional: true - '@vitest/ui': - optional: true - happy-dom: - optional: true - jsdom: - optional: true - - vscode-uri@3.1.0: - resolution: {integrity: sha512-/BpdSx+yCQGnCvecbyXdxHDkuk55/G3xwnC0GqY4gmQ3j+A+g8kzzgB4Nk/SINjqn6+waqw3EgbVF2QKExkRxQ==} - - vue-component-type-helpers@2.2.12: - resolution: {integrity: sha512-YbGqHZ5/eW4SnkPNR44mKVc6ZKQoRs/Rux1sxC6rdwXb4qpbOSYfDr9DsTHolOTGmIKgM9j141mZbBeg05R1pw==} - - vue-tsc@3.3.5: - resolution: {integrity: sha512-Rzh/G2MmNlMSAMTiQEjDrsb4dgB/jbtEM47rVN2NtidF1dfb/q4w4QvpQBtW5+y3y5H27Hjh7deVwk+YB02fNg==} + vscode-uri@3.1.0: + resolution: {integrity: sha512-/BpdSx+yCQGnCvecbyXdxHDkuk55/G3xwnC0GqY4gmQ3j+A+g8kzzgB4Nk/SINjqn6+waqw3EgbVF2QKExkRxQ==} + + vue-component-type-helpers@2.2.12: + resolution: {integrity: sha512-YbGqHZ5/eW4SnkPNR44mKVc6ZKQoRs/Rux1sxC6rdwXb4qpbOSYfDr9DsTHolOTGmIKgM9j141mZbBeg05R1pw==} + + vue-tsc@3.3.5: + resolution: {integrity: sha512-Rzh/G2MmNlMSAMTiQEjDrsb4dgB/jbtEM47rVN2NtidF1dfb/q4w4QvpQBtW5+y3y5H27Hjh7deVwk+YB02fNg==} hasBin: true peerDependencies: typescript: '>=5.0.0' @@ -4699,8 +4418,8 @@ packages: typescript: optional: true - vue@3.5.25: - resolution: {integrity: sha512-YLVdgv2K13WJ6n+kD5owehKtEXwdwXuj2TTyJMsO7pSeKw2bfRNZGjhB7YzrpbMYj5b5QsUebHpOqR3R3ziy/g==} + vue@3.5.41: + resolution: {integrity: sha512-2laE0p+aK+/AOPG/XL/WepOs/GlK755LJ1XECi9kDUrz1FKNw8rb2Xzlw9JS1rqEV55nb0ttsKxVlTCcd+R5cg==} peerDependencies: typescript: '*' peerDependenciesMeta: @@ -5433,10 +5152,14 @@ snapshots: '@babel/helper-string-parser@7.27.1': {} + '@babel/helper-string-parser@7.29.7': {} + '@babel/helper-string-parser@8.0.0': {} '@babel/helper-validator-identifier@7.28.5': {} + '@babel/helper-validator-identifier@7.29.7': {} + '@babel/helper-validator-identifier@8.0.2': {} '@babel/helper-validator-option@7.27.1': {} @@ -5450,6 +5173,10 @@ snapshots: dependencies: '@babel/types': 7.29.0 + '@babel/parser@7.29.8': + dependencies: + '@babel/types': 7.29.8 + '@babel/parser@8.0.0': dependencies: '@babel/types': 8.0.0 @@ -5481,6 +5208,11 @@ snapshots: '@babel/helper-string-parser': 7.27.1 '@babel/helper-validator-identifier': 7.28.5 + '@babel/types@7.29.8': + dependencies: + '@babel/helper-string-parser': 7.29.7 + '@babel/helper-validator-identifier': 7.29.7 + '@babel/types@8.0.0': dependencies: '@babel/helper-string-parser': 8.0.0 @@ -5682,101 +5414,11 @@ snapshots: tslib: 2.8.1 optional: true - '@esbuild/aix-ppc64@0.25.12': - optional: true - - '@esbuild/android-arm64@0.25.12': - optional: true - - '@esbuild/android-arm@0.25.12': - optional: true - - '@esbuild/android-x64@0.25.12': - optional: true - - '@esbuild/darwin-arm64@0.25.12': - optional: true - - '@esbuild/darwin-x64@0.25.12': - optional: true - - '@esbuild/freebsd-arm64@0.25.12': - optional: true - - '@esbuild/freebsd-x64@0.25.12': - optional: true - - '@esbuild/linux-arm64@0.25.12': - optional: true - - '@esbuild/linux-arm@0.25.12': - optional: true - - '@esbuild/linux-ia32@0.25.12': - optional: true - - '@esbuild/linux-loong64@0.25.12': - optional: true - - '@esbuild/linux-mips64el@0.25.12': - optional: true - - '@esbuild/linux-ppc64@0.25.12': - optional: true - - '@esbuild/linux-riscv64@0.25.12': - optional: true - - '@esbuild/linux-s390x@0.25.12': - optional: true - - '@esbuild/linux-x64@0.25.12': - optional: true - - '@esbuild/netbsd-arm64@0.25.12': - optional: true - - '@esbuild/netbsd-x64@0.25.12': - optional: true - - '@esbuild/openbsd-arm64@0.25.12': - optional: true - - '@esbuild/openbsd-x64@0.25.12': - optional: true - - '@esbuild/openharmony-arm64@0.25.12': - optional: true - - '@esbuild/sunos-x64@0.25.12': - optional: true - - '@esbuild/win32-arm64@0.25.12': - optional: true - - '@esbuild/win32-ia32@0.25.12': - optional: true - - '@esbuild/win32-x64@0.25.12': - optional: true - - '@grpc/grpc-js@1.13.4': - dependencies: - '@grpc/proto-loader': 0.7.13 - '@js-sdsl/ordered-map': 4.4.2 - '@grpc/grpc-js@1.14.4': dependencies: '@grpc/proto-loader': 0.8.1 '@js-sdsl/ordered-map': 4.4.2 - '@grpc/proto-loader@0.7.13': - dependencies: - lodash.camelcase: 4.3.0 - long: 5.3.2 - protobufjs: 7.4.0 - yargs: 17.7.2 - '@grpc/proto-loader@0.8.1': dependencies: lodash.camelcase: 4.3.0 @@ -5924,23 +5566,23 @@ snapshots: '@istanbuljs/schema@0.1.3': {} - '@jitl/quickjs-ffi-types@0.31.0': {} + '@jitl/quickjs-ffi-types@0.32.0': {} - '@jitl/quickjs-wasmfile-debug-asyncify@0.31.0': + '@jitl/quickjs-wasmfile-debug-asyncify@0.32.0': dependencies: - '@jitl/quickjs-ffi-types': 0.31.0 + '@jitl/quickjs-ffi-types': 0.32.0 - '@jitl/quickjs-wasmfile-debug-sync@0.31.0': + '@jitl/quickjs-wasmfile-debug-sync@0.32.0': dependencies: - '@jitl/quickjs-ffi-types': 0.31.0 + '@jitl/quickjs-ffi-types': 0.32.0 - '@jitl/quickjs-wasmfile-release-asyncify@0.31.0': + '@jitl/quickjs-wasmfile-release-asyncify@0.32.0': dependencies: - '@jitl/quickjs-ffi-types': 0.31.0 + '@jitl/quickjs-ffi-types': 0.32.0 - '@jitl/quickjs-wasmfile-release-sync@0.31.0': + '@jitl/quickjs-wasmfile-release-sync@0.32.0': dependencies: - '@jitl/quickjs-ffi-types': 0.31.0 + '@jitl/quickjs-ffi-types': 0.32.0 '@jridgewell/gen-mapping@0.3.13': dependencies: @@ -6028,10 +5670,10 @@ snapshots: '@microsoft/tsdoc@0.16.0': {} - '@milaboratories/columns-collection-driver@0.2.2': + '@milaboratories/columns-collection-driver@0.2.3': dependencies: '@milaboratories/helpers': 1.14.5 - '@milaboratories/pl-model-common': 1.47.2 + '@milaboratories/pl-model-common': 1.47.3 '@milaboratories/computable@2.9.8': dependencies: @@ -6044,13 +5686,13 @@ snapshots: '@milaboratories/helpers@1.14.5': {} - '@milaboratories/pf-driver@1.8.4(@bytecodealliance/preview2-shim@0.17.8)': + '@milaboratories/pf-driver@1.9.0(@bytecodealliance/preview2-shim@0.17.8)': dependencies: '@milaboratories/helpers': 1.14.5 + '@milaboratories/pf-spec': 1.0.0(@bytecodealliance/preview2-shim@0.17.8) '@milaboratories/pframes-rs-node': 1.1.56 - '@milaboratories/pframes-rs-wasm': 1.1.56(@bytecodealliance/preview2-shim@0.17.8)(@milaboratories/pl-model-common@1.47.2)(@milaboratories/pl-model-middle-layer@1.30.14) - '@milaboratories/pl-model-common': 1.47.2 - '@milaboratories/pl-model-middle-layer': 1.30.14 + '@milaboratories/pl-model-common': 1.47.3 + '@milaboratories/pl-model-middle-layer': 1.31.0 '@milaboratories/ts-helpers': 1.8.6 es-toolkit: 1.42.0 lru-cache: 11.2.4 @@ -6059,16 +5701,21 @@ snapshots: - encoding - supports-color - '@milaboratories/pf-spec-driver@1.4.23(@bytecodealliance/preview2-shim@0.17.8)': + '@milaboratories/pf-spec-driver@1.5.0(@bytecodealliance/preview2-shim@0.17.8)': dependencies: '@milaboratories/helpers': 1.14.5 - '@milaboratories/pframes-rs-wasm': 1.1.56(@bytecodealliance/preview2-shim@0.17.8)(@milaboratories/pl-model-common@1.47.2)(@milaboratories/pl-model-middle-layer@1.30.14) - '@milaboratories/pl-model-common': 1.47.2 - '@milaboratories/pl-model-middle-layer': 1.30.14 + '@milaboratories/pf-spec': 1.0.0(@bytecodealliance/preview2-shim@0.17.8) + '@milaboratories/pl-model-common': 1.47.3 '@noble/hashes': 2.2.0 transitivePeerDependencies: - '@bytecodealliance/preview2-shim' + '@milaboratories/pf-spec@1.0.0(@bytecodealliance/preview2-shim@0.17.8)': + dependencies: + '@bytecodealliance/preview2-shim': 0.17.8 + '@milaboratories/pl-model-common': 1.47.3 + '@milaboratories/pl-model-middle-layer': 1.31.0 + '@milaboratories/pframes-rs-node@1.1.56': dependencies: '@mapbox/node-pre-gyp': 2.0.3 @@ -6091,31 +5738,6 @@ snapshots: '@milaboratories/pframes-rs-wasip2@1.1.56': {} - '@milaboratories/pframes-rs-wasm@1.1.56(@bytecodealliance/preview2-shim@0.17.8)(@milaboratories/pl-model-common@1.47.2)(@milaboratories/pl-model-middle-layer@1.30.14)': - dependencies: - '@bytecodealliance/preview2-shim': 0.17.8 - '@milaboratories/pframes-rs-wasip2': 1.1.56 - '@milaboratories/pl-model-common': 1.47.2 - '@milaboratories/pl-model-middle-layer': 1.30.14 - - '@milaboratories/pl-client@3.14.3': - dependencies: - '@grpc/grpc-js': 1.13.4 - '@milaboratories/pl-http': 1.2.4 - '@milaboratories/pl-model-common': 1.47.2 - '@milaboratories/ts-helpers': 1.8.6 - '@protobuf-ts/grpc-transport': 2.11.1(@grpc/grpc-js@1.13.4) - '@protobuf-ts/runtime': 2.11.1 - '@protobuf-ts/runtime-rpc': 2.11.1 - canonicalize: 2.1.0 - denque: 2.1.0 - long: 5.3.2 - lru-cache: 11.2.4 - openapi-fetch: 0.15.0 - undici: 7.16.0 - utility-types: 3.11.0 - yaml: 2.8.1 - '@milaboratories/pl-client@3.14.6': dependencies: '@grpc/grpc-js': 1.14.4 @@ -6140,12 +5762,12 @@ snapshots: upath: 2.0.1 yaml: 2.8.1 - '@milaboratories/pl-deployments@3.0.13': + '@milaboratories/pl-deployments@3.0.15': dependencies: '@milaboratories/pl-config': 1.8.5 - '@milaboratories/pl-healthcheck': 1.0.4 + '@milaboratories/pl-healthcheck': 1.0.5 '@milaboratories/pl-http': 1.2.4 - '@milaboratories/pl-model-common': 1.47.2 + '@milaboratories/pl-model-common': 1.47.3 '@milaboratories/ts-helpers': 1.8.6 decompress: 4.2.1 ssh2: 1.16.0 @@ -6155,16 +5777,16 @@ snapshots: yaml: 2.8.1 zod: 3.25.76 - '@milaboratories/pl-drivers@1.16.11': + '@milaboratories/pl-drivers@1.16.15': dependencies: - '@grpc/grpc-js': 1.13.4 + '@grpc/grpc-js': 1.14.4 '@milaboratories/computable': 2.9.8 '@milaboratories/helpers': 1.14.5 - '@milaboratories/pl-client': 3.14.3 - '@milaboratories/pl-model-common': 1.47.2 - '@milaboratories/pl-tree': 1.13.2 + '@milaboratories/pl-client': 3.14.6 + '@milaboratories/pl-model-common': 1.47.3 + '@milaboratories/pl-tree': 1.13.6 '@milaboratories/ts-helpers': 1.8.6 - '@protobuf-ts/grpc-transport': 2.11.1(@grpc/grpc-js@1.13.4) + '@protobuf-ts/grpc-transport': 2.11.1(@grpc/grpc-js@1.14.4) '@protobuf-ts/plugin': 2.11.1 '@protobuf-ts/runtime': 2.11.1 '@protobuf-ts/runtime-rpc': 2.11.1 @@ -6189,17 +5811,17 @@ snapshots: json-stringify-safe: 5.0.1 zod: 3.23.8 - '@milaboratories/pl-errors@1.4.32': + '@milaboratories/pl-errors@1.4.35': dependencies: - '@milaboratories/pl-client': 3.14.3 + '@milaboratories/pl-client': 3.14.6 '@milaboratories/ts-helpers': 1.8.6 zod: 3.25.76 - '@milaboratories/pl-healthcheck@1.0.4': + '@milaboratories/pl-healthcheck@1.0.5': dependencies: - '@grpc/grpc-js': 1.13.4 + '@grpc/grpc-js': 1.14.4 '@milaboratories/ts-helpers': 1.8.6 - '@protobuf-ts/grpc-transport': 2.11.1(@grpc/grpc-js@1.13.4) + '@protobuf-ts/grpc-transport': 2.11.1(@grpc/grpc-js@1.14.4) '@protobuf-ts/runtime': 2.11.1 '@protobuf-ts/runtime-rpc': 2.11.1 @@ -6207,34 +5829,33 @@ snapshots: dependencies: undici: 7.16.0 - '@milaboratories/pl-middle-layer@1.66.8(@bytecodealliance/preview2-shim@0.17.8)(@types/node@25.3.2)': + '@milaboratories/pl-middle-layer@1.66.19(@bytecodealliance/preview2-shim@0.17.8)(@types/node@25.3.2)': dependencies: - '@milaboratories/columns-collection-driver': 0.2.2 + '@milaboratories/columns-collection-driver': 0.2.3 '@milaboratories/computable': 2.9.8 '@milaboratories/helpers': 1.14.5 - '@milaboratories/pf-driver': 1.8.4(@bytecodealliance/preview2-shim@0.17.8) - '@milaboratories/pf-spec-driver': 1.4.23(@bytecodealliance/preview2-shim@0.17.8) + '@milaboratories/pf-driver': 1.9.0(@bytecodealliance/preview2-shim@0.17.8) + '@milaboratories/pf-spec-driver': 1.5.0(@bytecodealliance/preview2-shim@0.17.8) '@milaboratories/pframes-rs-node': 1.1.56 - '@milaboratories/pframes-rs-wasm': 1.1.56(@bytecodealliance/preview2-shim@0.17.8)(@milaboratories/pl-model-common@1.47.2)(@milaboratories/pl-model-middle-layer@1.30.14) - '@milaboratories/pl-client': 3.14.3 - '@milaboratories/pl-deployments': 3.0.13 - '@milaboratories/pl-drivers': 1.16.11 - '@milaboratories/pl-errors': 1.4.32 + '@milaboratories/pl-client': 3.14.6 + '@milaboratories/pl-deployments': 3.0.15 + '@milaboratories/pl-drivers': 1.16.15 + '@milaboratories/pl-errors': 1.4.35 '@milaboratories/pl-http': 1.2.4 - '@milaboratories/pl-model-backend': 1.4.17 - '@milaboratories/pl-model-common': 1.47.2 - '@milaboratories/pl-model-middle-layer': 1.30.14 - '@milaboratories/pl-tree': 1.13.2 + '@milaboratories/pl-model-backend': 1.4.20 + '@milaboratories/pl-model-common': 1.47.3 + '@milaboratories/pl-model-middle-layer': 1.31.0 + '@milaboratories/pl-tree': 1.13.6 '@milaboratories/resolve-helper': 1.1.3 '@milaboratories/ts-helpers': 1.8.6 - '@platforma-sdk/block-tools': 2.12.8(@types/node@25.3.2) - '@platforma-sdk/model': 1.80.8 + '@platforma-sdk/block-tools': 2.13.0(@types/node@25.3.2) + '@platforma-sdk/model': 1.81.1 '@platforma-sdk/workflow-tengo': 6.8.2 canonicalize: 2.1.0 denque: 2.1.0 es-toolkit: 1.42.0 lru-cache: 11.2.4 - quickjs-emscripten: 0.31.0 + quickjs-emscripten: 0.32.0 semver: 7.8.5 undici: 7.16.0 utility-types: 3.11.0 @@ -6247,12 +5868,6 @@ snapshots: - encoding - supports-color - '@milaboratories/pl-model-backend@1.4.17': - dependencies: - '@milaboratories/pl-client': 3.14.3 - canonicalize: 2.1.0 - zod: 3.25.76 - '@milaboratories/pl-model-backend@1.4.20': dependencies: '@milaboratories/pl-client': 3.14.6 @@ -6272,14 +5887,6 @@ snapshots: canonicalize: 2.1.0 zod: 3.25.76 - '@milaboratories/pl-model-common@1.47.2': - dependencies: - '@milaboratories/helpers': 1.14.5 - '@milaboratories/pl-error-like': 1.12.10 - canonicalize: 2.1.0 - es-toolkit: 1.42.0 - zod: 3.25.76 - '@milaboratories/pl-model-common@1.47.3': dependencies: '@milaboratories/helpers': 1.14.5 @@ -6288,14 +5895,6 @@ snapshots: es-toolkit: 1.42.0 zod: 3.25.76 - '@milaboratories/pl-model-middle-layer@1.30.14': - dependencies: - '@milaboratories/helpers': 1.14.5 - '@milaboratories/pl-model-common': 1.47.2 - es-toolkit: 1.42.0 - utility-types: 3.11.0 - zod: 3.25.76 - '@milaboratories/pl-model-middle-layer@1.30.7': dependencies: '@milaboratories/helpers': 1.14.2 @@ -6312,11 +5911,11 @@ snapshots: utility-types: 3.11.0 zod: 3.25.76 - '@milaboratories/pl-tree@1.13.2': + '@milaboratories/pl-tree@1.13.6': dependencies: '@milaboratories/computable': 2.9.8 - '@milaboratories/pl-client': 3.14.3 - '@milaboratories/pl-errors': 1.4.32 + '@milaboratories/pl-client': 3.14.6 + '@milaboratories/pl-errors': 1.4.35 '@milaboratories/ts-helpers': 1.8.6 denque: 2.1.0 utility-types: 3.11.0 @@ -6326,9 +5925,9 @@ snapshots: dependencies: '@platforma-open/milaboratories.software-ptabler.schema': 1.13.2 - '@milaboratories/ptabler-expression-js@1.2.36': + '@milaboratories/ptabler-expression-js@1.2.37': dependencies: - '@platforma-open/milaboratories.software-ptabler.schema': 1.15.20 + '@platforma-open/milaboratories.software-ptabler.schema': 1.15.21 '@milaboratories/resolve-helper@1.1.3': {} @@ -6336,9 +5935,9 @@ snapshots: '@milaboratories/tengo-tester@1.6.4': {} - '@milaboratories/ts-builder@1.6.1(@types/node@25.3.2)(rollup@4.53.3)(vue@3.5.24(typescript@5.6.3))(yaml@2.8.1)': + '@milaboratories/ts-builder@1.6.2(@types/node@25.3.2)(rollup@4.53.3)(vue@3.5.24(typescript@5.6.3))(yaml@2.8.1)': dependencies: - '@milaboratories/ts-configs': 1.3.1 + '@milaboratories/ts-configs': 1.4.0 '@vitejs/plugin-vue': 6.0.6(vite@8.0.8(@types/node@25.3.2)(yaml@2.8.1))(vue@3.5.24(typescript@5.6.3)) commander: 15.0.0 jsonc-parser: 3.3.1 @@ -6377,10 +5976,10 @@ snapshots: - vue - yaml - '@milaboratories/ts-builder@1.6.1(@types/node@25.3.2)(rollup@4.53.3)(vue@3.5.25(typescript@5.6.3))(yaml@2.8.1)': + '@milaboratories/ts-builder@1.6.2(@types/node@25.3.2)(rollup@4.53.3)(vue@3.5.41(typescript@5.6.3))(yaml@2.8.1)': dependencies: - '@milaboratories/ts-configs': 1.3.1 - '@vitejs/plugin-vue': 6.0.6(vite@8.0.8(@types/node@25.3.2)(yaml@2.8.1))(vue@3.5.25(typescript@5.6.3)) + '@milaboratories/ts-configs': 1.4.0 + '@vitejs/plugin-vue': 6.0.6(vite@8.0.8(@types/node@25.3.2)(yaml@2.8.1))(vue@3.5.41(typescript@5.6.3)) commander: 15.0.0 jsonc-parser: 3.3.1 oxfmt: 0.35.0 @@ -6418,10 +6017,10 @@ snapshots: - vue - yaml - '@milaboratories/ts-builder@1.6.1(@types/node@25.3.2)(rollup@4.53.3)(vue@3.5.25(typescript@5.9.3))(yaml@2.8.1)': + '@milaboratories/ts-builder@1.6.2(@types/node@25.3.2)(rollup@4.53.3)(vue@3.5.41(typescript@5.9.3))(yaml@2.8.1)': dependencies: - '@milaboratories/ts-configs': 1.3.1 - '@vitejs/plugin-vue': 6.0.6(vite@8.0.8(@types/node@25.3.2)(yaml@2.8.1))(vue@3.5.25(typescript@5.9.3)) + '@milaboratories/ts-configs': 1.4.0 + '@vitejs/plugin-vue': 6.0.6(vite@8.0.8(@types/node@25.3.2)(yaml@2.8.1))(vue@3.5.41(typescript@5.9.3)) commander: 15.0.0 jsonc-parser: 3.3.1 oxfmt: 0.35.0 @@ -6459,7 +6058,7 @@ snapshots: - vue - yaml - '@milaboratories/ts-configs@1.3.1': {} + '@milaboratories/ts-configs@1.4.0': {} '@milaboratories/ts-helpers@1.8.6': dependencies: @@ -6467,10 +6066,10 @@ snapshots: canonicalize: 2.1.0 denque: 2.1.0 - '@milaboratories/uikit@2.15.17(typescript@5.6.3)': + '@milaboratories/uikit@2.15.23(typescript@5.6.3)': dependencies: '@milaboratories/helpers': 1.14.5 - '@platforma-sdk/model': 1.80.8 + '@platforma-sdk/model': 1.81.1 '@types/d3-array': 3.2.1 '@types/d3-axis': 3.0.6 '@types/d3-scale': 4.0.9 @@ -6801,30 +6400,45 @@ snapshots: '@pkgjs/parseargs@0.11.0': optional: true - '@platforma-open/milaboratories.runenv-python-3.12.10-atls@1.2.4': {} + '@platforma-open/milaboratories.runenv-python-3.12.10-atls@1.2.7': {} - '@platforma-open/milaboratories.runenv-python-3.12.10-h5ad@1.1.4': {} + '@platforma-open/milaboratories.runenv-python-3.12.10-clustering@0.1.1': {} + + '@platforma-open/milaboratories.runenv-python-3.12.10-h5ad@1.1.5': {} + + '@platforma-open/milaboratories.runenv-python-3.12.10-humanness@0.2.0': {} '@platforma-open/milaboratories.runenv-python-3.12.10-parapred@1.1.0': {} - '@platforma-open/milaboratories.runenv-python-3.12.10-rapids@1.4.4': {} + '@platforma-open/milaboratories.runenv-python-3.12.10-pgen@0.2.0': {} + + '@platforma-open/milaboratories.runenv-python-3.12.10-rapids@1.7.2': {} - '@platforma-open/milaboratories.runenv-python-3.12.10-sccoda@1.3.5': {} + '@platforma-open/milaboratories.runenv-python-3.12.10-sccoda@1.3.6': {} - '@platforma-open/milaboratories.runenv-python-3.12.10@1.3.6': {} + '@platforma-open/milaboratories.runenv-python-3.12.10-scientific-slim@1.1.0': {} - '@platforma-open/milaboratories.runenv-python-3@1.7.8': + '@platforma-open/milaboratories.runenv-python-3.12.10-torch-cuda@0.2.0': {} + + '@platforma-open/milaboratories.runenv-python-3.12.10@1.3.24': {} + + '@platforma-open/milaboratories.runenv-python-3@1.11.6': dependencies: - '@platforma-open/milaboratories.runenv-python-3.12.10': 1.3.6 - '@platforma-open/milaboratories.runenv-python-3.12.10-atls': 1.2.4 - '@platforma-open/milaboratories.runenv-python-3.12.10-h5ad': 1.1.4 + '@platforma-open/milaboratories.runenv-python-3.12.10': 1.3.24 + '@platforma-open/milaboratories.runenv-python-3.12.10-atls': 1.2.7 + '@platforma-open/milaboratories.runenv-python-3.12.10-clustering': 0.1.1 + '@platforma-open/milaboratories.runenv-python-3.12.10-h5ad': 1.1.5 + '@platforma-open/milaboratories.runenv-python-3.12.10-humanness': 0.2.0 '@platforma-open/milaboratories.runenv-python-3.12.10-parapred': 1.1.0 - '@platforma-open/milaboratories.runenv-python-3.12.10-rapids': 1.4.4 - '@platforma-open/milaboratories.runenv-python-3.12.10-sccoda': 1.3.5 + '@platforma-open/milaboratories.runenv-python-3.12.10-pgen': 0.2.0 + '@platforma-open/milaboratories.runenv-python-3.12.10-rapids': 1.7.2 + '@platforma-open/milaboratories.runenv-python-3.12.10-sccoda': 1.3.6 + '@platforma-open/milaboratories.runenv-python-3.12.10-scientific-slim': 1.1.0 + '@platforma-open/milaboratories.runenv-python-3.12.10-torch-cuda': 0.2.0 '@platforma-open/milaboratories.samples-and-data.model@1.11.2': dependencies: - '@platforma-sdk/model': 1.80.8 + '@platforma-sdk/model': 1.81.1 zod: 3.23.8 '@platforma-open/milaboratories.samples-and-data.model@2.5.3': @@ -6857,9 +6471,9 @@ snapshots: dependencies: '@milaboratories/pl-model-common': 1.23.0 - '@platforma-open/milaboratories.software-ptabler.schema@1.15.20': + '@platforma-open/milaboratories.software-ptabler.schema@1.15.21': dependencies: - '@milaboratories/pl-model-common': 1.47.2 + '@milaboratories/pl-model-common': 1.47.3 '@platforma-open/milaboratories.software-ptabler@1.14.0': {} @@ -6926,31 +6540,6 @@ snapshots: '@platforma-open/milaboratories.software-small-binaries.mnz-client': 1.6.5 '@platforma-open/milaboratories.software-small-binaries.table-converter': 1.3.5 - '@platforma-sdk/block-tools@2.12.8(@types/node@25.3.2)': - dependencies: - '@aws-sdk/client-ecr-public': 3.859.0 - '@aws-sdk/client-s3': 3.859.0 - '@inquirer/prompts': 7.10.1(@types/node@25.3.2) - '@milaboratories/pl-http': 1.2.4 - '@milaboratories/pl-model-backend': 1.4.17 - '@milaboratories/pl-model-common': 1.47.2 - '@milaboratories/pl-model-middle-layer': 1.30.14 - '@milaboratories/resolve-helper': 1.1.3 - '@milaboratories/ts-helpers': 1.8.6 - '@platforma-sdk/blocks-deps-updater': 2.2.0 - '@platforma-sdk/package-builder-lib': 1.2.1 - canonicalize: 2.1.0 - commander: 15.0.0 - lru-cache: 11.2.4 - mime-types: 2.1.35 - tar: 7.4.3 - undici: 7.16.0 - yaml: 2.8.1 - zod: 3.25.76 - transitivePeerDependencies: - - '@types/node' - - aws-crt - '@platforma-sdk/block-tools@2.13.0(@types/node@25.3.2)': dependencies: '@aws-sdk/client-ecr-public': 3.859.0 @@ -6990,13 +6579,13 @@ snapshots: utility-types: 3.11.0 zod: 3.23.8 - '@platforma-sdk/model@1.80.8': + '@platforma-sdk/model@1.81.1': dependencies: '@milaboratories/helpers': 1.14.5 '@milaboratories/pl-error-like': 1.12.10 - '@milaboratories/pl-model-common': 1.47.2 - '@milaboratories/pl-model-middle-layer': 1.30.14 - '@milaboratories/ptabler-expression-js': 1.2.36 + '@milaboratories/pl-model-common': 1.47.3 + '@milaboratories/pl-model-middle-layer': 1.31.0 + '@milaboratories/ptabler-expression-js': 1.2.37 canonicalize: 2.1.0 es-toolkit: 1.42.0 fast-json-patch: 3.1.1 @@ -7004,19 +6593,6 @@ snapshots: utility-types: 3.11.0 zod: 3.25.76 - '@platforma-sdk/package-builder-lib@1.2.1': - dependencies: - '@aws-sdk/client-s3': 3.859.0 - '@aws-sdk/lib-storage': 3.859.0(@aws-sdk/client-s3@3.859.0) - '@milaboratories/resolve-helper': 1.1.3 - archiver: 7.0.1 - undici: 7.16.0 - winston: 3.17.0 - yaml: 2.8.1 - zod: 3.25.76 - transitivePeerDependencies: - - aws-crt - '@platforma-sdk/package-builder-lib@1.3.0': dependencies: '@aws-sdk/client-s3': 3.859.0 @@ -7039,42 +6615,15 @@ snapshots: commander: 15.0.0 winston: 3.17.0 - '@platforma-sdk/test@1.80.9(@bytecodealliance/preview2-shim@0.17.8)(@types/node@25.3.2)(vite@7.2.7(@types/node@25.3.2)(lightningcss@1.32.0)(yaml@2.8.1))': + '@platforma-sdk/test@1.81.3(@bytecodealliance/preview2-shim@0.17.8)(@types/node@25.3.2)(vite@8.0.8(@types/node@25.3.2)(yaml@2.8.1))': dependencies: '@milaboratories/computable': 2.9.8 - '@milaboratories/pl-client': 3.14.3 - '@milaboratories/pl-middle-layer': 1.66.8(@bytecodealliance/preview2-shim@0.17.8)(@types/node@25.3.2) - '@milaboratories/pl-tree': 1.13.2 - '@platforma-sdk/model': 1.80.8 - '@vitest/coverage-istanbul': 4.1.4(vitest@4.0.18(@types/node@25.3.2)(lightningcss@1.32.0)(yaml@2.8.1)) - vitest: 4.1.4(@types/node@25.3.2)(@vitest/coverage-istanbul@4.1.4)(vite@7.2.7(@types/node@25.3.2)(lightningcss@1.32.0)(yaml@2.8.1)) - transitivePeerDependencies: - - '@bytecodealliance/preview2-shim' - - '@edge-runtime/vm' - - '@opentelemetry/api' - - '@types/node' - - '@vitest/browser-playwright' - - '@vitest/browser-preview' - - '@vitest/browser-webdriverio' - - '@vitest/coverage-v8' - - '@vitest/ui' - - aws-crt - - encoding - - happy-dom - - jsdom - - msw - - supports-color - - vite - - '@platforma-sdk/test@1.80.9(@bytecodealliance/preview2-shim@0.17.8)(@types/node@25.3.2)(vite@8.0.8(@types/node@25.3.2)(yaml@2.8.1))': - dependencies: - '@milaboratories/computable': 2.9.8 - '@milaboratories/pl-client': 3.14.3 - '@milaboratories/pl-middle-layer': 1.66.8(@bytecodealliance/preview2-shim@0.17.8)(@types/node@25.3.2) - '@milaboratories/pl-tree': 1.13.2 - '@platforma-sdk/model': 1.80.8 - '@vitest/coverage-istanbul': 4.1.4(vitest@4.1.4) - vitest: 4.1.4(@types/node@25.3.2)(@vitest/coverage-istanbul@4.1.4)(vite@8.0.8(@types/node@25.3.2)(yaml@2.8.1)) + '@milaboratories/pl-client': 3.14.6 + '@milaboratories/pl-middle-layer': 1.66.19(@bytecodealliance/preview2-shim@0.17.8)(@types/node@25.3.2) + '@milaboratories/pl-tree': 1.13.6 + '@platforma-sdk/model': 1.81.1 + '@vitest/coverage-istanbul': 4.1.4(vitest@4.1.10) + vitest: 4.1.10(@types/node@25.3.2)(@vitest/coverage-istanbul@4.1.4)(vite@8.0.8(@types/node@25.3.2)(yaml@2.8.1)) transitivePeerDependencies: - '@bytecodealliance/preview2-shim' - '@edge-runtime/vm' @@ -7093,13 +6642,13 @@ snapshots: - supports-color - vite - '@platforma-sdk/ui-vue@1.80.9(@bytecodealliance/preview2-shim@0.17.8)(typescript@5.6.3)': + '@platforma-sdk/ui-vue@1.81.1(@bytecodealliance/preview2-shim@0.17.8)(typescript@5.6.3)': dependencies: - '@milaboratories/columns-collection-driver': 0.2.2 - '@milaboratories/pf-spec-driver': 1.4.23(@bytecodealliance/preview2-shim@0.17.8) - '@milaboratories/pl-model-common': 1.47.2 - '@milaboratories/uikit': 2.15.17(typescript@5.6.3) - '@platforma-sdk/model': 1.80.8 + '@milaboratories/columns-collection-driver': 0.2.3 + '@milaboratories/pf-spec-driver': 1.5.0(@bytecodealliance/preview2-shim@0.17.8) + '@milaboratories/pl-model-common': 1.47.3 + '@milaboratories/uikit': 2.15.23(typescript@5.6.3) + '@platforma-sdk/model': 1.81.1 '@types/d3-format': 3.0.4 '@types/node': 24.5.2 '@types/semver': 7.7.0 @@ -7145,12 +6694,6 @@ snapshots: '@platforma-open/milaboratories.software-ptexter': 1.2.4 '@platforma-open/milaboratories.software-small-binaries': 2.1.1 - '@protobuf-ts/grpc-transport@2.11.1(@grpc/grpc-js@1.13.4)': - dependencies: - '@grpc/grpc-js': 1.13.4 - '@protobuf-ts/runtime': 2.11.1 - '@protobuf-ts/runtime-rpc': 2.11.1 - '@protobuf-ts/grpc-transport@2.11.1(@grpc/grpc-js@1.14.4)': dependencies: '@grpc/grpc-js': 1.14.4 @@ -7180,33 +6723,20 @@ snapshots: '@protobufjs/base64@1.1.2': {} - '@protobufjs/codegen@2.0.4': {} - '@protobufjs/codegen@2.0.5': {} - '@protobufjs/eventemitter@1.1.0': {} - '@protobufjs/eventemitter@1.1.1': {} - '@protobufjs/fetch@1.1.0': - dependencies: - '@protobufjs/aspromise': 1.1.2 - '@protobufjs/inquire': 1.1.0 - '@protobufjs/fetch@1.1.1': dependencies: '@protobufjs/aspromise': 1.1.2 '@protobufjs/float@1.0.2': {} - '@protobufjs/inquire@1.1.0': {} - '@protobufjs/path@1.1.2': {} '@protobufjs/pool@1.1.0': {} - '@protobufjs/utf8@1.1.0': {} - '@protobufjs/utf8@1.1.2': {} '@rolldown/binding-android-arm64@1.0.0-rc.15': @@ -7764,6 +7294,24 @@ snapshots: '@standard-schema/spec@1.1.0': {} + '@turbo/darwin-64@2.10.10': + optional: true + + '@turbo/darwin-arm64@2.10.10': + optional: true + + '@turbo/linux-64@2.10.10': + optional: true + + '@turbo/linux-arm64@2.10.10': + optional: true + + '@turbo/windows-64@2.10.10': + optional: true + + '@turbo/windows-arm64@2.10.10': + optional: true + '@tybys/wasm-util@0.10.3': dependencies: tslib: 2.8.1 @@ -7844,19 +7392,19 @@ snapshots: vite: 8.0.8(@types/node@25.3.2)(yaml@2.8.1) vue: 3.5.24(typescript@5.6.3) - '@vitejs/plugin-vue@6.0.6(vite@8.0.8(@types/node@25.3.2)(yaml@2.8.1))(vue@3.5.25(typescript@5.6.3))': + '@vitejs/plugin-vue@6.0.6(vite@8.0.8(@types/node@25.3.2)(yaml@2.8.1))(vue@3.5.41(typescript@5.6.3))': dependencies: '@rolldown/pluginutils': 1.0.0-rc.13 vite: 8.0.8(@types/node@25.3.2)(yaml@2.8.1) - vue: 3.5.25(typescript@5.6.3) + vue: 3.5.41(typescript@5.6.3) - '@vitejs/plugin-vue@6.0.6(vite@8.0.8(@types/node@25.3.2)(yaml@2.8.1))(vue@3.5.25(typescript@5.9.3))': + '@vitejs/plugin-vue@6.0.6(vite@8.0.8(@types/node@25.3.2)(yaml@2.8.1))(vue@3.5.41(typescript@5.9.3))': dependencies: '@rolldown/pluginutils': 1.0.0-rc.13 vite: 8.0.8(@types/node@25.3.2)(yaml@2.8.1) - vue: 3.5.25(typescript@5.9.3) + vue: 3.5.41(typescript@5.9.3) - '@vitest/coverage-istanbul@4.1.4(vitest@4.0.18(@types/node@25.3.2)(lightningcss@1.32.0)(yaml@2.8.1))': + '@vitest/coverage-istanbul@4.1.4(vitest@4.1.10)': dependencies: '@babel/core': 7.29.0 '@istanbuljs/schema': 0.1.3 @@ -7868,111 +7416,48 @@ snapshots: magicast: 0.5.2 obug: 2.1.3 tinyrainbow: 3.1.0 - vitest: 4.0.18(@types/node@25.3.2)(lightningcss@1.32.0)(yaml@2.8.1) + vitest: 4.1.10(@types/node@25.3.2)(@vitest/coverage-istanbul@4.1.4)(vite@8.0.8(@types/node@25.3.2)(yaml@2.8.1)) transitivePeerDependencies: - supports-color - '@vitest/coverage-istanbul@4.1.4(vitest@4.1.4)': - dependencies: - '@babel/core': 7.29.0 - '@istanbuljs/schema': 0.1.3 - '@jridgewell/gen-mapping': 0.3.13 - '@jridgewell/trace-mapping': 0.3.31 - istanbul-lib-coverage: 3.2.2 - istanbul-lib-report: 3.0.1 - istanbul-reports: 3.2.0 - magicast: 0.5.2 - obug: 2.1.3 - tinyrainbow: 3.1.0 - vitest: 4.1.4(@types/node@25.3.2)(@vitest/coverage-istanbul@4.1.4)(vite@7.2.7(@types/node@25.3.2)(lightningcss@1.32.0)(yaml@2.8.1)) - transitivePeerDependencies: - - supports-color - - '@vitest/expect@4.0.18': - dependencies: - '@standard-schema/spec': 1.1.0 - '@types/chai': 5.2.3 - '@vitest/spy': 4.0.18 - '@vitest/utils': 4.0.18 - chai: 6.2.2 - tinyrainbow: 3.1.0 - - '@vitest/expect@4.1.4': + '@vitest/expect@4.1.10': dependencies: '@standard-schema/spec': 1.1.0 '@types/chai': 5.2.3 - '@vitest/spy': 4.1.4 - '@vitest/utils': 4.1.4 + '@vitest/spy': 4.1.10 + '@vitest/utils': 4.1.10 chai: 6.2.2 tinyrainbow: 3.1.0 - '@vitest/mocker@4.0.18(vite@7.2.7(@types/node@25.3.2)(lightningcss@1.32.0)(yaml@2.8.1))': - dependencies: - '@vitest/spy': 4.0.18 - estree-walker: 3.0.3 - magic-string: 0.30.21 - optionalDependencies: - vite: 7.2.7(@types/node@25.3.2)(lightningcss@1.32.0)(yaml@2.8.1) - - '@vitest/mocker@4.1.4(vite@7.2.7(@types/node@25.3.2)(lightningcss@1.32.0)(yaml@2.8.1))': - dependencies: - '@vitest/spy': 4.1.4 - estree-walker: 3.0.3 - magic-string: 0.30.21 - optionalDependencies: - vite: 7.2.7(@types/node@25.3.2)(lightningcss@1.32.0)(yaml@2.8.1) - - '@vitest/mocker@4.1.4(vite@8.0.8(@types/node@25.3.2)(yaml@2.8.1))': + '@vitest/mocker@4.1.10(vite@8.0.8(@types/node@25.3.2)(yaml@2.8.1))': dependencies: - '@vitest/spy': 4.1.4 + '@vitest/spy': 4.1.10 estree-walker: 3.0.3 magic-string: 0.30.21 optionalDependencies: vite: 8.0.8(@types/node@25.3.2)(yaml@2.8.1) - '@vitest/pretty-format@4.0.18': + '@vitest/pretty-format@4.1.10': dependencies: tinyrainbow: 3.1.0 - '@vitest/pretty-format@4.1.4': - dependencies: - tinyrainbow: 3.1.0 - - '@vitest/runner@4.0.18': - dependencies: - '@vitest/utils': 4.0.18 - pathe: 2.0.3 - - '@vitest/runner@4.1.4': + '@vitest/runner@4.1.10': dependencies: - '@vitest/utils': 4.1.4 + '@vitest/utils': 4.1.10 pathe: 2.0.3 - '@vitest/snapshot@4.0.18': + '@vitest/snapshot@4.1.10': dependencies: - '@vitest/pretty-format': 4.0.18 + '@vitest/pretty-format': 4.1.10 + '@vitest/utils': 4.1.10 magic-string: 0.30.21 pathe: 2.0.3 - '@vitest/snapshot@4.1.4': - dependencies: - '@vitest/pretty-format': 4.1.4 - '@vitest/utils': 4.1.4 - magic-string: 0.30.21 - pathe: 2.0.3 - - '@vitest/spy@4.0.18': {} - - '@vitest/spy@4.1.4': {} - - '@vitest/utils@4.0.18': - dependencies: - '@vitest/pretty-format': 4.0.18 - tinyrainbow: 3.1.0 + '@vitest/spy@4.1.10': {} - '@vitest/utils@4.1.4': + '@vitest/utils@4.1.10': dependencies: - '@vitest/pretty-format': 4.1.4 + '@vitest/pretty-format': 4.1.10 convert-source-map: 2.0.0 tinyrainbow: 3.1.0 @@ -7990,17 +7475,17 @@ snapshots: '@vue/compiler-core@3.5.24': dependencies: - '@babel/parser': 7.29.0 + '@babel/parser': 7.29.8 '@vue/shared': 3.5.24 entities: 4.5.0 estree-walker: 2.0.2 source-map-js: 1.2.1 - '@vue/compiler-core@3.5.25': + '@vue/compiler-core@3.5.41': dependencies: - '@babel/parser': 7.29.0 - '@vue/shared': 3.5.25 - entities: 4.5.0 + '@babel/parser': 7.29.8 + '@vue/shared': 3.5.41 + entities: 7.0.1 estree-walker: 2.0.2 source-map-js: 1.2.1 @@ -8009,33 +7494,33 @@ snapshots: '@vue/compiler-core': 3.5.24 '@vue/shared': 3.5.24 - '@vue/compiler-dom@3.5.25': + '@vue/compiler-dom@3.5.41': dependencies: - '@vue/compiler-core': 3.5.25 - '@vue/shared': 3.5.25 + '@vue/compiler-core': 3.5.41 + '@vue/shared': 3.5.41 '@vue/compiler-sfc@3.5.24': dependencies: - '@babel/parser': 7.29.0 + '@babel/parser': 7.29.8 '@vue/compiler-core': 3.5.24 '@vue/compiler-dom': 3.5.24 '@vue/compiler-ssr': 3.5.24 '@vue/shared': 3.5.24 estree-walker: 2.0.2 magic-string: 0.30.21 - postcss: 8.5.10 + postcss: 8.5.26 source-map-js: 1.2.1 - '@vue/compiler-sfc@3.5.25': + '@vue/compiler-sfc@3.5.41': dependencies: - '@babel/parser': 7.29.0 - '@vue/compiler-core': 3.5.25 - '@vue/compiler-dom': 3.5.25 - '@vue/compiler-ssr': 3.5.25 - '@vue/shared': 3.5.25 + '@babel/parser': 7.29.8 + '@vue/compiler-core': 3.5.41 + '@vue/compiler-dom': 3.5.41 + '@vue/compiler-ssr': 3.5.41 + '@vue/shared': 3.5.41 estree-walker: 2.0.2 magic-string: 0.30.21 - postcss: 8.5.10 + postcss: 8.5.26 source-map-js: 1.2.1 '@vue/compiler-ssr@3.5.24': @@ -8043,10 +7528,10 @@ snapshots: '@vue/compiler-dom': 3.5.24 '@vue/shared': 3.5.24 - '@vue/compiler-ssr@3.5.25': + '@vue/compiler-ssr@3.5.41': dependencies: - '@vue/compiler-dom': 3.5.25 - '@vue/shared': 3.5.25 + '@vue/compiler-dom': 3.5.41 + '@vue/shared': 3.5.41 '@vue/compiler-vue2@2.7.16': dependencies: @@ -8056,9 +7541,9 @@ snapshots: '@vue/language-core@2.2.0(typescript@5.9.3)': dependencies: '@volar/language-core': 2.4.28 - '@vue/compiler-dom': 3.5.25 + '@vue/compiler-dom': 3.5.41 '@vue/compiler-vue2': 2.7.16 - '@vue/shared': 3.5.25 + '@vue/shared': 3.5.41 alien-signals: 0.4.14 minimatch: 9.0.5 muggle-string: 0.4.1 @@ -8069,8 +7554,8 @@ snapshots: '@vue/language-core@3.3.5': dependencies: '@volar/language-core': 2.4.28 - '@vue/compiler-dom': 3.5.25 - '@vue/shared': 3.5.25 + '@vue/compiler-dom': 3.5.41 + '@vue/shared': 3.5.41 alien-signals: 3.2.1 muggle-string: 0.4.1 path-browserify: 1.0.1 @@ -8080,33 +7565,33 @@ snapshots: dependencies: '@vue/shared': 3.5.24 - '@vue/reactivity@3.5.25': + '@vue/reactivity@3.5.41': dependencies: - '@vue/shared': 3.5.25 + '@vue/shared': 3.5.41 '@vue/runtime-core@3.5.24': dependencies: '@vue/reactivity': 3.5.24 '@vue/shared': 3.5.24 - '@vue/runtime-core@3.5.25': + '@vue/runtime-core@3.5.41': dependencies: - '@vue/reactivity': 3.5.25 - '@vue/shared': 3.5.25 + '@vue/reactivity': 3.5.41 + '@vue/shared': 3.5.41 '@vue/runtime-dom@3.5.24': dependencies: '@vue/reactivity': 3.5.24 '@vue/runtime-core': 3.5.24 '@vue/shared': 3.5.24 - csstype: 3.1.3 + csstype: 3.2.3 - '@vue/runtime-dom@3.5.25': + '@vue/runtime-dom@3.5.41': dependencies: - '@vue/reactivity': 3.5.25 - '@vue/runtime-core': 3.5.25 - '@vue/shared': 3.5.25 - csstype: 3.1.3 + '@vue/reactivity': 3.5.41 + '@vue/runtime-core': 3.5.41 + '@vue/shared': 3.5.41 + csstype: 3.2.3 '@vue/server-renderer@3.5.24(vue@3.5.24(typescript@5.6.3))': dependencies: @@ -8114,21 +7599,15 @@ snapshots: '@vue/shared': 3.5.24 vue: 3.5.24(typescript@5.6.3) - '@vue/server-renderer@3.5.25(vue@3.5.25(typescript@5.6.3))': + '@vue/server-renderer@3.5.41': dependencies: - '@vue/compiler-ssr': 3.5.25 - '@vue/shared': 3.5.25 - vue: 3.5.25(typescript@5.6.3) - - '@vue/server-renderer@3.5.25(vue@3.5.25(typescript@5.9.3))': - dependencies: - '@vue/compiler-ssr': 3.5.25 - '@vue/shared': 3.5.25 - vue: 3.5.25(typescript@5.9.3) + '@vue/compiler-ssr': 3.5.41 + '@vue/runtime-dom': 3.5.41 + '@vue/shared': 3.5.41 '@vue/shared@3.5.24': {} - '@vue/shared@3.5.25': {} + '@vue/shared@3.5.41': {} '@vue/test-utils@2.4.6': dependencies: @@ -8524,7 +8003,7 @@ snapshots: shebang-command: 2.0.0 which: 2.0.2 - csstype@3.1.3: {} + csstype@3.2.3: {} d3-array@3.2.4: dependencies: @@ -8650,41 +8129,14 @@ snapshots: entities@4.5.0: {} + entities@7.0.1: {} + es-module-lexer@1.7.0: {} es-module-lexer@2.0.0: {} es-toolkit@1.42.0: {} - esbuild@0.25.12: - optionalDependencies: - '@esbuild/aix-ppc64': 0.25.12 - '@esbuild/android-arm': 0.25.12 - '@esbuild/android-arm64': 0.25.12 - '@esbuild/android-x64': 0.25.12 - '@esbuild/darwin-arm64': 0.25.12 - '@esbuild/darwin-x64': 0.25.12 - '@esbuild/freebsd-arm64': 0.25.12 - '@esbuild/freebsd-x64': 0.25.12 - '@esbuild/linux-arm': 0.25.12 - '@esbuild/linux-arm64': 0.25.12 - '@esbuild/linux-ia32': 0.25.12 - '@esbuild/linux-loong64': 0.25.12 - '@esbuild/linux-mips64el': 0.25.12 - '@esbuild/linux-ppc64': 0.25.12 - '@esbuild/linux-riscv64': 0.25.12 - '@esbuild/linux-s390x': 0.25.12 - '@esbuild/linux-x64': 0.25.12 - '@esbuild/netbsd-arm64': 0.25.12 - '@esbuild/netbsd-x64': 0.25.12 - '@esbuild/openbsd-arm64': 0.25.12 - '@esbuild/openbsd-x64': 0.25.12 - '@esbuild/openharmony-arm64': 0.25.12 - '@esbuild/sunos-x64': 0.25.12 - '@esbuild/win32-arm64': 0.25.12 - '@esbuild/win32-ia32': 0.25.12 - '@esbuild/win32-x64': 0.25.12 - escalade@3.2.0: {} esprima@4.0.1: {} @@ -8745,10 +8197,6 @@ snapshots: dependencies: pend: 1.2.0 - fdir@6.5.0(picomatch@4.0.3): - optionalDependencies: - picomatch: 4.0.3 - fdir@6.5.0(picomatch@4.0.4): optionalDependencies: picomatch: 4.0.4 @@ -9191,6 +8639,8 @@ snapshots: nanoid@3.3.11: {} + nanoid@3.3.18: {} + napi-build-utils@2.0.0: {} nice-try@1.0.5: {} @@ -9221,8 +8671,6 @@ snapshots: object-assign@4.1.1: {} - obug@2.1.1: {} - obug@2.1.3: {} once@1.4.0: @@ -9360,8 +8808,6 @@ snapshots: picomatch@2.3.1: {} - picomatch@4.0.3: {} - picomatch@4.0.4: {} pify@2.3.0: {} @@ -9403,6 +8849,12 @@ snapshots: picocolors: 1.1.1 source-map-js: 1.2.1 + postcss@8.5.26: + dependencies: + nanoid: 3.3.18 + picocolors: 1.1.1 + source-map-js: 1.2.1 + prebuild-install@7.1.3: dependencies: detect-libc: 2.0.3 @@ -9426,21 +8878,6 @@ snapshots: proto-list@1.2.4: {} - protobufjs@7.4.0: - dependencies: - '@protobufjs/aspromise': 1.1.2 - '@protobufjs/base64': 1.1.2 - '@protobufjs/codegen': 2.0.4 - '@protobufjs/eventemitter': 1.1.0 - '@protobufjs/fetch': 1.1.0 - '@protobufjs/float': 1.0.2 - '@protobufjs/inquire': 1.1.0 - '@protobufjs/path': 1.1.2 - '@protobufjs/pool': 1.1.0 - '@protobufjs/utf8': 1.1.0 - '@types/node': 25.3.2 - long: 5.3.2 - protobufjs@7.6.5: dependencies: '@protobufjs/aspromise': 1.1.2 @@ -9472,17 +8909,17 @@ snapshots: queue-tick@1.0.1: {} - quickjs-emscripten-core@0.31.0: + quickjs-emscripten-core@0.32.0: dependencies: - '@jitl/quickjs-ffi-types': 0.31.0 + '@jitl/quickjs-ffi-types': 0.32.0 - quickjs-emscripten@0.31.0: + quickjs-emscripten@0.32.0: dependencies: - '@jitl/quickjs-wasmfile-debug-asyncify': 0.31.0 - '@jitl/quickjs-wasmfile-debug-sync': 0.31.0 - '@jitl/quickjs-wasmfile-release-asyncify': 0.31.0 - '@jitl/quickjs-wasmfile-release-sync': 0.31.0 - quickjs-emscripten-core: 0.31.0 + '@jitl/quickjs-wasmfile-debug-asyncify': 0.32.0 + '@jitl/quickjs-wasmfile-debug-sync': 0.32.0 + '@jitl/quickjs-wasmfile-release-asyncify': 0.32.0 + '@jitl/quickjs-wasmfile-release-sync': 0.32.0 + quickjs-emscripten-core: 0.32.0 rc@1.2.8: dependencies: @@ -9760,8 +9197,6 @@ snapshots: stackback@0.0.2: {} - std-env@3.10.0: {} - std-env@4.1.0: {} stream-browserify@3.0.0: @@ -9893,8 +9328,8 @@ snapshots: tinyglobby@0.2.15: dependencies: - fdir: 6.5.0(picomatch@4.0.3) - picomatch: 4.0.3 + fdir: 6.5.0(picomatch@4.0.4) + picomatch: 4.0.4 tinypool@2.1.0: {} @@ -9922,32 +9357,14 @@ snapshots: dependencies: safe-buffer: 5.2.1 - turbo-darwin-64@2.8.11: - optional: true - - turbo-darwin-arm64@2.8.11: - optional: true - - turbo-linux-64@2.8.11: - optional: true - - turbo-linux-arm64@2.8.11: - optional: true - - turbo-windows-64@2.8.11: - optional: true - - turbo-windows-arm64@2.8.11: - optional: true - - turbo@2.8.11: + turbo@2.10.10: optionalDependencies: - turbo-darwin-64: 2.8.11 - turbo-darwin-arm64: 2.8.11 - turbo-linux-64: 2.8.11 - turbo-linux-arm64: 2.8.11 - turbo-windows-64: 2.8.11 - turbo-windows-arm64: 2.8.11 + '@turbo/darwin-64': 2.10.10 + '@turbo/darwin-arm64': 2.10.10 + '@turbo/linux-64': 2.10.10 + '@turbo/linux-arm64': 2.10.10 + '@turbo/windows-64': 2.10.10 + '@turbo/windows-arm64': 2.10.10 tweetnacl@0.14.5: {} @@ -10037,20 +9454,6 @@ snapshots: picocolors: 1.1.1 vite: 8.0.8(@types/node@25.3.2)(yaml@2.8.1) - vite@7.2.7(@types/node@25.3.2)(lightningcss@1.32.0)(yaml@2.8.1): - dependencies: - esbuild: 0.25.12 - fdir: 6.5.0(picomatch@4.0.4) - picomatch: 4.0.4 - postcss: 8.5.10 - rollup: 4.53.3 - tinyglobby: 0.2.15 - optionalDependencies: - '@types/node': 25.3.2 - fsevents: 2.3.3 - lightningcss: 1.32.0 - yaml: 2.8.1 - vite@8.0.8(@types/node@25.3.2)(yaml@2.8.1): dependencies: lightningcss: 1.32.0 @@ -10063,80 +9466,15 @@ snapshots: fsevents: 2.3.3 yaml: 2.8.1 - vitest@4.0.18(@types/node@25.3.2)(lightningcss@1.32.0)(yaml@2.8.1): - dependencies: - '@vitest/expect': 4.0.18 - '@vitest/mocker': 4.0.18(vite@7.2.7(@types/node@25.3.2)(lightningcss@1.32.0)(yaml@2.8.1)) - '@vitest/pretty-format': 4.0.18 - '@vitest/runner': 4.0.18 - '@vitest/snapshot': 4.0.18 - '@vitest/spy': 4.0.18 - '@vitest/utils': 4.0.18 - es-module-lexer: 1.7.0 - expect-type: 1.3.0 - magic-string: 0.30.21 - obug: 2.1.1 - pathe: 2.0.3 - picomatch: 4.0.4 - std-env: 3.10.0 - tinybench: 2.9.0 - tinyexec: 1.0.2 - tinyglobby: 0.2.15 - tinyrainbow: 3.1.0 - vite: 7.2.7(@types/node@25.3.2)(lightningcss@1.32.0)(yaml@2.8.1) - why-is-node-running: 2.3.0 - optionalDependencies: - '@types/node': 25.3.2 - transitivePeerDependencies: - - jiti - - less - - lightningcss - - msw - - sass - - sass-embedded - - stylus - - sugarss - - terser - - tsx - - yaml - - vitest@4.1.4(@types/node@25.3.2)(@vitest/coverage-istanbul@4.1.4)(vite@7.2.7(@types/node@25.3.2)(lightningcss@1.32.0)(yaml@2.8.1)): - dependencies: - '@vitest/expect': 4.1.4 - '@vitest/mocker': 4.1.4(vite@7.2.7(@types/node@25.3.2)(lightningcss@1.32.0)(yaml@2.8.1)) - '@vitest/pretty-format': 4.1.4 - '@vitest/runner': 4.1.4 - '@vitest/snapshot': 4.1.4 - '@vitest/spy': 4.1.4 - '@vitest/utils': 4.1.4 - es-module-lexer: 2.0.0 - expect-type: 1.3.0 - magic-string: 0.30.21 - obug: 2.1.3 - pathe: 2.0.3 - picomatch: 4.0.4 - std-env: 4.1.0 - tinybench: 2.9.0 - tinyexec: 1.0.2 - tinyglobby: 0.2.15 - tinyrainbow: 3.1.0 - vite: 7.2.7(@types/node@25.3.2)(lightningcss@1.32.0)(yaml@2.8.1) - why-is-node-running: 2.3.0 - optionalDependencies: - '@types/node': 25.3.2 - '@vitest/coverage-istanbul': 4.1.4(vitest@4.0.18(@types/node@25.3.2)(lightningcss@1.32.0)(yaml@2.8.1)) - transitivePeerDependencies: - - msw - - vitest@4.1.4(@types/node@25.3.2)(@vitest/coverage-istanbul@4.1.4)(vite@8.0.8(@types/node@25.3.2)(yaml@2.8.1)): + vitest@4.1.10(@types/node@25.3.2)(@vitest/coverage-istanbul@4.1.4)(vite@8.0.8(@types/node@25.3.2)(yaml@2.8.1)): dependencies: - '@vitest/expect': 4.1.4 - '@vitest/mocker': 4.1.4(vite@8.0.8(@types/node@25.3.2)(yaml@2.8.1)) - '@vitest/pretty-format': 4.1.4 - '@vitest/runner': 4.1.4 - '@vitest/snapshot': 4.1.4 - '@vitest/spy': 4.1.4 - '@vitest/utils': 4.1.4 + '@vitest/expect': 4.1.10 + '@vitest/mocker': 4.1.10(vite@8.0.8(@types/node@25.3.2)(yaml@2.8.1)) + '@vitest/pretty-format': 4.1.10 + '@vitest/runner': 4.1.10 + '@vitest/snapshot': 4.1.10 + '@vitest/spy': 4.1.10 + '@vitest/utils': 4.1.10 es-module-lexer: 2.0.0 expect-type: 1.3.0 magic-string: 0.30.21 @@ -10152,7 +9490,7 @@ snapshots: why-is-node-running: 2.3.0 optionalDependencies: '@types/node': 25.3.2 - '@vitest/coverage-istanbul': 4.1.4(vitest@4.1.4) + '@vitest/coverage-istanbul': 4.1.4(vitest@4.1.10) transitivePeerDependencies: - msw @@ -10176,23 +9514,23 @@ snapshots: optionalDependencies: typescript: 5.6.3 - vue@3.5.25(typescript@5.6.3): + vue@3.5.41(typescript@5.6.3): dependencies: - '@vue/compiler-dom': 3.5.25 - '@vue/compiler-sfc': 3.5.25 - '@vue/runtime-dom': 3.5.25 - '@vue/server-renderer': 3.5.25(vue@3.5.25(typescript@5.6.3)) - '@vue/shared': 3.5.25 + '@vue/compiler-dom': 3.5.41 + '@vue/compiler-sfc': 3.5.41 + '@vue/runtime-dom': 3.5.41 + '@vue/server-renderer': 3.5.41 + '@vue/shared': 3.5.41 optionalDependencies: typescript: 5.6.3 - vue@3.5.25(typescript@5.9.3): + vue@3.5.41(typescript@5.9.3): dependencies: - '@vue/compiler-dom': 3.5.25 - '@vue/compiler-sfc': 3.5.25 - '@vue/runtime-dom': 3.5.25 - '@vue/server-renderer': 3.5.25(vue@3.5.25(typescript@5.9.3)) - '@vue/shared': 3.5.25 + '@vue/compiler-dom': 3.5.41 + '@vue/compiler-sfc': 3.5.41 + '@vue/runtime-dom': 3.5.41 + '@vue/server-renderer': 3.5.41 + '@vue/shared': 3.5.41 optionalDependencies: typescript: 5.9.3 diff --git a/pnpm-workspace.yaml b/pnpm-workspace.yaml index bce81ef..2054c86 100644 --- a/pnpm-workspace.yaml +++ b/pnpm-workspace.yaml @@ -8,19 +8,19 @@ packages: catalog: "@milaboratories/helpers": 1.14.5 - "@milaboratories/ts-builder": 1.6.1 - "@milaboratories/ts-configs": 1.3.1 + "@milaboratories/ts-builder": 1.6.2 + "@milaboratories/ts-configs": 1.4.0 typescript: ~5.9.3 "@platforma-sdk/workflow-tengo": 6.8.2 "@platforma-sdk/block-tools": 2.13.0 - "@platforma-sdk/model": 1.80.8 - "@platforma-sdk/ui-vue": 1.80.9 - "@platforma-sdk/test": 1.80.9 + "@platforma-sdk/model": 1.81.1 + "@platforma-sdk/ui-vue": 1.81.1 + "@platforma-sdk/test": 1.81.3 "@platforma-sdk/tengo-builder": 4.0.22 - "@platforma-sdk/package-builder": 3.14.2 + "@platforma-sdk/package-builder": 3.15.0 "@platforma-sdk/blocks-deps-updater": 2.2.0 - "@platforma-open/milaboratories.runenv-python-3": 1.7.8 + "@platforma-open/milaboratories.runenv-python-3": 1.11.6 "@platforma-open/milaboratories.software-mitool": 2.3.1-131-main # blocks used in integration tests. SND pinned EXACT to the last V1 (modelAPIVersion 1) release: @@ -29,11 +29,11 @@ catalog: "@platforma-open/milaboratories.samples-and-data": 1.13.3 "@platforma-open/milaboratories.samples-and-data.model": 1.11.2 - "turbo": 2.8.11 + "turbo": 2.10.10 "shx": 0.4.0 "@changesets/cli": 2.29.8 vue: 3.5.24 "ag-grid-enterprise": &ag-grid ~34.1.2 "ag-grid-vue3": *ag-grid - vitest: ~4.0.18 + vitest: ~4.1.10 From b3c5e716b73c3a70e9b29c3d3eaf1b0ba1db6518 Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Tue, 18 Aug 2026 09:25:37 +0200 Subject: [PATCH 065/282] MILAB-6496: keep the per-cell tables inside the block, per the spec update text commit 11594aac ("one grouping per block execution") inverted a requirement this block was built against. The interface atom previously obliged a reader to re-derive verdicts under another grouping WITHOUT re-running the block, which is why the per-cell tables were exports. It now states the opposite: the per-cell, per-tag states stay inside the block, and no consumer of the verdicts reads them. The atom's own rationale gives the reason, and it matches what is here: the old obligation contradicted the block-set atom, where regrouping is re-running this block, and it exported the largest artifact the block produces to a consumer that does not exist. Labelling and lead selection read verdicts, never cells. Checked before changing anything -- nothing in the model, the UI or the sibling block read either table. cellCounts and cellScalars move from the export frame to a block-local output, antigenCellTable. The per-cell reference readings keep their place among the run's own measurements, which is where a reader checks why a cell could or could not be compared. Three comments justifying them as exports are rewritten rather than left to mislead. The update also adds a requirement this block already met: the grouping enters after the counting, so a second execution over unchanged reads and an unchanged panel file re-does the verdict step alone. No reading parameter is in the fan-out allowlist and a tengo test enforces it -- written to keep the mitool fan-out cached, which turns out to be the same constraint. Adding a reading parameter to that allowlist would now break a spec requirement, not just a cache. --- software/per-cell-metrics/src/emit_verdicts.py | 11 ++++++----- workflow/src/column-specs.lib.tengo | 12 ++++++++---- workflow/src/main.tpl.tengo | 11 ++++++++++- workflow/src/verdict-import.tpl.tengo | 17 ++++++++++++----- 4 files changed, 36 insertions(+), 15 deletions(-) diff --git a/software/per-cell-metrics/src/emit_verdicts.py b/software/per-cell-metrics/src/emit_verdicts.py index 59efaa7..c0c9b57 100644 --- a/software/per-cell-metrics/src/emit_verdicts.py +++ b/software/per-cell-metrics/src/emit_verdicts.py @@ -624,11 +624,12 @@ def main() -> None: summary, summary_emitted = _pivot_identity_summary(verdicts, universe) _write_sorted(summary, f"{prefix}_identity_summary.csv", ["setId"]) - # The re-derivation material: the sparse per-tag counts and the per-cell - # scalars together reproduce every per-cell state exactly, at a small - # fraction of the size a per-cell-per-identity table would take. A reader - # regrouping the panel re-takes the highest member, re-scores against the - # same reference, and re-votes, without a re-run. + # The sparse per-tag counts and the per-cell scalars together carry every + # per-cell state, at a small fraction of the size a per-cell-per-identity + # table would take. They stay inside the block: reading the same experiment + # under another grouping is another execution rather than a re-derivation a + # reader performs, and the grouping enters after the counting, so a second + # execution over unchanged inputs re-does the verdict step alone. # With no list, membership is unknown rather than false: a barcode nobody # classified is not a barcode classified as "not a cell". "false" would be # a claim the run cannot support. diff --git a/workflow/src/column-specs.lib.tengo b/workflow/src/column-specs.lib.tengo index feb3c2b..9362772 100644 --- a/workflow/src/column-specs.lib.tengo +++ b/workflow/src/column-specs.lib.tengo @@ -698,10 +698,14 @@ identitySummaryImportSpec := func(setAxisSpec, identities, groupingId, served) { // // The block emits no dense per-cell-per-identity table: on a realistic run it is the largest artifact // the block would produce, and a pMHC panel does not fit at all. The sparse per-tag counts plus the -// per-cell scalars reproduce every per-cell state exactly at a small fraction of the size — a reader -// regrouping the panel re-takes the highest member, re-scores against the same reference and re-votes -// without a re-run. Both are EXPORTS: outputs are visible only to this block's own model, so -// re-derivation material returned as an output reaches nobody. +// per-cell scalars carry every per-cell state at a small fraction of that size. +// +// Both stay INSIDE the block, as outputs rather than exports. Reading the same experiment under another +// grouping is another execution of this block, not a re-derivation a reader performs, so no consumer +// across the boundary wants them — labelling and lead selection read verdicts, never cells. What makes +// re-execution cheap is where the grouping is consumed: it enters after the counting, so a second run +// over unchanged reads and an unchanged panel file reuses the cached counts and pays for the verdicts +// alone. // result_cell_counts.csv, keyed (sampleId, cellId, tag). // The CSV also repeats referenceCount and inCellList on every tag row. They are per-CELL facts and diff --git a/workflow/src/main.tpl.tengo b/workflow/src/main.tpl.tengo index 3951427..bbc4548 100644 --- a/workflow/src/main.tpl.tengo +++ b/workflow/src/main.tpl.tengo @@ -502,7 +502,12 @@ wf.body(func(args) { if !is_undefined(verdictImport) { // Everything a downstream block joins to: the verdicts, the set-keyed counts that are the only // family lead selection can see, the pivoted per-identity summary, the offered scope, the - // re-derivation material, the tag -> identity linker and the label columns. + // tag -> identity linker and the label columns. + // + // The per-cell tables are deliberately NOT among them. Reading the same experiment under another + // grouping is another execution of this block rather than a re-derivation a reader performs, so + // the per-cell per-tag states have no consumer across the boundary: labelling and lead selection + // both read verdicts, never cells. blockExports.antigenVerdicts = verdictImport.output("antigenVerdicts") // The same frame as an OUTPUT, because a block's own exports are not in its own result pool: without // this the block that produced the verdicts is the one place that cannot show them. The model reads @@ -511,6 +516,10 @@ wf.body(func(args) { // The run's own report. Outputs rather than exports: these are read by this block's model and UI. blockOutputs.antigenQcTable = pframes.exportFrame(verdictImport.output("qcTable")) blockOutputs.antigenPanelMismatchTable = pframes.exportFrame(verdictImport.output("panelMismatchTable")) + // The per-cell counts and reference readings, kept inside the block. The reference readings are part + // of what the run reports about itself -- they are where a reader checks why a cell could or could + // not be compared -- and the per-tag counts sit beside them at the grain they were read at. + blockOutputs.antigenCellTable = pframes.exportFrame(verdictImport.output("cellTable")) // What the run was answered under, including the comparator and cell list that actually served and // every parameter the reading used. Read as content by the model for the run summary. blockOutputs.antigenRunMeta = verdictRun.output("runMeta") diff --git a/workflow/src/verdict-import.tpl.tengo b/workflow/src/verdict-import.tpl.tengo index c151a1c..a532715 100644 --- a/workflow/src/verdict-import.tpl.tengo +++ b/workflow/src/verdict-import.tpl.tengo @@ -104,12 +104,18 @@ self.body(func(inputs) { columnSpecs.identitySummaryImportSpec(setAxis, summaryIdentities, served.groupingId, served)) } - // The re-derivation material. Both are EXPORTS rather than outputs: an output is visible only to this - // block's own model, so material a reader needs in order to regroup the panel without a re-run would - // reach nobody. - addTo(exportFb, "cellCounts", inputs.cellCounts, + // The per-cell material stays INSIDE the block: outputs, never exports. Reading the same experiment + // under another grouping is another execution of this block, not a re-derivation performed by a + // reader, so nothing downstream needs the per-cell per-tag states — and nothing downstream ever asked + // for them. Labelling and lead selection read verdicts, never cells. Exporting the largest artifact + // here to a consumer that does not exist is the cost this avoids. + // + // The per-cell reference readings keep their place among the run's own measurements, which is where + // a reader checks why a cell could or could not be compared. + cellFb := pframes.pFrameBuilder() + addTo(cellFb, "cellCounts", inputs.cellCounts, columnSpecs.cellTagCountsImportSpec(sampleAxis, cellAxis, tagAxis)) - addTo(exportFb, "cellScalars", inputs.cellScalars, + addTo(cellFb, "cellScalars", inputs.cellScalars, columnSpecs.cellScalarsImportSpec(sampleAxis, cellAxis, served)) // Which identities each sample was actually stained with — without it "never asked" is a claim a @@ -151,6 +157,7 @@ self.body(func(inputs) { return { antigenVerdicts: exportFb.build(), + cellTable: cellFb.build(), qcTable: qcFb.build(), panelMismatchTable: mismatchFb.build() } From f077bbd11f29d1ade87fdddbd0a027afa0caa289 Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Tue, 18 Aug 2026 10:09:52 +0200 Subject: [PATCH 066/282] MILAB-6496: retire the dominance assertions from the test suites The block test asserted a consensusFeature column that no longer exists, and the generator suite called consensus_category, which was removed with it. test/src/wf.test.ts - drop the dominant-feature assertion and invert it: a per-cell column calling one antigen is now a failure, since reintroducing it beside a verdict would give a reader two disagreeing answers. guardNoScore refuses the score annotation at build time; this checks what reached a table. - assert maxFraction instead, the surviving per-cell magnitude. - empty inputs now checks that every output a page reads resolves on a freshly added block. Five table outputs and a run record were added on this branch, each with its own undefined-until-computed guard, and an output that throws there breaks the page at block creation. - state at the top what this file does not cover: the verdict half needs a single-cell V(D)J dataset upstream that this chain cannot supply. - do not assert what a datasetless run emits beyond the per-cell table. Today the antigen stage is skipped entirely; the qc-measurement set requires the read-and-panel measurements to survive a run with no cell list. That gap is open, and encoding today's behaviour would have to be deleted to close it. software/test-data/manual/tests/test_panel.py - replace the consensus_category test with one that asserts the planted shape against the generator's own output: a cross-reactive cell carries a co-dominant pair of two on-target antigens. Verified by mutation - an uneven pair and a pair drawn from all antigens are both caught. - the evenness check passes vacuously on a bed where every cell is even, so it also asserts some ordinary binder is visibly uneven. - move the tags.csv-only tests onto the antigen-only scenario path. A full run builds the gex arm, which needs a downloaded gene-annotations asset, so it cannot run in a clean checkout. The four tests that genuinely need a full run now skip with the fetch command instead of failing with a bare subprocess error. - the out-of-range test asserted only a nonzero exit, which the missing asset satisfied just as well as the rejection it meant to check. It now names the rejection. --- software/test-data/manual/tests/test_panel.py | 102 ++++++++++++++---- test/src/wf.test.ts | 81 +++++++++++--- 2 files changed, 150 insertions(+), 33 deletions(-) diff --git a/software/test-data/manual/tests/test_panel.py b/software/test-data/manual/tests/test_panel.py index e778e86..eefefa4 100644 --- a/software/test-data/manual/tests/test_panel.py +++ b/software/test-data/manual/tests/test_panel.py @@ -4,13 +4,29 @@ counts must be rejected on BOTH the full-run and the --beam paths.""" import csv -import importlib.util import subprocess import sys from pathlib import Path +import pytest + HERE = Path(__file__).resolve().parent.parent # software/test-data/manual +# The gex arm annotates against a human gene-annotations table that is downloaded, not committed, so a +# full multiomic run cannot be built in a clean checkout. Tests needing only the antigen arm use the +# scenario path instead and are unaffected; the ones that genuinely need a full run say so rather than +# failing with a bare subprocess error that names no cause. +GENE_ANNOTATIONS = HERE / "assets" / "homo_sapiens_gene_annotations.csv" +needs_full_run = pytest.mark.skipif( + not GENE_ANNOTATIONS.exists(), + reason=( + f"missing {GENE_ANNOTATIONS.name}; fetch it with:\n" + " curl -sSL -o /tmp/hs.zip https://bin.pl-open.science/assets/platforma-open/" + "milaboratories.gene-annotations.homo-sapiens/main/1.1.0.zip" + f" && unzip -o /tmp/hs.zip -d {GENE_ANNOTATIONS.parent}" + ), +) + def _run(*args, out): subprocess.run( @@ -28,7 +44,9 @@ def _read_csv(path): def test_panel_has_type_species_class(tmp_path): - _run("tiny", "--offtarget-count", "2", out=tmp_path) + # tags.csv comes from the antigen arm, so the self-contained scenario bed suffices — no full run, + # no gene-annotations asset. + _run("--scenario", "errors", "--offtarget-count", "2", out=tmp_path) header, rows = _read_csv(tmp_path / "tags.csv") assert {"Type", "Species", "Class"} <= set(header) types = {r["Type"] for r in rows} @@ -37,6 +55,7 @@ def test_panel_has_type_species_class(tmp_path): assert {"Human", "Cyno"} <= {r["Species"] for r in rows} +@needs_full_run def test_multibarcode_combine_column(tmp_path): _run("tiny", "--multibarcode", out=tmp_path) with (tmp_path / "tags.csv").open() as fh: @@ -51,6 +70,7 @@ def test_multibarcode_combine_column(tmp_path): assert {"all", "sum"} <= {r["combine"] for r in rows} +@needs_full_run def test_messy_metadata_variants(tmp_path): _run("tiny", "--offtarget-count", "3", "--messy-metadata", out=tmp_path) with (tmp_path / "tags.csv").open() as fh: @@ -69,29 +89,65 @@ def test_beam_panel_has_type_species(tmp_path): assert "Target" in {r["Type"] for r in rows} -def _load_consensus(): - p = HERE.parent.parent / "per-cell-metrics" / "src" / "per_cell_metrics.py" - spec = importlib.util.spec_from_file_location("pcm", p) - mod = importlib.util.module_from_spec(spec) - spec.loader.exec_module(mod) - return mod +def _read_tsv(path): + with path.open() as fh: + return list(csv.DictReader(fh, delimiter="\t")) -def test_crossreactive_two_even_antigens(): - pcm = _load_consensus() - counts = {"AgA": 50.0, "AgB": 48.0, "ctrl": 3.0} - result = pcm.consensus_category( - counts, threshold=0.6, control="ctrl", offtargets=frozenset(), label_crossreactive=True - ) - assert result == "Target cross-reactive" +def _top_two_ratio(counts): + """Second-largest count over the largest, or None where the cell has fewer than two features.""" + ranked = sorted(counts, reverse=True) + return None if len(ranked) < 2 else ranked[1] / ranked[0] def test_generator_plants_crossreactive(tmp_path): - _run("tiny", "--offtarget-count", "1", "--crossreactive-frac", "0.1", out=tmp_path) - consensus = list(csv.DictReader((tmp_path / "truth" / "expected-consensus.tsv").open(), delimiter="\t")) - assert any(r.get("planted_consensus") == "crossreactive" for r in consensus) - - + """A planted cross-reactive cell carries a co-dominant pair of two ON-TARGET antigens. + + This asserts the generator against its own output and reads nothing from the block. It used to + check the block's `consensus_category` instead, which no longer exists: a single dominant antigen + per cell answers a different question from the four-state verdict and was removed with it. What + the bed still owes is the planted shape, because the gex and vdj arms are both built from this + truth file. + """ + # The antigen-only scenario bed rather than a full `tiny` run: a full run also builds the gex arm, + # which needs a gene-annotations asset that is downloaded rather than committed, so a full run + # cannot be generated in a clean checkout. This test needs the antigen arm alone. The scenario bed + # is self-contained and writes its truth files flat in the output directory. + _run("--scenario", "errors", "--offtarget-count", "1", "--crossreactive-frac", "0.1", out=tmp_path) + + consensus = _read_tsv(tmp_path / "expected-consensus.tsv") + crossreactive = {(r["sample"], r["cellId"]) for r in consensus if r["planted_consensus"] == "crossreactive"} + assert crossreactive, "no cross-reactive cell was planted at --crossreactive-frac 0.1" + + _, panel_rows = _read_csv(tmp_path / "tags.csv") + on_target = {r["feature"] for r in panel_rows if r["Type"] == "Target"} + + counts_by_cell = {} + for r in _read_tsv(tmp_path / "expected-abundance.tsv"): + counts_by_cell.setdefault((r["sample"], r["cellId"]), {})[r["feature"]] = int(r["planted_distinct_umis"]) + + # The pair is planted as second = first * U(0.85, 1.0), then truncated to an int, so 0.84 is the + # floor a correctly planted cell cannot fall below. + for key in crossreactive: + counts = counts_by_cell[key] + ratio = _top_two_ratio(counts.values()) + assert ratio is not None and ratio >= 0.84, f"{key} is labelled cross-reactive but its top two counts are {ratio}" + top_two = sorted(counts, key=counts.get, reverse=True)[:2] + assert set(top_two) <= on_target, f"{key}'s co-dominant pair includes a non-Target antigen: {top_two}" + + # The check above is only worth running if it can fail, and an evenness test passes vacuously on a + # bed where every cell is even. An ordinary binder plants one dominant antigen over background, so + # some non-cross-reactive cell must be visibly UNEVEN — otherwise the assertion above proves + # nothing about the label. + uneven = [ + key + for key, counts in counts_by_cell.items() + if key not in crossreactive and (_top_two_ratio(counts.values()) or 0) < 0.5 + ] + assert uneven, "every cell in the bed is co-dominant, so the cross-reactive assertion cannot discriminate" + + +@needs_full_run def test_heavy_only_airr(tmp_path): _run("tiny", "--heavy-only", out=tmp_path) tsvs = list((tmp_path / "vdj").glob("*.tsv")) @@ -104,6 +160,7 @@ def test_heavy_only_airr(tmp_path): assert loci == {"IGH"} +@needs_full_run def test_annotation_emitter(tmp_path): _run("tiny", "--with-annotations", out=tmp_path) tsvs = list((tmp_path / "annotations").glob("*.tsv")) @@ -140,4 +197,9 @@ def test_offtarget_count_out_of_range_errors(tmp_path): capture_output=True, text=True, ) + # Asserting only a nonzero exit would pass for any failure at all — a missing asset, a syntax + # error, a bad path — so it must name the rejection it is checking for. assert result.returncode != 0, f"expected nonzero exit for {extra}, got 0" + assert "--offtarget-count must be between" in (result.stderr + result.stdout), ( + f"exited nonzero for {extra}, but not because the count was out of range:\n{result.stderr}" + ) diff --git a/test/src/wf.test.ts b/test/src/wf.test.ts index 67d9655..d8b0b5e 100644 --- a/test/src/wf.test.ts +++ b/test/src/wf.test.ts @@ -10,12 +10,23 @@ import { FeatureIntegrationBlockPointer as myBlockSpec } from "this-block"; import type { InferBlockState, PTableHandle } from "@platforma-sdk/model"; import { createPlDataTableStateV2, wrapOutputs } from "@platforma-sdk/model"; -// Level-4 integration test (plan Task 7): a live end-to-end run emitting `pl7.app/feature/umiCount`. +// Block tests for the Feature Barcode Profiling block. +// +// WHAT THIS FILE COVERS, and what it does not. The block has two halves and this file reaches only +// one of them: +// +// * the per-sample counting half — FASTQ in, per-cell UMI counts out — is exercised end to end by +// the second test below, when a backend can run it. +// * the ANTIGEN VERDICT half is not covered here at all. It needs a single-cell V(D)J dataset +// upstream to supply the clonotype sets, and the samples-and-data chain this file uses cannot +// produce one. The verdict logic is covered by the Python suite instead +// (software/per-cell-metrics/test/, 259 tests) and by the Tengo suite for the p-column specs +// (workflow/src/*.test.tengo). Neither substitutes for a live run, which is why the block is +// verified by hand against software/test-data/fixtures/verdicts/ before release. // // Upstream chain follows the proven samples-and-data FASTQ pattern (blocks/mixcr-amplicon-alignment). -// The tag->feature CSV is a direct upload (M7 resolution): set as the block arg `tagFeatureCsvHandle` -// via a local file handle; the workflow imports it with file.importFile and shares the blob across the -// per-sample bodies. +// The tag->feature CSV is a direct upload: set as the block arg `tagFeatureCsvHandle` via a local file +// handle; the workflow imports it with file.importFile and shares the blob across the per-sample bodies. // // Golden (decoded from test/assets/fb_small_R{1,2}.fastq.gz; geometry CELL 16 + UMI 10 on R1, feature // 15 on R2; tags.csv: 15xG -> AGX, 15xC -> BGX): @@ -37,9 +48,31 @@ blockTest("empty inputs", { timeout: 20000 }, async ({ rawPrj: project, expect } project.getBlockState(blockId), 15000, )) as InferBlockState; - // With no upstream FASTQ column in the pool the option list is empty (args() throws, disabling Run, - // but outputs still resolve). - expect(stableState.outputs).toMatchObject({ fastqOptions: { ok: true, value: [] } }); + // With no upstream FASTQ column in the pool the option lists are empty (args() throws, disabling + // Run, but outputs still resolve). + expect(stableState.outputs).toMatchObject({ + fastqOptions: { ok: true, value: [] }, + datasetOptions: { ok: true, value: [] }, + }); + + // Every output a page reads must RESOLVE on a freshly added block, before anything has run. An + // output that throws here is not a failed computation — it breaks the page that reads it at the + // moment the block is created, which is the first thing a user sees. This block gained five table + // outputs and a run-record output on the verdict branch, each guarded by its own + // undefined-until-computed path, so the guards are what this asserts. Values are deliberately not + // asserted: `ok` with an undefined value is the correct empty-state answer for all of them. + const mustResolve = [ + "perCellTable", + "qcSummaryTable", + "verdictTable", + "antigenQcTable", + "antigenPanelMismatchTable", + "verdictRunMeta", + "isRunning", + "started", + ] as const; + const unresolved = mustResolve.filter((name) => stableState.outputs?.[name]?.ok !== true); + expect(unresolved, "outputs that failed to resolve on an empty block").toEqual([]); }); // Level-4 end-to-end run against the published mitool (software-mitool 2.3.1-129-main, carrying the @@ -120,9 +153,16 @@ blockTest.skip( // Configure the block. update-block-data must carry EVERY BlockArgsValid field, else the backend // reports "currentArgs not set". controlFeature is optional (no negative-control marker here), and - // so is datasetRef — with no single-cell V(D)J dataset the block skips the verdict stage and still - // emits everything this test reads. The reading's numeric parameters are required and carry the - // shipped defaults, the same values a freshly created block starts with. + // so is datasetRef: this run has no single-cell V(D)J dataset, so it exercises the counting half + // alone. The reading's numeric parameters are required and carry the shipped defaults, the same + // values a freshly created block starts with. + // + // What a datasetless run emits BESIDES the per-cell table is deliberately not asserted. Today the + // whole antigen stage is skipped, so nothing antigen-related is produced; the spec's qc-measurement + // set requires the eight read-and-panel measurements and the panel mismatch report to survive a run + // with no cell list, marking the rest not-evaluated. That gap is open (decision log O-4). Asserting + // today's behaviour would have to be deleted to fix it, so this test asserts only what both + // readings agree on. await project.mutateBlockStorage(fiBlockId, { operation: "update-block-data", value: { @@ -174,9 +214,24 @@ blockTest.skip( expect(maxUmiCounts).toEqual([1, 2]); expect(maxUmiCounts.reduce((a, b) => a + b, 0)).toBe(3); - // Consensus feature per cell: cellA dominant AGX (2 of 3 = 0.67 >= 0.6), cellB single-feature AGX. - // The consensusFeature String column is thus "AGX" for both cells. + // The cell's largest per-feature share of its own total: cellA 2/3, cellB 1/1. This is the + // surviving per-cell magnitude — it says how concentrated a cell's counts were, not which antigen + // it bound, so it is not a binding level and does not fall under the no-ordering prohibition. + const fractionColumns = data.filter((c) => c.type === "Double"); + expect(fractionColumns).toHaveLength(1); + const maxFractions = [...fractionColumns[0].data].map(Number).sort((a, b) => a - b); + expect(maxFractions[0]).toBeCloseTo(2 / 3, 5); + expect(maxFractions[1]).toBeCloseTo(1, 5); + + // The dominant-feature call is gone and must not come back through this table. It answered a + // different question from the four-state verdict — one antigen per cell, chosen by a share + // threshold — and reintroducing it beside a verdict would give a reader two disagreeing answers + // with no rule for which wins. `guardNoScore` in column-specs.lib.tengo refuses the score + // annotation at build time; this is the same claim checked against what actually reached a table. const stringColumnValues = data.filter((c) => c.type === "String").map((c) => [...c.data]); - expect(stringColumnValues.some((vals) => vals.every((v) => v === "AGX"))).toBe(true); + expect( + stringColumnValues.some((vals) => vals.every((v) => v === "AGX")), + "a per-cell column is calling one dominant feature — consensusFeature has returned", + ).toBe(false); }, ); From 3928f1a0d3db7218614810aeeb72c2f41d5976de Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Tue, 18 Aug 2026 11:12:23 +0200 Subject: [PATCH 067/282] MILAB-6496: cover the shape a real panel file arrives in MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every existing bed declares a role column, so every existing bed can name a comparator. A panel file observed in the field carries three columns and no fourth: sample, barcode sequence, antigen name. Two panel-rung tests already existed, but both still passed --role-column and reached the panel rung by having no row valued "Control" — so a file with no role column at all was untested. Three tests, all mutation-verified: - a three-column panel with no --role-column runs, resolves to the panel's own readings, and produces verdicts rather than reading unreliable throughout. Caught by raising the member minimum above the bed's tag count. - a barcode the two samples name differently carries no agreed antigen name, so its label falls back to the barcode itself and a reader meets a raw 15-mer where every other row shows an antigen. Caught both by keeping the first value instead of dropping a disagreement, and by replacing the fallback with a constant. - the same bed with the renaming removed, so the fallback test is a statement about disagreement rather than about this bed's barcodes. Deliberately not asserted: that a set spanning two samples should carry one verdict for a barcode naming two different antigens. Identities are keyed by tag while the panel is keyed by tag and sample, so the two do not compose when the name varies by sample. That question is open and a test fixing today's answer would have to be deleted to settle it. --- .../test/test_emit_verdicts.py | 118 ++++++++++++++++++ 1 file changed, 118 insertions(+) diff --git a/software/per-cell-metrics/test/test_emit_verdicts.py b/software/per-cell-metrics/test/test_emit_verdicts.py index f3dc9a2..1ac45ee 100644 --- a/software/per-cell-metrics/test/test_emit_verdicts.py +++ b/software/per-cell-metrics/test/test_emit_verdicts.py @@ -919,3 +919,121 @@ def test_no_qc_row_carries_a_null_panel_key(bed): panels = set(qc["panelId"].to_list()) assert "" in panels, "sample and capture rows belong to no panel and must carry an empty key" assert any(p for p in panels), "tag and identity rows must carry a real panel id" + + +# --- the shape a real panel file arrives in ----------------------------------------------------- +# +# Every bed above declares a role column, so every bed above can name a comparator. A panel file +# observed in the field carries three columns and no fourth: the sample, the barcode sequence, and the +# antigen's name. There is no role column to point `--role-column` at, so the declared rung is not +# reachable on it at all and the panel's own readings have to serve. It also reuses a barcode between +# samples under a different antigen name, which is the tag-inventory reuse the per-sample keying of the +# panel exists for. +# +# These tests fix what that file does today. They deliberately do NOT assert that a set spanning two +# samples should carry one verdict for a barcode that names two different antigens -- that question is +# open, and a test asserting today's answer would have to be deleted to settle it. + +CUSTOMER_TAGS = [f"SEQ{i:02d}" for i in range(1, 10)] # nine, against a shipped minimum of eight + + +def _customer_bed(root, *, renamed=2, span_samples=True): + """A three-column panel: sample, sequence, antigen. No role column, no grouping column. + + `renamed` barcodes carry a different antigen name in the second sample. `span_samples` puts every + cell in one clonotype set, so the set's cells come from both panels. + """ + rows = ["Sample,Sequence,Antigen"] + for sample, offset in (("SmpA", 0), ("SmpB", 100)): + for i, tag in enumerate(CUSTOMER_TAGS): + name = f"Ag{offset + i:03d}" if i < renamed else f"Ag{i:03d}" + rows.append(f"{sample},{tag},{name}") + (root / "panel.csv").write_text("\n".join(rows) + "\n") + + # SEQ01 is strong; the rest sit at 10, above the shipped floor of 4 so nothing is floored away and + # the panel median stays a real number. A background of 3 would floor to zero, drag the median to + # zero, and make every identity unreliable for a reason unrelated to the comparator. + counts = ["sampleId,cellId,tag,umiCount"] + linker = ["sampleId,cellId,setId"] + for sample in ("SmpA", "SmpB"): + for cell in ("c1", "c2", "c3"): + counts.append(f"{sample},{cell},{CUSTOMER_TAGS[0]},900") + counts.extend(f"{sample},{cell},{t},10" for t in CUSTOMER_TAGS[1:]) + linker.append(f"{sample},{cell},{'K1' if span_samples else 'K' + sample}") + (root / "counts.csv").write_text("\n".join(counts) + "\n") + (root / "linker.csv").write_text("\n".join(linker) + "\n") + return root + + +CUSTOMER_ARGS = [ + "counts.csv", + "panel.csv", + "--linker", + "linker.csv", + "--barcode-col", + "Sequence", + "--feature-col", + "Antigen", + "--sample-col", + "Sample", + "--output-prefix", + "result", +] + + +def test_a_panel_with_no_role_column_still_produces_verdicts(bed): + # No --role-column and no --reference-values, because the file has no column to name. The run must + # not fail and must not read unreliable throughout: nine tags clear the minimum of eight, so the + # panel's own readings serve. + _customer_bed(bed) + r = _run(bed, *CUSTOMER_ARGS) + assert r.returncode == 0, r.stderr + + meta = json.loads((bed / "result_run_meta.json").read_text()) + assert meta["referenceChoice"] == ReferenceChoice.PANEL.value + assert meta["referenceValues"] == [], "nothing can be declared without a role column" + # With no grouping column either, every barcode is its own identity. + assert meta["identities"] == CUSTOMER_TAGS + + states = pl.read_csv(bed / "result_verdicts.csv", infer_schema_length=0) + assert set(states["state"].to_list()) != {"unreliable"}, "the panel could serve and was not asked to" + # The strong barcode reads bound and the background does not, or the bed cannot tell a working + # comparator from a broken one. + by_identity = dict(zip(states["identity"].to_list(), states["state"].to_list(), strict=True)) + assert by_identity[CUSTOMER_TAGS[0]] == "bound" + assert {by_identity[t] for t in CUSTOMER_TAGS[1:]} == {"not bound"} + + +def test_a_barcode_renamed_between_samples_falls_back_to_its_sequence_as_a_label(bed): + # A barcode the two samples name differently carries no agreed antigen name, so the label column + # has nothing to show and falls back to the barcode itself. A scientist then reads a raw 15-mer + # where every other row shows an antigen. + _customer_bed(bed, renamed=2) + assert _run(bed, *CUSTOMER_ARGS).returncode == 0 + + labels = pl.read_csv(bed / "result_identity_labels.csv", infer_schema_length=0) + by_identity = dict(zip(labels["identity"].to_list(), labels["label"].to_list(), strict=True)) + + renamed = CUSTOMER_TAGS[:2] + for tag in renamed: + assert by_identity[tag] == tag, f"{tag} disagrees across samples, so it has no name to show" + # The consistently-named barcodes DO keep their antigen name. Without this the assertion above + # would also pass on a build that had simply stopped emitting labels at all. + for i, tag in enumerate(CUSTOMER_TAGS[2:], start=2): + assert by_identity[tag] == f"Ag{i:03d}", f"{tag} agrees across samples and must show its name" + + +def test_the_label_fallback_is_caused_by_the_disagreement_and_nothing_else(bed): + # Same bed with the renaming removed: every barcode now agrees across both samples, so no label + # falls back. This is what makes the previous test a statement about disagreement rather than about + # this bed's barcodes. + _customer_bed(bed, renamed=0) + assert _run(bed, *CUSTOMER_ARGS).returncode == 0 + + labels = pl.read_csv(bed / "result_identity_labels.csv", infer_schema_length=0) + fell_back = [ + identity + for identity, label in zip(labels["identity"].to_list(), labels["label"].to_list(), strict=True) + if identity == label + ] + assert fell_back == [], "no barcode disagrees here, so no label should fall back" From d500fb8565df6698ee3c275392997fa6c5eda0b3 Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Tue, 18 Aug 2026 14:32:00 +0200 Subject: [PATCH 068/282] MILAB-6496: add the two shapes a real panel file arrives in MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both were observed in use at one account at the same time, on two of its projects, so neither is a legacy form of the other. Added as projections of the bed's existing slots, samples and names, so counts.csv and linker.csv apply to all five panels unchanged and the only thing that varies is the shape of the declaration. The three existing panels regenerate byte-identically. panel_narrow.csv — three columns: sample, barcode, antigen name. No role column, so nothing can be named a comparator and the panel's own readings serve. The control sits in it as an ordinary row nothing marks, which is how the observed file carries it. panel_wide.csv — seven columns, adding three things the narrow shape cannot show: a catalogue id 1:1 with the sequence, so pointing the barcode role at the wrong one of the two joins to nothing; a channel column holding four values that are three channels, one spelled two ways; and a constant column, a declared property carrying no information. Its role column declares target versus off-target and carries no comparator value at all. It also carries case-variant role values, because the observed file held six values that were three roles. Two failure modes, kept separate so a test can tell them apart: one barcode reading two spellings across samples loses the property entirely and ends up with no role, while a barcode consistently spelled the other way keeps its role but no longer matches the first. Three tests, all mutation-verified: - the narrow shape resolves to the panel's own readings, and exactly the barcodes the samples name differently lose their label — the count derived from the bed, not written down, so a reseeded bed still asserts the shape. - naming the off-target role as the comparator DELETES the off-target questions. Reference tags are held out of the identity universe, so the role column is not merely the wrong source for a comparator: it removes the question an off-target exists to pose. Compared against a run of the same panel with no role column, so it is a statement about the naming rather than about this bed. - a role value differing only in case is not matched, silently, so it stays a question while its identically-roled siblings become comparators. Caught by folding case in the reference match, and by keeping reference tags in the identity universe. --- .../test/test_emit_verdicts.py | 128 +++++++++++++++++- .../test-data/fixtures/verdicts/README.md | 36 +++++ .../test-data/fixtures/verdicts/generate.py | 95 +++++++++++++ .../fixtures/verdicts/panel_narrow.csv | 21 +++ .../fixtures/verdicts/panel_wide.csv | 21 +++ 5 files changed, 300 insertions(+), 1 deletion(-) create mode 100644 software/test-data/fixtures/verdicts/panel_narrow.csv create mode 100644 software/test-data/fixtures/verdicts/panel_wide.csv diff --git a/software/per-cell-metrics/test/test_emit_verdicts.py b/software/per-cell-metrics/test/test_emit_verdicts.py index 1ac45ee..d785e4e 100644 --- a/software/per-cell-metrics/test/test_emit_verdicts.py +++ b/software/per-cell-metrics/test/test_emit_verdicts.py @@ -602,7 +602,15 @@ def test_the_floor_runs_before_tags_combine(bed): # one comparator and two, and a barcode declared on one sample and read on another. VERDICT_BED = Path(__file__).resolve().parents[2] / "test-data" / "fixtures" / "verdicts" -VERDICT_BED_FILES = ("counts.csv", "linker.csv", "panel.csv", "panel_with_reference.csv", "panel_multi_reference.csv") +VERDICT_BED_FILES = ( + "counts.csv", + "linker.csv", + "panel.csv", + "panel_with_reference.csv", + "panel_multi_reference.csv", + "panel_narrow.csv", + "panel_wide.csv", +) NAME_GROUPING = ("--grouping", json.dumps({"by": "property", "column": "Name"})) @@ -1037,3 +1045,121 @@ def test_the_label_fallback_is_caused_by_the_disagreement_and_nothing_else(bed): if identity == label ] assert fell_back == [], "no barcode disagrees here, so no label should fall back" + + +# --- the two shapes, run against the committed bed ---------------------------------------------- +# +# The three tests further up use an inline bed to fix what a role-less panel does. These two run the +# committed bed's own projections of the same slots, so they can be compared against each other and +# against the four-column panels — the only thing that varies is the shape of the declaration. + +NARROW_COLS = ["--barcode-col", "Sequence", "--feature-col", "Antigen", "--sample-col", "Sample"] +WIDE_COLS = ["--barcode-col", "Sequence", "--feature-col", "Name", "--sample-col", "Samples"] + + +def _wide_roles(bed): + """tag -> the set of Type values it is declared with, from the seven-column panel.""" + panel = pl.read_csv(bed / "panel_wide.csv", infer_schema_length=0) + roles: dict[str, set[str]] = {} + for row in panel.iter_rows(named=True): + roles.setdefault(row["Sequence"], set()).add(row["Type"]) + return roles + + +def test_the_narrow_shape_loses_a_label_for_every_barcode_the_samples_name_differently(wide_bed): + # Nine tags and no role column: the panel's own readings serve, and every barcode two samples name + # differently has no agreed name to show. The count is derived from the bed rather than written + # down, so a bed regenerated under another seed still asserts the same shape. + shape = _bed_shape(wide_bed) + r = _run( + wide_bed, "counts.csv", "panel_narrow.csv", "--linker", "linker.csv", *NARROW_COLS, "--output-prefix", "result" + ) + assert r.returncode == 0, r.stderr + meta = json.loads((wide_bed / "result_run_meta.json").read_text()) + assert meta["referenceChoice"] == ReferenceChoice.PANEL.value + + labels = pl.read_csv(wide_bed / "result_identity_labels.csv", infer_schema_length=0) + fell_back = {r["identity"] for r in labels.iter_rows(named=True) if r["identity"] == r["label"]} + # Exactly the renamed barcodes, and nothing else: the ones the samples agree on keep their names. + assert fell_back == shape["renamed"] + assert fell_back, "the bed must rename at least one barcode or this test asserts nothing" + assert len(fell_back) < labels.height, "and must not rename all of them" + + +def test_naming_the_off_target_role_as_the_comparator_deletes_the_off_target_questions(wide_bed): + # The role column says what a member is TO THE QUESTION; the comparator is a different axis. Naming + # the off-target role as the comparator does not merely move a baseline — reference tags are held + # out of the identity universe, so the off-targets stop being asked about at all. + roles = _wide_roles(wide_bed) + off_target = {tag for tag, values in roles.items() if values == {"Off-Target"}} + assert off_target, "the bed must declare at least one off-target for this test to mean anything" + + assert ( + _run( + wide_bed, "counts.csv", "panel_wide.csv", "--linker", "linker.csv", *WIDE_COLS, "--output-prefix", "plain" + ).returncode + == 0 + ) + asked_without = {identity for _, identity in _states_prefix(wide_bed, "plain")} + + assert ( + _run( + wide_bed, + "counts.csv", + "panel_wide.csv", + "--linker", + "linker.csv", + *WIDE_COLS, + "--role-column", + "Type", + "--reference-values", + "Off-Target", + "--output-prefix", + "named", + ).returncode + == 0 + ) + asked_with = {identity for _, identity in _states_prefix(wide_bed, "named")} + + # Without the naming they are questions; with it they are gone. + assert off_target <= asked_without, "an off-target is an identity when nothing names it a comparator" + assert not (off_target & asked_with), "naming the role deleted the off-target questions" + assert asked_with, "and must not delete every question, or the bed says nothing about which went" + + +def test_a_role_value_differing_only_in_case_is_not_matched(wide_bed): + # The observed file held six Type values that were three roles. A tag whose role is spelled + # `Off-target` is not selected by `Off-Target`, silently — so it stays a question while its + # identically-roled siblings become comparators. + roles = _wide_roles(wide_bed) + variant = { + tag + for tag, values in roles.items() + if len(values) == 1 and (v := next(iter(values))) != "Off-Target" and v.lower() == "off-target" + } + assert variant, "the bed must carry a case variant of the off-target role" + + assert ( + _run( + wide_bed, + "counts.csv", + "panel_wide.csv", + "--linker", + "linker.csv", + *WIDE_COLS, + "--role-column", + "Type", + "--reference-values", + "Off-Target", + "--output-prefix", + "named", + ).returncode + == 0 + ) + asked = {identity for _, identity in _states_prefix(wide_bed, "named")} + assert variant <= asked, "the case-variant tag was silently left as a question" + + +def _states_prefix(bed, prefix): + v = pl.read_csv(bed / f"{prefix}_verdicts.csv", infer_schema_length=0) + return {(r["setId"], r["identity"]) for r in v.iter_rows(named=True)} diff --git a/software/test-data/fixtures/verdicts/README.md b/software/test-data/fixtures/verdicts/README.md index 30fc315..96cfe23 100644 --- a/software/test-data/fixtures/verdicts/README.md +++ b/software/test-data/fixtures/verdicts/README.md @@ -15,6 +15,42 @@ samples are `SNN`, and no real sequence, antigen or sample identifier appears an | `panel.csv` | Four samples, panels of 3, 4, 4 and 5 tags. **No** comparator tag. | | `panel_with_reference.csv` | The same panels plus **one** comparator tag (`Ctrl1`) on every sample. | | `panel_multi_reference.csv` | The same panels plus **two** comparator tags (`Ctrl1`, `Ctrl2`) on every sample. | +| `panel_narrow.csv` | The **three-column** shape: sample, barcode, antigen name, and no fourth column. No role column, so nothing can be named as a comparator and the panel's own readings serve. The control is an ordinary row nothing marks. | +| `panel_wide.csv` | The **seven-column** shape: sample, name, catalogue id, barcode, channel, a constant column, role. The role column declares target vs off-target and carries **no** comparator value. Includes case-variant role values. | + +### The two customer shapes, and what they are for + +Both were observed in use at one account at the same time, on two of its projects — so neither is a +legacy form of the other. They are projections of the same slots, samples and names as the three panels +above, which is what lets `counts.csv` and `linker.csv` apply to all five unchanged: the panels differ +only in the shape of the declaration. + +**Neither carries a value meaning "comparator."** In both, the negative control is one antigen the +scientist points at by name in the interface. So a run over either resolves to the panel's own readings. + +`panel_narrow.csv` reproduces, on nine tags, what the observed file does on seventeen: four barcodes +carry a different antigen name in different samples, so four identity labels fall back to the raw +barcode. Run it with `--barcode-col Sequence --feature-col Antigen --sample-col Sample` and no +`--role-column`. + +`panel_wide.csv` adds three things the narrow shape cannot show. A **catalogue id** 1:1 with the +sequence, so pointing the barcode role at the wrong one of the two joins to nothing. A **channel** +column holding four values that are three channels, one of them spelled two ways. A **constant** +column, which is a declared property carrying no information — group on it and every tag becomes one +identity. + +And its `Type` column carries **case variants**, deliberately, because the observed file held six values +that were three roles. Two failure modes, kept separate so a test can tell them apart: + +- `A0`'s slot reads `Target (Primary)` in two samples and `Target (primary)` in the other two. One + barcode, two values, so the property is **dropped for that tag entirely** and it ends up with no role. +- `A5`'s slot reads `Off-target` wherever it appears — self-consistent, so it keeps its role, but it no + longer matches the `Off-Target` written elsewhere. Selecting one value silently misses the other. + +**What running the wide panel with `--reference-values "Off-Target"` demonstrates** is why the role +column is the wrong source for a comparator: reference tags are held out of the identity universe, so +the identity count drops from nine to seven and **the off-targets stop being asked about at all**. The +question an off-target exists to pose is deleted rather than answered. | `counts.csv` | Sparse per-(sample, cell, barcode) UMI counts for all eleven cells. | | `linker.csv` | Cell to clonotype set: `K01`, `K02`, `K03` (spanning two samples), `K04` (a singleton). | diff --git a/software/test-data/fixtures/verdicts/generate.py b/software/test-data/fixtures/verdicts/generate.py index b4b7789..412cd48 100644 --- a/software/test-data/fixtures/verdicts/generate.py +++ b/software/test-data/fixtures/verdicts/generate.py @@ -211,11 +211,106 @@ def write_panel(path: str, seq: dict[str, str], controls: list[str]) -> None: f.write(f"{sample},{CONTROL_NAMES[slot]},{seq[slot]},Control\n") +# --- the two shapes real panel files arrive in -------------------------------------------------- +# +# Projections of the same slots, samples and names as the panels above, so counts.csv and linker.csv +# apply to them unchanged and the three panels differ only in the shape of the declaration. Both +# shapes were observed in use at one account, at the same time, on two of its projects. +# +# NEITHER carries a value meaning "comparator", and that is the point of them. In both, the negative +# control is one antigen the scientist points at by name in the interface -- so a run over either +# resolves to the panel's own readings, and `--reference-values` has nothing correct to name. The +# `Type` column of the wide shape declares what a member is TO THE QUESTION (a target, an off-target), +# which is a different axis from what a count is read against. Naming `Off-Target` as the comparator +# does not merely mis-set a baseline: reference tags are held out of the identity universe, so every +# off-target stops being asked about at all -- the question an off-target exists to pose is deleted. + +# A per-slot catalogue id, 1:1 with the sequence. Real files carry both, and a reader who points the +# barcode role at the catalogue id instead of the sequence joins to nothing. +CATALOGUE_IDS = {slot: f"T{100 + i:04d}" for i, slot in enumerate(ANTIGEN_SLOTS + CONTROL_SLOTS)} + +# Four distinct values, two of which are one channel spelled two ways -- so grouping on this column +# splits one channel in two. +CHANNELS = { + "A0": "PE", "A1": "PE", + "A2": "APC", "A3": "APC", + "A4": "PE Dazzle", "A5": "PE Dazzle", + "A6": "PE-Dazzle 5120", "A7": "PE-Dazzle 5120", + "R0": "APC", "R1": "APC", +} + +# One value on every row: a declared column that carries no information at all. Grouping on it puts +# every tag in one identity, which is legal and useless. +RESIDUES = "ECD protein" + +# What each member is to the question. No "Control" anywhere, deliberately. +TYPES = { + "A0": "Target (Primary)", "A1": "Target (Primary)", + "A2": "Off-Target", "A3": "Target (Secondary)", + "A4": "Target (Secondary)", "A5": "Off-Target", + "A6": "Target (Primary)", "A7": "Target (Primary)", + "R0": "Off-Target", "R1": "Off-Target", +} + +# Case variants, because the observed file carried six values that were three roles. Two failure +# modes, kept separate so a test can tell them apart: +# +# A0 reads "Target (Primary)" in S01/S03 and "Target (primary)" in S02/S04. One barcode, two +# values, so the property is dropped for that tag entirely -- it ends up with no role at all. +# +# A5 reads "Off-target" everywhere it appears. Self-consistent, so it keeps its role, but it no +# longer matches A2's "Off-Target" -- so selecting one role value silently misses the other. +LOWERCASED_IN = {"A0": {"S02", "S04"}} +ALWAYS_LOWERCASED = {"A5"} + + +def _typed(slot: str, sample: str) -> str: + role = TYPES[slot] + if slot in ALWAYS_LOWERCASED or sample in LOWERCASED_IN.get(slot, set()): + # Lowercase only the parenthesised qualifier or the word after the hyphen, which is how the + # observed variants differed -- not a blanket .lower(). + return role.replace("(P", "(p").replace("(S", "(s").replace("-Target", "-target") + return role + + +def write_panel_narrow(path: str, seq: dict[str, str]) -> None: + """Three columns and no fourth: sample, barcode, antigen name. + + The declared control is present as an ordinary antigen row, exactly as it is in the observed file + -- nothing in the table says it is the control. + """ + with open(path, "w") as f: + f.write("Sample,Sequence,Antigen\n") + for sample, name, slot in PANEL: + f.write(f"{sample},{seq[slot]},{name}\n") + for sample in SAMPLES: + f.write(f"{sample},{seq['R0']},{CONTROL_NAMES['R0']}\n") + + +def write_panel_wide(path: str, seq: dict[str, str]) -> None: + """Seven columns: sample, name, catalogue id, barcode, channel, a constant, and the role.""" + with open(path, "w") as f: + f.write("Samples,Name,Barcode,Sequence,Channel,Residues,Type\n") + for sample, name, slot in PANEL: + f.write( + f"{sample},{name},{CATALOGUE_IDS[slot]},{seq[slot]}," + f"{CHANNELS[slot]},{RESIDUES},{_typed(slot, sample)}\n" + ) + for sample in SAMPLES: + slot = "R0" + f.write( + f"{sample},{CONTROL_NAMES[slot]},{CATALOGUE_IDS[slot]},{seq[slot]}," + f"{CHANNELS[slot]},{RESIDUES},{_typed(slot, sample)}\n" + ) + + def main() -> None: seq = barcodes() write_panel("panel.csv", seq, []) write_panel("panel_with_reference.csv", seq, ["R0"]) write_panel("panel_multi_reference.csv", seq, ["R0", "R1"]) + write_panel_narrow("panel_narrow.csv", seq) + write_panel_wide("panel_wide.csv", seq) with open("counts.csv", "w") as f: f.write("sampleId,cellId,tag,umiCount\n") diff --git a/software/test-data/fixtures/verdicts/panel_narrow.csv b/software/test-data/fixtures/verdicts/panel_narrow.csv new file mode 100644 index 0000000..19965a2 --- /dev/null +++ b/software/test-data/fixtures/verdicts/panel_narrow.csv @@ -0,0 +1,21 @@ +Sample,Sequence,Antigen +S01,AGAACCCCCCTT,Ag01 +S01,AGTTAAGAACAA,Ag02 +S01,AAGCAACAATCT,Ag03 +S02,AGAACCCCCCTT,Ag11 +S02,AGTTAAGAACAA,Ag02 +S02,TCGTGGTCCTGG,Ag04 +S02,TCCGTGACTTTG,Ag05 +S03,AGAACCCCCCTT,Ag01 +S03,AAGCAACAATCT,Ag03 +S03,TCGTGGTCCTGG,Ag14 +S03,ACCTTACGGGCT,Ag06 +S04,AGAACCCCCCTT,Ag11 +S04,AGTTAAGAACAA,Ag12 +S04,TCCGTGACTTTG,Ag15 +S04,CTTTTTGCCGTT,Ag07 +S04,CATCTCTAGTCT,Ag07 +S01,TGTAGACGCATA,Ctrl1 +S02,TGTAGACGCATA,Ctrl1 +S03,TGTAGACGCATA,Ctrl1 +S04,TGTAGACGCATA,Ctrl1 diff --git a/software/test-data/fixtures/verdicts/panel_wide.csv b/software/test-data/fixtures/verdicts/panel_wide.csv new file mode 100644 index 0000000..608581e --- /dev/null +++ b/software/test-data/fixtures/verdicts/panel_wide.csv @@ -0,0 +1,21 @@ +Samples,Name,Barcode,Sequence,Channel,Residues,Type +S01,Ag01,T0100,AGAACCCCCCTT,PE,ECD protein,Target (Primary) +S01,Ag02,T0101,AGTTAAGAACAA,PE,ECD protein,Target (Primary) +S01,Ag03,T0102,AAGCAACAATCT,APC,ECD protein,Off-Target +S02,Ag11,T0100,AGAACCCCCCTT,PE,ECD protein,Target (primary) +S02,Ag02,T0101,AGTTAAGAACAA,PE,ECD protein,Target (Primary) +S02,Ag04,T0103,TCGTGGTCCTGG,APC,ECD protein,Target (Secondary) +S02,Ag05,T0104,TCCGTGACTTTG,PE Dazzle,ECD protein,Target (Secondary) +S03,Ag01,T0100,AGAACCCCCCTT,PE,ECD protein,Target (Primary) +S03,Ag03,T0102,AAGCAACAATCT,APC,ECD protein,Off-Target +S03,Ag14,T0103,TCGTGGTCCTGG,APC,ECD protein,Target (Secondary) +S03,Ag06,T0105,ACCTTACGGGCT,PE Dazzle,ECD protein,Off-target +S04,Ag11,T0100,AGAACCCCCCTT,PE,ECD protein,Target (primary) +S04,Ag12,T0101,AGTTAAGAACAA,PE,ECD protein,Target (Primary) +S04,Ag15,T0104,TCCGTGACTTTG,PE Dazzle,ECD protein,Target (Secondary) +S04,Ag07,T0106,CTTTTTGCCGTT,PE-Dazzle 5120,ECD protein,Target (Primary) +S04,Ag07,T0107,CATCTCTAGTCT,PE-Dazzle 5120,ECD protein,Target (Primary) +S01,Ctrl1,T0108,TGTAGACGCATA,APC,ECD protein,Off-Target +S02,Ctrl1,T0108,TGTAGACGCATA,APC,ECD protein,Off-Target +S03,Ctrl1,T0108,TGTAGACGCATA,APC,ECD protein,Off-Target +S04,Ctrl1,T0108,TGTAGACGCATA,APC,ECD protein,Off-Target From cef16bdf50760c34fb74b53a3a57b20f5f48a8e0 Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Tue, 18 Aug 2026 14:43:15 +0200 Subject: [PATCH 069/282] MILAB-6496: reshape a generated run's panel into the two real shapes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit generate.py emits one panel shape with a role column, one panel for every sample, and the control carrying its own Decoy role. Neither shape observed in use at the account looks like that, and both were in use at the same time on two of its projects. reshape_panel.py rewrites a run's tags.csv into both, keeping every barcode unchanged so either uploads against the same FASTQs — a reshaped panel with a changed barcode joins to nothing, which is why nothing touches the tag column. Sample names come from the antigen arm's filenames so they match Samples & Data. tags_narrow.csv three columns and no fourth. Nothing declares a role, so nothing can be named the comparator. tags_wide.csv seven columns: a catalogue id 1:1 with the sequence, a channel column holding four values that are three channels, a constant column, and a role column declaring target vs off-target with no comparator value anywhere in it. Both rename a barcode between samples, so one sequence carries two antigen names and those identities lose their label to a raw 15-mer. --drop-from-later makes a later sample declare fewer tags, which is what makes never-asked reachable. Also recorded a trap that makes the block look broken when it is not. This bed plants background at 1-3 UMIs, and 253 of 432 readings in a tiny --panel-size 12 run fall below the shipped count floor of 4. Neither shape declares a comparator, so the panel's own readings serve; with the floor at 4 that background is zeroed, the panel median collapses to 0, and 0 is under the reference thin line of 2 — so every cell with signal reads impossible to compare and nothing binds at all. At --floor 1 the same run gives 27 bound. Both numbers are defensible on their own and simply do not compose. --- software/test-data/manual/README.md | 45 ++++++ software/test-data/manual/reshape_panel.py | 170 +++++++++++++++++++++ 2 files changed, 215 insertions(+) create mode 100644 software/test-data/manual/reshape_panel.py diff --git a/software/test-data/manual/README.md b/software/test-data/manual/README.md index 0dd0500..0ff7ff6 100644 --- a/software/test-data/manual/README.md +++ b/software/test-data/manual/README.md @@ -375,3 +375,48 @@ real BEAM-T run) — a non-regenerable fallback pool used only if the full list that the `realistic` calibration targets. - `design-and-schemas.md` — design rationale, the join-spine axis contract, per-arm file schemas, and the biology/coherence model. + +## The Two Shapes A Real Panel File Arrives In + +`generate.py` emits one panel shape: `tag,feature,Type,Species,Class`, one panel for every sample, with +the control carrying its own `Decoy` role. Two other shapes were observed in use at one account at the +same time, on two of its projects, and neither looks like that. `reshape_panel.py` rewrites a generated +run's `tags.csv` into both, **keeping every barcode unchanged** so either can be uploaded against the +same FASTQs: + +```bash +python3 generate.py tiny --arm antigen --panel-size 12 --offtarget-count 3 +python3 generate.py tiny --arm vdj --panel-size 12 +python3 reshape_panel.py runs/tiny +``` + +| File | Shape | What it exercises | +|---|---|---| +| `tags_narrow.csv` | `Sample,Sequence,Antigen` | No role column at all, so nothing can be named as the comparator and the panel's own readings serve. The control is an ordinary row nothing marks. | +| `tags_wide.csv` | `Samples,Name,Barcode,Sequence,Channel,Residues,Type` | A role column that declares target vs off-target and carries **no** comparator value; a catalogue id 1:1 with the sequence; a channel column holding four values that are three channels; a constant column; and case-variant role values. | + +Both rename a barcode between samples (`--rename`, default 2), so the same sequence carries a different +antigen name in different samples. Under the per-tag grouping the identity is the barcode, so those +identities lose their label and show a raw 15-mer. `--drop-from-later N` makes a later sample declare +fewer tags, which is what makes *never asked* reachable. + +**A panel below 8 tags cannot serve as its own comparator**, so generate at least that many +(`--panel-size 12` gives 12 + 1 control). The script warns if you are under. + +### ⚠️ Set the count floor to 1 for these two shapes + +This bed plants background at 1–3 UMIs per barcode — **253 of 432 readings in a `tiny --panel-size 12` +run sit below the shipped count floor of 4**. Neither shape declares a comparator, so the panel's own +readings have to serve, and with the floor at 4 that background is zeroed, the panel median collapses to +0, and 0 is below the reference thin line of 2. Every cell carrying signal then reads *impossible to +compare*: + +``` +--floor 4 not bound 702, unreliable 260, bound 0 <- looks broken, is not +--floor 1 not bound 909, bound 27, unreliable 26 +``` + +So set **Advanced → count floor = 1** in the block when uploading either shape. The two numbers are each +defensible and simply do not compose: the bed's background is calibrated to a real 5k BEAM-T library, +and the floor of 4 comes from the antibody-side lineage. A declared comparator would sidestep it, which +is exactly what neither of these shapes can supply. diff --git a/software/test-data/manual/reshape_panel.py b/software/test-data/manual/reshape_panel.py new file mode 100644 index 0000000..6a133c4 --- /dev/null +++ b/software/test-data/manual/reshape_panel.py @@ -0,0 +1,170 @@ +#!/usr/bin/env python3 +"""Rewrite a generated run's tags.csv into the two shapes a real panel file arrives in. + + python3 reshape_panel.py runs/tiny + +Writes `tags_narrow.csv` and `tags_wide.csv` beside `tags.csv`, both carrying **the same barcodes**, so +either can be uploaded to the block against the same FASTQs. A reshaped panel with a changed barcode +joins to nothing, which is why nothing here touches the `tag` column. + +Why this exists. `generate.py` emits one panel shape: `tag,feature,Type,Species,Class`, one panel for +every sample, with the control carrying its own `Decoy` role. Two other shapes were observed in use at +one account at the same time, on two of its projects, and neither looks like that: + + narrow sample, barcode, antigen name — and no fourth column. Nothing declares a role, so nothing can + be named as the comparator and the panel's own readings have to serve. + wide sample, name, catalogue id, barcode, channel, a constant, role. The role column declares what + a member is TO THE QUESTION (target, off-target) and carries **no** comparator value. + +In both, the negative control is one antigen the scientist points at by name in the interface. So +neither shape can reach the declared-comparator path, for two different reasons — which is the thing +these files exist to make visible in the app rather than only in a CSV. + +Both shapes rename a barcode between samples: the same sequence carries a different antigen name in +different samples, which is the tag-inventory reuse the per-sample keying of the panel exists for. Under +the per-tag grouping the identity is the barcode, so those identities lose their label and a reader +meets a raw 15-mer where every other row shows an antigen. + +Deterministic: every choice below is positional, so a rerun over the same tags.csv is byte-identical. +Stdlib only, like the rest of this bed. +""" + +import argparse +import csv +import os +import sys + +# Four values that are three channels — one of them spelled two ways, so grouping on this column splits +# one channel in two. Assigned by position, cycling. +CHANNELS = ["PE", "PE", "APC", "APC", "PE Dazzle", "PE Dazzle", "PE-Dazzle 5120", "PE-Dazzle 5120"] + +# One value on every row: a declared property carrying no information at all. Group on it and every tag +# lands in one identity, which is legal and useless. +RESIDUES = "ECD protein" + +# The control's own role is folded into the off-target set on purpose. The observed wide file had no +# value meaning "comparator" anywhere in its role column, and that is the whole point of the shape. +ROLE_OF = {"Target": "Target (Primary)", "Off-Target": "Off-Target", "Decoy": "Off-Target"} + + +def _lowercased(role: str) -> str: + """The role as the observed file also spelled it — the qualifier or the word after the hyphen.""" + return role.replace("(P", "(p").replace("(S", "(s").replace("-Target", "-target") + + +def read_tags(run_dir: str) -> list[dict]: + path = os.path.join(run_dir, "tags.csv") + if not os.path.exists(path): + sys.exit(f"no tags.csv in {run_dir} — generate a run first (python3 generate.py tiny --arm antigen)") + with open(path, newline="") as f: + rows = list(csv.DictReader(f)) + for col in ("tag", "feature"): + if not rows or col not in rows[0]: + sys.exit(f"{path} has no {col!r} column; columns are {list(rows[0]) if rows else '[]'}") + return rows + + +def read_samples(run_dir: str) -> list[str]: + """Sample names from the antigen arm's filenames, so they match Samples & Data exactly.""" + antigen = os.path.join(run_dir, "antigen") + if not os.path.isdir(antigen): + sys.exit(f"no antigen/ arm in {run_dir} — run: python3 generate.py --arm antigen") + names = sorted({f.split("_R")[0] for f in os.listdir(antigen) if f.endswith(".fastq.gz")}) + if not names: + sys.exit(f"no FASTQs in {antigen}") + return names + + +def _renamed(name: str, sample_index: int) -> str: + """A plainly different antigen name for a reused barcode in a later sample.""" + return f"{name}__alt{sample_index}" + + +def write_narrow(path: str, tags: list[dict], samples: list[str], rename: int, drop: int) -> int: + with open(path, "w", newline="") as f: + w = csv.writer(f) + w.writerow(["Sample", "Sequence", "Antigen"]) + rows = 0 + for s_i, sample in enumerate(samples): + # A later sample may declare fewer tags, which is what makes *never asked* reachable: a set + # whose cells sit only in that sample was never offered the dropped identities. + offered = tags[: len(tags) - drop] if s_i else tags + for t_i, tag in enumerate(offered): + name = _renamed(tag["feature"], s_i) if (s_i and t_i < rename) else tag["feature"] + w.writerow([sample, tag["tag"], name]) + rows += 1 + return rows + + +def write_wide(path: str, tags: list[dict], samples: list[str], rename: int, drop: int) -> int: + with open(path, "w", newline="") as f: + w = csv.writer(f) + w.writerow(["Samples", "Name", "Barcode", "Sequence", "Channel", "Residues", "Type"]) + rows = 0 + for s_i, sample in enumerate(samples): + offered = tags[: len(tags) - drop] if s_i else tags + for t_i, tag in enumerate(offered): + name = _renamed(tag["feature"], s_i) if (s_i and t_i < rename) else tag["feature"] + role = ROLE_OF.get(tag.get("Type", "Target"), "Target (Primary)") + # Two case-variant failure modes, kept apart so each can be told from the other: + # tag 0 reads one spelling in the first sample and another in the rest, so it carries + # two values, the property is dropped for it, and it ends up with no role at all; + # tag 1 reads the other spelling everywhere, so it keeps its role but no longer + # matches the same role written normally elsewhere. + if (t_i == 0 and s_i) or t_i == 1: + role = _lowercased(role) + w.writerow( + [ + sample, + name, + f"T{100 + t_i:04d}", + tag["tag"], + CHANNELS[t_i % len(CHANNELS)], + RESIDUES, + role, + ] + ) + rows += 1 + return rows + + +def main() -> None: + p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + p.add_argument("run_dir", help="a generated run directory, e.g. runs/tiny") + p.add_argument( + "--rename", + type=int, + default=2, + help="barcodes carrying a different antigen name in later samples (default 2; 0 disables)", + ) + p.add_argument( + "--drop-from-later", + type=int, + default=0, + help="tags a later sample does not declare, making *never asked* reachable (default 0)", + ) + args = p.parse_args() + + tags = read_tags(args.run_dir) + samples = read_samples(args.run_dir) + if args.rename > len(tags) or args.drop_from_later >= len(tags): + sys.exit(f"--rename/--drop-from-later exceed the panel's {len(tags)} tags") + + narrow = os.path.join(args.run_dir, "tags_narrow.csv") + wide = os.path.join(args.run_dir, "tags_wide.csv") + n_rows = write_narrow(narrow, tags, samples, args.rename, args.drop_from_later) + w_rows = write_wide(wide, tags, samples, args.rename, args.drop_from_later) + + kept = len(tags) - args.drop_from_later + print(f"{len(tags)} tags x {len(samples)} samples -> {samples}") + print(f" {narrow} ({n_rows} rows, 3 columns, no role column)") + print(f" {wide} ({w_rows} rows, 7 columns, role column with no comparator value)") + print(f" {args.rename} barcode(s) renamed in later samples; later samples declare {kept} of {len(tags)}") + if len(tags) < 8: + print(f" WARNING: {len(tags)} tags is below the shipped panel minimum of 8, so the panel's own") + print(" readings cannot serve and every verdict will read unreliable. Regenerate with") + print(" a larger --panel-size, or lower the minimum in the block's settings.") + + +if __name__ == "__main__": + main() From 6c1f944703a042245e2f02e09739edc113da712c Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Tue, 18 Aug 2026 15:13:58 +0200 Subject: [PATCH 070/282] MILAB-6496: defer the contending-antigen editor MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Commented out, with the reasoning in place so it can be restored by uncommenting. Only the editor is deferred. contendingGroups stays in the block data, the args projection still passes it and the workflow still threads --contending, so a project that already carries groups keeps its competitor notes and nothing about the emitted verdicts changes. Two reasons. contending-grouping-chosen-at-annotation holds that which members form a contending group is chosen "over whatever properties the scientist's panel file carries" — a declared property such as a binding-site column, chosen at annotation and never frozen. Hand-entered lists are a different thing: they ask the scientist to retype a grouping the panel file is supposed to carry. And it was close to unusable at the default grouping, which its own warning admitted: under one-identity-per-tag the identities ARE the barcodes, and the panel is read column by column before the run, so no barcode-to-name pairing exists yet and the picker could only offer 15-mers. Nothing in the corpus is contradicted. contention-travels-with-the-negative requires the note to travel with the verdict where a group exists, and it still does; it does not require this block to offer a way to type one in. --- .changeset/defer-contending-editor.md | 8 ++++++++ ui/src/components/VerdictSettings.vue | 29 +++++++++++++++++++++++++++ 2 files changed, 37 insertions(+) create mode 100644 .changeset/defer-contending-editor.md diff --git a/.changeset/defer-contending-editor.md b/.changeset/defer-contending-editor.md new file mode 100644 index 0000000..a25607a --- /dev/null +++ b/.changeset/defer-contending-editor.md @@ -0,0 +1,8 @@ +--- +'@platforma-open/milaboratories.feature-integration.ui': patch +'@platforma-open/milaboratories.feature-integration': patch +--- + +The contending-antigen editor is not offered for now. Only the editor is deferred: a project that already carries contending groups keeps them, the args projection still passes them, and the emitted verdicts and their competitor notes are unchanged. + +It asked the scientist to retype by hand a grouping the panel file is meant to declare, and at the default one-identity-per-tag grouping the identities are the barcodes themselves — so the picker could only offer raw 15-mers. What it should become is contention derived from a declared panel column, alongside the existing grouping choice. diff --git a/ui/src/components/VerdictSettings.vue b/ui/src/components/VerdictSettings.vue index 9b0260e..5b2d76a 100644 --- a/ui/src/components/VerdictSettings.vue +++ b/ui/src/components/VerdictSettings.vue @@ -224,6 +224,34 @@ function removeContendingGroup(index: number) { + Date: Tue, 18 Aug 2026 15:56:46 +0200 Subject: [PATCH 071/282] MILAB-6496: replace the verdict and quality tables with a punchcard MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Binding verdicts and Quality checks pages are removed. A punchcard takes their place, in the shape the use case's own figure gives it (assets/punch-card.svg): one row per clonotype set, one column per antigen identity, and one punch per cell — a filled dot for bound, a blank cell for not bound, a dashed ring for a position never offered. The fourth state, which the figure predates, gets a solid ring. The identities on show come from a dropdown. Every identity is already in the pivoted frame, so picking one costs a redraw rather than a run, and a thousand-antigen panel does not render a thousand columns nobody asked for. The punch's size carries what the verdict rests on. support-travels-with-the-reading requires both counts to travel with a verdict wherever it appears, and its reason is about the page: a reading resting on three cells must not look like one resting on forty. The pivot's cell therefore carries state, cells that answered and cells that could have answered together — one column per identity, because a grid pairs a cell with another column's cell only by position, and because a column name here is an antigen name from a customer's panel file, so any suffix marking a second family is a name some panel is entitled to use. Both removed pages' artifacts are still emitted. The verdicts still export to downstream blocks and the quality frames are still built; verdict-block-interface names the run-level measurements as one of this block's two artifacts, and dropping a view does not release it from producing them. What no longer exists is the pair of grids. Whether those frames should instead cross the boundary as exports is left open rather than settled by leaving an unread output behind. The reading itself is untouched: no threshold, default or verdict moves. Block data migrates to v4, dropping the three grid states the removed views owned and adding the punchcard's state and its identity selection. --- .../punchcard-replaces-verdict-tables.md | 24 +++ model/src/index.ts | 187 ++++++++---------- model/src/types.ts | 16 +- .../per-cell-metrics/src/emit_verdicts.py | 44 ++++- .../test/test_emit_verdicts.py | 76 +++++++ test/src/wf.test.ts | 18 +- ui/src/app.ts | 6 +- ui/src/components/PunchCell.vue | 124 ++++++++++++ ui/src/pages/PunchcardPage.vue | 137 +++++++++++++ ui/src/pages/QualityChecksPage.vue | 127 ------------ ui/src/pages/VerdictsPage.vue | 100 ---------- workflow/src/column-specs.lib.tengo | 46 ++++- workflow/src/main.tpl.tengo | 5 + workflow/src/verdict-import.tpl.tengo | 20 +- workflow/src/verdict-run.tpl.tengo | 1 + 15 files changed, 563 insertions(+), 368 deletions(-) create mode 100644 .changeset/punchcard-replaces-verdict-tables.md create mode 100644 ui/src/components/PunchCell.vue create mode 100644 ui/src/pages/PunchcardPage.vue delete mode 100644 ui/src/pages/QualityChecksPage.vue delete mode 100644 ui/src/pages/VerdictsPage.vue diff --git a/.changeset/punchcard-replaces-verdict-tables.md b/.changeset/punchcard-replaces-verdict-tables.md new file mode 100644 index 0000000..18ed9cb --- /dev/null +++ b/.changeset/punchcard-replaces-verdict-tables.md @@ -0,0 +1,24 @@ +--- +'@platforma-open/milaboratories.feature-integration.per-cell-metrics': minor +'@platforma-open/milaboratories.feature-integration.workflow': minor +'@platforma-open/milaboratories.feature-integration.model': minor +'@platforma-open/milaboratories.feature-integration.ui': minor +'@platforma-open/milaboratories.feature-integration': minor +--- + +Replace the Binding verdicts and Quality checks tables with a punchcard + +The two result tables are removed as views. A punchcard takes their place: rows are clonotype sets, +columns are the antigen identities picked from a dropdown, and a cell is one punch whose colour is the +verdict and whose size is the support behind it. Every identity is already in the result, so picking one +costs a redraw rather than a run. + +Both artifacts are still emitted. The verdicts and the run's own measurements are what the block owes, +and dropping a view does not release it from producing them — the verdicts still export to downstream +blocks, and the quality frames are still built by the workflow. What no longer exists is the pair of +grids that presented them. + +The reading itself is unchanged: no threshold, default or verdict moves. + +Block data migrates to v4, dropping the three grid states the removed views owned and adding the +punchcard's own state and its identity selection. diff --git a/model/src/index.ts b/model/src/index.ts index 2337954..e6dff53 100644 --- a/model/src/index.ts +++ b/model/src/index.ts @@ -1,4 +1,4 @@ -import type { BlockRenderCtx, InferOutputsType } from "@platforma-sdk/model"; +import type { BlockRenderCtx, InferOutputsType, PlDataTableStateV2 } from "@platforma-sdk/model"; import { BlockModelV3, createPlDataTableStateV2, @@ -29,15 +29,16 @@ const DEFAULT_PANEL_REFERENCE_MIN_MEMBERS = 8; const DEFAULT_REFERENCE_THIN_LINE = 2; const DEFAULT_HIGH_REFERENCE_LINE = 100; -// The two axes the verdict view is assembled from. The exported verdict frame carries every table the -// reading emitted — per-cell counts keyed by cell, the offered scope keyed by sample, the tag→identity -// linker keyed by tag — and any of them joined into this view fans one verdict row into many. So the view -// is pinned to its own key: what a verdict is ABOUT (the identity) and who it is about it FOR (the -// clonotype set). The set axis name is the one the datasetOptions query requires of an anchor, so a -// dataset that could be picked here always carries it. -const IDENTITY_AXIS = "pl7.app/antigen/identityId"; -const CLONOTYPE_SET_AXIS = "pl7.app/vdj/scClonotypeKey"; -const PANEL_AXIS = "pl7.app/antigen/panelId"; +// The punchcard's frame is keyed on the clonotype set alone, and each identity is a COLUMN rather than an +// axis value — which is what a punchcard needs and what a (set, identity) frame cannot give a table. The +// identity therefore travels in the column's DOMAIN, which is how the model reads which identity a column +// belongs to without parsing a label. +// +// One column per identity, its value carrying the state and both support counts together (see +// identityPunchImportSpec). The pairing is inside the value because a grid pairs a cell with another +// column's cell only by position, which no import guarantees. +const IDENTITY_PUNCH_COLUMN = "pl7.app/antigen/identityPunch"; +const IDENTITY_ID_DOMAIN = "pl7.app/antigen/identityId"; // The run record emit_verdicts.py writes (result_run_meta.json), read as content. Only the fields the UI // states back to the user are typed here; the file carries every parameter the reading used. @@ -195,25 +196,6 @@ function parseQcRows(ctx: BlockRenderCtx) { return (qcMap?.data ?? []).filter((e) => e.value != null); } -// The readable name of each declared tag set, for the two report tables keyed on the panel axis. -// -// A panel id is a twelve-character hash of the sorted tag list — stable across re-runs of the same -// declaration, and unreadable on sight. The name emit_verdicts.py writes for it (" tags: ") is a label column in the EXPORTED verdict frame, so it has to travel in the columns list -// like the identity label does on the verdict table: createPlDataTable discovers label columns in the -// result pool, and a block's own exports are not in its own pool. -function panelLabelColumns(ctx: BlockRenderCtx) { - const pCols = ctx.outputs - ?.resolve({ field: "antigenVerdictsTable", allowPermanentAbsence: true }) - ?.getPColumns(); - return (pCols ?? []).filter( - (c) => - c.spec.name === "pl7.app/label" && - c.spec.axesSpec.length === 1 && - c.spec.axesSpec[0].name === PANEL_AXIS, - ); -} - // Tag→feature CSV metadata from the prerun (emit-csv-meta), or undefined until staging has produced it. // Shared by the two column dropdowns, the control dropdown, and the csvColumnsLoading signal. function readCsvMeta(ctx: BlockRenderCtx): CsvMeta | undefined { @@ -264,11 +246,19 @@ function suggestSampleColumn(ctx: BlockRenderCtx): string return best?.col; } +// v3 data shape: the reading's parameters, with the three grid states the two removed result views owned. +// v4 replaces them with the punchcard's own state and its identity picker. +type BlockDataV3 = Omit & { + verdictTableState: PlDataTableStateV2; + antigenQcTableState: PlDataTableStateV2; + panelMismatchTableState: PlDataTableStateV2; +}; + // v2 data shape: the preset selector + pattern string, with the dominance-era parameters still on it. // The dominant-feature readout, the off-target designation and the specificity score they fed are gone // from per_cell_metrics.py, so nothing consumes these three any more. type BlockDataV2 = Omit< - BlockData, + BlockDataV3, | "datasetRef" | "roleColumn" | "referenceValues" @@ -326,7 +316,7 @@ const dataModel = new DataModelBuilder() // The new numeric parameters are seeded with the shipped defaults so a migrated project renders the // same run a fresh one would — a parameter left undefined here would reach the CLI as its argparse // default, which is the same number arrived at without anyone choosing it. - .migrate( + .migrate( "v3", ({ dominanceThreshold: _d, offtargetProperty: _p, offtargetValues: _v, ...rest }) => ({ ...rest, @@ -341,6 +331,23 @@ const dataModel = new DataModelBuilder() panelMismatchTableState: createPlDataTableStateV2(), }), ) + // v3 -> v4: the flat verdict table and the quality-report tables are gone as VIEWS, and the punchcard + // takes their place. The three grid states go with them rather than being carried: a saved column set or + // filter is meaningful only against the frame it was saved on, and none of these three frames is on + // screen any more. The punchcard's own state starts fresh, and its identity picker starts empty — the + // page fills the first columns itself rather than a migration guessing which antigens matter. + // + // What the removed pages showed is still EMITTED: the verdicts and the run's measurements are both + // artifacts `verdict-block-interface` obliges this block to produce, and dropping a view does not + // release it from producing them. + .migrate( + "v4", + ({ verdictTableState: _v, antigenQcTableState: _q, panelMismatchTableState: _m, ...rest }) => ({ + ...rest, + punchcardTableState: createPlDataTableStateV2(), + punchcardIdentities: [], + }), + ) .init(() => ({ runMode: "full" as const, // full run by default; "dry" = read-limited Preview // Default preset = the geometry the block shipped with: 10x 5' v2 BEAM (16 / 10 / 15). @@ -357,9 +364,8 @@ const dataModel = new DataModelBuilder() highReferenceLine: DEFAULT_HIGH_REFERENCE_LINE, tableState: createPlDataTableStateV2(), qcSummaryTableState: createPlDataTableStateV2(), - verdictTableState: createPlDataTableStateV2(), - antigenQcTableState: createPlDataTableStateV2(), - panelMismatchTableState: createPlDataTableStateV2(), + punchcardTableState: createPlDataTableStateV2(), + punchcardIdentities: [], })); export const platforma = BlockModelV3.create(dataModel) @@ -958,86 +964,63 @@ export const platforma = BlockModelV3.create(dataModel) }, { retentive: true, withStatus: true }, ) - // The verdict view: one row per (clonotype set, antigen identity), carrying the four-state verdict and - // the support behind it. Assembled from the exported verdict frame, which the workflow also surfaces as - // an output because a block's own exports are not in its own result pool. + // Every combined identity the punchcard could show, in the order the workflow gave them. Options for the + // picker, and the source of the page's own fallback: with nothing picked the page opens on the first few + // of these rather than on an empty grid. // - // The identity LABEL column travels in the columns list rather than being discovered: createPlDataTable - // looks for label columns in the result pool, and this block's are not there. It is what puts the - // antigen's readable name in the row — emit_verdicts.py writes the panel's feature name, the tag itself - // where the panel names none, and "name (tag)" where two tags would otherwise share one label. + // Read from the pivot's own columns rather than from the run record's identity list, because the two can + // disagree in exactly one way that matters — the pivot is size-gated upstream, so a run over a large + // panel names its identities in the record and emits no columns at all. Reading the columns means the + // picker offers what the punchcard can actually draw. + .retentiveOutput("punchcardIdentityOptions", (ctx): { value: string; label: string }[] => { + const pCols = ctx.outputs + ?.resolve({ field: "antigenPunchcardTable", allowPermanentAbsence: true }) + ?.getPColumns(); + if (pCols === undefined) return []; + const seen = new Set(); + const options: { value: string; label: string }[] = []; + for (const c of pCols) { + if (c.spec.name !== IDENTITY_PUNCH_COLUMN) continue; + const identity = c.spec.domain?.[IDENTITY_ID_DOMAIN]; + if (identity === undefined || seen.has(identity)) continue; + seen.add(identity); + options.push({ value: identity, label: identity }); + } + return options; + }) + // The punchcard: one row per clonotype set, one column per picked identity, each cell carrying the + // four-state verdict and the count of cells that answered it. The pivoted shape comes from the workflow + // because a table cannot pivot a (set, identity) frame into columns. + // + // Only the picked identities' columns are passed. Every identity is in the frame either way, so the + // picker costs a re-render rather than a run — and a panel of a thousand antigens would otherwise render + // a thousand columns nobody asked for. // // createPlDataTableV2 rather than V3 for the same reason as perCellTable above: V3's discovery walks the // whole result pool and hangs on the upstream Samples&Data File dataset. V2 takes the columns as given. - // Filtering, ordering and default visibility all come from the specs the workflow built — the four - // states and wasCompeted carry pl7.app/isDiscreteFilter, and no column carries an orderable annotation, - // which column-specs.lib.tengo enforces rather than assumes. .output( - "verdictTable", + "punchcardTable", (ctx) => { const pCols = ctx.outputs - ?.resolve({ field: "antigenVerdictsTable", allowPermanentAbsence: true }) + ?.resolve({ field: "antigenPunchcardTable", allowPermanentAbsence: true }) ?.getPColumns(); if (pCols === undefined) return undefined; + const picked = new Set(ctx.data.punchcardIdentities); const cols = pCols.filter((c) => { - const axes = c.spec.axesSpec.map((a) => a.name); - if (c.spec.name === "pl7.app/label") return axes.length === 1 && axes[0] === IDENTITY_AXIS; - return axes.length === 2 && axes[0] === CLONOTYPE_SET_AXIS && axes[1] === IDENTITY_AXIS; + const identity = c.spec.domain?.[IDENTITY_ID_DOMAIN]; + return identity !== undefined && picked.has(identity); }); if (cols.length === 0) return undefined; - return createPlDataTableV2(ctx, cols, ctx.data.verdictTableState); - }, - { retentive: true, withStatus: true }, - ) - // The run's own quality report: one row per (level, panel, measured thing, measurement), carrying the - // measurement's status, the coverage triple beside it and — where nothing computed the measurement — the - // reason it was deferred. Every declared measurement keeps its row whether or not this run could compute - // it, so the frame is complete by construction and the view needs no handling for a measurement that is - // simply absent. - // - // The panel is part of the KEY here, not a value beside it: the same reagent stained into several samples - // writes one row per panel at the same (level, measured thing, measurement), and rows sharing an axis key - // are silently collapsed on import to whichever one survives. - // - // createPlDataTableV2 rather than V3 for the reason recorded on perCellTable above: V3's discovery walks - // the entire result pool and hangs on the upstream Samples & Data FASTQ dataset. Every column is keyed on - // this frame's own four axes, so no filtering by axis shape is needed — unlike verdictTable, which reads a - // frame holding several tables at once. - .output( - "antigenQcTable", - (ctx) => { - const pCols = ctx.outputs - ?.resolve({ field: "antigenQcTable", allowPermanentAbsence: true }) - ?.getPColumns(); - if (pCols === undefined || pCols.length === 0) return undefined; - return createPlDataTableV2( - ctx, - [...pCols, ...panelLabelColumns(ctx)], - ctx.data.antigenQcTableState, - ); - }, - { retentive: true, withStatus: true }, - ) - // The panel-versus-reads check: one row per (panel, tag), with the direction of the mismatch and the - // samples that reported it. Keyed on the panel rather than on the sample because a declared tag the reads - // never carried is a property of the declared tag set, not of any one sample; the samples travel in the - // row so nothing about where it was seen is lost. Both directions live in the one frame under the - // direction column, which is what lets a single table show them both. - .output( - "antigenPanelMismatchTable", - (ctx) => { - const pCols = ctx.outputs - ?.resolve({ field: "antigenPanelMismatchTable", allowPermanentAbsence: true }) - ?.getPColumns(); - if (pCols === undefined || pCols.length === 0) return undefined; - return createPlDataTableV2( - ctx, - [...pCols, ...panelLabelColumns(ctx)], - ctx.data.panelMismatchTableState, - ); + return createPlDataTableV2(ctx, cols, ctx.data.punchcardTableState); }, { retentive: true, withStatus: true }, ) + // The run's quality report and the panel-versus-reads check have no model output. Both are still emitted + // by the workflow — `verdict-block-interface` names the run-level measurements as one of this block's two + // artifacts, and a removed view does not release it from producing them — but nothing in this block reads + // them any more, because the tables that did were removed as the wrong way to deliver the quality + // end-goal. Whether they should instead cross the boundary as exports is an open question, not something + // to settle by leaving an unread output behind. // What the reading was actually answered under. The page states the comparator that SERVED rather than // the one that was requested, because the software degrades a request it cannot honour and a reader // meeting an all-unreliable table otherwise has no way to learn that happened. Absent until a run with a @@ -1125,11 +1108,7 @@ export const platforma = BlockModelV3.create(dataModel) // Shown for every run, including one with no V(D)J dataset. That run produces no antigen // columns at all, and the page saying so is the only place a user learns why — hiding the // tab would leave the absence unexplained. - { type: "link" as const, href: "/verdicts" as const, label: "Binding verdicts" }, - // Shown on the same terms as the verdict tab, for the same reason: a run with no V(D)J - // dataset produced neither the measurements nor the panel check, and the page saying so is - // the only place a user learns why. - { type: "link" as const, href: "/checks" as const, label: "Quality checks" }, + { type: "link" as const, href: "/punchcard" as const, label: "Punchcard" }, ] : []), ]; diff --git a/model/src/types.ts b/model/src/types.ts index 43e50ea..00567c6 100644 --- a/model/src/types.ts +++ b/model/src/types.ts @@ -142,15 +142,17 @@ export type BlockData = { * data it feeds — a write-on-read loop, and a write race between two open clients. */ contendingGroups?: string[][]; - verdictTableState: PlDataTableStateV2; // verdict grid state (UI-only, never projected to args) + punchcardTableState: PlDataTableStateV2; // punchcard grid state (UI-only, never projected to args) /** - * Grid state for the two halves of the run's own report — the quality measurements and the - * panel-versus-reads check. Separate states because they are separate frames on separate keys: the - * measurements are keyed (level, panel, measured thing, measurement) and the check is keyed - * (panel, tag), so a column set or filter saved for one means nothing in the other. UI-only. + * The identities whose columns the punchcard shows, picked one at a time. A view filter and nothing + * more: every identity is already in the pivoted frame, so adding one costs a column render rather + * than a run, and an empty list means "none picked yet" rather than "all". + * + * Written on a user gesture only. The choices come from the punchcardIdentityOptions output, and a + * watcher copying that output into data would make the output depend on the data feeding it — a + * write-on-read loop, and a write race between two open clients. */ - antigenQcTableState: PlDataTableStateV2; - panelMismatchTableState: PlDataTableStateV2; + punchcardIdentities: string[]; presetId?: string; pattern?: string; diff --git a/software/per-cell-metrics/src/emit_verdicts.py b/software/per-cell-metrics/src/emit_verdicts.py index c0c9b57..725f110 100644 --- a/software/per-cell-metrics/src/emit_verdicts.py +++ b/software/per-cell-metrics/src/emit_verdicts.py @@ -306,20 +306,51 @@ def _cells_by_set(linker: pl.DataFrame) -> dict[str, list[CellKey]]: return {set_id: sorted(keys) for set_id, keys in sorted(members.items())} -def _pivot_identity_summary(verdicts: pl.DataFrame, universe: set[str]) -> tuple[pl.DataFrame, bool]: - """The per-set verdict row, one column per identity. +def _pivot_identity_summary(verdicts: pl.DataFrame, universe: set[str]) -> tuple[pl.DataFrame, pl.DataFrame, bool]: + """The per-set verdict row and its support, one column per identity in each. Pivoted onto the set axis alone because a column carrying an axis the clonotype anchor does not have is dropped with no error by the block that consumes this, so a `(set, identity)` column is invisible there. Gated on identity count: the pivot costs a column per identity and a large panel would turn one artifact into a thousand. + + The second frame is the punchcard's, and its cell carries the state AND the + support in one value, `state|answered|couldAnswer`. Two reasons it is one + column rather than three: + + `support-travels-with-the-reading` obliges both counts to travel with a + verdict *wherever it appears*, and its reason is about the page rather than + the artifact — a reading resting on three cells must not look like one + resting on forty. A punchcard drawn from the state pivot alone would be + exactly that, so the support has to reach the same cell. + + And it cannot reach it as sibling columns. A column name here IS an antigen + name from a customer's panel file, so any suffix marking a support column is + a name some panel is entitled to use; and a grid pairs a cell to another + column's cell only by position, which no import guarantees. One value per + identity removes both problems, and keeps the pivot one column wide per + identity so the size gate above still means what it says. + + The state pivot is left exactly as it was, because lead selection reads it + and a compound value would not filter. """ if len(universe) > IDENTITY_SUMMARY_MAX_IDENTITIES or verdicts.height == 0: sets = verdicts.select("setId").unique() if verdicts.height else pl.DataFrame(schema={"setId": pl.String}) - return sets, False - wide = verdicts.pivot(on="identity", index="setId", values="state") - return wide.select(["setId", *sorted(universe)]), True + return sets, sets, False + ordered = ["setId", *sorted(universe)] + states = verdicts.pivot(on="identity", index="setId", values="state").select(ordered) + punch = verdicts.with_columns( + pl.concat_str( + [ + pl.col("state"), + pl.col("cellsAnswered").cast(pl.String), + pl.col("cellsCouldAnswer").cast(pl.String), + ], + separator="|", + ).alias("punch") + ).pivot(on="identity", index="setId", values="punch") + return states, punch.select(ordered), True def _leaf(level, entity, measurement, value, detail, panel_id, status: Status) -> QcRow: @@ -621,8 +652,9 @@ def main() -> None: _write_sorted(verdicts, f"{prefix}_verdicts.csv", ["setId", "identity"]) _write_sorted(set_counts(verdicts), f"{prefix}_set_counts.csv", ["setId"]) - summary, summary_emitted = _pivot_identity_summary(verdicts, universe) + summary, punch, summary_emitted = _pivot_identity_summary(verdicts, universe) _write_sorted(summary, f"{prefix}_identity_summary.csv", ["setId"]) + _write_sorted(punch, f"{prefix}_identity_punch.csv", ["setId"]) # The sparse per-tag counts and the per-cell scalars together carry every # per-cell state, at a small fraction of the size a per-cell-per-identity diff --git a/software/per-cell-metrics/test/test_emit_verdicts.py b/software/per-cell-metrics/test/test_emit_verdicts.py index d785e4e..886ad6c 100644 --- a/software/per-cell-metrics/test/test_emit_verdicts.py +++ b/software/per-cell-metrics/test/test_emit_verdicts.py @@ -1163,3 +1163,79 @@ def test_a_role_value_differing_only_in_case_is_not_matched(wide_bed): def _states_prefix(bed, prefix): v = pl.read_csv(bed / f"{prefix}_verdicts.csv", infer_schema_length=0) return {(r["setId"], r["identity"]) for r in v.iter_rows(named=True)} + + +# --- the punchcard's pivot --------------------------------------------------------- +# +# All of these run against the COMMITTED bed rather than the small inline one, and that is +# load-bearing. On the inline bed every row has cellsAnswered == cellsCouldAnswer and the +# panel yields a single identity, so swapping the two counts and shuffling the column +# order are both invisible: mutating either passed the first version of these tests. The +# committed bed carries several identities, readings whose support is short of what could +# have answered, and *never asked* positions where couldAnswer is zero. +# +# Verified by mutation: swapping the two counts, dropping the state from the value, and +# changing the separator are each caught. Dropping the `select(ordered)` that aligns the +# punch pivot with the state pivot is NOT caught and cannot be here — polars pivots columns +# in order of first appearance, which on this bed already equals sorted order. That +# alignment is enforced by construction rather than observed by a test. + + +def _punch_bed(bed): + r = _run(bed, *_bed_args("panel_with_reference.csv")) + assert r.returncode == 0, r.stderr + return ( + pl.read_csv(bed / "result_verdicts.csv", infer_schema_length=0), + pl.read_csv(bed / "result_identity_punch.csv", infer_schema_length=0), + ) + + +def test_punch_bed_can_tell_the_two_counts_apart(wide_bed): + # The guard on the tests below. If every row answered exactly as many cells as could + # have, swapping the two counts is undetectable and the agreement test below passes + # while the punch draws the wrong size everywhere. + verdicts, punch = _punch_bed(wide_bed) + differing = verdicts.filter(pl.col("cellsAnswered") != pl.col("cellsCouldAnswer")) + assert differing.height > 0, "bed no longer distinguishes answered from could-answer" + assert len([c for c in punch.columns if c != "setId"]) > 1, "bed no longer has several identities" + + +def test_punch_pivot_agrees_with_the_long_verdicts(wide_bed): + # The punchcard's cell is the only place its three facts meet, so this is the one check + # that they are the SAME three facts the long frame carries. A pivot that dropped a + # field, swapped the counts, or paired a state with another identity's numbers would + # still write a well-formed file. + verdicts, punch = _punch_bed(wide_bed) + identities = sorted(set(verdicts["identity"].to_list())) + assert punch.columns == ["setId", *identities] + + expected = { + (r["setId"], r["identity"]): f"{r['state']}|{r['cellsAnswered']}|{r['cellsCouldAnswer']}" + for r in verdicts.iter_rows(named=True) + } + for row in punch.iter_rows(named=True): + for identity in identities: + assert row[identity] == expected[(row["setId"], identity)], (row["setId"], identity) + + +def test_punch_pivot_keys_and_order_match_the_state_pivot(wide_bed): + # Both pivots are gated together and ordered together: the punchcard reads one and lead + # selection reads the other, and a reader comparing them must not meet a set or an + # identity present in one and absent from the other -- or in a different column order, + # which is what makes the two frames comparable side by side at all. + _punch_bed(wide_bed) + states = pl.read_csv(wide_bed / "result_identity_summary.csv", infer_schema_length=0) + punch = pl.read_csv(wide_bed / "result_identity_punch.csv", infer_schema_length=0) + assert states.columns == punch.columns + assert states["setId"].to_list() == punch["setId"].to_list() + + +def test_punch_state_is_the_state_the_long_frame_gives(wide_bed): + # The state is the half of the cell that carries the answer, so it is asserted on its + # own: a punch whose counts are right and whose state is another identity's would still + # draw a glyph, in the wrong colour, with nothing to catch it. + verdicts, punch = _punch_bed(wide_bed) + by_key = {(r["setId"], r["identity"]): r["state"] for r in verdicts.iter_rows(named=True)} + for row in punch.iter_rows(named=True): + for identity in [c for c in punch.columns if c != "setId"]: + assert row[identity].split("|")[0] == by_key[(row["setId"], identity)] diff --git a/test/src/wf.test.ts b/test/src/wf.test.ts index d8b0b5e..6faf20f 100644 --- a/test/src/wf.test.ts +++ b/test/src/wf.test.ts @@ -57,16 +57,15 @@ blockTest("empty inputs", { timeout: 20000 }, async ({ rawPrj: project, expect } // Every output a page reads must RESOLVE on a freshly added block, before anything has run. An // output that throws here is not a failed computation — it breaks the page that reads it at the - // moment the block is created, which is the first thing a user sees. This block gained five table - // outputs and a run-record output on the verdict branch, each guarded by its own - // undefined-until-computed path, so the guards are what this asserts. Values are deliberately not - // asserted: `ok` with an undefined value is the correct empty-state answer for all of them. + // moment the block is created, which is the first thing a user sees. Each verdict-branch output is + // guarded by its own undefined-until-computed path, so the guards are what this asserts. Values are + // deliberately not asserted: `ok` with an undefined value is the correct empty-state answer for all of + // them — including punchcardIdentityOptions, which answers with an empty list before any run. const mustResolve = [ "perCellTable", "qcSummaryTable", - "verdictTable", - "antigenQcTable", - "antigenPanelMismatchTable", + "punchcardTable", + "punchcardIdentityOptions", "verdictRunMeta", "isRunning", "started", @@ -177,9 +176,8 @@ blockTest.skip( highReferenceLine: 100, tableState: createPlDataTableStateV2(), qcSummaryTableState: createPlDataTableStateV2(), - verdictTableState: createPlDataTableStateV2(), - antigenQcTableState: createPlDataTableStateV2(), - panelMismatchTableState: createPlDataTableStateV2(), + punchcardTableState: createPlDataTableStateV2(), + punchcardIdentities: [], } satisfies BlockData, }); diff --git a/ui/src/app.ts b/ui/src/app.ts index aab38de..c9697f5 100644 --- a/ui/src/app.ts +++ b/ui/src/app.ts @@ -3,9 +3,8 @@ import { defineAppV3 } from "@platforma-sdk/ui-vue"; import { watchEffect } from "vue"; import MainPage from "./pages/MainPage.vue"; import QcSummaryPage from "./pages/QcSummaryPage.vue"; -import QualityChecksPage from "./pages/QualityChecksPage.vue"; +import PunchcardPage from "./pages/PunchcardPage.vue"; import ResultsPage from "./pages/ResultsPage.vue"; -import VerdictsPage from "./pages/VerdictsPage.vue"; export const sdkPlugin = defineAppV3(platforma, (app) => { // Block-label pattern: mirror the model's suggestedBlockLabel (" / → ") @@ -32,8 +31,7 @@ export const sdkPlugin = defineAppV3(platforma, (app) => { "/": () => MainPage, "/qc": () => QcSummaryPage, "/results": () => ResultsPage, - "/verdicts": () => VerdictsPage, - "/checks": () => QualityChecksPage, + "/punchcard": () => PunchcardPage, }, }; }); diff --git a/ui/src/components/PunchCell.vue b/ui/src/components/PunchCell.vue new file mode 100644 index 0000000..0794756 --- /dev/null +++ b/ui/src/components/PunchCell.vue @@ -0,0 +1,124 @@ + + + + + diff --git a/ui/src/pages/PunchcardPage.vue b/ui/src/pages/PunchcardPage.vue new file mode 100644 index 0000000..8f47533 --- /dev/null +++ b/ui/src/pages/PunchcardPage.vue @@ -0,0 +1,137 @@ + + + diff --git a/ui/src/pages/QualityChecksPage.vue b/ui/src/pages/QualityChecksPage.vue deleted file mode 100644 index 9d4dfaf..0000000 --- a/ui/src/pages/QualityChecksPage.vue +++ /dev/null @@ -1,127 +0,0 @@ - - - diff --git a/ui/src/pages/VerdictsPage.vue b/ui/src/pages/VerdictsPage.vue deleted file mode 100644 index 462409b..0000000 --- a/ui/src/pages/VerdictsPage.vue +++ /dev/null @@ -1,100 +0,0 @@ - - - diff --git a/workflow/src/column-specs.lib.tengo b/workflow/src/column-specs.lib.tengo index 9362772..b40a366 100644 --- a/workflow/src/column-specs.lib.tengo +++ b/workflow/src/column-specs.lib.tengo @@ -665,24 +665,27 @@ setCountsImportSpec := func(setAxisSpec, served) { // The label is the identity string itself. The readable name lives in result_identity_labels.csv, // which is imported as a column rather than read as a value, so it is not available while these // specs are built. -identitySummaryImportSpec := func(setAxisSpec, identities, groupingId, served) { +// Both identity pivots — the state and the support behind it — are keyed the same way, carry the same +// column order and the same labels, and differ only in what a column holds. One builder produces both so +// the pair cannot drift: the punchcard pairs a state with its support by identity, and a divergence in +// naming or ordering here would pair a state with a different identity's support with nothing to signal it. +// The id prefix differs between the two families because both frames name their columns after the same +// identities, and the punchcard frame holds BOTH at once: one prefix would give a state column and its +// support column the same id inside one frame, where one of the pair silently wins. +identityPivotImportSpec := func(setAxisSpec, identities, groupingId, served, colName, valueType, basePriority, idPrefix, extra) { cols := [] for i, identity in identities { cols = append(cols, { column: identity, - id: "identity_" + strings.substituteSpecialCharacters(identity), + id: idPrefix + strings.substituteSpecialCharacters(identity), spec: { - name: "pl7.app/antigen/identityVerdict", - valueType: "String", + name: colName, + valueType: valueType, domain: maps.merge(servedDomain(served), { "pl7.app/antigen/identityId": identity, "pl7.app/antigen/groupingId": groupingId }), - annotations: a(92000 - i, false, { - "pl7.app/label": identity, - "pl7.app/isDiscreteFilter": "true", - "pl7.app/discreteValues": VERDICT_STATES - }) + annotations: a(basePriority - i, false, maps.merge({ "pl7.app/label": identity }, extra)) } }) } @@ -694,6 +697,30 @@ identitySummaryImportSpec := func(setAxisSpec, identities, groupingId, served) { } } +// --- The punchcard's pivot: result_identity_punch.csv, keyed (setId) -------------------- +// +// One column per identity whose value carries the state AND the support behind it, `state|answered| +// couldAnswer`. The UI reads all three from the one cell: the state picks the punch's colour, the two +// counts its size and its tooltip. +// +// Compound rather than three columns because `support-travels-with-the-reading` requires both counts to +// travel with the verdict wherever it appears — a punch showing state alone would make a reading resting +// on three cells look like one resting on forty — and a grid can only pair a cell with another column's +// cell by position, which no import guarantees. Carrying it as an opaque String is also why this family is +// kept out of the exported frame: lead selection filters the state pivot, and a compound value would not. +identityPunchImportSpec := func(setAxisSpec, identities, groupingId, served) { + return identityPivotImportSpec(setAxisSpec, identities, groupingId, served, + "pl7.app/antigen/identityPunch", "String", 92000, "identityPunch_", {}) +} + +identitySummaryImportSpec := func(setAxisSpec, identities, groupingId, served) { + return identityPivotImportSpec(setAxisSpec, identities, groupingId, served, + "pl7.app/antigen/identityVerdict", "String", 92000, "identity_", { + "pl7.app/isDiscreteFilter": "true", + "pl7.app/discreteValues": VERDICT_STATES + }) +} + // --- Re-derivation material ------------------------------------------------------------- // // The block emits no dense per-cell-per-identity table: on a realistic run it is the largest artifact @@ -1048,6 +1075,7 @@ export { verdictsImportSpec: verdictsImportSpec, setCountsImportSpec: setCountsImportSpec, identitySummaryImportSpec: identitySummaryImportSpec, + identityPunchImportSpec: identityPunchImportSpec, cellTagCountsImportSpec: cellTagCountsImportSpec, cellScalarsImportSpec: cellScalarsImportSpec, offeredImportSpec: offeredImportSpec, diff --git a/workflow/src/main.tpl.tengo b/workflow/src/main.tpl.tengo index bbc4548..5ee5e57 100644 --- a/workflow/src/main.tpl.tengo +++ b/workflow/src/main.tpl.tengo @@ -439,6 +439,7 @@ wf.body(func(args) { verdicts: verdictRun.output("verdicts"), setCounts: verdictRun.output("setCounts"), identitySummary: verdictRun.output("identitySummary"), + identityPunch: verdictRun.output("identityPunch"), cellCounts: verdictRun.output("cellCounts"), cellScalars: verdictRun.output("cellScalars"), offered: verdictRun.output("offered"), @@ -513,6 +514,10 @@ wf.body(func(args) { // this the block that produced the verdicts is the one place that cannot show them. The model reads // it, keeps the columns keyed (clonotype set, identity) plus the identity label, and drops the rest. blockOutputs.antigenVerdictsTable = pframes.exportFrame(verdictImport.output("antigenVerdicts")) + // The punchcard's source: the same verdicts pivoted onto the clonotype set, one column per identity + // for the state and one for the support behind it. A table cannot pivot a (set, identity) frame into + // columns, so the shape the punchcard needs is made here rather than in the model. + blockOutputs.antigenPunchcardTable = pframes.exportFrame(verdictImport.output("punchcard")) // The run's own report. Outputs rather than exports: these are read by this block's model and UI. blockOutputs.antigenQcTable = pframes.exportFrame(verdictImport.output("qcTable")) blockOutputs.antigenPanelMismatchTable = pframes.exportFrame(verdictImport.output("panelMismatchTable")) diff --git a/workflow/src/verdict-import.tpl.tengo b/workflow/src/verdict-import.tpl.tengo index a532715..841f433 100644 --- a/workflow/src/verdict-import.tpl.tengo +++ b/workflow/src/verdict-import.tpl.tengo @@ -24,7 +24,8 @@ verdictLinker := import(":verdict-linker") json := import("json") -// Returns three frames: antigenVerdicts (exported), qcTable and panelMismatchTable (this block's own). +// Returns five frames: antigenVerdicts (exported), and punchcard, cellTable, qcTable and +// panelMismatchTable (this block's own). // defineOutputs is deliberately absent — an ephemeral template takes its output names from what the body // returns and never checks a declared list, so declaring one here would be inert. self.awaitState("columns", "PColumnBundle") @@ -104,6 +105,22 @@ self.body(func(inputs) { columnSpecs.identitySummaryImportSpec(setAxis, summaryIdentities, served.groupingId, served)) } + // --- the punchcard's own frame, as an OUTPUT -------------------------------------------------------- + // + // The punchcard is the view `block-set` calls this block's own — every clonotype against every identity, + // each position in one of the four states. It needs the pivoted shape, because a table cannot pivot a + // (set, identity) frame into columns, and it needs it as an output rather than an export: a block's own + // exports are not in its own result pool. + // + // Its cell carries the state and the two support counts together, so no glyph on that page states a + // verdict without what it rests on. Gated on the same identity count as the summary above — either both + // pivots are present or neither is, and the page says so rather than rendering an empty grid. + punchFb := pframes.pFrameBuilder() + if len(summaryIdentities) > 0 { + addTo(punchFb, "identityPunch", inputs.identityPunch, + columnSpecs.identityPunchImportSpec(setAxis, summaryIdentities, served.groupingId, served)) + } + // The per-cell material stays INSIDE the block: outputs, never exports. Reading the same experiment // under another grouping is another execution of this block, not a re-derivation performed by a // reader, so nothing downstream needs the per-cell per-tag states — and nothing downstream ever asked @@ -157,6 +174,7 @@ self.body(func(inputs) { return { antigenVerdicts: exportFb.build(), + punchcard: punchFb.build(), cellTable: cellFb.build(), qcTable: qcFb.build(), panelMismatchTable: mismatchFb.build() diff --git a/workflow/src/verdict-run.tpl.tengo b/workflow/src/verdict-run.tpl.tengo index 4669e02..ec025c4 100644 --- a/workflow/src/verdict-run.tpl.tengo +++ b/workflow/src/verdict-run.tpl.tengo @@ -31,6 +31,7 @@ RESULT_TABLES := [ { out: "verdicts", file: "result_verdicts.csv" }, { out: "setCounts", file: "result_set_counts.csv" }, { out: "identitySummary", file: "result_identity_summary.csv" }, + { out: "identityPunch", file: "result_identity_punch.csv" }, { out: "cellCounts", file: "result_cell_counts.csv" }, { out: "cellScalars", file: "result_cell_scalars.csv" }, { out: "offered", file: "result_offered.csv" }, From 1a3535e6cea9f3a3b5862c653a1511d86c5b65d2 Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Tue, 18 Aug 2026 16:52:48 +0200 Subject: [PATCH 072/282] MILAB-6496: pair each cell's admissibility with its own row cell_scalars built its admissibility column as a positional pl.Series in analysed_cells order and attached it AFTER two joins on reference_frame. Polars does not promise a left frame's row order survives a join (maintain_order defaults to "none"), so a reordering join hands cells each other's labels, and _write_sorted then sorts the file, which hides the swap rather than repairing it. Nothing downstream can tell: the column is present, its values are the right multiset, and only the pairing is wrong. Admissibility is now computed in the same tuple as its own cell, when reference_frame is constructed, so the pairing is a property of construction rather than of whatever polars did in between. The test asserts the mapping keyed by cell, over a bed carrying three distinct labels at once - admissible, thin comparator, and gated. Fewer labels cannot distinguish a correct pairing from a permutation of it. Verified by mutation: reversing the positional attach fails the test, and no assertion on the column's presence or on its value multiset does. NO_COMPARATOR is deliberately absent from that bed. With a declared comparator, reference_by_cell zero-fills every analysed cell it read nothing for, so a cell missing its comparator row reads THIN, not NO_COMPARATOR - which is the documented contract, and was my first wrong expectation of it. --- .../per-cell-metrics/src/emit_verdicts.py | 32 ++++++++------ .../test/test_emit_verdicts.py | 43 +++++++++++++++++++ 2 files changed, 61 insertions(+), 14 deletions(-) diff --git a/software/per-cell-metrics/src/emit_verdicts.py b/software/per-cell-metrics/src/emit_verdicts.py index 725f110..e1b3704 100644 --- a/software/per-cell-metrics/src/emit_verdicts.py +++ b/software/per-cell-metrics/src/emit_verdicts.py @@ -671,10 +671,26 @@ def main() -> None: orient="row", schema={"sampleId": pl.String, "cellId": pl.String, "inCellList": pl.String}, ) + + def _admissibility(key: CellKey) -> str: + reason = _cell_admissibility_reason(key, admissibility) + return "admissible" if reason is None else reason.value + + # Admissibility is built HERE, in the same row as its own cell, and not attached to a later frame as a + # positional column. Polars does not promise a left frame's row order survives a join + # (`maintain_order` defaults to "none"), so a positional attach after the joins below can give cells + # each other's labels -- and `_write_sorted` then sorts the file, which hides it rather than repairing + # it. Carrying the value in the tuple makes the pairing a property of construction instead of a + # property of whatever polars did in between. reference_frame = pl.DataFrame( - [(s, c, reference.by_cell.get((s, c))) for s, c in analysed_cells], + [(s, c, reference.by_cell.get((s, c)), _admissibility((s, c))) for s, c in analysed_cells], orient="row", - schema={"sampleId": pl.String, "cellId": pl.String, "referenceCount": pl.Int64}, + schema={ + "sampleId": pl.String, + "cellId": pl.String, + "referenceCount": pl.Int64, + "admissibility": pl.String, + }, ) cell_counts = ( non_reference.join(reference_frame, on=["sampleId", "cellId"], how="left") @@ -687,18 +703,6 @@ def main() -> None: cell_scalars = ( reference_frame.join(in_list, on=["sampleId", "cellId"], how="left") .with_columns(pl.col("inCellList").fill_null(unlisted_reads)) - .with_columns( - pl.Series( - "admissibility", - [ - (lambda reason: "admissible" if reason is None else reason.value)( - _cell_admissibility_reason(key, admissibility) - ) - for key in analysed_cells - ], - dtype=pl.String, - ) - ) .select(["sampleId", "cellId", "referenceCount", "admissibility", "inCellList"]) ) _write_sorted(cell_scalars, f"{prefix}_cell_scalars.csv", ["sampleId", "cellId"]) diff --git a/software/per-cell-metrics/test/test_emit_verdicts.py b/software/per-cell-metrics/test/test_emit_verdicts.py index 886ad6c..02bb92b 100644 --- a/software/per-cell-metrics/test/test_emit_verdicts.py +++ b/software/per-cell-metrics/test/test_emit_verdicts.py @@ -1239,3 +1239,46 @@ def test_punch_state_is_the_state_the_long_frame_gives(wide_bed): for row in punch.iter_rows(named=True): for identity in [c for c in punch.columns if c != "setId"]: assert row[identity].split("|")[0] == by_key[(row["setId"], identity)] + + +def test_cell_scalars_pairs_each_cell_with_its_own_admissibility(tmp_path): + """Every cell's admissibility must be ITS OWN, not the row next to it. + + The frame this comes from is built in one order and then joined twice before + the admissibility column is attached. Polars does not promise a left frame's + row order survives a join (`maintain_order` defaults to "none"), so a + positional attach can hand cells each other's labels -- and because the file + is sorted on write, nothing downstream can tell. The keyed assertion below is + what makes the pairing observable at all: asserting the column's PRESENCE, or + the multiset of its values, passes just as happily when every label has moved + one row down. + + Three distinct labels appear on purpose. A bed where every cell reads the + same label cannot tell a correct pairing from any permutation of it, and two + labels only catch permutations that cross the boundary between them. + + `no comparator for this cell` is deliberately NOT among them: with a declared + comparator, `reference_by_cell` zero-fills every analysed cell it read + nothing for (verdict.py:212-221), so a cell missing its comparator row reads + THIN rather than NO_COMPARATOR. That reason needs a run with no comparator at + all, where it is the answer for every cell and so distinguishes nothing here. + """ + (tmp_path / "counts.csv").write_text( + "sampleId,cellId,tag,umiCount\n" + "S1,ok1,AAAA,500\nS1,ok1,CTRL,6\n" # comparable + "S1,ok2,AAAA,500\nS1,ok2,CTRL,6\n" # comparable + "S1,thin,AAAA,500\nS1,thin,CTRL,1\n" # comparator below the thin line of 2 + "S1,hi,AAAA,500\nS1,hi,CTRL,400\n" # comparator above the gate -> set aside + ) + (tmp_path / "panel.csv").write_text("Samples,Name,Sequence,Type\nS1,AgA,AAAA,Target\nS1,Ctrl,CTRL,Control\n") + (tmp_path / "linker.csv").write_text("sampleId,cellId,setId\nS1,ok1,K1\nS1,ok2,K1\nS1,thin,K2\nS1,hi,K2\n") + _run(tmp_path, *BASE, "--gate-threshold", "100") + + scalars = pl.read_csv(tmp_path / "result_cell_scalars.csv", infer_schema_length=0) + by_cell = {r["cellId"]: r["admissibility"] for r in scalars.iter_rows(named=True)} + assert by_cell == { + "ok1": "admissible", + "ok2": "admissible", + "thin": "the comparator rests on too little to compare against", + "hi": "cell set aside by the admissibility gate", + } From b11965585c08c57bb42b3b04b07e72744c1c519d Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Tue, 18 Aug 2026 16:54:53 +0200 Subject: [PATCH 073/282] MILAB-6496: stop emit_panel crashing on a ragged CSV row, and give it tests csv.DictReader fills a short row's missing columns with None rather than omitting them, so `row.get(tag_col, "")` returned None -- the default never fires because the key exists -- and .strip() raised AttributeError. One malformed line in a user-supplied tag->feature CSV took the whole run down with a traceback. emit_panel.py had no tests at all. The new file covers the three properties of its output that are contract rather than convenience -- deduplicated, sorted, one barcode per line, all asserted on exact bytes because mitool matches this file verbatim and the pure-template dedup rests on the bytes being stable -- plus the renamed column, both refusal paths, whitespace trimming, and the ragged row. The ragged-row test puts the barcode column SECOND, deliberately. A short row truncates only trailing columns, so with the barcode first the missing key is the other column and nothing reads it: the first version of the test passed against the unfixed code and proved nothing. --- software/per-cell-metrics/src/emit_panel.py | 6 +- .../per-cell-metrics/test/test_emit_panel.py | 99 +++++++++++++++++++ 2 files changed, 104 insertions(+), 1 deletion(-) create mode 100644 software/per-cell-metrics/test/test_emit_panel.py diff --git a/software/per-cell-metrics/src/emit_panel.py b/software/per-cell-metrics/src/emit_panel.py index bd9c553..5117842 100644 --- a/software/per-cell-metrics/src/emit_panel.py +++ b/software/per-cell-metrics/src/emit_panel.py @@ -25,7 +25,11 @@ def main() -> None: raise SystemExit( f"column {args.tag_col!r} not found in {args.tag_feature_csv} (columns: {reader.fieldnames})" ) - seqs = {row[args.tag_col].strip() for row in reader if row.get(args.tag_col, "").strip()} + # `or ""` rather than a get() default: a short row's missing columns are + # present-and-None in DictReader's output, not absent, so the default never + # fires and .strip() met None. One malformed line of a user-supplied CSV + # took the whole run down with a traceback. + seqs = {seq for seq in ((row.get(args.tag_col) or "").strip() for row in reader) if seq} if not seqs: raise SystemExit(f"no feature barcodes found in column {args.tag_col!r}") diff --git a/software/per-cell-metrics/test/test_emit_panel.py b/software/per-cell-metrics/test/test_emit_panel.py new file mode 100644 index 0000000..cfdcfae --- /dev/null +++ b/software/per-cell-metrics/test/test_emit_panel.py @@ -0,0 +1,99 @@ +"""Behavioral tests for emit_panel.py (Feature Integration software). + +Writes the panel's barcode column out as a plain one-per-line list for mitool's +refine-tags whitelist (`-t FEATURE#file:panel.txt`). Three properties of that +output are contract rather than convenience, and each has a test below: +deduplicated, sorted, and one barcode per line. mitool matches against this file +verbatim, and the workflow's pure-template dedup rests on the bytes being stable, +so an unsorted or duplicated file is not cosmetically wrong -- it changes a +resource handle and silently costs every downstream node its cache. + +Run through the CLI like the other tool tests: this is a subprocess entry point, +and a caller reaching past it into `main()` would not exercise the argparse and +SystemExit behavior that is most of what the file does. +""" + +import pathlib +import subprocess +import sys + +SRC = pathlib.Path(__file__).parents[1] / "src" / "emit_panel.py" + + +def _run(tmp_path, csv_text, *args, expect_failure=False): + """Run the tool over `csv_text`, returning the output file's text. + + Asserts success by default: a tool that exits non-zero has written nothing, + so a test that goes on to assert emptiness of the output would pass for the + wrong reason. + """ + src_csv = tmp_path / "tags.csv" + src_csv.write_text(csv_text) + out = tmp_path / "panel.txt" + r = subprocess.run( + [sys.executable, str(SRC), str(src_csv), str(out), *args], + capture_output=True, + text=True, + ) + if expect_failure: + assert r.returncode != 0, f"expected failure, got 0. stdout={r.stdout!r}" + return r.stderr + assert r.returncode == 0, f"exited {r.returncode}. stderr={r.stderr!r}" + return out.read_text() + + +def test_dedupes_sorts_and_writes_one_barcode_per_line(tmp_path): + # The whole contract in one assertion, on exact bytes: AAAA appears twice in + # the input under two different antigen names and must appear once here, and + # CCCC must follow it rather than lead. Asserting the set of lines instead + # would pass on an unsorted file, which is the case that breaks dedup + # downstream rather than anything a reader would notice. + text = _run(tmp_path, "tag,feature\nCCCC,x\nAAAA,y\nAAAA,z\n") + assert text == "AAAA\nCCCC\n" + + +def test_tag_col_selects_a_renamed_column(tmp_path): + # The panel file's barcode column is whatever the user pointed the block at, + # so the default name is a default and not an assumption. + text = _run(tmp_path, "Sequence,Name\nGGGG,x\nTTTT,y\n", "--tag-col", "Sequence") + assert text == "GGGG\nTTTT\n" + + +def test_missing_column_names_the_column_it_wanted(tmp_path): + stderr = _run(tmp_path, "sequence,feature\nAAAA,x\n", expect_failure=True) + assert "tag" in stderr + + +def test_header_only_input_is_refused(tmp_path): + # An empty whitelist is not an empty correction -- mitool given a whitelist + # of nothing corrects every barcode to nothing. Failing here is the point. + stderr = _run(tmp_path, "tag,feature\n", expect_failure=True) + assert "no feature barcodes" in stderr + + +def test_all_blank_tag_cells_are_refused(tmp_path): + stderr = _run(tmp_path, "tag,feature\n ,x\n,y\n", expect_failure=True) + assert "no feature barcodes" in stderr + + +def test_padded_cells_are_trimmed_and_blank_cells_skipped(tmp_path): + # The barcode is a join key against the counts, whose reader strips for the + # same reason: " AAAA " and "AAAA" are one barcode, and a whitelist carrying + # the padded form matches nothing. + text = _run(tmp_path, "tag,feature\n AAAA ,x\n,y\nCCCC,z\n") + assert text == "AAAA\nCCCC\n" + + +def test_a_ragged_short_row_is_skipped_rather_than_crashing(tmp_path): + # csv.DictReader fills a short row's missing keys with None, so the tag cell + # is present-and-None rather than absent. `row.get(col, "")` returns None + # there -- the default never fires, because the key exists -- and .strip() + # raised AttributeError, taking the whole run down on one malformed line of a + # user-supplied CSV. + # + # The barcode column is deliberately SECOND here. A short row only truncates + # the trailing columns, so with the barcode first the missing key is the other + # column and nothing reads it -- the first version of this test put it first, + # passed against the unfixed code, and proved nothing. + text = _run(tmp_path, "feature,tag\nx,AAAA\ny\nz,GGGG\n", "--tag-col", "tag") + assert text == "AAAA\nGGGG\n" From 78c99545c2fab8875fdd6162aeb87a41052cfdb3 Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Tue, 18 Aug 2026 16:57:03 +0200 Subject: [PATCH 074/282] MILAB-6496: a corrupt QC number must never read acceptable Every < and > comparison against NaN is False, so a NaN value fell through to `bad = False` and the measurement read ACCEPTABLE. One NaN among a tag's peers made np.quantile return NaN quartiles, with the same result. For QC code corrupt-input-reads-green is the worst available failure mode, because acceptable is the one status a reader will not investigate. Two guards, and they read differently on purpose: - a non-finite VALUE is treated exactly as an absent one, not evaluated. +inf read acceptable against an at-least line and -inf happened to alert; one rule for "not a finite number" is easier to defend than a rule whose answer depends on the sign, and neither is a measurement. - non-finite FENCES leave the comparison unjudged. There the value is a real number and the measurement was computed; what cannot be defended is the distribution it would be measured against, which is what unjudged says. Neither status enters a rollup, so a corrupt reading can no longer leave its panel or its capture looking clean. DECISION, flagged for veto: the review recommended unjudged for both. Absent values already map to not-evaluated, and a NaN is likewise no number arriving, so matching them keeps one meaning per status; the fence case stays unjudged because there the measurement did compute. --- software/per-cell-metrics/src/qc_measures.py | 20 +++++++++- .../per-cell-metrics/test/test_qc_measures.py | 38 +++++++++++++++++++ 2 files changed, 56 insertions(+), 2 deletions(-) diff --git a/software/per-cell-metrics/src/qc_measures.py b/software/per-cell-metrics/src/qc_measures.py index 8972a4d..b285783 100644 --- a/software/per-cell-metrics/src/qc_measures.py +++ b/software/per-cell-metrics/src/qc_measures.py @@ -24,6 +24,7 @@ from __future__ import annotations +import math from dataclasses import dataclass from enum import Enum @@ -279,7 +280,13 @@ def status_for(measurement: str, value: float | None, lines: dict[str, float]) - f"{measurement!r} is judged against the run itself, not against a line: " "call outlier_status(value, peers) with the measurement's peers in the same panel" ) - if measurement in _DEFERRED or value is None: + # A non-finite value is treated exactly as an absent one. Every `<` and `>` against + # NaN is False, so without this a NaN fell through to `bad = False` and the + # measurement read ACCEPTABLE -- corrupt input reading green, which is the one status + # a reader will not investigate. +inf read green too, against an at-least line; -inf + # happened to alert. One rule for "not a finite number" is easier to defend than a + # rule whose answer depends on the sign, and neither is a measurement. + if measurement in _DEFERRED or value is None or not math.isfinite(value): return Status.NOT_EVALUATED if measurement not in lines: return Status.UNJUDGED @@ -322,11 +329,20 @@ def outlier_status( Unjudged below `MIN_PEERS_TO_COMPARE` peers, where a quartile is not a distribution but an arithmetic accident of two or three numbers. """ - if value is None: + # Same rule as `status_for` for the value itself: not a finite number is not a + # measurement, and a NaN compared against any fence is False, which read ACCEPTABLE. + if value is None or not math.isfinite(value): return Status.NOT_EVALUATED if len(peers) < MIN_PEERS_TO_COMPARE: return Status.UNJUDGED q1, q3 = (float(q) for q in np.quantile(peers, [0.25, 0.75])) + # A non-finite fence is a different failure from a non-finite value, and reads + # differently. One NaN among the peers makes np.quantile return NaN quartiles, so + # every comparison went False and the tag read ACCEPTABLE. The value here is a real + # number and the measurement WAS computed -- what cannot be defended is the + # distribution it would be measured against, which is what unjudged says. + if not (math.isfinite(q1) and math.isfinite(q3)): + return Status.UNJUDGED return Status.ALERTING if value > q3 + (q3 - q1) * fence else Status.ACCEPTABLE diff --git a/software/per-cell-metrics/test/test_qc_measures.py b/software/per-cell-metrics/test/test_qc_measures.py index ca0ff61..965d6da 100644 --- a/software/per-cell-metrics/test/test_qc_measures.py +++ b/software/per-cell-metrics/test/test_qc_measures.py @@ -581,3 +581,41 @@ def test_the_minimum_peer_count_is_satisfied_at_the_named_value(): # to compare against, two is not. Nothing else pins this boundary. assert outlier_status(0.9, [0.01, 0.02, 0.03]) is not Status.UNJUDGED assert outlier_status(0.9, [0.01, 0.02]) is Status.UNJUDGED + + +# --- corrupt numbers must never read green ------------------------------------------- +# +# Every `<` and `>` comparison against NaN is False, so before this was fixed a NaN +# value fell through to `bad = False` and the measurement read ACCEPTABLE -- and a NaN +# among the peers made np.quantile return NaN fences, with the same result. For QC +# code, corrupt-input-reads-green is the worst available failure mode: it is the one +# state a reader will not investigate. + + +def test_a_nan_value_is_not_evaluated_rather_than_acceptable(): + assert status_for("readsPerCell", float("nan"), DEFAULT_LINES) is Status.NOT_EVALUATED + + +def test_infinite_values_are_not_evaluated_rather_than_judged(): + # +inf would have read ACCEPTABLE against an at-least line, which is the green + # reading again. -inf happens to alert, so only one direction was dangerous -- but + # neither is a measurement, and one rule for "not a finite number" is easier to + # defend than a rule that depends on the sign. + assert status_for("readsPerCell", float("inf"), DEFAULT_LINES) is Status.NOT_EVALUATED + assert status_for("readsPerCell", float("-inf"), DEFAULT_LINES) is Status.NOT_EVALUATED + + +def test_a_nan_value_is_not_evaluated_against_its_peers(): + assert outlier_status(float("nan"), [0.1, 0.2, 0.3, 0.4]) is Status.NOT_EVALUATED + + +def test_nan_among_the_peers_leaves_the_comparison_unjudged(): + # The value is a real number here; what cannot be defended is the distribution it + # would be measured against. That is unjudged, not not-evaluated -- the + # measurement was computed, and only the comparison is unavailable. + assert outlier_status(0.9, [0.1, float("nan"), 0.3, 0.4]) is Status.UNJUDGED + + +def test_a_clear_outlier_still_alerts_with_finite_peers(): + # The guard above must not swallow the case the measure exists for. + assert outlier_status(0.9, [0.01, 0.02, 0.03, 0.04]) is Status.ALERTING From 57a14745c3f4365d65cf1057f08705b36945bc7b Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Tue, 18 Aug 2026 16:59:19 +0200 Subject: [PATCH 075/282] MILAB-6496: make the verdict CLI helper assert the run succeeded _run captured output and returned it unchecked, and most tests in this file never looked at the returncode. That is invisible rather than merely lax: the tool writes into cwd, so a run that dies before writing leaves the previous run's files in place and every read of them still succeeds. test_output_is_byte_stable_across_runs was the live false-pass. It compares two runs' bytes and asserted neither returncode, so a second invocation crashing on startup compared the first run's files against themselves. With a tool that cannot run at all it compared {} against {} and passed. _run now asserts success by default and takes expect_failure=True for the two cutoff-refusal cases, with stderr in the failure message so a dead run says why. Verified by mutation: with the entrypoint raising immediately, byte-stability now fails where it previously passed. --- .../test/test_emit_verdicts.py | 31 ++++++++++++++----- 1 file changed, 24 insertions(+), 7 deletions(-) diff --git a/software/per-cell-metrics/test/test_emit_verdicts.py b/software/per-cell-metrics/test/test_emit_verdicts.py index 02bb92b..e4e0886 100644 --- a/software/per-cell-metrics/test/test_emit_verdicts.py +++ b/software/per-cell-metrics/test/test_emit_verdicts.py @@ -11,10 +11,28 @@ SRC = Path(__file__).resolve().parents[1] / "src" -def _run(cwd, *args): - return subprocess.run( +def _run(cwd, *args, expect_failure=False): + """Run the CLI, asserting it succeeded unless the caller wants a failure. + + Success is asserted HERE rather than left to each test, because a crashed run + is invisible to most assertions in this file: the tool writes into `cwd`, so a + run that dies before writing leaves the PREVIOUS run's files in place and + every read of them still succeeds. `test_output_is_byte_stable_across_runs` + was the live case -- it compares two runs' bytes and asserted neither + returncode, so a second invocation crashing on startup compared the first + run's files against themselves and passed. + + stderr rides along in the message because a bare `assert returncode == 0` + tells you the run died and not why. + """ + r = subprocess.run( [sys.executable, str(SRC / "emit_verdicts.py"), *map(str, args)], cwd=cwd, capture_output=True, text=True ) + if expect_failure: + assert r.returncode != 0, f"expected a non-zero exit, got 0. stdout={r.stdout!r}" + else: + assert r.returncode == 0, f"exited {r.returncode}. stderr={r.stderr!r}" + return r BASE = [ @@ -348,12 +366,11 @@ def test_a_cutoff_at_the_analytic_floor_is_refused(bed): bound = float(specificity_score(0, 0)) - r = _run(bed, *BASE, "--cutoff", "0.04") - assert r.returncode != 0 + r = _run(bed, *BASE, "--cutoff", "0.04", expect_failure=True) assert "0.042" in (r.stderr + r.stdout) - assert _run(bed, *BASE, "--cutoff", repr(bound)).returncode != 0, "the bound itself must be refused" - assert _run(bed, *BASE, "--cutoff", repr(bound * 1.001)).returncode == 0 - assert _run(bed, *BASE, "--cutoff", "0.05").returncode == 0 + _run(bed, *BASE, "--cutoff", repr(bound), expect_failure=True) # the bound itself is refused + _run(bed, *BASE, "--cutoff", repr(bound * 1.001)) + _run(bed, *BASE, "--cutoff", "0.05") def test_rows_are_sorted_on_a_bed_wide_enough_for_order_to_show(bed): From aa5e51f36e18897807096c15b1ce2e2401dce3c5 Mon Sep 17 00:00:00 2001 From: Paul Newling Date: Tue, 18 Aug 2026 17:03:21 +0200 Subject: [PATCH 076/282] MILAB-6496: carry the comparator choice as a machine token, not a sentence ReferenceChoice's values were display prose, and that prose crossed three boundaries: the run-meta JSON, a p-column DOMAIN, and a UI branch that string-matched `=== "no comparator available"` to decide whether to warn that every reading was taken against nothing. Rewording that sentence for readability would have silently removed the warning, with nothing failing. The enum now carries `declared` / `panel` / `none`, identical to the model's ReferenceSource union. Nothing in Python compared the values -- every branch uses the members -- so the change is confined to what crosses a boundary. Display wording moves to the model, next to the labels its referenceSources output already offers before a run, so a comparator does not change its name once it has served. VerdictRunMeta's two fields are typed ReferenceSource rather than string, which is what let a display sentence be used as a control-flow token. DELIBERATE consequence: the p-column domain value changes with it, so verdict columns emitted before and after do not union in one pool. A domain value is a stable key that outlives any wording; moving it once, on an unreleased branch, to stop being a sentence is better than a key that moves whenever a sentence is reworded. Side effect worth naming: test_emit_verdicts.py's `referenceChoice != "panel"` assertion was vacuous before -- no value was ever that token -- and now actually checks that a degraded request is not reported as served. --- model/src/index.ts | 40 +++++++++++++++++------ software/per-cell-metrics/src/verdict.py | 17 ++++++++-- ui/src/pages/PunchcardPage.vue | 41 ++++++++++++++++-------- workflow/src/column-specs.lib.tengo | 8 ++++- 4 files changed, 78 insertions(+), 28 deletions(-) diff --git a/model/src/index.ts b/model/src/index.ts index e6dff53..9230e01 100644 --- a/model/src/index.ts +++ b/model/src/index.ts @@ -40,13 +40,28 @@ const DEFAULT_HIGH_REFERENCE_LINE = 100; const IDENTITY_PUNCH_COLUMN = "pl7.app/antigen/identityPunch"; const IDENTITY_ID_DOMAIN = "pl7.app/antigen/identityId"; +// How each comparator choice is written for a reader. The single place the wording lives: the Python +// enum, the run-meta JSON and the p-column domain all carry the machine token, so rewording a sentence +// here cannot break a branch anywhere. The three strings match the labels the `referenceSources` output +// offers before a run, so the same choice does not change its name once it has served. +export const REFERENCE_SOURCE_LABELS: Record = { + declared: "Declared reference tag", + panel: "The panel's own readings", + none: "No comparator", +}; + // The run record emit_verdicts.py writes (result_run_meta.json), read as content. Only the fields the UI // states back to the user are typed here; the file carries every parameter the reading used. export type VerdictRunMeta = { - /** The comparator that actually SERVED — a request the panel cannot honour degrades to none. */ - referenceChoice: string; + /** + * The comparator that actually SERVED — a request the panel cannot honour degrades to none. A + * `ReferenceSource` rather than a bare string: the value crosses from the Python enum through the + * run-meta JSON into a UI branch, and typing it as `string` is what let a display sentence be used as + * a control-flow token. + */ + referenceChoice: ReferenceSource; /** The comparator that was ASKED for, so a degraded run can say what it lost. */ - referenceSourceRequested: string; + referenceSourceRequested: ReferenceSource; referenceTags: string[]; identityCount: number; setCount: number; @@ -992,9 +1007,9 @@ export const platforma = BlockModelV3.create(dataModel) // four-state verdict and the count of cells that answered it. The pivoted shape comes from the workflow // because a table cannot pivot a (set, identity) frame into columns. // - // Only the picked identities' columns are passed. Every identity is in the frame either way, so the - // picker costs a re-render rather than a run — and a panel of a thousand antigens would otherwise render - // a thousand columns nobody asked for. + // All of them by default, and the selection only narrows. Every identity is in the frame either way, so + // narrowing costs a re-render rather than a run — which is what makes it safe to open on the whole panel + // and let a reader cut it down, rather than the reverse. // // createPlDataTableV2 rather than V3 for the same reason as perCellTable above: V3's discovery walks the // whole result pool and hangs on the upstream Samples&Data File dataset. V2 takes the columns as given. @@ -1005,11 +1020,16 @@ export const platforma = BlockModelV3.create(dataModel) ?.resolve({ field: "antigenPunchcardTable", allowPermanentAbsence: true }) ?.getPColumns(); if (pCols === undefined) return undefined; + // Every antigen gets a column. The selection NARROWS that, and an empty selection means "all" rather + // than "none": the punchcard's whole job is the full grid of clonotypes against the panel, so a page + // that opens empty and waits to be told which antigens matter has inverted its own purpose. const picked = new Set(ctx.data.punchcardIdentities); - const cols = pCols.filter((c) => { - const identity = c.spec.domain?.[IDENTITY_ID_DOMAIN]; - return identity !== undefined && picked.has(identity); - }); + const identityOf = (c: (typeof pCols)[number]) => c.spec.domain?.[IDENTITY_ID_DOMAIN]; + const punchCols = pCols.filter((c) => identityOf(c) !== undefined); + const cols = + picked.size === 0 + ? punchCols + : punchCols.filter((c) => picked.has(identityOf(c) as string)); if (cols.length === 0) return undefined; return createPlDataTableV2(ctx, cols, ctx.data.punchcardTableState); }, diff --git a/software/per-cell-metrics/src/verdict.py b/software/per-cell-metrics/src/verdict.py index dea7d97..cbd71c0 100644 --- a/software/per-cell-metrics/src/verdict.py +++ b/software/per-cell-metrics/src/verdict.py @@ -136,9 +136,20 @@ class ReferenceChoice(str, Enum): software cannot serve would put a crashing option in the dropdown. """ - DECLARED = "declared reference tag" - PANEL = "the panel's own readings" - NONE = "no comparator available" + # Machine tokens, not prose, and deliberately identical to the model's + # `ReferenceSource` union. These values cross three boundaries -- the run-meta + # JSON, a p-column DOMAIN, and a UI branch -- and prose crossing a boundary + # makes rewording a sentence a breaking change: the "every reading is + # unreliable" banner was a `=== "no comparator available"` string match, so + # editing this line for readability would have silently removed the warning. + # Display wording lives in the model, which already owns the labels for these + # three choices in its `referenceSources` output. + # + # `UnreliableReason` deliberately does the opposite -- its value IS the prose a + # reader sees -- because nothing branches on it. + DECLARED = "declared" + PANEL = "panel" + NONE = "none" def resolve_default_source( diff --git a/ui/src/pages/PunchcardPage.vue b/ui/src/pages/PunchcardPage.vue index 8f47533..1075f6e 100644 --- a/ui/src/pages/PunchcardPage.vue +++ b/ui/src/pages/PunchcardPage.vue @@ -9,6 +9,7 @@ import { PlSlideModal, usePlDataTableSettingsV2, } from "@platforma-sdk/ui-vue"; +import { REFERENCE_SOURCE_LABELS } from "@platforma-open/milaboratories.feature-integration.model"; import { computed, ref } from "vue"; import { useApp } from "../app"; import PunchCell from "../components/PunchCell.vue"; @@ -46,7 +47,18 @@ const noDataset = computed(() => app.model.data.datasetRef === undefined); // choice that SERVED is the only one worth stating: a reader meeting a grid of rings otherwise has nothing // telling them the comparator they asked for was never available. const runMeta = computed(() => app.model.outputs.verdictRunMeta); -const noComparator = computed(() => runMeta.value?.referenceChoice === "no comparator available"); +// Compared against the machine token, not against a sentence. This branch used to string-match the +// display prose the Python enum happened to carry, so rewording that sentence for readability would have +// silently removed the warning below with nothing failing. +const noComparator = computed(() => runMeta.value?.referenceChoice === "none"); +// Display wording comes from the model, which owns it for the pre-run dropdown too, so a comparator does +// not change its name once it has served. +const requestedLabel = computed(() => + runMeta.value ? REFERENCE_SOURCE_LABELS[runMeta.value.referenceSourceRequested] : "", +); +const servedLabel = computed(() => + runMeta.value ? REFERENCE_SOURCE_LABELS[runMeta.value.referenceChoice] : "", +); const comparatorDegraded = computed( () => runMeta.value !== undefined && @@ -58,13 +70,15 @@ const comparatorDegraded = computed( const ungroupedTags = computed(() => runMeta.value?.tagsWithoutGroupingValue ?? []); const identityOptions = computed(() => app.model.outputs.punchcardIdentityOptions ?? []); -const picked = computed(() => app.model.data.punchcardIdentities); // The pivot is size-gated upstream: a panel above the limit emits no identity columns at all, so a run can -// have produced verdicts and still offer nothing to draw. Told apart from "nothing picked yet" because the -// two need opposite things from the reader — one is a panel too wide for this view, the other is one click. +// have produced verdicts and still have nothing here to draw. That is a different thing from a narrowed +// view, and it needs saying, because an empty grid looks the same either way. const nothingToOffer = computed(() => !noDataset.value && identityOptions.value.length === 0); -const nothingPicked = computed(() => identityOptions.value.length > 0 && picked.value.length === 0); + +// Empty means every antigen, which is the default the page opens on. Stated because an empty multi-select +// usually means the opposite, and a reader who assumes "none" will not trust a full grid. +const narrowed = computed(() => app.model.data.punchcardIdentities.length > 0); - + diff --git a/ui/src/components/PunchCell.vue b/ui/src/components/PunchCell.vue index e13939f..da5eafc 100644 --- a/ui/src/components/PunchCell.vue +++ b/ui/src/components/PunchCell.vue @@ -16,8 +16,8 @@ import { PUNCH_DIAMETER_PX, PUNCH_PAINT, parsePunch } from "./punchMarks"; // blank-for-negative is unreadable at real density, because a clonotype binds one antigen out of the panel // and the grid is therefore ~92% negative before anything goes wrong. Blank is reserved for the one state // that genuinely has no answer in it, a position the experiment never put to this clonotype. The three -// states that ARE answers read as dots of one family, so the card is a field of colour and a reader is -// never asked to tell a shape from an absence. +// states that ARE answers read as dots of one family, so the card is a field of colour and a reader is never +// asked to tell a shape from an absence. // // EVERY mark is one size, and a large one. Never size a mark by the share of a clonotype's cells that // answered. `support-travels-with-the-reading` is a DELIVERY obligation: it fixes that the scientist is @@ -51,8 +51,8 @@ const glyph = computed(() => punch.value.kind === "read" ? GLYPH_OF[punch.value.state] : "unknown", ); -// Painted from the shared map so the legend above the card cannot describe a colour the card does not -// draw. See punchMarks.ts for why these are inline values rather than CSS classes. +// Painted from the shared map so the legend above the card cannot describe a colour the card does not draw. +// See punchMarks.ts for why these are inline values rather than CSS classes. const punchStyle = computed(() => ({ display: "inline-block", boxSizing: "border-box", @@ -62,9 +62,9 @@ const punchStyle = computed(() => ({ ...(glyph.value === "none" ? {} : PUNCH_PAINT[glyph.value]), })); -// Fills the cell, so the hover target is the whole cell rather than the dot. The measured wrapper was -// 130px inside a 161px cell, which leaves a strip on each side where a reader aiming at the column would -// get nothing back. +// Fills the cell, so the hover target is the whole cell rather than the dot. The measured wrapper was 130px +// inside a 161px cell, which leaves a strip on each side where a reader aiming at the column would get +// nothing back. const cellStyle: CSSProperties = { display: "flex", alignItems: "center", @@ -73,13 +73,14 @@ const cellStyle: CSSProperties = { height: "100%", }; -// Why this mark is this colour, in the order a reader asks it: what the verdict is, what it rests on, and -// - where the verdict is unsettled - which of the six ways it failed to settle. The reason tokens are -// machine values (`no-comparator`, `tie`, ...), so each is expanded here rather than shown raw. A token -// is a key, not a sentence. -// Each line reads as its own sentence, so each is capitalised at the source rather than by a transform -// over `lines`. A blanket transform would also capitalise the antigen name, which is panel data — the -// panel says `gp120`, and a tooltip is not the place to start editing what the scientist declared. +// Why this mark is this colour, in the order a reader asks it: what the verdict is, what it rests on, and -- +// where the verdict is unsettled -- which of the six ways it failed to settle. The reason tokens are machine +// values (`no-comparator`, `tie`, ...), so each is expanded here rather than shown raw. A token is a key, +// not a sentence. +// +// Each line reads as its own sentence, so each is capitalised at the source rather than by a transform over +// `lines`. A blanket transform would also capitalise the antigen name, which is panel data -- the panel says +// `gp120`, and a tooltip is not the place to start editing what the scientist declared. const WHY_UNSETTLED: Record = { "never-offered": "No sample holding these cells declared this antigen", "no-comparator": "No baseline reading existed for these cells", @@ -103,15 +104,15 @@ const lines = computed(() => { const p = punch.value; if (p.kind !== "read") return ["No readable verdict for this clonotype at this identity"]; - // The antigen first. The header row scrolls out of view on a long grid, so the panel must say which - // column this dot belongs to before it says anything about the verdict. + // The antigen first. The header row scrolls out of view on a long grid, so the panel must say which column + // this dot belongs to before it says anything about the verdict. const out = props.params.antigen === undefined ? [] : [props.params.antigen]; out.push(p.state.toUpperCase(), EXPLANATION[p.state]); if (p.state !== "never asked") { - // How many COULD answer is shown only where the run carried panels that differ, which is where - // it varies. Under one panel it is the clonotype's own cell count at every identity, already - // beside its name in the grid, and repeating it here would teach a reader to skip the line that - // separates a verdict resting on three cells from one resting on forty. + // How many COULD answer is shown only where the run carried panels that differ, which is where it + // varies. Under one panel it is the clonotype's own cell count at every identity, already beside its + // name in the grid, and repeating it here would teach a reader to skip the line that separates a verdict + // resting on three cells from one resting on forty. out.push( props.params.showCouldAnswer ? `${p.answered} of ${p.couldAnswer} cells answered` @@ -120,25 +121,25 @@ const lines = computed(() => { if (p.agreement !== undefined) out.push(`${Math.round(p.agreement * 100)}% of them agreed`); } if (p.reason !== undefined) out.push(WHY_UNSETTLED[p.reason] ?? p.reason); - // Last, because it is about the COLUMN rather than this verdict: why this identity is one merged - // reagent while its neighbours are single antigens. + // Last, because it is about the COLUMN rather than this verdict: why this identity is one merged reagent + // while its neighbours are single antigens. if (props.params.mergedNote !== undefined) out.push(props.params.mergedNote); return out; }); -// The panel is rendered by this component and teleported to , rather than left to the browser's -// native `title`. Three reasons, and the first is decisive: a `title` inside a virtualised grid cell did -// not fire at all in the app, which is what a reader reported after the tooltip was "verified" by -// inspecting attributes in the DOM. A title also cannot be styled or laid out - a five-line explanation -// arrives as one run-together string - and it appears only after the OS hover delay, which for a grid a -// reader is scanning is long enough to feel absent. +// The panel is rendered by this component and teleported to , rather than left to the browser's native +// `title`. Three reasons, and the first is decisive: a `title` inside a virtualised grid cell did not fire at +// all in the app, which is what a reader reported after the tooltip was "verified" by inspecting attributes +// in the DOM. A title also cannot be styled or laid out -- a five-line explanation arrives as one +// run-together string -- and it appears only after the OS hover delay, which for a grid a reader is scanning +// is long enough to feel absent. // -// Teleported because the cell clips: ag-grid gives each cell `overflow: hidden`, so a panel rendered in -// place is cut to a 40px row. +// Teleported because the cell clips: ag-grid gives each cell `overflow: hidden`, so a panel rendered in place +// is cut to a 40px row. const hover = ref<{ x: number; y: number } | null>(null); -// Positioned beside the cursor and flipped when it would leave the window, so a punch in the last column -// or the bottom row still shows its whole explanation. +// Positioned beside the cursor and flipped when it would leave the window, so a punch in the last column or +// the bottom row still shows its whole explanation. const panelStyle = computed(() => { const at = hover.value; if (at === null) return { display: "none" }; diff --git a/ui/src/components/PunchLegend.vue b/ui/src/components/PunchLegend.vue index d96f079..f07f193 100644 --- a/ui/src/components/PunchLegend.vue +++ b/ui/src/components/PunchLegend.vue @@ -15,9 +15,9 @@ import { PUNCH_LEGEND_DIAMETER_PX, PUNCH_PAINT, type PunchGlyph } from "./punchM // // Which card this legend is above. The four glyphs are the same on both faces, which is the point of the // shared paint map, but what a glyph MEANS is not: a mark on the card is a verdict a majority of cells -// produced, and a mark on the by-cell face is one cell's own reading. "A majority read it as bound" is -// false of a single cell, and a legend that said it would teach the wrong thing about the very view a -// reader opened to see individual cells. +// produced, and a mark on the by-cell face is one cell's own reading. "A majority read it as bound" is false +// of a single cell, and a legend that said it would teach the wrong thing about the very view a reader opened +// to see individual cells. // // Both wordings live here, in the one component, so a glyph can never be paired with the wrong swatch. const props = withDefaults(defineProps<{ variant?: "set" | "cell" }>(), { variant: "set" }); diff --git a/ui/src/components/QcSection.vue b/ui/src/components/QcSection.vue index f5470b2..59cd917 100644 --- a/ui/src/components/QcSection.vue +++ b/ui/src/components/QcSection.vue @@ -3,8 +3,8 @@ import { PlStatusTag } from "@platforma-sdk/ui-vue"; import { reactive } from "vue"; import type { QcCheck } from "../results"; -// One QC check row in the sample report's Quality Checks tab: status tag, label, the measured value, -// and a description that stays folded until the reader asks for it. Same shape and styling as +// One QC check row in the sample report's Quality Checks tab: status tag, label, the measured value, and a +// description that stays folded until the reader asks for it. Same shape and styling as // blocks/mixcr-clonotyping's components/QcSection.vue, so the two blocks' reports read alike. const props = defineProps<{ value: QcCheck; @@ -19,9 +19,9 @@ const data = reactive({
- + NOT EVALUATED
diff --git a/ui/src/components/punchMarks.ts b/ui/src/components/punchMarks.ts index 9779924..87b23d7 100644 --- a/ui/src/components/punchMarks.ts +++ b/ui/src/components/punchMarks.ts @@ -3,27 +3,27 @@ import type { CSSProperties } from "vue"; /** * The punchcard's marks, in one place. * - * The cell renderer and the legend both paint from this map, so the legend cannot describe a colour the - * card does not draw. Two hand-maintained copies is the ordinary way a legend goes wrong, and it is - * invisible when it does: both halves look deliberate, and only a reader comparing them closely would - * notice that the swatch and the cell disagree. + * The cell renderer and the legend both paint from this map, so the legend cannot describe a colour the card + * does not draw. Two hand-maintained copies is the ordinary way a legend goes wrong, and it is invisible + * when it does: both halves look deliberate, and only a reader comparing them closely would notice that the + * swatch and the cell disagree. * * Styles are inline values rather than CSS classes because these are consumed inside an ag-grid cell - * renderer, which is instantiated outside Vue's scope-id context: a scoped stylesheet emits every rule - * with a `[data-v-…]` attribute the rendered elements do not carry, so not one rule matches and the card - * paints blank. That is not hypothetical — it shipped, and the card was reported as empty. + * renderer, which is instantiated outside Vue's scope-id context: a scoped stylesheet emits every rule with + * a `[data-v-...]` attribute the rendered elements do not carry, so not one rule matches and the card paints + * blank. That is not hypothetical -- it shipped, and the card was reported as empty. */ export type PunchGlyph = "bound" | "not-bound" | "unreliable" | "unknown"; /** * One diameter for every mark on the card. * - * The card used to size bound and not-bound by how many of a clonotype's cells answered, on the reading - * that a verdict resting on three cells must not look like one resting on forty. The obligation behind - * that is `support-travels-with-the-reading`, and it is a delivery obligation: it fixes that the scientist - * is HANDED the two counts, not that a dot encode them. They are handed over twice already -- in the - * per-cell tooltip, and as columns in the clonotype expansion -- so the card is free to be a field of - * flat colour, which is what it is actually read as at panel density. + * The card used to size bound and not-bound by how many of a clonotype's cells answered, on the reading that + * a verdict resting on three cells must not look like one resting on forty. The obligation behind that is + * `support-travels-with-the-reading`, and it is a delivery obligation: it fixes that the scientist is HANDED + * the two counts, not that a dot encode them. They are handed over twice already -- in the per-cell tooltip, + * and as columns in the clonotype expansion -- so the card is free to be a field of flat colour, which is + * what it is actually read as at panel density. */ export const PUNCH_DIAMETER_PX = 22; @@ -31,9 +31,9 @@ export const PUNCH_DIAMETER_PX = 22; * The legend's swatches, smaller than the card's marks on purpose. * * Two numbers rather than one is safe here for the reason the encoding was removed: a diameter carries no - * meaning any more, so a swatch drawn at a different size cannot misreport anything -- it is an example of - * a COLOUR, at the scale a line of text wants. While the card sized its dots by evidence this would have - * been a real hazard, because the swatch would have been read as one particular amount of support. + * meaning any more, so a swatch drawn at a different size cannot misreport anything -- it is an example of a + * COLOUR, at the scale a line of text wants. While the card sized its dots by evidence this would have been + * a real hazard, because the swatch would have been read as one particular amount of support. */ export const PUNCH_LEGEND_DIAMETER_PX = 11; @@ -46,16 +46,16 @@ export const PUNCH_PAINT: Record = { unknown: { border: "1.5px dotted #d94438", opacity: "0.7" }, }; -// ONE decoder for the punch value, shared by the grid cell and the clonotype expansion. The value is a -// single `|`-joined string because a grid pairs a cell with another column's cell only by position, and no -// import guarantees that, so everything a position needs travels together. Two readers of one format would -// be two chances to disagree about it, which is why this lives here rather than in a component. +// ONE decoder for the punch value, shared by the grid cell and the clonotype expansion. The value is a single +// `|`-joined string because a grid pairs a cell with another column's cell only by position, and no import +// guarantees that, so everything a position needs travels together. Two readers of one format would be two +// chances to disagree about it, which is why this lives here rather than in a component. // // state | cellsAnswered | cellsCouldAnswer | agreement | unreliableReason | cellsBound // // `cellsBound` is the sixth field and was appended, so a value written before it existed has five and still -// decodes. Anything that does not decode is reported as such rather than guessed at: an unreadable value -// must never pass for an answer. +// decodes. Anything that does not decode is reported as such rather than guessed at: an unreadable value must +// never pass for an answer. export const VERDICT_STATES = ["bound", "not bound", "unreliable", "never asked"] as const; export type VerdictState = (typeof VERDICT_STATES)[number]; diff --git a/ui/src/csvMeta.ts b/ui/src/csvMeta.ts index 9b5851b..08a7495 100644 --- a/ui/src/csvMeta.ts +++ b/ui/src/csvMeta.ts @@ -2,28 +2,27 @@ import type { CsvMeta } from "@platforma-open/milaboratories.feature-integration import { parse } from "csv-parse/browser/esm/sync"; /** - * Reads the tag→feature CSV's headers, each header's distinct values, and its row count. + * Reads the tag->feature CSV's headers, each header's distinct values, and its row count. * - * This is the block's ONLY panel parser. Until 2026-08 the same job ran in the workflow, as the - * emit-csv-meta Python entrypoint, and the dropdowns waited for an upload and a staging exec to fill - * them. Reading the file here fills them on the pick instead. Nothing downstream re-derives this, so the - * semantics below are the block's definition of what a panel column and a panel row ARE, not an - * approximation of some other parser. + * This is the block's ONLY panel parser. Until 2026-08 the same job ran in the workflow, as the emit-csv-meta + * Python entrypoint, and the dropdowns waited for an upload and a staging exec to fill them. Reading the file + * here fills them on the pick instead. Nothing downstream re-derives this, so the semantics below are the + * block's definition of what a panel column and a panel row ARE, not an approximation of some other parser. * - * Parsing is delegated to `csv-parse`, as in blocks/xsv-import — RFC 4180 quoting, doubled quotes, - * commas and newlines inside quoted fields, and both LF and CRLF endings. Real panel files use CRLF, so - * that last one is load-bearing. The shape of this function follows readFileForImport in - * blocks/samples-and-data: bytes in, a value out, and a throw where the file has no header to read. + * Parsing is delegated to `csv-parse`, as in blocks/xsv-import: RFC 4180 quoting, doubled quotes, commas and + * newlines inside quoted fields, and both LF and CRLF endings. Real panel files use CRLF, so that last one is + * load-bearing. The shape of this function follows readFileForImport in blocks/samples-and-data: bytes in, a + * value out, and a throw where the file has no header to read. */ export function parseTagCsvMeta(bytes: Uint8Array): CsvMeta { - // Decoding is done here rather than left to csv-parse so the block owns the one decision that a BOM - // forces. TextDecoder strips a UTF-8 BOM, which is what an Excel-exported panel needs: left in place it - // becomes part of the first header's name, and every later match against that name fails. + // Decoding is done here rather than left to csv-parse so the block owns the one decision that a BOM forces. + // TextDecoder strips a UTF-8 BOM, which is what an Excel-exported panel needs: left in place it becomes + // part of the first header's name, and every later match against that name fails. const text = new TextDecoder("utf-8").decode(bytes); - // relax_column_count: a panel whose rows are shorter or longer than its header is readable — the value - // loop below simply finds nothing at the missing indices. Refusing the file would be worse than - // reading the columns that ARE there. + // relax_column_count: a panel whose rows are shorter or longer than its header is readable, since the value + // loop below simply finds nothing at the missing indices. Refusing the file would be worse than reading the + // columns that ARE there. const records: string[][] = parse(text, { relax_column_count: true, skip_empty_lines: true, @@ -32,13 +31,13 @@ export function parseTagCsvMeta(bytes: Uint8Array): CsvMeta { if (records.length === 0) throw new Error("The panel CSV is empty: it has no header row and no data rows."); - // Blank header cells are dropped, so a trailing comma on the header line does not become a nameless - // column in three dropdowns. The INDEX is kept from the original header, not from the compacted list: - // dropping cell 1 of `Barcode,,Name` must not make `Name` look like column 1 when its values are at 2. + // Blank header cells are dropped, so a trailing comma on the header line does not become a nameless column + // in three dropdowns. The INDEX is kept from the original header, not from the compacted list: dropping + // cell 1 of `Barcode,,Name` must not make `Name` look like column 1 when its values are at 2. // - // A repeated header keeps both entries in `columns` and resolves to its LAST index for values. Both - // halves are deliberate: the dropdowns show the file's headers as the file has them, and a later - // column silently shadowing an earlier one of the same name is the same rule a spreadsheet applies. + // A repeated header keeps both entries in `columns` and resolves to its LAST index for values. Both halves + // are deliberate: the dropdowns show the file's headers as the file has them, and a later column silently + // shadowing an earlier one of the same name is the same rule a spreadsheet applies. const columns: string[] = []; const indexByColumn = new Map(); records[0].forEach((cell, index) => { @@ -56,9 +55,9 @@ export function parseTagCsvMeta(bytes: Uint8Array): CsvMeta { const distinct = new Map>(); for (const name of indexByColumn.keys()) distinct.set(name, new Set()); - // A row whose every cell is blank is not a row. Panels exported from a spreadsheet routinely carry a - // few of these at the end, and counting them would make rowCount disagree with the number of barcodes - // declared — the comparison the duplicate-mapping gate is built on. + // A row whose every cell is blank is not a row. Panels exported from a spreadsheet routinely carry a few of + // these at the end, and counting them would make rowCount disagree with the number of barcodes declared -- + // the comparison the duplicate-mapping gate is built on. let rowCount = 0; for (let r = 1; r < records.length; r++) { const row = records[r]; @@ -74,8 +73,8 @@ export function parseTagCsvMeta(bytes: Uint8Array): CsvMeta { } } - // Sorted so the dropdowns are stable: the same panel read twice must offer its values in the same - // order, whatever order the rows happened to be in. + // Sorted so the dropdowns are stable: the same panel read twice must offer its values in the same order, + // whatever order the rows happened to be in. const valuesByColumn: Record = {}; for (const [name, seen] of distinct) valuesByColumn[name] = [...seen].sort(); diff --git a/ui/src/csvSource.ts b/ui/src/csvSource.ts index bb27217..54de23a 100644 --- a/ui/src/csvSource.ts +++ b/ui/src/csvSource.ts @@ -13,10 +13,10 @@ import { parseTagCsvMeta } from "./csvMeta"; /** * The panel CSV's metadata, read straight off the user's disk. * - * Returns undefined for a REMOTE pick, which is not a failure: an `index://` handle names a file in - * remote storage that this machine cannot open, so those picks are served by the blob path below - * instead. Any real failure — the file vanished between the pick and the read, the bytes are not a - * readable CSV — throws, and the caller shows it. + * Returns undefined for a REMOTE pick, which is not a failure: an `index://` handle names a file in remote + * storage that this machine cannot open, so those picks are served by the blob path below instead. Any real + * failure -- the file vanished between the pick and the read, the bytes are not a readable CSV -- throws, and + * the caller shows it. * * Same shape as blocks/immune-assay-data (setFile) and blocks/synthetic-repertoire-profiler: guard on * isImportFileHandleUpload, then read through the ls driver. @@ -24,12 +24,12 @@ import { parseTagCsvMeta } from "./csvMeta"; export async function readLocalCsvMeta(handle: ImportFileHandle): Promise { if (!isImportFileHandleUpload(handle)) return undefined; - // The cast is unavoidable and is the one assumption this module makes. isImportFileHandleUpload proves - // the handle is an `upload://` one, but LocalImportFileHandle is a SEPARATE brand meaning "openable on - // this machine, in this session", and no SDK predicate tests for it. What makes the cast sound is the - // caller: this runs synchronously from the file-picker gesture, so the handle came from the dialog this - // session just opened. Never call this with a handle read back out of `data` — a project reopened on - // another machine carries handles whose files are not here. + // The cast is unavoidable and is the one assumption this module makes. isImportFileHandleUpload proves the + // handle is an `upload://` one, but LocalImportFileHandle is a SEPARATE brand meaning "openable on this + // machine, in this session", and no SDK predicate tests for it. What makes the cast sound is the caller: + // this runs synchronously from the file-picker gesture, so the handle came from the dialog this session + // just opened. Never call this with a handle read back out of `data` -- a project reopened on another + // machine carries handles whose files are not here. const localHandle = handle as LocalImportFileHandle; const bytes = await getRawPlatformaInstance().lsDriver.getLocalFileContent(localHandle); return parseTagCsvMeta(bytes); @@ -38,12 +38,12 @@ export async function readLocalCsvMeta(handle: ImportFileHandle): Promise LocalBlobHandleAndSize | undefined, diff --git a/ui/src/pages/AntigenQcPage.vue b/ui/src/pages/AntigenQcPage.vue index 02c5cbb..8ffb433 100644 --- a/ui/src/pages/AntigenQcPage.vue +++ b/ui/src/pages/AntigenQcPage.vue @@ -11,15 +11,15 @@ import { useApp } from "../app"; const app = useApp(); -// Two readings of the same run, on one page because a reader checking whether a run can be trusted asks -// both questions at once: did the measurements pass, and did the panel we declared match the barcodes the +// Two readings of the same run, on one page because a reader checking whether a run can be trusted asks both +// questions at once: did the measurements pass, and did the panel we declared match the barcodes the // sequencer actually returned. // // This page is the RUN's quality, never the sample's. The "Per-sample QC" page above shows the mitool -// per-sample stats — reads parsed and matched, cells and features detected — one row per sample. What is -// below is keyed (level, panel, entity, measurement): the measurements the verdict stage takes over the -// whole run. The two pages are named apart for that reason, since "QC" alone would read as two views of -// one set of numbers. +// per-sample stats -- reads parsed and matched, cells and features detected -- one row per sample. What is +// below is keyed (level, panel, entity, measurement): the measurements the verdict stage takes over the whole +// run. The two pages are named apart for that reason, since "QC" alone would read as two views of one set of +// numbers. const qcSettings = usePlDataTableSettingsV2({ model: () => app.model.outputs.runQualityTable, }); @@ -29,20 +29,20 @@ const mismatchSettings = usePlDataTableSettingsV2({ }); // A missing V(D)J dataset is a legitimate state rather than a half-filled form: the block runs, and the -// verdict stage alone is skipped — so neither table below has a source. Read from data rather than from an -// output, because the point is what the user has chosen, including before the next run. Same device, and -// the same reason, as the explore readout's own empty state. +// verdict stage alone is skipped, so neither table below has a source. Read from data rather than from an +// output, because the point is what the user has chosen, including before the next run. Same device, and the +// same reason, as the explore readout's own empty state. const noDataset = computed(() => app.model.data.datasetRef === undefined); // An absent frame and an empty frame are different facts and get different words. Absent means the verdict -// stage produced no report at all, so the frame is not there to read. Empty means it ran, imported its -// frame and put no rows in it, which for the mismatch check is the wanted outcome and for the measurements -// is a sign something went wrong upstream. So absence is answered here, by drawing no grid at all, and -// emptiness inside the grid through `noRowsText`. Neither ends up as a bare empty table. +// stage produced no report at all, so the frame is not there to read. Empty means it ran, imported its frame +// and put no rows in it, which for the mismatch check is the wanted outcome and for the measurements is a +// sign something went wrong upstream. So absence is answered here, by drawing no grid at all, and emptiness +// inside the grid through `noRowsText`. Neither ends up as a bare empty table. // -// `ok === false` is deliberately NOT treated as absence. An errored output belongs to the grid, which -// renders the error it was handed. Swallowing it into "the stage did not run" would report a failure as a -// choice the user made. +// `ok === false` is deliberately NOT treated as absence. An errored output belongs to the grid, which renders +// the error it was handed. Swallowing it into "the stage did not run" would report a failure as a choice the +// user made. const qcAbsent = computed(() => { const output = app.model.outputs.runQualityTable; return output === undefined || (output.ok && output.value === undefined); @@ -56,8 +56,8 @@ const mismatchAbsent = computed(() => { // Status is rendered as the plain string the workflow emitted, with the discrete filter its spec declares. // Deliberately not a status tag. The vocabulary is `acceptable` and `alerting` as a ranked pair PLUS // `unjudged` and `not evaluated`, and those last two are states rather than degrees of badness: `unjudged` -// means no line exists to judge against, and `not evaluated` means nothing computed it. A tag vocabulary -// of ALERT / WARN / OK / HOLD cannot carry that. It would either rank the two non-ranks as mild badness or +// means no line exists to judge against, and `not evaluated` means nothing computed it. A tag vocabulary of +// ALERT / WARN / OK / HOLD cannot carry that. It would either rank the two non-ranks as mild badness or // collapse them into one another, and both readings are the mistake this vocabulary exists to prevent. diff --git a/ui/src/pages/PunchcardPage.vue b/ui/src/pages/PunchcardPage.vue index 94f73ae..7c92a45 100644 --- a/ui/src/pages/PunchcardPage.vue +++ b/ui/src/pages/PunchcardPage.vue @@ -28,7 +28,7 @@ import VerdictSettings from "../components/VerdictSettings.vue"; const app = useApp(); // Every clonotype set against every picked identity: rows are the sets, columns are the identities, and a -// cell is one punch. This is the reading `block-set` calls this block's own view — every clonotype against +// cell is one punch. This is the reading `block-set` calls this block's own view -- every clonotype against // every identity, each position in one of the four states with what it rests on beside it. const tableSettings = usePlDataTableSettingsV2({ model: () => app.model.outputs.punchcardTable, @@ -36,19 +36,20 @@ const tableSettings = usePlDataTableSettingsV2({ // Only the antigen columns are punches. The grid applies a renderer through `defaultColDef`, so a selector // that answers unconditionally replaces EVERY cell, and the row number and the clonotype label render as -// "unreadable value" marks: a clone id is not a verdict and never parses as one. This table carries the -// punch family plus the clonotype axis and whatever label columns the pool supplies for it, and only the -// first should be drawn. +// "unreadable value" marks: a clone id is not a verdict and never parses as one. This table carries the punch +// family plus the clonotype axis and whatever label columns the pool supplies for it, and only the first +// should be drawn. // -// Identified from the column's own SPEC, which the grid hands back on `colDef.context`. Never by matching -// the identity against the column id, which is wrong twice over: an id is +// Identified from the column's own SPEC, which the grid hands back on `colDef.context`. Never by matching the +// identity against the column id, which is wrong twice over. An id is // `identityPunch_`, and the substitution rewrites `-`, space, `.`, -// `/`, `(`, `)` and more to `_`, so an identity carrying any of them — every antigen name under a property -// grouping — never matches its OWN column; and `.includes()` is a substring test, so `SpikeWT` also -// matches `identityPunch_SpikeWT_alt` and the hover panel names the wrong antigen with no sign anything is -// amiss. The spec is the exact handle: the identity travels in the column's domain, put there by +// `/`, `(`, `)` and more to `_`, so an identity carrying any of them -- every antigen name under a property +// grouping -- never matches its OWN column. And `.includes()` is a substring test, so `SpikeWT` also matches +// `identityPunch_SpikeWT_alt` and the hover panel names the wrong antigen with no sign anything is amiss. +// +// The spec is the exact handle: the identity travels in the column's domain, put there by // identityPivotImportSpec, and reading it needs no knowledge of how an id is spelled. Should the grid ever -// stop supplying `context`, the punch renderer does not apply and the card renders raw values — visibly +// stop supplying `context`, the punch renderer does not apply and the card renders raw values -- visibly // broken rather than quietly mislabelled. type PunchColumnContext = { type?: string; @@ -73,10 +74,10 @@ const identityOfColumn = (params: { colDef?: { context?: PunchColumnContext }; }): string | undefined => identityOfColumnNamed(params, PUNCH_COLUMN_NAME); -// An unplaced identity gets a note explaining ITSELF. Its column header reads as a bare barcode where -// every other header is an antigen, and nothing on the header says why. The banner above the card says -// some barcodes carry no grouping value, but it sits far from the column it is about. Attaching the note -// to the column's cells puts the explanation where the reader's cursor already is. +// An unplaced identity gets a note explaining ITSELF. Its column header reads as a bare barcode where every +// other header is an antigen, and nothing on the header says why. The banner above the card says some +// barcodes carry no grouping value, but it sits far from the column it is about. Attaching the note to the +// column's cells puts the explanation where the reader's cursor already is. // // An unplaced identity IS its barcode, since a tag the grouping column says nothing about becomes its own // identity, so this is an exact set membership test rather than a search. @@ -92,21 +93,21 @@ const mergedNote = (identity: string | undefined): string | undefined => { ); }; -// Identity -> full label, from the identity options output (which carries the workflow's label). +// Identity -> full label, from the identity options output, which carries the workflow's label. const labelOf = computed(() => { const m: Record = {}; for (const o of identityOptions.value) m[o.value] = o.label; return m; }); -// The clonotype's own column, which is where the row gets its button. The grid hands this column a context -// of `{type: "column", spec: {name: "pl7.app/label", axesSpec: [the clonotype axis]}}`. NOT an axis -// context, even though the value shown is the axis's label: the pool-resolved label column stands in for -// the axis and is handed over as an ordinary column. +// The clonotype's own column, which is where the row gets its button. The grid hands this column a context of +// `{type: "column", spec: {name: "pl7.app/label", axesSpec: [the clonotype axis]}}`. NOT an axis context, +// even though the value shown is the axis's label: the pool-resolved label column stands in for the axis and +// is handed over as an ordinary column. // -// Matched by AXIS as well as by name, against the same axis id the expansion filters on, so the two -// provably agree. Name alone breaks the moment a second label column reaches this frame, which is what -// happens on the by-identity face. +// Matched by AXIS as well as by name, against the same axis id the expansion filters on, so the two provably +// agree. Name alone breaks the moment a second label column reaches this frame, which is what happens on the +// by-identity face. const isClonotypeLabelColumn = (params: { colDef?: { context?: PunchColumnContext } }): boolean => { const spec = params.colDef?.context?.spec; const axisName = app.model.outputs.clonotypeAxisId?.name; @@ -120,13 +121,13 @@ const isClonotypeLabelColumn = (params: { colDef?: { context?: PunchColumnContex const cellRendererSelector = (params: { colDef?: { context?: PunchColumnContext } }) => { // The affordance. `invokeRowsOnDoubleClick` makes the button fire the ROW's double-click event, so it - // routes through the same `openExpansion` handler as a double-click anywhere on the row: one path, not - // two, and clicking the row keeps working. The button exists because nothing on a grid of coloured dots - // says it can be opened, and a reader who does not already know does not find out. + // routes through the same `openExpansion` handler as a double-click anywhere on the row: one path, not two, + // and clicking the row keeps working. The button exists because nothing on a grid of coloured dots says it + // can be opened, and a reader who does not already know does not find out. // - // Not `showCellButtonForAxisId`, which renders nothing here and no error: the SDK matches that prop - // against an axis column's own id or a one-axis label column's id with `isJsonEqual`, and neither branch - // matches. This route replaces the cell's renderer instead, the same mechanism the punch glyphs use. + // Not `showCellButtonForAxisId`, which renders nothing here and no error: the SDK matches that prop against + // an axis column's own id or a one-axis label column's id with `isJsonEqual`, and neither branch matches. + // This route replaces the cell's renderer instead, the same mechanism the punch glyphs use. if (isClonotypeLabelColumn(params)) { return { component: PlAgTextAndButtonCell, params: { invokeRowsOnDoubleClick: true } }; } @@ -135,8 +136,8 @@ const cellRendererSelector = (params: { colDef?: { context?: PunchColumnContext return { component: PunchCell, params: { - // The full name travels to the cell because a reader who hovers a dot far down a long grid cannot - // see the header row at all. The options output supplies the label. + // The full name travels to the cell because a reader who hovers a dot far down a long grid cannot see + // the header row at all. The options output supplies the label. antigen: labelOf.value[identity] ?? identity, mergedNote: mergedNote(identity), showCouldAnswer: panelsDiffer.value, @@ -145,8 +146,8 @@ const cellRendererSelector = (params: { colDef?: { context?: PunchColumnContext }; // The by-cell face's renderer. Matched on the cell punch's own column NAME, so a column of one card can -// never be drawn by the other card's renderer: the two share an identity domain key but nothing else, and -// the set-level renderer handed a two-field value would report it unreadable. +// never be drawn by the other card's renderer: the two share an identity domain key but nothing else, and the +// set-level renderer handed a two-field value would report it unreadable. const cellPunchRendererSelector = (params: { colDef?: { context?: PunchColumnContext } }) => { const identity = identityOfColumnNamed(params, CELL_PUNCH_COLUMN_NAME); if (identity === undefined) return undefined; @@ -156,18 +157,17 @@ const cellPunchRendererSelector = (params: { colDef?: { context?: PunchColumnCon // The reading's own settings, reachable from the page they explain. const settingsOpen = ref(false); -// The expansion: one clonotype's identities read DOWN, which `the-explore-readout` puts opposite this -// card's read ACROSS. The card stays a field of colour with no number in any position, and every number -// the atom asks for lives in here. -// -// The gesture is a double-click on the row, matching the Main page's own way of opening a sample report. +// The expansion: one clonotype's identities read DOWN, which `the-explore-readout` puts opposite this card's +// read ACROSS. The card stays a field of colour with no number in any position, and every number the atom +// asks for lives in here. The gesture is a double-click on the row, matching the Main page's own way of +// opening a sample report. // // `showCellButtonForAxisId` renders NOTHING here, with no error, and is worth re-testing only if the SDK's -// label-column branch changes. The SDK matches that prop with `isJsonEqual` against either an axis -// column's own id or the id of a one-axis LABEL column (`table-source-v2.ts:296-315`), and this card -// displays the clonotype axis through a pool-resolved label column, so neither branch matches, the -// selector returns undefined and the cell renders as plain text. The axis id is not the problem: it is -// derived from an emitted column, domain and all. +// label-column branch changes. The SDK matches that prop with `isJsonEqual` against either an axis column's +// own id or the id of a one-axis LABEL column (`table-source-v2.ts:296-315`), and this card displays the +// clonotype axis through a pool-resolved label column, so neither branch matches, the selector returns +// undefined and the cell renders as plain text. The axis id is not the problem: it is derived from an emitted +// column, domain and all. // // The key is all the event carries. `cellButtonClicked` emits a `PTableKey` and nothing else, because the // table's values live in the pFrame and a detail view is expected to re-query. So the key goes into block @@ -186,8 +186,8 @@ const expansionOpen = computed({ // identity -- the same card, one clonotype deep. // // Local state rather than block data, and reset on every open. A tab is a glance rather than a setting: -// nothing downstream reads it, no other client needs it, and reopening on whichever face was last used -// would answer a question the reader did not ask. Block data would also make it a migration. +// nothing downstream reads it, no other client needs it, and reopening on whichever face was last used would +// answer a question the reader did not ask. Block data would also make it a migration. const EXPANSION_TABS = [ { label: "By identity", value: "identity" as const }, { label: "By cell", value: "cell" as const }, @@ -201,9 +201,9 @@ function openExpansion(key?: PTableKey) { expansionTab.value = "identity"; } -// Seeded on first use rather than required in block data: a required field would need every stored -// project migrated to carry it, and a project saved before the expansion existed would otherwise open -// with an undefined grid state bound to v-model. +// Seeded on first use rather than required in block data: a required field would need every stored project +// migrated to carry it, and a project saved before the expansion existed would otherwise open with an +// undefined grid state bound to v-model. const expansionTableState = computed({ get: () => app.model.data.expansionTableState ?? createPlDataTableStateV2(), set: (value) => { @@ -211,8 +211,8 @@ const expansionTableState = computed({ }, }); -// Its own grid state, and its own `sourceId`. Two tables over different axes: sharing either would carry -// one face's column order and filters into the other, where none of the column ids resolve. +// Its own grid state, and its own `sourceId`. Two tables over different axes: sharing either would carry one +// face's column order and filters into the other, where none of the column ids resolve. const cellExpansionTableState = computed({ get: () => app.model.data.cellExpansionTableState ?? createPlDataTableStateV2(), set: (value) => { @@ -228,9 +228,9 @@ const cellExpansionSettings = usePlDataTableSettingsV2({ const expansionSettings = usePlDataTableSettingsV2({ model: () => app.model.outputs.expansionTable, // The expansion's data source changes on a DOUBLE-CLICK, not on a run: the model rebuilds the table - // filtered to whichever clonotype was chosen. The SDK documents `sourceId` as mandatory for exactly - // that case — "when the table can change without block run" — and without it the component holds the - // previous source's cached state and the grid sits in Loading until the whole window is reloaded. + // filtered to whichever clonotype was chosen. The SDK documents `sourceId` as mandatory for exactly that + // case -- "when the table can change without block run" -- and without it the component holds the previous + // source's cached state and the grid sits in Loading until the whole window is reloaded. sourceId: () => app.model.data.expandedSet?.join(" "), }); @@ -240,9 +240,9 @@ const expansionSettings = usePlDataTableSettingsV2({ // reader. // // The design asks for `C-ZDKEZ — 4 cells`. Both are pFrame values and the row event carries only a key, so -// that needs a value route which does not exist: a Parquet p-column cannot be read in the model, and no -// block in this workspace builds a header out of row values. The clonotype's name stays available as an -// optional column of the panel's own table, one click away in the Columns picker. +// that needs a value route which does not exist: a Parquet p-column cannot be read in the model, and no block +// in this workspace builds a header out of row values. The clonotype's name stays available as an optional +// column of the panel's own table, one click away in the Columns picker. // // The clonotype's readable name, fetched through the card's own pFrame handle. `fullPframeHandle` is the // frame the grid already joined the upstream label column into, so the title and the card cannot disagree @@ -251,55 +251,54 @@ const labelsPframe = computed(() => { const out = app.model.outputs.punchcardTable; return out?.ok === true ? out.value?.fullPframeHandle : undefined; }); -// The clonotype axis, derived in the model from an emitted column so its domain is exact. The same id -// the expansion's filter uses, which is what keeps the label lookup and the filter talking about one -// axis. +// The clonotype axis, derived in the model from an emitted column so its domain is exact. The same id the +// expansion's filter uses, which is what keeps the label lookup and the filter talking about one axis. const clonotypeAxisId = computed(() => app.model.outputs.clonotypeAxisId); const { resolveTitle } = useClonotypeLabels(labelsPframe, clonotypeAxisId); -// The panel's title. The name when it is known, and the generic word until then — never the raw +// The panel's title. The name when it is known, and the generic word until then -- never the raw // scClonotypeKey, which names nothing to a reader and appears nowhere else in the block. The lookup is a -// driver call, so there IS a first frame with no name yet; "Clonotype" carries that frame rather than +// driver call, so there IS a first frame with no name yet, and "Clonotype" carries that frame rather than // flashing a key. const expansionTitle = computed(() => resolveTitle(app.model.data.expandedSet?.[0]) ?? "Clonotype"); // A missing V(D)J dataset is a legitimate state rather than a half-filled form: the block runs, and the // verdict stage alone is skipped. Read from data rather than from an output, because the point is what the -// user has chosen — including before the next run. +// user has chosen, including before the next run. // -// This is a limit of how the stage is currently WIRED, not of the view. A row here is a clonotype or -// whatever else was rolled up, the read is taken per cell before anything is combined, and the software -// already accepts a cell list in place of a linker — so rows of cells are drawable in principle. What -// stands in the way is that main.tpl gates the whole verdict stage on the dataset ref. The page says what -// is true today and does not dress it up as a property of the punchcard. +// This is a limit of how the stage is currently WIRED, not of the view. A row here is a clonotype or whatever +// else was rolled up, the read is taken per cell before anything is combined, and the software already +// accepts a cell list in place of a linker -- so rows of cells are drawable in principle. What stands in the +// way is that main.tpl gates the whole verdict stage on the dataset ref. The page says what is true today +// and does not dress it up as a property of the punchcard. const noDataset = computed(() => app.model.data.datasetRef === undefined); -// What the run was actually answered under. A rung that cannot serve refuses the run rather than falling -// to another, so what served always equals what was asked for -- and where no baseline could be -// established at all, this record is what says so. +// What the run was actually answered under. A rung that cannot serve refuses the run rather than falling to +// another, so what served always equals what was asked for -- and where no baseline could be established at +// all, this record is what says so. const runMeta = computed(() => app.model.outputs.verdictRunMeta); -// A run that established no baseline read no verdicts, so no punchcard is drawn and this reason is shown -// in its place. Only the tag-distribution rung reaches here: its conditions are properties of the data, -// so the run had to proceed before it could learn them. Read from the boolean the record carries, never -// from a string match -- rewording the reason must not silently remove the branch. +// A run that established no baseline read no verdicts, so no punchcard is drawn and this reason is shown in +// its place. Only the tag-distribution rung reaches here: its conditions are properties of the data, so the +// run had to proceed before it could learn them. Read from the boolean the record carries, never from a +// string match -- rewording the reason must not silently remove the branch. const noBaseline = computed(() => runMeta.value?.baselineEstablished === false); const noBaselineReason = computed(() => runMeta.value?.noBaselineReason ?? ""); -// Whether the run carried panels that differ. `the-explore-readout` shows the per-identity -// could-answer count only then, because only then does it vary: under one panel it is the clonotype's -// own cell count at every identity, and the grid already carries that beside the name. Taken from the -// run record rather than guessed in the UI — what panels a run carried is the run's fact. +// Whether the run carried panels that differ. `the-explore-readout` shows the per-identity could-answer count +// only then, because only then does it vary: under one panel it is the clonotype's own cell count at every +// identity, and the grid already carries that beside the name. Taken from the run record rather than guessed +// in the UI -- what panels a run carried is the run's fact. const panelsDiffer = computed(() => (runMeta.value?.samplePanelCount ?? 1) > 1); -// How many of this clonotype's cells the gate set aside, stated ONCE for the clonotype. 206 puts it -// here rather than at every identity: a set-aside cell answers nothing anywhere, so a number repeated -// down the identity column would read as a per-identity failure that did not happen. +// How many of this clonotype's cells the gate set aside, stated ONCE for the clonotype, rather than at every +// identity: a set-aside cell answers nothing anywhere, so a number repeated down the identity column would +// read as a per-identity failure that did not happen. // -// Read from the run record, which is the only route available — a Parquet p-column's values cannot be -// read in the model, so a set-grain number reaches the UI either through a table (which would put it on -// every row) or through this file. +// Read from the run record, which is the only route available -- a Parquet p-column's values cannot be read +// in the model, so a set-grain number reaches the UI either through a table, which would put it on every row, +// or through this file. // -// Undefined unless a gate was declared, which is the atom's condition. An absent entry under a declared -// gate is a real zero: the map is sparse. +// Undefined unless a gate was declared. An absent entry under a declared gate is a real zero: the map is +// sparse. const setAsideLine = computed(() => { const meta = runMeta.value; const key = app.model.data.expandedSet; @@ -308,34 +307,34 @@ const setAsideLine = computed(() => { const count = meta.cellsSetAsideBySet?.[String(key[0])] ?? 0; return `Cells set aside: ${count}. A set-aside cell answers nothing at any identity.`; }); -// Which baseline served is NOT shown here, and does not need to be. Nothing substitutes for a rung -// that cannot serve, so what served always equals what was asked for and the Settings field already -// states it. More to the point, it travels with the verdicts structurally: `servedDomain` puts it in -// the DOMAIN of every emitted column, so two runs answered under different baselines emit columns of -// different identity and cannot be silently unioned in a pool holding both. A banner would be a -// weaker copy of a guarantee the data already carries. - -// Tags the grouping column said nothing about stand as their own identity under a bare barcode. The -// software reports this to stderr. A column a reader cannot place needs saying on the page too. +// Which baseline served is NOT shown here, and does not need to be. Nothing substitutes for a rung that +// cannot serve, so what served always equals what was asked for and the Settings field already states it. +// More to the point, it travels with the verdicts structurally: `servedDomain` puts it in the DOMAIN of every +// emitted column, so two runs answered under different baselines emit columns of different identity and +// cannot be silently unioned in a pool holding both. A banner would be a weaker copy of a guarantee the data +// already carries. + +// Tags the grouping column said nothing about stand as their own identity under a bare barcode. The software +// reports this to stderr. A column a reader cannot place needs saying on the page too. const ungroupedTags = computed(() => runMeta.value?.tagsWithoutGroupingValue ?? []); const identityOptions = computed(() => app.model.outputs.punchcardIdentityOptions ?? []); // The pivot is size-gated upstream: a panel above the limit emits no identity columns at all, so a run can -// have produced verdicts and still have nothing here to draw. That is a different thing from a narrowed -// view, and it needs saying, because an empty grid looks the same either way. +// have produced verdicts and still have nothing here to draw. That is a different thing from a narrowed view, +// and it needs saying, because an empty grid looks the same either way. const nothingToOffer = computed(() => !noDataset.value && identityOptions.value.length === 0); -// Headers carry the identity's full name, never a truncation: the identity a column holds is the one thing -// a reader needs from a header. The grid auto-sizes every column to its contents and exposes no width a -// block can set, so a long label does make its column wide. Every column is resizable, and the hover -// below names the identity as well. +// Headers carry the identity's full name, never a truncation: the identity a column holds is the one thing a +// reader needs from a header. The grid auto-sizes every column to its contents and exposes no width a block +// can set, so a long label does make its column wide. Every column is resizable, and the hover below names +// the identity as well. // Nothing here narrows the card to a subset of identities, and nothing should. PlAgDataTableV2 ships a -// columns panel and a filters panel, both live on this table, so such a control re-implements in block -// state what the grid already does, and two narrowing mechanisms can disagree where the grid's own cannot -// disagree with itself. Every identity column the pivot produced renders, and narrowing is done in the -// grid. The options output stays, because the card reads two other things off it. +// columns panel and a filters panel, both live on this table, so such a control re-implements in block state +// what the grid already does, and two narrowing mechanisms can disagree where the grid's own cannot disagree +// with itself. Every identity column the pivot produced renders, and narrowing is done in the grid. The +// options output stays, because the card reads two other things off it.