diff --git a/.changeset/agreement-as-a-percentage.md b/.changeset/agreement-as-a-percentage.md new file mode 100644 index 0000000..c031eb3 --- /dev/null +++ b/.changeset/agreement-as-a-percentage.md @@ -0,0 +1,13 @@ +--- +'@platforma-open/milaboratories.feature-integration.ui': minor +--- + +The agreement limit is a percentage, and the form's sections are named for what they hold. + +"Share of voting cells that must agree (0–1)" becomes "Voting cells that must agree (%)". A share is a number a reader has to translate, and this one has a floor most readers do not expect: agreement is measured among the cells that answered and the verdict takes the majority, so it can never fall to half or below. The field now runs from 51 to 100 and says why. The data keeps the 0–1 share, so `--min-agreement` and every stored project are unchanged. + +Its tooltip now leads with the default, which is **off**: no agreement test runs, a narrow majority stands, and the verdict reports how narrow it was. + +The tag-barcode FASTQ dataset is marked required, which `args()` has always enforced. The single-cell V(D)J dataset moves up beside it, so the two dataset inputs sit together above the panel. + +Three sections are renamed: a new **Panel Settings** header over the panel file and its columns, **Baseline (background) level** becomes **Baseline (Background) Parameters**, and **The reading** becomes **Threshold Parameters**. diff --git a/.changeset/antigen-barcode-binding-profiling.md b/.changeset/antigen-barcode-binding-profiling.md new file mode 100644 index 0000000..4df866f --- /dev/null +++ b/.changeset/antigen-barcode-binding-profiling.md @@ -0,0 +1,318 @@ +--- +'@platforma-open/milaboratories.feature-integration': major +'@platforma-open/milaboratories.feature-integration.model': major +'@platforma-open/milaboratories.feature-integration.ui': minor +'@platforma-open/milaboratories.feature-integration.workflow': minor +'@platforma-open/milaboratories.feature-integration.per-cell-metrics': minor +--- + +Antigen barcode binding profiling: verdicts, the baseline ladder, the quality layer and the readout. + +This release reconciles the block to the antigen-barcode binding-profiling spec. The entries below were written one change at a time and are grouped here; each keeps its own wording. + +## Verdicts and the baseline ladder + +Antigen binding is reported as a four-state verdict per clonotype set and antigen identity: **bound**, **not bound**, **never asked**, or **unreliable**. The last two are not kinds of "not bound" — *never asked* means the experiment did not put that antigen to those cells, and *unreliable* means it did and the data cannot settle the result. Both are emitted as rows rather than left absent, so a reader can tell an unanswered question from a negative answer. + +An identity is a group of tags, not a feature name: the same barcode carries different names in different samples' panels, so name-keying splits one reagent and can merge two. Tags combine into an identity by the highest of their counts, and the grouping is a rule over the panel's declared properties rather than a frozen map, so the same run can be read at more than one grouping. + +**What each verdict rests on travels with it.** Every row carries how many of the set's cells could have answered at that identity, how many did, and the agreement among them — so a verdict resting on three cells is distinguishable from one resting on forty. Where an antigen read *not bound* while something it was declared to compete with read *bound* for the same clonotype, the row says so, and a downstream filter can test it. + +**Nothing is orderable.** No score, rank or per-antigen magnitude leaves the block, and a build-time assertion refuses any score annotation on an emitted column. A verdict is a statement about what the experiment could establish; ranking clonotypes by it is a downstream block's job, from these outputs plus other assays. + +**Quality measurements ship with the reading.** Fifteen measurements across sample, tag, identity, panel and capture levels — the spec's fourteen, with its single saturation-and-depth row shipping as two, because reads-per-cell is computable from what the parse step already reports while saturation is not, and one measurement cannot be half *not evaluated*. Each states what it counts and — only where a line can be defended — what a bad value implies. A measurement with no defensible line reads *unjudged* rather than being given an invented threshold, and one the run could not supply inputs for reads *not evaluated* with its reason. Every level reports coverage beside its status, so a run states both what is wrong and how much of it was actually checked. Sample and panel roll up as separate axes: a bad sample is prepared again, a bad reagent is replaced. + +**The panel's declarations travel with the verdicts.** Whatever the panel file says consistently about an identity's tags is exported beside that identity's readings — role, species, carrier, whatever the scientist's own table carries — each as its own filterable column keyed by identity. A property an identity's member tags disagree about is omitted rather than resolved to a winner: an identity read as one thing whose members disagree about what that thing is has no declaration to travel. Without this a reader can see that an identity was bound and not what it was declared to be, and cannot ask whether a clonotype bound its target and nothing in the control group. + +The panel-versus-reads check is emitted as a p-column in both directions — barcodes the reads carry that no panel declared, and tags a panel declared that the reads never showed. + +**Removed:** the dominant-feature ("consensus") call and the per-cell specificity score. A single dominant antigen per cell answers a different question from the one this block now answers, and a specificity magnitude is exactly the narrowing the four-state verdict replaces. The `pl7.app/feature/consensusFeature` and specificity p-columns are no longer emitted. No block in this workspace consumes them. + +**Unchanged:** the per-cell UMI count and fraction columns, the negative-control marker column, and the combine-mode settings. + +**A single-cell V(D)J dataset is required for the antigen stage.** Without one the block still runs and still emits its per-cell UMI counts and fractions, per-sample QC and per-feature properties exactly as before — but it produces no verdicts, no per-antigen columns and no panel check, rather than producing empty ones. The Verdicts page says so instead of showing an empty table. + +The baseline is the scientist's choice. The block no longer picks one. + +`what-plays-the-baseline` requires that the scientist selects among the rungs and that nothing selects for them: a baseline nobody chose is a methodology nobody knows they used, and two runs of one experiment would otherwise be answered by different rules with nobody choosing either. The block derived it in two layers. Neither derives now. + +**This changes what an existing project computes.** A project that never touched the baseline field was silently answered under a derived rung — the declared tag where one existed, else the panel's own readings. It is now answered under the bottom rung: no baseline, and every verdict that needs one reads *unreliable*. The settings page says so in a warning while the field is unchosen. Choosing the rung that was being derived restores the previous numbers exactly. + +An unselected run is not refused. Refusing to start would be the block deciding a scientist's methodology by withholding the run, which is the same act as choosing one for them. It completes, and the run record carries both what was asked for and what served. + +"No baseline" is now on the list. It was withheld on the reasoning that nobody would choose a run with no answers, which was right about the consequence and wrong about the status: it is a position held in print, by scientists who argue that a tag declared to be bound by nothing is not truly negative and that a reference chosen that way lends false confidence. On that view the absence is a design choice rather than an omission. + +A rung that stops being serviceable is still never swapped for another. The run reports no baseline and records the request beside it. + +Stop the software picking a baseline rung, and require the choice on the command line. + +`what-plays-the-baseline` requires that the scientist selects among the rungs and that nothing selects for them: a baseline nobody chose is a methodology nobody knows they used, and two runs of one experiment would otherwise be answered by different rules with nobody choosing either. + +The software's own three-rung default is removed rather than left unused. Leaving it in place was a trap: the workflow omits `--reference-source` whenever the model's value is empty, so removing the derivation in the model alone would have silently promoted this one to the live rule, deriving in the layer furthest from the reader. `--reference-source` is now required, so there is nothing left to promote. + +`served_source` is unaffected and still degrades a rung that cannot serve to *none*, never to a different rung. + +The model still derives, which is a known deviation and the last one left. Removing it needs a ruling the spec does not settle — whether a run with nothing selected refuses to start or completes with every verdict that needs a baseline reading unreliable — because the two need different code. The docstring there now states this, and no longer claims such verdicts read *not evaluated*, which is a quality-measurement status and never a verdict state. + +Read counts against one declared baseline tag, or none. Several are refused, not combined. + +The block used to take the highest reading among several declared baseline tags. `baseline-scope` states that references are never combined, and taking the highest is a combination — so a panel declaring several no longer gets a comparator nobody specified. + +Reading against several is **deferred to a later version**, not dropped. That atom builds the reference as a grouping over a declared panel property: each reference serves the group its declaration scopes it to. This block has no group-by half — a tag is a comparator for the whole panel or for none of it — so it cannot say which antigens a second comparator belongs to. Refusing is also what the field does: the ordinary antibody run rejects a second control outright, and the T-cell run requires one control per allele and rejects two. + +The run stops and names the tags it found, rather than falling silently to no comparator. This is a panel a scientist fixes in a minute, and a silent fall to *unreliable* everywhere would not tell them how. + +Scoped to the rung that reads a declared tag **as** the comparator. Under the panel rung and the tag-distribution rung, several declared tags are unambiguous — the first treats them as ordinary readings in its median, the second never sees the role at all — so those runs are unaffected. + +One role value can mark several tags, so this is a limit on the tags found, not on how many values are picked. The settings tooltip says so. + +One value marks the baseline tag, not several. + +"Values that mark the baseline tag" was a multi-select. It is now a single-select, "Value that marks the baseline tag". + +`040-glossary` splits the two cardinalities. Being a control is a property of the tag, and a panel may carry several controls that are never nominated. Being the reference is a job given to one of them, declared with the run. So the value that nominates the baseline is singular. + +Several values only ever described a panel whose role column spells one role more than one way. A panel that does that is asking to be corrected, not accommodated. + +`referenceValues` stays a list in the block's data, so the `--reference-values` flag and every stored project keep their shape. A project that had picked several values keeps them until the field is next touched, and the run still refuses if they mark more than one tag. That refusal is unchanged and is now the only one that can fire: one value can still mark several tags, which is a panel fact the control cannot see. + +Whether the minimum count reaches the baseline tag is now a setting, off by default. + +The block exempted the declared baseline tag from the minimum count and hard-coded that. It is now "Apply the minimum count to the baseline tag" under Advanced reading settings, unticked. + +**It changes no verdict, and that is checked rather than asserted.** Each baseline source reads its own counts before the minimum, so the level a count is judged against is the same either way. An end-to-end test runs the same bed with the setting off and on and requires the verdicts, the per-cell counts and the per-cell scalars to be byte-identical, while requiring the removed-readings count to differ — so the test cannot pass on a bed where the setting reaches nothing. + +What it does change is the run's own accounting: how many readings the run reports as removed, how many cells it reports as emptied, and through those, which of a clonotype's cells count as empty. + +The emptied-cell population follows the same switch. With the baseline exempt, a cell holding only a below-minimum baseline reading never had evidence of binding for the minimum to remove. With the baseline subject to it, that cell has been emptied. Scoping the population one way while flooring the other would report a cell as keeping evidence it no longer has. + +One stale piece of reasoning is retired with this. The exemption's second stated ground was that flooring the comparator "lowers every denominator and shifts the whole run toward bound". Since each rung reads its own source raw, flooring here reaches no denominator at all, and that clause no longer holds. + +Add the tag-distribution baseline, and build every baseline from raw counts. + +A run with no declared control tag can now read each count against that tag's own distribution across the sample's cells, split into two components. It serves where the sample holds at least 300 cells and the tag's counts actually separate; a tag that does not separate reports no comparator rather than an invented one, and only the identities built from that tag are affected. Both conditions are settings on the CLI (`--distribution-min-cells`, `--distribution-separation`) and both are recorded in the run record, alongside a per-tag list of what could not be fitted. + +This is the first comparator that varies by identity rather than by cell, so the shared admissibility bundle now carries the identity a comparison is being made about. + +Separately, and independent of the new rung: every comparator is now computed from the raw counts rather than the floored ones. The minimum count applies to the reading being judged, never to what it is judged against — and because reference tags are exempt from the minimum and antigen tags are not, the panel comparator was previously a median taken over a mixture of raw and floored values. Panel comparators rise, so fewer cells read *bound*. + +Offer the tag-distribution baseline as a source, with its two conditions as settings. + +"Each tag's own distribution" joins the baseline dropdown. It reads each count against the lower of two components fitted to that tag's counts across the sample's cells, which is what serves a panel that declares no baseline tag and is too small to stand in for one — the shape every antibody kit has, since they cap at fifteen tags. + +It is the only source offered unconditionally. Whether it can serve turns on the sample's cell count and on whether each tag's counts separate, and the second is answered per tag rather than per run, so the conditions are stated in the option's description and the run reports what it managed: which tags fitted, which did not, and why. A tag that did not separate takes only the antigens it carries with it; every other antigen in the same cells is answered normally. + +Two new settings under "Baseline thresholds": the cells a sample needs before the rung may serve (300, from the study the method comes from) and how deep the dip between the two components must be (0.5, this block's choice — nothing published sets it). Both are sent on every run, so the record states the numbers a reading would have used whichever source served. + +Raise the panel rung's member minimum from 8 to 25. + +The figure comes from one preprint, whose own panels held 50 and 100 members, and nothing validates it lower. It gates the method rather than tuning it: below it, comparing a count against a handful of other antigens is not a background estimate, so the baseline it permits is not conservative but wrong. + +At 8 the rung was within reach of an antibody panel. It is not meant to be — those kits cap at fifteen tags — so a panel that declares no baseline tag no longer stands in as its own background. Such a run reads each tag against its own distribution instead, which is what that source was added for. A run that wants the old behaviour can still lower the setting, and says so wherever its verdicts appear. + +Carry a third number with every clonotype: how many of its cells were left with no count on any tag. + +`support-travels-with-the-reading` asks for it beside the two counts a verdict already ships — how many of the clonotype's cells could have answered at an identity, and how many did. This one is a property of the cell rather than of a position, so it is counted once for the clonotype: a cell with nothing left is empty at every identity, and repeating the subtraction per position would report a per-identity failure that did not happen. + +**It changes no verdict.** Those cells vote *not bound* like any other. What it carries is whether a negative rests on cells that read something or on cells that read nothing. + +- **The baseline is part of the test, and that is the whole discriminator.** A cell whose antigen tags all fell below the minimum count while its baseline reading survived took up reagent and none of it was antigen — a real negative and a real vote. Only a cell with nothing anywhere read nothing. The existing per-sample `cellsEmptied` counter cannot see this: while the baseline is exempt from the minimum, that counter is scoped to the readings the minimum was allowed to remove, so the baseline is invisible to it. This is a new tally rather than a rename. +- **`Cells that read nothing` ships off by default**, in the table's column chooser, ordered next to `Cells` because it qualifies it — forty cells of which thirty-eight read nothing is a different clonotype from forty that all read something. +- **Turning `Apply the minimum count to the baseline` on moves this number**, and moves nothing else. The verdicts, the per-cell counts and the per-cell scalars are byte-identical across that switch, which is now pinned by a test. +- An emptied cell stays in the clonotype's cell count and stays in the vote. Dropping such cells would shrink the denominator and make verdicts more positive, and filtering them from the cell list is the same effect by another route. + +Still to come: the per-sample alert for a run carrying many such cells, which fires whether or not a reader turned the column on. Where it lives and what counts as "many" are open. + +## Declarations and identity grouping + +Group tags into identities per tag and sample. The panel file declares what a tag carries in each sample, so a barcode reused across panels now resolves to the antigen its own sample declared instead of standing alone under its raw sequence. On a per-sample panel the punchcard renders identities across rather than tags across, and each cell's reading combines only the tags its own sample offered. + +A panel member that contradicts itself no longer lets one member's declaration stand for the whole identity. + +A property holds of a grouped identity only where its member tags agree. A tag whose own rows contradict each other has no agreed value, so it reached that test as an empty string and was filtered out exactly like a tag whose cell was blank — and a blank member is deliberately not allowed to veto its neighbours. + +On a panel with barcode reuse that inverts the outcome. Measured on a real sixteen-row panel grouped on its role column: an identity whose five member tags declared six different antigen names between them came back carrying **one member's name**, because four of the five had contradicted themselves into silence and the survivor then agreed with nobody but itself. Nothing in the export marked it partial. + +A member that contradicted itself is a disagreement, not a silence, and now blocks the property. That is the direction the tag-grain rule already takes — it keeps disagreements rather than dropping them, because with barcode reuse an inconsistent declaration is the expected case and dropping it silently breaks the panel file's no-silent-drop rule. This stops that guarantee being undone one grain higher. + +Strictly more omission, never more assertion. Panels with heavy barcode reuse will carry fewer declarations on grouped identities than before, which is the correction rather than a side effect. A member that genuinely declared nothing still does not block its neighbours, and a column the identity was grouped on is still settled by construction. + +No new computation: the call site already built the disagreement map for its warnings and simply did not pass it down. + +A panel may mark several control features, not one. + +"Control feature marker (output only)" was a single-select. It is now "Control feature markers (output only)", a multi-select, and every chosen feature is marked in the `pl7.app/feature/negativeControl` column that downstream reads. + +`040-glossary` separates the two cardinalities. Being a control is a property of the tag, and a panel may carry several controls that are never nominated. Being the reference that supplies the baseline is a job given to exactly one of them. This setting marks controls and nominates nothing, so it takes as many as the panel has. The nomination is `referenceValues`, which stays singular. + +`--control-feature` is now repeatable. It is repeated rather than comma-joined because a feature name may contain a comma, and joining would split one name into two features that do not exist. Duplicates are dropped, so a feature is marked once and the axis it keys on cannot carry it twice. + +`controlFeature` is the shape a project saved before the setting took a list. It is still read: every reader goes through `controlFeatures()`, which reads either, so no stored project needs a migration. Nothing writes the singular form now. + +Emit the negative control on the feature axis. The chosen control feature is now surfaced as a dedicated hidden per-feature marker (`pl7.app/feature/negativeControl`), so VDJ Multiomic Integration can remove the control from its antigen metrics (restriction index, antigen breadth, per-antigen fraction columns, and the dominant call). No user-facing change — the marker is hidden and is not offered as a per-feature property. + +Rename the co-binding cell label from "cross-reactive" to "Target cross-reactive", making explicit that it means a cell binding two or more on-target antigens — distinct from unwanted, nonspecific polyreactivity. Also drop the remaining internal "Decoy" examples (retired in favour of "off-target"). + +## The quality layer + +Reconcile the quality layer: one rollup level, twelve measurements, and no invented lines. + +The quality output changes shape, so a reader of a previous run's report will find rows missing and one +figure computing differently. Every change removes a claim the run could not support. + +**Only the sample rolls up.** The panel and capture statuses are gone. A panel status assumed its +per-tag measurements would mostly carry statuses, and they do not — one is categorical and the rest are +read as comparisons against the other tags in the same panel, which cannot be rolled into a severity +without discarding the comparison that made them findings. A capture status was then the worst of every +sample and every panel, which reduces to the worst of every sample: a statement that only repeats what +sits beside it. Nothing hides, because a reagent finding states itself on its own per-tag row, keyed by +the panel that has it. `--capture-map` is still accepted and is not read. + +**Three measurements are now stated exclusions rather than rows.** Sequencing saturation goes because a +scientist cannot act on it for the run already collected, and whether the run was deep enough is +answered by reads per cell against the vendor's recommendation. The known-answer check goes because +nothing declares a known answer — no surface asks which clonotype the scientist already knows — so +building the measurement means building that declaration first. Self-disagreement at an identity goes +because it has nothing to compare against, and so cannot separate a faulty reagent from a panel full of +weak binders. The obligation to show identity figures beside an alerting tag goes with it. + +**Self-disagreement is computed by pooling cells.** For one tag: every set with two or more cells that +could answer contributes all of them, and the cells sitting in the minority of their own set are the +numerator. The previous form scored sets — what share of sets disagreed at all — which needed a +small-set cutoff, since a share over three cells takes only four values and would otherwise set the +figure. **This changes the number on every run.** Two states cap the new figure at half. + +**A comparison is not a line, so it cannot produce a status.** The against-the-run route is removed +along with the interquartile fence behind it. Per-tag self-disagreement now reads *unjudged* and carries +its value for a reader to compare against the tag's siblings. What that costs is real and accepted: a +barcoded reagent binding something other than the receptor no longer announces itself, and a reader who +does not scan the column sees a bad tag and a good one alike. The alternative was a multiplier nobody +published, which moves the invention up a level rather than removing it — and an outlier rule fires on +healthy runs, because marginal binding inflates disagreement across a whole panel. + +**The per-sample checks in the interface drop two invented cutoffs.** Reads assigned to the panel keeps +its inherited 0.50 line and loses the second tier below it, which had no published source. Reads +matching the read pattern now carries no status at all, the matched share being none of the four numbers +the field publishes for this assay. Cells detected is unchanged and is now described as what it is, a +categorical fact rather than a quantity judged against a cutoff. + +**A tag the reads never show is now *never asked*, not *not bound*.** Zero reads across a sample is +categorical and cannot arise from biology: ambient reagent reaches every cell, so a tag that bound +nothing still returns counts. What zero reads means is a reagent never added, a barcode mis-declared, +or a library that failed — and none of those put the question the panel file says was put. Those cells +now leave that identity's denominator instead of voting a confident negative on every clonotype in the +run. A per-cell absence is unchanged: a cell that read nothing for a tag its sample did measure still +votes *not bound*, which is a reading that happened and failed. `declaredNeverSeen` carries no status +now, the verdict having taken that job. + +**A baseline is required, and a run without one does not happen.** The bottom rung is gone — "no +baseline" is no longer a value a scientist can select, and an unselected baseline is refused rather +than answered. The alternative was every position reading *unreliable*, which is honest and useless: a +full punchcard of non-answers costs what a real run costs and looks like a result at a glance. + +Where the refusal falls follows from when the condition becomes knowable. A missing baseline tag and a +panel below the tag count are properties of the **settings**, so they are caught before anything runs +and the message names the condition that failed. Whether a sample holds enough cells whose counts +separate is a property of the **data**, so a run on that rung proceeds, finishes, reports that no +baseline could be established, and draws no punchcard. Its answer frames keep their headers and carry +no rows; the frames describing the run's structure are written in full. + +A **Run quality** page shows the run's own quality report and the panel-versus-reads check. Both were computed by the verdict stage on every run and emitted, and neither had a page — so a measurement that came out alerting, and a barcode the panel declared that no read carried, were reported to nobody. + +The measurements table carries each measurement's status beside the coverage triple behind it — how many were judged, how many unjudged, how many not evaluated — and, where nothing computed a measurement, the reason it was deferred. All of those are shown without opening the column chooser: a status reading "nothing here is wrong" and a level where almost nothing was checkable must not look the same. Status is the plain word the run emitted, filterable through the discrete filter its column already declares, because `unjudged` and `not evaluated` are states rather than degrees of badness and no four-rank tag vocabulary can say that. + +Under a **Panel versus reads** separator, the mismatch check shows both directions in one table — barcodes the panel declared that no read carried, and barcodes the reads carried that the panel never declared — told apart by a filterable direction column. + +An absent report and an empty one say different things and are answered differently. No report at all means the verdict stage never ran, which happens when no single-cell V(D)J dataset was picked; the page says that instead of drawing a grid. An empty report means the stage ran and found nothing, which for the mismatch check is the outcome you want; the grid renders and says so in place of its rows. + +This page is the run's quality. The existing **Per-sample QC** page is unchanged and still shows the per-sample read statistics. + +Run quality rows carry the measurement's readable name, and the two columns explaining what a measurement counts and what a bad value means are shown by default rather than hidden behind the column chooser. + +## The readout and its pages + +Replace the Binding verdicts and Quality checks tables with a punchcard + +The two result tables are removed as views. A punchcard takes their place: rows are clonotype sets, +columns are the antigen identities picked from a dropdown, and a cell is one punch whose colour is the +verdict and whose size is the support behind it. Every identity is already in the result, so picking one +costs a redraw rather than a run. + +Both artifacts are still emitted. The verdicts and the run's own measurements are what the block owes, +and dropping a view does not release it from producing them — the verdicts still export to downstream +blocks, and the quality frames are still built by the workflow. What no longer exists is the pair of +grids that presented them. + +The reading itself is unchanged: no threshold, default or verdict moves. + +Block data migrates to v4, dropping the three grid states the removed views owned and adding the +punchcard's own state and its identity selection. + +Punchcard column headers show the identity's full name. They were cut to 20 characters to stop a long label auto-sizing its column off screen, but the cut fell on the barcode suffix that distinguishes two tags sharing a joined label, so distinct columns read as duplicates. + +The punchcard renders every identity column and is narrowed with the grid's own columns and filters panels. The "Antigens shown" multi-select is removed, along with the `punchcardIdentities` view state behind it. + +The per-sample slide-over is tabbed — Visual Report, Quality Checks and Log — following mixcr-clonotyping. Read recovery and the per-check quality statuses are shown for the selected sample instead of logs alone. + +## Settings and config-time guards + +Read the tag-feature panel in the UI, so the column dropdowns fill on the pick. + +Choosing a panel CSV used to start a round trip — upload the blob, run a staging exec, read its JSON — before the barcode-sequence, feature-name and negative-control dropdowns had anything to offer. The exec itself was 59 lines of stdlib Python, but its artifact shared the package's `requirements.txt`, so the backend built a venv holding polars, numpy and scipy to run it, and paid that again after every version bump. + +The UI now reads the file directly. A local pick is read from disk on the gesture and the dropdowns fill immediately. A pick from remote storage, or a project opened where the original file never existed, is read from the CSV blob the prerun already exports — the same parser over the same bytes, so there is no second implementation to keep in agreement. + +- **`emit-csv-meta` is gone**: the entrypoint, `emit_csv_meta.py`, and its tests. The prerun now has no exec at all, so nothing builds a venv during staging. It still imports and exports the CSV, which is what drives the upload. +- **`csvMetaSnapshot` in block data** carries the parsed panel, tagged with the handle it was read from. `readCsvMeta` returns it only while that tag matches the CSV currently picked, so a snapshot cannot be read against a different file. +- **A failure is now shown, not logged.** With no workflow-side parser left to fall back on, a discarded parse error would leave empty dropdowns and no explanation, so the reason appears next to the file input. +- Parsing uses `csv-parse`, as in the xsv-import block: RFC 4180 quoting, and both LF and CRLF endings. Real panel files are CRLF. + +The `prerunArgs` projection is unchanged and must stay that way — it is what keeps the UI's write to `csvMetaSnapshot` from re-rendering staging. The comment above it now says so. + +No change to `args()`, to the production workflow, or to any exported column. Existing projects re-read their panel from the exported blob on open. + +Warn at config time when the chosen barcode-sequence column holds no nucleotide sequences. A panel CSV often carries an identifier column beside the sequence column, and picking the identifier previously failed several stages into the run, inside barcode correction, with a Java stack trace. The block now names the offending values and the column that would work. The duplicate-barcode warning stays silent while this one shows, so the two never disagree about the fix. + +Clarify Settings tooltips. Each optional field now leads with when to use it (or to leave it blank), and the control, sample-column, off-target, combine-mode, dominance, and min-UMI descriptions are tightened for readability. + +The Tag-feature CSV tooltip drops its closing sentence, which pointed at the two labelled fields directly below it, so the tooltip is short enough to stay on screen. The Barcode sequence column tooltip now says the `FEATURE` tag captures the barcode on Read 2 and names Read 2 as the second read of each pair, because the assembled pattern also holds a sibling group called `R2` and "the `FEATURE` capture on Read 2" could be read as containment. + +Clarify off-target tooltips. The off-target property/values tooltips drop the retired "Decoy" term (in favour of "off-target antigen"), and the off-target-values tooltip now correctly states that value matching trims surrounding spaces but is case-sensitive — it previously claimed matching ignores case, which contradicted the shipped behaviour. + +The contending-antigen editor is not offered for now. Only the editor is deferred: a project that already carries contending groups keeps them, the args projection still passes them, and the emitted verdicts and their competitor notes are unchanged. + +It asked the scientist to retype by hand a grouping the panel file is meant to declare, and at the default one-identity-per-tag grouping the identities are the barcodes themselves — so the picker could only offer raw 15-mers. What it should become is contention derived from a declared panel column, alongside the existing grouping choice. + +Hide the Combine-mode column selector. It is not exposed to users for now; the control, its validation, and the workflow's combine-mode logic are kept for later re-enable. With the selector hidden, `combineColumn` stays unset and every antigen uses the default "sum" mode. + +## Exported columns + +The verdict stage's cell artifact is split, so that no frame leaving the block carries a column keyed on `(sample, cell, tag)`. + +One frame previously bundled two differently-keyed columns and exported the pair: the per-cell, per-tag counts, and the per-cell scalars — the reference reading and whether a declared gate set the cell aside. Only the second belongs outside. The per-cell per-tag states stay inside the block, because labelling and lead selection read verdicts and never cells, so exporting them shipped the run's largest artifact across the boundary to a consumer that does not exist. The per-cell reference readings are the opposite case: the block is required to report the cells carrying a high reference reading whether or not a gate is declared, and which of them a declared gate set aside. + +**The per-cell per-tag counts are no longer imported at all**, not merely un-exported. Nothing read them on either side of the boundary, so importing them built the run's biggest p-frame for no reader on every verdict run — their grain is cell × tag, which on a realistic panel is 11-20× the rows of the sparse reads they derive from. The Python is untouched: it still writes the counts table and the exec template still collects it, so the states are computed and exist within the run. What stops is turning them into p-columns. + +**Renamed output:** `antigenCellTable` → `antigenCellReference`. What remains in the frame is per-cell reference readings and gate outcomes, so the old name described a shape the frame no longer has. + +Renaming an output is breaking for any downstream consumer, which is why this is worth checking rather than taking on faith — but there are none, so this ships as a patch. The claim is verifiable in one command: `git grep -n -E "antigenCellTable|cellCounts|cellScalars"` over `model/src`, `ui/src` and `workflow/src` returns hits only inside the workflow that builds them. Neither the model nor the UI reads the frame; the model's `perCellTable` output resolves a different, identically-named workflow output and is unaffected. The punchcard, verdicts, QC and panel-mismatch frames are unchanged. + +Align the exported columns' reader-facing labels to the spec glossary. + +Labels and descriptions only. No p-column name, domain or axis spec changes, so nothing downstream re-binds and no column identity moves. + +- **"Reference count" → "Baseline reading"** on the per-cell comparator. The glossary defines *baseline* as the reading a count is measured against, and *reference* as the tag rather than the reading. Every other reader-facing surface moved to "baseline" already; this one lives in the workflow and was missed. +- **"Antigens bound / offered / settled / unsettled" → "Identities …"** on the clonotype counts. They count identities, never tags — the module's own comment said so two lines above the labels — and the glossary separates an identity, "a group of tags read as one thing", from the antigen a tag carries. +- **The cell-grain sibling becomes "Identities this cell bound"**, because the clonotype-grain count now carries the plain name. The two are different numbers over different populations, and a reader meeting both under one name would take the smaller for a subset of the larger, which it is not. +- **"Panel" → "Panel used"** on the per-sample column. The panel axis's own label column is the other "Panel", and it names the panel itself. +- **"Measured thing" → "Measurement subject"**, and two Title Case stragglers to sentence case. + +Checked while doing this and deliberately left alone: "Feature" and "Tag" are not two names for one layer. The feature axis keys by antigen name and the tag axis carries barcode sequences, they are separate axes on purpose, and the module already guards against a new column keying on the legacy feature axis while holding barcodes. + +No two exported columns now share a label. + +## Build and packaging + +Lower the per-sample mitool memory floor from 64 GiB to 16 GiB. The 64 GiB floor was applied on every run regardless of input size, and mitool's memory-from-limits launcher turns the grant into a JVM with `-Xms` = 50% of it — a ~32 GiB initial heap even for tiny datasets, which swaps on typical desktop RAM and stalls the "parsing reads" step. The `size("reads")*4` term still scales large inputs up (cap 256 GiB), so only small runs are affected. Also lower the per-sample mitool CPU default from 16 to 8 (matching peptide-extraction; 16 exceeded the core count on typical desktop machines) and fix the "mitool CPUs per sample" tooltip, which stated the default was 4. + +Fix the block changelog pointer. `block.meta.changelog` pointed at `file:../CHANGELOG.md` (the repo-root "Initial release" stub), so every published block-pack shipped the 1.0.0 stub and the desktop update view showed no release notes. Point it at `file:./CHANGELOG.md` — the changesets-generated block changelog. diff --git a/.changeset/each-baseline-brings-its-own-rule.md b/.changeset/each-baseline-brings-its-own-rule.md new file mode 100644 index 0000000..04abebc --- /dev/null +++ b/.changeset/each-baseline-brings-its-own-rule.md @@ -0,0 +1,18 @@ +--- +'@platforma-open/milaboratories.feature-integration.model': major +'@platforma-open/milaboratories.feature-integration.ui': minor +'@platforma-open/milaboratories.feature-integration.workflow': major +'@platforma-open/milaboratories.feature-integration.per-cell-metrics': major +--- + +The tag-distribution baseline brings its own rule, and it is the one its method was published with. + +`what-plays-the-baseline` fixes that rule: on the raw counts, drop the counts above the 99th percentile, fit a two-component negative binomial mixture, label the higher-median component the signal one, and give each cell the probability that its count belongs to it. A cell reads bound at 0.9 or above. `count-becomes-a-state` is explicit that the score and cutoff of the declared rung do not apply — each baseline brings its own rule, a run selects one baseline, so exactly one rule calls the state. + +**This changes every verdict on a distribution-rung run.** The rung previously fitted a kernel density over log2 counts, split it at the deepest trough, took the median of the background as a comparator count, and then handed that count to the declared rung's score. The published method it followed never defines that number, and no run was reading the rule the rung was validated with. + +The separation test goes with it, and that is also the spec's instruction rather than an omission. The old fit rejected a tag whose two components did not stand far enough apart, on a threshold this block invented. `what-plays-the-baseline` refuses exactly that: the method assumes two components exist, a tag that bound nothing will be split anyway and its upper slice called signal, no published test replaces the eye, and "a test invented here would be this corpus doing the thing it refuses everywhere else". So the run shows the fit instead of judging it. **A tag nothing bound can now report bound cells.** + +`Maximum dip height` is removed from the settings, along with `--distribution-separation`, because a mixture has no trough to measure. The 300-cell condition stays. + +`referenceCount` is now null on every distribution-rung row. That rung's comparator is a fitted distribution rather than a reading, and a number there would read as a comparator that served. diff --git a/.changeset/settings-named-from-the-glossary.md b/.changeset/settings-named-from-the-glossary.md new file mode 100644 index 0000000..6c6d8ba --- /dev/null +++ b/.changeset/settings-named-from-the-glossary.md @@ -0,0 +1,13 @@ +--- +'@platforma-open/milaboratories.feature-integration.ui': minor +--- + +Every setting is named from the spec glossary, and the form is grouped by what a setting decides. + +Seven labels change. "Admissibility gate (baseline UMIs)" and "Min UMIs per barcode" say **unique counts**, which is the glossary's word for a count of distinct molecules. "Minimum cell agreement" becomes "Share of voting cells that must agree", since the glossary's term is a **vote**. "Bound cutoff" becomes "Score at which a cell reads bound", naming the **score** it acts on. "Panel columns that define an identity" becomes "Panel columns that group tags into identities", naming the **grouping**. "High baseline reading" becomes "Line where a baseline reading counts as high", which says it is a line rather than a reading. "Cells needed to fit a tag's own distribution" becomes "Cells a sample needs for this baseline", dropping the method jargon. + +Four moves. The sample column joins the other panel-column pickers at the top, where it belongs: it is a panel input, and it was sitting below the baseline among the reading settings. The baseline's own cell condition moves up beside the baseline choice, because it is a condition on that choice. "Optional settings" becomes **The reading** and takes the agreement share, which was alone behind an "Advanced reading settings" accordion that now goes. The "Baseline thresholds" accordion goes too: the gate and the high-reading line both read a cell's own baseline reading, so both exist only under a declared tag, and they now sit in the baseline section with the rung they belong to. A header for two fields that vanish with the rung above them was a section the reader met empty more often than not. + +Both of those fields were gated on "not the fitted rung", which also showed them before any baseline was chosen. They are now gated on the declared rung itself. + +Three tooltips follow the same words, and the sticky-cell one now says what sticky means and that counting such cells is a measurement while setting them aside is the gate's job. diff --git a/.changeset/the-minimum-never-reaches-the-baseline.md b/.changeset/the-minimum-never-reaches-the-baseline.md new file mode 100644 index 0000000..bc442fa --- /dev/null +++ b/.changeset/the-minimum-never-reaches-the-baseline.md @@ -0,0 +1,14 @@ +--- +'@platforma-open/milaboratories.feature-integration.model': major +'@platforma-open/milaboratories.feature-integration.ui': minor +'@platforma-open/milaboratories.feature-integration.workflow': minor +'@platforma-open/milaboratories.feature-integration.per-cell-metrics': minor +--- + +The minimum count never reaches the baseline tag, and the setting that could make it is gone. + +`minimum-count-before-any-reference` states it as a rule rather than a preference: the minimum asks whether a count is evidence of binding, a tag declared to be bound by nothing is never evidence of binding, so the question does not arise for it. A cell's reference reading enters the comparison as it came back, and a small one is the measurement rather than noise to be cleared away. + +"Apply the minimum count to the baseline tag" is removed, along with `--minimum-applies-to-baseline` and the `minimumAppliesToBaseline` field. The exemption is now unconditional in `apply_floor`. + +**No verdict changes.** The setting was off by default and every rung already built its comparator from raw counts, so the default behaviour is what shipped. What changes is that the other behaviour can no longer be selected. A stored project that had switched it on now runs with the exemption in force, and reports fewer removed readings and fewer emptied cells than it did. diff --git a/.changeset/the-quality-layer-reads-three-statuses.md b/.changeset/the-quality-layer-reads-three-statuses.md new file mode 100644 index 0000000..a0a4da2 --- /dev/null +++ b/.changeset/the-quality-layer-reads-three-statuses.md @@ -0,0 +1,41 @@ +--- +'@platforma-open/milaboratories.feature-integration.model': minor +'@platforma-open/milaboratories.feature-integration.workflow': minor +'@platforma-open/milaboratories.feature-integration.per-cell-metrics': minor +--- + +The quality layer reads three statuses, and the reagent table reads the frame each figure is about. + +A measurement is now **OK**, **warn** or **alert** and nothing else. One with no line behind it carries no status, and which of the two cases it is reads from the value: a number means nothing judges it, a reason in place of a number means nothing computed it. The row is there either way, and the coverage triple beside it still separates the two. + +**Warn is new.** Every inherited line arrives with a warn threshold and an error threshold, and the block held one number per measurement, so each pair was collapsed and a calibrated distinction discarded. Reads per cell at 4,000 read *alerting* and now warns, since one published number gives one boundary. The panel-assigned fraction at 0.49 read *alerting* and now warns, since only a wholly failed sample alerts. + +The two thresholds of a line are read independently, because the field warns on a direction and puts error at total failure for three of its four lines. + +**The reagent table's figures now come from the right side of the minimum.** Cells with any count and the median count per cell are read from the raw counts, cells called bound from the post-minimum states. The median was taken over bound cells alone, where it could only ever print a number above the cutoff's floor, so a half-degraded reagent showed a healthy figure. Every declared tag keeps a row, so a dead reagent reads as a zero rather than as an absence, and reference tags keep a row whose bound count is empty rather than zero. + +**Two measurements changed hands.** The panel-assigned fraction was filed against the *usable antigen reads* row and satisfies the *undeclared barcodes* row: its complement is that quantity exactly, and the line transfers with it. Its status no longer reaches its sample's rollup, because a reagent belongs to the run rather than to any one sample. + +**One measurement is new.** The fraction of reads whose cell barcode the chemistry could have produced, warning below 0.75 and alerting below 0.50. The refine-tags report already carried the step it reads. It is the one inherited line with a gradient at both ends rather than a catastrophe, and the reason a third status level exists. + +The per-sample quality frame gains a **Valid cell-barcode fraction** column. + +**The fitted background now leaves the function that fits it.** Under a population baseline the block fits a two-component mixture per tag and per sample, and kept only the failures. The background component's mean, its share of cells, and the signal mean beside it now reach the measurement set as one tag-level row: the median over the panel's samples that fitted, with the spread and the unfitted count in its detail. It is what a scientist reads to see whether a tag's counts separated at all, and it depends on no cutoff — which matters, because it is read in order to settle one. + +Under a declared baseline nothing is fitted, and every row says that rather than going missing. + +**The run's scores now have a spread.** The score is computed for every cell and identity, used for one comparison against the cutoff, and was then dropped. It reaches the measurement set as deciles: one figure for the whole run, because the cutoff is one number for the run. A scientist may move that cutoff to where their own run's scores separate, and that licence is unusable unless the scores are in front of them — until now it was set blind. + +The measurement carries no line, since a line here would be the block placing the cutoff instead. A population baseline yields a probability rather than a score, and under it the row says so rather than printing a number from the wrong rule. The measurement set gains a **run** grain, which is a grain of one. + +**The sticky measurement takes two forms, and the gate decides which.** The block carried two thresholds on a cell's reference reading: the admissibility gate, which sets a cell aside, and a separate observation line defaulted to 100, which counted cells as sticky. Only one exists in the spec. `060-parameter-set` lists seven parameters and a sticky line is not among them, and `290-reference-two-roles` is explicit that *how many are high* needs a high, and only a declared gate supplies one. + +So **Sticky cell threshold** is gone from the settings, along with its parameter and its `--high-reference-line` flag. With a gate declared, the cells counted high are the cells it set aside — one number, both jobs. With no gate declared, which is the default, there is no *high* to count and the measurement is the **spread of the reference readings** instead. That spread is what a scientist reads in order to place a gate, and until now a first run offered a count against a line nobody had chosen. + +A stored project carrying the old value keeps running; the field is simply no longer read. + +**Run quality is visible again, and it draws the three distributions.** The tab was hidden while the quality layer was being re-cut. It returns with the measurement table, the panel-versus-reads check, and a new Distributions section holding the three plots `330-the-quality-readout` asks for: the spread of the run's scores, the reference reading across cells, and the fitted background for each tag. + +The deciles and the fitted parameters go out as plottable frames rather than only as detail strings on a measurement row, because a number encoded in a string is one nobody can draw. Both decile sets are taken over the whole run: the cutoff is one number for the run and so is the gate, so each plot shows every cell its number will act on. The backgrounds are emitted at the fit's own `(sample, tag)` grain, since aggregating to the tag would hide a reagent that separated in one sample and not another. + +Each plot says so where it cannot exist: a run read against a population baseline has no scores to spread, and one read against a declared baseline tag fits no background. diff --git a/.changeset/the-sample-report-is-the-measurement-set.md b/.changeset/the-sample-report-is-the-measurement-set.md new file mode 100644 index 0000000..942f801 --- /dev/null +++ b/.changeset/the-sample-report-is-the-measurement-set.md @@ -0,0 +1,22 @@ +--- +'@platforma-open/milaboratories.feature-integration.model': minor +'@platforma-open/milaboratories.feature-integration.workflow': minor +'@platforma-open/milaboratories.feature-integration.per-cell-metrics': minor +'@platforma-open/milaboratories.feature-integration.ui': minor +--- + +A sample's report is its own measurement set, and the Quality tag is that set's rollup. + +The Quality Checks tab listed **three** hand-written checks against a threshold the UI held itself, while the software computed **nine** sample-level measurements with their own statuses and their own line provenance. A reader opened a sample, met three rows and a green badge, and concluded the sample had been checked. Six measurements were omitted without saying so, and a list that silently drops what it could not check reads exactly like a list that checked everything and found nothing wrong. + +**The tab now lists every sample-level measurement the software declares**, in declaration order, including the ones nothing computed. Each row carries its status where a line stands behind it, its value, and — where there is no value — the reason in place of one. A measurement with no line carries no status and shows an em-dash: there is no fourth status word, and which of the two no-status cases applies is read from the value. + +**A blank and a zero are opposite findings, so nothing is ever blank.** Every valueless row states why: the aggregate-barcode fraction because nothing in this block detects aggregates, the sticky measurement because no cell of the sample carries a comparator reading, the read-level fractions because the refine-tags report supplied no step with input reads. The reasons travel in `result_qc.csv` too, on a `reason` column that previously carried only the deferred ones. + +**The sample's rolled-up status sits at the top of the list, with its coverage beside it** — how many measurements were judged, how many were computed with no line to judge them against, and how many nothing computed. Whether something is wrong and whether anybody looked are different questions and are answered separately. + +**The Main grid's Quality tag is that same rollup and is computed nowhere else.** The UI no longer holds a QC threshold of its own: `PANEL_ASSIGNED_LINE`, `qcChecks`, `qualityStatus` and `QcCheck` are gone, and with them the second copy of a line that could drift from the software's. The tag and the report beside it cannot disagree about one sample. + +A measurement whose finding belongs to a reagent rather than to the sample keeps its own status on its row and stays out of the sample's rollup, and its row says so — otherwise a reader meets a status on the page that the tag does not carry. + +The verdict step emits `result_qc_by_sample.json` alongside `result_qc.csv`, and the model reads it as `sampleQcReport`. The frame remains the artefact every other reader takes; the report is the same measurements keyed by sample, for a view that holds one sample at a time. diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml index 14226e6..42a8beb 100644 --- a/.github/workflows/build.yaml +++ b/.github/workflows/build.yaml @@ -24,7 +24,7 @@ jobs: with: app-name: 'Block: Feature Integration' app-name-slug: 'block-feature-integration' - node-version: '20.x' + node-version: '22.x' gha-runner-label: hz-ubuntu-dind build-script-name: 'build:dev-local' build-before-publish-script-name: 'build:release' diff --git a/.github/workflows/mark-stable.yaml b/.github/workflows/mark-stable.yaml index 1af2df2..e255c10 100644 --- a/.github/workflows/mark-stable.yaml +++ b/.github/workflows/mark-stable.yaml @@ -15,7 +15,7 @@ jobs: uses: milaboratory/github-ci/.github/workflows/block-mark-stable.yaml@v4 with: app-name: 'Block: Feature Integration - Mark Stable' - node-version: '20.x' + node-version: '22.x' npmrc-config: | { "registries": { diff --git a/.github/workflows/python-tests.yaml b/.github/workflows/python-tests.yaml index 2e1c8e8..ea0f0e0 100644 --- a/.github/workflows/python-tests.yaml +++ b/.github/workflows/python-tests.yaml @@ -51,7 +51,7 @@ jobs: - uses: actions/checkout@v4 - uses: pnpm/action-setup@v4 - uses: actions/setup-node@v4 - with: { node-version: '20.x', cache: 'pnpm' } + with: { node-version: '22.x', cache: 'pnpm' } - run: pnpm install --frozen-lockfile - uses: astral-sh/setup-uv@v5 with: { version: "${{ env.UV_VERSION }}" } diff --git a/.gitignore b/.gitignore index d9096dd..05b8d09 100644 --- a/.gitignore +++ b/.gitignore @@ -29,7 +29,15 @@ software/test-data/manual/** !software/test-data/manual/**/ !software/test-data/manual/**/*.py !software/test-data/manual/**/*.md +# ...except these dev design/calibration notes — kept on disk, not version-controlled. +software/test-data/manual/design-and-schemas.md +software/test-data/manual/real-data-calibration.md !software/test-data/manual/assets/whitelist_cells.txt +# Generated runs are ignored WHATEVER their extension. The `*.md` exception above exists for the +# hand-written docs beside the generators, and a generated run carries its own RUN.md report — which +# describes the panel that drove it, and that panel may be a real one supplied from outside the repo. +# Nothing a generator writes is recipe, so nothing under runs/ is tracked. +software/test-data/manual/runs/** # Local working docs / scratch — not version-controlled docs/superpowers/ diff --git a/block/package.json b/block/package.json index d78915f..473ce19 100644 --- a/block/package.json +++ b/block/package.json @@ -48,7 +48,7 @@ "support": "mailto:support@milaboratories.com", "description": "Assign antigens to single cells from BEAM / LIBRA-seq style feature-barcode reads", "longDescription": "file:../docs/description.md", - "changelog": "file:../CHANGELOG.md", + "changelog": "file:./CHANGELOG.md", "tags": [ "single-cell", "vdj", diff --git a/block/src/agents-extra.ts b/block/src/agents-extra.ts index 228811f..cf19ec2 100644 --- a/block/src/agents-extra.ts +++ b/block/src/agents-extra.ts @@ -3,6 +3,6 @@ // surface should expose. // // In the future this file will host JS functions the MCP runtime -// can execute. The `.d.ts` declarations stay visible to the agent; -// the JS bodies execute in the MCP code-execution context (the -// agent sees the types but not the implementation). +// can execute. The `.d.ts` declarations stay visible to the agent. +// The JS bodies execute in the MCP code-execution context, so the +// agent sees the types but not the implementation. diff --git a/model/package.json b/model/package.json index 298dbab..7df3cc7 100644 --- a/model/package.json +++ b/model/package.json @@ -22,6 +22,7 @@ "check": "ts-builder check --target block-model" }, "dependencies": { + "@milaboratories/graph-maker": "catalog:", "@milaboratories/helpers": "catalog:", "@platforma-sdk/model": "catalog:" }, diff --git a/model/src/index.ts b/model/src/index.ts index 70bc7f9..88a97bb 100644 --- a/model/src/index.ts +++ b/model/src/index.ts @@ -1,26 +1,256 @@ -import type { BlockRenderCtx, InferOutputsType } from "@platforma-sdk/model"; +import type { + AxisId, + BlockRenderCtx, + InferOutputsType, + PlDataTableStateV2, +} from "@platforma-sdk/model"; import { BlockModelV3, createPlDataTableStateV2, + createPFrameForGraphs, createPlDataTableV2, + createPlDataTableV3, + DataColumn, DataModelBuilder, isPColumnSpec, parseResourceMap, + getAxisId, } from "@platforma-sdk/model"; import { assemblePattern, CELL_TAG, FEATURE_TAG, UMI_TAG, validatePattern } from "./pattern"; import { getPreset } from "./presets"; -import type { BlockArgs, BlockData } from "./types"; +import type { BlockArgs, BlockData, CsvMeta, GroupingRule, ReferenceSource } from "./types"; export { assemblePattern, parsePattern, validatePattern } from "./pattern"; export type { PatternParts } from "./pattern"; export { allPresets, getPreset } from "./presets"; export type { Preset } from "./presets"; -export type { BlockArgs, BlockData } from "./types"; +export type { BlockArgs, BlockData, CsvMeta, GroupingRule, ReferenceSource } from "./types"; + +// Re-exported so the UI can seed a grid state without a direct @platforma-sdk/model dependency. The ui +// package depends only on ui-vue, which does not carry this factory. +export { createPlDataTableStateV2 } from "@platforma-sdk/model"; +export type { PTableKey } from "@platforma-sdk/model"; + +// The reading's shipped defaults, restating the Python's own (verdict.py DEFAULT_FLOOR, BOUND_CUTOFF, +// DEFAULT_PANEL_MIN_MEMBERS, combine.py DEFAULT_MIN_VOTERS) so +// the user can see and change the value that produced a run. Each is a declared default, not a +// calibrated line: nothing published sets any of them. +const DEFAULT_COUNT_FLOOR = 4; +const DEFAULT_BOUND_CUTOFF = 75; +const DEFAULT_MIN_VOTING_CELLS = 1; +// From one preprint, whose own panels held fifty and a hundred members. Nothing validates it lower. It +// GATES rather than tunes: below it, a count compared against a handful of other antigens is not a +// background estimate. Keep it above the fifteen-tag cap of an antibody kit, so such a panel falls to +// the tag-distribution rung instead. +const DEFAULT_PANEL_REFERENCE_MIN_MEMBERS = 25; +// Both from the study the tag-distribution rung comes from. The first is that study's own bootstrapping +// figure. The second has no published value -- the paper shows the trough in a figure and never says how +// deep one must be -- so it ships as a declared default a run can move and report moving. Mirrors +// tag_distribution.py. +const DEFAULT_DISTRIBUTION_MIN_CELLS = 300; + +// The four inherited lines and the three aggregate-barcode knobs (see BlockArgs) are projected straight +// through in args() below, undefined included. An undefined line reaches the CLI as the shipped default +// substituted by verdict-args.lib.tengo's `_num`; an undefined knob reaches qc_report.py's argparse +// default. Those two remain the values a run is scored against. +// +// The two maps below are read by the settings fields ONLY, to display the number already in force where +// the stored value is undefined. They are never seeded into `data` and never projected into args, so a +// field showing 0.75 and a field holding 0.75 produce the same command line and the same args hash. +// +// Each value MUST equal its counterpart in verdict-args.lib.tengo (the lines) and qc_measures.py (the +// knobs). `test/src/qcDefaults.test.ts` asserts both sets against those files. +export const QC_LINE_DEFAULTS = { + cellBarcodeValidWarn: 0.75, + cellBarcodeValidError: 0.5, + readsPerCellWarn: 5000, + aggregateBarcodeWarn: 0.05, + aggregateBarcodeError: 1.0, + undeclaredBarcodeWarn: 0.5, + undeclaredBarcodeError: 1.0, + usableReadWarn: 0.2, + usableReadError: 0.0, +} as const; + +export const AGGREGATE_DETECTION_DEFAULTS = { + aggregateBarcodeIqrMultiplier: 3.0, + aggregateBarcodeMinUmiThreshold: 1000.0, + aggregateBarcodeTopN: 100, +} as const; + +// The punchcard's frame is keyed on the clonotype set alone, and each identity is a COLUMN rather than +// an axis value: that is what a punchcard needs, and a (set, identity) frame cannot give it to a table. +// The identity travels in the column's DOMAIN, so the model reads a column's identity without parsing a +// label. +// +// One column per identity. Its value carries the state and both support counts together (see +// identityPunchImportSpec), because a grid pairs one column's cell with another's by position alone, +// which no import guarantees. +// +// The UI identifies a punch column by these two -- the column name and the domain key its identity +// travels under -- read off the spec the grid hands back on `colDef.context`. Never by the column id: an +// id is mangled by `substituteSpecialCharacters`, and a substring test lets `SpikeWT` match +// `SpikeWT_alt` and name the wrong antigen. +export const PUNCH_COLUMN_NAME = "pl7.app/antigen/identityPunch"; +export const PUNCH_IDENTITY_DOMAIN = "pl7.app/antigen/identityId"; +// The by-cell face's punch column. Same identity domain key, so one column-matching helper serves both +// cards. Only the column NAME separates a set's verdict from a cell's own reading. +export const CELL_PUNCH_COLUMN_NAME = "pl7.app/antigen/cellPunch"; +// The clonotype's cell count, carried in the punchcard's own frame so the grid can read it. A block's own +// exports are not in its own result pool, so the copy in the exported setCounts family is unreachable +// from here. +export const PUNCH_CELL_COUNT_COLUMN = "pl7.app/antigen/cellCount"; + +// How each comparator choice is written for a reader, and the single place that wording lives. The Python +// enum, the run-meta JSON and the p-column domain all carry the machine token, so rewording a sentence +// here cannot break a branch. These strings match the labels `referenceSources` offers before a run, so a +// choice does not change its name once it has served. User-facing names only: the DATA layer keeps +// `declared`/`panel`/`none`, which are p-column domain values, and domain is part of column identity. +export const REFERENCE_SOURCE_LABELS: Record = { + declared: "Declared baseline tag", + panel: "The panel's own readings", + distribution: "Each tag's own distribution", +}; + +// The run record emit_verdicts.py writes (result_run_meta.json), read as content. Only the fields the UI +// states back to the user are typed here. The file carries every parameter the reading used. +export type VerdictRunMeta = { + /** + * The comparator that actually SERVED -- a request the panel cannot honour degrades to none. A + * `ReferenceSource` rather than a bare string: the value crosses from the Python enum through the + * run-meta JSON into a UI branch, and typing it as `string` is what let a display sentence be used as + * a control-flow token. + */ + referenceChoice: ReferenceSource; + /** + * Whether the run established a baseline, and where it did not, why. + * + * Only the tag-distribution rung can reach false. Its conditions -- enough cells in the sample, and + * counts that actually separate -- are properties of the DATA, so a run resting on it proceeds and + * reports afterwards. The other rungs are refused from the settings before anything is read. + * + * False means the run finished and read no verdicts. The punchcard is not drawn, and the reason is + * shown in its place: a full grid of *unreliable* costs what a real run costs and looks like a result + * at a glance. + */ + baselineEstablished: boolean; + noBaselineReason: string | null; + /** The comparator that was ASKED for, so a degraded run can say what it lost. */ + referenceSourceRequested: ReferenceSource; + referenceTags: string[]; + identityCount: number; + setCount: number; + cellsAnalysed: number; + /** Tags the grouping column said nothing about. Each stands as its own identity, under a bare barcode. */ + tagsWithoutGroupingValue: string[]; + /** + * How many DISTINCT panels the run carried. One means every sample was stained with the same tags, and + * then how many of a clonotype's cells could answer does not vary by identity -- it is the clonotype's + * own cell count, which the grid carries beside its name. Optional because a run record written before + * this field existed does not have it, and a reader treats absent as one. + */ + samplePanelCount?: number; + /** + * The read limit the run applied, or absent/null where none was declared. Present because it is the one + * signal for "a gate was declared" -- `the-explore-readout` shows set-aside cells only then, and a gate + * that set nothing aside must still say so rather than look like no gate at all. + */ + gateThreshold?: number | null; + /** + * How many of each clonotype's cells the gate set aside, keyed by set id. Absent when no gate was + * declared, and SPARSE -- a clonotype that lost nothing carries no entry, so an absent key reads as + * zero. Sparse because this record is parsed on every render. + * + * Set grain, deliberately, and not a column of the expansion table: a set-aside cell answers nothing at + * any identity, so repeating the subtraction at every position would imply a per-identity failure that + * did not happen. + */ + cellsSetAsideBySet?: Record; + /** + * Which cell list every figure was computed against. + * + * Two runs whose lists came from different sources do not share a denominator, so a fraction of cells + * carries no meaning without it. Optional because a run record written before this field existed does + * not carry it. The three values are `emit_verdicts.py`'s own, not prose: renaming one there without + * renaming it here leaves this undefined rather than failing. + */ + cellListSource?: CellListSource; +}; + +/** Where the run's cell list came from. Mirrors `emit_verdicts.py`'s `cell_list_source`. */ +export type CellListSource = "cell list" | "clonotype linker" | "none"; -const DOMINANCE_FLOOR = 0.5; // threshold is user-adjustable down to 0.5, never lower +// A measurement's status. Three words and no fourth: a measurement carries one only where a line stands +// behind it, and null covers both cases where none does. Which of the two happened is read from the value -- +// a number means computed and unjudged, its absence means nothing computed it, and the reason says which. +export type QcMeasurementStatus = "OK" | "warn" | "alert"; + +// One sample-level quality measurement, as emit_verdicts.py writes it into result_qc_by_sample.json. +export type SampleQcMeasurement = { + /** The measurement id. Stable: it is also a value on the `measurement` axis and a p-column name. */ + id: string; + /** The readable name, carried beside the id rather than instead of it. */ + label: string; + /** The number, or null where the run could not compute one. Then `reason` says why, and is never empty. */ + value: number | null; + /** What went in, carried alongside a number. Null where there is nothing to add. */ + detail: string | null; + /** Why there is no number. Non-null exactly when `value` is null. */ + reason: string | null; + /** Null where no line stands behind the measurement, or where there is no value to judge. */ + status: QcMeasurementStatus | null; + /** What the measurement counts. */ + counts: string; + /** What a bad value means, where a line exists to make that claim. */ + implies: string | null; + /** + * Whether this measurement's status reaches the sample's rollup. False for a measurement whose finding + * belongs to a reagent rather than to the sample it was measured on, which is why a row can carry a + * status the sample's own tag does not. + */ + rollsUp: boolean; +}; + +// One sample's quality report: every sample-level measurement, and the rollup over those that roll up. +export type SampleQcReport = { + /** The worst status among the measurements that carry one. Null where none did. */ + status: QcMeasurementStatus | null; + /** How many measurements carried a status. */ + judged: number; + /** How many were computed with no line to judge them against. */ + unjudged: number; + /** How many the run could not compute at all. */ + notEvaluated: number; + /** Every sample-level measurement in declaration order, including the ones nothing computed. */ + measurements: SampleQcMeasurement[]; +}; + +// What the software resolves an unset reference source to, restated so the dropdown can say it. Mirrors +// verdict.py resolve_default_source: a declared reagent, else the panel's own readings where the panel is +// big enough, else nothing. +export type ReferenceSourceChoices = { + /** + * EVERY rung, always, in ladder order, and every one selectable. The scientist picks the rung first and + * then supplies what it needs -- which is why an unmet requirement cannot withhold the option. Offering + * only the rungs already satisfied made the declared rung unreachable until its own sub-settings were + * filled in, with nothing on screen saying which those were. + */ + options: { + value: ReferenceSource; + label: string; + description: string; + /** What this rung still needs before it can serve, as a sentence. Undefined once it can. */ + needs?: string; + }[]; + /** + * What a run with nothing chosen is answered under, as a sentence. Constant now that nothing derives, + * and kept because the sentence is what a reader needs rather than the token. + */ + fallback: string; +}; // Ordinal step key -> the step a sample is CURRENTLY on once that report has settled. A stepReports entry -// appears when its step finishes, so the furthest-present report implies the next running step +// appears when its step finishes, so the furthest-present report implies the next running step. export type SampleStep = "parsing" | "refining" | "counting" | "metrics"; const STEP_AFTER: Record = { "1-parse": "refining", @@ -29,15 +259,15 @@ const STEP_AFTER: Record = { }; const STEP_ORDER = ["1-parse", "2-refine", "3-tagstat"]; -// mitool prefixes its progress lines with this marker (set via MI_PROGRESS_PREFIX in the workflow -// step templates); the model scrapes matching lines for a live per-sample 0–100% bar. ProgressPattern -// pulls the stage name + percent + ETA out of a marked line. Same values as blocks/peptide-extraction. +// mitool prefixes its progress lines with this marker, set via MI_PROGRESS_PREFIX in the workflow step +// templates. The model scrapes matching lines for a live per-sample 0-100% bar, and ProgressPattern pulls +// the stage name, percent and ETA out of one. Same values as blocks/peptide-extraction. export const ProgressPrefix = "[==PROGRESS==]"; export const ProgressPattern = /(?[^:]*):(?: *(?[0-9.]+)%)?(?: *ETA: *(?.+))?/; -// Per-sample QC metrics as emitted by qc_report.py (result_qc.json), read by the analysisLog output -// and (per sample) by the Main grid's Quality + Read recovery columns (derived in ui/src/results.ts). +// Per-sample QC metrics as emitted by qc_report.py (result_qc.json), read by the analysisLog output and, +// per sample, by the Main grid's Quality + Read recovery columns (derived in ui/src/results.ts). export type QcRow = { readsTotal: number; readsMatched: number; @@ -47,29 +277,21 @@ export type QcRow = { totalUniqueUmis: number; medianUmisPerCell: number; panelAssignedFraction: number | ""; // "" when no refine report (qc_report leaves it blank) + cellBarcodeValidFraction: number | ""; // same blank rule; the refine report's CELL step }; // Panel-assigned fraction below this flags a sample in the analysis log (panel / read-geometry issue). const PANEL_ASSIGNED_FLOOR = 0.5; -// Tag→feature CSV metadata emitted by the prerun's single emit-csv-meta exec (emit_csv_meta.py): the -// column headers (-> the barcode/feature column dropdowns) and each column's distinct values (-> the -// negative-control dropdown, indexed by the chosen feature column). One upload-triggered exec feeds all -// three CSV-derived dropdowns; picking the feature column is then a pure model recompute (no rerun). -// rowCount (total data rows) is emitted by emit-csv-meta so the model can detect a feature barcode that -// appears on more than one row (distinct barcode values < rowCount) — the sample-specific-mapping case -// per_cell_metrics.py guards at the end of the run. Optional so a prerun output predating rowCount still -// parses (the duplicate check then simply skips). -type CsvMeta = { - columns: string[]; - valuesByColumn: Record; - rowCount?: number; -}; +// CsvMeta -- the panel's headers, each header's distinct values, and its row count -- lives in types.ts +// beside the data field that carries it. It feeds the barcode/feature column dropdowns, the +// negative-control dropdown indexed by the chosen feature column, and the duplicate-mapping gate, which +// compares distinct barcode values against rowCount to spot a barcode declared on more than one row. -// mitool tag-stat emits these columns (the CELL/FEATURE/UMI tags — see pattern.ts — plus tag-stat's -// count/totalWeight/unique_ outputs). A user-mapped CSV barcode/feature column that names one of -// these would corrupt the join or crash group_by in per_cell_metrics.py — which guards it too, but only -// after the full mitool chain has run. args() rejects it here so Run is disabled up front. +// mitool tag-stat emits these columns: the CELL/FEATURE/UMI tags (see pattern.ts) plus tag-stat's +// count/totalWeight/unique_ outputs. A user-mapped CSV barcode/feature column naming one of these +// would corrupt the join or crash group_by in per_cell_metrics.py -- which guards it too, but only after +// the full mitool chain has run. args() rejects it here so Run is disabled up front. const RESERVED_TAGSTAT_COLUMNS = new Set([ CELL_TAG, FEATURE_TAG, @@ -86,8 +308,8 @@ function median(xs: number[]): number | undefined { } // sampleId -> display name from the upstream pl7.app/label column whose axis matches the input FASTQ's -// sample axis. Shared by the sampleLabels and analysisLog outputs — kept as a module helper (not one -// output reading another) because each block output is an independent pure function of ctx. +// sample axis. Shared by the sampleLabels and analysisLog outputs. A module helper rather than one output +// reading another, because each block output is an independent pure function of ctx. function resolveSampleLabels( ctx: BlockRenderCtx, ): Record | undefined { @@ -103,7 +325,7 @@ function resolveSampleLabels( try { datasetSampleIds = new Set((JSON.parse(axisKeys0) as unknown[]).map(String)); } catch { - datasetSampleIds = undefined; // malformed → don't scope (fall back to the full map below) + datasetSampleIds = undefined; // malformed: do not scope, fall back to the full map below } } const obj = ctx.resultPool.getData().entries.find((f) => { @@ -126,7 +348,7 @@ function resolveSampleLabels( e[1], ]), ) as Record; - // Restrict to the selected dataset's samples (fall back to the full map if the annotation was missing). + // Restrict to the selected dataset's samples, falling back to the full map if the annotation was missing. return datasetSampleIds ? Object.fromEntries( Object.entries(full).filter(([sampleId]) => datasetSampleIds.has(sampleId)), @@ -135,9 +357,9 @@ function resolveSampleLabels( } // Per-sample QC rows from qcJson (workflow saveFileContent -> inline JSON content, read synchronously), -// filtered to settled samples (qcJson is the last per-sample step, so a present entry = that sample -// finished). Shared by completedSamples / sampleQc / analysisLog. Returns [] when outputs haven't -// settled; callers that need to distinguish "not started" map that to undefined themselves. +// filtered to settled samples: qcJson is the last per-sample step, so a present entry means that sample +// finished. Shared by completedSamples / sampleQc / analysisLog. Returns [] when outputs have not settled. +// A caller that must tell "not started" apart maps that to undefined itself. function parseQcRows(ctx: BlockRenderCtx) { const outputs = ctx.outputs; if (outputs === undefined) return []; @@ -149,21 +371,91 @@ function parseQcRows(ctx: BlockRenderCtx) { return (qcMap?.data ?? []).filter((e) => e.value != null); } -// Tag→feature CSV metadata from the prerun (emit-csv-meta), or undefined until staging has produced it. -// Shared by the two column dropdowns, the control dropdown, and the csvColumnsLoading signal. +// Tag->feature CSV metadata, or undefined until the UI has read the file. Shared by the two column +// dropdowns, the control dropdown, and the csvColumnsLoading signal. +// +// The snapshot is read only while its handle matches the CSV currently picked. That comparison is the +// whole guard against a stale read: every path that swaps the CSV clears the snapshot, and if one ever +// fails to, the mismatch makes the metadata absent rather than wrong. function readCsvMeta(ctx: BlockRenderCtx): CsvMeta | undefined { - return ctx.prerun - ?.resolve({ field: "csvMeta", allowPermanentAbsence: true }) - ?.getDataAsJsonOrUndefined(); + const snap = ctx.data.csvMetaSnapshot; + if (snap === undefined) return undefined; + return snap.handle === ctx.data.tagFeatureCsvHandle ? snap.meta : undefined; +} + +// The grouping columns a rule names, whichever shape it is stored in. A project saved before the rule took +// a list carries `column` rather than `columns`. Reading both here costs one function, where a data +// migration would have to run against every stored project. Every reader goes through this. +export function groupingColumns(rule: GroupingRule | undefined): string[] { + if (rule === undefined || rule.by !== "property") return []; + if (rule.columns !== undefined) return rule.columns.filter((c: string) => c !== ""); + return rule.column ? [rule.column] : []; +} + +// `referenceRungsAvailable` stood here and is gone with the derivation it fed. Which rungs this data could +// serve is still worth SAYING -- the dropdown's option list says it -- but it must not decide anything. A +// helper shared between a display and a projection is how the deciding crept back in. + +/** + * The baseline rung this run is answered under: the scientist's choice, and nothing else. + * + * `what-plays-the-baseline` requires that the scientist selects among the rungs and that nothing selects + * for them. A baseline nobody chose is a methodology nobody knows they used, and two runs of one + * experiment would otherwise be answered by different rules. There is exactly one place a rung comes + * from: `data.referenceSource`. Never derive it from what the panel can serve. + * + * An unselected run IS refused, and undefined is what carries that. There is no bottom rung answering + * every position *unreliable* -- that output is honest and useless, costing what a real run costs while + * looking like a result at a glance. `args()` throws on undefined. This function only reports it. + * + * A stored choice is passed through even where this data cannot serve it, such as a declared tag whose + * values were cleared. The refusal then comes from `args()` where it can see the reason, and from the + * software otherwise. Nothing here writes to `data`, so a choice that becomes serviceable again revives + * on its own. + */ +export function resolveReferenceSource(data: BlockData): ReferenceSource | undefined { + return data.referenceSource; +} + +// A/C/G/T plus N (ambiguous base), case-insensitive. +const isDnaValue = (v: string) => /^[ACGTN]+$/i.test(v); + +// Evidence that the chosen barcode-sequence column does NOT hold nucleotide sequences. Undefined when it +// does, or when the CSV meta has not resolved. Blank cells are ignored rather than counted against the +// column: a trailing empty row is a CSV artefact, not evidence about the contents. A module helper because +// a block output cannot read another output, and two need this -- barcodeAlphabetIssue reports it, and +// barcodeMappingIssue stays silent while it holds, so the two never hand the reader contradictory fixes. +function barcodeAlphabetProblem( + ctx: BlockRenderCtx, +): { offenders: string[]; checked: number; alternative: string | undefined } | undefined { + if (!ctx.data.tagFeatureCsvHandle) return undefined; + const barcodeCol = ctx.data.barcodeSeqColumn; + if (!barcodeCol) return undefined; + const meta = readCsvMeta(ctx); + if (!meta) return undefined; + const values = meta.valuesByColumn?.[barcodeCol]; + if (values === undefined) return undefined; + const clean = (xs: string[]) => xs.map((v) => v.trim()).filter((v) => v !== ""); + const checked = clean(values); + if (checked.length === 0) return undefined; + const offenders = checked.filter((v) => !isDnaValue(v)); + if (offenders.length === 0) return undefined; + // Name a column that would work, if the CSV has one. The mistake is nearly always "picked the ID column + // when the sequences are one over", so naming the alternative saves the user a guess. + const alternative = meta.columns.find((c) => { + if (c === barcodeCol || c === ctx.data.featureNameColumn) return false; + const candidate = clean(meta.valuesByColumn?.[c] ?? []); + return candidate.length > 0 && candidate.every(isDnaValue); + }); + return { offenders, checked: checked.length, alternative }; } // The tag CSV column that looks like it names the dataset's samples, or undefined. A CSV is sample-aware -// when the same barcode maps to different features per sample; the tell is a column whose distinct values -// cover the dataset's sample names. Return the column whose distinct values are a SUPERSET of the dataset -// sample names (preferring exact set-equality, then fewest extra values), excluding the columns already -// bound to the barcode / feature roles. Shared by the suggestedSampleColumn output (UI suggestion) and -// barcodeMappingIssue (names the fix in the duplicate-barcode message) — kept a module helper because a -// block output cannot read another output. undefined until both the CSV meta and sample labels resolve. +// when the same barcode maps to different features per sample, and the tell is a column whose distinct +// values cover the dataset's sample names. Returns the column whose distinct values are a SUPERSET of the +// dataset sample names, preferring exact set-equality then fewest extra values, excluding the columns +// already bound to the barcode / feature roles. Shared by the suggestedSampleColumn output and +// barcodeMappingIssue. Undefined until both the CSV meta and sample labels resolve. function suggestSampleColumn(ctx: BlockRenderCtx): string | undefined { const meta = readCsvMeta(ctx); const labels = resolveSampleLabels(ctx); @@ -188,7 +480,7 @@ function suggestSampleColumn(ctx: BlockRenderCtx): string if (!isSuperset) continue; const exact = values.size === datasetNames.size; const extra = values.size - datasetNames.size; - // Prefer exact set-equality; among equals, prefer the fewest extra values. + // Prefer exact set-equality. Among equals, prefer the fewest extra values. if ( best === undefined || (exact && !best.exact) || @@ -199,9 +491,70 @@ function suggestSampleColumn(ctx: BlockRenderCtx): string return best?.col; } +// GraphMaker keeps its own chart configuration here, one per plot: a chart's saved state is about that +// chart, so sharing one would make picking an axis on the score plot move the background plot too. Shared +// between `init` and the v4 -> v5 migration so a migrated project opens on the same chart a new one does. +const INITIAL_GRAPH_STATES = { + scoreDistributionGraphState: { title: "Spread of the run's scores", template: "line" }, + referenceReadingGraphState: { title: "Reference reading across cells", template: "line" }, + fittedBackgroundGraphState: { title: "Fitted background per tag", template: "dots" }, +} as const satisfies Pick< + BlockData, + "scoreDistributionGraphState" | "referenceReadingGraphState" | "fittedBackgroundGraphState" +>; + +// v6 data shape: everything the current shape has except the undeclared-barcode grid's state, which +// arrived with that table. +type BlockDataV6 = Omit; + +// v5 data shape: v6 without the reagent grid's state, which arrived with the reagent table. +type BlockDataV5 = Omit; + +// v4 data shape: v5 without the three GraphMaker states, which arrived with the distribution plots. +type BlockDataV4 = Omit< + BlockDataV5, + "scoreDistributionGraphState" | "referenceReadingGraphState" | "fittedBackgroundGraphState" +>; + +// v3 data shape: the reading's parameters, with the three grid states the two removed result views owned. +// v4 replaced them with the punchcard's own state. `punchcardIdentities` is dead on the right-hand side of +// the Omit and harmless on the left. +type BlockDataV3 = Omit & { + verdictTableState: PlDataTableStateV2; + antigenQcTableState: PlDataTableStateV2; + panelMismatchTableState: PlDataTableStateV2; +}; + +// v2 data shape: the preset selector and pattern string, with the dominance-era parameters still on it. +// The dominant-feature readout, the off-target designation and the specificity score they fed are gone +// from per_cell_metrics.py, so nothing consumes these three any more. +type BlockDataV2 = Omit< + BlockDataV3, + | "datasetRef" + | "roleColumn" + | "referenceValues" + | "referenceSource" + | "panelReferenceMinMembers" + | "distributionMinCells" + | "countFloor" + | "boundCutoff" + | "minVotingCells" + | "minAgreement" + | "gateThreshold" + | "grouping" + | "contendingGroups" + | "verdictTableState" + | "antigenQcTableState" + | "panelMismatchTableState" +> & { + dominanceThreshold: number; + offtargetProperty?: string; + offtargetValues?: string[]; +}; + // v1 (pre-preset) data shape: read geometry was three explicit length fields. v2 replaces them with a -// preset selector + a mitool tag-pattern string (see model/src/pattern.ts, model/src/presets). -type BlockDataV1 = Omit & { +// preset selector and a mitool tag-pattern string (see model/src/pattern.ts, model/src/presets). +type BlockDataV1 = Omit & { cellLen: number; umiLen: number; featureLen: number; @@ -209,9 +562,9 @@ type BlockDataV1 = Omit & { const dataModel = new DataModelBuilder() .from("v1") - .migrate("v2", ({ cellLen, umiLen, featureLen, ...rest }) => { - // The shipped default (16/10/15) maps to the fixed BEAM preset; any other geometry maps to the - // generic preset carrying the assembled pattern (offset 0 — the only layout the v1 UI could express). + .migrate("v2", ({ cellLen, umiLen, featureLen, ...rest }) => { + // The shipped default (16/10/15) maps to the fixed BEAM preset. Any other geometry maps to the generic + // preset carrying the assembled pattern -- offset 0, the only layout the v1 UI could express. const isBeamDefault = cellLen === 16 && umiLen === 10 && featureLen === 15; return isBeamDefault ? { ...rest, presetId: "tenx-beam" } @@ -227,15 +580,95 @@ const dataModel = new DataModelBuilder() }), }; }) + // v2 -> v3: the dominance parameters go and the reading's own arrive. The three dropped fields are + // dropped rather than carried, because a field kept "just in case" still travels in the args hash and + // stales the block on an edit that changes no computation. The new numeric parameters are seeded with + // the shipped defaults, so a migrated project renders the same run a fresh one would. A parameter left + // undefined here would reach the CLI as its argparse default -- the same number arrived at without + // anyone choosing it. + .migrate( + "v3", + ({ dominanceThreshold: _d, offtargetProperty: _p, offtargetValues: _v, ...rest }) => ({ + ...rest, + countFloor: DEFAULT_COUNT_FLOOR, + boundCutoff: DEFAULT_BOUND_CUTOFF, + minVotingCells: DEFAULT_MIN_VOTING_CELLS, + panelReferenceMinMembers: DEFAULT_PANEL_REFERENCE_MIN_MEMBERS, + distributionMinCells: DEFAULT_DISTRIBUTION_MIN_CELLS, + verdictTableState: createPlDataTableStateV2(), + antigenQcTableState: createPlDataTableStateV2(), + panelMismatchTableState: createPlDataTableStateV2(), + }), + ) + // v3 -> v4: the flat verdict table and the quality-report tables are gone as VIEWS, and the punchcard + // takes their place. The three grid states go with them rather than being carried: a saved column set or + // filter is meaningful only against the frame it was saved on. The punchcard's own state starts fresh, + // on the whole panel. + // + // What the removed pages showed is still EMITTED: the verdicts and the run's measurements are both + // artifacts `verdict-block-interface` obliges this block to produce, and dropping a view does not + // release it from producing them. + // + // The Run quality page's two grid states are `runQualityTableState` / `runQualityMismatchTableState`, + // NOT the two keys stripped here. Never reuse a stripped key: a saved column set and filter from a + // removed view would reappear under a grid it was never saved against. + .migrate( + "v4", + ({ verdictTableState: _v, antigenQcTableState: _q, panelMismatchTableState: _m, ...rest }) => ({ + ...rest, + punchcardTableState: createPlDataTableStateV2(), + }), + ) + // v4 -> v5: the three GraphMaker states arrive. `init` seeds a NEW project only, so a project created + // before these keys existed carries none of them, and GraphMaker renders nothing when its model is + // undefined. A migration is the only route that reaches a stored project. + .migrate("v5", (data) => ({ + ...data, + ...INITIAL_GRAPH_STATES, + })) + // v5 -> v6: the reagent grid's state arrives. `init` seeds a NEW project only, and a + // `PlAgDataTableV2` bound to an undefined state renders nothing and reports no error. + .migrate("v6", (data) => ({ + ...data, + reagentTableState: createPlDataTableStateV2(), + })) + // v6 -> v7: the undeclared-barcode grid's state arrives, for the same reason the reagent + // grid's did one version earlier. + .migrate("v7", (data) => ({ + ...data, + undeclaredBarcodesTableState: createPlDataTableStateV2(), + })) + // v7 -> v8: the fitted-background plot gains a sample facet. `:default-options` seeds a plot with + // no saved state and never overwrites one, so a reader who opened that tab before keeps the pooled + // single-panel chart. The fit runs per (tag, sample) and one panel reads every sample's fits as one + // population, so this resets that one plot's state. The other two are left alone. + .migrate("v8", (data) => ({ + ...data, + fittedBackgroundGraphState: INITIAL_GRAPH_STATES.fittedBackgroundGraphState, + })) .init(() => ({ - dominanceThreshold: 0.6, - runMode: "full" as const, // full run by default; "dry" = read-limited Preview - // Default preset = the geometry the block shipped with: 10x 5' v2 BEAM (16 / 10 / 15). + runMode: "full" as const, // full run by default. "dry" = read-limited Preview + // Default preset: the geometry the block shipped with, 10x 5' v2 BEAM (16 / 10 / 15). presetId: "tenx-beam", cellWhitelist: "", // de-novo CELL correction by default defaultBlockLabel: "", + // The reading's parameters. minAgreement and gateThreshold are deliberately absent: both are off by + // default, and off means absent rather than zero (see the args projection). + countFloor: DEFAULT_COUNT_FLOOR, + boundCutoff: DEFAULT_BOUND_CUTOFF, + minVotingCells: DEFAULT_MIN_VOTING_CELLS, + panelReferenceMinMembers: DEFAULT_PANEL_REFERENCE_MIN_MEMBERS, + distributionMinCells: DEFAULT_DISTRIBUTION_MIN_CELLS, tableState: createPlDataTableStateV2(), qcSummaryTableState: createPlDataTableStateV2(), + punchcardTableState: createPlDataTableStateV2(), + // These names avoid the two keys the v3 -> v4 migration strips. Both were added before any surviving + // project was created, so no migration carries them. + runQualityTableState: createPlDataTableStateV2(), + ...INITIAL_GRAPH_STATES, + runQualityMismatchTableState: createPlDataTableStateV2(), + reagentTableState: createPlDataTableStateV2(), + undeclaredBarcodesTableState: createPlDataTableStateV2(), })); export const platforma = BlockModelV3.create(dataModel) @@ -243,14 +676,32 @@ export const platforma = BlockModelV3.create(dataModel) if (!data.fbFastqRef) throw new Error("Select the feature-barcode FASTQ"); if (!data.tagFeatureCsvHandle) throw new Error("Upload the tag→feature CSV"); if (!data.barcodeSeqColumn) throw new Error("Select the barcode-sequence column in the CSV"); - if (!data.featureNameColumn) throw new Error("Select the feature-name column in the CSV"); - // The barcode-sequence and feature-name roles must map to different CSV columns. The Python guards - // this too (per_cell_metrics.py), but only after the full mitool chain runs; rejecting it here - // disables Run up front instead of burning the pipeline to fail at the end. + if (!data.featureNameColumn) throw new Error("Select the panel column naming each antigen"); + // REQUIRED. The block used to run without one and emit counts, per-cell values and per-sample quality + // with no verdicts at all. Verdicts are what the block is for, so that run was a stage sold as a + // deliverable: nobody adds a binding-profiling block to obtain read statistics. + // The panel rung is retired, and a project stored under it is refused rather than moved. `args()` is + // where a settings-knowable refusal belongs: the scientist changes the configuration instead of waiting + // for a run to tell them. + if (data.referenceSource === "panel") + throw new Error( + "The baseline source \u201CThe panel's own readings\u201D is no longer available. It read each " + + "count against the median of the cell's other tags, which needs a decision at the clonotype " + + "rather than at the cell, and every other rule here reads a cell first. Choose a declared " + + "baseline tag, or each tag's own distribution, under \u201CBaseline source\u201D.", + ); + if (!data.datasetRef) + throw new Error( + "Select the single-cell V(D)J dataset the verdicts are about. Every verdict is about one " + + "clonotype, so the block cannot produce any without it.", + ); + // The barcode-sequence and feature-name roles must map to different CSV columns. The Python guards this + // too, but only after the full mitool chain runs. Rejecting it here disables Run up front instead of + // burning the pipeline to fail at the end. if (data.barcodeSeqColumn === data.featureNameColumn) throw new Error("Barcode-sequence and feature-name columns must be different"); - // Reject a CSV column that collides with a reserved tag-stat column up front (same reason as the - // barcode≠feature guard above: the Python guards it too, but only after the full mitool chain runs). + // Reject a CSV column colliding with a reserved tag-stat column up front, for the same reason as the + // barcode-not-feature guard above. for (const [role, col] of [ ["Barcode-sequence", data.barcodeSeqColumn], ["Feature-name", data.featureNameColumn], @@ -260,11 +711,10 @@ export const platforma = BlockModelV3.create(dataModel) `${role} column "${col}" collides with a reserved tag-stat column; pick another`, ); } - // Optional combine-mode column: its values are per-feature modes ("sum"/"all"), so it must be its - // OWN CSV column — distinct from the barcode-sequence and feature-name roles, and not a reserved - // tag-stat column. Python guards this too, but only after the mitool chain runs; reject up front so a - // mis-picked column (e.g. the barcode column, whose values are DNA sequences) disables Run with a - // clear message instead of failing the pipeline at the end. + // Optional combine-mode column: its values are per-feature modes ("sum"/"all"), so it must be its OWN + // CSV column -- distinct from the barcode-sequence and feature-name roles, and not a reserved tag-stat + // column. Python guards this too, but only after the mitool chain runs. Reject up front so a mis-picked + // column, such as the barcode column of DNA sequences, disables Run with a clear message. if (data.combineColumn) { if ( data.combineColumn === data.barcodeSeqColumn || @@ -282,26 +732,26 @@ export const platforma = BlockModelV3.create(dataModel) // clear message rather than start a run with no reads to cap. if (data.runMode === "dry" && (data.limitInput == null || data.limitInput < 1)) throw new Error("Enter a read limit (≥ 1) for Preview mode, or switch to a full run"); - // Read geometry: resolve the selected preset to its effective pattern (fixed preset owns it; the - // generic preset carries it in data.pattern), validate it loosely (the required CELL/UMI/FEATURE - // tags + R2 capture must be present — the workflow's refine-tags/tag-stat reference them by name; - // anything else is passed to mitool verbatim), then hand the string to the workflow directly. + // Read geometry: resolve the selected preset to its effective pattern (a fixed preset owns it, the + // generic preset carries it in data.pattern), validate it loosely, then hand the string to the + // workflow. Loose means only the CELL/UMI/FEATURE tags and the R2 capture must be present, since + // refine-tags/tag-stat reference them by name. Anything else goes to mitool verbatim. const preset = getPreset(data.presetId); if (!preset) throw new Error("Select a read-geometry preset"); const pattern = preset.userConfigurable ? (data.pattern ?? "") : preset.pattern; const patternError = validatePattern(pattern); if (patternError) throw new Error(patternError); - // Sample-aware mapping (optional): when a sample column is chosen, the per-sample workflow body - // filters the CSV to its own sample's rows, so pass the column name + the sampleId→name snapshot it - // needs to translate its iteration key. The snapshot is taken on the same gesture that sets the - // column (MainPage.setSampleColumn); require it here so a stale/half-set state disables Run. + // Sample-aware mapping, optional: when a sample column is chosen, the per-sample workflow body filters + // the CSV to its own sample's rows. Pass the column name and the sampleId->name snapshot it needs to + // translate its iteration key. That snapshot is taken on the same gesture that sets the column + // (MainPage.setSampleColumn). Require it here so a half-set state disables Run. const sampleAware = !!data.sampleColumn; if (sampleAware) { if (!data.sampleLabelSnapshot || Object.keys(data.sampleLabelSnapshot).length === 0) throw new Error("Re-select the sample column (sample labels not captured)"); - // Block Run when a dataset sample has no rows in the CSV's sample column — it would silently get - // no features. Gate purely from the snapshots taken when the column was picked (args is data-only). + // Block Run when a dataset sample has no rows in the CSV's sample column -- it would silently get no + // features. Gate purely from the snapshots taken when the column was picked, since args is data-only. const csvValues = new Set(data.sampleColumnValues ?? []); const missing = Object.values(data.sampleLabelSnapshot).filter((n) => !csvValues.has(n)); if (missing.length > 0) @@ -312,39 +762,217 @@ export const platforma = BlockModelV3.create(dataModel) ); } + // A barcode on more than one row with no sample column is not a warning. It is a run that will stop: + // per_cell_metrics.py refuses to map one barcode to two antigens, and it refuses at the END, after + // every sample has been parsed. Blocking Run here costs the user a second instead of the whole run. + // Both numbers are snapshots taken when the barcode column was picked. Absent means the meta had not + // resolved then, and the gate stays out of the way rather than guessing. + if ( + !sampleAware && + data.panelRowCount !== undefined && + data.panelBarcodeDistinct !== undefined && + data.panelBarcodeDistinct < data.panelRowCount + ) + throw new Error( + `The tag CSV has ${data.panelRowCount} rows but only ${data.panelBarcodeDistinct} distinct ` + + `barcodes, so one barcode maps to more than one antigen. Set the sample column if the CSV ` + + `lists each barcode once per sample, or remove the duplicate rows.`, + ); + + // The reading's own parameters. The single-cell V(D)J dataset is deliberately NOT required: without it + // the block still emits the tag counts, the per-cell scalars, the panel-versus-reads check and the + // per-sample QC, none of which need a clonotype set. A missing input narrows what can be answered. + if (data.countFloor < 0) throw new Error("The count floor cannot be negative"); + if (data.boundCutoff < 0 || data.boundCutoff > 100) + throw new Error("The bound cutoff is a score between 0 and 100"); + if (data.minVotingCells < 1) throw new Error("At least one cell must vote"); + // A role column names WHERE each tag's role is written. The role values are what actually marks one. + // Named alone the column is inert: emit_verdicts.py reads it only under `if args.role_column and + // reference_values`, so it is validated, recorded in the run meta, and changes no number. The run then + // reads against the panel's own readings while the form says a baseline tag is declared -- a wrong + // answer wearing the look of a configured one. Requiring the values costs no expressiveness: a panel + // that declares no baseline leaves this column blank. + // + // A baseline is required, and a run without one does not happen. The alternative is a full punchcard + // of *unreliable*, which costs what a real run costs and looks like a result at a glance. + // + // Refused HERE, before anything is read, because which rung was chosen and whether a baseline tag is + // declared are properties of the settings. The scientist changes the configuration instead of waiting + // for a run to tell them, and the message names the condition that failed. + if (!data.referenceSource) + throw new Error( + "Choose what the counts are read against, under “Baseline source”. Every verdict is a reading " + + "against a baseline, so a run without one produces no answers at all. Which baselines this " + + "panel can serve is listed with each option.", + ); + if (data.referenceSource === "declared" && !data.roleColumn) + throw new Error( + "The declared-baseline option reads every count against one tag marked as the baseline, and no " + + "panel column is set to say which tag that is. Choose the column under “Role column”, or pick " + + "a different baseline source.", + ); + // + // The panel-size condition is NOT checked here and cannot be: it needs the count of distinct barcodes, + // which lives in the CSV metadata, and this projection must not read that (see the note on the staging + // projection below). The software refuses it instead, naming the same condition, and the + // `referenceSources` output marks the option unserviceable. + // + // The cell-count condition is not checked anywhere before the run, and cannot be: whether a sample + // holds enough cells whose counts separate is a property of the DATA. A run on that rung proceeds and + // reports afterwards that no baseline could be established. + // + // Scoped to the DECLARED rung, and that scope is load-bearing. A role column only names a baseline + // tag for the rung that reads one, so under any other rung it changes nothing and must not refuse a + // run. Unscoped, a project that once chose a role column and later moved to the tag-distribution + // rung kept the stale column, threw here, and greyed out Run -- while the Role column field is + // hidden under that rung, so the value doing it was invisible and unclearable. + if (data.referenceSource === "declared" && data.roleColumn && !data.referenceValues?.length) + throw new Error( + `The panel column "${data.roleColumn}" declares each tag's role, but no value of it is marked ` + + `as the baseline, so the column changes nothing. Under "Baseline value", choose the value that ` + + `marks it, or change the baseline source.`, + ); + // Every panel column the verdict settings name, each with the label the user sees. Two different things + // can be wrong with one of these, so both checks below walk this same list. Check each grouping column + // on its own: a grouping may name several, and joining them would compare "Identity, Channel" against + // the panel's headers, match nothing, throw here, and take the whole block to Limbo, refs and all. + const named: [string, string | undefined][] = [ + ["Baseline role", data.roleColumn], + ...groupingColumns(data.grouping).map((c): [string, string] => ["Grouping", c]), + ]; + + // First: a column the panel reader consumes as a KEY is not a property column, so naming one here ends + // the run at the exec. emit_verdicts.py raises on a grouping column the panel does not declare, and on + // a role column wherever role values are set. Where they are NOT set it raises nothing and the baseline + // falls back to the panel's own readings -- a wrong answer rather than no answer. + // + // The way in is reassigning a key column WITHIN one panel file. The settings dropdowns stop offering + // it, the pick already stored survives, and the field reads empty while the data is not. Checked + // against data rather than the header snapshot, because a key column IS a real header. + const keyColumns: [string, string | undefined][] = [ + ["barcode sequence", data.barcodeSeqColumn], + ["sample", data.sampleColumn], + ]; + for (const [role, column] of named) { + if (!column) continue; + for (const [key, keyColumn] of keyColumns) { + if (column === keyColumn) + throw new Error( + `The ${role} column "${column}" is also the ${key} column. The panel reader consumes that ` + + `column as a key rather than a property, so choose a different column for one of them.`, + ); + } + } + + // Second: a role column or a grouping column the panel does not carry at all ends the whole run at the + // exec too, and the user meets that as a dead run with no hint of which setting caused it. The check is + // against the headers snapshotted when the column was picked -- args reads data only -- so a panel swap + // that leaves the pick behind disables Run with a message naming the column instead. + const panelColumns = data.panelColumnSnapshot; + if (panelColumns?.length) { + for (const [role, column] of named) { + if (column && !panelColumns.includes(column)) + throw new Error( + `The ${role} column "${column}" is not in the uploaded panel file. Select a column from the new panel.`, + ); + } + } + + // Contending groups, canonicalised here rather than in the editor: the args value is a cache key, so + // the same declaration written in a different order must produce the same string or the block goes + // stale and re-runs the whole reading for nothing. A group of fewer than two members is dropped -- one + // identity contends with nothing, and an empty group is that same case. + const contendingGroups = (data.contendingGroups ?? []) + .map((group) => [...new Set(group)].sort()) + .filter((group) => group.length > 1) + .sort((a, b) => a.join(" ").localeCompare(b.join(" "))); + return { fbFastqRef: data.fbFastqRef, tagFeatureCsvHandle: data.tagFeatureCsvHandle, barcodeSeqColumn: data.barcodeSeqColumn, featureNameColumn: data.featureNameColumn, - controlFeature: data.controlFeature, - // canonicalize + clamp to the 0.5 floor - dominanceThreshold: Math.max(DOMINANCE_FLOOR, data.dominanceThreshold ?? 0.6), // Optional multi-barcode antigen combine mode. combineColumn names a tag-CSV column giving each - // feature's mode (sum = OR, the default; all = AND, feature called only when every member barcode - // fires). Projected only when set so the workflow default (every feature OR) is untouched otherwise. - // minUmi is the AND per-barcode "fired" floor (integer >= 1; default 1 in the workflow/Python), - // projected only alongside combineColumn — the workflow passes --min-umi only with --combine-col, - // so without a combine mode it would only stale the block with no computational effect. + // feature's mode: sum = OR, the default, and all = AND, where a feature is called only when every + // member barcode fires. Projected only when set, so the workflow default stands otherwise. minUmi is + // the AND per-barcode "fired" floor, an integer >= 1 defaulting to 1, projected only alongside + // combineColumn because the workflow passes --min-umi only with --combine-col. ...(data.combineColumn ? { combineColumn: data.combineColumn } : {}), ...(data.combineColumn && typeof data.minUmi === "number" && data.minUmi >= 1 ? { minUmi: Math.round(data.minUmi) } : {}), - // Optional off-target designation (F2). offtargetProperty names an imported per-feature property - // column (e.g. antigen_class); offtargetValues are that column's values marking a feature as - // off-target. Such features are excluded from the dominant call (like the control) and turn on the - // cross-reactive label. Projected only when both are set, so the dominant call is unchanged - // otherwise (empty column / values → workflow leaves the rule untouched). - ...(data.offtargetProperty && data.offtargetValues && data.offtargetValues.length > 0 - ? { - offtargetProperty: data.offtargetProperty, - // Sort + dedup: the Python treats these as a set, so canonicalize here so re-selecting the - // same values in a different order yields the same args hash (no needless stale / re-run). - offtargetValues: [...new Set(data.offtargetValues)].sort(), - } - : {}), - // Preview: cap reads only in dry mode; a full run omits it (all reads). Projected only when dry, so - // toggling back to full changes the args hash and re-runs on the complete input. + // The aggregate-barcode detection knobs. Undefined projects as undefined, and the workflow's own + // default stands. Passed through raw, never gated on a positivity check: unlike minAgreement/ + // gateThreshold below, none of these three is an "off means absent" switch. + aggregateBarcodeIqrMultiplier: data.aggregateBarcodeIqrMultiplier, + aggregateBarcodeMinUmiThreshold: data.aggregateBarcodeMinUmiThreshold, + aggregateBarcodeTopN: + typeof data.aggregateBarcodeTopN === "number" + ? Math.round(data.aggregateBarcodeTopN) + : undefined, + // --- the binding reading --- + // The dataset anchor. Absent is a legitimate state, not a half-filled form, so it projects as absent + // and the workflow skips the verdict stage alone. + datasetRef: data.datasetRef, + // Empty and absent are the same claim for both of these, so an empty selection projects as absent + // rather than as "" / [] -- two spellings of one request would otherwise be two cache keys. + // + // Sent only for the rung that reads one. A project that once chose a role column and later moved + // to another rung keeps the stale value in `data`, and projecting it would put it in the argument + // vector, where it changes the cache key and re-runs the reading for a setting nothing consults. + roleColumn: data.referenceSource === "declared" ? data.roleColumn || undefined : undefined, + // Sorted and de-duplicated: the Python reads these as a set, so re-picking the same values in a + // different order must not re-run the reading. + referenceValues: + data.referenceSource === "declared" && data.referenceValues?.length + ? [...new Set(data.referenceValues)].sort() + : undefined, + // Always concrete, because the software has no default: --reference-source is required there, and + // nothing below this line picks a rung. An unselected choice reaches the run as "none", which is a + // rung rather than a refusal. `served_source` only ever drops a rung to none, never substitutes a + // different one, so what this sends is what the run is answered under. The run record carries both. + referenceSource: resolveReferenceSource(data), + panelReferenceMinMembers: Math.round(data.panelReferenceMinMembers), + distributionMinCells: Math.round(data.distributionMinCells), + countFloor: Math.round(data.countFloor), + // A switch, so off means ABSENT rather than false -- the same rule the two thresholds below follow. + // It keeps the args vector of every project that never touched it byte-identical. + boundCutoff: data.boundCutoff, + minVotingCells: Math.round(data.minVotingCells), + // Off by default, and off means ABSENT: a minimum agreement of 0 passes every majority instead of + // skipping the check, and a gate of 0 sets aside every cell instead of gating none. Both are + // different claims from "off", so neither is projected as zero. + minAgreement: + typeof data.minAgreement === "number" && data.minAgreement > 0 + ? data.minAgreement + : undefined, + gateThreshold: + typeof data.gateThreshold === "number" && data.gateThreshold > 0 + ? Math.round(data.gateThreshold) + : undefined, + // A rule over declared panel properties, never a tag->identity map. Absent means one identity per tag, + // which is the reading's own default, so no hand-built { by: "tag" } is sent in its place. Normalised + // to a list here so the software receives one shape. It reads the older `column` too, but a run + // record naming `columns` is what every future reader should see. + grouping: + data.grouping?.by === "property" + ? { by: "property" as const, columns: groupingColumns(data.grouping) } + : data.grouping, + contendingGroups: contendingGroups.length > 0 ? contendingGroups : undefined, + // The four inherited lines. Each undefined projects as undefined, and emit_verdicts.py's own + // shipped default stands -- passed through raw rather than gated on positivity, since 0.0 is a + // real published threshold (usableReadError) and not an "off" state. + cellBarcodeValidWarn: data.cellBarcodeValidWarn, + cellBarcodeValidError: data.cellBarcodeValidError, + readsPerCellWarn: data.readsPerCellWarn, + aggregateBarcodeWarn: data.aggregateBarcodeWarn, + aggregateBarcodeError: data.aggregateBarcodeError, + undeclaredBarcodeWarn: data.undeclaredBarcodeWarn, + undeclaredBarcodeError: data.undeclaredBarcodeError, + usableReadWarn: data.usableReadWarn, + usableReadError: data.usableReadError, + // Preview: cap reads only in dry mode. A full run omits it. Projected only when dry, so toggling back + // to full changes the args hash and re-runs on the complete input. ...(data.runMode === "dry" && data.limitInput ? { limitInput: Math.round(data.limitInput) } : {}), @@ -353,11 +981,11 @@ export const platforma = BlockModelV3.create(dataModel) ...(sampleAware ? { sampleColumn: data.sampleColumn, sampleLabels: data.sampleLabelSnapshot } : {}), - // CELL whitelist: "" = de-novo CELL correction (default; no external whitelist). + // CELL whitelist: "" = de-novo CELL correction, with no external whitelist. cellWhitelist: data.cellWhitelist ?? "", - // Optional mitool resource overrides (Advanced Settings). Project only positive integers so a blank - // or zero field falls through to the workflow defaults (4 CPUs; formula-sized RAM) instead of - // sending a meaningless request or staling the block on an empty edit. + // Optional mitool resource overrides (Advanced Settings). Project only positive integers so a blank or + // zero field falls through to the workflow defaults (4 CPUs, formula-sized RAM) instead of sending a + // meaningless request or staling the block on an empty edit. ...(typeof data.perProcessCPUs === "number" && data.perProcessCPUs >= 1 ? { perProcessCPUs: Math.round(data.perProcessCPUs) } : {}), @@ -366,22 +994,26 @@ export const platforma = BlockModelV3.create(dataModel) : {}), }; }) - // Staging depends only on the CSV: emit-csv-meta emits every column's values in one exec, so the - // negative-control dropdown no longer needs a rerun when the feature column changes (the model indexes - // the already-emitted map). featureNameColumn is deliberately NOT a prerun arg — and neither is - // fbFastqRef: the CSV metadata is independent of the FASTQ, so keying staging on it would re-run the - // emit-csv-meta step and blank the column dropdowns (csvColumnsLoading → tagMappingDisabled) every time - // the FASTQ changes or a PlRef re-resolves on reload. Key on the CSV alone. + // Staging depends only on the CSV. It imports the file and exports the blob, and nothing else. + // featureNameColumn is deliberately NOT a prerun arg, and neither is fbFastqRef: the panel is independent + // of the FASTQ, so keying staging on it would re-import the CSV on every FASTQ change or PlRef re-resolve. + // + // THIS PROJECTION MUST NOT GROW TO INCLUDE csvMetaSnapshot, csvImportError, OR ANYTHING DERIVED FROM + // THEM. The UI reads the panel from the blob this staging exports and writes the result into + // csvMetaSnapshot. A data write re-renders staging only when the canonical JSON of THIS projection + // changes -- that comparison in pl-middle-layer's setStates is what gates renderStagingFor -- so today + // that write cannot re-run staging. Add the snapshot here and it can: the write re-renders staging, which + // re-exports the blob, which re-triggers the write. And because a staging re-render calls resetStaging + // first, every turn of that loop would discard the uploaded CSV, not merely waste work. .prerunArgs((data) => ({ tagFeatureCsvHandle: data.tagFeatureCsvHandle, })) - // NOTE on enrichments (.enriches): intentionally NOT declared. `.enriches(args => PlRef[])` is for a - // block that produces columns sharing the key space of a ref it holds (clonotype-browser enriches its - // inputAnchor; cell-browser enriches its countsRef). This block introduces a NEW cell/feature key - // space [sampleId, cellId, featureId] off a FASTQ input, and holds no ref to the downstream VDJ - // dataset it would enrich — so there is nothing to enrich here. VDJ Multiomic Integration discovers - // these columns under its VDJ anchor via the pl7.app/sc/cellLinker (linker traversal), not via - // enrichment. Revisit only if the live cross-block discovery check shows otherwise. + // Enrichments (.enriches): intentionally NOT declared. `.enriches(args => PlRef[])` is for a block that + // produces columns sharing the key space of a ref it holds (clonotype-browser enriches its inputAnchor, + // cell-browser its countsRef). This block introduces a NEW cell/feature key space + // [sampleId, cellId, featureId] off a FASTQ input and holds no ref to the downstream VDJ dataset, so + // there is nothing to enrich. VDJ Multiomic Integration discovers these columns under its VDJ anchor + // through the pl7.app/sc/cellLinker, not through enrichment. // feature-barcode FASTQ options (file-valued sequencing columns, fastq / fastq.gz) .output("fastqOptions", (ctx) => @@ -395,9 +1027,41 @@ export const platforma = BlockModelV3.create(dataModel) ); }), ) - // Suggested block label for the sidebar subtitle: " / - ", derived from - // the current inputs. Computed here (not in .subtitle) because the subtitle context has no result - // pool; a UI watchEffect copies this into data.defaultBlockLabel. Each part is dropped until set. + // The single-cell V(D)J dataset the verdicts are keyed by: columns on [sampleId, scClonotypeKey] flagged + // as anchors -- the same query VDJ Multiomic Integration uses, so the two blocks offer the user the same + // list. There is deliberately no linkerOptions beside it: the cell linker carries pl7.app/isLinkerColumn + // and is hidden in tables, so it is not a column a user can pick, and the workflow resolves it from this + // anchor by name. + .output("datasetOptions", (ctx) => + ctx.resultPool.getOptions([ + { + axes: [{ name: "pl7.app/sampleId" }, { name: "pl7.app/vdj/scClonotypeKey" }], + annotations: { "pl7.app/isAnchor": "true" }, + }, + ]), + ) + // The identities the contending-groups editor picks from, live from the uploaded panel. An identity is + // whatever the grouping rule groups tags by: the tag itself under the default per-tag rule, and the + // property's value under a property rule -- so the option list is the distinct values of the barcode + // column or of the chosen property column. Under the per-tag rule the ids ARE the barcode sequences and + // are their own labels, because the panel metadata is column-wise and carries no tag->name pairing. + // Retentive so the editor does not blank on a rerun. + // + // This output exists so that only the USER'S PICKS are ever written to data. A watcher copying this list + // into data would make the output depend on data derived from it, and two open clients would race. + .retentiveOutput("identityOptions", (ctx): { value: string; label: string }[] => { + const grouped = groupingColumns(ctx.data.grouping); + // Nothing to offer under a grouping on SEVERAL columns. An identity is then the combination of their + // values, and the prerun CSV meta is column-wise, with no pairing between columns. Crossing them would + // invent combinations the panel never declared, and a fabricated identity is worse than none. + if (grouped.length > 1) return []; + const column = grouped[0] ?? ctx.data.barcodeSeqColumn; + if (!column) return []; + return (readCsvMeta(ctx)?.valuesByColumn?.[column] ?? []).map((v) => ({ value: v, label: v })); + }) + // Suggested block label for the sidebar subtitle: " / - ", derived from the + // current inputs. Computed here rather than in .subtitle because the subtitle context has no result pool. + // A UI watchEffect copies this into data.defaultBlockLabel. Each part is dropped until set. .output("suggestedBlockLabel", (ctx): string | undefined => { const parts: string[] = []; const ref = ctx.data?.fbFastqRef; @@ -420,32 +1084,23 @@ export const platforma = BlockModelV3.create(dataModel) } if (parts.length === 0) return undefined; // The default subtitle must never render with dots (Stan's request, S1). Periods come from a dotted - // dataset/file label; the " / " and " - " separators are a slash and hyphen, not periods, so - // stripping "." leaves them intact. Replace periods with spaces and collapse the doubles they create. - // A subtitle the user types in the sidebar is not routed through this output, so overrides are safe. + // dataset or file label. The " / " and " - " separators are a slash and a hyphen, so stripping "." + // leaves them intact. Replace periods with spaces and collapse the doubles that creates. A subtitle the + // user types in the sidebar does not pass through this output, so an override is safe. return parts.join(" / ").replace(/\./g, " ").replace(/ {2,}/g, " ").trim(); }) - // Negative-control dropdown options: the distinct values of the chosen feature-name - // column, from the prerun's emit-csv-meta valuesByColumn map. No rerun on column change — the map - // already carries every column's values, so picking the feature column just re-indexes here. - // Retentive avoids a flicker to [] on rerun; empty until the CSV is uploaded and staging completes. - .retentiveOutput("controlOptions", (ctx): { value: string; label: string }[] => { - const col = ctx.data.featureNameColumn; - const names = col ? (readCsvMeta(ctx)?.valuesByColumn?.[col] ?? []) : []; - return names.map((name) => ({ value: name, label: name })); - }) - // CSV column headers (from the prerun emit-csv-meta step) → the barcode/feature column dropdowns - // Retentive so the dropdowns don't blank on rerun; empty until the CSV is uploaded + parsed. + // The panel's column headers, feeding the barcode/feature column dropdowns. Retentive so the dropdowns do + // not blank on rerun. Empty until the panel has been read. .retentiveOutput("csvColumnOptions", (ctx): { value: string; label: string }[] => (readCsvMeta(ctx)?.columns ?? []).map((c) => ({ value: c, label: c })), ) - // Every CSV column's distinct values (from the prerun emit-csv-meta step). The UI reads this when the - // sample column is picked, to snapshot that column's values into data (args gates Run on them). + // Every panel column's distinct values. The UI reads this when the sample column is picked, to snapshot + // that column's values into data, which args gates Run on. .retentiveOutput( "csvValuesByColumn", (ctx): Record => readCsvMeta(ctx)?.valuesByColumn ?? {}, ) - // Sample-aware mapping sanity check (UI warning only; args is the authoritative gate). When a sample + // Sample-aware mapping sanity check, a UI warning only -- args is the authoritative gate. When a sample // column is chosen, compare its CSV values against the dataset's sample names: flag dataset samples // absent from the CSV (they would get no features) and CSV values matching no dataset sample (typos). .retentiveOutput("sampleMappingWarning", (ctx): string[] | undefined => { @@ -459,68 +1114,133 @@ export const platforma = BlockModelV3.create(dataModel) const fmt = (xs: string[]) => `${xs.slice(0, 5).join(", ")}${xs.length > 5 ? "…" : ""}`; const missing = [...datasetNames].filter((n) => !csvSamples.has(n)); const extra = [...csvSamples].filter((s) => !datasetNames.has(s)); - // One line per issue (the UI renders each on its own line). Missing samples block Run (args throws); - // extra CSV values are only informational (those rows are simply never used). + // One line per issue, each rendered on its own line. Missing samples block Run, because args throws. + // Extra CSV values are informational, since those rows are never used. Counted into a real plural + // rather than written "sample(s)": these lines are read while something is already wrong. const lines: string[] = []; if (missing.length > 0) lines.push( - `${missing.length} dataset sample(s) have no rows in the CSV — Run is blocked until every sample is mapped (or the sample column is cleared): ${fmt(missing)}.`, + `${missing.length} ${missing.length === 1 ? "sample" : "samples"} in your dataset have no rows in the CSV: ${fmt(missing)}. The block disables Run until every sample has rows, or until you clear the sample column.`, ); if (extra.length > 0) lines.push( - `${extra.length} CSV sample value(s) match no dataset sample (ignored): ${fmt(extra)}.`, + `${extra.length} sample ${extra.length === 1 ? "value" : "values"} in the CSV match no sample in your dataset: ${fmt(extra)}. The block ignores those rows.`, ); return lines.length > 0 ? lines : undefined; }) - // The tag CSV column that looks like it names the dataset's samples (or undefined). The UI offers it as - // a one-click "use sample-aware mapping" suggestion. Purely advisory — the user must still pick it (a - // gesture that snapshots the sample map into data); this output never writes data. Excludes the columns - // already bound to the barcode / feature roles. See suggestSampleColumn for the superset/equality rule. + // The tag CSV column that looks like it names the dataset's samples, or undefined. The UI offers it as a + // one-click "use sample-aware mapping" suggestion. Purely advisory: the user must still pick it, the + // gesture that snapshots the sample map into data, and this output never writes data. Excludes the + // columns already bound to the barcode and feature roles. See suggestSampleColumn for the rule. .retentiveOutput("suggestedSampleColumn", (ctx): string | undefined => suggestSampleColumn(ctx)) - // Duplicate-barcode detection at config time (UI warning only; the Python guards it authoritatively at - // the end of the run). Fires when a CSV is uploaded, the barcode column is chosen, no sample column is - // set, and that barcode column has fewer distinct values than the CSV has data rows — i.e. some barcode - // maps on more than one row, which would fan the per-cell join and double molecule counts. Names the - // fix (set the Sample column, suggesting the likely one; else remove the duplicate rows). Skipped when - // rowCount is absent (prerun predates it) — then the check can't run and we defer to the Python guard. + // Alphabet check on the chosen barcode-sequence column, a UI warning only. mitool guards the same + // condition, but by failing refine-tags in the middle of the run. + // + // A panel CSV often carries BOTH an identifier column and the nucleotide column: "Barcode" holds T0100 + // and "Sequence" holds CGATGCCGGACGATC. The identifier column has the name a user is more likely to + // select, and that choice writes a panel.txt of non-nucleotide strings. The run then fails several stages + // later, inside barcode correction, with "Error while loading sequence set from ./panel.txt" and a Java + // stack trace, after the reads are parsed. + // + // The args guard cannot catch this: it sees only `data`, and the values live in the prerun CSV meta. + // Deliberately not gated on sampleColumn -- a per-sample filter narrows which rows reach the panel, and + // never turns an identifier into a sequence. + .retentiveOutput("barcodeAlphabetIssue", (ctx): string | undefined => { + const problem = barcodeAlphabetProblem(ctx); + if (problem === undefined) return undefined; + const { offenders, checked, alternative } = problem; + return ( + `Column "${ctx.data.barcodeSeqColumn}" does not hold nucleotide sequences. ${offenders.length} ` + + `of ${checked} distinct values contain characters outside A/C/G/T/N, for example ` + + `"${offenders[0]}". The block builds the feature-barcode panel from this column, so the run ` + + "would fail during barcode correction. " + + (alternative !== undefined + ? `Column "${alternative}" holds sequences. Pick that one.` + : "Pick the column that holds the barcode nucleotide sequences.") + ); + }) + // Duplicate-barcode detection at config time, a UI warning only. The Python guards it authoritatively at + // the end of the run. Fires when a CSV is uploaded, the barcode column is chosen, no sample column is + // set, and that barcode column has fewer distinct values than the CSV has data rows. Some barcode then + // maps on more than one row, which would fan the per-cell join and double molecule counts. Names the fix: + // set the Sample column, suggesting the likely one, or remove the duplicate rows. Skipped where rowCount + // is absent, which defers to the Python guard. .retentiveOutput("barcodeMappingIssue", (ctx): string | undefined => { if (!ctx.data.tagFeatureCsvHandle) return undefined; const barcodeCol = ctx.data.barcodeSeqColumn; if (!barcodeCol) return undefined; - if (ctx.data.sampleColumn) return undefined; // already sample-aware — the per-sample filter fixes it + if (ctx.data.sampleColumn) return undefined; // already sample-aware: the per-sample filter fixes it + // Silent while the column holds no sequences at all. "Some barcode sits on two rows" would direct the + // reader to the sample column. The mistake is in the barcode column itself. + if (barcodeAlphabetProblem(ctx) !== undefined) return undefined; const meta = readCsvMeta(ctx); if (!meta || meta.rowCount === undefined) return undefined; const distinct = meta.valuesByColumn?.[barcodeCol]?.length ?? 0; if (distinct >= meta.rowCount) return undefined; // no duplicate barcodes const suggested = suggestSampleColumn(ctx); return ( - "Some feature barcodes appear on multiple rows, so a single mapping is ambiguous. " + - `If this CSV is sample-specific, set the Sample column${suggested ? ` (looks like "${suggested}")` : ""}; ` + - "otherwise remove the duplicate rows." + `The CSV has ${meta.rowCount} rows but only ${distinct} distinct barcodes. ` + + "The block cannot map one barcode to two antigens, so the run will stop. " + + `If the CSV lists each barcode once per sample, set the Sample column${suggested ? ` ("${suggested}")` : ""}. ` + + "If it does not, remove the duplicate rows." + ); + }) + // The case the two checks either side of this one cannot see: a panel CSV that IS sample-keyed, with no + // sample column set, and no barcode repeated to give it away. `barcodeMappingIssue` needs a duplicate + // barcode, which a fully disjoint panel -- sample A stained with one set, sample B with another -- never + // supplies. `sampleMappingWarning` validates a column that has been chosen and returns nothing when none + // has. + // + // The tell is a column whose values cover every dataset sample, which is what `suggestSampleColumn` + // already looks for. Guarded against `barcodeMappingIssue`'s condition so the two never fire together: + // that one is the louder problem and already names this fix. + // + // Worth a warning rather than a tooltip because, read as one panel, every sample is offered every + // antigen. An antigen a sample was never stained with then comes back NOT BOUND instead of NEVER ASKED -- + // the collapse of a non-answer into a negative that the four-state verdict exists to prevent, and nothing + // else on the page would say it happened. + .retentiveOutput("unkeyedSamplePanel", (ctx): string | undefined => { + if (!ctx.data.tagFeatureCsvHandle || !ctx.data.barcodeSeqColumn) return undefined; + if (ctx.data.sampleColumn) return undefined; + const meta = readCsvMeta(ctx); + if (!meta || meta.rowCount === undefined) return undefined; + const distinct = meta.valuesByColumn?.[ctx.data.barcodeSeqColumn]?.length ?? 0; + if (distinct < meta.rowCount) return undefined; // barcodeMappingIssue owns this one + const suggested = suggestSampleColumn(ctx); + if (!suggested) return undefined; + return ( + `The CSV has a column that names your samples ("${suggested}"), and you have not set the sample ` + + "column. The block therefore reads the CSV as a single panel and applies that one panel to every " + + "sample. It then judges every sample on antigens it was never stained with, and those antigens " + + 'come back as "not bound" instead of "never asked". If the CSV is sample-specific, set the ' + + "Sample column." ); }) - // True while the uploaded CSV is still being parsed by staging (handle set, but emit-csv-meta hasn't - // produced csvMeta yet) — lets the UI show a "reading columns…" state instead of silent empty + // Total data rows in the panel, so the UI can snapshot it alongside the barcode column's distinct count. + // Those two numbers are what args() needs to refuse a duplicate mapping. + .retentiveOutput("csvRowCount", (ctx): number | undefined => readCsvMeta(ctx)?.rowCount) + // True while the panel has been picked but not yet read: the handle is set and no snapshot matches it. A + // local pick closes this window within a tick. A remote pick holds it until the upload lands and the UI + // parses the exported blob. Lets the UI show a "reading columns..." state instead of silent empty // dropdowns. NOT retentive: it must report the live loading state, including on a CSV swap. .output( "csvColumnsLoading", (ctx): boolean => !!ctx.data.tagFeatureCsvHandle && readCsvMeta(ctx) === undefined, ) - // Drives the tag→feature CSV upload: getImportProgress() registers the import handle with the - // middle-layer upload driver so the CSV bytes are actually pushed; isActive keeps it computing even - // when the block isn't being viewed. Without this the CSV never uploads and every per-sample body - // hangs on __extra_tagsCsv (mirrors immune-assay-data index.ts / samples-and-data). + // Drives the tag->feature CSV upload: getImportProgress() registers the import handle with the + // middle-layer upload driver, so the CSV bytes are pushed. isActive keeps it computing while the block is + // not being viewed. Without this the CSV never uploads and every per-sample body hangs on __extra_tagsCsv + // (mirrors immune-assay-data index.ts / samples-and-data). .output( "tagFeatureCsvImportHandle", (ctx) => ctx.outputs?.resolve("tagFeatureCsvImportHandle")?.getImportProgress(), { isActive: true }, ) - // Same upload driver, but resolved from the PRERUN (staging) render — this is the one that fires - // before Run. The CSV-derived dropdowns (csvColumnOptions / controlOptions) are populated by the prerun - // reading the uploaded CSV, and their values are REQUIRED by args(). The main driver above only - // fires once args() passes, so on its own it deadlocks: no upload → empty dropdowns → args() throws - // → no main render → no upload. Driving the upload from staging breaks the cycle (mirrors - // samples-and-data's "Drives prerun file uploads" getImportProgress). + // Same upload driver, resolved from the PRERUN (staging) render, which is the one that fires before Run. + // The prerun reads the uploaded CSV to populate the CSV-derived dropdowns (csvColumnOptions / + // controlOptions), and args() REQUIRES their values. The main driver above fires only once args() passes, + // so on its own it deadlocks: no upload, empty dropdowns, args() throws, no main render, no upload. + // Driving the upload from staging breaks the cycle (mirrors samples-and-data). .output( "tagFeatureCsvImportHandlePrerun", (ctx) => @@ -529,14 +1249,20 @@ export const platforma = BlockModelV3.create(dataModel) ?.getImportProgress(), { isActive: true }, ) - // True while the main run is executing (no output/context field settled yet) — drives the block - // spinner via the app.ts progress callback. + // The uploaded tag->feature CSV as a downloadable blob handle, resolved from the PRERUN's csvFile export. + // This is what lets the UI read the CSV's bytes for a REMOTE (index://) pick, where it cannot read the + // user's disk: the same client-side parser then runs on these bytes instead. The prerun already exported + // csvFile to make staging demand the blob, so this adds no work to the workflow. `traverse` rather than + // `resolve`, because it does not assert a field type. + .output("csvFileHandle", (ctx) => ctx.prerun?.traverse({ field: "csvFile" })?.getFileHandle()) + // True while the main run is executing, with no output or context field settled yet. Drives the block + // spinner through the app.ts progress callback. .output("isRunning", (ctx) => ctx.outputs?.getIsReadyOrError() === false) - // True once the main workflow has begun producing outputs (ctx.outputs settles) — lets the Main page + // True once the main workflow has begun producing outputs (ctx.outputs settles), so the Main page can // swap the static "run the block" hint for the live per-sample progress grid. .output("started", (ctx) => ctx.outputs !== undefined) - // Per-sample current step, derived from which stepReports entries have settled (a report appears when - // its step finishes). + // Per-sample current step, derived from which stepReports entries have settled. A report appears when its + // step finishes. .output("sampleStep", (ctx): Record | undefined => { if (ctx.outputs === undefined) return undefined; const reports = parseResourceMap( @@ -563,14 +1289,14 @@ export const platforma = BlockModelV3.create(dataModel) }) // Per-[sampleId, step] live log handles (parse / refine / tag-stat stdout streams), bound by the // per-sample Logs tab (PlLogView) so the user can read each mitool step's output as it runs. A no-match - // sample carries only its 1-parse entry (the map key set is variable — see fb-refine-tagstat). + // sample carries only its 1-parse entry, so the map key set is variable -- see fb-refine-tagstat. .output("stepLogs", (ctx) => ctx.outputs !== undefined ? parseResourceMap(ctx.outputs.resolve("stepLogs"), (acc) => acc.getLogHandle(), false) : undefined, ) // Per-[sampleId] log handle for the Python per-cell-metrics step (the "4-metrics" step). Surfaced - // separately from stepLogs because it's produced after the mitool stepLogs map is built; the UI's + // separately from stepLogs because it is produced after the mitool stepLogs map is built. The UI's // per-step Logs panel reads it when the "4-metrics" step is selected. .output("metricsLog", (ctx) => ctx.outputs !== undefined @@ -581,9 +1307,21 @@ export const platforma = BlockModelV3.create(dataModel) ) : undefined, ) - // Live per-sample parse progress (0–100%) — reads the flat parseLogStream Log, registered the moment - // the per-sample body runs (before parse finishes). Kept mainly as an EARLY roster signal (it appears - // before the stepLogs map fills); the per-step bar detail comes from stepProgress below. + // The Python per-cell-metrics step's live progress, scraped from the same stream `metricsLog` exposes + // as a handle. That step is the slowest one on a large run and the only one whose bar used to hold at + // its band floor from start to finish, because nothing read its stdout for a percentage. + .output("metricsProgress", (ctx) => + ctx.outputs !== undefined + ? parseResourceMap( + ctx.outputs.resolve("metricsLogStream"), + (acc) => acc.getProgressLogWithInfo(ProgressPrefix), + false, + ) + : undefined, + ) + // Live per-sample parse progress (0-100%), read from the flat parseLogStream Log, registered the moment + // the per-sample body runs and before parse finishes. Mainly an EARLY roster signal: it appears before the + // stepLogs map fills. The per-step bar detail comes from stepProgress below. .output("parseProgress", (ctx) => ctx.outputs !== undefined ? parseResourceMap( @@ -594,9 +1332,9 @@ export const platforma = BlockModelV3.create(dataModel) : undefined, ) // Per-[sampleId, step] live progress line (parse / refine / tag-stat), scraped from each step's stdout - // stream. Drives the rich per-step text in the grid Progress cell (which tag is being corrected, sort - // vs write phase, live %). ui/src/progress.ts composes these into a MONOTONIC cumulative bar (each step - // owns a quarter of the bar) so it never resets to zero between steps. Same source as stepLogs. + // stream. Drives the rich per-step text in the grid Progress cell: which tag is being corrected, sort vs + // write phase, live %. ui/src/progress.ts composes these into a MONOTONIC cumulative bar, each step owning + // a quarter, so it never resets to zero between steps. Same source as stepLogs. .output("stepProgress", (ctx) => ctx.outputs !== undefined ? parseResourceMap( @@ -606,16 +1344,16 @@ export const platforma = BlockModelV3.create(dataModel) ) : undefined, ) - // sampleIds whose per-sample pipeline has finished. qcJson is the LAST per-sample step and is inline - // JSON content, so getDataAsJsonOrUndefined reads it synchronously — the done-set drives the grid's - // "Done" state (a sample not in this set is still Processing). + // sampleIds whose per-sample pipeline has finished. qcJson is the LAST per-sample step and is inline JSON + // content, so getDataAsJsonOrUndefined reads it synchronously. The done-set drives the grid's "Done" + // state: a sample not in this set is still Processing. .output("completedSamples", (ctx): string[] | undefined => { if (ctx.outputs === undefined) return undefined; return parseQcRows(ctx).map((e) => String(e.key[0])); }) - // Per-sample QC metrics (from qcJson) keyed by sampleId — drives the Main grid's Quality + Read - // recovery columns (derived in ui/src/results.ts). Present per sample once its qc step settles (same - // source as completedSamples), so the two columns fill in as each sample finishes. + // Per-sample QC metrics from qcJson, keyed by sampleId, driving the Main grid's Quality + Read recovery + // columns (derived in ui/src/results.ts). Present per sample once its qc step settles, the same source as + // completedSamples, so the two columns fill in as each sample finishes. .output("sampleQc", (ctx): Record | undefined => { if (ctx.outputs === undefined) return undefined; const out: Record = {}; @@ -623,30 +1361,30 @@ export const platforma = BlockModelV3.create(dataModel) return out; }) // sampleId -> display name (upstream pl7.app/label), for the progress grid's Sample column. Shares - // resolveSampleLabels with analysisLog; kept as its own output because outputs cannot read one another. + // resolveSampleLabels with analysisLog. Its own output because outputs cannot read one another. .output("sampleLabels", (ctx): Record | undefined => resolveSampleLabels(ctx)) - // The block's single "Analysis logs" (lines shown in the UI's wide slide-over), built from the - // per-sample QC JSON (qcJson), which settles incrementally as each sample's qc step finishes: - // - while the run is in progress → a live count of samples finished so far ("Processing… N …"); - // - when every sample is done → a run-level summary (aggregate reads/panel-assigned/cells + - // any samples flagged for a panel-assigned fraction below PANEL_ASSIGNED_FLOOR, by name). - // One area regardless of sample count; detailed per-sample stats live on the QC page (qcSummaryTable). + // The block's single "Analysis logs", shown in the UI's wide slide-over, built from the per-sample QC + // JSON (qcJson), which settles incrementally as each sample's qc step finishes: + // - while the run is in progress: a live count of samples finished so far ("Processing... N ...") + // - when every sample is done: a run-level summary (aggregate reads / panel-assigned / cells, plus any + // samples flagged for a panel-assigned fraction below PANEL_ASSIGNED_FLOOR, by name) + // One area regardless of sample count. Detailed per-sample stats live on the QC page (qcSummaryTable). .output("analysisLog", (ctx): string[] | undefined => { if (ctx.outputs === undefined) return undefined; - // Sample labels (sampleId -> name) from the upstream pl7.app/label column — display names for - // flagged samples. Shared resolver with the sampleLabels output. + // Sample labels (sampleId -> name) from the upstream pl7.app/label column, for flagged samples. Shared + // resolver with the sampleLabels output. const labels = resolveSampleLabels(ctx); - // Per-sample QC metrics; each entry appears as that sample's qc step finishes (shared with - // completedSamples / sampleQc). qcJson is inline JSON content read synchronously. + // Per-sample QC metrics. Each entry appears as that sample's qc step finishes, shared with + // completedSamples / sampleQc. qcJson is inline JSON content read synchronously. const entries = parseQcRows(ctx); const done = entries.length; const running = ctx.outputs.getIsReadyOrError() === false; - // While the run is in progress → a live count of samples finished so far. No fixed denominator: - // the block only processes the samples present in its feature-barcode dataset, which isn't reliably - // known until the run completes (a project-wide sample total would over-count and make a finished - // run look stuck). On a crash the count freezes at how far it got, next to the block's error state. + // While the run is in progress: a live count of samples finished so far. No fixed denominator, because + // the block processes only the samples present in its feature-barcode dataset, which is not reliably + // known until the run completes. A project-wide total would over-count and make a finished run look + // stuck. On a crash the count freezes where it got to, next to the block's error state. if (running) { return done === 0 ? ["Processing…"] @@ -705,23 +1443,20 @@ export const platforma = BlockModelV3.create(dataModel) lines.push("", "Analysis complete. Full per-sample statistics are on the QC page."); return lines; }) - // The Main table is ONE ROW PER CELL [sampleId, cellId]. The per-(cell x feature) rows - // moved OUT of this table into the collapsed workflow frame (consensus + the per-cell summary - // columns: Max Feature UMI count, Max Feature Fraction, Max Specificity score, and a "Feature - // breakdown" string listing every feature as "feature (fraction%, umi)" sorted by descending fraction). The - // per-feature matrix is not lost: it is still exported to the result pool (perCellFeatures, the - // per-cell export contract) for VDJ Multiomic Integration. This output resolves the workflow's collapsed - // perCellTable PFrame; undefined until the workflow emits it (guarded by the UI). + // The Main table is ONE ROW PER CELL [sampleId, cellId]. The collapsed workflow frame carries the + // per-cell summary columns: Max feature UMI count, Max feature fraction, and a "Feature breakdown" string + // listing every feature as "feature (fraction%, umi)" sorted by descending fraction. The per-feature + // matrix is still exported to the result pool (perCellFeatures) for VDJ Multiomic Integration. This + // output resolves the workflow's collapsed perCellTable PFrame, undefined until the workflow emits it. // // Uses createPlDataTableV2 (columns passed directly via getPColumns), NOT V3. This frame is our OWN - // self-contained, non-batch processColumn output. createPlDataTableV3's discovery cannot render it: - // the object (scoped-sources) form returns undefined for this frame regardless of anchor/maxHops - // config, and the array-columns form runs discoverLabelColumnVariants over the - // ENTIRE result pool and hangs forever on the upstream Samples&Data FASTQ File-dataset - // (no_data::pf.dataset.*). V2 takes the columns as-is and auto-joins the sampleId label — - // the pattern blocks/peptide-extraction uses for the same non-batch processColumn + samples-and-data - // setup. retentive avoids blanking the grid on recompute; withStatus feeds PlAgDataTableV2 the - // OutputWithStatus envelope it renders loading/error from. + // self-contained, non-batch processColumn output, and createPlDataTableV3's discovery cannot render it: + // the object (scoped-sources) form returns undefined whatever the anchor/maxHops config, and the + // array-columns form runs discoverLabelColumnVariants over the ENTIRE result pool and hangs forever on + // the upstream Samples&Data FASTQ File-dataset (no_data::pf.dataset.*). V2 takes the columns + // as-is and auto-joins the sampleId label, the pattern blocks/peptide-extraction uses for the same + // non-batch processColumn plus samples-and-data setup. retentive avoids blanking the grid on recompute. + // withStatus feeds PlAgDataTableV2 the OutputWithStatus envelope it renders loading and error from. .output( "perCellTable", (ctx) => { @@ -732,11 +1467,10 @@ export const platforma = BlockModelV3.create(dataModel) { retentive: true, withStatus: true }, ) // Per-sample QC summary table: reads parsed/matched, cells/features detected, UMI totals, and - // panel-assigned fraction. Uses createPlDataTableV2 (columns passed directly via getPColumns) like - // perCellTable. The earlier V3 form used selector { mode: "enrichment", maxHops: 0 }, - // which never traverses to the upstream pl7.app/label column — so the sampleId axis rendered the raw - // sample hash instead of the human sample name. createPlDataTableV2 runs getAllLabelColumns over the - // result pool and auto-joins the matching sampleId label, giving the real sample name. + // panel-assigned fraction. Uses createPlDataTableV2 like perCellTable, because it runs getAllLabelColumns + // over the result pool and auto-joins the matching sampleId label. A V3 selector of + // { mode: "enrichment", maxHops: 0 } never traverses to the upstream pl7.app/label column, and the + // sampleId axis then renders the raw sample hash. .output( "qcSummaryTable", (ctx) => { @@ -746,17 +1480,502 @@ export const platforma = BlockModelV3.create(dataModel) }, { retentive: true, withStatus: true }, ) + // Every combined identity the punchcard could show, in the order the workflow gave them, each with the + // label the workflow put on its column. Two things on the card read it, and neither narrows anything: the + // punch hover, because a reader hovering a dot far down a long grid cannot see the header row, and the + // card's empty state, to tell "the pivot emitted no identity columns" apart from "this run has no rows". + // + // Read from the pivot's own columns, not from the run record's identity list. The pivot is size-gated + // upstream, so a run over a large panel names its identities in the record and emits no columns at all. + // Reading the columns lists what the punchcard can actually draw. + .retentiveOutput("punchcardIdentityOptions", (ctx): { value: string; label: string }[] => { + const pCols = ctx.outputs + ?.resolve({ field: "antigenPunchcardTable", allowPermanentAbsence: true }) + ?.getPColumns(); + if (pCols === undefined) return []; + const seen = new Set(); + const options: { value: string; label: string }[] = []; + for (const c of pCols) { + if (c.spec.name !== PUNCH_COLUMN_NAME) continue; + const identity = c.spec.domain?.[PUNCH_IDENTITY_DOMAIN]; + if (identity === undefined || seen.has(identity)) continue; + seen.add(identity); + // The label the workflow put on the column, which for a merged identity is the joined name rather + // than the barcode. The identity itself is a raw sequence under the per-tag grouping, so carrying the + // label is what keeps the punch hover naming an antigen the reader recognises. + const label = c.spec.annotations?.["pl7.app/label"]; + options.push({ value: identity, label: label ?? identity }); + } + return options; + }) + // The punchcard: one row per clonotype set, one column per identity, each cell carrying the four-state + // verdict and the count of cells that answered it. The pivoted shape comes from the workflow, because a + // table cannot pivot a (set, identity) frame into columns. + // + // The whole panel, every time. A reader who wants fewer columns hides them in the grid's columns panel or + // filters in its filters panel, and this output second-guesses neither. + // + // V3 here, and V2 everywhere else in this block, for a reason particular to this table. V2 cannot build it + // at all: `Cannot produce a Vec1 with a length of zero`. These columns are keyed on ONE axis, + // `pl7.app/vdj/scClonotypeKey`, and the result pool holds a label column for exactly that axis. V2 + // discovers the label, the frame's only axis is consumed, and the engine is handed an empty key vector. + // One axis is inherent to a punchcard, so there is nothing to tune. V3's `primaryColumns` form takes the + // columns as given and runs NO data-column discovery, so it never walks the result pool and cannot hang + // on the upstream Samples & Data FASTQ dataset -- the hazard the other tables here chose V2 to avoid, and + // the reason this is not a blanket migration. V3 still resolves label columns for the axes it was handed, + // which is wanted: a clonotype row reads better under its clonotype label than under a raw key. + // + // V2 is deprecated SDK-side in favour of this call, so the rest of this model's tables will follow. Each + // needs its own check against the discovery hazard first. + .output( + "punchcardTable", + (ctx) => { + const pCols = ctx.outputs + ?.resolve({ field: "antigenPunchcardTable", allowPermanentAbsence: true }) + ?.getPColumns(); + if (pCols === undefined) return undefined; + const identityOf = (c: (typeof pCols)[number]) => c.spec.domain?.[PUNCH_IDENTITY_DOMAIN]; + // Every identity the pivot produced, always. Narrowing is the grid's job: PlAgDataTableV2 ships a + // columns panel and a filters panel, so a stored server-side filter cannot disagree with what the + // column chooser shows. + const cols = pCols.filter((c) => identityOf(c) !== undefined); + if (cols.length === 0) return undefined; + // The clonotype's cell count, beside its name. It carries no identity domain, so the filter above + // drops it -- and `primaryColumns` runs no discovery, so a column not listed here never reaches the + // grid. It is keyed on the same single axis as the punch columns, which is what makes adding it safe: + // a column carrying an axis the others lack would widen the join. + const cellCount = pCols.filter((c) => c.spec.name === PUNCH_CELL_COUNT_COLUMN); + // Alphabetical by the name a READER sees. The workflow emits these sorted by identity, which under + // the per-tag grouping is the barcode, and a panel's names never sort the same as its sequences. + // Numeric collation, so `antigen_9` precedes `antigen_10`. The clonotype column is not among these + // (`columns: null` brings it in separately), so it keeps its place at the front. + const labelOf = (c: (typeof cols)[number]) => + c.spec.annotations?.["pl7.app/label"] ?? (identityOf(c) as string); + const ordered = [...cols].sort((a, b) => + labelOf(a).localeCompare(labelOf(b), undefined, { sensitivity: "base", numeric: true }), + ); + // Headers carry the identity's full name, never a truncation: which identity a column is, is the one + // thing a reader needs from a header. Correct a too-long label where it is produced. + // + // Column ORDER comes from the `pl7.app/table/orderPriority` annotation on each spec, and from nothing + // else. The cell count carries 96000, between the clonotype label's 100000 and the punches' 92000, and + // lands at position 3: row number, clonotype, cell count, then the identities. A `displayOptions.ordering` + // rule and this array's own order are BOTH inert here. If you are fixing a column that "renders last", + // measure first, and measure with `aria-colindex`: `querySelectorAll('[role="columnheader"]')` returns + // AG Grid's recycled header nodes in an order unrelated to column position. + return createPlDataTableV3(ctx, { + primaryColumns: [...cellCount, ...ordered].map((c) => DataColumn.fromColumn(c)), + columns: null, + tableState: ctx.data.punchcardTableState, + }); + }, + { retentive: true, withStatus: true }, + ) + // The clonotype axis id, DERIVED from an emitted column and never written out by hand. The page hangs the + // expansion's row button on this axis, and `showCellButtonForAxisId` is matched with `isJsonEqual` -- + // exact JSON equality, domain and all. A hand-written `{type, name}` misses the domain this axis carries, + // matches nothing, and renders no button and no error. Deriving it from the same spec the filter reads + // also makes the two provably agree. + .output("clonotypeAxisId", (ctx): AxisId | undefined => { + const pCols = ctx.outputs + ?.resolve({ field: "antigenPunchcardTable", allowPermanentAbsence: true }) + ?.getPColumns(); + const axis = pCols?.[0]?.spec.axesSpec[0]; + return axis === undefined ? undefined : getAxisId(axis); + }) + // The expansion: ONE clonotype's identities, read down. `the-explore-readout` puts this opposite the card + // -- the grid is read across a row to see what a clone bound, and the expansion down to see what those + // verdicts rest on, which is where a number belongs. A number in every position of the card would compete + // with reading it across, so this is where cellsBound and the support counts surface. + // + // Reads `antigenVerdictsTable`: the LONG verdicts family at (set, identity) grain, held open by main.tpl + // for exactly this, so this page costs no workflow change and no second import. Its rows are identities, + // the shape the expansion wants and the shape a pivot cannot give it. + // + // NOT gated on the identity count that gates the card's pivots. That gate exists because a pivot costs a + // COLUMN per identity. Here an identity costs a ROW, and only one clonotype's rows are ever fetched, so a + // panel too wide for the card is precisely where this view still reads. + // + // The filter is pushed down, not applied after the fact: `createPlDataTableV3` puts it in the PTable def, + // `createPTableDefV3` wraps the join in a `{type:"filter", predicate}` query node, and the engine lowers + // that into the data query (pframes-rs `visit_filter`). One clonotype's rows are what crosses the + // boundary, whatever the run's size. + .output( + "expansionTable", + (ctx) => { + // Undefined until a clonotype is chosen, and that is the point rather than a convenience: a table + // built with no filter is EVERY clonotype's identities at once, which on a real run is the exact cost + // this design exists to avoid. No selection, no table. + const chosen = ctx.data.expandedSet; + if (chosen === undefined || chosen.length === 0) return undefined; + const frame = ctx.outputs + ?.resolve({ field: "antigenVerdictsTable", allowPermanentAbsence: true }) + ?.getPColumns(); + if (frame === undefined || frame.length === 0) return undefined; + // The columns the design asks for -- identity, state, bound, and could-answer only where the run + // carried panels that differ -- NAMED explicitly, which is the whole correctness of this call. + // `antigenVerdictsTable` surfaces the entire export frame, whose families are keyed on five different + // axes: tag, panel, sample, set, and (set, identity). Handing all of them to one table is a malformed + // join, and the SDK answers `discoverColumns failed` out of `discoverLabelColumns`, which reads as an + // SDK fault and is not one. Only the (set, identity) family belongs here. + // + // The identity's readable name comes FIRST, and it has to be named here rather than left to + // `columns: null`. That option resolves label columns from the result pool, and this label lives in + // `exportFb` -- a block's own exports are not in its own result pool. Without it every row prints the + // same clonotype with nothing telling them apart. + // + // Could-answer is CONDITIONAL. Under one panel it is the clonotype's own cell count at every identity, + // which the grid already carries beside its name, so a column of it repeats one number down the page. + // The whole argument for carrying the number is that a verdict from three cells and one from forty + // print the same word, and a number that never varies defeats that argument. + // + // Read from the run RECORD, not from current args: what panels a run carried is a fact about that run, + // the same discipline the comparator uses (served, never requested). + const runMeta = ctx.outputs + ?.resolve({ field: "antigenRunMeta", allowPermanentAbsence: true }) + ?.getDataAsJsonOrUndefined(); + // Absent reads as one panel: a run record written before the field existed has no opinion, and one + // panel is the ordinary case. + const panelsDiffer = (runMeta?.samplePanelCount ?? 1) > 1; + // Not-bound is deliberately absent. A cell's vote is exactly one of bound or not-bound, so a third + // column is `answered - bound` printed out, and a reader who wants it subtracts two numbers already on + // the row. It stays in the EXPORT, which has its own readers. Only this panel drops it. + const WANTED = [ + "pl7.app/label", + "pl7.app/antigen/verdict", + ...(panelsDiffer ? ["pl7.app/antigen/cellsCouldAnswer"] : []), + "pl7.app/antigen/cellsAnswered", + "pl7.app/antigen/cellsBound", + ]; + // One axis, the identity axis, is what makes `pl7.app/label` a label column rather than a name + // collision -- the frame carries other one-axis labels (the panel's, the tag's), and a label on the + // wrong axis would join nothing. Filtered on the axis rather than trusted by name. + const identityAxisName = "pl7.app/antigen/identityId"; + const pCols = WANTED.flatMap((name) => + frame.filter( + (c) => + c.spec.name === name && + (name !== "pl7.app/label" || + (c.spec.axesSpec.length === 1 && c.spec.axesSpec[0].name === identityAxisName)), + ), + ); + // The identity's name has to be one of them, and a count is not enough to know that. Where only the + // label fails to match -- an axis name drifting at export time is all it takes -- the verdict and bound + // columns still match, the count is still non-zero, and the panel renders anonymous rows with nothing + // to say it regressed. No panel is a visible failure. A nameless one reads as working. + if (!pCols.some((c) => c.spec.name === "pl7.app/label")) return undefined; + // The set axis is the first of the (set, identity) pair, taken from the verdict column and not from + // `pCols[0]`: the identity label column sorts first and carries only the identity axis, so + // `pCols[0].spec.axesSpec[0]` would hand the filter that axis and resolve nothing. An axis assembled + // here would be a lookalike with a different identity and filter nothing. + const verdictCol = pCols.find((c) => c.spec.name === "pl7.app/antigen/verdict"); + // Checked directly, and before `setAxis` is derived from it: the filter below reads `verdictCol.id`, + // and narrowing only `setAxis` would leave `verdictCol` typed as possibly undefined at that use. + if (verdictCol === undefined) return undefined; + const setAxis = verdictCol.spec.axesSpec[0]; + if (setAxis === undefined) return undefined; + return createPlDataTableV3(ctx, { + primaryColumns: pCols.map((c) => DataColumn.fromColumn(c)), + columns: null, + tableState: ctx.data.expansionTableState, + // The label column is SUPPLIED here as a primary column, filtered out of the frame above -- it is + // not discovered from a pool. `PlAgDataTableV2` drops any axis that has a label column and renders + // the label column in its place, subject to that column's own visibility. + // + // Two rules, and the order matters -- the first match wins. Both columns the table would show as a + // name are called `pl7.app/label`, so they are told apart by the axis each one labels. + displayOptions: { + visibility: [ + // The identity's name. This is the row's subject. Required despite the spec already + // annotating "default": the catch-all rule below matches this column too, and first match wins. + { + match: { + name: "^pl7\\.app/label$", + axes: [{ name: "^pl7\\.app/antigen/identityId$" }], + partialAxesMatch: false, + }, + visibility: "default", + }, + // Any other label column here labels the CLONOTYPE axis, and the panel is about one clonotype, + // which the reader chose by clicking it. Printing its name down every row is repetition. + // Optional rather than hidden, so the Columns picker can bring it back. + { match: { name: "^pl7\\.app/label$" }, visibility: "optional" }, + ], + // Cells-that-answered sits LAST, behind the count it contains. Its annotation puts it at 98000, + // ahead of cells-that-read-bound at 97500, which is the right default everywhere else: a + // denominator reads before the number it divides. In this panel the bound count is what the reader + // came for and the answered count is the context, so the two swap. Overridden here rather than in + // the workflow spec because those columns are EXPORTS with downstream readers, and a priority is + // global. + // + // This rule only reaches a clonotype the grid has not drawn before. + // `expansionTableState.stateCache` keeps one grid state PER `sourceId`, and `sourceId` here is the + // expanded clonotype, so every clonotype opened once has its own frozen `columnOrder.orderedColIds`. + // A stored order is an explicit list of column ids and it beats anything the model asks for. A + // reorder that has to reach already-opened clonotypes therefore needs the cache invalidated, the + // device the v3 -> v4 migration used. Not done here: the order is a preference, and resetting every + // reader's saved columns and filters to move one column right is the more expensive mistake. + ordering: [{ match: { name: "^pl7\\.app/antigen/cellsAnswered$" }, priority: 90000 }], + }, + filters: { + type: "and", + filters: [ + { + type: "patternEquals", + // The FULL axis id, domain included. Dropping the domain leaves an id that + // `remapFilterColumnIds` cannot resolve against the table's columns, and the SDK's + // unresolved-leaf path calls `console`, which does not exist in the model's QuickJS sandbox. + // The symptom is then `ReferenceError: 'console' is not defined` from deep inside the SDK, + // naming nothing about the filter. That error means an unresolvable filter column. + column: { + type: "axis", + id: + setAxis.domain === undefined + ? { name: setAxis.name, type: setAxis.type } + : { name: setAxis.name, type: setAxis.type, domain: setAxis.domain }, + }, + value: String(chosen[0]), + }, + { + type: "patternNotEquals", + // A never-asked position is not a reading, and the panel keeps the numbers to the identities + // the experiment actually put to these cells. Filtered by the verdict's own value rather than + // by a count, because a bound count of 0 is a real reading and must stay. + column: { type: "column", id: verdictCol.id }, + value: "never asked", + }, + ], + }, + }); + }, + { retentive: true, withStatus: true }, + ) + // The expansion's BY-CELL face: one row per cell of the chosen clonotype, one column per identity, + // carrying that cell's own reading rather than its set's verdict. This is where a reader sees WHY a + // verdict came out as it did: an `unreliable` on the card is cells disagreeing, and nothing but this shows + // the disagreement. + // + // Filtered on `setId`, a COLUMN here rather than an axis. The frame is keyed (sampleId, cellId) because + // that is what a cell is, and the clonotype is a property of the row. So the filter leaf is a + // `{type: "column"}` one, and `PColumn.id` is the `ColumnUniversalId` it wants. Never a hand-built id. + // + // Same push-down as the by-identity face, and it matters more here, where the frame's grain is every cell + // of the run. + .output( + "cellExpansionTable", + (ctx) => { + const chosen = ctx.data.expandedSet; + if (chosen === undefined || chosen.length === 0) return undefined; + const frame = ctx.outputs + ?.resolve({ field: "antigenCellReference", allowPermanentAbsence: true }) + ?.getPColumns(); + if (frame === undefined || frame.length === 0) return undefined; + // The set column has to be found before anything else: without it there is no filter, and an + // unfiltered table here is every cell in the run against every identity. Absent means the software + // gated the pivot away, a legitimate state and not an error. So no table, and the page says why from + // the run record. + const setCol = frame.find((c) => c.spec.name === "pl7.app/antigen/cellSetId"); + if (setCol === undefined) return undefined; + const punchCols = frame.filter((c) => c.spec.name === "pl7.app/antigen/cellPunch"); + if (punchCols.length === 0) return undefined; + const boundCount = frame.filter((c) => c.spec.name === "pl7.app/antigen/boundIdentities"); + // No ordering rule, deliberately. The bound count sits immediately right of the axes because its own + // annotation priority (95000) outranks every identity column (94000 and down), and that is where it + // belongs: it is the one number that summarises the row, and a matrix a hundred columns wide puts its + // far edge off screen. The by-identity face carries a rule only because there the annotation put + // cells-answered in the wrong place. + return createPlDataTableV3(ctx, { + primaryColumns: [setCol, ...boundCount, ...punchCols].map((c) => DataColumn.fromColumn(c)), + columns: null, + tableState: ctx.data.cellExpansionTableState, + filters: { + type: "and", + filters: [ + { + type: "patternEquals", + column: { type: "column", id: setCol.id }, + value: String(chosen[0]), + }, + ], + }, + }); + }, + { retentive: true, withStatus: true }, + ) + // The run's own quality report: every declared measurement with its status, the coverage triple behind it, + // and -- where nothing computed it -- the reason it was deferred. This block is obliged to produce the + // run-level measurements, and the obligation is that every measurement that CAN be computed is computed + // and SHOWN. Read from `outputs` and not from the exports, because a block's own exports are not in its + // own result pool. + // + // `allowPermanentAbsence` for the same reason punchcardTable needs it: the whole verdict stage is gated on + // a V(D)J dataset being picked, so on a run without one this field never appears, and a resolve that + // treats a permanent absence as a pending one waits forever instead of returning undefined. + // + // A frame with no rows is deliberately NOT folded into undefined. Absent means the verdict stage did not + // run. Empty means it ran and had nothing to report, which for the mismatch check is the good outcome. + .output( + "runQualityTable", + (ctx) => { + const pCols = ctx.outputs + ?.resolve({ field: "antigenQcTable", allowPermanentAbsence: true }) + ?.getPColumns(); + if (pCols === undefined) return undefined; + return createPlDataTableV2(ctx, pCols, ctx.data.runQualityTableState); + }, + { retentive: true, withStatus: true }, + ) + // The three distributions `330-the-quality-readout` puts last in the readout, as ONE p-frame for + // GraphMaker rather than as rows in the measurement table. Two of them exist so a scientist can + // place the cutoff and the gate, and a decile encoded in a detail string is a number nobody can + // draw. `allowPermanentAbsence` for the same reason the tables above need it. + .output( + "runQualityDistributions", + (ctx) => { + const pCols = ctx.outputs + ?.resolve({ field: "antigenQcDistributions", allowPermanentAbsence: true }) + ?.getPColumns(); + if (pCols === undefined) return undefined; + return createPFrameForGraphs(ctx, pCols); + }, + { retentive: true, withStatus: true }, + ) + // One row per (panel, tag, identity), carrying the figures the measurement table holds long-format. + // A tag carrying two identities takes a row under each. `allowPermanentAbsence` for the same reason + // the tables above need it. + .output( + "reagentTable", + (ctx) => { + const pCols = ctx.outputs + ?.resolve({ field: "antigenReagentTable", allowPermanentAbsence: true }) + ?.getPColumns(); + if (pCols === undefined) return undefined; + return createPlDataTableV2(ctx, pCols, ctx.data.reagentTableState); + }, + { retentive: true, withStatus: true }, + ) + // The panel-versus-reads check: every barcode the panel declared that no read carried, and every barcode + // the reads carried that the panel never declared. Both directions are in the one frame, told apart by the + // direction column, which carries a discrete filter so either half is reachable on its own. A mismatch + // report the user cannot see defeats its purpose, which is why the workflow emits it into `outputs` and + // not only into the exports. + .output( + "runQualityMismatchTable", + (ctx) => { + const pCols = ctx.outputs + ?.resolve({ field: "antigenPanelMismatchTable", allowPermanentAbsence: true }) + ?.getPColumns(); + if (pCols === undefined) return undefined; + return createPlDataTableV2(ctx, pCols, ctx.data.runQualityMismatchTableState); + }, + { retentive: true, withStatus: true }, + ) + // 330's own table: barcodes the reads carried that no panel declares, keyed by sequence. It carries + // the one status this run's quality surface publishes outside the measurement list -- the share of a + // sample's reads landing in undeclared barcodes -- and that status is the barcode's, never rolled into + // any sample's own. Usually empty, which is the outcome the field wants. `allowPermanentAbsence` for + // the same reason the tables above need it: the verdict stage is gated on a chosen V(D)J dataset. + .output( + "undeclaredBarcodesTable", + (ctx) => { + const pCols = ctx.outputs + ?.resolve({ field: "antigenUndeclaredBarcodesTable", allowPermanentAbsence: true }) + ?.getPColumns(); + if (pCols === undefined) return undefined; + return createPlDataTableV2(ctx, pCols, ctx.data.undeclaredBarcodesTableState); + }, + { retentive: true, withStatus: true }, + ) + // What the reading was actually answered under. The page states the comparator that SERVED rather than the + // one that was requested, because the software degrades a request it cannot honour and a reader meeting an + // all-unreliable table otherwise has no way to learn that happened. Absent until a run with a V(D)J + // dataset has produced it. + .output("verdictRunMeta", (ctx): VerdictRunMeta | undefined => + ctx.outputs + ?.resolve({ field: "antigenRunMeta", allowPermanentAbsence: true }) + ?.getDataAsJsonOrUndefined(), + ) + // Every sample-level quality measurement, keyed by sample, with that sample's rolled-up status. The single + // source for both the Main grid's Quality tag and the sample detail view's Quality Checks tab, so the tag + // and the list beside it cannot disagree about one sample. Absent until a run with a V(D)J dataset has + // produced it: the sample report is written by the verdict step. + .output("sampleQcReport", (ctx): Record | undefined => + ctx.outputs + ?.resolve({ field: "antigenSampleQc", allowPermanentAbsence: true }) + ?.getDataAsJsonOrUndefined>(), + ) + // The rung the run WILL be answered under, for the settings field to show. The same call `args()` projects, + // so the field cannot show one rule while the workflow receives another. Keep it that way: the last + // divergence between a shown rung and a sent one came from two copies of one rule. Nothing writes back, so + // there is no hairpin. + .output("effectiveReferenceSource", (ctx): ReferenceSource | undefined => + resolveReferenceSource(ctx.data), + ) + // Every baseline rung, each with what it still needs. Two rungs remain: a declared reference tag, and a + // tag's own distribution across the sample's cells. The panel's own readings are retired, and empty + // droplets need gene expression this block does not read. + // + // Whether the declared rung can serve is knowable before a run, from the panel metadata staging already + // emits: it needs a role column and values of it the column actually carries. The distribution rung's + // conditions are properties of the DATA, so they are stated in its description and reported by the run. + .retentiveOutput("referenceSources", (ctx): ReferenceSourceChoices => { + const meta = readCsvMeta(ctx); + const roleColumn = ctx.data.roleColumn; + const roleValues = new Set(roleColumn ? (meta?.valuesByColumn?.[roleColumn] ?? []) : []); + const declaredTags = (ctx.data.referenceValues ?? []).filter((v) => roleValues.has(v)); + + // EVERY rung is offered and every one is selectable, whether or not it can serve yet. The scientist + // chooses the rung and then supplies what it needs: withholding the option until its requirements were + // already met made the declared rung unreachable, because the requirements only appear once it is + // chosen. `needs` carries what is still missing, and the form shows it against the chosen rung. + const options: ReferenceSourceChoices["options"] = [ + { + value: "declared", + label: "Declared baseline tag", + description: + "The block judges each count against the tag your panel marks as the baseline, in the " + + "same cell. Verdicts read this way compare across runs.", + needs: + declaredTags.length > 0 + ? undefined + : "Name the panel column that declares each tag's role, then pick the value of it that " + + "marks the baseline tag. Both fields are below.", + }, + // No `needs`. Whether this rung can serve turns on the sample's cell count and on whether each tag's + // counts separate. This block has read neither, and the second is answered per tag rather than per + // run. So the conditions live in the description, and the RUN reports which tags fitted and why not. + { + value: "distribution", + label: "Each tag's own distribution", + description: + `The block splits each tag's counts across a sample's cells into two components and judges ` + + `counts against the lower one. It needs at least ${Math.round(ctx.data.distributionMinCells)} ` + + `cells in the sample, and it needs that tag's counts to actually separate. A tag whose counts ` + + `do not separate gets no baseline, and only the antigens that tag carries read unreliable. ` + + `Pick this where your panel declares no baseline tag and is too small to stand in for one.`, + }, + ]; + + // `none` is NOT offered, and there is no fourth option. A baseline is required and a run without one + // does not happen, so "no baseline" is not a position a scientist can select here -- it is a + // configuration `args()` refuses. The published view that a tag declared to be bound by nothing is not + // truly negative is served by the two rungs that need no such tag, rather than by an option that + // produces no answers. + + // What an unselected run is answered under. Nothing falls anywhere, so this states the consequence of + // leaving the field alone rather than naming a fallback. + const fallback = "no baseline -- every verdict that needs one reads unreliable"; + return { options, fallback }; + }) .title(() => "Feature Barcode Profiling") - // Standard block-label subtitle. The subtitle render context is args-only (no result pool / outputs - // — touching them renders "Invalid subtitle"), so the dynamic " / - " - // string is derived in the `suggestedBlockLabel` OUTPUT (which HAS the pool) and copied into - // `defaultBlockLabel` by a UI watchEffect (the sanctioned block-label pattern). The subtitle only - // reads `ctx.data`. Guard `ctx.data` — it can be undefined before block storage is parsed. + // Standard block-label subtitle. The subtitle render context is args-only -- touching the result pool or + // outputs renders "Invalid subtitle" -- so the dynamic " / - " string is + // derived in the `suggestedBlockLabel` OUTPUT, which HAS the pool, and copied into `defaultBlockLabel` by a + // UI watchEffect. Guard `ctx.data`: it can be undefined before block storage is parsed. .subtitle((ctx) => ctx.data?.defaultBlockLabel || "Feature-barcode - per-cell antigen counts") - // Main (the per-sample progress grid) is always shown. The result tabs — Per-sample QC and the - // per-cell results table — appear only once the block has produced outputs, so a fresh/unrun block - // shows only Main. ctx.outputs settles when the workflow starts emitting (the same signal as the - // `started` output). The Graph and Raw tag-stat views were removed. + // Main (the per-sample progress grid) is always shown. The result tabs -- Per-sample QC and the per-cell + // results table -- appear only once the block has produced outputs, so an unrun block shows only Main. + // ctx.outputs settles when the workflow starts emitting, the same signal as the `started` output. .sections((ctx) => { const hasRun = ctx.outputs !== undefined; return [ @@ -765,6 +1984,14 @@ export const platforma = BlockModelV3.create(dataModel) ? [ { type: "link" as const, href: "/qc" as const, label: "Per-sample QC" }, { type: "link" as const, href: "/results" as const, label: "Per-cell results" }, + // Shown for every run, including one with no V(D)J dataset. That run produces no antigen + // columns, and the page saying so is the only place a user learns why. Hiding the tab would leave + // the absence unexplained. + { type: "link" as const, href: "/punchcard" as const, label: "Explore readout" }, + // "Run quality" rather than "QC": that page is per SAMPLE, this one is per run, and one + // label for both invites a reader to take them as two views of one set of numbers. Shown for + // every run, including one with no V(D)J dataset, for the same reason the readout is. + { type: "link" as const, href: "/antigen-qc" as const, label: "Run quality" }, ] : []), ]; diff --git a/model/src/pattern.ts b/model/src/pattern.ts index 1f159c8..a3e7540 100644 --- a/model/src/pattern.ts +++ b/model/src/pattern.ts @@ -1,36 +1,37 @@ // mitool tag-pattern (read-geometry) model for the Feature Integration block. // // Feature-barcode reads have a FIXED single-cell layout across every documented 10x 5' antigen-capture -// variant (verified against the 10x docs): +// variant, verified against the 10x docs: // Read 1: CELL barcode + UMI Read 2: [optional leading skip] + FEATURE barcode + remainder -// so — unlike the general peptide-amplicon builder in blocks/peptide-extraction — there are no anchors, -// no per-read insert assignment, no reverse-complement mirroring, and no single-end case. The only -// user-tunable numbers are the three barcode lengths plus a Read 2 offset (the TotalSeq-C / next-gen -// antigen-barcoding case, where the 15 nt barcode sits behind a 10 nt lead: `5PNNNNNNNNNN(BC)`). +// so, unlike the general peptide-amplicon builder in blocks/peptide-extraction, there are no anchors, no +// per-read insert assignment, no reverse-complement mirroring, and no single-end case. The only +// user-tunable numbers are the three barcode lengths plus a Read 2 offset. That offset serves the +// TotalSeq-C and next-gen antigen-barcoding case, where the 15 nt barcode sits behind a 10 nt lead: +// `5PNNNNNNNNNN(BC)`. -// Tag names mitool registers for this pipeline. This is the SINGLE source of truth: they are baked into -// every assembled pattern here AND sent to the workflow in args.tags, so the downstream commands -// (`refine-tags -t CELL -t FEATURE -u UMI`, `tag-stat`, per_cell) reference exactly the names the -// pattern declares — no independent copy in the workflow to drift. +// Tag names mitool registers for this pipeline, and the SINGLE source of truth. They are baked into every +// assembled pattern here AND sent to the workflow in args.tags, so the downstream commands +// (`refine-tags -t CELL -t FEATURE -u UMI`, `tag-stat`, per_cell) reference exactly the names the pattern +// declares. The workflow keeps no independent copy that could drift. export const CELL_TAG = "CELL"; export const UMI_TAG = "UMI"; export const FEATURE_TAG = "FEATURE"; -/** Structured read geometry for the UI pattern builder. UI-facing only: never projected into BlockArgs - * (the workflow consumes the assembled pattern string / derived lengths) and never persisted on its own - * (it is derived from the pattern string on demand). `featureOffset` is an anonymous `N{skip}` — it is - * not a named tag, so it appears here (a builder field that must round-trip) but nowhere in the - * workflow commands. */ +/** Structured read geometry for the UI pattern builder. UI-facing only. Never projected into BlockArgs, + * since the workflow consumes the assembled pattern string and the derived lengths, and never persisted + * on its own, since it is derived from the pattern string on demand. `featureOffset` is an anonymous + * `N{skip}` rather than a named tag, so it appears here as a builder field that must round-trip, and + * nowhere in the workflow commands. */ export type PatternParts = { cellLen: number; // CELL barcode length on Read 1 umiLen: number; // UMI length on Read 1 - r1TrailingWildcard: boolean; // trailing `*` on Read 1 — tolerate sequence past CELL+UMI (e.g. 28 nt R1) + r1TrailingWildcard: boolean; // trailing `*` on Read 1, to tolerate sequence past CELL+UMI (28 nt R1) featureLen: number; // FEATURE barcode length on Read 2 featureOffset: number; // leading N-skip before the feature barcode on Read 2 (0 = R2 position 0) }; -// The fixed BEAM shape. The trailing `*` on Read 1 is captured (group 3) so the builder can round-trip -// it; it tolerates R1 sequenced longer than CELL+UMI (e.g. the 28 nt R1 common in 5' runs). +// The fixed BEAM shape. The trailing `*` on Read 1 is captured as group 3 so the builder can round-trip +// it. It tolerates R1 sequenced longer than CELL+UMI, such as the 28 nt R1 common in 5' runs. const PATTERN_RE = /^\^\(CELL:N\{(\d+)\}\)\(UMI:N\{(\d+)\}\)(\*)?\\\^(?:N\{(\d+)\})?\(FEATURE:N\{(\d+)\}\)\(R2:\*\)$/; @@ -47,10 +48,10 @@ export function parsePattern(s: string): PatternParts | null { }; } -/** Loose validation for a user-supplied pattern (write mode + args). mitool does the real parsing; this - * only enforces what the block's downstream commands depend on: `refine-tags`/`tag-stat`/`per_cell` - * reference the CELL, UMI, and FEATURE tags plus the R2 capture by name, so those must be present. Any - * other content (constant flanks, an N-spacer, anchors) is passed to mitool verbatim. Returns null when +/** Loose validation for a user-supplied pattern (write mode and args). mitool does the real parsing. This + * enforces only what the block's downstream commands depend on: `refine-tags`, `tag-stat` and `per_cell` + * reference the CELL, UMI and FEATURE tags plus the R2 capture by name, so those must be present. Any + * other content -- constant flanks, an N-spacer, anchors -- goes to mitool verbatim. Returns null when * valid, else a message naming what is missing. */ export function validatePattern(s: string): string | null { const p = s.trim(); @@ -73,9 +74,9 @@ export function validatePattern(s: string): string | null { /** Assemble the mitool pattern string from structured parts (R1 trailing `*` and R2 offset optional). */ export function assemblePattern(p: PatternParts): string { const trailing = p.r1TrailingWildcard ? "*" : ""; - // Anonymous N-skip — bare (no parentheses). mitool reads `(...)` as a `(TAG:pattern)` group, so a - // parenthesized `(N{n})` is rejected with "Unexpected character in tag identifier"; a bare `N{n}` is - // matched but not captured, which is exactly what an offset should be. Verified against mitool 2.3.1. + // Anonymous N-skip, bare and without parentheses. mitool reads `(...)` as a `(TAG:pattern)` group, so it + // rejects a parenthesized `(N{n})` with "Unexpected character in tag identifier". A bare `N{n}` is + // matched but not captured, which is what an offset should be. Verified against mitool 2.3.1. const skip = p.featureOffset > 0 ? `N{${p.featureOffset}}` : ""; return ( `^(${CELL_TAG}:N{${p.cellLen}})(${UMI_TAG}:N{${p.umiLen}})${trailing}` + diff --git a/model/src/presets/generic/feature-barcode-umi.ts b/model/src/presets/generic/feature-barcode-umi.ts index 20ee56b..fc476a6 100644 --- a/model/src/presets/generic/feature-barcode-umi.ts +++ b/model/src/presets/generic/feature-barcode-umi.ts @@ -1,9 +1,9 @@ import type { Preset } from "../types"; // User-configurable single-cell feature-barcode kit. Covers non-BEAM-Core reagents on the same 5' -// chemistry — e.g. TotalSeq-C / next-gen antigen barcoding, whose 15 nt barcode sits behind a 10 nt -// lead on Read 2 (set the Read 2 offset to 10). "Feature barcode", not "amplicon": these are single-cell -// tag reads, not targeted-insert amplicons. +// chemistry, such as TotalSeq-C and next-gen antigen barcoding, whose 15 nt barcode sits behind a 10 nt +// lead on Read 2. Set the Read 2 offset to 10 for those. "Feature barcode", not "amplicon": these are +// single-cell tag reads, not targeted-insert amplicons. const preset: Preset = { id: "generic-fb-umi", vendor: "", diff --git a/model/src/presets/tenx/beam.ts b/model/src/presets/tenx/beam.ts index 9e7087c..af5a6d5 100644 --- a/model/src/presets/tenx/beam.ts +++ b/model/src/presets/tenx/beam.ts @@ -1,14 +1,14 @@ import type { Preset } from "../types"; -// Chromium Single Cell 5′ Barcode Enabled Antigen Mapping (BEAM) — the block's default and the only -// chemistry 10x supports for BEAM (unsupported on GEM-X/v3). 5′ v2: CELL 16 + UMI 10 on Read 1, 15 nt -// feature barcode at the start of Read 2. Geometrically identical for BEAM-Ab and BEAM-T. +// Chromium Single Cell 5′ Barcode Enabled Antigen Mapping (BEAM), the block's default and the only +// chemistry 10x supports for BEAM. GEM-X and v3 do not support it. 5′ v2: CELL 16 + UMI 10 on Read 1, +// 15 nt feature barcode at the start of Read 2. Geometrically identical for BEAM-Ab and BEAM-T. // Ref: https://www.10xgenomics.com/support/universal-five-prime-gene-expression/documentation/steps/experimental-design-and-planning/chromium-single-cell-5-barcode-enabled-antigen-mapping-beam-–-experimental-planning-guide const preset: Preset = { id: "tenx-beam", vendor: "10x Genomics", kit: "BEAM", - // `label` is the product name only; the UI appends " — {vendor}" for the dropdown. + // `label` is the product name only. The UI appends " — {vendor}" for the dropdown. label: "Chromium Single Cell 5′ BEAM", description: "Chromium Single Cell 5′ Barcode Enabled Antigen Mapping (BEAM): 16 nt cell barcode + 10 nt UMI on Read 1, 15 nt feature barcode at the start of Read 2. 5′ v2 chemistry; covers BEAM-Ab and BEAM-T.", diff --git a/model/src/presets/types.ts b/model/src/presets/types.ts index 11c2af7..68cc97e 100644 --- a/model/src/presets/types.ts +++ b/model/src/presets/types.ts @@ -1,13 +1,13 @@ -/** A read-geometry preset for the Feature Integration pattern builder (mirrors the shape used by - * blocks/peptide-extraction). A fixed-kit preset carries its `pattern` directly; a user-configurable - * preset leaves `pattern` empty and drives it from `data.pattern` via the UI builder. */ +/** A read-geometry preset for the Feature Integration pattern builder, mirroring the shape + * blocks/peptide-extraction uses. A fixed-kit preset carries its `pattern` directly. A user-configurable + * preset leaves `pattern` empty and drives it from `data.pattern` in the UI builder. */ export type Preset = { id: string; vendor: string; kit: string; label: string; // dropdown text description: string; - pattern: string; // "" for userConfigurable presets (pattern lives in data.pattern) + pattern: string; // "" for a userConfigurable preset, whose pattern lives in data.pattern notes?: string; userConfigurable?: boolean; }; diff --git a/model/src/types.ts b/model/src/types.ts index 3a01e09..8fa67ad 100644 --- a/model/src/types.ts +++ b/model/src/types.ts @@ -1,78 +1,306 @@ +import type { GraphMakerState } from "@milaboratories/graph-maker"; import type { ImportFileHandle, PlDataTableStateV2, PlRef } from "@platforma-sdk/model"; +/** + * Which baseline a count is read against. Selected, never inferred: two runs answered by different rules + * produce numbers that do not compare, and a scientist who did not choose the rule cannot know that + * happened. + * + * There is no "none". A baseline is required and a run without one does not happen, so an unselected + * choice is undefined here and `args()` refuses it -- rather than a fourth value meaning "answer every + * position unreliable", which costs what a real run costs and looks like a result at a glance. + */ +/** + * `"panel"` is RETIRED and no longer offered. `292-what-plays-the-baseline@8.2.1` names it "a fourth + * possibility ... named and not built", and the reason it gives is not panel size: the one tool that + * implements it decides at the CLONOTYPE, pooling a clone's cells into one vector before anything is + * tested, so no cell ever holds a state and there is nothing to vote on. Adopting it would carve a + * clonotype-level exception through the middle of the verdict model. `060-parameter-set@2.0.0` drops the + * 25-tag parameter with it. + * + * The member stays in the union so a project stored under it still parses. `args()` refuses such a run and + * names the replacement, rather than moving the choice itself -- a baseline nobody chose is a methodology + * nobody knows they used. + */ +export type ReferenceSource = "declared" | "panel" | "distribution"; + +/** + * How tags become identities. A RULE over declared properties, never a tag->identity map: a map is keyed + * by tags, which are known only after the block runs, so any editor for it writes an output back into + * data. Property column names are knowable at prerun, from the panel header the block already enumerates. + * Absent means one identity per tag. + * + * Several columns may be named, and the identity is the distinct combination of their values. Name + * antigen and concentration together, and the same antigen at two concentrations is two identities. + * + * `column` is the shape this rule had before it took a list. It stays readable because a project stored + * under it must keep running, and `groupingColumns()` is the one place that reads either. Never write it. + */ +export type GroupingRule = + | { by: "tag" } + | { by: "property"; columns: string[]; column?: never } + | { by: "property"; column: string; columns?: never }; + /** Workflow inputs (projected from BlockData by the args lambda; validated there). */ export type BlockArgs = { fbFastqRef: PlRef; // feature-barcode FASTQ column (from samples-and-data, result pool) tagFeatureCsvHandle: ImportFileHandle; // tag->feature CSV, user-uploaded barcodeSeqColumn: string; // CSV column holding the feature barcode (whitelist/panel) featureNameColumn: string; // CSV column holding the feature/antigen name - controlFeature?: string; // negative-control feature name; omitted -> no score - dominanceThreshold: number; // default 0.6, floor 0.5 pattern: string; // Mitool tag pattern // mitool tag names baked into `pattern` tags: { cell: string; umi: string; feature: string }; - // Sample-aware tag→feature mapping (optional). When set, the same feature barcode may map to different + // Sample-aware tag->feature mapping (optional). When set, the same feature barcode may map to different // features per sample. sampleColumn?: string; // the CSV column holding the (user-friendly) sample name - sampleLabels?: Record; // a snapshot of sampleId→name - // Cell-barcode whitelist for refine-tags CELL correction. "" = de-novo (default; non-10x/synthetic). - // A 10x built-in name (e.g. 737K-august-2016) makes cellIds match the VDJ producer by construction. + sampleLabels?: Record; // a snapshot of sampleId->name + // Cell-barcode whitelist for refine-tags CELL correction. "" = de-novo, the default for non-10x and + // synthetic input. A 10x built-in name such as 737K-august-2016 makes cellIds match the VDJ producer by + // construction. cellWhitelist: string; - // Optional mitool resource overrides (Advanced Settings). Undefined -> workflow defaults (4 CPUs; RAM - // sized by the input-blob formula). When set, perProcessMemGB is a hard fixed RAM request per sample. + // Optional mitool resource overrides (Advanced Settings). Undefined means workflow defaults: 8 CPUs, and + // RAM sized by the input-blob formula. When set, perProcessMemGB is a fixed RAM request per sample. perProcessCPUs?: number; perProcessMemGB?: number; - // Preview (dry-run): when set, mitool parse processes only the first `limitInput` reads per sample so - // the user can sanity-check settings before the full run. Omitted -> full run (all reads). Mirrors + // Preview (dry-run): when set, mitool parse processes only the first `limitInput` reads per sample, so + // the user can check settings before the full run. Omitted means a full run, every read. Mirrors // mixcr-clonotyping / demultiplex-fastq "Preview" mode. limitInput?: number; // Optional multi-barcode antigen combine mode. combineColumn names a tag-CSV column giving each - // feature's mode ("sum" = OR, the default; "all" = AND, feature called only when every member barcode - // fires). minUmi is the AND per-barcode "fired" floor (integer >= 1; workflow default 1). + // feature's mode: "sum" = OR, the default, and "all" = AND, where a feature is called only when every + // member barcode fires. minUmi is the AND per-barcode "fired" floor, an integer >= 1 defaulting to 1 in + // the workflow. combineColumn?: string; minUmi?: number; - // Optional off-target designation (F2). offtargetProperty names an imported per-feature property column - // (e.g. antigen_class); offtargetValues are that column's values marking a feature as off-target. Such - // features are excluded from the dominant call (like the control) and enable the "cross-reactive" label. - // Both present -> off-target-aware; omitted -> unchanged dominant call. - offtargetProperty?: string; - offtargetValues?: string[]; + // The aggregate-barcode detection knobs (qc_measures.py AGGREGATE_BARCODE_*). Undefined means the + // shipped default. These run inside qc_report.py, which is part of the per-sample mitool fan-out, so + // they belong here rather than beside the reading's own parameters below: moving one re-runs that + // sample's parse/refine-tags/tag-stat chain, the same cost combineColumn or minUmi already carry. + aggregateBarcodeIqrMultiplier?: number; + aggregateBarcodeMinUmiThreshold?: number; + aggregateBarcodeTopN?: number; + + // --- the binding reading ------------------------------------------------------------------------- + // Everything below reaches emit_verdicts.py through verdict-args.lib.tengo, and nothing below reaches + // the per-sample mitool fan-out. A change to how the counts are READ therefore recovers every per-sample + // body from cache and re-runs the verdict stage alone. + + // The single-cell V(D)J dataset ANCHOR (axes [pl7.app/sampleId, pl7.app/vdj/scClonotypeKey], + // pl7.app/isAnchor). Not a linker ref: the cell linker carries pl7.app/isLinkerColumn and is hidden in + // tables, so no user can pick it, and the workflow resolves it from this anchor by name. The anchor is + // receptor-scoped, so choosing the dataset is choosing the receptor, which is what lets a BCR + TCR run + // bring two linkers without a panic. REQUIRED: `args()` refuses a run without one. It stays optional in + // this type because the workflow still carries its no-dataset branch, which is now unreachable and left + // as a guard rather than deleted. + datasetRef?: PlRef; + // The panel column declaring each tag's role, and the values of it that mark a tag as the comparator. + roleColumn?: string; + referenceValues?: string[]; + referenceSource?: ReferenceSource; + panelReferenceMinMembers: number; // members the panel needs before its own readings can serve + // The one condition on reading a count against that tag's own distribution across the sample's cells. + // It GATES rather than tunes: below it the baseline the rung would produce is wrong rather than + // conservative, which is why it has no "off". There is no separation condition -- the rung's own atom + // refuses one, because no published test tells a real split from a dent and inventing one here would be + // this block deciding what the method leaves to the eye. + distributionMinCells: number; // cells a sample needs before the rung may serve + countFloor: number; // counts below this are not evidence of binding + boundCutoff: number; // specificity score (0-100) at or above which a cell binds + minVotingCells: number; // a verdict may rest on one cell and say so + // Share (0-1) of answering cells the majority must reach. Off by default, and off means ABSENT rather + // than zero: a floor of 0 passes every majority instead of skipping the check. + minAgreement?: number; + // The admissibility gate, in comparator UMIs. Undefined means off. Zero would set aside every cell, so + // the args lambda projects it only when positive. + gateThreshold?: number; + grouping?: GroupingRule; + // Identities declared to contend for one binding site. Canonicalised by the args lambda: each group + // sorted, groups sorted, groups of fewer than two members dropped. + contendingGroups?: string[][]; + + // The four inherited lines (`315-where-the-lines-come-from`), each undefined meaning the shipped + // default. All four are round numbers carried over from the field rather than calibrated for this + // assay; error is absent for readsPerCellWarn because the field published one boundary. + cellBarcodeValidWarn?: number; + cellBarcodeValidError?: number; + readsPerCellWarn?: number; + aggregateBarcodeWarn?: number; + aggregateBarcodeError?: number; + undeclaredBarcodeWarn?: number; + undeclaredBarcodeError?: number; + usableReadWarn?: number; + usableReadError?: number; +}; + +/** + * What the block knows about the tag->feature CSV without running anything: the panel's headers, the + * distinct values of each header, and how many non-blank data rows there were. + * + * Parsed in the UI, from the file itself. `rowCount` stays optional because a project stored before the + * count existed must keep opening, and the duplicate-mapping gate skips where it is absent. + */ +export type CsvMeta = { + columns: string[]; + valuesByColumn: Record; + rowCount?: number; }; /** Unified persisted UI state. */ export type BlockData = { fbFastqRef?: PlRef; tagFeatureCsvHandle?: ImportFileHandle; + /** + * The panel CSV's metadata, parsed in the UI, tagged with the handle it was read from. + * + * The handle tag is what makes it safe to persist: a snapshot is read only while it matches the CSV + * currently picked, so a stale one left by a failed clear can never be read against a different file. + * + * This is the ONLY source of the metadata -- no workflow step parses the panel. The UI fills it from the + * user's disk on a local pick, and from the prerun-imported blob for a remote pick. Absent means the + * bytes have not arrived yet, which the "Reading columns..." alert reports. + */ + csvMetaSnapshot?: { handle: ImportFileHandle; meta: CsvMeta }; + /** + * Why the panel CSV could not be read, or undefined where it could. Shown to the user rather than + * logged: with no workflow-side parser to fall back on, a discarded parse error leaves empty dropdowns + * and nothing that says why. + */ + csvImportError?: string; barcodeSeqColumn?: string; featureNameColumn?: string; - controlFeature?: string; sampleColumn?: string; sampleLabelSnapshot?: Record; - // Distinct values of the chosen sample column at pick time — snapshotted alongside the label map so - // args() can gate Run purely from data (block when a dataset sample has no rows in the CSV). + // Distinct values of the chosen sample column at pick time, snapshotted alongside the label map so + // args() can gate Run purely from data. Run is blocked when a dataset sample has no rows in the CSV. sampleColumnValues?: string[]; - dominanceThreshold: number; - // Preview (dry-run) mode. "full" (default) processes all reads; "dry" caps mitool parse to `limitInput` - // reads per sample so the user can check settings first. Mirrors mixcr-clonotyping / demultiplex-fastq. + // Preview (dry-run) mode. "full", the default, processes every read. "dry" caps mitool parse to + // `limitInput` reads per sample so the user can check settings first. Mirrors mixcr-clonotyping and + // demultiplex-fastq. runMode?: "dry" | "full"; limitInput?: number; // Optional multi-barcode antigen combine mode. combineColumn names a tag-CSV column giving each - // feature's mode ("sum" = OR, the default; "all" = AND, feature called only when every member - // barcode fires). minUmi is the AND per-barcode "fired" floor (integer >= 1; workflow default 1). + // feature's mode: "sum" = OR, the default, and "all" = AND, where a feature is called only when every + // member barcode fires. minUmi is the AND per-barcode "fired" floor, an integer >= 1 defaulting to 1. combineColumn?: string; minUmi?: number; - // Optional off-target designation (F2). offtargetProperty names an imported per-feature property column - // (e.g. antigen_class); offtargetValues are that column's values marking a feature as off-target. Both - // present -> the dominant call excludes those features and enables the "cross-reactive" label. - offtargetProperty?: string; - offtargetValues?: string[]; + // The aggregate-barcode detection knobs. See BlockArgs for what each means to the reading. + aggregateBarcodeIqrMultiplier?: number; + aggregateBarcodeMinUmiThreshold?: number; + aggregateBarcodeTopN?: number; + + // --- the binding reading ------------------------------------------------------------------------- + // See BlockArgs for what each one means to the reading. The notes here are about the DATA layer only. + + /** + * The single-cell V(D)J dataset anchor, and the block's one optional input. A missing dataset narrows + * what the block can answer, since no clonotype set means no verdict, and stops nothing. The args lambda + * never throws on its absence. + */ + datasetRef?: PlRef; + roleColumn?: string; + /** + * The panel's headers as they stood when the role column or the grouping column was picked. Both of + * those name a panel column, and emit_verdicts.py exits the whole run when the panel does not carry the + * one it was given. The user meets that as a dead run, not as a message about the setting that caused + * it. args() validates from data alone, so the headers have to BE in data. They are snapshotted on the + * pick gesture, exactly as sampleColumnValues is. + */ + panelColumnSnapshot?: string[]; + referenceValues?: string[]; + referenceSource?: ReferenceSource; + panelReferenceMinMembers: number; + distributionMinCells: number; + countFloor: number; + boundCutoff: number; + minVotingCells: number; + minAgreement?: number; + gateThreshold?: number; + grouping?: GroupingRule; + // The four inherited lines. See BlockArgs for what each means to the reading. + cellBarcodeValidWarn?: number; + cellBarcodeValidError?: number; + readsPerCellWarn?: number; + aggregateBarcodeWarn?: number; + aggregateBarcodeError?: number; + undeclaredBarcodeWarn?: number; + undeclaredBarcodeError?: number; + usableReadWarn?: number; + usableReadError?: number; + /** + * Written on a user gesture only. The identities to choose from come from the identityOptions model + * output, and a watcher that copied that output into data would make the model output depend on the data + * it feeds -- a write-on-read loop, and a write race between two open clients. + */ + contendingGroups?: string[][]; + punchcardTableState: PlDataTableStateV2; // punchcard grid state (UI-only, never projected to args) + /** + * The clonotype whose expansion is open, as the readout grid's own row key, or undefined when none is. + * UI-only, never projected to args: opening an expansion must not re-run anything. + * + * A whole key rather than a bare string, because the grid hands back a `PTableKey` and `expansionTable` + * turns it straight into an axis filter. Undefined is load-bearing rather than an empty state: the + * output returns no table while it holds, because a table built with no filter would be every + * clonotype's rows at once, the one outcome the expansion exists to avoid. + */ + expandedSet?: (string | number)[]; + /** + * Grid state for the expansion table. UI-only, never projected to args. + * + * Optional, unlike the card's own state beside it. A required field would need every stored project + * migrated to carry it, and `createPlDataTableV3` already takes `tableState` as optional. A project that + * predates the expansion opens with a default grid instead of failing to open. + */ + expansionTableState?: PlDataTableStateV2; + /** + * Grid state for the expansion's BY-CELL face. It has to be separate from the state beside it: the two + * tabs are different tables over different axes, one row per identity against one row per cell, so a + * shared state would carry one tab's column order and filters into the other, where none of the column + * ids resolve. Optional for the same reason as above. + */ + cellExpansionTableState?: PlDataTableStateV2; + // No field narrows which identity columns the punchcard shows, and none should be added. + // PlAgDataTableV2 ships a columns panel and a filters panel, so such a field re-implements in block state + // what the grid already does, and two narrowing mechanisms can disagree where the grid's own cannot + // disagree with itself. Every identity column is rendered. + // + // No field truncates the punch headers either. A cut header hides which identity a column is, which is + // what a reader needs from it most. Every column is resizable, and the punch hover carries the name. A + // `punchcardIdentities` list or a `punchcardFullLabels` flag stored by an older project is ignored. + + // Snapshotted on the gesture that picks the barcode column, so args() can refuse a mapping that is + // certain to fail without reading an output. args is data-only, and these are read from the + // csvValuesByColumn / csvRowCount OUTPUTS, which lag a gesture by one round trip even though the metadata + // they derive from is now in data. Same device as sampleColumnValues, for the same reason. Absent where + // the metadata had not arrived at pick time, or predates rowCount. The gate then does not fire, and the + // Python guard catches it at the end of the run. + // + // Both could be dropped now that csvMetaSnapshot puts the same numbers in data, where args() could read + // them directly. That is a migration and a change to the gate, so it is deliberately left alone here. + panelRowCount?: number; + panelBarcodeDistinct?: number; + presetId?: string; pattern?: string; cellWhitelist?: string; // optional (defaults to "" = de-novo); see BlockArgs.cellWhitelist - // Optional mitool resource overrides (Advanced Settings); undefined = workflow defaults. + // Optional mitool resource overrides (Advanced Settings). Undefined means workflow defaults. perProcessCPUs?: number; perProcessMemGB?: number; defaultBlockLabel?: string; // UI-only: sidebar subtitle, mirrored from the suggestedBlockLabel output tableState: PlDataTableStateV2; // per-cell results grid state (UI-only, never projected to args) qcSummaryTableState: PlDataTableStateV2; // per-sample QC summary grid state (UI-only) + // The Run quality page's two grids (UI-only). Deliberately NOT named `antigenQcTableState` / + // `panelMismatchTableState`: the v3 -> v4 migration strips those two keys. Reusing them would either + // fight that strip or resurrect a column set and filter saved against a frame nobody has looked at + // since. A stored grid state means something only against the frame it was saved on. + runQualityTableState: PlDataTableStateV2; // run-level quality measurements grid state + // GraphMaker's own chart configuration, one per plot. Opaque to this block: the widget owns the + // shape and reads it back. Separate rather than shared, because picking an axis on one chart must + // not move another. + scoreDistributionGraphState: GraphMakerState; + referenceReadingGraphState: GraphMakerState; + fittedBackgroundGraphState: GraphMakerState; + runQualityMismatchTableState: PlDataTableStateV2; // panel-versus-reads mismatch grid state + reagentTableState: PlDataTableStateV2; // per (tag, identity) reagent grid state + // 330's own table: barcodes the reads carried that no panel declares, keyed by sequence. + undeclaredBarcodesTableState: PlDataTableStateV2; }; diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 1b1a6b4..e4abd41 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -9,18 +9,21 @@ catalogs: '@changesets/cli': specifier: 2.29.8 version: 2.29.8 + '@milaboratories/graph-maker': + specifier: 1.4.8 + version: 1.4.8 '@milaboratories/helpers': specifier: 1.14.5 version: 1.14.5 '@milaboratories/ts-builder': - specifier: 1.6.1 - version: 1.6.1 + specifier: 1.6.2 + version: 1.6.2 '@milaboratories/ts-configs': - specifier: 1.3.1 - version: 1.3.1 + specifier: 1.4.0 + version: 1.4.0 '@platforma-open/milaboratories.runenv-python-3': - specifier: 1.7.8 - version: 1.7.8 + specifier: 1.11.6 + version: 1.11.6 '@platforma-open/milaboratories.samples-and-data': specifier: 1.13.3 version: 1.13.3 @@ -31,20 +34,20 @@ catalogs: specifier: 2.3.1-131-main version: 2.3.1-131-main '@platforma-sdk/block-tools': - specifier: 2.12.8 - version: 2.12.8 + specifier: 2.14.0 + version: 2.14.0 '@platforma-sdk/model': - specifier: 1.80.8 - version: 1.80.8 + specifier: 1.81.1 + version: 1.81.1 '@platforma-sdk/tengo-builder': - specifier: 4.0.19 - version: 4.0.19 + specifier: 4.0.22 + version: 4.0.22 '@platforma-sdk/test': - specifier: 1.80.9 - version: 1.80.9 + specifier: 1.81.3 + version: 1.81.3 '@platforma-sdk/ui-vue': - specifier: 1.80.9 - version: 1.80.9 + specifier: 1.81.1 + version: 1.81.1 '@platforma-sdk/workflow-tengo': specifier: 6.8.2 version: 6.8.2 @@ -54,18 +57,21 @@ catalogs: ag-grid-vue3: specifier: ~34.1.2 version: 34.1.2 + csv-parse: + specifier: ^6.1.0 + version: 6.2.1 shx: specifier: 0.4.0 version: 0.4.0 turbo: - specifier: 2.8.11 - version: 2.8.11 + specifier: 2.10.10 + version: 2.10.10 typescript: specifier: ~5.9.3 version: 5.9.3 vitest: - specifier: ~4.0.18 - version: 4.0.18 + specifier: ~4.1.10 + version: 4.1.10 vue: specifier: 3.5.24 version: 3.5.24 @@ -86,25 +92,25 @@ importers: version: 2.29.8(@types/node@25.3.2) '@milaboratories/ts-builder': specifier: 'catalog:' - version: 1.6.1(@types/node@25.3.2)(rollup@4.53.3)(vue@3.5.25(typescript@5.9.3))(yaml@2.8.1) + version: 1.6.2(@types/node@25.3.2)(rollup@4.53.3)(vue@3.5.41(typescript@5.9.3))(yaml@2.8.1) '@platforma-sdk/block-tools': specifier: 'catalog:' - version: 2.12.8(@types/node@25.3.2) + version: 2.14.0(@types/node@25.3.2) shx: specifier: 'catalog:' version: 0.4.0 turbo: specifier: 'catalog:' - version: 2.8.11 + version: 2.10.10 block: devDependencies: '@milaboratories/ts-builder': specifier: 'catalog:' - version: 1.6.1(@types/node@25.3.2)(rollup@4.53.3)(vue@3.5.25(typescript@5.9.3))(yaml@2.8.1) + version: 1.6.2(@types/node@25.3.2)(rollup@4.53.3)(vue@3.5.41(typescript@5.9.3))(yaml@2.8.1) '@milaboratories/ts-configs': specifier: 'catalog:' - version: 1.3.1 + version: 1.4.0 '@platforma-open/milaboratories.feature-integration.model': specifier: workspace:* version: link:../model @@ -116,10 +122,10 @@ importers: version: link:../workflow '@platforma-sdk/block-tools': specifier: 'catalog:' - version: 2.12.8(@types/node@25.3.2) + version: 2.14.0(@types/node@25.3.2) '@platforma-sdk/model': specifier: 'catalog:' - version: 1.80.8 + version: 1.81.1 shx: specifier: 'catalog:' version: 0.4.0 @@ -129,12 +135,15 @@ importers: model: dependencies: + '@milaboratories/graph-maker': + specifier: 'catalog:' + version: 1.4.8(@milaboratories/pl-model-common@1.48.0)(@platforma-sdk/model@1.81.1)(@platforma-sdk/ui-vue@1.81.1(@bytecodealliance/preview2-shim@0.17.8)(typescript@5.6.3))(d3-dispatch@3.0.1)(d3-path@3.1.0)(d3-scale-chromatic@3.1.0)(typescript@5.6.3) '@milaboratories/helpers': specifier: 'catalog:' version: 1.14.5 '@platforma-sdk/model': specifier: 'catalog:' - version: 1.80.8 + version: 1.81.1 '@types/node': specifier: '*' version: 25.3.2 @@ -144,22 +153,22 @@ importers: devDependencies: '@milaboratories/ts-builder': specifier: 'catalog:' - version: 1.6.1(@types/node@25.3.2)(rollup@4.53.3)(vue@3.5.25(typescript@5.6.3))(yaml@2.8.1) + version: 1.6.2(@types/node@25.3.2)(rollup@4.53.3)(vue@3.5.41(typescript@5.6.3))(yaml@2.8.1) '@milaboratories/ts-configs': specifier: 'catalog:' - version: 1.3.1 + version: 1.4.0 '@platforma-sdk/block-tools': specifier: 'catalog:' - version: 2.12.8(@types/node@25.3.2) + version: 2.14.0(@types/node@25.3.2) software/per-cell-metrics: devDependencies: '@platforma-open/milaboratories.runenv-python-3': specifier: 'catalog:' - version: 1.7.8 + version: 1.11.6 '@platforma-sdk/block-tools': specifier: 'catalog:' - version: 2.12.8(@types/node@25.3.2) + version: 2.14.0(@types/node@25.3.2) test: dependencies: @@ -174,7 +183,7 @@ importers: version: 1.11.2 '@platforma-sdk/model': specifier: 'catalog:' - version: 1.80.8 + version: 1.81.1 this-block: specifier: workspace:@platforma-open/milaboratories.feature-integration@* version: link:../block @@ -184,25 +193,31 @@ importers: devDependencies: '@milaboratories/ts-builder': specifier: 'catalog:' - version: 1.6.1(@types/node@25.3.2)(rollup@4.53.3)(vue@3.5.25(typescript@5.6.3))(yaml@2.8.1) + version: 1.6.2(@types/node@25.3.2)(rollup@4.53.3)(vue@3.5.41(typescript@5.6.3))(yaml@2.8.1) '@milaboratories/ts-configs': specifier: 'catalog:' - version: 1.3.1 + version: 1.4.0 '@platforma-sdk/test': specifier: 'catalog:' - version: 1.80.9(@bytecodealliance/preview2-shim@0.17.8)(@types/node@25.3.2)(vite@7.2.7(@types/node@25.3.2)(lightningcss@1.32.0)(yaml@2.8.1)) + version: 1.81.3(@bytecodealliance/preview2-shim@0.17.8)(@types/node@25.3.2)(vite@8.0.8(@types/node@25.3.2)(yaml@2.8.1)) vitest: specifier: 'catalog:' - version: 4.0.18(@types/node@25.3.2)(lightningcss@1.32.0)(yaml@2.8.1) + version: 4.1.10(@types/node@25.3.2)(@vitest/coverage-istanbul@4.1.4)(vite@8.0.8(@types/node@25.3.2)(yaml@2.8.1)) ui: dependencies: + '@milaboratories/graph-maker': + specifier: 'catalog:' + version: 1.4.8(@milaboratories/pl-model-common@1.48.0)(@platforma-sdk/model@1.81.1)(@platforma-sdk/ui-vue@1.81.1(@bytecodealliance/preview2-shim@0.17.8)(typescript@5.6.3))(d3-dispatch@3.0.1)(d3-path@3.1.0)(d3-scale-chromatic@3.1.0)(typescript@5.6.3) '@platforma-open/milaboratories.feature-integration.model': specifier: workspace:* version: link:../model + '@platforma-sdk/model': + specifier: 'catalog:' + version: 1.81.1 '@platforma-sdk/ui-vue': specifier: 'catalog:' - version: 1.80.9(@bytecodealliance/preview2-shim@0.17.8)(typescript@5.6.3) + version: 1.81.1(@bytecodealliance/preview2-shim@0.17.8)(typescript@5.6.3) '@types/node': specifier: '*' version: 25.3.2 @@ -212,6 +227,9 @@ importers: ag-grid-vue3: specifier: 'catalog:' version: 34.1.2(vue@3.5.24(typescript@5.6.3)) + csv-parse: + specifier: 'catalog:' + version: 6.2.1 typescript: specifier: '*' version: 5.6.3 @@ -221,10 +239,10 @@ importers: devDependencies: '@milaboratories/ts-builder': specifier: 'catalog:' - version: 1.6.1(@types/node@25.3.2)(rollup@4.53.3)(vue@3.5.24(typescript@5.6.3))(yaml@2.8.1) + version: 1.6.2(@types/node@25.3.2)(rollup@4.53.3)(vue@3.5.24(typescript@5.6.3))(yaml@2.8.1) '@milaboratories/ts-configs': specifier: 'catalog:' - version: 1.3.1 + version: 1.4.0 workflow: dependencies: @@ -240,16 +258,19 @@ importers: devDependencies: '@platforma-sdk/tengo-builder': specifier: 'catalog:' - version: 4.0.19 + version: 4.0.22 '@platforma-sdk/test': specifier: 'catalog:' - version: 1.80.9(@bytecodealliance/preview2-shim@0.17.8)(@types/node@25.3.2)(vite@8.0.8(@types/node@25.3.2)(yaml@2.8.1)) + version: 1.81.3(@bytecodealliance/preview2-shim@0.17.8)(@types/node@25.3.2)(vite@8.0.8(@types/node@25.3.2)(yaml@2.8.1)) shx: specifier: 'catalog:' version: 0.4.0 packages: + '@ag-grid-community/core@32.3.9': + resolution: {integrity: sha512-oZeAEPgaJVMzfKqbAPCyadcN5+iy+tjvhRLqEYJdBxtLgW/s2s0qXcXQvnrz7eUMD3Z7h3BQRVt2h/p0T6Ox/w==} + '@ast-grep/napi-darwin-arm64@0.36.3': resolution: {integrity: sha512-uM0Hrm5gcHqaBL64ktmPBFMTorTlPKWsUfi0E2Cg09GJfeYWvZmicCqgd7qVtjURmQvFQdb4JSqHIkJvws6Uqw==} engines: {node: '>= 10'} @@ -511,6 +532,10 @@ packages: resolution: {integrity: sha512-qMlSxKbpRlAridDExk92nSobyDdpPijUq2DW6oDnUqd0iOGxmQjyqhMIihI9+zv4LPyZdRje2cavWPbCbWm3eA==} engines: {node: '>=6.9.0'} + '@babel/helper-string-parser@7.29.7': + resolution: {integrity: sha512-Pb5ijPrZ89GDH8223L4UP8i6QApWxs04RbPQJTeWDV0/keR2E36MeKnyr6LYmUUvqRRI+Iv87SuF1W6ErINzYw==} + engines: {node: '>=6.9.0'} + '@babel/helper-string-parser@8.0.0': resolution: {integrity: sha512-6mJgmFFFIIO82vvoLt9XtRC7/TkzXfts1t/SpRX4IHSzMgqoPYCWesVu1udUPUWioAE/2fcG6WuI8zrkE1gwrg==} engines: {node: ^22.18.0 || >=24.11.0} @@ -519,6 +544,10 @@ packages: resolution: {integrity: sha512-qSs4ifwzKJSV39ucNjsvc6WVHs6b7S03sOh2OcHF9UHfVPqWWALUsNUVzhSBiItjRZoLHx7nIarVjqKVusUZ1Q==} engines: {node: '>=6.9.0'} + '@babel/helper-validator-identifier@7.29.7': + resolution: {integrity: sha512-qehxGkRj55h/ff8EMaJ+cYhyaKlHIxqYDn682wQD7RNp9UujOQsHog2uS0r2vzr4pW+sXf90NeeayjcNaX3fFg==} + engines: {node: '>=6.9.0'} + '@babel/helper-validator-identifier@8.0.2': resolution: {integrity: sha512-9Fr9QeyCAyi1BR1jKZ6uYQ24EIhQUx5ReHfQU7drOE+TPOb+w11/dsqLkMOT2U29OdCT71XajrOT8xDc1C7orA==} engines: {node: ^22.18.0 || >=24.11.0} @@ -536,6 +565,11 @@ packages: engines: {node: '>=6.0.0'} hasBin: true + '@babel/parser@7.29.8': + resolution: {integrity: sha512-E8lTAYNB1KW+FH+VGJuZM1ioAx2E6oVlvQFRrf5P8ZZmsiJXYAD9vTFV7yyEURNzgh1dFqMZuO6tUwcARbqFCA==} + engines: {node: '>=6.0.0'} + hasBin: true + '@babel/parser@8.0.0': resolution: {integrity: sha512-aLxAE+imI9bCcyaPrUDjBv3uSkWieifjLe0kuFOZF0zli0L6GCsTmsePnTr55adbIAgYz2zhN1vnFimCBUYcRQ==} engines: {node: ^22.18.0 || >=24.11.0} @@ -557,6 +591,10 @@ packages: resolution: {integrity: sha512-LwdZHpScM4Qz8Xw2iKSzS+cfglZzJGvofQICy7W7v4caru4EaAmyUuO6BGrbyQ2mYV11W0U8j5mBhd14dd3B0A==} engines: {node: '>=6.9.0'} + '@babel/types@7.29.8': + resolution: {integrity: sha512-Vj1jF3cPfxg7OAfoI7QnVKLoILlm2JF9pnVHrX8qx7AHMiYWT+NDAA7jChlNgRS4WTLc/fD1lXLmPixluj+3Gg==} + engines: {node: '>=6.9.0'} + '@babel/types@8.0.0': resolution: {integrity: sha512-K8ponJDxBwDHigkeFqaqT5wLGl4bTlwMafR8k7b5CPxr6Ww+UG9ls8Yx6Tcpboxu97eeGVEEyKcHmEyOwN1vSw==} engines: {node: ^22.18.0 || >=24.11.0} @@ -629,6 +667,56 @@ packages: resolution: {integrity: sha512-Ir+AOibqzrIsL6ajt3Rz3LskB7OiMVHqltZmspbW/TJuTVuyOMirVqAkjfY6JISiLHgyNqicAC8AyHHGzNd/dA==} engines: {node: '>=0.1.90'} + '@d3fc/d3fc-axis@3.0.7': + resolution: {integrity: sha512-S4pILxkQUkD7WQmimWxIEHfXUYEonlXWuWvMP6iq3KXL3d+cj4flvwNqLYZvSVaQDdK990cKkjrK/ZAKnReGlg==} + peerDependencies: + d3-scale: '*' + d3-selection: '*' + d3-shape: '*' + + '@d3fc/d3fc-chart@5.1.9': + resolution: {integrity: sha512-xFO9lDUi2wAkjfyWlVZuFwSWEgnk0WOEagR2R9KzQ+uxeg3oZOPzwE+AZbpALyQbjjQ2uIxcOk56xnmnnayk3w==} + peerDependencies: + d3-scale: '*' + d3-selection: '*' + + '@d3fc/d3fc-data-join@6.0.3': + resolution: {integrity: sha512-fd1D2Cl4YGjzl3gBhcrvTl/VxaSncY0ZcokWsN8ahtmk9DZK4DnAgHGrdecnXVLkOx+ANDcqxqscYz6MWXLbcA==} + peerDependencies: + d3-selection: '*' + + '@d3fc/d3fc-element@6.2.0': + resolution: {integrity: sha512-AvdZ3V4mVxF9dGYLiDCoqr3GhrFOUQEc1FcP20QEhQ3fJ3qYRwx7/uhL7G/L2xbe6k4delPgnLOvtoaDenhpZw==} + + '@d3fc/d3fc-pointer@3.0.3': + resolution: {integrity: sha512-hXY7LqliDEJBH/do4YZusdLoikLYlWoN7efPC7YKYJ8igoEQFa48BqEHcANLs1qH+r7vzL2SS8V39MzgnJ1yQA==} + peerDependencies: + d3-dispatch: '*' + d3-selection: '*' + + '@d3fc/d3fc-rebind@6.0.1': + resolution: {integrity: sha512-+ryBZ53ALMffbADwnFAtTYQJcT7PE5BwpducGYS0X6Jux6ESnp+fP+cDQvBGbDBOVqaziGnfeLeJXjtMnZujmQ==} + + '@d3fc/d3fc-series@6.1.3': + resolution: {integrity: sha512-OSbt60SohTIib1xihX9ufneyJY7s9Feg9hvXVyEBZEBkwNE1NaeesZJ4nkmslu39RWuwsvpK3apL/XuBw7i5WA==} + peerDependencies: + d3-array: '*' + d3-scale: '*' + d3-scale-chromatic: '*' + d3-selection: '*' + d3-shape: '*' + + '@d3fc/d3fc-shape@6.0.1': + resolution: {integrity: sha512-/dD3S8BWrOjO2mSptUmwe38V7KG4Kw6liIE5NXZJjX/XidfZhuDu7WWuya3i90HeNYDZNcs6Z+4qM3FnvlZf8g==} + peerDependencies: + d3-path: '*' + + '@d3fc/d3fc-webgl@3.2.1': + resolution: {integrity: sha512-yNYHW/tC05rrJs5fpVM5zdmInL8NH6UP09ZJ2tmKw62ELYHWySV8vuXOVZN3V/3WihPa60p2w23H0Hrp2b5qeg==} + peerDependencies: + d3-scale: '*' + d3-shape: '*' + '@dabh/diagnostics@2.0.3': resolution: {integrity: sha512-hrlQOIi7hAfzsMqlGSFyVucrx38O+j6wiGOf//H2ecvIEqYN4ADBSS2iLMh5UFyDunCNniUIPk/q3riFv45xRA==} @@ -650,168 +738,12 @@ packages: '@emnapi/wasi-threads@1.2.2': resolution: {integrity: sha512-c95qOXkHdydNKhscBTebqEC1CVAZpyqOfVfBzQ1qgzyl3gfeldUjIggDbIZgDKsHLgnsM+igH7TJ/eAasaVuMA==} - '@esbuild/aix-ppc64@0.25.12': - resolution: {integrity: sha512-Hhmwd6CInZ3dwpuGTF8fJG6yoWmsToE+vYgD4nytZVxcu1ulHpUQRAB1UJ8+N1Am3Mz4+xOByoQoSZf4D+CpkA==} - engines: {node: '>=18'} - cpu: [ppc64] - os: [aix] - - '@esbuild/android-arm64@0.25.12': - resolution: {integrity: sha512-6AAmLG7zwD1Z159jCKPvAxZd4y/VTO0VkprYy+3N2FtJ8+BQWFXU+OxARIwA46c5tdD9SsKGZ/1ocqBS/gAKHg==} - engines: {node: '>=18'} - cpu: [arm64] - os: [android] - - '@esbuild/android-arm@0.25.12': - resolution: {integrity: sha512-VJ+sKvNA/GE7Ccacc9Cha7bpS8nyzVv0jdVgwNDaR4gDMC/2TTRc33Ip8qrNYUcpkOHUT5OZ0bUcNNVZQ9RLlg==} - engines: {node: '>=18'} - cpu: [arm] - os: [android] - - '@esbuild/android-x64@0.25.12': - resolution: {integrity: sha512-5jbb+2hhDHx5phYR2By8GTWEzn6I9UqR11Kwf22iKbNpYrsmRB18aX/9ivc5cabcUiAT/wM+YIZ6SG9QO6a8kg==} - engines: {node: '>=18'} - cpu: [x64] - os: [android] - - '@esbuild/darwin-arm64@0.25.12': - resolution: {integrity: sha512-N3zl+lxHCifgIlcMUP5016ESkeQjLj/959RxxNYIthIg+CQHInujFuXeWbWMgnTo4cp5XVHqFPmpyu9J65C1Yg==} - engines: {node: '>=18'} - cpu: [arm64] - os: [darwin] - - '@esbuild/darwin-x64@0.25.12': - resolution: {integrity: sha512-HQ9ka4Kx21qHXwtlTUVbKJOAnmG1ipXhdWTmNXiPzPfWKpXqASVcWdnf2bnL73wgjNrFXAa3yYvBSd9pzfEIpA==} - engines: {node: '>=18'} - cpu: [x64] - os: [darwin] - - '@esbuild/freebsd-arm64@0.25.12': - resolution: {integrity: sha512-gA0Bx759+7Jve03K1S0vkOu5Lg/85dou3EseOGUes8flVOGxbhDDh/iZaoek11Y8mtyKPGF3vP8XhnkDEAmzeg==} - engines: {node: '>=18'} - cpu: [arm64] - os: [freebsd] - - '@esbuild/freebsd-x64@0.25.12': - resolution: {integrity: sha512-TGbO26Yw2xsHzxtbVFGEXBFH0FRAP7gtcPE7P5yP7wGy7cXK2oO7RyOhL5NLiqTlBh47XhmIUXuGciXEqYFfBQ==} - engines: {node: '>=18'} - cpu: [x64] - os: [freebsd] - - '@esbuild/linux-arm64@0.25.12': - resolution: {integrity: sha512-8bwX7a8FghIgrupcxb4aUmYDLp8pX06rGh5HqDT7bB+8Rdells6mHvrFHHW2JAOPZUbnjUpKTLg6ECyzvas2AQ==} - engines: {node: '>=18'} - cpu: [arm64] - os: [linux] - - '@esbuild/linux-arm@0.25.12': - resolution: {integrity: sha512-lPDGyC1JPDou8kGcywY0YILzWlhhnRjdof3UlcoqYmS9El818LLfJJc3PXXgZHrHCAKs/Z2SeZtDJr5MrkxtOw==} - engines: {node: '>=18'} - cpu: [arm] - os: [linux] - - '@esbuild/linux-ia32@0.25.12': - resolution: {integrity: sha512-0y9KrdVnbMM2/vG8KfU0byhUN+EFCny9+8g202gYqSSVMonbsCfLjUO+rCci7pM0WBEtz+oK/PIwHkzxkyharA==} - engines: {node: '>=18'} - cpu: [ia32] - os: [linux] - - '@esbuild/linux-loong64@0.25.12': - resolution: {integrity: sha512-h///Lr5a9rib/v1GGqXVGzjL4TMvVTv+s1DPoxQdz7l/AYv6LDSxdIwzxkrPW438oUXiDtwM10o9PmwS/6Z0Ng==} - engines: {node: '>=18'} - cpu: [loong64] - os: [linux] - - '@esbuild/linux-mips64el@0.25.12': - resolution: {integrity: sha512-iyRrM1Pzy9GFMDLsXn1iHUm18nhKnNMWscjmp4+hpafcZjrr2WbT//d20xaGljXDBYHqRcl8HnxbX6uaA/eGVw==} - engines: {node: '>=18'} - cpu: [mips64el] - os: [linux] - - '@esbuild/linux-ppc64@0.25.12': - resolution: {integrity: sha512-9meM/lRXxMi5PSUqEXRCtVjEZBGwB7P/D4yT8UG/mwIdze2aV4Vo6U5gD3+RsoHXKkHCfSxZKzmDssVlRj1QQA==} - engines: {node: '>=18'} - cpu: [ppc64] - os: [linux] - - '@esbuild/linux-riscv64@0.25.12': - resolution: {integrity: sha512-Zr7KR4hgKUpWAwb1f3o5ygT04MzqVrGEGXGLnj15YQDJErYu/BGg+wmFlIDOdJp0PmB0lLvxFIOXZgFRrdjR0w==} - engines: {node: '>=18'} - cpu: [riscv64] - os: [linux] - - '@esbuild/linux-s390x@0.25.12': - resolution: {integrity: sha512-MsKncOcgTNvdtiISc/jZs/Zf8d0cl/t3gYWX8J9ubBnVOwlk65UIEEvgBORTiljloIWnBzLs4qhzPkJcitIzIg==} - engines: {node: '>=18'} - cpu: [s390x] - os: [linux] - - '@esbuild/linux-x64@0.25.12': - resolution: {integrity: sha512-uqZMTLr/zR/ed4jIGnwSLkaHmPjOjJvnm6TVVitAa08SLS9Z0VM8wIRx7gWbJB5/J54YuIMInDquWyYvQLZkgw==} - engines: {node: '>=18'} - cpu: [x64] - os: [linux] - - '@esbuild/netbsd-arm64@0.25.12': - resolution: {integrity: sha512-xXwcTq4GhRM7J9A8Gv5boanHhRa/Q9KLVmcyXHCTaM4wKfIpWkdXiMog/KsnxzJ0A1+nD+zoecuzqPmCRyBGjg==} - engines: {node: '>=18'} - cpu: [arm64] - os: [netbsd] - - '@esbuild/netbsd-x64@0.25.12': - resolution: {integrity: sha512-Ld5pTlzPy3YwGec4OuHh1aCVCRvOXdH8DgRjfDy/oumVovmuSzWfnSJg+VtakB9Cm0gxNO9BzWkj6mtO1FMXkQ==} - engines: {node: '>=18'} - cpu: [x64] - os: [netbsd] - - '@esbuild/openbsd-arm64@0.25.12': - resolution: {integrity: sha512-fF96T6KsBo/pkQI950FARU9apGNTSlZGsv1jZBAlcLL1MLjLNIWPBkj5NlSz8aAzYKg+eNqknrUJ24QBybeR5A==} - engines: {node: '>=18'} - cpu: [arm64] - os: [openbsd] - - '@esbuild/openbsd-x64@0.25.12': - resolution: {integrity: sha512-MZyXUkZHjQxUvzK7rN8DJ3SRmrVrke8ZyRusHlP+kuwqTcfWLyqMOE3sScPPyeIXN/mDJIfGXvcMqCgYKekoQw==} - engines: {node: '>=18'} - cpu: [x64] - os: [openbsd] - - '@esbuild/openharmony-arm64@0.25.12': - resolution: {integrity: sha512-rm0YWsqUSRrjncSXGA7Zv78Nbnw4XL6/dzr20cyrQf7ZmRcsovpcRBdhD43Nuk3y7XIoW2OxMVvwuRvk9XdASg==} - engines: {node: '>=18'} - cpu: [arm64] - os: [openharmony] - - '@esbuild/sunos-x64@0.25.12': - resolution: {integrity: sha512-3wGSCDyuTHQUzt0nV7bocDy72r2lI33QL3gkDNGkod22EsYl04sMf0qLb8luNKTOmgF/eDEDP5BFNwoBKH441w==} - engines: {node: '>=18'} - cpu: [x64] - os: [sunos] - - '@esbuild/win32-arm64@0.25.12': - resolution: {integrity: sha512-rMmLrur64A7+DKlnSuwqUdRKyd3UE7oPJZmnljqEptesKM8wx9J8gx5u0+9Pq0fQQW8vqeKebwNXdfOyP+8Bsg==} - engines: {node: '>=18'} - cpu: [arm64] - os: [win32] - - '@esbuild/win32-ia32@0.25.12': - resolution: {integrity: sha512-HkqnmmBoCbCwxUKKNPBixiWDGCpQGVsrQfJoVGYLPT41XWF8lHuE5N6WhVia2n4o5QK5M4tYr21827fNhi4byQ==} - engines: {node: '>=18'} - cpu: [ia32] - os: [win32] - - '@esbuild/win32-x64@0.25.12': - resolution: {integrity: sha512-alJC0uCZpTFrSL0CCDjcgleBXPnCrEAhTBILpeAp7M/OFgoqtAetfBzX0xM00MUsVVPpVjlPuMbREqnZCXaTnA==} - engines: {node: '>=18'} - cpu: [x64] - os: [win32] - - '@grpc/grpc-js@1.13.4': - resolution: {integrity: sha512-GsFaMXCkMqkKIvwCQjCrwH+GHbPKBjhwo/8ZuUkWHqbI73Kky9I+pQltrlT0+MWpedCoosda53lgjYfyEPgxBg==} + '@grpc/grpc-js@1.14.4': + resolution: {integrity: sha512-k9Dj3DV/itK9D06Y8f190Qgop7/Ui+D0njFV3LHMPwPT75DpXLQohE9Wmz0QElrJnzsjB7KPWiKJbOl7IPDArQ==} engines: {node: '>=12.10.0'} - '@grpc/proto-loader@0.7.13': - resolution: {integrity: sha512-AiXO/bfe9bmxBjxxtYxFAXGZvMaN5s8kO+jBHAJCON8rJoB5YS/D6X7ZNc6XQkuHNmyl4CYaMI1fJ/Gn27RGGw==} + '@grpc/proto-loader@0.8.1': + resolution: {integrity: sha512-wtF6h+DY6M3YaDBPAmvuuA6jV8Sif9MjtOI5euKFWRgCDl5PeDpPsHR9u2l6St5ceY8AZgoNDww5+HvEsXFsGg==} engines: {node: '>=6'} hasBin: true @@ -961,20 +893,20 @@ packages: resolution: {integrity: sha512-ZXRY4jNvVgSVQ8DL3LTcakaAtXwTVUxE81hslsyD2AtoXW/wVob10HkOJ1X/pAlcI7D+2YoZKg5do8G/w6RYgA==} engines: {node: '>=8'} - '@jitl/quickjs-ffi-types@0.31.0': - resolution: {integrity: sha512-1yrgvXlmXH2oNj3eFTrkwacGJbmM0crwipA3ohCrjv52gBeDaD7PsTvFYinlAnqU8iPME3LGP437yk05a2oejw==} + '@jitl/quickjs-ffi-types@0.32.0': + resolution: {integrity: sha512-v9T+GQpmk43VDJ7d72sf0Nexhk+ArvtUihW27dy7lqAl0zBObFKtSBBIm5RBjwIhE8VwsPPm9PNuvPvNqLWUEg==} - '@jitl/quickjs-wasmfile-debug-asyncify@0.31.0': - resolution: {integrity: sha512-YkdzQdr1uaftFhgEnTRjTTZHk2SFZdpWO7XhOmRVbi6CEVsH9g5oNF8Ta1q3OuSJHRwwT8YsuR1YzEiEIJEk6w==} + '@jitl/quickjs-wasmfile-debug-asyncify@0.32.0': + resolution: {integrity: sha512-EX8zbXwGqCgAE764M+qvkHtyXDi/FUoMBea0JnES7vCM3P7a2+EOZOjGv85wtZ2sJhI1oJ+nekmqpOODFDY+hw==} - '@jitl/quickjs-wasmfile-debug-sync@0.31.0': - resolution: {integrity: sha512-8XvloaaWBONqcHXYs5tWOjdhQVxzULilIfB2hvZfS6S+fI4m2+lFiwQy7xeP8ExHmiZ7D8gZGChNkdLgjGfknw==} + '@jitl/quickjs-wasmfile-debug-sync@0.32.0': + resolution: {integrity: sha512-LeYWrPGC1uNCTBWvibo3ZLJj0CSVNYUXvJpXMCmuQ5Sap2cCACc3uvGvYV4homHHBAzfw5akoTqMMS4YFRtw+Q==} - '@jitl/quickjs-wasmfile-release-asyncify@0.31.0': - resolution: {integrity: sha512-uz0BbQYTxNsFkvkurd7vk2dOg57ElTBLCuvNtRl4rgrtbC++NIndD5qv2+AXb6yXDD3Uy1O2PCwmoaH0eXgEOg==} + '@jitl/quickjs-wasmfile-release-asyncify@0.32.0': + resolution: {integrity: sha512-3oSwPfja12ICz4aIblB58cuY8JlEq5Txt8Cut4VLo+LH47QN+mzCnSgnbB03hWzg1LBcc+VyyI9UOag7a1NF+Q==} - '@jitl/quickjs-wasmfile-release-sync@0.31.0': - resolution: {integrity: sha512-hYduecOByj9AsAfsJhZh5nA6exokmuFC8cls39+lYmTCGY51bgjJJJwReEu7Ff7vBWaQCL6TeDdVlnp2WYz0jw==} + '@jitl/quickjs-wasmfile-release-sync@0.32.0': + resolution: {integrity: sha512-BKNDI/TPBfGlLNGYpLrhcDGXmIk4xHm4MRAisOBnOzpXVn9HZWsfmMAc9WMBrAHjvvds6HOikKeaOBKdPdpVrg==} '@jridgewell/gen-mapping@0.3.13': resolution: {integrity: sha512-2kkt/7niJ6MgEPxF0bYdQ6etZaA+fQvDcLKckhy1yIQOzaoKjBBjSj63/aLVjYE3qhRt5dvM+uUyfCg6UKCBbA==} @@ -1019,13 +951,19 @@ packages: '@microsoft/tsdoc@0.16.0': resolution: {integrity: sha512-xgAyonlVVS+q7Vc7qLW0UrJU7rSFcETRWsqdXZtjzRU8dF+6CkozTK4V4y1LwOX7j8r/vHphjDeMeGI4tNGeGA==} - '@milaboratories/columns-collection-driver@0.2.2': - resolution: {integrity: sha512-wezb0Fu7iqwykw8ZRwMge72qXVr+eH2UEYsYGm7X38KQ3DVim2eWw3bKW+IcooFt/X18egsN8j4MPxUxUuVxcw==} + '@milaboratories/columns-collection-driver@0.2.3': + resolution: {integrity: sha512-3rNWmuQGvEaBEzGMHIWeOi1p8j8sMDpnPKNUQvT0Ji7/ArvnfM5HK8dywxB+n3mLK8c11lW16+auM5xuydJXGw==} '@milaboratories/computable@2.9.8': resolution: {integrity: sha512-X0ZtxOnIJlAd9Y7CyBtWSKHgVuTKXbIryMwJaoYSEz0gY3JZVnsD0f59J/lwe3Key0/lSYh3EbL/pP5jACIzfQ==} engines: {node: '>=22.19.0'} + '@milaboratories/graph-maker@1.4.8': + resolution: {integrity: sha512-Fo3wjr5lqSciVJddD+GbdzecK7+Oqab1d1SMWcDmS5Vr5j0djpPkG9pPpOUn1aQlrYV9Lj9yWte6pkzHJ1+1+Q==} + peerDependencies: + '@platforma-sdk/model': 1.79.14 + '@platforma-sdk/ui-vue': 1.79.14 + '@milaboratories/helpers@1.14.2': resolution: {integrity: sha512-zOkD4aWNbembwI+AsPTz7nmWC1VPA4rwGBc+Nd5jPpKVh7rCtZwQrlOP5mVqRVMKIAjb1nU8kzzCGfqNPqfJjA==} engines: {node: '>=22'} @@ -1034,12 +972,26 @@ packages: resolution: {integrity: sha512-Kiy0g7sEmFQxDwtnmTrdJ8XdUK0IDAB5ZnPCNEbK7lPGHIa+xG3kzfeR4y44QBdrYaK0NwG63D8Fc7dlv9KItg==} engines: {node: '>=22'} - '@milaboratories/pf-driver@1.8.4': - resolution: {integrity: sha512-ouTnOC+QfAF5AHLw6vAOoxwl/Kr8a3YvvBAmo5FstmVTzL57laVIu/QGpPTNBwwW5adUT7cv7Uou09c3t7uayw==} + '@milaboratories/miplots4@1.2.3': + resolution: {integrity: sha512-rhoF7iRcJWKUqJMQq8y84vlkqZn4Yk7tOV8Zef+W+GqVt9UYgV1XLJu/+Wp1GcsUVtPP8Xs5CRgfA/YfV+a4Hg==} + + '@milaboratories/pf-driver@1.9.0': + resolution: {integrity: sha512-OhMsf1jNMp31wRCrXHMU75WQj48Nc/lDeYTo4/6R6+FLYBYkK9yWgsSH+02V4WCKbUqsyF1Xy0+O+qsYjpqUDA==} engines: {node: '>=22.19.0'} - '@milaboratories/pf-spec-driver@1.4.23': - resolution: {integrity: sha512-/madO+vexJH92Utbz5Nh7rxKz9SnYUPZlsLZInWrMAJaXrIqcYZnCS1h2kBcNWJ4/9/YO/si7XtM1EDAmvImWw==} + '@milaboratories/pf-plots@1.4.6': + resolution: {integrity: sha512-NxUr9fFW1CP9CrV3osd2mjPNor2REawnwNuOLfCOJZvIqZj9hcoWJl8V76Zz6U8ruT7xo3od/rYMf0VUfsCktQ==} + peerDependencies: + '@milaboratories/pl-model-common': 1.46.1 + '@platforma-sdk/model': 1.79.14 + + '@milaboratories/pf-spec-driver@1.5.0': + resolution: {integrity: sha512-XWXrKCyCwcqS8EcukuyDqvMq0D6qt61zrkb3qGpWKyy0SVQNzXC2JL0ThCM2uGI7QQelgV0aagcg8xU5pGKWPw==} + + '@milaboratories/pf-spec@1.0.0': + resolution: {integrity: sha512-JIOhJdDHcKDItjlxOT4wVmnjejoazM69qK5ylu+g5q65RIcH/R5MF2OGiu9nV7QbLEXszll+EVnvx6jYhuE44w==} + peerDependencies: + '@bytecodealliance/preview2-shim': ^0.20.1 '@milaboratories/pframes-rs-node@1.1.56': resolution: {integrity: sha512-H6qltcR+HHb2kAy0U0zP90rqmv/MZGGkdXIyf89FUZTpoHOlXG4Ahxq7wXp9vviUczci4cLl2L0Q+dL2XGcsUA==} @@ -1051,26 +1003,23 @@ packages: '@milaboratories/pframes-rs-wasip2@1.1.56': resolution: {integrity: sha512-54bhC6XCAVO09J/sqVwEKA4hhTa27BDDNdH8BE+6+LvjBVkg/qq2/f0MzdrVijrmagbr97F1zgj5wIgUqwCSoA==} - '@milaboratories/pframes-rs-wasm@1.1.56': - resolution: {integrity: sha512-k/RQqiF+SwoOtFnYQ+i34Lzna8prOoFbLHaPY5oEUOrB/czehosFMzRQJAeFDKOxI/SbrH4D5jmWzTgUuhavLQ==} - peerDependencies: - '@bytecodealliance/preview2-shim': 0.17.9 - '@milaboratories/pl-model-common': 1.46.2 - '@milaboratories/pl-model-middle-layer': 1.30.7 + '@milaboratories/pl-client@3.14.6': + resolution: {integrity: sha512-IPzh5bSibr02DMKpwlCW6TuHVxaDe2o6tjaEJ4P2y5lshRbe5SBR/ne/G/N7E7KEKxmpWq7ZapOCLkLyvqFFDw==} + engines: {node: '>=22.19.0'} - '@milaboratories/pl-client@3.14.3': - resolution: {integrity: sha512-yyep+I0FozB1pbf22maRCVjgrO3AEB8f88PyTmJM0K7Y5bfq0T/MgprTyPwMB34u1XfKI9M5yqHWmyuYdzq08w==} + '@milaboratories/pl-client@3.14.7': + resolution: {integrity: sha512-HIjPfGAYRRD0hbq5YSTkWt5XgBiv+yYtw73EjWLxui8eUEeB2ffdgsvH7IhVx2oXjUOL4p7M4i8PBnmdJ3830w==} engines: {node: '>=22.19.0'} '@milaboratories/pl-config@1.8.5': resolution: {integrity: sha512-XnfYXSSkRxeImQ21k6I8y5apisvagcSgMGfEeyRxNdSGwUVJbbI8TwJk+XBEDQ4lErW8oqtLxKjFQuHJMzRUoQ==} - '@milaboratories/pl-deployments@3.0.13': - resolution: {integrity: sha512-uoM1fX6D8/3y6DmHHCj5A3aZVrmir5bZk+AgfdMp/QgCTwKyhvvE4uGUBMn27/uEJk54kFJ15KathxBA2UlD9A==} + '@milaboratories/pl-deployments@3.0.15': + resolution: {integrity: sha512-+i3bdvfPCXLZ789f5fRdtrbil3/RFyzSBNMkULcnyKY5u6GRZLjWDBC5aSE75PRTWx3Fxf743Fj9L3B6loMSWw==} engines: {node: '>=22.19.0'} - '@milaboratories/pl-drivers@1.16.11': - resolution: {integrity: sha512-lCmCIAtjybdJHW3ZBJvoDzbY9jyZQZeL/OawM9MqKOXBwZQy+E2Bi7G5IHJqG13hy8i6FvJQs2/IaWxRpfuInQ==} + '@milaboratories/pl-drivers@1.16.15': + resolution: {integrity: sha512-mVKIEHVOeAPdwSZZk+URAFD6zoyPiAARinpDfihDvkFcrlTLIzeoOvnQNPG5a8jqc0GiW+a9mnTrfjG7+n82Ww==} engines: {node: '>=22'} '@milaboratories/pl-error-like@1.12.10': @@ -1079,22 +1028,25 @@ packages: '@milaboratories/pl-error-like@1.12.5': resolution: {integrity: sha512-opYP4OrB6JBMsH9RMRmAH44+MG7PWiV08dHW9+RsXGOaqX+rYXs9TTBXYRhlVMDLwwefSKzelvDg8HL748aM+A==} - '@milaboratories/pl-errors@1.4.32': - resolution: {integrity: sha512-57bft0ieS2sYek6g+FiF5FMeBV1cyVgOchI8sfJAtc2gTlTH5Py8m/z13ZweV4+mGmIGDLNakzQvYGlgloCDmg==} + '@milaboratories/pl-errors@1.4.35': + resolution: {integrity: sha512-rq5Lg9G7ax+LtMPdrkZg+Pdqk28JViiS/KIcQfk0Cg1ttZoAnhDhFIh6rAFy2eGybPtlzXqM32/ODKUSMawc8Q==} - '@milaboratories/pl-healthcheck@1.0.4': - resolution: {integrity: sha512-VLoF7iW7px8BG+vTT/nQ+qkJDNSsXUivRsyZlbM7VCxKdVrZwPfn/rqQIJ4g6daBONVtCqUhAFi52IeXRg5mxg==} + '@milaboratories/pl-healthcheck@1.0.5': + resolution: {integrity: sha512-ZkQti4VU2FapJBFRtZGfR5NDPXEAEFgTf/NfemypgY0aA75fFPi4/c3BpXX5K7dht+JOgHqkMrBHxKIuE2T+fA==} engines: {node: '>=22.19.0'} '@milaboratories/pl-http@1.2.4': resolution: {integrity: sha512-QKmhx+WEvJCV9dUy/SBdQk/ApaJ5ewBFgm/b+XPlS10SusAdqUUTGvK5+hq8YSuUMXlHb/dk++UtI5YlDuDl2Q==} - '@milaboratories/pl-middle-layer@1.66.8': - resolution: {integrity: sha512-VJ0t6g7x917xGN7qKvG5KHcezmzp/6b/+/Wer+QLLJueEb5gmMpWvjtLyfjNbJG8v2X85IhUQKdXETVsR3l2nw==} + '@milaboratories/pl-middle-layer@1.66.19': + resolution: {integrity: sha512-rFBmJDdBcIrdDHz730p3sL8+7uUs7VimXYGQo9IbayxmAlp5mnft1XlPUTuVbwx5QeRNGv4Cwya1T+7WL1wSmw==} engines: {node: '>=22.19.0'} - '@milaboratories/pl-model-backend@1.4.17': - resolution: {integrity: sha512-ZoxNLmWs+MQxCU/s4ml5DHV9htu0xk9NHYQ+07AtWNhLRCjSeFcVTVQlt0EvFMlzytOKGjmMbCMhaKYUuh/wrA==} + '@milaboratories/pl-model-backend@1.4.20': + resolution: {integrity: sha512-AkV+PhmQms6WPwR3E75THrkD9fFE7OpfsIUC297HvnFUoCFXR6GOMGPworA/qipXReTmv192AqBpuaLdgAngyg==} + + '@milaboratories/pl-model-backend@1.4.21': + resolution: {integrity: sha512-Z2z5J8bglslgXXoi1aySi1ho+/d+ylJiuEaRw9XTNuE5iBM3EQPe0rw095pXGOinTlMpmvUcCI2wkVCraUQ1dA==} '@milaboratories/pl-model-common@1.23.0': resolution: {integrity: sha512-1uHb2pS+hWJyBKvfOlMYmjbFuWn+tNOULWj84mWASdqSjdYyHfVYbLq5esawM+dnZ2GBQIzJRbXrZYIMqH4Peg==} @@ -1102,24 +1054,30 @@ packages: '@milaboratories/pl-model-common@1.46.2': resolution: {integrity: sha512-VEeauisApYScvCS8lnK3zpFJ520xuTAodKJmjR8ulHcMrWMyWMfHEdGb7j5OMD0mM/OwTgmQrrJ5eB7Xd+xoOQ==} - '@milaboratories/pl-model-common@1.47.2': - resolution: {integrity: sha512-XCEcL+CHYxZ/S/y9zrpzzaeZO+ZKQZ3Y9/pEKEsgDZy4VLtXZGG0RHV9byNmMGz8Y+7BahQ9tAf1d26TdxRVZQ==} + '@milaboratories/pl-model-common@1.47.3': + resolution: {integrity: sha512-tXmKujm+6ru/Fh9hr9H3iRBWbS+JE0Q7FNualtzJpijaB3SNtScR4erLTamdANv5RYNn7kbYpDWr6wAjAOajrg==} - '@milaboratories/pl-model-middle-layer@1.30.14': - resolution: {integrity: sha512-gZwF0ux28uOnKFfCwjcD8beJCfzdyq1mqLgd/+T5zYCFsomTvPCMp2HYwzXmUp7poO/HTIAxSA6qT8As2q2WWg==} + '@milaboratories/pl-model-common@1.48.0': + resolution: {integrity: sha512-oCVrjFNmjQolb7YWnbSGHnp6GGVm1PhG1lnNABp9lqYjvA2ISjIPQLIo/vT2ca0mqwLrEvbuRqiqSFOg7sQ5tQ==} '@milaboratories/pl-model-middle-layer@1.30.7': resolution: {integrity: sha512-rs9x3Ron4ujR/UOdEgB8WUB1SvZ8ZAScT1Av/e4or+iiQ/CzhmK9nqtYamHVwKV+JgwIFbXQwxGvIHDVz955dQ==} - '@milaboratories/pl-tree@1.13.2': - resolution: {integrity: sha512-zFN+CGNyxSlv7cLcFBYMWreneXqrLNWqOkZlXQmHmqfocVrTZ5EYa9nE7L/6opwzF2VaYTYD/iaxNcUmpD68pA==} + '@milaboratories/pl-model-middle-layer@1.31.0': + resolution: {integrity: sha512-D1vyGABBtCYyp2IhKd9eMoZtKUj1wUYpIzz4Xz5k+bfJWGEF8GmZGiOvO4oOvrsEsXlG4NyO249MnBKmHFcppQ==} + + '@milaboratories/pl-model-middle-layer@1.32.0': + resolution: {integrity: sha512-X1iLGgOwzkw8mQ/GfTxfOTJakBJ26np85h5HqUziMbVkFL0SR7wpd7xpgUs6TKVkYNM8viAv3iA2k63Yc5SahQ==} + + '@milaboratories/pl-tree@1.13.6': + resolution: {integrity: sha512-R3wnMjbCNfAsyGJe3FQOqEyrNePd9ZCwkjDj2ps7d9oeiJ7QX6hYkIVuMlq8IQNYOJybfHc/dFEvWeYE3uAAZg==} engines: {node: '>=22.19.0'} '@milaboratories/ptabler-expression-js@1.1.9': resolution: {integrity: sha512-fH0gix6ObRI9/TgPk1S40EkdFLskSdh6AloNBhkedH7WyTwOmM7zmtjmp+n7TiRHzRfufqzwLOZ9TFb7qAXQRA==} - '@milaboratories/ptabler-expression-js@1.2.36': - resolution: {integrity: sha512-LiqCbEMc8VVSXfoFsZSTpSQyoxb5L5FQAe46EcS2vRMrmMcTZwR2QGXH8nfr0oeS4MyjZepsh+Iv+H7VWK7bkA==} + '@milaboratories/ptabler-expression-js@1.2.37': + resolution: {integrity: sha512-urVRk4b5Jse555euNNITlOJ437McZVtBnC5h/E6O+iHm+hfNIi6OP1aAD5ai4jOpHllm85zW5Ne7hVyyd71bJw==} '@milaboratories/resolve-helper@1.1.3': resolution: {integrity: sha512-38/dW/XRZQREOxAOOKtO0lzEWPCP/DH0qhB3q1kYcGoN++5V92/zbVwbYrMDeDcjTyo+D62iIep+sKXeWHa7Uw==} @@ -1132,19 +1090,19 @@ packages: os: [darwin, linux, win32] hasBin: true - '@milaboratories/ts-builder@1.6.1': - resolution: {integrity: sha512-0m+I8bdxw6mGTfPt+xW8OGvburrxLUpI/QZsyACHsXwyfVRfDKvWB3/9Hm053KzwHqrzpeuvidEBdKRVRAi9KQ==} + '@milaboratories/ts-builder@1.6.2': + resolution: {integrity: sha512-6pS36U5EL2IBLIpoP0/5oSFA9jzjcJmYhgEODzt5kglovTzJuh15nEmJMeChr0OpFtGFjQT6o37qHerDGETIpA==} hasBin: true - '@milaboratories/ts-configs@1.3.1': - resolution: {integrity: sha512-MfLF+qgDwnD2BuncGzFqQxKuqq/0KtXcXXftcvp8E08xY9cl5kkmBHX/H8RYAH4FvF6ghb56c3I6iaxIa5xIUw==} + '@milaboratories/ts-configs@1.4.0': + resolution: {integrity: sha512-VzU9D+RiggsG4VYteJSBN4o8IjYae9GbZ8MofikMJQLOI1Ir9/pVyPXXGnG/TAGvmcjbK/psAiZVGSvI5MBPGg==} '@milaboratories/ts-helpers@1.8.6': resolution: {integrity: sha512-ef01tARUl+0Urt3x8HHAByrhYHg4Rnn7WiFyJ+joZFZl1T1pUKTgfB7Zxc8Cn06HIl4i6H7s5OSymjZePIGqfQ==} engines: {node: '>=22.19.0'} - '@milaboratories/uikit@2.15.17': - resolution: {integrity: sha512-P0DRxZY0y2nYpoISTuKtsuTLPsetdPnS8Z0FWJglDqrh3JBkhdyFd2WbPMukhZXY+FldklgmXvEmNLUis7dnKA==} + '@milaboratories/uikit@2.15.23': + resolution: {integrity: sha512-FrZPJsa3jry4juVofR42IPCHv+oKPAcdBL+OidX0/S2lhIAsCIrEKBcAN3yzWd2yvyAPb3qzt0hbwpuTY0UVlw==} '@napi-rs/wasm-runtime@1.1.6': resolution: {integrity: sha512-ZLv/JdUfkvOy9eCnnBaGfiO+XimbjebAeO+MRQqD/B+FR1tnRN0tpKSJHRbE8sFfS6aqsXZ67TQjfwfsxULVbg==} @@ -1561,26 +1519,41 @@ packages: resolution: {integrity: sha512-+1VkjdD0QBLPodGrJUeqarH8VAIvQODIbwh9XpP5Syisf7YoQgsJKPNFoqqLQlu+VQ/tVSshMR6loPMn8U+dPg==} engines: {node: '>=14'} - '@platforma-open/milaboratories.runenv-python-3.12.10-atls@1.2.4': - resolution: {integrity: sha512-yeklY7ISewNOQ5TWlNLRi6VSOGfcdTJPoDWtf5Z8na3xPP56fWZFuckX/ixZpGvcoyBSQf1kFBFnHPginX7jCw==} + '@platforma-open/milaboratories.runenv-python-3.12.10-atls@1.2.7': + resolution: {integrity: sha512-vJiU68NKgyupW5meeWB/pZZ8y5wewPV6/EcRQ2BwUhSyikDQrn83XzBlc/nTu9UmbYmBqQTrt59x0qKtcC0ILw==} + + '@platforma-open/milaboratories.runenv-python-3.12.10-clustering@0.1.1': + resolution: {integrity: sha512-hnr1tf0dwoqTyjtUVZcwQm13O0V8kz4jCUz+yvOgdvWMrbg4o/h1KbghtM8HivI+tRoowc629QJWdy0bc7inQA==} - '@platforma-open/milaboratories.runenv-python-3.12.10-h5ad@1.1.4': - resolution: {integrity: sha512-Yo47x/rm/FF8x/7q2o7TXdAS7pEWqBd/+aHnQzgNAF0DYljzln99mq1L4lMrQx08SJBGrh84ZtgQl/weTzWSIw==} + '@platforma-open/milaboratories.runenv-python-3.12.10-h5ad@1.1.5': + resolution: {integrity: sha512-cuGk6NMBocvNePoMTnDW4z8eIF5RQdDcgwuPHg9atW9/veOOp02p5jk0EwxmnbMSVV0yDwRMd+0crQAQqW9bCA==} + + '@platforma-open/milaboratories.runenv-python-3.12.10-humanness@0.2.0': + resolution: {integrity: sha512-YjKFPXe+caABLre5oZTwAxFHvVCEb5K8rVTEIUBhFYMtfLipVIix0LXi6LxWoOYdhoEeKgIc4zlWFDKs1xaEoQ==} '@platforma-open/milaboratories.runenv-python-3.12.10-parapred@1.1.0': resolution: {integrity: sha512-nEM4eEj7pFT6yi+P5028qtgK5v9A9H0XdVHDtrKX7xW5Jipc1RIliOEU3gzaH0E7UAzPFQGSqShlwakRcGXwdQ==} - '@platforma-open/milaboratories.runenv-python-3.12.10-rapids@1.4.4': - resolution: {integrity: sha512-Ebg+bTaBwsG+WGUSKNgibhg+Cq3TqyrqPkWOkNMa9a2cMDCerYMpKQkRQ/uMWY/uO0jpuW75ck3Lpxv9mCiuXg==} + '@platforma-open/milaboratories.runenv-python-3.12.10-pgen@0.2.0': + resolution: {integrity: sha512-+4pEVwRZaBAm0716NoNdK9cfF4E7jtsRUpcAWYCuPbbZJ3Q7bgfgq8bEhcck8No97l9jWN2rAeJegA9UQy5FgQ==} + + '@platforma-open/milaboratories.runenv-python-3.12.10-rapids@1.7.2': + resolution: {integrity: sha512-rCG+R9WjdopEtEgJcSWNRIWhpo1ABGnA7rPn0BF6ufvLOtKXOGtaGENI7BJlkEaLb2Cbh2XUUgS5l2r5WGhIRw==} - '@platforma-open/milaboratories.runenv-python-3.12.10-sccoda@1.3.5': - resolution: {integrity: sha512-oNkt0u8fgZACcL3XmAu4v0tDGnTSc3qRQs07U8OFX+paQk7qH4ckt0psLGN1iHDKZ3Gw3nlgel0D6Lst02WahQ==} + '@platforma-open/milaboratories.runenv-python-3.12.10-sccoda@1.3.6': + resolution: {integrity: sha512-NAacPe2uFxf5zqDAa70bblP4ZicvA52lVyzhV7IzlyeEIClAsGatLbitMnRghc2TepwYhQY4EbvElkKDFaSSGw==} - '@platforma-open/milaboratories.runenv-python-3.12.10@1.3.6': - resolution: {integrity: sha512-Bv4IF0PtbnyKJmlW/ygcGdhJS3cSCsOFZWR9buzr+VteDITKS1xyISW6tkO3glVBgInITGrtaDPKXTeiurhOHA==} + '@platforma-open/milaboratories.runenv-python-3.12.10-scientific-slim@1.1.0': + resolution: {integrity: sha512-CSLEjBYUdHDf66QAfgQ9jo+MoW63Wr0ygdIncJE2siO2jMUzYCqgCNd8bMjaMfZa3YBC9bHca3Cjxbp4VYc54A==} - '@platforma-open/milaboratories.runenv-python-3@1.7.8': - resolution: {integrity: sha512-vEvyMG30Q94Kp/9vSxElaDvvzkTXwLT8KbavRP0UwAeU7vpXdAuFhDEigrXFE/uMPlQJDqOL4Vx2xG5Tsr2DWw==} + '@platforma-open/milaboratories.runenv-python-3.12.10-torch-cuda@0.2.0': + resolution: {integrity: sha512-PRGkgsVY08tQvoucU8fJmeCF+x9XjiHHjNTbxWqddqx2EZpiUvmylkxtU8i8xJW13cesAI+xHv3NsVUXQaRDqA==} + + '@platforma-open/milaboratories.runenv-python-3.12.10@1.3.24': + resolution: {integrity: sha512-5ne8Fmlhu1YKB5RYgq4c/AvYzb+sUY4hzv/bKIj0E3aDgJdMYO6M6lMicDzQTMagrGRu9FIhDewIgo++TX3+Yw==} + + '@platforma-open/milaboratories.runenv-python-3@1.11.6': + resolution: {integrity: sha512-H3IR86pq0T4UOR66BZEYIkMmlhuyAiOnAP3O8y5M6A2C2UH/oCLGVYqrFF7XHWgjmacvZ7Qcz4KDtIxgJpFCMw==} '@platforma-open/milaboratories.samples-and-data.model@1.11.2': resolution: {integrity: sha512-y9r9LcvBGFmg1LPo+6MP8AWVTigYqJVzKwj1K/z1eJBGDASdroIFTtGB+3CEltOEOt6oDGAzbJC2oRQKMJGTYQ==} @@ -1609,8 +1582,8 @@ packages: '@platforma-open/milaboratories.software-ptabler.schema@1.13.2': resolution: {integrity: sha512-xQ5eD6WNLL490bQY2kEH/Dz+0SE5uyLexwOQzoUpKhBqdYb4eIKGA2NnZ9SDOgfDoiNFzTeYGQhuI+R3W6P0Og==} - '@platforma-open/milaboratories.software-ptabler.schema@1.15.20': - resolution: {integrity: sha512-+7PcB53IdEGZ9wS88UtPIH3tHk4dIZbk4RdUSywXCdnjzTC2qipK92zPMsC/vZE4cUo8XnETL6Ia+nSXrxwy9A==} + '@platforma-open/milaboratories.software-ptabler.schema@1.15.21': + resolution: {integrity: sha512-3EWO64poe5VcBIgVIdIFHwN7nPi000kPuaRmlVt3xnnQZqXUYDFNZfZ2Mc9/mE1Eq3FLwQxcy/5mng7LvgWnXg==} '@platforma-open/milaboratories.software-ptabler@1.14.0': resolution: {integrity: sha512-bkQvykUBygav4y5/GCZbAbwoU4z29AJhVufhAvEYSEMQtozw++CPXcci2OQJaz6+N5OClmznHEwDOMcU4KfRFQ==} @@ -1681,8 +1654,12 @@ packages: '@platforma-open/milaboratories.software-small-binaries@2.1.1': resolution: {integrity: sha512-KN1PR7YgUUfx1dxh/TtcoWpSZbOXbRxXzQuJ505qHuXuE0spYwC7bXnvJsEYbEwRcPMa0foTrjBnPNORE4yr+Q==} - '@platforma-sdk/block-tools@2.12.8': - resolution: {integrity: sha512-UI/gnQuRO+mt7Vy6ootQN1KP0H2GdxGQxtcppGQI6CEAEfTWYmCk8f1C7QoWTgS6tjQkkKmLE8iDzAdbZqzchg==} + '@platforma-sdk/block-tools@2.13.0': + resolution: {integrity: sha512-xKiTmNLjfxCnnHRr9Q4ct3HdjLfkbLztvxpU++/pek0R8Bj/Lezeoji+eLpxAAYYUxjMr8jLj935eLK5MqKrSg==} + hasBin: true + + '@platforma-sdk/block-tools@2.14.0': + resolution: {integrity: sha512-sAsSDAHVAqlDn/QO+KuBnaVy/VomiHM4cI4ghylcY0Fq2CroVyz8Cm5YNw0SPINuUYmpzvVQuG9ili9Fn7uOog==} hasBin: true '@platforma-sdk/blocks-deps-updater@2.2.0': @@ -1692,22 +1669,22 @@ packages: '@platforma-sdk/model@1.51.2': resolution: {integrity: sha512-AFCQus1HYOW/TwrYYSKetgqW39S2XUCGI856H0dxjArJiTHzroUpGtUv+8CkklBCTH5KhWjd7+8BMMRx/MOZpQ==} - '@platforma-sdk/model@1.80.8': - resolution: {integrity: sha512-/Vp9U1EvEa3g0geh29g+KKkJ0n5WKB48fB1tJ3XJtTOTUTpK7thu9GEgnbOARLJ95SXxTiZzUKIY9xYa7YtpTQ==} + '@platforma-sdk/model@1.81.1': + resolution: {integrity: sha512-pyCcCzMu+L0ELnNQkpmYMNw+A70WWCXng29WxOVllMTvFvr6/BaH4OUOA2q8ocseUlKm6qFQiRxmZTe53AcvlQ==} - '@platforma-sdk/package-builder-lib@1.2.1': - resolution: {integrity: sha512-H6weitj7JxbiJSlteEFLafTJ+tfty6iv/imf3ysy8oCS8AZIRJk2VMW3M/aAc+xVkQeX7oVIwMwFMYrJIoFsAg==} + '@platforma-sdk/package-builder-lib@1.3.0': + resolution: {integrity: sha512-CdBjmNo6E1fBxKYWaXa49L/L2WLURxs2f1TAqxLIZlHRE4DZ6E1TEj3jNNKESWp+/9rwtLkTAzmTzNPrDgz+2Q==} - '@platforma-sdk/tengo-builder@4.0.19': - resolution: {integrity: sha512-N4Koocvmfe2D4yOnNkCTiRiz++AyinA9kZrSQ1cmVD2uofwaJXRBEmVsc029VciWSCrn8FmFt3KjZ1JPE4IieA==} + '@platforma-sdk/tengo-builder@4.0.22': + resolution: {integrity: sha512-8+zDYDFFI2tQS7dplTcxgQdUwzt8+IO++sJiVbwBgwSg1Giyc2qMThEYPLK3QNFArgCsIDR6wQuS8t66pJsRPQ==} engines: {node: '>=22'} hasBin: true - '@platforma-sdk/test@1.80.9': - resolution: {integrity: sha512-MN31C6VnvIP/yXxeZhf1iyDAZK3V6NjxkbR3ToJnUJrBvCvXp33KY8IV8f6H7UOP72+cAfrwXM4vW2+rDyxXzg==} + '@platforma-sdk/test@1.81.3': + resolution: {integrity: sha512-LSDJoQqpxX2G8W7ZshnYWroqNQiFoxXQlka4pEJTnPEO8YISW3lWknBhqsxMp2GyepQNoyl2WO1bfiQz+ssT5A==} - '@platforma-sdk/ui-vue@1.80.9': - resolution: {integrity: sha512-dvLA4Fil37gU2XK5n0MTqzmAYoKR6TWG+t2RBFqnTJzyWl9ZZGS0wWSp3hbpkcUIjj2edM35OdFVhqmKWoE/tg==} + '@platforma-sdk/ui-vue@1.81.1': + resolution: {integrity: sha512-xFnehLOXDHYwDFrdtiU/NBjV0l6pEVLkthfnBjR+g2pSrHJGEL9JDrY3DdzoNUSxU4fal+Ak5aoAJR5v8mOeLg==} '@platforma-sdk/workflow-tengo@5.8.0': resolution: {integrity: sha512-OJnnjBXt1VcS1QNMC90PlkXrWJoKQ/Nl3/NeTqzTmq1j5gT3cZXoIQYu1c31KZXBungUo8TphymBcs+qDGpVgA==} @@ -1740,29 +1717,26 @@ packages: '@protobufjs/base64@1.1.2': resolution: {integrity: sha512-AZkcAA5vnN/v4PDqKyMR5lx7hZttPDgClv83E//FMNhR2TMcLUhfRUBHCmSl0oi9zMgDDqRUJkSxO3wm85+XLg==} - '@protobufjs/codegen@2.0.4': - resolution: {integrity: sha512-YyFaikqM5sH0ziFZCN3xDC7zeGaB/d0IUb9CATugHWbd1FRFwWwt4ld4OYMPWu5a3Xe01mGAULCdqhMlPl29Jg==} + '@protobufjs/codegen@2.0.5': + resolution: {integrity: sha512-zgXFLzW3Ap33e6d0Wlj4MGIm6Ce8O89n/apUaGNB/jx+hw+ruWEp7EwGUshdLKVRCxZW12fp9r40E1mQrf/34g==} - '@protobufjs/eventemitter@1.1.0': - resolution: {integrity: sha512-j9ednRT81vYJ9OfVuXG6ERSTdEL1xVsNgqpkxMsbIabzSo3goCjDIveeGv5d03om39ML71RdmrGNjG5SReBP/Q==} + '@protobufjs/eventemitter@1.1.1': + resolution: {integrity: sha512-vW1GmwMZNnL+gMRaovlh9yZX74kc+TTU3FObkkurpMaRtBfLP3ldjS9KQWlwZgraRE0+dheEEoAxdzcJQ8eXZg==} - '@protobufjs/fetch@1.1.0': - resolution: {integrity: sha512-lljVXpqXebpsijW71PZaCYeIcE5on1w5DlQy5WH6GLbFryLUrBD4932W/E2BSpfRJWseIL4v/KPgBFxDOIdKpQ==} + '@protobufjs/fetch@1.1.1': + resolution: {integrity: sha512-GpptLrs57adMSuHi3VNj0mAF8dwh36LMaYF6XyJ6JMWlVsc+t42tm1HSEDmOs3A8fC9yyeisgLhsTVQokOZ0zw==} '@protobufjs/float@1.0.2': resolution: {integrity: sha512-Ddb+kVXlXst9d+R9PfTIxh1EdNkgoRe5tOX6t01f1lYWOvJnSPDBlG241QLzcyPdoNTsblLUdujGSE4RzrTZGQ==} - '@protobufjs/inquire@1.1.0': - resolution: {integrity: sha512-kdSefcPdruJiFMVSbn801t4vFK7KB/5gd2fYvrxhuJYg8ILrmn9SKSX2tZdV6V+ksulWqS7aXjBcRXl3wHoD9Q==} - '@protobufjs/path@1.1.2': resolution: {integrity: sha512-6JOcJ5Tm08dOHAbdR3GrvP+yUUfkjG5ePsHYczMFLq3ZmMkAD98cDgcT2iA1lJ9NVwFd4tH/iSSoe44YWkltEA==} '@protobufjs/pool@1.1.0': resolution: {integrity: sha512-0kELaGSIDBKvcgS4zkjz1PeddatrjYcmMWOlAuAPwAeccUrPHdUqo/J6LiymHHEiJT5NrF1UVwxY14f+fy4WQw==} - '@protobufjs/utf8@1.1.0': - resolution: {integrity: sha512-Vvn3zZrhQZkkBE8LSuW3em98c0FwgO4nxzv6OdSxPKJIEKY2bGbHn+mhGIPerzI4twdxaP8/0+06HBpwf345Lw==} + '@protobufjs/utf8@1.1.2': + resolution: {integrity: sha512-b1UQwcEZ4yCnMCD8DAL1VlbvBJE9/IX4FTIp7BG1xYpf29SLazLSrqUkj4w7Y5y7cCVP6E5tcqqcI0xemPkHug==} '@rolldown/binding-android-arm64@1.0.0-rc.15': resolution: {integrity: sha512-YYe6aWruPZDtHNpwu7+qAHEMbQ/yRl6atqb/AhznLTnD3UY99Q1jE7ihLSahNWkF4EqRPVC4SiR4O0UkLK02tA==} @@ -2315,6 +2289,1676 @@ packages: '@standard-schema/spec@1.1.0': resolution: {integrity: sha512-l2aFy5jALhniG5HgqrD6jXLi/rUWrKvqN/qJx6yoJsgKhblVd+iqqU4RCXavm/jPityDo5TCvKMnpjKnOriy0w==} + '@stdlib/array-base-accessor-getter@0.2.3': + resolution: {integrity: sha512-/2tQbhLpgBMfAafB9I8MWFkn3dYUUGcUUiZcbPHaAloV22G6W2HU+gUk0l4i++w/34Y9HzxjF1pevMi9wPizMQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/array-base-accessor-setter@0.2.3': + resolution: {integrity: sha512-NMvp8JNmvI9mCWmenPX749j28IBvUflsQ9WVhCRwmzrBDgf1FMUvvFFyWpcjnZI+8luwW97o9e3EzWBUIxeLEA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/array-base-arraylike2object@0.2.2': + resolution: {integrity: sha512-k5dDtnpL33vgSuHkIJjsMcNWT75SCD3daeU+S9Ucwb8GW9UJh8iWRmm1/F4OLD0dRlQx1Jrkf7Z3fUxIb8aaqA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/array-base-assert-contains@0.2.3': + resolution: {integrity: sha512-0SFOYk9DozEa/8zzKIds78ubLHP1bnL6RuqXi5Af167N8BmSUFzkl3dI/CYhGOjxOyBUIZxe38AsngkVCNYtCQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/array-base-assert-is-accessor-array@0.2.3': + resolution: {integrity: sha512-crAa2IZOPCqQMLRfIVkFMX6tJFnzTKfhCNzRXOGBmx7tOfoZoI/KHHVct1yZLv6uBD7v6hvegKdJcGk1BgMTEw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/array-base-assert-is-booleanarray@0.0.3': + resolution: {integrity: sha512-10sn1QJMZUahwCy2QTujv6Mwx+fikqVhO7m1uywozBRALltEIq04r0CKhXPWxJS7HjQ+CYg9Lx2f0l19avR91w==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/array-base-assert-is-complex-typed-array@0.1.3': + resolution: {integrity: sha512-Rx9edezX8Suxh2+zQ3JtuJSiXKWTvS9drj8zWUhbdOBodtMSiqWH4UH9YM8N7BcZiYXWL99cnnIKDPXxz5wPfQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/array-base-assert-is-complex128array@0.2.3': + resolution: {integrity: sha512-C2B/mcZ9S1I2KSsWkAwIIoW0ekDXcMao0Eoa8moS6xlpX5sgDVhmB/DkL0sb+z0E3GRp6/axiqyNTmEl1dhKeA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/array-base-assert-is-complex64array@0.2.3': + resolution: {integrity: sha512-CtI7sc1pcXB2yqQxHbz6mglOHK243HV+/hO2jAqDBPpaRiY+/z6+NHuhd1lrynJBSJdFzDyTatQaLTsfhLPDfw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/array-base-filled@0.2.3': + resolution: {integrity: sha512-+npZqPn297Gv+D6B/OUkDT/3JGFdrr31+d/SGTeCzm7JQ22tGCUDmixDXwqVp9fWwB+LbjrOWrXMKluUTLcNWg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/array-base-getter@0.2.3': + resolution: {integrity: sha512-guM8IURCdDGEo71vvk4S0ZZUWVP3GvzdtYhWLJnDQPW4axLdfWTIUf13qcrUV5TRSMpfMOftSuQZf7dBGAOE2Q==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/array-base-incrspace@0.2.3': + resolution: {integrity: sha512-uzaxYUiYd8bk/wJdSBuE/Mx/jc1vr/7CtnVu9522QRq/cE3ZUPqLW12pBYo+oEoBOfHNA4CGBtdK2yNMI3zrbQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/array-base-index-of-same-value@0.1.1': + resolution: {integrity: sha512-Eyn/DU1tmb/LOTdTahIB1fEUGX50UzTNW2dzmwTVBbjHk1mN25vvjzkn0H1Eba5HF+bHdVrKXS6X+/Jt/28JcA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/array-base-resolve-getter@0.2.3': + resolution: {integrity: sha512-nkaDnhbUYe53FZYxITFaxXLgn0WELnhjvspi2XvExuSa2h+chqVKlOKj4hvzffYaYtiONlpBzURpa8Mu19AQPQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/array-base-setter@0.2.3': + resolution: {integrity: sha512-jq27dgj4h9YyT4ZUubG7jP0KrNQP1MY2aLyieHGXv9Yx+1ysf8FLBRkekbOl1MJXXE7o0bOGAsiywS+olCgBGA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/array-base-zeros@0.2.3': + resolution: {integrity: sha512-C+eo5MFy37TR7BJ5EDK/5O83/x1n4pozCRj3fcL3ZrURTwUHq0nwRCVCmym/DrmZuxb7MM4xHyYto/suoQOm3A==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/array-bool@0.1.2': + resolution: {integrity: sha512-u+e5+u/zCSNM6914c+3dVjfR3HWJzZvME84FyuETS1WkSIueHYYjbL/Y3Mwrrj8UufOdQCJEFDsNHUPBRWSqmQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/array-complex128@0.3.2': + resolution: {integrity: sha512-DPmQ8+MIxJBvusFNXDJMxqHCvMezhq0BGHcZoPeQYn5EDr6PN27VhyxgSWALzmHl72432e6ukeg+KQPKJQ1fcg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/array-complex64@0.3.2': + resolution: {integrity: sha512-gYh7d2yCq4EELHP/8PKNWz2NO+x6ln5VPXbd4logAVZg4EhYW3rl9jkU6W0KGPncwEDIw+QuHcIAogYmPFnFDQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/array-dtype@0.3.1': + resolution: {integrity: sha512-D/tBahfw2W0gVKPX89YbUA52SYOeGS2kqdOMgf2lJD/U1vuJehLq145dajQEGhjc60wdigM2VMjfCrME2MGfSQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/array-float32@0.2.3': + resolution: {integrity: sha512-LxdKGrpsCehFDgU+nw7r3/NL+g8pe9zcDn/6fvNwiuy0AiunX/+c8lin9qUy/FEhrbU6aFXepFM2Ql/9YQD+TA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/array-float64@0.2.3': + resolution: {integrity: sha512-LtQOcxfE2zX5QnYfLNfptSV9q3JUGkOvGhvtO94iPXEvYTvJSEWLK7G97AJR9MK3rdL1+USjGQzUPiUd1/kAbw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/array-int16@0.2.3': + resolution: {integrity: sha512-yNoSOvDGUwHlO5g3wyEM5KYIRaHB5N/PNQHhOTbuvjdsvBhPgmyJr/TQABnSipKsS6hOEveI1SxyONfZGD2QkQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/array-int32@0.2.3': + resolution: {integrity: sha512-m+uRYfsP2XDMCgCN0EMsxUNha4SQ29htPqFC9KZPgnVvvmjr5gsTHiZ95Io/ErgLxFCjRajpi9stjZ7f9Xeiug==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/array-int8@0.2.3': + resolution: {integrity: sha512-RZFDdqpFwNrIUob38519AL6tNWGQb/Nqo5TnocCgLPRVFa722DF2MI/DMjhQBbp9RhI2gHAz/gtqfa3CIhsM7Q==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/array-uint16@0.2.3': + resolution: {integrity: sha512-ceDGvNGOxR2kUO30USUWL8ezKw+R6wBwQNJmPOfT1HJRmCxio+eRMBygmxoRmWwoj2pj8IHC/GlC3N2kSNN0Iw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/array-uint32@0.2.3': + resolution: {integrity: sha512-zZGjkJjPsgp3WyKOCICOGaJ3OoVyzwgyZgyEHl3wlEqfPhUPpVJGLhy9mFx8WIoTGHsCWHs1COWZfh88bL/pSA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/array-uint8@0.2.3': + resolution: {integrity: sha512-+UVsdR94Qe1zXEbpE+rpXFxp/VB8+zGxH3d7RH2cUpy5eFeqcSQNvxaxfqZnbmqk2Gu5tJEv/ZBcTXkpeKH7HA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/array-uint8c@0.2.3': + resolution: {integrity: sha512-bTK5NJeXPo3bpiSu7qebFjGogu9CJKCNmmooAjPcq1Kj/S5C+vVamJjHx/C32euvlynwRiF0fJjcerSVvggsZQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-contains@0.3.1': + resolution: {integrity: sha512-CDAVa9BxxFN3XfcgNDlbZj/sY3AyWmX0oz/5xFj/V9SRgHeOn/NeDu+CAW/Ph6GW1xKNVMeva6tSFjNWdsG6Xg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-has-float32array-support@0.2.3': + resolution: {integrity: sha512-nNRO8I4dp3CtxFXCD901IBmVc+/otaFMr/Mx1jN956nkaXaL3GZR6p12IGgN535Nj3QILoKPmOG7gOjts+uQOA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-has-float64array-support@0.2.3': + resolution: {integrity: sha512-BbHjShic68woFydK2s8ECn4uKM+Cr3lB6kBP+9wK7wkUGVzq4lo7n+Twt0cBbgwJOGLlQayJQOQAVVq/nu/wDg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-has-generator-support@0.2.3': + resolution: {integrity: sha512-wr4w9/ufSBHSj+x3WNGK2WdEA9u+itVeY2Tx6ExN7w5avBAFe5EF8/ndeta4iQd8Su6YKJCqCUXNVJ3Dq4gl5g==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-has-int16array-support@0.2.3': + resolution: {integrity: sha512-b/2LkEPpg4jmrZZYVF0kmjsThEolINpj7TkKLQORxWeFKlULHX+rEVBwnUxGqW7Z3P6xwklbSlzzhlCU0xXc4w==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-has-int32array-support@0.2.3': + resolution: {integrity: sha512-FjGlpOYwUV/DbWpX14ZirgxMRgkwe6KKsN+/DEllu/w0bLIR4uBVBOQk8XDnExSVd72CiOH9+HOx6QTjHlhnMg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-has-int8array-support@0.2.3': + resolution: {integrity: sha512-DYq1D2vpVO9NnRQa+BQ6JeZy4SX4yps2Tusalnri7M9s9BNzgzH2rqB1gdcUmSLQ8vLLaHNVsHPGoiPqh9tJBw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-has-iterator-symbol-support@0.2.3': + resolution: {integrity: sha512-m06tXK3ErFF3O3pFrgfbCmtH7UpGDspqAoE260Gw+75JY7LkX6cl/uGoGOmxq0WZPyom/R4UJCdDkB7J15DKYQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-has-own-property@0.2.3': + resolution: {integrity: sha512-BnX1Lpvd9YaucQLokQrf7ppLwa3V+nTUQ9qoys5SloYhjwbYWtO61SJVT2JK+cfsCxUAx/HAh3dN1qTKxx1PIg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-has-symbol-support@0.2.3': + resolution: {integrity: sha512-yNsJnCb7HWye5xhZ/eRIpp28HwOVFF5gUCKeT1p4haoDJOn+3YEPYaXMn4mYK6kRN96tUhovlJQv9H/9jwnLpA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-has-to-primitive-symbol-support@0.1.1': + resolution: {integrity: sha512-nobeU8aB76XlDtKCC4DZWRdWSbPCoOGOOwEuVoyJ058gNacR4lRPoCA/7cxAVl3UqqLwhX3E1MSOMxNTnMyTGA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-has-tostringtag-support@0.2.3': + resolution: {integrity: sha512-9sULfRKYneF/Mq6F96xqeQrf0a9wdJc1lNZf2wfs1EtRwQHQfx5+QlYl7hp3gWI/iid5M4Npme86T0413eaBfA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-has-uint16array-support@0.2.3': + resolution: {integrity: sha512-7PjQTyXmpIczUz7Vx8c7BcdroJmMmQjyNQDF9a46Ya+nZxqH7g9jRsxmvzqVrSyXhkIyAbw1Ao2QyKocz+hZhg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-has-uint32array-support@0.2.3': + resolution: {integrity: sha512-xa/0yWs+fhp/yUZPD3/ZmQCXqv7haJGGJE75HBX9tw9CHgmqpgAihD4yGy982CtQaxEltH8icNOX8CQzZZS31w==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-has-uint8array-support@0.2.3': + resolution: {integrity: sha512-cgXcNtv5PS+LEV0tj8VOZ8WVjLJuvSzG0GolF+2M0u5le2hHk7BfKH9bqmrk0AzpipAEEdQgp4CGc/Mmji5s8g==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-has-uint8clampedarray-support@0.2.3': + resolution: {integrity: sha512-Lp2sWfsU5Saqm4IQ+ARuUFc3o6qOJoYxzhyI9nAHC1yy8bgvwNkjFiGs+fJcfrtFxeNDC8WIs+IbzuiT3HSoJw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-is-accessor-array@0.2.3': + resolution: {integrity: sha512-Ll7rZqXnwT5v5ys01rnUxfJFxtqVprJtb0NMyF3Rw9QXJtPsVcaWMOeL5FDZhBqlw8d6Lq+VkVA6WabgtPQuzA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-is-arguments@0.2.3': + resolution: {integrity: sha512-J3cf5pEZmJkjUi2MV/uldPe+rOpC9Y7pLAtJHY7BkCTKxZCyn16KpX9PuqeMXp+H04GdD9wxuhESGdqoXPa7pg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-is-array-like-object@0.2.3': + resolution: {integrity: sha512-Vu+ekWIc3srgjwWZSUl7FK+kj0HVwWdcAHNDsUon6mcvzltkTm+fYS9q/233L9U8BanhfORMtVcQ6hLX0Cnnkw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-is-array-like@0.2.3': + resolution: {integrity: sha512-HXjWVOiyYwjSOAhq5LDXWcZDuUwHRcOWKY9s0PqeMS7/15smJJCMc6y1yg/aXae0qzHdex4UmCB4zIxXOftJvA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-is-array@0.2.3': + resolution: {integrity: sha512-ayRsLGmssNO+8SR63tPP2+LZCxFVsSXtdN0ToSdm0kAOaGT4e0FoQus+AlFdhW5xWtvoxL8r5WKTVqJ514/rvQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-is-arraybuffer@0.2.3': + resolution: {integrity: sha512-1uNY2fnLP1bsHDqydcuxhj3SdJrMMvKdZvGVCSoT8H0HRMoStED3FS/zXzSLTGHXcExYYSHXWyOrvieLWNIfQg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-is-big-endian@0.2.3': + resolution: {integrity: sha512-+/X7ZcXFdboHIKDtxnx25fKmf9EhPRyp37qp9H9tUb0RnPT8SgmQKtYZxovp7EeBbESZOKSousg0jZKLbPqRAg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-is-boolean@0.2.3': + resolution: {integrity: sha512-36BETlRDrBeCOBLowW1/vqk9wD+OA3Wp9eQxFij/7icKiMDURH+Fiw2hfbhaZ3c3Y9uEN0rAdOT7AJlwWhXc7Q==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-is-buffer@0.2.3': + resolution: {integrity: sha512-ppnL0sC5XHSk221AYjT4zD33TDB/XuSkhbbbr5sviTH5+88KAqy7SjFlvcpqpvB2BvEVzqRE1fgVso8SjGQD+A==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-is-collection@0.2.3': + resolution: {integrity: sha512-mAL86cRZhTeqCHiLDynOFm2S3DRUaB9NWPbUSFELmoWZ9wV0JeCIfNRYRyzWRPthMsyTYACAhLx8UV8IGJjsmQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-is-complex-like@0.2.4': + resolution: {integrity: sha512-BTrhcwlCRb/6NRqlB6Ap5WpniqaawCZOWnJuAJy16lbrvYvFP09f7UNNmxrxFl8BCg9hhyZQTIfdaF9CMgN9wQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-is-enumerable-property@0.2.3': + resolution: {integrity: sha512-UtqsBVUEAnPjfbOduQ8CT/YwUFLiXiapjSnKaGeLX000qBHCW07uOL1nIuE/vN61SMB+yvZuTZl9xUux/Adtew==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-is-float32array@0.2.3': + resolution: {integrity: sha512-fHPem8taZPd1YoYIB1lqju9HEPDUuY/UFKG0/PquCZtA/g32TEyMM7tb/GCcDQuxno9jnH4+6r7CTfHaa9758w==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-is-float64array@0.2.3': + resolution: {integrity: sha512-GEuYmjWDT2PY530fIO4qBDr4xO9vYWTjyszkyma0RWKzyGBdCV6GYdMj8qO+W7bOVsqeuoYNTCNebOYfpgZbSQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-is-function@0.2.3': + resolution: {integrity: sha512-6EdSkdJ1KtILdGb84MhherNtitGgzHH+u6rs1gGj+294wg/9IzLYaaHDktJVRN/viD3XLEQjPGzSTq5x5DzPbQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-is-int16array@0.2.3': + resolution: {integrity: sha512-H5OohxNtQJ/i6esWE5R01TmXl0M5gCdvHP3+02ahiXbq6PBOJ5bJQVvw2M/j0jO93tS6d+5OcVGAKGazSn76RQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-is-int32array@0.2.3': + resolution: {integrity: sha512-ATPkqHYsQnOYs2Vz3KudJd/N79GFsiKzEOiTMzl5ILcN6bdW28ZNjfh2eIJ/o8NR22rfPQOiUCvxJHw2hpAYKw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-is-int8array@0.2.3': + resolution: {integrity: sha512-eK9SvSUVTTQ0yGgWF9BI/hvqs6pu77NoVRwfQLLvjzcpI3hRm5RyzhHyoNe6Q7FQ1obxvVYsCjysRmC+xmV36Q==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-is-integer@0.2.3': + resolution: {integrity: sha512-HfJg0x6YWW6aP3JSONoI5bNXFGqtiNqWj0L0ex+oDpzrVbXQOu9GNlRKP5P5cDgfI+3rpZMCINX52IGKyCiUJw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-is-little-endian@0.2.3': + resolution: {integrity: sha512-TvKdQhKwrWf0rik5iiurlI1hpJru+bbYGByoSJ6SScTvZIgf54MTYEsWgVyoNqMw8AQFxOyDbMRee+je65ZbMA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-is-nan@0.2.3': + resolution: {integrity: sha512-gFTy7KCz0/bqf0pnEBihtpVz8QYZHiYDuv8wvTZsybMz8tDOE5LnNPJ1klCsWgRLgCy9MA7uec3xwfst9n+71A==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-is-nonnegative-integer@0.2.3': + resolution: {integrity: sha512-3Goiu4Kum1Xm9NbvC23GZOj1a6aVrGF1D8CkZkKDNp8kiYgKSKonvcbf3JeVrr/fLxsdfjnprbzrPOYsMOsRYw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-is-number-array@0.2.3': + resolution: {integrity: sha512-jpodfINEvUe6lQWxurZduf2aiaAoBD+KZsXq4Pn8sas6XDjPRmHkqth6dJFz8QcyzDdCmq6CbyPOEs7e/q192g==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-is-number@0.2.3': + resolution: {integrity: sha512-6i2RoG5TYn7mfKnPmvAA1Gdhn3PxvQegb2bh2pi5k/+xXgEHoubpqBSVxZBTZ70GIkPwNFJQDRWqwOwHet6enQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-is-object-like@0.2.3': + resolution: {integrity: sha512-K6G58h5euEVSEZvFZSHADoYL7sixNTRy+ezHl/I3byJpC9PJdvTO0XWYD0CT0yVXIaqmCYU2NsPpgGdcPLELFg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-is-object@0.2.3': + resolution: {integrity: sha512-H9NaVGuPl4qA3J4gDAzy+sHCkTImVycoNd7izPF63JuAlEO6KTdpVK7stQBXgcWf1pyRHem7ZTJCzht2UMLzcA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-is-plain-object@0.2.3': + resolution: {integrity: sha512-MRdn9kuzC4Hf/2Z/911yZUso6s6f048Ck3dO1j9scHk9EKnU404XrC5NJHeZ9OaI/689Fs+JKDWyKyqyHYRV8A==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-is-positive-integer@0.2.3': + resolution: {integrity: sha512-lGd068beyzC9O1cAlyI5vKlCEUyJwwTrSdLkfVqNdqppiX64lOngjoAhu4BmHScuWSiJz9Zz39YnQ/mA6cH/kg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-is-regexp@0.2.3': + resolution: {integrity: sha512-+QlQMHrmSmF++7gK0Jjgy7W5aa91gAfpFWPyY4gUwkrSc+mY+JPPYW4peAf+v6dXABOgfKH/JOaFPSv227SSTQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-is-same-value@0.2.3': + resolution: {integrity: sha512-YC69SQUS5RJ4gjYrgzBTV42VLNm4geENhA5dHfS3JfdwruzG+AGBbhupG+DLRQII9Mpv8qui2ik4EXQCKU99bQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-is-string-array@0.2.3': + resolution: {integrity: sha512-OZhH3sJ4xUP1iGEvpG4IGIr9B9pWWv1CsvK1bSsNunvCEa5fi6XPJvXeP5NbgTIgT4JhmM5fxgaoT74S/CW3Cw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-is-string@0.2.3': + resolution: {integrity: sha512-KC2sHwnIo775uEPRG7miERHb3DMABf37jS6o2bmcMHyME+gl+HC/bnqgwYTfgOsM0YOvYMSgA1HJo6iPDXdFrw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-is-typed-array-like@0.2.3': + resolution: {integrity: sha512-TOb9CCwNGikBm2KD6cpiAJSq7FoD0amdt9zsBMRQUJEgfjKxcRGORx5i73GmkJUoFT0uJa2pnRRlGJp+4bZmoA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-is-uint16array@0.2.3': + resolution: {integrity: sha512-gHnj9FpIpyiIgsDfA7exvfqqn91THgGJD6BAWqiBgQQccK++K5q3ARc31ysTBuIea1FqYfpy30vt2ZW46lIMcw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-is-uint32array@0.2.3': + resolution: {integrity: sha512-Jt9hp3iolCoQC6zC/yisTvxcuxhZ9jyiYwNzyEzgoY4GdV/WETuTpQKwTKbgQkJdUbcl/3VIQaVopEJSUaoV5w==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-is-uint8array@0.2.3': + resolution: {integrity: sha512-a+CzS7ffk2D8oFsswqGJk3L8zpqcnzuB5rwolilRVQtrWjQT/s2ydv5b1HbMB2rEcT6h64rYJDNAITeetGmzBA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-is-uint8clampedarray@0.2.3': + resolution: {integrity: sha512-tGyJ74IqIFczWUZAAaR/xlGSzh6vO8t8SYV33ruNPjah/d69uijXHRVohx/aO3xaQ7bbVapk/WbGosmILI7sIw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-napi-equal-typedarray-types@0.2.3': + resolution: {integrity: sha512-eBfODSc+zsPnsIXuO1cmWZ6AgGFXPpx0YEApBFzG6akllcPssYuOIJHuE9xBdgL2c8NAGKl91tMitlkmEulG8w==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-napi-equal-types@0.2.3': + resolution: {integrity: sha512-JfESREBknrUOorPQp6y0vQicG5oSrzopRs2/Uas4tXRWRWe8J+WI60W/HTNAbfow03a5k30M9b7Vg81N6zf2mw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-napi-is-type@0.2.3': + resolution: {integrity: sha512-EajA0tANjWepz0MVE0zZU45CXLA12/uuRpQ+680ZBfULsVc0plipSQfIMePBgHueJe9YMgyqm3r25RXzf98i7g==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-napi-is-typedarray@0.2.3': + resolution: {integrity: sha512-szTmOUQmxi7wi+sJjOpmqBeKdaTHXJAYaBAlrT4Lm8KPjIGr+RP2Fvi2CfFqQu4faekHEM/AXuKbujZnc/P7Xw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-napi-status-ok@0.2.3': + resolution: {integrity: sha512-rA+0Vo1ULn4uBHQ+FfagPAcikFT3W/t9I7HdP7SMjz71IZpkWyuxVFZi5MTv2+7hGMYA8be5FibLz20uzbZ9pA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-tools-array-function@0.2.3': + resolution: {integrity: sha512-z2a2G6f9Exz/V0UfE00TLm1sp6W+pOdYltekRRgdEgFNPPP66WxaJfFLkXwqZv0X75U6TuQ0pp7tFq6Ac0lf5A==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/assert-tools-array-like-function@0.2.3': + resolution: {integrity: sha512-OjU5pTqnG5vVLQnQNhbtuENciUJHOo8KO3QiUPUcvEY72bkF9ONNthYUl4VmKEEeHZnyrHIeIpjOQk18JNTKeA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/blas-base-gcopy@0.2.3': + resolution: {integrity: sha512-W1vwLtEIfty7iCJZZ3OhcpNw6o0HwGMXdLWLEDlhYpQ+eBKPBpa8FQducrGSbYDI0sys+ZBMq/oRv9dGIyH2IA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/blas-ext-base-gapxsumpw@0.3.1': + resolution: {integrity: sha512-PAe5tuC7LtUdCpa3w2Up7n8axhavCsRjrgB+GUcpyW5CwhjahTqqve4QAkcdZt6Y530dlh31q50I5IQG80RDiw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/blas-ext-base-gsumpw@0.3.1': + resolution: {integrity: sha512-LARELhFnqFuAQiJVsg2MxXTQvR74+SCu56kzsBK3Wxrc8KQKzmzmWHPJ2y9beZNkzPLmpWNPiCyQUULvmUd8lQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/boolean-ctor@0.2.3': + resolution: {integrity: sha512-JmVu1SdJHYK5ubLl8nqi0gfYX5kAcSRRJYNQ7JuG8oU6WtkIMsC0ahQ8+3G673czRaQ7HGE+pL0IZiH/s8HSrQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/complex-float32-ctor@0.1.1': + resolution: {integrity: sha512-P5aJ7kJ3VkOCvtwIipYH2t/vENK8XnwQGDChQiZAgkJadrF2dKYOXa8t1vWEsy/AMVujAjamHlCXQffhafYm8A==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/complex-float32-imag@0.1.2': + resolution: {integrity: sha512-RwwHXMocm0/1wMB3NI7oJR2MNA1avqisqh58oFR95Kq35leSIrevSHw2HbW3P1QG1uyYCbOTp5V2zv1alwXAzg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/complex-float32-real@0.1.2': + resolution: {integrity: sha512-KyUfN8nm9zyfNwi9O37u1oq3hsAi8uG1sQX+lI49PGxqkxfIEoMwHclYrs2lgQ02EWD6t/Coj+8dbzKwuj21ug==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/complex-float32-reim@0.1.4': + resolution: {integrity: sha512-PXnims1rRAVGARrpoeiwxkYcThjCoXv3iLhUhz8oyulcHteGkJuBAmhnrLANfkh2ro+bFlk6lC9znLt13SOM2g==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/complex-float64-base-assert-is-same-value@0.0.3': + resolution: {integrity: sha512-Arz1sXdgWNUY9Fl5FWpGM/4KPU7xH/1vEPjf2DlnESsHOuNXjPfaox5Lw9BcAHqVs/Va34HO2LA9XLcbXVI9zA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/complex-float64-ctor@0.1.2': + resolution: {integrity: sha512-xCVE2Lv1/I7A/2mDloT3hDCpA9NP9XOqZKpyKhg3D/9D/vJnh1rnn9CK8+xzVfRborsyQbJF7OI+Ijc3szDI1A==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/complex-float64-imag@0.1.2': + resolution: {integrity: sha512-uYmpvw8723pXVMaPIyoDWh53OLBk3F5HQyCFrgJsFCfuM0e718yd778ltMOyl7kem3M87ealSuxBwFX5SLFxcA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/complex-float64-real@0.1.2': + resolution: {integrity: sha512-m8kJIckqStBS7WzmGe5X8A1CCooGmLToXRo1qq4EydA976FFMFuYrsI4XuByt1e9zp9HEGeoywyelaqyeeWtHw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/complex-float64-reim@0.1.4': + resolution: {integrity: sha512-5IxS+GqkVezTGh57Gp7pIyAHS+obDqUpq5y7KpAjrH0s6gp6oN/m+tmPh7K6Yz8yX+7FLVn5b9lZxY1otBBEJQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-array-max-array-length@0.2.3': + resolution: {integrity: sha512-f6X1YlF9RnN5HkeFZFr8V4iSpV5968/4V5j6RFKY7gwr3jxjiIBar4CFja8W0f5eRUsuAlxf3HQ+Y0syYKpLuQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-array-max-typed-array-length@0.2.3': + resolution: {integrity: sha512-Npc2GG28d1+dgdb2akvzDNZZpiGTIAEe1vvJzBLfmVnDkqxYSuTu55N9jcXtTvrPOLqxwWy7K8AMAKboMFEFVA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float16-eps@0.2.3': + resolution: {integrity: sha512-gdDE9eYNuJqIxmpWFfFsTx24uJv3BV/S+MKi3vj/TgWFeGMnsqHP+xIf46Hby2OwRbr6yYzsVwfJMAzF7ItCQQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float16-exponent-bias@0.3.1': + resolution: {integrity: sha512-fUqcun9eGniI8sqsZf06qa064cKcp6cARfX9dXuGnCQWSWEwn8Q/LKr8zEgUVEu6cJCVyCMHgm8VPpkTpnFPAQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float16-exponent-mask@0.1.1': + resolution: {integrity: sha512-5J38bgxW+UlF9AHG+C9Z7ilx25aPSOr3ECyK0LThG9H8Kw2zL+9cNbt72vRnVr0BSCv3NKvgxPza+B/bB7IJpg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float16-max@0.2.3': + resolution: {integrity: sha512-M1Hgv/fUdnuV6uw9yyXXOAf7Bzf7I/8naaYolXGX4h7B6XSuCT6Of/0BRxXlU9D+V6zPniGo2zdBaXUGfek28g==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float16-num-significand-bits@0.0.2': + resolution: {integrity: sha512-qml2Wmm1/gHOGmEfrR1w5G2wYsBHKfCLBaJ8idZ4IMcBfmR8Ez4I9K3uQ3EZ6mW8QU2nUIgp0Fc99lnzLoH+7A==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float16-sign-mask@0.1.1': + resolution: {integrity: sha512-LpLyCmkhzwotXb1mAAkAZ4sTk5lh5lW3Q1pOp6jtepEFT/marQsoeTe+0r0mIwSncGbym/eEB455qUZQsdiTBg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float16-significand-mask@0.1.1': + resolution: {integrity: sha512-+IA0fHkBV7X00ZTqAo0ZZSSuh9fjLGB8+4ot8iR0irC0YxPLkvyPzFHujnWFu8UD1Eu3SZcyaCY20CLcgsGr9Q==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float16-smallest-normal@0.2.3': + resolution: {integrity: sha512-u/EzlthAeRba4TePLQzTyivIAp4GB9jLdOwE9FdxSYmqUwAbEIODkd7vft1ArzuCILRHM4NNNESJeaLlv9Qd7w==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float32-abs-mask@0.2.3': + resolution: {integrity: sha512-owGGn8KvmHmne0u5ItTvOpHL6lUJP+tuUe2wzrKYiZuoUTg+7cOhNW4fPWCK8AFYxs11Ha3qPA7oAzjqlq+Log==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float32-eps@0.2.3': + resolution: {integrity: sha512-yJY7ipPOriceMh2zQvNVDYRrBJlQoBN+7cejox5DxTTn5zytyJn9sfdBQs2buGBqT7BiV+RJE7iE1vQdhuslcQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float32-exponent-bias@0.2.3': + resolution: {integrity: sha512-AdJFdh75o8cBwT/zjRebtZGGz0lYY2P/ITKrxHo4snqinm+DoMFRChkYM2s0QYn8ZgFBHQ1iEPB0cYzxTLk3tg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float32-exponent-mask@0.2.3': + resolution: {integrity: sha512-mkSAIakaixXJy45Ss8G7QbrS6eQemGjfIZhwmwlWDHxCJEVcfpuIijO/N0gtlF/MnGSOACwgn3Qkt1t3Eg30IQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float32-max@0.2.3': + resolution: {integrity: sha512-b+G9qRNbN4/Ke1BKK3fP7+nWahbWJhcCpbieaiAsbdJyucx7oIaHyzcnNDSmPYhVP57iGEqL9RzhVRd0u4+u4w==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float32-ninf@0.2.3': + resolution: {integrity: sha512-X0t2gw6UKfzDRRjYH8k8dqRv2C6iI2sn3UrunRIzCL4WYYR+QWn1UwjZWG4fviIE+QuEGTyoKKb4wG6gTiS8yw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float32-num-significand-bits@0.1.1': + resolution: {integrity: sha512-pb5IMIdVb22DP6yJc1PmsqTc3mmn02ze4P1oE5aZY/0v4bPUf9yegUmYIyCEWPfM1HX/xnW4GwXDZ3QptSgLqg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float32-pinf@0.2.3': + resolution: {integrity: sha512-ofzg0/NDAjALtXw0GPGzsfwpTEziENlSNln2PqoPphno6u7yfCD7BWblIinvDALFwbpsUVWuxAvarL8XnXl4mA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float32-sign-mask@0.2.3': + resolution: {integrity: sha512-CdGdK5nHesLEkS6TeQ/jdl3iThAZYrNCysRVshkpz15ORgvbFyHiXaYFIGkwt0a5RnA4i11e3nioHBNWxITGxQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float32-significand-mask@0.2.4': + resolution: {integrity: sha512-6oBi94bfEEqEPHyzgS3YNBK0fjCRsh/8A6xyaPuHBP+ltGmGfNsZmsZ5vQ+vChszKVyJG1/Bd/j4wif7GGJ/3A==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float32-smallest-normal@0.2.3': + resolution: {integrity: sha512-NHUPJr9+Bwcr0lMZ2goX7T5jWpel9AQcv4fZfnlabQ3HSYDkAhLBmoCUJh2jxjItR4SbWXzpIEKL3RNDeiKEeQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float64-e@0.2.3': + resolution: {integrity: sha512-8nQCMqyeLPdNN2Kzc+tSeoksZbvyfRpqUgB2xCujfWIxdsPKvAXclWpMQ0EZuk19MBpNPSOEbiXbUiAIrk1Ksg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float64-eps@0.2.3': + resolution: {integrity: sha512-TB4+YU9vc9RwMmSrCt9+UuFeXp+a29Q1KrzcBTcxU8uB/2totg6yYAFwhs8+48Vr5T/bQOsTQpuxXA5KEaThrg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float64-eulergamma@0.2.3': + resolution: {integrity: sha512-8scw10W2H/rgE2gSmW3Ubp310dsxW0UGppUoDcm5aJNOu/KgCUMqStoPLU653f5AbPZ//hSBpRh2+Nv31WCieg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float64-exponent-bias@0.2.3': + resolution: {integrity: sha512-GjhdDNlHP96WooqvgEvsCPcK64ZsfpU6DlAsjRMJAPuh/QJ+Ut4WC7vMxuLp/Qio/N3AEsRvCOWIqa5jTvEWUw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float64-fourth-pi@0.2.3': + resolution: {integrity: sha512-Ib/W5K0Pf/sDLjnGclxigJkGQJ/1Jtb+id4kuyRrf1qrlQwiP7tZq60prd+czL14r+1r2y3MYE/7ow4Fc1chUA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float64-gamma-lanczos-g@0.2.3': + resolution: {integrity: sha512-bAC0FhhxMDeU6UsCGsAzVZXjnJOc65StXINkvIzhKTw8SjrlQGifNNcanOVtrwKMnpq13MLsVcG+m0R7BmiUZA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float64-half-ln-two@0.2.3': + resolution: {integrity: sha512-5FcB6dmhWYNSvCRIukJ3fdd5kNH2eTbpgKo1pYqksp3dwMD6GKR3i/832maewutCvcT72Baj95Si1bqX037AXQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float64-half-pi@0.2.3': + resolution: {integrity: sha512-JBDU7c03I5H0kvmOM0ZyxwVxK6kZTLogYSXAS8ho3jkHU8P+DFHXxn6X9DhGKKlTaCTHCAEYQ7zSmV8H886TMA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float64-high-word-abs-mask@0.2.3': + resolution: {integrity: sha512-AXa286EisR1y71U71M18QvzldmIWsh1TNimoBjrUdCQn5jELOJAQPriTKt0H8kngfOUYfy/+EJwCpuQEL/VWrg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float64-high-word-exponent-mask@0.2.3': + resolution: {integrity: sha512-/q5YMZwugOvQGCXp2kusJY04/KHDzAKMv/QEFBB98fDhN6Ux2VWbVyBESJzn55zVGQvb9XhULwkBy/KdB6YuqQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float64-high-word-sign-mask@0.2.1': + resolution: {integrity: sha512-Fep/Ccgvz5i9d5k96zJsDjgXGno8HJfmH7wihLmziFmA2z9t7NSacH4/BH4rPJ5yXFHLkacNLDxaF1gO1XpcLA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float64-high-word-significand-mask@0.2.3': + resolution: {integrity: sha512-SGQfHBnS71PFGQt7Tv6ptAHStJy0sTXmUEjOOXoJlnlClizsgdHXwkeho4xIvycxyNe7L4nT34cFPDMRFguT7w==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float64-ln-sqrt-two-pi@0.2.3': + resolution: {integrity: sha512-+VE4Z3OoPmyVzYJ7mbQUkKdkT7+uHeSG821U0hZLtUQAD2AmPWQNUzKk+G6IF8FUQv03Ly6U4lLL9aoXxXjz0A==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float64-ln-two@0.2.3': + resolution: {integrity: sha512-pCNVc7Cmr56sg+jOrN2yLSWNHdN0UECAZIJ79sDaLBpF2FqeL6EI52EJesLmD64oGSNUC4yZNdGBe+DpKc1ZXw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float64-max-base10-exponent@0.2.3': + resolution: {integrity: sha512-hM5h/ZFk/XM9bpb5NZcpsRcVQLL5PRdXveVise/FS6NbKEkOvvFqZ0FBXeAYkA1mvnGSPE90gKFvF32/3kv83A==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float64-max-base2-exponent-subnormal@0.2.1': + resolution: {integrity: sha512-D1wBNn54Hu2pK6P/yBz0FtPBI3/7HdgK8igYjWDKWUKzC92R/6PHZ9q5NzedcGxoBs8MUk1zNpP0tZyYj9Y4YQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float64-max-base2-exponent@0.2.3': + resolution: {integrity: sha512-DXOhw/QF63NCeB7o7zAgq2g6IeT7jGDT5kBoRP67qbOeHm+NpZdrFdHXo7acH+pwLJXf5iECnVdyQ/axDLQ5mQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float64-max-ln@0.2.3': + resolution: {integrity: sha512-M/qzo1CWhgX84vCuF6Nba/Jf+doaQKxQHNXr3yXKL7BEatz7MWweEMaU2pxfKWa5wADMDvLttskYm5jLWnkiNg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float64-max-nth-factorial@0.1.1': + resolution: {integrity: sha512-krwhCczEOQQDllianEG3LMR7m887BBMZV/udK1/0bYhXJPA2joOEp3RNz3F+IOMJZgPwMnK/x5P66Yp44jeydg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float64-max-safe-integer@0.2.3': + resolution: {integrity: sha512-T1CHYahzAHGLEIS5NMiHjOkGzvgZ0KFtBvFB62shkfMbnkc2gHxVuUzmbxaIm8jQO9X4qafQkRpHFg8AE/e4kg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float64-max@0.2.3': + resolution: {integrity: sha512-UiN+AZ8FuWCq0vXjTYRzQVpMA5Dnx5lBGaYP46/XiT9DY5v4I4ooJiSGcRpnMa/AcFvM+ZLVq/K/2c1VdL8h3g==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float64-min-base10-exponent-subnormal@0.2.1': + resolution: {integrity: sha512-AghpVQcTuhPiLE1IAk+e8HNU+wce/132Sg3MsL60CQaRVnG8X/VdivzEJGt2Nw9uH5rctvVINFrNnM0gxtmzJg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float64-min-base10-exponent@0.2.3': + resolution: {integrity: sha512-IECHt7a8dZloMWBzs8/05OC09K6Fi5J4rkNnayWVZbfHUHpls8bMqibcjgHPmebGWHCFkHcyKYCL73Sb0+Lang==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float64-min-base2-exponent-subnormal@0.2.1': + resolution: {integrity: sha512-fTXfvctXWj/48gK+gbRBrHuEHEKY4QOJoXSGp414Sz6vUxHusHJJ686p8ze3XqM7CY6fmL09ZgdGz/uhJl/7lw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float64-min-base2-exponent@0.2.3': + resolution: {integrity: sha512-N6Xh+ZCnSCMC+ImkgmYAPHFn4V+94pA5KM2ekhNUC3as1pZVJRkJVacogGnqJ9vnoAbZ0QgZpkJpap8Jy7Pldw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float64-min-ln@0.2.3': + resolution: {integrity: sha512-xiA2KS0yVt69fPehhEgl2glBJJrMLH/0q6eornMRkrZuG+o03wLKaqAqTcrkm2nhIMmhNXEVm+gQU0nfiI8Cmw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float64-ninf@0.2.3': + resolution: {integrity: sha512-1yBmgdNkvxWR0IfebDnhS4KOHPNgTpe3j6CCRt3qUW98l45tUkjuqyKKu8jNvjJCq+qKaNHNNk8xAl+/eDuABg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float64-num-high-word-significand-bits@0.1.1': + resolution: {integrity: sha512-qyZWHlGYIS6Hjxnt8TTBWlvkGQlFJT2UVQhaoBrQfp456YF8aghgiwHiaUTXzngMH1llcEq2tyPjCxBdTLs6Hg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float64-pi@0.2.3': + resolution: {integrity: sha512-1s/4TBYzAj+tOgi2yRX6Iecos8SUn968Vymsl8U7+5pDpJzicfBpB9ZIbWNVMQVkBRWFnMx/T3BUzW1+g/dEsA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float64-pinf@0.2.3': + resolution: {integrity: sha512-8kRy0XOvW7QiJlxdy8MXx7rM3S4H/ZsUI+q9dNoarKVDBtWzkqnEG/u5QYouVjGHaSVL8C7b7qwL1FdpSD+24g==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float64-smallest-normal@0.2.3': + resolution: {integrity: sha512-z2IDWJD7s6QwrfzI5o4q3OnZ7mEzUIs1V1PdTuHDofyWkpFGEr7GgwGI/0cZbuhW3aY3xydvM3rZ5SxT+6KQoQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float64-smallest-subnormal@0.2.3': + resolution: {integrity: sha512-NcXOJRA9vuhWcPPvwQf+c86Oab8UtJ3P8Y2DwKt9eRzjRdFFMw73pDmYXEOHSwczn/mGaTHG4HxmVBTugh/ezw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float64-sqrt-eps@0.2.3': + resolution: {integrity: sha512-N6ga0XNLkDgrGCs1cgajXzTYFgl6gDrgWOJW3AQhZPaFEswTtD+yMmcw3p4ye1LZpU3JyDr8hcY2iwL8whX+nA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float64-sqrt-two-pi@0.2.3': + resolution: {integrity: sha512-XVCJ0voDf2U/MMqqnLOo9CB8IEC4odQoercGolrJBud6MXmWp1eVrRYM6y0ocJuyaAPgsYSoPKp0+Gc907InTQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float64-sqrt-two@0.2.3': + resolution: {integrity: sha512-vehmaN+P1IYJ6HhbH70ebhLAZOYnuTm1ezWpFBm/IEC9NGiKKmyrtO+z1xp6iWGl1VrAAhWJE+bBQEP4wVQsCw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-float64-two-pi@0.2.3': + resolution: {integrity: sha512-KoNeAgq+jSDT0lYYHAYosZJwRSH9VlykKjI7BK8TVox7z5QqlXG4Sa94NeGwP8qhp+g4N3Mb7BqXlol3t4vPfg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-int16-max@0.2.3': + resolution: {integrity: sha512-mPHYy0jMD8ALGd5QDOAYU8mO57sbYsivYUVZq7+/mJDXldVprD+fUJFX8QOkuRW1LIsVSqHA1F4oGrnjFM+lVA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-int16-min@0.2.3': + resolution: {integrity: sha512-FZrIJecQGw5qEYSrd4inQgx9xe5igDb+UmZZDKccv39xwWpPPfprt7Sr+zholUdcGGpoJdftBI9QbbLQRTOqcw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-int32-max@0.3.1': + resolution: {integrity: sha512-ZAhWfNaBhVzDatkha1q0vu7tO6XRbP6kxJk/Erh4emgE/4JpC9WPlXfkZvhOoCWQ1Rsll7tevxN3rfu6oGN7OA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-int32-min@0.2.3': + resolution: {integrity: sha512-xXa5t+tlRuwi+cHru4p8gFPnXqa2hI1bn+K6IoTd8Pj+ySKTkR0evSgNIrO6kcEf+kj2YuNXZZM3sqK33SH9vw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-int8-max@0.2.3': + resolution: {integrity: sha512-TD5BzgiZshK/LD/3Ya4fp2mI0j/XGm+XH7TOWLjMWrH3JYbW/GIQfTTDsXTEu52Te0M64Xy5WBmvrXoObFkqSw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-int8-min@0.2.3': + resolution: {integrity: sha512-sD5yCZIujDG0DKLBSj+r3UzxQ04yTyQ4yT/umQrtTHMHAGmZnNZ0Jab8VmxBxfZUC2PAVc841hS/XKQglw5Fsg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-uint16-max@0.2.3': + resolution: {integrity: sha512-QLqt85JMBnL8ozliswgAeR9oWrsmWeaeWjr10P0zbGV1dKhM2HoRK3VXjO4SCtNWUwiV4w8Wa5ylSYdJhORDjA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-uint32-max@0.2.3': + resolution: {integrity: sha512-NYQIGMA3Z8+1gp2qUfmr90whvZJLDq737ipNeujrFm98/RcbBlCRmXCLlj0WMwxniw2T35ysfclAcFSE7caUCA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/constants-uint8-max@0.2.3': + resolution: {integrity: sha512-B4VzHho0T43dJk3m1x9V3GZ1OOK9AKNojo3jOweixBLLlofaEIe0E9f4Gihc2ef7+M4azYptsJdPY1jPy7kK2g==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/error-tools-fmtprodmsg@0.2.3': + resolution: {integrity: sha512-mt4YRp52oCkNWhxkUIJkeGaFnSvZkyfl4rOJX8Drz99xFiIdanXcSacckhrXXu6NQvMDKidCGr6DvrhHHx15LQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/fs-exists@0.2.3': + resolution: {integrity: sha512-kfHok9sKxvCftgWriVtNjqezNZA72Q/rJh78lcrGCLPvSHGIAL460olzl7D1OgAgQvM3Cwdoupay0o3dBCCwVQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/fs-resolve-parent-path@0.2.3': + resolution: {integrity: sha512-bof7pQQewv8LLaSi2Ut77mkEv4v45Z/w/hYocG0UUzmXhu4lXIiCcXz9ZAMJC4tdN5hwBReJGfJ0mQryddhGGA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/function-ctor@0.2.3': + resolution: {integrity: sha512-/Ny143oEz3YAwauaPZ+J9A3IGFdE+0dam5/ATJPGLcl2nDj8P2shAsMp30coyEyzKDoMZqYSKTFFTV5rxyChAA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-assert-is-even@0.2.5': + resolution: {integrity: sha512-EtCGbIAnrhgYpCrCrYNQC6aqOPviTdU2Ui1t9grPG2e1eE8cMjzSpTg1TJmcZeMShzlWl9BcgeCS2GQPaYhkCQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-assert-is-finite@0.2.3': + resolution: {integrity: sha512-PqqxkiubjHe9jdIfAf/PTHZLbYIk1wcFwPGALM+PR3kjRngEkti7NbbkTQX3sByZCCfRwOf7VxLPWDlGo3EmLw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-assert-is-finitef@0.2.3': + resolution: {integrity: sha512-FiP4w+PjwD6zapXgrD0hLEtnUMi+uCcBQw+tU9cAGyB+7NrhnUoKSWgHsjtLGLMD08gWsFOIjZzZRavx4PocfA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-assert-is-infinite@0.2.3': + resolution: {integrity: sha512-tWQoN9z7j56DIop5cBnbdkP/o1MGQ0K8bX4v6Nxsgqngk7YBXYlxGRVw+46FVyxNButJY20VwxpMhrv4fQg35g==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-assert-is-integer@0.2.7': + resolution: {integrity: sha512-rsxiM7N9Y5kd9elTEJpx1kAcHBeHETH46vn2s5FoKyNt7oZLqv9iE5oKF3oNznA3H47o7SNyQSdcJpUDqS1W3w==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-assert-is-nan@0.2.3': + resolution: {integrity: sha512-mj1p+JUSfbsiBI3+hwRV7bS0qgI2nWdzrFtBJUSLk+17MJSdA4aboUsXM12wIV1kg5JNfEAn+5eh4NF/sMvaoQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-assert-is-nanf@0.2.3': + resolution: {integrity: sha512-sWMMFhTrkeYu/t5bv7KZ8Rm7H4pD+O1+wYYNu1CsJ4y1Q6HwSDeHBNIH7Z4zHOOSFiOidb04jnqMGSsL5UCFRg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-assert-is-negative-zero@0.2.3': + resolution: {integrity: sha512-5XzTwloDrG60ZRd5vgbw/lXKn794s2q8Hb0pT2LUy0WG2j98LhAQNEv0f0a7brhHlGOMTX5Hmg1DZXw+ve2u5Q==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-assert-is-odd@0.3.2': + resolution: {integrity: sha512-AVtda9LMQ7WgracZDygZzh7rBDys8KKn4qzAIVv2hWG3ns8X8btQ46plekP44iRyGjYagXMzLJn2jo/Irkf71Q==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-assert-is-positive-integer@0.3.2': + resolution: {integrity: sha512-+FLU8JowmOzk2wiUI/PJzKXXPj2F2BFiNhJihpYv7mm3SCoSyCgRgPG3wzVmPieOY1T3TFBF0ITZy1eqQQCC1Q==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-assert-is-positive-zero@0.2.3': + resolution: {integrity: sha512-V+f8enSK2dEk2V/nHMezyJaUfSC0QJNUD8+mQYON7JtFaxq/+xURnPMllF9h84cFqQ9SAuxWXP+Y7MoE+ENORw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-napi-binary@0.3.3': + resolution: {integrity: sha512-rLBojVTK0GRhy5v0MEKLmamggOrosEMaDcMS5ITuUe6Jgz4DM3lsiB/EHhd0TamPdqo+5HPjqSo10zpFVbLZbA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-napi-ternary@0.3.2': + resolution: {integrity: sha512-DUu7zbd692l86VnUaav4dgIVIkYnxklDMMKpmGT+mi30QH959324tjl3dsthYy5iySnKWg/V0z54opu0FC8jXA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-napi-unary@0.2.7': + resolution: {integrity: sha512-eEu0nAq8WzCGe15ppGJWtLGhKOoG4OcMJQpsfbjwaa3sZ9h1sREmV1iiAFWyi/mHy8C0y6PiG2szx4D/F0ywmg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-special-abs@0.2.3': + resolution: {integrity: sha512-0n2Jb1NQv/siPL5EJFf6/cCuK63zqgqFjQwPujJI8B+dDizlHJKF4jR6yOtavrB5RvVf37/3Ibpdjf8UPf3uzA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-special-absf@0.2.3': + resolution: {integrity: sha512-lUEEnIo6jpR397bn4mphlihPBMncdgKYQQYZ8nwnL/wGNE9InwqLqrO6otM+CO2nkG/kOyt0dSAhSSffqBZU6g==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-special-acos@0.2.4': + resolution: {integrity: sha512-lidI9Q1No+jYM7glXFXMPzxQKwrzqV/QuAp4hVDmidDTL1d6NW0jPBAiaRL7RLcqXqd8oYVirTrQIga8RhMatQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-special-asin@0.2.4': + resolution: {integrity: sha512-HM/x7n25umG0plZrAil2wq21m6+9rA9TDTXErYXAv27HEM5NZGCZn9i+kpTog5/IDSdM58OkTsEB5FOfaBCjuw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-special-beta@0.3.1': + resolution: {integrity: sha512-DabfQZifUcNF529IYtj/9A5oAmZD6g4JhQt1drx8sMMLUJ89r8TZLIyMZU/bgp4cpzSkRGRSPjfXWL/98y5Y7w==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-special-betainc@0.2.3': + resolution: {integrity: sha512-yTGYcCptTWVFxL9hv4Uc221kb7tULU5JkRp5oklPhZmg2uj/eRIvaGYtYvzuTnEU2HpdgWcTVwdDaWsG68F6NQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-special-binomcoef@0.3.1': + resolution: {integrity: sha512-ywcno89Y7ntRdj3NdAVfVzYoIrJb5lILhq+K+NqhgrJBOcrE8ZF+/yPhyVwjUvksfGy990tBnyhx+06VLVFAQg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-special-ceil@0.2.3': + resolution: {integrity: sha512-QJsGAxftN8RLjDkqKqyOcWh8tGFauyXMvqLxoq4C22tm+1ABJ+1rU5wEATOgvOHaDVc7HvHA7fakHTB0UsDUyw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-special-copysign@0.2.3': + resolution: {integrity: sha512-kOPVuWHQht53FcFXO4L1yqXSZy4emdI6qITJAWRENDuW9YZ6QRTONBqOaDC5uU/dRZg3LcMUpQ7iuqYqa/pbeQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-special-cos@0.3.1': + resolution: {integrity: sha512-efthz58G0wf8rXh4+jbYeRuRCagNlYb8AL3es1GxwV9P9/8ZpKAlkQiRPY3AI3ZJMB17aR/KHpVw7Pz+rw4Fhg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-special-erfc@0.2.5': + resolution: {integrity: sha512-qA2iPLKr8SjYpI9TCT7LoQCbUrbx17kmiRidDewWk/aHL0pcpSm8KITCjBUtsmletSEUNUS+8MyAPqisXT3N1g==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-special-erfcinv@0.2.4': + resolution: {integrity: sha512-oF/3UXs2zZFAOKe19rRcWg2SFZFHQbnFJ4ucbSzYykM79pDnvVtSP/dRPtDy3uevUix6BzeloFyK0dLbQsYt+w==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-special-exp@0.2.5': + resolution: {integrity: sha512-1xZWUc+uDQHRD/mHJMsAjenRq7w6ZZqAUc8Jtxwcd5Ccv23CckP0TvDNkC6RqVWWPJploxoxDArQ+BUXgHbpeA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-special-expm1@0.2.4': + resolution: {integrity: sha512-/2HSORRdM0vAdiEEILreYV8cKtnUfW2OrxKLc5dbkCMHe895gURlrpTA507lF3x02uAnHh8PiasMsYTOXcGeCA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-special-factorial@0.3.2': + resolution: {integrity: sha512-YOqPZBtZPP+Pa6jQn/IY0JjGx1sShxQ27XYbpgupg9yZCvabLzhHg+J6/mnjqf4eti6UcVtCITFUZOZ671GyOQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-special-floor@0.2.4': + resolution: {integrity: sha512-YSDt9gHSp8SeVBVzHfpZZ1HFEQDPihHUF92zTDCRGKeKBC4BTFlw9Q4/fNWqS1mtuBGli1zgpRbiTv3atQ5EjQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-special-fmod@0.1.1': + resolution: {integrity: sha512-pW0Jn1S9D8Eh34tEmEJUzZlkfN2nhOmi0dCO8xDWhgdit3+qG/U85t0bULT+/5kRezy8SspnTFkGOoopHztzQQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-special-gamma-delta-ratio@0.3.1': + resolution: {integrity: sha512-Tekp7Flw36nZPZ8fQilkPRiGyMbYyzWjRV/HZWuBaqqDWE3+J2W68qflGFCad0M6qkWDxLV65TJ9E8yJiTYxAQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-special-gamma-lanczos-sum-expg-scaled@0.2.2': + resolution: {integrity: sha512-P8exugMuGijA+9FzG1krmfPIHWqRolFm9xws+H1OGTw9lq6c+sPAny/1KNLo/QbL/UaKrjhbeUDVHOjN3wj2Hw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-special-gamma-lanczos-sum@0.3.2': + resolution: {integrity: sha512-wIH23KUPBKhu0nCraZ3aY882ICDtLIDxeiiBY4F1hNeBBrMeODcCWJc3+jwXvzjigBXPRFHWJqjOyL57FyoGBQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-special-gamma1pm1@0.3.1': + resolution: {integrity: sha512-cZ6m+qn350Hi6nBXoqH/4WLRInDRJ4cZLVgfzuP/49waT9iOGX+kszKAVnMjvp4AYe96ZEBJ1DwUcMYDrX3S1A==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-special-gamma@0.3.1': + resolution: {integrity: sha512-WONtgh00BdeBbgvhjjmn4kviBXb2ybFa4IDOhknxvKS3m8mIVJU59tzLHVa9XiGcomq1H7Mm6CeGPmjmkD+Rpw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-special-gammainc@0.3.1': + resolution: {integrity: sha512-w9R2wnk+5gvZaWiPsnlpZVELYvQ+6ZccgBY2JmRmwBLbvj5t/u8M0jC7MM3CaMCIt5jBmocCD4kCtconzx4WuQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-special-gammaincinv@0.2.3': + resolution: {integrity: sha512-eH3TCXZdH4WvtgKc+jI3ATeaYq87eKcActfSpV0ZZE6Otc326vmNzhv7dXnjVxgSY81bKgKeBzfaTW/fGyXT7w==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-special-gammaln@0.3.1': + resolution: {integrity: sha512-ptzVz18xkfxzgekgAbk6ytMoDc7MYAUY4xZuAW/SyFGO6sCPVbwU4GH1/1IQAGLrDKYizKeVoTsO5NGK5Zpvyw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-special-gcd@0.3.2': + resolution: {integrity: sha512-eQKqRFOlHGw5Ce4mo91uilqywONEp/7L4d/bDEUqL1sRj3bTAIZeCa6ap7wl8vnMXCHLGI0ve7xp6wxVHo4Bdg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-special-kernel-betainc@0.2.3': + resolution: {integrity: sha512-ho7hVn2GAnHsoj+SKJxrgo5rvftmIinE2H2J9/NWXYU6ubjQJ1EmJAzZ6fMSED++lK+eARS1kUl8hUeKRuIbDQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-special-kernel-betaincinv@0.2.3': + resolution: {integrity: sha512-k93/sg4xC/eIsUUw1OD3Rwb8Y2Bl5moTMuOoKeSo5aogqpP6mFwSKOkoW9VS4iAHFUHkjPLgpaN7eRStm1/Deg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-special-kernel-cos@0.2.4': + resolution: {integrity: sha512-DFaoDJCFhPXiOuiGPSU1tmlUQ2UzRWMMB7lKKARjUoxbFpNLwB1TAqAnlx/rguQcCMf7uSZYALpTfxYlJHDPyg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-special-kernel-sin@0.2.4': + resolution: {integrity: sha512-mMXh47MXZd4gB45747Y+6Z5G4oQW0Uaqo5sx+HmJ7JPol/Sp3B6JSqahT+OtAf7oZVBAJ7qWmNlmib6IiueosQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-special-ldexp@0.2.5': + resolution: {integrity: sha512-ZbuCayeY8zsmptOxDsTnDmRGmEt10wjOXOdsKTCNGZwLP7h7e0ImTRbJ8wpKiLWv8JKef/xE+rp61wQrjZHS3A==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-special-ln@0.2.5': + resolution: {integrity: sha512-ki+/tyoi767DsNyaw95O5NDMt1MbmRp5PI5ZfZo11AE6yUDKeS1H5cm3N3SE2GdwtuKyT3ZBc5/d8MtWzTPObA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-special-log1p@0.2.4': + resolution: {integrity: sha512-x7YYsmM3NqfcZlDwZ4x88e5MtkXHTtUAroPhuDgDk2Ez9J/ZWQY4/t3RHb+wwEYtZU1WG4PQ7VMDlHOJUvEhBQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-special-log1pmx@0.2.4': + resolution: {integrity: sha512-CrF4ucxzA1hkR4zSHaXWZGhfFuHS9A80nZP8cvIBvrtU8Dl21fHqCegOSSZBNvzL7WPssCXNzgkfdf5GUGvwlw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-special-max@0.3.1': + resolution: {integrity: sha512-TV4GK3gt5BCy4se6Rn59zSbcL2EQBYynJ4g15KeniZWrQbvpm7WdjdJRJk/Ly78ESAoM6Qd2KEaP/bTGukGTqA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-special-maxabs@0.3.1': + resolution: {integrity: sha512-nvlh0Pkwrmh3E6Nva+o0dbAksJ7BErsp9v3Q783TGNlwsRNovOc8FmrkqvmuDhWkSoUUuYtefYyxz1HZ+CzvDg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-special-min@0.2.4': + resolution: {integrity: sha512-RjwMYI+CqdD8Pg41RBsvv0g50nl3d2a5SJBXDcjzHIs0Lj4+qdOXSYQYbn2IQH0dVCNlZrBwaGX3L8O9tu5umQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-special-minabs@0.2.4': + resolution: {integrity: sha512-aRQBXaoSBj026u90ZTreKa88GAF8SWSqm2WD43v22TFJVJ0qhjtSmQXRLs+tD2ug7LDUHa8Bf1Ui2DYuVw41uQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-special-pow@0.3.1': + resolution: {integrity: sha512-8i5zbz0KfhnNeACMsT+94/iv1/zCNj5JurXvrNHfLg3eLwvKIvwI28G+anpaqGZdPX3aJq66M5+dKDMSWvUvyw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-special-powm1@0.3.2': + resolution: {integrity: sha512-GwjXuzaKy4uO+DDFirt1HIKlvL5359ssx4AgMhYuPnwcHvq+B8NSo8gld/jgcshTXelzkSHvupOcgZrBO8gZVQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-special-rempio2@0.3.1': + resolution: {integrity: sha512-tHqU2T0ewJKOUhlAqV5hc1iQzPC2KoBn88ahOBndQqfJUKUG0vg/sDPfi6LF35RVo+yW03pDMzeZlAmCi4bSzg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-special-round@0.3.1': + resolution: {integrity: sha512-YoFhBJb0J21QLG1sEHoC8Pxnb+eMfu+pAeT40jF1vlC6p18MFEDVgcX/iBpJg52jPal7CHJF86X6XxlbJCh3Yg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-special-roundn@0.2.3': + resolution: {integrity: sha512-u3G0ZUNJJzG8BFHmy+YhJpBFklTzF7I+Jz502bhv84GBjfwaaFOaJP1/RADmjcZsWU+ZuH/5G/kmFSwMFtKc8A==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-special-signum@0.2.3': + resolution: {integrity: sha512-a4Ar0QsFqsie6dOAvJQyb7OWpIZKIbaa6EVn/ZF+B/CPfczQvqrv7UujeH5tfwZYOXqFbKXAOXv//nQm8/4yHQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-special-sin@0.3.1': + resolution: {integrity: sha512-t1mU2N6muFdXaafeYtxB3NQ9udxzL7gwFtKyph+GaMWJKLg1EAZ9G31StMFh5dqxmY0IjHLdkDUhwwh3pXxgPQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-special-sinpi@0.3.1': + resolution: {integrity: sha512-IHBiUy9KmG0pR6nqCaVktw/7j0IestUk6gIqMIQtQSyC11sgByhNJFKEBbIwqLgPGnLJ10B8Y3iN5jgBC7lRpQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-special-sqrt@0.2.3': + resolution: {integrity: sha512-Pdgq9Imx4lxsmM4jouu4b2ynB1osgrKLkJtdsaNUjrA3kTxc16FALkAJVEdKA9/7FxEFLji9CvJ0VBOWXzoyTQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-special-trunc@0.2.3': + resolution: {integrity: sha512-ga3rqr82G7Q3eqpF/aFR4v2Bj3meKhbuWRiEWrJwWARTqQa0e8aWTI7p9qYeQ0mFsKVoXH21oESjAHuJ1WfsQg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-tools-continued-fraction@0.2.3': + resolution: {integrity: sha512-EyzelNGzuL7wFfzbpuDFbf7//a3Z9iSRtIEbVbulduXQ8uYbB3YGMbbuKES2nG+WBeawpKFAgMDTC6siPdo1Vw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-tools-evalpoly@0.2.3': + resolution: {integrity: sha512-SJFnTcVUeDqVp+wJIZe9yj6Ddxn7kaf+aCepbmnxqdJLtvOtCsEyvOYX5q26x6Ik/JJULTNiOMQqWm2VInNAzQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/math-base-tools-sum-series@0.2.3': + resolution: {integrity: sha512-2Tc5fjKCXizXLb266n+AxNu0tTXau3DLd2lVPDhyBSqXoCqT0N/PZBmxWujaRuTpr7WxmEMzoMTxegwz487oaA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/napi-argv-bool@0.1.1': + resolution: {integrity: sha512-xW+BJ8wYeJAs3GPhu8SdlZur9WgzfKwFESJR8gzsaKkUy9+1POMs3jZA++I+Ote7CNY0AmFgC3/5v+qLNcb+Xw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/napi-argv-double@0.2.2': + resolution: {integrity: sha512-QJrnt4CTS8x7WHny4s8eHvGT2u5x8BSH6RWVaPg6ZrnHwqz9Jz48JGTawFKLgcyvZyeE9kCwKDkPIN7MbXcHKQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/napi-argv-float64array@0.2.3': + resolution: {integrity: sha512-0uZaYZC/sV6UN/rDNJxQVn5U4CsdjPGztT2c0D7i0bPPwLscv4THKyLge6x41acHx1kjPuEvq03l+Wg94/DL+A==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/napi-argv-float@0.2.3': + resolution: {integrity: sha512-2hNFgenY7M5iVfIbRw+xccutxtKfk89VQb1RMGbSAOrRcR53J5OENAaddohrCp1zT1FKe14LCG/k+LlZR5jEdA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/napi-argv-int32@0.2.3': + resolution: {integrity: sha512-Set5AdSIUePU/Y0gT+WC4lerzahwaAQJd1USCgnKQD55xmDfXAPMyJliZnZPNUyjQ6SsWN6PaiOfJluq+oVcVQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/napi-argv@0.2.3': + resolution: {integrity: sha512-JQe8cqcCmxVhAMZ6z7Ysx0hmY8DtIGEXcM7Ymmjxl5VNaVXq/grw3HEQm5Fcb+TdqJIb7xEbsRHCiowRzvk0Pg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/napi-create-double@0.0.3': + resolution: {integrity: sha512-6JcSG40t8HQGOgRaaxpxMWKrkqqZbSnUIKi9qx8R4g1gWWzbP5/aRMmMcLpanmXP61PjryfpnjhtUC27WMFNfQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/napi-create-int32@0.0.3': + resolution: {integrity: sha512-tIT9BWA3jmOjzz4XTwVplkKETI3dyTK4BZoMtbQGnTxfsHz+pNdgepbp8hEx7wJADHKO2L8YTqddTHij4H3Ptw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/napi-export@0.3.1': + resolution: {integrity: sha512-JQaR2FYyIGprLgjg02JwgGGbjaYgRsWJpF72T748AiUQMd8SEN/mjUkqPq4zYLRjYATpKxnDUQNvP642z9JnIg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/number-ctor@0.2.3': + resolution: {integrity: sha512-T4xeLGny/gBRe4ZJcS5ugluT8E7Y/LCEHycTQingYWqbBK+5XJED7zJrcawkneewXd7CgqypCtVuI0BOxBBzcQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/number-float16-base-to-float32@0.1.2': + resolution: {integrity: sha512-OClMQYz6GJEiSxrzhNp2LK4K8j75Rq6HCk+ReRBNQjXYdvNfzzaYoqnv3ZSzrXdvqUoGz+19XaVTLLzkKT1GCg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/number-float16-base-to-float64@0.1.2': + resolution: {integrity: sha512-1MNdMfThDFEIMeJHKmum343x9sbiRplRmsjEHojVGyMQNuxOamzbzw+Gw3xl04GrfOIUG673KZEljwUwDFgtUA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/number-float16-ctor@0.1.2': + resolution: {integrity: sha512-bdBhgwgtRClxBU9Ef8puY/WQ+DMqUr0oXfkfCrYKUATmqVQghv3bDw/dMggz01AguJo2ZtdjGp3PjQwNZlmHcg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/number-float32-base-exponent@0.2.4': + resolution: {integrity: sha512-JUuSZCPD6yWA2jJhtS5MIYSZGerJJCXfwStz0Mqsh76x166mSkiZ4dH3VpDz2OMrZk6rvt35UB2HhAZecTqFAw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/number-float32-base-to-float16@0.1.1': + resolution: {integrity: sha512-phYWBtfheX0c/pXtbiK/xr6lVwU5HKYVNgmol8y0HE8NvwXG3oCXLySa2p0jr+Xi7G9xtMdXW4Eq0rShs4lTmQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/number-float32-base-to-word@0.2.3': + resolution: {integrity: sha512-E42KOQ+jqVDElwvFtrDMEkXYl03btIPan3g/YAScCibfmYDkYRDT9iuaydyYS/O3+m/MPM1Jhn+U/B8+f+C1HQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/number-float64-base-assert-is-same-value@0.2.2': + resolution: {integrity: sha512-kE5uKSaMeE/V/zJf3B1jd1OOZ4biwXOp3hPuaOYm4s9J53Kq+53Hbq0wLLAVta/KZ1cbTp72nOjdufiMefyABQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/number-float64-base-exponent@0.2.3': + resolution: {integrity: sha512-KRicBsOMrVCIB2veebT+mPpTrxyXCP6eknJTs8A59E/ATU2/gKYbMKzkdEX/FzxSBcdw5thvuuqmmwJKGZYSkA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/number-float64-base-from-words@0.2.3': + resolution: {integrity: sha512-7v0bOW1AjsoWqkiVFqljLxCWmGA+iZ3OA/NwuIFBsYOEOpZqMIevu6TkJkWYaP6YYUJUrSG0w6t7b1P9JuzBgw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/number-float64-base-get-high-word@0.2.3': + resolution: {integrity: sha512-yyJJR8+8BK88ldB5rM3GaM6S7vfUXZ3ib2VPse7Jb/ecd9r9gE6ojJv9v7N/M4STmKVrWWcggFEHzAVe/KMhrQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/number-float64-base-get-low-word@0.2.3': + resolution: {integrity: sha512-e2OcnZ8wAYU5w/sFXY5d8bOFyjWYHW4UxdxLUsf24nVKyBPvS2Rhib8NbOETPeT8WYqFN6yrL2L4VlXO3Mz6ig==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/number-float64-base-normalize@0.2.4': + resolution: {integrity: sha512-WVtJYkAt1zZXRlcjmbGEYVlZUd6dpSsOKACmVLEWyI9kMlegRAwibAKGMRtAcuYFSOyRVONwJbYar35McmabHg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/number-float64-base-set-high-word@0.2.3': + resolution: {integrity: sha512-tWyEAJlbINvxcGMK5wgOlfFcZP9ILpqdBWLrc9HCpbUul41pmWEZi7+zI7dwe5zWIlACzguykdY0cEe7raceIw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/number-float64-base-set-low-word@0.2.3': + resolution: {integrity: sha512-lmZvzkVlPqM+viBpmI0lBzArRb2+ifBMhuZ/Rtla+O91bcXgBtx/+L4UuqofAfQ5PRRnCW82r3kzILEKsxSoaw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/number-float64-base-to-float16@0.1.2': + resolution: {integrity: sha512-d+ZJYGoykrzI3H0GRCI3qqyjspgNcIvECQ4OjyC1qdy1haT3cgoj1rwWkJS9uFKD/QgWlFGMrcmRzf0MVj17zg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/number-float64-base-to-float32@0.2.3': + resolution: {integrity: sha512-+Lu8vk1oLCxO8RaRzw8pEY+BP2TWROweNFiML0w7TP4i9ZyG5E6EJdxZ2J3c29LVPfsf+oLOOxyySHH2kjwIsw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/number-float64-base-to-words@0.2.3': + resolution: {integrity: sha512-G63YuysfTKRP3Tkn19zBxTseS+uYnnOYZSN9VRTWrx5iNE8IYDzanAJIolSBZ7PuFMoE+HmjEZ0E9SX3NsLKvg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/number-uint32-base-to-int32@0.2.3': + resolution: {integrity: sha512-C6jGEzdnoiy9Er7mTwWO34r+6DVPRRVKRGVtxSQW3yAiQOnLS1ZvimqvgZtl0PYu1OnE02ReegpOMkpqSGO3vQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/object-ctor@0.2.2': + resolution: {integrity: sha512-3zkUkzZ9LdX+J5fEyU/D7I2c+0qG+ejNI3tCtSFek35bezdqWs9tOAAUiU6HeQJcGgqOmkakB4apIxvN+eehCA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/os-byte-order@0.2.3': + resolution: {integrity: sha512-QU4JY7WN4inoLVz+ML92TOevgyxZa+d2RcA1EmaDC0y0TNrXez1GmIn1t7+L7UTJEdf39IUm6VfdMmYdcDwFTw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/os-float-word-order@0.2.3': + resolution: {integrity: sha512-5dLZBCNn9Fge3kq7uHIh7S1qshfe5L7pjLYLSUGxxKa7gecAbsaQBPLxDkFiFwMytr6tnGGjWQfyVbhq/+RjlQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/process-cwd@0.2.3': + resolution: {integrity: sha512-T/sYVJjs9iHAOK9B8yGymGY+59eAz20G7eL/G4cObT6sWssAF7s/2+KgzAvWB0maE9qX6qnP4zBg2msCsWwvVg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/regexp-extended-length-path@0.2.3': + resolution: {integrity: sha512-3HYXiSzBpKz4nSOCtrYtMw9/OXN4EmTvq0owXv6HMvvDweFskieBcdNlVcOl0ViezjvUuvUL0DHJHdGp6SQEIw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/regexp-function-name@0.2.3': + resolution: {integrity: sha512-ER1C2rUW5Kzu5w4W0gbrJdIj5STNO5RfNy4GeiqycezeiXT99G393QeN4irqcOItYkeZhwbpY2ZzwlLxQvod1A==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/stats-anova1@0.2.3': + resolution: {integrity: sha512-UuDnctJuggfwuFmFJS0XGVN8x9S+lXCov7CvqkBaCThJH0k8iqOD7aLKR1L7qP9O+jSxuyNkk9CYtLqQ0J2rxA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/stats-base-dists-chisquare-cdf@0.3.1': + resolution: {integrity: sha512-j0X9V9FO0kwpdmhaa38+JuEjkHws1b3SWJUGuCx9C/eUdTY4soi9EwjdXheDRu0ww9qRhrECpXobHUiLJEdnTw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/stats-base-dists-degenerate-cdf@0.3.1': + resolution: {integrity: sha512-i6qu8JegKFJ/xJcnJH4x+wMTxKbHtMG2bOVjdFswxt5MHGEob0bmpZMCPaUQ99KDXx1ntntbAzqURt8Bwhi12g==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/stats-base-dists-f-cdf@0.2.3': + resolution: {integrity: sha512-wl74kdmaGjUlFkb2VI5ml/VyZsUB89EkHJi+tTvr281MyM9D0JYd0xaDKai7OnJ4R3S5YhFueQ2hkAjnIDwEkg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/stats-base-dists-gamma-cdf@0.3.1': + resolution: {integrity: sha512-L+S2aLWixmdF+hHDt+R72yduF+jvDljTU1tpcevBD1CxMNuGliShgsbU9/w10odi30SW1HtaS9cRA99kWJmm/A==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/stats-base-dists-normal-cdf@0.3.1': + resolution: {integrity: sha512-a4xGRE5ZKaB1J9wGEXu7gforIA9mGkR3BqlHPaaUuasDBdYxsI6NH7mZr/zeWaT4c176iiZzJjYMPWUAdV4aFQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/stats-base-dists-signrank-cdf@0.2.3': + resolution: {integrity: sha512-i6XUeuXXjCw9WIabx1wouBJ4RDxmkH85hYT3JrQWJwUxvDRUypYaI4V5abjeNYwlFByo2wiKhhjYe4nCXKg6hg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/stats-base-dists-t-cdf@0.2.3': + resolution: {integrity: sha512-Y6LUuwknLaewhjnJgMwcZuiD4adx1hzEkbnnrwlqh7Qz+CFyh1D5Ov7ZoyHkcs5djAZcOTaVbn8Ktc+EQ6A7mA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/stats-base-dists-t-quantile@0.2.3': + resolution: {integrity: sha512-HT6cfDL8EKt7H5rpA5x4YHOfXNAOyxToijTBiQSEvHoGnRoULcc0RFm+KF3kI85HP55AVkg6/KIuukdpRICxUg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/stats-incr-stdev@0.2.3': + resolution: {integrity: sha512-vjYynYt2/phYnn05EWQeVCXpssy+aE9rIeYpwNtiu24yR3BZ4scAYoxn/O6toG/hnmxAW/DZl4qAvhuTUq7hiQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/stats-kruskal-test@0.2.3': + resolution: {integrity: sha512-T9fgDFwGS0R7BCCi9Gp+AKGX8F3vHWOw4zp24d35XlD35trl7eYtrg35kYxBD6khsmP0Fd3oioYCWbj+eYzJHw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/stats-ranks@0.2.3': + resolution: {integrity: sha512-SoXELH42IHmW8gdpqLGnMt6PNCH3A9CqXcXojh5Ff/RGhZMrRTdQphTjNfqfNdwjmM8eqlI7SbOG93Alx7QoHg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/stats-strided-mean@0.1.1': + resolution: {integrity: sha512-45UdxpAPbb3d+8obiaHOeUBie0GlB6J0xV+vMhkGR71AO5UaEPLxFnIRgx9dz1KLuThw8Ms/TQXTvXV3Z8YnvQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/stats-strided-meanpn@0.1.1': + resolution: {integrity: sha512-Mu6bByEITlud9K/nrO7ICA0o5K+CP09GhWJoXS8BJWi2qmw/h/3O3RzBKkinKkUhoW9LVNSI+wmng4fU+rsXOQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/stats-strided-variance@0.1.1': + resolution: {integrity: sha512-nrO5mvyRUV8Ta4Y4HsTGY8uQ97+e0QZyt+T6ASjqvqKe6B+KC1yBGXKAcxxa698Pyhx2eFHTrvpm39GGI7SQUQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/stats-strided-variancepn@0.1.1': + resolution: {integrity: sha512-F2GPQSda5jxsZ0kZhYXLc/RJ4uMknWJqYx420/Ed5Du1NIDMLxy4mCleJxgJ/zVLS9t8AgAq/EOAx/QOY78l2g==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/stats-ttest2@0.2.3': + resolution: {integrity: sha512-CJIrZTDmyJi0TrscYO2qiE9M+rsWcKEa2a0nwh1e4VD6glJxO5q53t4y1Yr0D3OyI17ZMIhS7o1zJZnxD3mDVg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/stats-ttest@0.2.3': + resolution: {integrity: sha512-sbwGkIvLNTi77qJ2fwscQH+wT7XGv5gM2MmE6HTJsDk4XTmhyPTlSsE4mbuGKz5UXlnVCrL2qAp5KaW+TlTHlg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/stats-wilcoxon@0.2.3': + resolution: {integrity: sha512-w0+WZ8TDbM9jW1ASnjDSYFHSO3xJZuGkr1ojt/DUufB0KMNjhrsoiQmljPN4QhKI4d/DRHU+ZR9MrxSBn123XQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/strided-base-reinterpret-boolean@0.0.3': + resolution: {integrity: sha512-co8rBbg9FYbz4X2ZGMMlt/+MpC4gq/JhxaSYMby2n92hvK57Ru62L35Kq1nngBHFb6hshhCRbnFXMyVzL6OYwQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/strided-base-reinterpret-complex128@0.2.3': + resolution: {integrity: sha512-7y/Phr1B+d2GBnyv9a0DQBwsZIvDstLXNuEfQ+Y9NOzl0NG7pQvzlTkm6MTtxIbsYoQO44GTpIHEK8ag/cpa/A==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/strided-base-reinterpret-complex64@0.2.2': + resolution: {integrity: sha512-BgbLBqLqoc0GfdyHPI2xYuTOqQkAB2ivBw7rN35ev0J2GN0X9rnUtRKvl5V2H3WFqsvujieS6HwGZ+fmF2Tocg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/strided-base-reinterpret-complex@0.1.3': + resolution: {integrity: sha512-3mG3Uqyvvr1NXQ/EAg3Yy5V5VAWLBuZ4K3/cbaxXJTl4DL5r+Uh1Taus4pgy4j5Z2FVkK/4iqjpnRWx8sj0q6g==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/strided-base-stride2offset@0.1.1': + resolution: {integrity: sha512-c5GkXFQOKEbfWM3f8db1AyDCWswazdgTKJrIj7DZZJSBUUECmg5avHPJy0juFHn1Sp3JfzQUx6ViQXqm6VoarQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/string-base-format-interpolate@0.2.4': + resolution: {integrity: sha512-POG725+DPEmzEJbMFTFPdKM4vA5i+t7juNYs+H2cQz5JXiPV1+5+P3epnO+/DqHWzLhiMlsKDCZq03C3WcBkSA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/string-base-format-tokenize@0.2.4': + resolution: {integrity: sha512-qYHSmqFv7vloLFvjBAbZCn5qtxj7M+Qru3VqBooJTxWMcUeZOEBZg18/PFgfUrZhji5MVWDXbRTAx5Bu5M5nQA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/string-base-lowercase@0.4.1': + resolution: {integrity: sha512-3NcDy2j6HtvKrC2GRCt5mKiYaWWHLLSQRSbuu28WEzA98/2izmiHYkR0i9IeEd4Tqrxqof/FXP4gWj1f14RXTQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/string-base-repeat@0.2.3': + resolution: {integrity: sha512-Cd9shcpBgwM0MsYCcE3sLmtEhph1tQDK+JsfvMmUpdAOdR4fIVtbT7W+6LT/dSXxCqCsrdDypBY148lITzZ8zA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/string-base-replace@0.2.3': + resolution: {integrity: sha512-FJdh2GzIkgMlr0v/ZZKD5cWRs+SMOhBTWxAexmWoYFL3WL1YMDgwI5hRtYh/ySyOy94MjJmYwRMpIVdxeclrAw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/string-format@0.2.3': + resolution: {integrity: sha512-uE0LHUUnWKfxFeipyfb9yWbs+iczz6dHaolSGC1WBzMVyeHqx8xvq+yP3f2POYWgEcknmxNgU21WSqToSNrxdA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/string-repeat@0.2.3': + resolution: {integrity: sha512-6Bh4BCnldYilcc0uwX2CK3ujuBQNI9uoAPgEHVI012hTpsT610o5Zxcis3JLsDgG/IWG/Ig+vYnmIt989nn6Gw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/string-replace@0.2.3': + resolution: {integrity: sha512-CXJhl/L2TfRrkVUNVp8OfNsKeuUzOZXzJLUDoopHtQalbWBww1fDSDj1fwGpzgZV4IODs8LCsiIxLBnLRP6emg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/symbol-ctor@0.2.3': + resolution: {integrity: sha512-wnuFrxnmhg2p6QQP0B/j97LD5ys2d1bHuwVvh9yuUSCLoX/GankaVyGSufk2ROlsxrcm2HPhSDB/ILBHeUmamA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/symbol-iterator@0.2.3': + resolution: {integrity: sha512-HN609e5u8+O3VtiHbQeuGeKMgR1/bZ0E/qwObihTA5Yca5ATDJsTZOxY463JNU41x+6xlptO24l4OJE+t3IBAA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/symbol-to-primitive@0.1.2': + resolution: {integrity: sha512-//r52ighL35HprZrK5Pdiz1L27l47X9o+0LOJ84mTk5eB89i9shVRjOpHauKyx/uy9VkDoxUkMY0PwPrAIp4aw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/types@0.4.3': + resolution: {integrity: sha512-9GCqS2eni2VSwa5/CCniAJ4I1eWLtvior1z6hoPJp6VTNMgW9Lh4BiI84IO0xun/0qAclJ+mTGTiCTZxZLK13A==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/utils-constant-function@0.2.3': + resolution: {integrity: sha512-mQ6606O8JKq4tgL508it1ro7hx+RK6j36URSjbRI7Dje72YYpdQJGnN6UJ183Im5TT0fN2s4oOCM6+SAGcQi2w==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/utils-constructor-name@0.2.3': + resolution: {integrity: sha512-/ApogDUIFxndcmnEeI6ctsqWt66c40g1T7R0nS9aOoYgnDOdmpUFSflzf++MDWksBycGsqgm1UD7IU54m4u7jg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/utils-convert-path@0.2.3': + resolution: {integrity: sha512-fmqJ6Qvce0HV7D+uYrram/+iAHB3u2gMys9zeh0uc/h411sB8Hw9X7i1w2cJevCpouMzqTnpzp75C+swkoQlKA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/utils-count-by@0.2.3': + resolution: {integrity: sha512-cK/P0JXJHFC8jrONRnedhSJhHTlvG641Ay5Q1kpjum+R2ryBkVcgxNAfpfrSaiLOTDKRmj9dFuNv1cXN1h2S4Q==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/utils-define-nonenumerable-read-only-accessor@0.2.4': + resolution: {integrity: sha512-ZxHDIuA6d1Wr10Nkav5ejS53LS0eAwXV9IijQLQ/vD/rCRMNnLYT1hC2a0ZpDRc+lHQ0xsyHQvmYQJy9GBFkpg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/utils-define-nonenumerable-read-only-property@0.2.3': + resolution: {integrity: sha512-W0rqnRsXgW3GjcwEcOMi0mI/CXn3TVqmFvcxItLUZPJNYmwrU6+t+9kGldhMw845DyOtv8uYASrnNoVE/VfiDw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/utils-define-property@0.2.5': + resolution: {integrity: sha512-WpbXc2uq8vtSqmWFzVrFifNj6HGoGHIZHolxX5GfRFbj4RNme1u/wtklmiOvcHE0s6SwPX764tuFXIgHBR5Vag==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/utils-define-read-only-property@0.2.3': + resolution: {integrity: sha512-3sGMpmmSlErHup+Y6YwIgN075GW3t1FD8ARS+zHaU1U1IKEaiOVKMH49WfBGiSsVfV7sfieoZC5B//MrSWQjaA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/utils-escape-regexp-string@0.2.3': + resolution: {integrity: sha512-ZQAq5HHF+C1fmK+RvqeMO0iIQf5kGl2Ofx9LYloURUPZD2zVMX+2hxDLFBTPAusqYxffQAjj3ap4LE/J0+YiRQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/utils-eval@0.2.3': + resolution: {integrity: sha512-T9rL6lEX58My6GcZ0bXD7reJoBEzoO/r5i3U/lj6dC94immyQtbqoz0djHfpCVrCnoK4Mzbq3t813cBQS7yfRg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/utils-get-prototype-of@0.2.3': + resolution: {integrity: sha512-unEoKhhAnV4AbQ87oPo1v+4xLjd00nQxJLSvl43PkdoBn/GKNv/B6SMnQWmM5b62esWTbxBu/uWCYoglbprxkA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/utils-global@0.2.3': + resolution: {integrity: sha512-e8gyuENhSeF8udH7vlZMDbNVQ/ZxZ4vWOcpB2iw06z1HSFZWc31Coh19ScfgagLfofo2GQYPJI3tJtZQYsUS6A==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/utils-identity-function@0.2.3': + resolution: {integrity: sha512-XZNW1j/mWbRTP8WeIVCQX9o/2HOudrCIgRjte5EJsDjZp2fKW6mTjm96s3P9NGd4YQi3jzDm36VWPjLRQziJRg==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/utils-index-of@0.2.3': + resolution: {integrity: sha512-R5lBztlvTWiH0CFFSYsSDGDexQH5y9UVIAHNevp/uS3jE0+nMHREWEkIwCHxkAsYlbi1Dlbyq8+vapswxgBSNw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/utils-keys@0.2.3': + resolution: {integrity: sha512-pH7orA2LGYVQ/uTDtNFPocdpF8AfiZIcJpGR1178OhoaMv7lXa4avVHCNY5TT+GHZebJxIWJum1zmi8onYuVwQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/utils-library-manifest@0.2.4': + resolution: {integrity: sha512-sNzEclCbpRRa35DBv8ZOcOjCesBLczkOFX8o7mMrG9VWhyZuJUCBtgSPmzGIIBLAsVEYJuguEk4iYoMqFnCwnw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/utils-memoize@0.2.3': + resolution: {integrity: sha512-H8OGqss0lsbTAFKxbaQ+xywRgtcn9WV4w6p59tjMVmsDqQNGGrwvf6TuljUJY3Wc21E5VK4Vcs2O/MxBIAJsZw==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/utils-native-class@0.2.3': + resolution: {integrity: sha512-UGXCPhgmOjwMpEQ5fFiifcqrhUnP8ICyWzEkHphUrh2/0pHyK0zilZvcODfROgzoy6L9ivUUuR6NjrAI+j14hA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/utils-noop@0.2.3': + resolution: {integrity: sha512-kZQWwUwq+RwmKBZZD+6dnRwhjfut5fBgbEappugvVjFGp12wDKXCj6D6Y/xTmBdKFRumO8/gzkj9n6DKNDchpA==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/utils-tabulate@0.2.3': + resolution: {integrity: sha512-00HL6s5KPcAqwobAjmaTlsGK7i6FVb1IFqivaKoIhn8rGPuFTYmeo4Kc2ZyXGitteQJjXABl0qjF+yei813OVQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@stdlib/utils-type-of@0.2.3': + resolution: {integrity: sha512-2KmDUfqIv0g4lrWtnyR5xw82NvdsEwUHON43JAe55XRlLWnk1V1troiyJiM9fccENtl524kVRBnD7RMohmUyyQ==} + engines: {node: '>=0.10.0', npm: '>2.7.0'} + os: [aix, darwin, freebsd, linux, macos, openbsd, sunos, win32, windows] + + '@turbo/darwin-64@2.10.10': + resolution: {integrity: sha512-gFDD+wRP5hWxBRghGyEbjpbLOY7aIU/wvsnKdMM7odQcp/wHMrnI83p0FyxxMRZnFH9ZD+S59MvcpOC5b+nrCA==} + cpu: [x64] + os: [darwin] + + '@turbo/darwin-arm64@2.10.10': + resolution: {integrity: sha512-VZYsxZ6yjyDosUqtiroAVSXPLmx/qBxdHJgIxdMH9RyNmLdOLOWtJnYMnI4qckwCgQMK85G3fu94/xk5+iBCgw==} + cpu: [arm64] + os: [darwin] + + '@turbo/linux-64@2.10.10': + resolution: {integrity: sha512-lAvW+yEnmsCKMEIwNugjozawvYytHKPhU0kfLBizu83MIs8OUb9KobYvkZ56L5akSM6K7+gBFLEIfQkaceh90g==} + cpu: [x64] + os: [android, linux] + + '@turbo/linux-arm64@2.10.10': + resolution: {integrity: sha512-MSJ+NkRTd79Z9+YEZpUV9VOWVOOigFhE+v/ETNYJEuTJp3r00y9YgFvDXrmM+DP8Kal6tk3U6xSugD2/Ojh+Jg==} + cpu: [arm64] + os: [android, linux] + + '@turbo/windows-64@2.10.10': + resolution: {integrity: sha512-ycWpXDkUfnDFDY9d+4Qna/UZotDB0wj+s9agrlmNt0Q7a3XHORhK8GPKJdzgzeutXu9EW5P/jyabTEHlohuDXw==} + cpu: [x64] + os: [win32] + + '@turbo/windows-arm64@2.10.10': + resolution: {integrity: sha512-PMk6zQN0csUFklLe+1hz/5G9uU1YmV0cEIey2R/bSeA6o69qcBTlN4A3jOqkgenOO5dOpMHmq2sEUZo8r1+Ssg==} + cpu: [arm64] + os: [win32] + '@tybys/wasm-util@0.10.3': resolution: {integrity: sha512-F3fo1MYrRJYL3zER0OUOmkutjr1Vp23m7OsSgp7nq4SP6OqX6C/56XFIPAl5bt3zaBRjmW7SGz3u/6LwFpYcOg==} @@ -2333,6 +3977,9 @@ packages: '@types/d3-format@3.0.4': resolution: {integrity: sha512-fALi2aI6shfg7vM5KiR1wNJnZ7r6UuggVqtDA+xiEdPZQwy/trcQaHnwShLuLdta2rTymCNpxYTiMZX/e09F4g==} + '@types/d3-hierarchy@3.1.7': + resolution: {integrity: sha512-tJFtNoYBtRtkNysX1Xq4sxtjK8YgoWUNpIiUee0/jHGRwqvzYxkq0hGVbbOGSz+JgFxxRu4K8nb3YpG3CMARtg==} + '@types/d3-scale@4.0.9': resolution: {integrity: sha512-dLmtwB8zkAeO/juAMfnV+sItKjlsw2lKdZVVy6LRr0cBmegxSABiLEpGVmSJJ8O08i4+sGR6qQtb6WtuwJdvVw==} @@ -2402,25 +4049,11 @@ packages: peerDependencies: vitest: 4.1.4 - '@vitest/expect@4.0.18': - resolution: {integrity: sha512-8sCWUyckXXYvx4opfzVY03EOiYVxyNrHS5QxX3DAIi5dpJAAkyJezHCP77VMX4HKA2LDT/Jpfo8i2r5BE3GnQQ==} - - '@vitest/expect@4.1.4': - resolution: {integrity: sha512-iPBpra+VDuXmBFI3FMKHSFXp3Gx5HfmSCE8X67Dn+bwephCnQCaB7qWK2ldHa+8ncN8hJU8VTMcxjPpyMkUjww==} - - '@vitest/mocker@4.0.18': - resolution: {integrity: sha512-HhVd0MDnzzsgevnOWCBj5Otnzobjy5wLBe4EdeeFGv8luMsGcYqDuFRMcttKWZA5vVO8RFjexVovXvAM4JoJDQ==} - peerDependencies: - msw: ^2.4.9 - vite: ^6.0.0 || ^7.0.0-0 - peerDependenciesMeta: - msw: - optional: true - vite: - optional: true + '@vitest/expect@4.1.10': + resolution: {integrity: sha512-YsCn+qAk1GWjQOWFEsEcL2gNQ0zmVmQu3T03qP6UyjhtmdtwtbuI+DASn/7iQB3HGTXkdBwGddzxPlmiql5vlA==} - '@vitest/mocker@4.1.4': - resolution: {integrity: sha512-R9HTZBhW6yCSGbGQnDnH3QHfJxokKN4KB+Yvk9Q1le7eQNYwiCyKxmLmurSpFy6BzJanSLuEUDrD+j97Q+ZLPg==} + '@vitest/mocker@4.1.10': + resolution: {integrity: sha512-v0xaezt+DKEmKfaxg133ldzADrwLGd7Ze1MfQQTYfvs8OqZIwbxyxaYURivwV7sWy5fqn3rH5uOrSp07bp44Ow==} peerDependencies: msw: ^2.4.9 vite: ^6.0.0 || ^7.0.0 || ^8.0.0 @@ -2430,35 +4063,20 @@ packages: vite: optional: true - '@vitest/pretty-format@4.0.18': - resolution: {integrity: sha512-P24GK3GulZWC5tz87ux0m8OADrQIUVDPIjjj65vBXYG17ZeU3qD7r+MNZ1RNv4l8CGU2vtTRqixrOi9fYk/yKw==} - - '@vitest/pretty-format@4.1.4': - resolution: {integrity: sha512-ddmDHU0gjEUyEVLxtZa7xamrpIefdEETu3nZjWtHeZX4QxqJ7tRxSteHVXJOcr8jhiLoGAhkK4WJ3WqBpjx42A==} + '@vitest/pretty-format@4.1.10': + resolution: {integrity: sha512-W1HsjSH4MXQ9YfmmhLAoIYf1HRfekQCGngeIgcei6MP5QQGWUe0gkopdZQaVCFO+JDJMrAJGwa5pRpNpvy4P8Q==} - '@vitest/runner@4.0.18': - resolution: {integrity: sha512-rpk9y12PGa22Jg6g5M3UVVnTS7+zycIGk9ZNGN+m6tZHKQb7jrP7/77WfZy13Y/EUDd52NDsLRQhYKtv7XfPQw==} + '@vitest/runner@4.1.10': + resolution: {integrity: sha512-IKI6kpIH+LmpROplyLwBBaCfMgOZOMsygVa6BARD6ahA04VRuJSa6OaVG7kRvSEMD870Vd91rSSw0eegtWyLGg==} - '@vitest/runner@4.1.4': - resolution: {integrity: sha512-xTp7VZ5aXP5ZJrn15UtJUWlx6qXLnGtF6jNxHepdPHpMfz/aVPx+htHtgcAL2mDXJgKhpoo2e9/hVJsIeFbytQ==} + '@vitest/snapshot@4.1.10': + resolution: {integrity: sha512-xRkfOT1qpTAi/Ti4Y1LtfRc3kEuqxGw59eN2jN9pRWMtS/XDevekhcFSqvQqjUNGksfjMJu3Y+oJ+4Ypn2OaJw==} - '@vitest/snapshot@4.0.18': - resolution: {integrity: sha512-PCiV0rcl7jKQjbgYqjtakly6T1uwv/5BQ9SwBLekVg/EaYeQFPiXcgrC2Y7vDMA8dM1SUEAEV82kgSQIlXNMvA==} + '@vitest/spy@4.1.10': + resolution: {integrity: sha512-PLf/Ugvoq5wO/b4rwYCR1h2PSIdXz7wnkQFMiUpLdtM7l6pqVFcQIBEHyT1+l+cj7mNwAfZHzqXqDyjvOuwbDw==} - '@vitest/snapshot@4.1.4': - resolution: {integrity: sha512-MCjCFgaS8aZz+m5nTcEcgk/xhWv0rEH4Yl53PPlMXOZ1/Ka2VcZU6CJ+MgYCZbcJvzGhQRjVrGQNZqkGPttIKw==} - - '@vitest/spy@4.0.18': - resolution: {integrity: sha512-cbQt3PTSD7P2OARdVW3qWER5EGq7PHlvE+QfzSC0lbwO+xnt7+XH06ZzFjFRgzUX//JmpxrCu92VdwvEPlWSNw==} - - '@vitest/spy@4.1.4': - resolution: {integrity: sha512-XxNdAsKW7C+FLydqFJLb5KhJtl3PGCMmYwFRfhvIgxJvLSXhhVI1zM8f1qD3Zg7RCjTSzDVyct6sghs9UEgBEQ==} - - '@vitest/utils@4.0.18': - resolution: {integrity: sha512-msMRKLMVLWygpK3u2Hybgi4MNjcYJvwTb0Ru09+fOyCXIgT5raYP041DRRdiJiI3k/2U6SEbAETB3YtBrUkCFA==} - - '@vitest/utils@4.1.4': - resolution: {integrity: sha512-13QMT+eysM5uVGa1rG4kegGYNp6cnQcsTc67ELFbhNLQO+vgsygtYJx2khvdt4gVQqSSpC/KT5FZZxUpP3Oatw==} + '@vitest/utils@4.1.10': + resolution: {integrity: sha512-fy9am/HWxbaGt/Sawrp90vt6Y6jQwf1RX77cz3uwoJwJVMli/e1IEwRPnMNJ7vKfPTwo0diXifkpPvwH9v7nGA==} '@volar/language-core@2.4.28': resolution: {integrity: sha512-w4qhIJ8ZSitgLAkVay6AbcnC7gP3glYM3fYwKV3srj8m494E3xtrCv6E+bWviiK/8hs6e6t1ij1s2Endql7vzQ==} @@ -2472,26 +4090,26 @@ packages: '@vue/compiler-core@3.5.24': resolution: {integrity: sha512-eDl5H57AOpNakGNAkFDH+y7kTqrQpJkZFXhWZQGyx/5Wh7B1uQYvcWkvZi11BDhscPgj8N7XV3oRwiPnx1Vrig==} - '@vue/compiler-core@3.5.25': - resolution: {integrity: sha512-vay5/oQJdsNHmliWoZfHPoVZZRmnSWhug0BYT34njkYTPqClh3DNWLkZNJBVSjsNMrg0CCrBfoKkjZQPM/QVUw==} + '@vue/compiler-core@3.5.41': + resolution: {integrity: sha512-q0Xtv/F9w2YO/7htQhtiL+Ev2WCJbe5N2hc+XfgyKkEKqWpSxknmT8QOuGdEKNdjPq0c3F7rNpFkTo3Kfrm7pg==} '@vue/compiler-dom@3.5.24': resolution: {integrity: sha512-1QHGAvs53gXkWdd3ZMGYuvQFXHW4ksKWPG8HP8/2BscrbZ0brw183q2oNWjMrSWImYLHxHrx1ItBQr50I/q2zw==} - '@vue/compiler-dom@3.5.25': - resolution: {integrity: sha512-4We0OAcMZsKgYoGlMjzYvaoErltdFI2/25wqanuTu+S4gismOTRTBPi4IASOjxWdzIwrYSjnqONfKvuqkXzE2Q==} + '@vue/compiler-dom@3.5.41': + resolution: {integrity: sha512-oKacVfNglLvGjnS6BXOlGL7EyG2h8X03pqXCjzotRZUaXGjbrTJUnVAQjrCqUnS+lyu31nwQjZY/d817GmCnfw==} '@vue/compiler-sfc@3.5.24': resolution: {integrity: sha512-8EG5YPRgmTB+YxYBM3VXy8zHD9SWHUJLIGPhDovo3Z8VOgvP+O7UP5vl0J4BBPWYD9vxtBabzW1EuEZ+Cqs14g==} - '@vue/compiler-sfc@3.5.25': - resolution: {integrity: sha512-PUgKp2rn8fFsI++lF2sO7gwO2d9Yj57Utr5yEsDf3GNaQcowCLKL7sf+LvVFvtJDXUp/03+dC6f2+LCv5aK1ag==} + '@vue/compiler-sfc@3.5.41': + resolution: {integrity: sha512-XJhip7R2wy6vX3knCxdZN4KracFaZUef58s1KYewqluedHIJaPIVfXoYT7MF1F8nCvv6k8bWWxDC8opMkg1VTQ==} '@vue/compiler-ssr@3.5.24': resolution: {integrity: sha512-trOvMWNBMQ/odMRHW7Ae1CdfYx+7MuiQu62Jtu36gMLXcaoqKvAyh+P73sYG9ll+6jLB6QPovqoKGGZROzkFFg==} - '@vue/compiler-ssr@3.5.25': - resolution: {integrity: sha512-ritPSKLBcParnsKYi+GNtbdbrIE1mtuFEJ4U1sWeuOMlIziK5GtOL85t5RhsNy4uWIXPgk+OUdpnXiTdzn8o3A==} + '@vue/compiler-ssr@3.5.41': + resolution: {integrity: sha512-U3v5OejKEGqOI0Wy0+Sz7hGuIFZHA4LSXzrNM3IMIeDyJEBBfTpX26n3SDgToRpP2bLc9FfI2j/kSgcJ8Emq5A==} '@vue/compiler-vue2@2.7.16': resolution: {integrity: sha512-qYC3Psj9S/mfu9uVi5WvNZIzq+xnXMhOwbTFKKDD7b1lhpnn71jXSFdTQ+WsIEk0ONCd7VV2IMm7ONl6tbQ86A==} @@ -2510,36 +4128,34 @@ packages: '@vue/reactivity@3.5.24': resolution: {integrity: sha512-BM8kBhtlkkbnyl4q+HiF5R5BL0ycDPfihowulm02q3WYp2vxgPcJuZO866qa/0u3idbMntKEtVNuAUp5bw4teg==} - '@vue/reactivity@3.5.25': - resolution: {integrity: sha512-5xfAypCQepv4Jog1U4zn8cZIcbKKFka3AgWHEFQeK65OW+Ys4XybP6z2kKgws4YB43KGpqp5D/K3go2UPPunLA==} + '@vue/reactivity@3.5.41': + resolution: {integrity: sha512-rznsqKM0np0x18EjzF8x88MpEhdNsffbvFbckLL5+oUKz1BxAImEmO7J1ArRYSyo6aQaVoBDp7jEkT91OOxydA==} '@vue/runtime-core@3.5.24': resolution: {integrity: sha512-RYP/byyKDgNIqfX/gNb2PB55dJmM97jc9wyF3jK7QUInYKypK2exmZMNwnjueWwGceEkP6NChd3D2ZVEp9undQ==} - '@vue/runtime-core@3.5.25': - resolution: {integrity: sha512-Z751v203YWwYzy460bzsYQISDfPjHTl+6Zzwo/a3CsAf+0ccEjQ8c+0CdX1WsumRTHeywvyUFtW6KvNukT/smA==} + '@vue/runtime-core@3.5.41': + resolution: {integrity: sha512-Vcry58hiAKwGen9Z1jUZE0feFsNArPCMOImYI8el48A9Idf6DuQYD0U05zZIF2Iad1hGhPSvcbBbAOhNr55fhg==} '@vue/runtime-dom@3.5.24': resolution: {integrity: sha512-Z8ANhr/i0XIluonHVjbUkjvn+CyrxbXRIxR7wn7+X7xlcb7dJsfITZbkVOeJZdP8VZwfrWRsWdShH6pngMxRjw==} - '@vue/runtime-dom@3.5.25': - resolution: {integrity: sha512-a4WrkYFbb19i9pjkz38zJBg8wa/rboNERq3+hRRb0dHiJh13c+6kAbgqCPfMaJ2gg4weWD3APZswASOfmKwamA==} + '@vue/runtime-dom@3.5.41': + resolution: {integrity: sha512-3vVBahVBS9+U6cmXBLyb8nE6/yYo4J/CGI9eVFs3KiMc0YHuudwKyShTD65jtJy/L9PUUxNAFu4cj4LiJ0UFbw==} '@vue/server-renderer@3.5.24': resolution: {integrity: sha512-Yh2j2Y4G/0/4z/xJ1Bad4mxaAk++C2v4kaa8oSYTMJBJ00/ndPuxCnWeot0/7/qafQFLh5pr6xeV6SdMcE/G1w==} peerDependencies: vue: 3.5.24 - '@vue/server-renderer@3.5.25': - resolution: {integrity: sha512-UJaXR54vMG61i8XNIzTSf2Q7MOqZHpp8+x3XLGtE3+fL+nQd+k7O5+X3D/uWrnQXOdMw5VPih+Uremcw+u1woQ==} - peerDependencies: - vue: 3.5.25 + '@vue/server-renderer@3.5.41': + resolution: {integrity: sha512-n6hx/pNFfbD6SuyeuMVkvqox8bwf/ET9JlA/kAz/imw8sw++wkqKe2mHX5KutjPpbKE4Z56yTHszoOjGMI9igQ==} '@vue/shared@3.5.24': resolution: {integrity: sha512-9cwHL2EsJBdi8NY22pngYYWzkTDhld6fAD6jlaeloNGciNSJL6bLpbxVgXl96X00Jtc6YWQv96YA/0sxex/k1A==} - '@vue/shared@3.5.25': - resolution: {integrity: sha512-AbOPdQQnAnzs58H2FrrDxYj/TJfmeS2jdfEEhgiKINy+bnOANmVizIEgq1r+C5zsbs6l1CCQxtcj71rwNQ4jWg==} + '@vue/shared@3.5.41': + resolution: {integrity: sha512-IOnwSCma8j+9xJT6b8H0dEYidC80NsYmNMlZxRsukYcSoGaDBohog5hDxzeUXdFeGWFA++vWvxqOmrr96VlqMA==} '@vue/test-utils@2.4.6': resolution: {integrity: sha512-FMxEjOpYNYiFe0GkaHsnJPXFHxQ6m4t8vI/ElPGpMWxZKpmRvQ33OIrvRXemy6yha03RxhOlQuy+gZMC3CQSow==} @@ -2632,6 +4248,9 @@ packages: ag-charts-locale@12.1.2: resolution: {integrity: sha512-GFAq4KZit3CHT3Br/LGruas75HqsvV+VZV6qz5hLrmiovJSSVfzPirzf/hUGDTCHqIOS7CdxmpdKUu8pKv1LGw==} + ag-charts-types@10.3.9: + resolution: {integrity: sha512-drcRiJVencliC8LnRwk4MmeQDNNBg5GzmOoLFihO3/k0CUK0VF/N+2nc7iFozwaNG0btSB9vAhYuJLjqHMtRrQ==} + ag-charts-types@12.1.2: resolution: {integrity: sha512-B5IEMzaDxpJRko9sREIBEZOn44tlMX353tMJ1RGYHZrjDFx4on/p6q3YxBvway3nViAM2POmFOS45+1fF9YC8g==} @@ -2902,6 +4521,9 @@ packages: colorspace@1.1.4: resolution: {integrity: sha512-BgvKJiuVu1igBUF2kEjRCZXol6wiiGbY5ipL/oVPwm0BL9sIpMIzM8IK7vwuxIIzOXMV3Ey5w+vxhm0rR/TN8w==} + comlink@4.4.2: + resolution: {integrity: sha512-OxGdvBmJuNKSCMO4NTl1L47VRp6xn2wG4F/2hYzB6tiCb709otOxtEYCSvK80PtjODfXXZu8ds+Nw5kVCjqd2g==} + commander@10.0.1: resolution: {integrity: sha512-y4Mg2tXshplEbSGzx7amzPwKKOCGuoSRP/CjEdwwk0FOGlUbq6lKuoyDZTNZkmxHdJtp54hdfY/JUrdL7Xfdug==} engines: {node: '>=14'} @@ -2967,8 +4589,11 @@ packages: resolution: {integrity: sha512-uV2QOWP2nWzsy2aMp8aRibhi9dlzF5Hgh5SHaB9OiTGEyDTiJJyx0uy51QXdyWbtAHNua4XJzUKca3OzKUd3vA==} engines: {node: '>= 8'} - csstype@3.1.3: - resolution: {integrity: sha512-M1uQkMl8rQK/szD0LNhtqxIPLpimGm8sOBwU7lLnCpSbTyY3yeU1Vc7l4KT5zT4s/yOxHH5O7tIuuLOCnLADRw==} + csstype@3.2.3: + resolution: {integrity: sha512-z1HGKcYy2xA8AGQfwrn0PAy+PB7X/GSj3UVJW9qKyn43xWa+gl5nXmU4qqLMRzWVLFC8KusUX8T/0kCiOYpAIQ==} + + csv-parse@6.2.1: + resolution: {integrity: sha512-LRLMV+UCyfMokp8Wb411duBf1gaBKJfOfBWU9eHMJ+b+cJYZsNu3AFmjJf3+yPGd59Exz1TsMjaSFyxnYB9+IQ==} d3-array@3.2.4: resolution: {integrity: sha512-tdQAmyA18i4J7wprpYq8ClcxZy3SC31QMeByyCFyRt7BVHdREQZ5lpzoe5mFEYZUWe+oq8HBvk9JjpibyEV4Jg==} @@ -2982,14 +4607,50 @@ packages: resolution: {integrity: sha512-zg/chbXyeBtMQ1LbD/WSoW2DpC3I0mpmPdW+ynRTj/x2DAWYrIY7qeZIHidozwV24m4iavr15lNwIwLxRmOxhA==} engines: {node: '>=12'} + d3-dispatch@3.0.1: + resolution: {integrity: sha512-rzUyPU/S7rwUflMyLc1ETDeBj0NRuHKKAcvukozwhshr6g6c5d8zh4c2gQjY2bZ0dXeGLWc1PF174P2tVvKhfg==} + engines: {node: '>=12'} + + d3-drag@3.0.0: + resolution: {integrity: sha512-pWbUJLdETVA8lQNJecMxoXfH6x+mO2UQo8rSmZ+QqxcbyA3hfeprFgIT//HW2nlHChWeIIMwS2Fq+gEARkhTkg==} + engines: {node: '>=12'} + + d3-ease@3.0.1: + resolution: {integrity: sha512-wR/XK3D3XcLIZwpbvQwQ5fK+8Ykds1ip7A2Txe0yxncXSdq1L9skcG7blcedkOX+ZcgxGAmLX1FrRGbADwzi0w==} + engines: {node: '>=12'} + d3-format@3.1.0: resolution: {integrity: sha512-YyUI6AEuY/Wpt8KWLgZHsIU86atmikuoOmCfommt0LYHiQSPjvX2AcFc38PX0CBpr2RCyZhjex+NS/LPOv6YqA==} engines: {node: '>=12'} + d3-hierarchy@3.1.2: + resolution: {integrity: sha512-FX/9frcub54beBdugHjDCdikxThEqjnR93Qt7PvQTOHxyiNCAlvMrHhclk3cD5VeAaq9fxmfRp+CnWw9rEMBuA==} + engines: {node: '>=12'} + d3-interpolate@3.0.1: resolution: {integrity: sha512-3bYs1rOD33uo8aqJfKP3JWPAibgw8Zm2+L9vBKEHJ2Rg+viTR7o5Mmv5mZcieN+FRYaAOWX5SJATX6k1PWz72g==} engines: {node: '>=12'} + d3-path@3.1.0: + resolution: {integrity: sha512-p3KP5HCf/bvjBSSKuXid6Zqijx7wIfNW+J/maPs+iwR35at5JCbLUT0LzF1cnjbCHWhqzQTIN2Jpe8pRebIEFQ==} + engines: {node: '>=12'} + + d3-polygon@3.0.1: + resolution: {integrity: sha512-3vbA7vXYwfe1SYhED++fPUQlWSYTTGmFmQiany/gdbiWgU/iEyQzyymwL9SkJjFFuCS4902BSzewVGsHHmHtXg==} + engines: {node: '>=12'} + + d3-quadtree@3.0.1: + resolution: {integrity: sha512-04xDrxQTDTCFwP5H6hRhsRcb9xxv2RzkcsygFzmkSIOJy3PeRJP7sNk3VRIbKXcog561P9oU0/rVH6vDROAgUw==} + engines: {node: '>=12'} + + d3-random@3.0.1: + resolution: {integrity: sha512-FXMe9GfxTxqd5D6jFsQ+DJ8BJS4E/fT5mqqdjovykEB2oFbTMDVdg1MGFxfQW+FBOGoB++k8swBrgwSHT1cUXQ==} + engines: {node: '>=12'} + + d3-scale-chromatic@3.1.0: + resolution: {integrity: sha512-A3s5PWiZ9YCXFye1o246KoscMWqf8BsD9eRiJ3He7C9OBaxKhAd5TFCdEx/7VbKtxxTsu//1mMJFrEt572cEyQ==} + engines: {node: '>=12'} + d3-scale@4.0.2: resolution: {integrity: sha512-GZW464g1SH7ag3Y7hXjf8RoUuAFIqklOAq3MRl4OaWabTFJY9PN/E1YklhXLh+OQ3fM9yS2nOkCoS+WLZ6kvxQ==} engines: {node: '>=12'} @@ -2998,6 +4659,10 @@ packages: resolution: {integrity: sha512-fmTRWbNMmsmWq6xJV8D19U/gw/bwrHfNXxrIN+HfZgnzqTHp9jOmKMhsTUjXOJnZOdZY9Q28y4yebKzqDKlxlQ==} engines: {node: '>=12'} + d3-shape@3.2.0: + resolution: {integrity: sha512-SaLBuwGm3MOViRq2ABk3eLoxwZELpH6zhl3FbAoJ7Vm1gofKx6El1Ib5z23NUEhF9AsGl7y+dzLe5Cw2AArGTA==} + engines: {node: '>=12'} + d3-time-format@4.1.0: resolution: {integrity: sha512-dJxPBlzC7NugB2PDLwo9Q8JiTR3M3e4/XANkreKSUxF8vvXKqm1Yfq4Q5dl8budlunRVlUUaDUgFt7eA8D6NLg==} engines: {node: '>=12'} @@ -3006,9 +4671,31 @@ packages: resolution: {integrity: sha512-VqKjzBLejbSMT4IgbmVgDjpkYrNWUYJnbCGo874u7MMKIWsILRX+OpX/gTk8MqjpT1A/c6HY2dCA77ZN0lkQ2Q==} engines: {node: '>=12'} + d3-timer@3.0.1: + resolution: {integrity: sha512-ndfJ/JxxMd3nw31uyKoY2naivF+r29V+Lc0svZxe1JvvIRmi8hUsrMvdOwgS1o6uBHmiz91geQ0ylPP0aj1VUA==} + engines: {node: '>=12'} + + d3-transition@3.0.1: + resolution: {integrity: sha512-ApKvfjsSR6tg06xrL434C0WydLr7JewBB3V+/39RMHsaXTOG0zmt/OAXeng5M5LBm0ojmxJrpomQVZ1aPvBL4w==} + engines: {node: '>=12'} + peerDependencies: + d3-selection: 2 - 3 + + d3-zoom@3.0.0: + resolution: {integrity: sha512-b8AmV3kfQaqWAuacbPuNbL6vahnOJflOhexLzMMNLga62+/nh0JzvJ0aO/5a5MVgUFGS7Hu1P9P03o3fJkDCyw==} + engines: {node: '>=12'} + de-indent@1.0.2: resolution: {integrity: sha512-e/1zu3xH5MQryN2zdVaF0OrdNLUbvWxzMbi+iNA6Bky7l1RoP8a2fIbRocyHclXt/arDrrR6lL3TqFD9pMQTsg==} + debug@2.6.9: + resolution: {integrity: sha512-bC7ElrdJaJnPbAP+1EotYvqZsb3ecl5wi6Bfi6BJTUcNowp6cvspg0jXznRTKDjm/E7AdgFBVeAPVMNcKGsHMA==} + peerDependencies: + supports-color: '*' + peerDependenciesMeta: + supports-color: + optional: true + debug@4.4.0: resolution: {integrity: sha512-6WTZ/IxCY/T6BALoZHaE4ctp9xm+Z5kY/pzYaCHRFeyVhojxlrm+46y68HA6hr0TcwEssoxNiDEUJQjfPZ/RYA==} engines: {node: '>=6.0'} @@ -3106,6 +4793,10 @@ packages: resolution: {integrity: sha512-V0hjH4dGPh9Ao5p0MoRY6BVqtwCjhz6vI5LT8AJ55H+4g9/4vbHx1I54fS0XuclLhDHArPQCiMjDxjaL8fPxhw==} engines: {node: '>=0.12'} + entities@7.0.1: + resolution: {integrity: sha512-TWrgLOFUQTH994YUyl1yT4uyavY5nNB5muff+RtWaqNVCAK408b5ZnnbNAUEWLTCpum9w6arT70i1XdQ4UeOPA==} + engines: {node: '>=0.12'} + es-module-lexer@1.7.0: resolution: {integrity: sha512-jEQoCwk8hyb2AZziIOLhDqpm5+2ww5uIE6lkO/6jcOCusfk6LhMHpXXfBLXTZ7Ydyt0j4VoUQv6uGNYbdW+kBA==} @@ -3115,11 +4806,6 @@ packages: es-toolkit@1.42.0: resolution: {integrity: sha512-SLHIyY7VfDJBM8clz4+T2oquwTQxEzu263AyhVK4jREOAwJ+8eebaa4wM3nlvnAqhDrMm2EsA6hWHaQsMPQ1nA==} - esbuild@0.25.12: - resolution: {integrity: sha512-bbPBYYrtZbkt6Os6FiTLCTFxvq4tt3JKall1vRwshA3fdVztsLAatFaZobhkBC8/BrPetoa0oksYoKXoG4ryJg==} - engines: {node: '>=18'} - hasBin: true - escalade@3.2.0: resolution: {integrity: sha512-WUj2qlxaQtO4g6Pq5c29GTcWGDyd8itL8zTlipgECz3JesAiiOKotd8JU6otB3PACgG6xkJUyVhboMS+bje/jA==} engines: {node: '>=6'} @@ -3476,6 +5162,9 @@ packages: jsonfile@6.2.0: resolution: {integrity: sha512-FGuPw30AdOIUTRMC2OMRtQV+jkVj2cfPqSeWXv1NEAJ1qZ5zb1X6z1mFhbfOB/iy3ssJCD+3KuZ8r8C3uVFlAg==} + kdbush@4.1.0: + resolution: {integrity: sha512-e9vurzrXJQrFX6ckpHP3bvj5l+9CnYzkxDNnNQ1h2QTqdWsUAJgXiKdGNcOa1EY85dU8KbQ+z/FdQdB7P+9yfQ==} + kolorist@1.8.0: resolution: {integrity: sha512-Y+60/zizpJ3HRH8DCss+q95yr6145JXZo46OTpFvDZWLfRCE4qChOyk1b26nMaNpfHHgxagk9dXT5OP0Tfe+dQ==} @@ -3673,6 +5362,9 @@ packages: resolution: {integrity: sha512-tzzskb3bG8LvYGFF/mDTpq3jpI6Q9wc3LEmBaghu+DdCssd1FakN7Bc0hVNmEyGq1bq3RgfkCb3cmQLpNPOroA==} engines: {node: '>=4'} + ms@2.0.0: + resolution: {integrity: sha512-Tpp60P6IUJDTuOq/5Z8cdskzJujfwqfOTkrwIwj7IRISpnkJnT6SyJ4PCPnGMoFjC9ddhal5KVIYtAt97ix05A==} + ms@2.1.3: resolution: {integrity: sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==} @@ -3691,6 +5383,11 @@ packages: engines: {node: ^10 || ^12 || ^13.7 || ^14 || >=15.0.1} hasBin: true + nanoid@3.3.18: + resolution: {integrity: sha512-DTg4MJbGMWkfi6VZFdNt2/caMbQy4Ou+Op/hJQvGEWcnVfoA1QA+xzRKAzw9jD6+GVOOeYr/mIcuDSdug6F6+w==} + engines: {node: ^10 || ^12 || ^13.7 || ^14 || >=15.0.1} + hasBin: true + napi-build-utils@2.0.0: resolution: {integrity: sha512-GEbrYkbfF7MoNaoh2iGG84Mnf/WZfB0GdGEsM8wz7Expx/LlWf5U8t9nvJKXSp3qr5IsEbK04cBGhol/KwOsWA==} @@ -3735,9 +5432,6 @@ packages: resolution: {integrity: sha512-rJgTQnkUnH1sFw8yT6VSU3zD3sWmu6sZhIseY8VX+GRu3P6F7Fu+JNDoXfklElbLJSnc3FUQHVe4cU5hj+BcUg==} engines: {node: '>=0.10.0'} - obug@2.1.1: - resolution: {integrity: sha512-uTqF9MuPraAQ+IsnPf366RG4cP9RtUi7MLO1N3KEc+wb0a6yKpeL0lmk2IB1jY5KHPAlTc6T/JRdC/YqxHNwkQ==} - obug@2.1.3: resolution: {integrity: sha512-9miFgM2OFba7hB+pRgvtV84pYTBaoTHohvmIgiRt6dRIzbwEOIaNaP+dIlGs2fNFoB0SeISs0Jz5WFVRid6Xyg==} engines: {node: '>=12.20.0'} @@ -3859,10 +5553,6 @@ packages: resolution: {integrity: sha512-JU3teHTNjmE2VCGFzuY8EXzCDVwEqB2a8fsIvwaStHhAWJEeVd1o1QD80CU6+ZdEXXSLbSsuLwJjkCBWqRQUVA==} engines: {node: '>=8.6'} - picomatch@4.0.3: - resolution: {integrity: sha512-5gTmgEY/sqK6gFXLIsQNH19lWb4ebPDLA4SdLP7dsWkIXHWlG66oPuVvXSGFPppYZz8ZDZq0dYYrbHfBCVUb1Q==} - engines: {node: '>=12'} - picomatch@4.0.4: resolution: {integrity: sha512-QP88BAKvMam/3NxH6vj2o21R6MjxZUAd6nlwAS/pnGvN9IVLocLHxGYIzFhg6fUQ+5th6P4dv4eW9jX3DSIj7A==} engines: {node: '>=12'} @@ -3901,6 +5591,10 @@ packages: resolution: {integrity: sha512-pMMHxBOZKFU6HgAZ4eyGnwXF/EvPGGqUr0MnZ5+99485wwW41kW91A4LOGxSHhgugZmSChL5AlElNdwlNgcnLQ==} engines: {node: ^10 || ^12 || >=14} + postcss@8.5.26: + resolution: {integrity: sha512-u82N74LFzG8ca+dD8puPnplTXoGH4fTPpVGuIbt36G3qvNlkvfD0lEAZSxaly3KX8TS/L1A1gsCEmvKmBcVbkQ==} + engines: {node: ^10 || ^12 || >=14} + prebuild-install@7.1.3: resolution: {integrity: sha512-8Mf2cbV7x1cXPUILADGI3wuhfqWvtiLA1iclTDbFRZkgRQS0NqsPZphna9V+HyTEadheuPmjaJMsbzKQFOzLug==} engines: {node: '>=10'} @@ -3922,8 +5616,8 @@ packages: proto-list@1.2.4: resolution: {integrity: sha512-vtK/94akxsTMhe0/cbfpR+syPuszcuwhqVjJq26CuNDgFGj682oRBXOP5MJpv2r7JtE8MsiepGIqvvOTBwn2vA==} - protobufjs@7.4.0: - resolution: {integrity: sha512-mRUWCc3KUU4w1jU8sGxICXH/gNS94DvI1gxqDvBzhj1JpcsimQkYiOJfwsPUykUI5ZaspFbSgmBLER8IrQ3tqw==} + protobufjs@7.6.5: + resolution: {integrity: sha512-/FPD0nUc9jH6rfFjji9IBqOz4pcSE3CsT1m7Ep6Mdb0LxSUMj8hgl6GomOvZzpNpAqqGaXA0P3VSrZLFzIhQrw==} engines: {node: '>=12.0.0'} pump@3.0.2: @@ -3945,17 +5639,32 @@ packages: queue-tick@1.0.1: resolution: {integrity: sha512-kJt5qhMxoszgU/62PLP1CJytzd2NKetjSRnyuj31fDd3Rlcz3fzlFdFLD1SItunPwyqEOkca6GbV612BWfaBag==} - quickjs-emscripten-core@0.31.0: - resolution: {integrity: sha512-oQz8p0SiKDBc1TC7ZBK2fr0GoSHZKA0jZIeXxsnCyCs4y32FStzCW4d1h6E1sE0uHDMbGITbk2zhNaytaoJwXQ==} + quickjs-emscripten-core@0.32.0: + resolution: {integrity: sha512-QFnPfjFey8EqknSrSxe1hZrf1/8z7/6s1QzGOmKo6++02r7QRRX7ZoyNaZh7JuVjWsVW87KnQrbZqnHkOAzUyg==} - quickjs-emscripten@0.31.0: - resolution: {integrity: sha512-K7Yt78aRPLjPcqv3fIuLW1jW3pvwO21B9pmFOolsjM/57ZhdVXBr51GqJpalgBlkPu9foAvhEAuuQPnvIGvLvQ==} + quickjs-emscripten@0.32.0: + resolution: {integrity: sha512-So0Sqw869y/S2oE3Nuc0uT3Dhqgvsj8FSrwBdsuTosVsG8ME5/OcudU1GxsrIFdFABgy17GHnTVO9TYV/bLQcA==} engines: {node: '>=16.0.0'} + quickselect@3.0.0: + resolution: {integrity: sha512-XdjUArbK4Bm5fLLvlm5KpTFOiOThgfWWI4axAZDWg4E/0mKdZyI9tNEfds27qCi1ze/vwTR16kvmmGhRra3c2g==} + + rbush@4.0.1: + resolution: {integrity: sha512-IP0UpfeWQujYC8Jg162rMNc01Rf0gWMMAb2Uxus/Q0qOFw4lCcq6ZnQEZwUoJqWyUGJ9th7JjwI4yIWo+uvoAQ==} + rc@1.2.8: resolution: {integrity: sha512-y3bGgqKj3QBdxLbLkomlohkvsA8gdAiUQlSBJnBhfn+BPxg4bc62d8TcBW15wavDfgexCgccckhcZvywyQYPOw==} hasBin: true + react-dom@19.2.8: + resolution: {integrity: sha512-rVprimfGBG3DR+Tq0IQG2DT5PxKth1WIGDmj5yPmlzr4YBe7uyE+Du4oVqTDXZSHGGGXRtTJEGSSePyQCMBglQ==} + peerDependencies: + react: ^19.2.8 + + react@19.2.8: + resolution: {integrity: sha512-PWaYA1L/q9u2u7xYQi+Y3L3Yfnie7XyLeaJICV1MGD6LprsBxcAqGjYyr0eY3p+QdsA+x/Irkt4Qif8D63+Sbw==} + engines: {node: '>=0.10.0'} + read-yaml-file@1.1.0: resolution: {integrity: sha512-VIMnQi/Z4HT2Fxuwg5KrY174U1VdUIASQVWXXyqtNRtxSr9IYkn1rsI6Tb6HsrHCmB7gVpNwX6JxPTHcH6IoTA==} engines: {node: '>=6'} @@ -4079,6 +5788,9 @@ packages: safer-buffer@2.1.2: resolution: {integrity: sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg==} + scheduler@0.27.0: + resolution: {integrity: sha512-eNv+WrVbKu1f3vbYJT/xtiF5syA5HPIMtf9IgY/nKg0sWqzAUEvqY/xm7OcZc/qafLx/iO9FgOmeSAp4v5ti/Q==} + seek-bzip@1.0.6: resolution: {integrity: sha512-e1QtP3YL5tWww8uKaOCQ18UxIT2laNBXHjV/S2WYCiK4udiv8lkG89KRIoCjUagnAmCBurjF4zEVX2ByBbnCjQ==} hasBin: true @@ -4186,9 +5898,6 @@ packages: stackback@0.0.2: resolution: {integrity: sha512-1XMJE5fQo1jGH6Y/7ebnwPOBEkIEnT4QF32d5R1+VXdXveM0IBMJt8zfaxX1P3QhVwrYe+576+jkANtSS2mBbw==} - std-env@3.10.0: - resolution: {integrity: sha512-5GS12FdOZNliM5mAOxFRg7Ir0pWz8MdpYm6AY6VPkGpbA7ZzmbzNcBJQ0GPvvyWgcY7QAhCgf9Uy89I03faLkg==} - std-env@4.1.0: resolution: {integrity: sha512-Rq7ybcX2RuC55r9oaPVEW7/xu3tj8u4GeBYHBWCychFtzMIr86A7e3PPEBPT37sHStKX3+TiX/Fr/ACmJLVlLQ==} @@ -4339,47 +6048,17 @@ packages: tunnel-agent@0.6.0: resolution: {integrity: sha512-McnNiV1l8RYeY8tBgEpuodCC1mLUdbSN+CYBL7kJsJNInOP8UjDDEwdk6Mw60vdLLrr5NHKZhMAOSrR2NZuQ+w==} - turbo-darwin-64@2.8.11: - resolution: {integrity: sha512-XKaCWaz4OCt77oYYvGCIRpvYD4c/aNaKjRkUpv+e8rN3RZb+5Xsyew4yRO+gaHdMIUhQznXNXfHlhs+/p7lIhA==} - cpu: [x64] - os: [darwin] - - turbo-darwin-arm64@2.8.11: - resolution: {integrity: sha512-VvynLHGUNvQ9k7GZjRPSsRcK4VkioTfFb7O7liAk4nHKjEcMdls7GqxzjVWgJiKz3hWmQGaP9hRa9UUnhVWCxA==} - cpu: [arm64] - os: [darwin] + turbo@2.10.10: + resolution: {integrity: sha512-/90KTW+USzvYOPmafRZHVKLBsHXQ5810Ao/HdtJYAqguIhZ+XruS6eIUjqJUDtrSxaZYynNFht68qckGKAOWTA==} + hasBin: true - turbo-linux-64@2.8.11: - resolution: {integrity: sha512-cbSn37dcm+EmkQ7DD0euy7xV7o2el4GAOr1XujvkAyKjjNvQ+6QIUeDgQcwAx3D17zPpDvfDMJY2dLQadWnkmQ==} - cpu: [x64] - os: [linux] + tweetnacl@0.14.5: + resolution: {integrity: sha512-KXXFFdAbFXY4geFIwoyNK+f5Z1b7swfXABfL7HXCmoIWMKU3dmS26672A4EeQtDzLKy7SXmfBu51JolvEKwtGA==} - turbo-linux-arm64@2.8.11: - resolution: {integrity: sha512-+trymp2s2aBrhS04l6qFxcExzZ8ffndevuUB9c5RCeqsVpZeiWuGQlWNm5XjOmzoMayxRARZ5ma7yiWbGMiLqQ==} - cpu: [arm64] - os: [linux] - - turbo-windows-64@2.8.11: - resolution: {integrity: sha512-3kJjFSM4yw1n9Uzmi+XkAUgCae19l/bH6RJ442xo7mnZm0tpOjo33F+FYHoSVpIWVMd0HG0LDccyafPSdylQbA==} - cpu: [x64] - os: [win32] - - turbo-windows-arm64@2.8.11: - resolution: {integrity: sha512-JOM4uF2vuLsJUvibdR6X9QqdZr6BhC6Nhlrw4LKFPsXZZI/9HHLoqAiYRpE4MuzIwldCH/jVySnWXrI1SKto0g==} - cpu: [arm64] - os: [win32] - - turbo@2.8.11: - resolution: {integrity: sha512-H+rwSHHPLoyPOSoHdmI1zY0zy0GGj1Dmr7SeJW+nZiWLz2nex8EJ+fkdVabxXFMNEux+aywI4Sae8EqhmnOv4A==} - hasBin: true - - tweetnacl@0.14.5: - resolution: {integrity: sha512-KXXFFdAbFXY4geFIwoyNK+f5Z1b7swfXABfL7HXCmoIWMKU3dmS26672A4EeQtDzLKy7SXmfBu51JolvEKwtGA==} - - typescript@3.9.10: - resolution: {integrity: sha512-w6fIxVE/H1PkLKcCPsFqKE7Kv7QUwhU8qQY2MueZXWx5cPZdwFupLgKK3vntcK98BtNHZtAF4LA/yl2a7k8R6Q==} - engines: {node: '>=4.2.0'} - hasBin: true + typescript@3.9.10: + resolution: {integrity: sha512-w6fIxVE/H1PkLKcCPsFqKE7Kv7QUwhU8qQY2MueZXWx5cPZdwFupLgKK3vntcK98BtNHZtAF4LA/yl2a7k8R6Q==} + engines: {node: '>=4.2.0'} + hasBin: true typescript@5.4.5: resolution: {integrity: sha512-vcI4UpRgg81oIRUFwR0WSIHKt11nJ7SAVlYNIu+QpqeyXP+gpQJy/Z4+F0aGxSE4MqwjyXvW/TzgkLAx2AGHwQ==} @@ -4476,46 +6155,6 @@ packages: peerDependencies: vite: '*' - vite@7.2.7: - resolution: {integrity: sha512-ITcnkFeR3+fI8P1wMgItjGrR10170d8auB4EpMLPqmx6uxElH3a/hHGQabSHKdqd4FXWO1nFIp9rRn7JQ34ACQ==} - engines: {node: ^20.19.0 || >=22.12.0} - hasBin: true - peerDependencies: - '@types/node': ^20.19.0 || >=22.12.0 - jiti: '>=1.21.0' - less: ^4.0.0 - lightningcss: ^1.21.0 - sass: ^1.70.0 - sass-embedded: ^1.70.0 - stylus: '>=0.54.8' - sugarss: ^5.0.0 - terser: ^5.16.0 - tsx: ^4.8.1 - yaml: ^2.4.2 - peerDependenciesMeta: - '@types/node': - optional: true - jiti: - optional: true - less: - optional: true - lightningcss: - optional: true - sass: - optional: true - sass-embedded: - optional: true - stylus: - optional: true - sugarss: - optional: true - terser: - optional: true - tsx: - optional: true - yaml: - optional: true - vite@8.0.8: resolution: {integrity: sha512-dbU7/iLVa8KZALJyLOBOQ88nOXtNG8vxKuOT4I2mD+Ya70KPceF4IAmDsmU0h1Qsn5bPrvsY9HJstCRh3hG6Uw==} engines: {node: ^20.19.0 || >=22.12.0} @@ -4559,54 +6198,20 @@ packages: yaml: optional: true - vitest@4.0.18: - resolution: {integrity: sha512-hOQuK7h0FGKgBAas7v0mSAsnvrIgAvWmRFjmzpJ7SwFHH3g1k2u37JtYwOwmEKhK6ZO3v9ggDBBm0La1LCK4uQ==} - engines: {node: ^20.0.0 || ^22.0.0 || >=24.0.0} - hasBin: true - peerDependencies: - '@edge-runtime/vm': '*' - '@opentelemetry/api': ^1.9.0 - '@types/node': ^20.0.0 || ^22.0.0 || >=24.0.0 - '@vitest/browser-playwright': 4.0.18 - '@vitest/browser-preview': 4.0.18 - '@vitest/browser-webdriverio': 4.0.18 - '@vitest/ui': 4.0.18 - happy-dom: '*' - jsdom: '*' - peerDependenciesMeta: - '@edge-runtime/vm': - optional: true - '@opentelemetry/api': - optional: true - '@types/node': - optional: true - '@vitest/browser-playwright': - optional: true - '@vitest/browser-preview': - optional: true - '@vitest/browser-webdriverio': - optional: true - '@vitest/ui': - optional: true - happy-dom: - optional: true - jsdom: - optional: true - - vitest@4.1.4: - resolution: {integrity: sha512-tFuJqTxKb8AvfyqMfnavXdzfy3h3sWZRWwfluGbkeR7n0HUev+FmNgZ8SDrRBTVrVCjgH5cA21qGbCffMNtWvg==} + vitest@4.1.10: + resolution: {integrity: sha512-R9jUTe5S4Qb0HCd4TNqpC7oGcrMssMRGXLW80ubjWsW9VH5GF8y1Y0SFLY9AbqSk6nt0PnOx4H4WNJYZ13GUPw==} engines: {node: ^20.0.0 || ^22.0.0 || >=24.0.0} hasBin: true peerDependencies: '@edge-runtime/vm': '*' '@opentelemetry/api': ^1.9.0 '@types/node': ^20.0.0 || ^22.0.0 || >=24.0.0 - '@vitest/browser-playwright': 4.1.4 - '@vitest/browser-preview': 4.1.4 - '@vitest/browser-webdriverio': 4.1.4 - '@vitest/coverage-istanbul': 4.1.4 - '@vitest/coverage-v8': 4.1.4 - '@vitest/ui': 4.1.4 + '@vitest/browser-playwright': 4.1.10 + '@vitest/browser-preview': 4.1.10 + '@vitest/browser-webdriverio': 4.1.10 + '@vitest/coverage-istanbul': 4.1.10 + '@vitest/coverage-v8': 4.1.10 + '@vitest/ui': 4.1.10 happy-dom: '*' jsdom: '*' vite: ^6.0.0 || ^7.0.0 || ^8.0.0 @@ -4654,8 +6259,8 @@ packages: typescript: optional: true - vue@3.5.25: - resolution: {integrity: sha512-YLVdgv2K13WJ6n+kD5owehKtEXwdwXuj2TTyJMsO7pSeKw2bfRNZGjhB7YzrpbMYj5b5QsUebHpOqR3R3ziy/g==} + vue@3.5.41: + resolution: {integrity: sha512-2laE0p+aK+/AOPG/XL/WepOs/GlK755LJ1XECi9kDUrz1FKNw8rb2Xzlw9JS1rqEV55nb0ttsKxVlTCcd+R5cg==} peerDependencies: typescript: '*' peerDependenciesMeta: @@ -4755,6 +6360,11 @@ packages: snapshots: + '@ag-grid-community/core@32.3.9': + dependencies: + ag-charts-types: 10.3.9 + tslib: 2.8.1 + '@ast-grep/napi-darwin-arm64@0.36.3': optional: true @@ -5388,10 +6998,14 @@ snapshots: '@babel/helper-string-parser@7.27.1': {} + '@babel/helper-string-parser@7.29.7': {} + '@babel/helper-string-parser@8.0.0': {} '@babel/helper-validator-identifier@7.28.5': {} + '@babel/helper-validator-identifier@7.29.7': {} + '@babel/helper-validator-identifier@8.0.2': {} '@babel/helper-validator-option@7.27.1': {} @@ -5405,6 +7019,10 @@ snapshots: dependencies: '@babel/types': 7.29.0 + '@babel/parser@7.29.8': + dependencies: + '@babel/types': 7.29.8 + '@babel/parser@8.0.0': dependencies: '@babel/types': 8.0.0 @@ -5436,6 +7054,11 @@ snapshots: '@babel/helper-string-parser': 7.27.1 '@babel/helper-validator-identifier': 7.28.5 + '@babel/types@7.29.8': + dependencies: + '@babel/helper-string-parser': 7.29.7 + '@babel/helper-validator-identifier': 7.29.7 + '@babel/types@8.0.0': dependencies: '@babel/helper-string-parser': 8.0.0 @@ -5599,6 +7222,67 @@ snapshots: '@colors/colors@1.6.0': {} + '@d3fc/d3fc-axis@3.0.7(d3-scale@4.0.2)(d3-selection@3.0.0)(d3-shape@3.2.0)': + dependencies: + '@d3fc/d3fc-data-join': 6.0.3(d3-selection@3.0.0) + '@d3fc/d3fc-rebind': 6.0.1 + d3-scale: 4.0.2 + d3-selection: 3.0.0 + d3-shape: 3.2.0 + + '@d3fc/d3fc-chart@5.1.9(d3-array@3.2.4)(d3-path@3.1.0)(d3-scale-chromatic@3.1.0)(d3-scale@4.0.2)(d3-selection@3.0.0)(d3-shape@3.2.0)': + dependencies: + '@d3fc/d3fc-axis': 3.0.7(d3-scale@4.0.2)(d3-selection@3.0.0)(d3-shape@3.2.0) + '@d3fc/d3fc-data-join': 6.0.3(d3-selection@3.0.0) + '@d3fc/d3fc-element': 6.2.0 + '@d3fc/d3fc-rebind': 6.0.1 + '@d3fc/d3fc-series': 6.1.3(d3-array@3.2.4)(d3-path@3.1.0)(d3-scale-chromatic@3.1.0)(d3-scale@4.0.2)(d3-selection@3.0.0)(d3-shape@3.2.0) + d3-scale: 4.0.2 + d3-selection: 3.0.0 + transitivePeerDependencies: + - d3-array + - d3-path + - d3-scale-chromatic + - d3-shape + + '@d3fc/d3fc-data-join@6.0.3(d3-selection@3.0.0)': + dependencies: + d3-selection: 3.0.0 + + '@d3fc/d3fc-element@6.2.0': {} + + '@d3fc/d3fc-pointer@3.0.3(d3-dispatch@3.0.1)(d3-selection@3.0.0)': + dependencies: + '@d3fc/d3fc-rebind': 6.0.1 + d3-dispatch: 3.0.1 + d3-selection: 3.0.0 + + '@d3fc/d3fc-rebind@6.0.1': {} + + '@d3fc/d3fc-series@6.1.3(d3-array@3.2.4)(d3-path@3.1.0)(d3-scale-chromatic@3.1.0)(d3-scale@4.0.2)(d3-selection@3.0.0)(d3-shape@3.2.0)': + dependencies: + '@d3fc/d3fc-data-join': 6.0.3(d3-selection@3.0.0) + '@d3fc/d3fc-rebind': 6.0.1 + '@d3fc/d3fc-shape': 6.0.1(d3-path@3.1.0) + '@d3fc/d3fc-webgl': 3.2.1(d3-scale@4.0.2)(d3-shape@3.2.0) + d3-array: 3.2.4 + d3-scale: 4.0.2 + d3-scale-chromatic: 3.1.0 + d3-selection: 3.0.0 + d3-shape: 3.2.0 + transitivePeerDependencies: + - d3-path + + '@d3fc/d3fc-shape@6.0.1(d3-path@3.1.0)': + dependencies: + d3-path: 3.1.0 + + '@d3fc/d3fc-webgl@3.2.1(d3-scale@4.0.2)(d3-shape@3.2.0)': + dependencies: + '@d3fc/d3fc-rebind': 6.0.1 + d3-scale: 4.0.2 + d3-shape: 3.2.0 + '@dabh/diagnostics@2.0.3': dependencies: colorspace: 1.1.4 @@ -5637,94 +7321,16 @@ snapshots: tslib: 2.8.1 optional: true - '@esbuild/aix-ppc64@0.25.12': - optional: true - - '@esbuild/android-arm64@0.25.12': - optional: true - - '@esbuild/android-arm@0.25.12': - optional: true - - '@esbuild/android-x64@0.25.12': - optional: true - - '@esbuild/darwin-arm64@0.25.12': - optional: true - - '@esbuild/darwin-x64@0.25.12': - optional: true - - '@esbuild/freebsd-arm64@0.25.12': - optional: true - - '@esbuild/freebsd-x64@0.25.12': - optional: true - - '@esbuild/linux-arm64@0.25.12': - optional: true - - '@esbuild/linux-arm@0.25.12': - optional: true - - '@esbuild/linux-ia32@0.25.12': - optional: true - - '@esbuild/linux-loong64@0.25.12': - optional: true - - '@esbuild/linux-mips64el@0.25.12': - optional: true - - '@esbuild/linux-ppc64@0.25.12': - optional: true - - '@esbuild/linux-riscv64@0.25.12': - optional: true - - '@esbuild/linux-s390x@0.25.12': - optional: true - - '@esbuild/linux-x64@0.25.12': - optional: true - - '@esbuild/netbsd-arm64@0.25.12': - optional: true - - '@esbuild/netbsd-x64@0.25.12': - optional: true - - '@esbuild/openbsd-arm64@0.25.12': - optional: true - - '@esbuild/openbsd-x64@0.25.12': - optional: true - - '@esbuild/openharmony-arm64@0.25.12': - optional: true - - '@esbuild/sunos-x64@0.25.12': - optional: true - - '@esbuild/win32-arm64@0.25.12': - optional: true - - '@esbuild/win32-ia32@0.25.12': - optional: true - - '@esbuild/win32-x64@0.25.12': - optional: true - - '@grpc/grpc-js@1.13.4': + '@grpc/grpc-js@1.14.4': dependencies: - '@grpc/proto-loader': 0.7.13 + '@grpc/proto-loader': 0.8.1 '@js-sdsl/ordered-map': 4.4.2 - '@grpc/proto-loader@0.7.13': + '@grpc/proto-loader@0.8.1': dependencies: lodash.camelcase: 4.3.0 long: 5.3.2 - protobufjs: 7.4.0 + protobufjs: 7.6.5 yargs: 17.7.2 '@inquirer/ansi@1.0.2': {} @@ -5867,23 +7473,23 @@ snapshots: '@istanbuljs/schema@0.1.3': {} - '@jitl/quickjs-ffi-types@0.31.0': {} + '@jitl/quickjs-ffi-types@0.32.0': {} - '@jitl/quickjs-wasmfile-debug-asyncify@0.31.0': + '@jitl/quickjs-wasmfile-debug-asyncify@0.32.0': dependencies: - '@jitl/quickjs-ffi-types': 0.31.0 + '@jitl/quickjs-ffi-types': 0.32.0 - '@jitl/quickjs-wasmfile-debug-sync@0.31.0': + '@jitl/quickjs-wasmfile-debug-sync@0.32.0': dependencies: - '@jitl/quickjs-ffi-types': 0.31.0 + '@jitl/quickjs-ffi-types': 0.32.0 - '@jitl/quickjs-wasmfile-release-asyncify@0.31.0': + '@jitl/quickjs-wasmfile-release-asyncify@0.32.0': dependencies: - '@jitl/quickjs-ffi-types': 0.31.0 + '@jitl/quickjs-ffi-types': 0.32.0 - '@jitl/quickjs-wasmfile-release-sync@0.31.0': + '@jitl/quickjs-wasmfile-release-sync@0.32.0': dependencies: - '@jitl/quickjs-ffi-types': 0.31.0 + '@jitl/quickjs-ffi-types': 0.32.0 '@jridgewell/gen-mapping@0.3.13': dependencies: @@ -5971,10 +7577,10 @@ snapshots: '@microsoft/tsdoc@0.16.0': {} - '@milaboratories/columns-collection-driver@0.2.2': + '@milaboratories/columns-collection-driver@0.2.3': dependencies: '@milaboratories/helpers': 1.14.5 - '@milaboratories/pl-model-common': 1.47.2 + '@milaboratories/pl-model-common': 1.47.3 '@milaboratories/computable@2.9.8': dependencies: @@ -5983,17 +7589,79 @@ snapshots: '@types/node': 24.5.2 utility-types: 3.11.0 + '@milaboratories/graph-maker@1.4.8(@milaboratories/pl-model-common@1.48.0)(@platforma-sdk/model@1.81.1)(@platforma-sdk/ui-vue@1.81.1(@bytecodealliance/preview2-shim@0.17.8)(typescript@5.6.3))(d3-dispatch@3.0.1)(d3-path@3.1.0)(d3-scale-chromatic@3.1.0)(typescript@5.6.3)': + dependencies: + '@ag-grid-community/core': 32.3.9 + '@milaboratories/helpers': 1.14.5 + '@milaboratories/miplots4': 1.2.3(d3-dispatch@3.0.1)(d3-path@3.1.0)(d3-scale-chromatic@3.1.0) + '@milaboratories/pf-plots': 1.4.6(@milaboratories/pl-model-common@1.48.0)(@platforma-sdk/model@1.81.1) + '@platforma-sdk/model': 1.81.1 + '@platforma-sdk/ui-vue': 1.81.1(@bytecodealliance/preview2-shim@0.17.8)(typescript@5.6.3) + '@types/d3-hierarchy': 3.1.7 + '@types/d3-scale': 4.0.9 + '@vueuse/core': 13.8.0(vue@3.5.41(typescript@5.6.3)) + ag-grid-vue3: 34.1.2(vue@3.5.41(typescript@5.6.3)) + canonicalize: 2.1.0 + d3-hierarchy: 3.1.2 + d3-scale: 4.0.2 + vue: 3.5.41(typescript@5.6.3) + transitivePeerDependencies: + - '@milaboratories/pl-model-common' + - d3-dispatch + - d3-path + - d3-scale-chromatic + - supports-color + - typescript + '@milaboratories/helpers@1.14.2': {} '@milaboratories/helpers@1.14.5': {} - '@milaboratories/pf-driver@1.8.4(@bytecodealliance/preview2-shim@0.17.8)': + '@milaboratories/miplots4@1.2.3(d3-dispatch@3.0.1)(d3-path@3.1.0)(d3-scale-chromatic@3.1.0)': + dependencies: + '@d3fc/d3fc-chart': 5.1.9(d3-array@3.2.4)(d3-path@3.1.0)(d3-scale-chromatic@3.1.0)(d3-scale@4.0.2)(d3-selection@3.0.0)(d3-shape@3.2.0) + '@d3fc/d3fc-pointer': 3.0.3(d3-dispatch@3.0.1)(d3-selection@3.0.0) + '@d3fc/d3fc-series': 6.1.3(d3-array@3.2.4)(d3-path@3.1.0)(d3-scale-chromatic@3.1.0)(d3-scale@4.0.2)(d3-selection@3.0.0)(d3-shape@3.2.0) + '@d3fc/d3fc-webgl': 3.2.1(d3-scale@4.0.2)(d3-shape@3.2.0) + '@stdlib/stats-anova1': 0.2.3 + '@stdlib/stats-base-dists-f-cdf': 0.2.3 + '@stdlib/stats-kruskal-test': 0.2.3 + '@stdlib/stats-ttest': 0.2.3 + '@stdlib/stats-ttest2': 0.2.3 + '@stdlib/stats-wilcoxon': 0.2.3 + comlink: 4.4.2 + d3-array: 3.2.4 + d3-axis: 3.0.0 + d3-color: 3.1.0 + d3-drag: 3.0.0 + d3-format: 3.1.0 + d3-hierarchy: 3.1.2 + d3-polygon: 3.0.1 + d3-quadtree: 3.0.1 + d3-random: 3.0.1 + d3-scale: 4.0.2 + d3-selection: 3.0.0 + d3-shape: 3.2.0 + d3-zoom: 3.0.0 + kdbush: 4.1.0 + lodash: 4.17.23 + rbush: 4.0.1 + react: 19.2.8 + react-dom: 19.2.8(react@19.2.8) + zod: 3.25.76 + transitivePeerDependencies: + - d3-dispatch + - d3-path + - d3-scale-chromatic + - supports-color + + '@milaboratories/pf-driver@1.9.0(@bytecodealliance/preview2-shim@0.17.8)': dependencies: '@milaboratories/helpers': 1.14.5 + '@milaboratories/pf-spec': 1.0.0(@bytecodealliance/preview2-shim@0.17.8) '@milaboratories/pframes-rs-node': 1.1.56 - '@milaboratories/pframes-rs-wasm': 1.1.56(@bytecodealliance/preview2-shim@0.17.8)(@milaboratories/pl-model-common@1.47.2)(@milaboratories/pl-model-middle-layer@1.30.14) - '@milaboratories/pl-model-common': 1.47.2 - '@milaboratories/pl-model-middle-layer': 1.30.14 + '@milaboratories/pl-model-common': 1.47.3 + '@milaboratories/pl-model-middle-layer': 1.31.0 '@milaboratories/ts-helpers': 1.8.6 es-toolkit: 1.42.0 lru-cache: 11.2.4 @@ -6002,16 +7670,29 @@ snapshots: - encoding - supports-color - '@milaboratories/pf-spec-driver@1.4.23(@bytecodealliance/preview2-shim@0.17.8)': + '@milaboratories/pf-plots@1.4.6(@milaboratories/pl-model-common@1.48.0)(@platforma-sdk/model@1.81.1)': + dependencies: + '@milaboratories/helpers': 1.14.5 + '@milaboratories/pl-model-common': 1.48.0 + '@platforma-sdk/model': 1.81.1 + canonicalize: 2.1.0 + lodash: 4.17.23 + + '@milaboratories/pf-spec-driver@1.5.0(@bytecodealliance/preview2-shim@0.17.8)': dependencies: '@milaboratories/helpers': 1.14.5 - '@milaboratories/pframes-rs-wasm': 1.1.56(@bytecodealliance/preview2-shim@0.17.8)(@milaboratories/pl-model-common@1.47.2)(@milaboratories/pl-model-middle-layer@1.30.14) - '@milaboratories/pl-model-common': 1.47.2 - '@milaboratories/pl-model-middle-layer': 1.30.14 + '@milaboratories/pf-spec': 1.0.0(@bytecodealliance/preview2-shim@0.17.8) + '@milaboratories/pl-model-common': 1.47.3 '@noble/hashes': 2.2.0 transitivePeerDependencies: - '@bytecodealliance/preview2-shim' + '@milaboratories/pf-spec@1.0.0(@bytecodealliance/preview2-shim@0.17.8)': + dependencies: + '@bytecodealliance/preview2-shim': 0.17.8 + '@milaboratories/pl-model-common': 1.47.3 + '@milaboratories/pl-model-middle-layer': 1.31.0 + '@milaboratories/pframes-rs-node@1.1.56': dependencies: '@mapbox/node-pre-gyp': 2.0.3 @@ -6034,20 +7715,31 @@ snapshots: '@milaboratories/pframes-rs-wasip2@1.1.56': {} - '@milaboratories/pframes-rs-wasm@1.1.56(@bytecodealliance/preview2-shim@0.17.8)(@milaboratories/pl-model-common@1.47.2)(@milaboratories/pl-model-middle-layer@1.30.14)': + '@milaboratories/pl-client@3.14.6': dependencies: - '@bytecodealliance/preview2-shim': 0.17.8 - '@milaboratories/pframes-rs-wasip2': 1.1.56 - '@milaboratories/pl-model-common': 1.47.2 - '@milaboratories/pl-model-middle-layer': 1.30.14 + '@grpc/grpc-js': 1.14.4 + '@milaboratories/pl-http': 1.2.4 + '@milaboratories/pl-model-common': 1.47.3 + '@milaboratories/ts-helpers': 1.8.6 + '@protobuf-ts/grpc-transport': 2.11.1(@grpc/grpc-js@1.14.4) + '@protobuf-ts/runtime': 2.11.1 + '@protobuf-ts/runtime-rpc': 2.11.1 + canonicalize: 2.1.0 + denque: 2.1.0 + long: 5.3.2 + lru-cache: 11.2.4 + openapi-fetch: 0.15.0 + undici: 7.16.0 + utility-types: 3.11.0 + yaml: 2.8.1 - '@milaboratories/pl-client@3.14.3': + '@milaboratories/pl-client@3.14.7': dependencies: - '@grpc/grpc-js': 1.13.4 + '@grpc/grpc-js': 1.14.4 '@milaboratories/pl-http': 1.2.4 - '@milaboratories/pl-model-common': 1.47.2 + '@milaboratories/pl-model-common': 1.48.0 '@milaboratories/ts-helpers': 1.8.6 - '@protobuf-ts/grpc-transport': 2.11.1(@grpc/grpc-js@1.13.4) + '@protobuf-ts/grpc-transport': 2.11.1(@grpc/grpc-js@1.14.4) '@protobuf-ts/runtime': 2.11.1 '@protobuf-ts/runtime-rpc': 2.11.1 canonicalize: 2.1.0 @@ -6065,12 +7757,12 @@ snapshots: upath: 2.0.1 yaml: 2.8.1 - '@milaboratories/pl-deployments@3.0.13': + '@milaboratories/pl-deployments@3.0.15': dependencies: '@milaboratories/pl-config': 1.8.5 - '@milaboratories/pl-healthcheck': 1.0.4 + '@milaboratories/pl-healthcheck': 1.0.5 '@milaboratories/pl-http': 1.2.4 - '@milaboratories/pl-model-common': 1.47.2 + '@milaboratories/pl-model-common': 1.47.3 '@milaboratories/ts-helpers': 1.8.6 decompress: 4.2.1 ssh2: 1.16.0 @@ -6080,16 +7772,16 @@ snapshots: yaml: 2.8.1 zod: 3.25.76 - '@milaboratories/pl-drivers@1.16.11': + '@milaboratories/pl-drivers@1.16.15': dependencies: - '@grpc/grpc-js': 1.13.4 + '@grpc/grpc-js': 1.14.4 '@milaboratories/computable': 2.9.8 '@milaboratories/helpers': 1.14.5 - '@milaboratories/pl-client': 3.14.3 - '@milaboratories/pl-model-common': 1.47.2 - '@milaboratories/pl-tree': 1.13.2 + '@milaboratories/pl-client': 3.14.6 + '@milaboratories/pl-model-common': 1.47.3 + '@milaboratories/pl-tree': 1.13.6 '@milaboratories/ts-helpers': 1.8.6 - '@protobuf-ts/grpc-transport': 2.11.1(@grpc/grpc-js@1.13.4) + '@protobuf-ts/grpc-transport': 2.11.1(@grpc/grpc-js@1.14.4) '@protobuf-ts/plugin': 2.11.1 '@protobuf-ts/runtime': 2.11.1 '@protobuf-ts/runtime-rpc': 2.11.1 @@ -6114,17 +7806,17 @@ snapshots: json-stringify-safe: 5.0.1 zod: 3.23.8 - '@milaboratories/pl-errors@1.4.32': + '@milaboratories/pl-errors@1.4.35': dependencies: - '@milaboratories/pl-client': 3.14.3 + '@milaboratories/pl-client': 3.14.6 '@milaboratories/ts-helpers': 1.8.6 zod: 3.25.76 - '@milaboratories/pl-healthcheck@1.0.4': + '@milaboratories/pl-healthcheck@1.0.5': dependencies: - '@grpc/grpc-js': 1.13.4 + '@grpc/grpc-js': 1.14.4 '@milaboratories/ts-helpers': 1.8.6 - '@protobuf-ts/grpc-transport': 2.11.1(@grpc/grpc-js@1.13.4) + '@protobuf-ts/grpc-transport': 2.11.1(@grpc/grpc-js@1.14.4) '@protobuf-ts/runtime': 2.11.1 '@protobuf-ts/runtime-rpc': 2.11.1 @@ -6132,34 +7824,33 @@ snapshots: dependencies: undici: 7.16.0 - '@milaboratories/pl-middle-layer@1.66.8(@bytecodealliance/preview2-shim@0.17.8)(@types/node@25.3.2)': + '@milaboratories/pl-middle-layer@1.66.19(@bytecodealliance/preview2-shim@0.17.8)(@types/node@25.3.2)': dependencies: - '@milaboratories/columns-collection-driver': 0.2.2 + '@milaboratories/columns-collection-driver': 0.2.3 '@milaboratories/computable': 2.9.8 '@milaboratories/helpers': 1.14.5 - '@milaboratories/pf-driver': 1.8.4(@bytecodealliance/preview2-shim@0.17.8) - '@milaboratories/pf-spec-driver': 1.4.23(@bytecodealliance/preview2-shim@0.17.8) + '@milaboratories/pf-driver': 1.9.0(@bytecodealliance/preview2-shim@0.17.8) + '@milaboratories/pf-spec-driver': 1.5.0(@bytecodealliance/preview2-shim@0.17.8) '@milaboratories/pframes-rs-node': 1.1.56 - '@milaboratories/pframes-rs-wasm': 1.1.56(@bytecodealliance/preview2-shim@0.17.8)(@milaboratories/pl-model-common@1.47.2)(@milaboratories/pl-model-middle-layer@1.30.14) - '@milaboratories/pl-client': 3.14.3 - '@milaboratories/pl-deployments': 3.0.13 - '@milaboratories/pl-drivers': 1.16.11 - '@milaboratories/pl-errors': 1.4.32 + '@milaboratories/pl-client': 3.14.6 + '@milaboratories/pl-deployments': 3.0.15 + '@milaboratories/pl-drivers': 1.16.15 + '@milaboratories/pl-errors': 1.4.35 '@milaboratories/pl-http': 1.2.4 - '@milaboratories/pl-model-backend': 1.4.17 - '@milaboratories/pl-model-common': 1.47.2 - '@milaboratories/pl-model-middle-layer': 1.30.14 - '@milaboratories/pl-tree': 1.13.2 + '@milaboratories/pl-model-backend': 1.4.20 + '@milaboratories/pl-model-common': 1.47.3 + '@milaboratories/pl-model-middle-layer': 1.31.0 + '@milaboratories/pl-tree': 1.13.6 '@milaboratories/resolve-helper': 1.1.3 '@milaboratories/ts-helpers': 1.8.6 - '@platforma-sdk/block-tools': 2.12.8(@types/node@25.3.2) - '@platforma-sdk/model': 1.80.8 + '@platforma-sdk/block-tools': 2.13.0(@types/node@25.3.2) + '@platforma-sdk/model': 1.81.1 '@platforma-sdk/workflow-tengo': 6.8.2 canonicalize: 2.1.0 denque: 2.1.0 es-toolkit: 1.42.0 lru-cache: 11.2.4 - quickjs-emscripten: 0.31.0 + quickjs-emscripten: 0.32.0 semver: 7.8.5 undici: 7.16.0 utility-types: 3.11.0 @@ -6172,9 +7863,15 @@ snapshots: - encoding - supports-color - '@milaboratories/pl-model-backend@1.4.17': + '@milaboratories/pl-model-backend@1.4.20': + dependencies: + '@milaboratories/pl-client': 3.14.6 + canonicalize: 2.1.0 + zod: 3.25.76 + + '@milaboratories/pl-model-backend@1.4.21': dependencies: - '@milaboratories/pl-client': 3.14.3 + '@milaboratories/pl-client': 3.14.7 canonicalize: 2.1.0 zod: 3.25.76 @@ -6191,7 +7888,7 @@ snapshots: canonicalize: 2.1.0 zod: 3.25.76 - '@milaboratories/pl-model-common@1.47.2': + '@milaboratories/pl-model-common@1.47.3': dependencies: '@milaboratories/helpers': 1.14.5 '@milaboratories/pl-error-like': 1.12.10 @@ -6199,12 +7896,12 @@ snapshots: es-toolkit: 1.42.0 zod: 3.25.76 - '@milaboratories/pl-model-middle-layer@1.30.14': + '@milaboratories/pl-model-common@1.48.0': dependencies: '@milaboratories/helpers': 1.14.5 - '@milaboratories/pl-model-common': 1.47.2 + '@milaboratories/pl-error-like': 1.12.10 + canonicalize: 2.1.0 es-toolkit: 1.42.0 - utility-types: 3.11.0 zod: 3.25.76 '@milaboratories/pl-model-middle-layer@1.30.7': @@ -6215,11 +7912,27 @@ snapshots: utility-types: 3.11.0 zod: 3.25.76 - '@milaboratories/pl-tree@1.13.2': + '@milaboratories/pl-model-middle-layer@1.31.0': + dependencies: + '@milaboratories/helpers': 1.14.5 + '@milaboratories/pl-model-common': 1.47.3 + es-toolkit: 1.42.0 + utility-types: 3.11.0 + zod: 3.25.76 + + '@milaboratories/pl-model-middle-layer@1.32.0': + dependencies: + '@milaboratories/helpers': 1.14.5 + '@milaboratories/pl-model-common': 1.48.0 + es-toolkit: 1.42.0 + utility-types: 3.11.0 + zod: 3.25.76 + + '@milaboratories/pl-tree@1.13.6': dependencies: '@milaboratories/computable': 2.9.8 - '@milaboratories/pl-client': 3.14.3 - '@milaboratories/pl-errors': 1.4.32 + '@milaboratories/pl-client': 3.14.6 + '@milaboratories/pl-errors': 1.4.35 '@milaboratories/ts-helpers': 1.8.6 denque: 2.1.0 utility-types: 3.11.0 @@ -6229,9 +7942,9 @@ snapshots: dependencies: '@platforma-open/milaboratories.software-ptabler.schema': 1.13.2 - '@milaboratories/ptabler-expression-js@1.2.36': + '@milaboratories/ptabler-expression-js@1.2.37': dependencies: - '@platforma-open/milaboratories.software-ptabler.schema': 1.15.20 + '@platforma-open/milaboratories.software-ptabler.schema': 1.15.21 '@milaboratories/resolve-helper@1.1.3': {} @@ -6239,9 +7952,9 @@ snapshots: '@milaboratories/tengo-tester@1.6.4': {} - '@milaboratories/ts-builder@1.6.1(@types/node@25.3.2)(rollup@4.53.3)(vue@3.5.24(typescript@5.6.3))(yaml@2.8.1)': + '@milaboratories/ts-builder@1.6.2(@types/node@25.3.2)(rollup@4.53.3)(vue@3.5.24(typescript@5.6.3))(yaml@2.8.1)': dependencies: - '@milaboratories/ts-configs': 1.3.1 + '@milaboratories/ts-configs': 1.4.0 '@vitejs/plugin-vue': 6.0.6(vite@8.0.8(@types/node@25.3.2)(yaml@2.8.1))(vue@3.5.24(typescript@5.6.3)) commander: 15.0.0 jsonc-parser: 3.3.1 @@ -6280,10 +7993,10 @@ snapshots: - vue - yaml - '@milaboratories/ts-builder@1.6.1(@types/node@25.3.2)(rollup@4.53.3)(vue@3.5.25(typescript@5.6.3))(yaml@2.8.1)': + '@milaboratories/ts-builder@1.6.2(@types/node@25.3.2)(rollup@4.53.3)(vue@3.5.41(typescript@5.6.3))(yaml@2.8.1)': dependencies: - '@milaboratories/ts-configs': 1.3.1 - '@vitejs/plugin-vue': 6.0.6(vite@8.0.8(@types/node@25.3.2)(yaml@2.8.1))(vue@3.5.25(typescript@5.6.3)) + '@milaboratories/ts-configs': 1.4.0 + '@vitejs/plugin-vue': 6.0.6(vite@8.0.8(@types/node@25.3.2)(yaml@2.8.1))(vue@3.5.41(typescript@5.6.3)) commander: 15.0.0 jsonc-parser: 3.3.1 oxfmt: 0.35.0 @@ -6321,10 +8034,10 @@ snapshots: - vue - yaml - '@milaboratories/ts-builder@1.6.1(@types/node@25.3.2)(rollup@4.53.3)(vue@3.5.25(typescript@5.9.3))(yaml@2.8.1)': + '@milaboratories/ts-builder@1.6.2(@types/node@25.3.2)(rollup@4.53.3)(vue@3.5.41(typescript@5.9.3))(yaml@2.8.1)': dependencies: - '@milaboratories/ts-configs': 1.3.1 - '@vitejs/plugin-vue': 6.0.6(vite@8.0.8(@types/node@25.3.2)(yaml@2.8.1))(vue@3.5.25(typescript@5.9.3)) + '@milaboratories/ts-configs': 1.4.0 + '@vitejs/plugin-vue': 6.0.6(vite@8.0.8(@types/node@25.3.2)(yaml@2.8.1))(vue@3.5.41(typescript@5.9.3)) commander: 15.0.0 jsonc-parser: 3.3.1 oxfmt: 0.35.0 @@ -6362,7 +8075,7 @@ snapshots: - vue - yaml - '@milaboratories/ts-configs@1.3.1': {} + '@milaboratories/ts-configs@1.4.0': {} '@milaboratories/ts-helpers@1.8.6': dependencies: @@ -6370,10 +8083,10 @@ snapshots: canonicalize: 2.1.0 denque: 2.1.0 - '@milaboratories/uikit@2.15.17(typescript@5.6.3)': + '@milaboratories/uikit@2.15.23(typescript@5.6.3)': dependencies: '@milaboratories/helpers': 1.14.5 - '@platforma-sdk/model': 1.80.8 + '@platforma-sdk/model': 1.81.1 '@types/d3-array': 3.2.1 '@types/d3-axis': 3.0.6 '@types/d3-scale': 4.0.9 @@ -6704,30 +8417,45 @@ snapshots: '@pkgjs/parseargs@0.11.0': optional: true - '@platforma-open/milaboratories.runenv-python-3.12.10-atls@1.2.4': {} + '@platforma-open/milaboratories.runenv-python-3.12.10-atls@1.2.7': {} + + '@platforma-open/milaboratories.runenv-python-3.12.10-clustering@0.1.1': {} + + '@platforma-open/milaboratories.runenv-python-3.12.10-h5ad@1.1.5': {} - '@platforma-open/milaboratories.runenv-python-3.12.10-h5ad@1.1.4': {} + '@platforma-open/milaboratories.runenv-python-3.12.10-humanness@0.2.0': {} '@platforma-open/milaboratories.runenv-python-3.12.10-parapred@1.1.0': {} - '@platforma-open/milaboratories.runenv-python-3.12.10-rapids@1.4.4': {} + '@platforma-open/milaboratories.runenv-python-3.12.10-pgen@0.2.0': {} - '@platforma-open/milaboratories.runenv-python-3.12.10-sccoda@1.3.5': {} + '@platforma-open/milaboratories.runenv-python-3.12.10-rapids@1.7.2': {} - '@platforma-open/milaboratories.runenv-python-3.12.10@1.3.6': {} + '@platforma-open/milaboratories.runenv-python-3.12.10-sccoda@1.3.6': {} - '@platforma-open/milaboratories.runenv-python-3@1.7.8': + '@platforma-open/milaboratories.runenv-python-3.12.10-scientific-slim@1.1.0': {} + + '@platforma-open/milaboratories.runenv-python-3.12.10-torch-cuda@0.2.0': {} + + '@platforma-open/milaboratories.runenv-python-3.12.10@1.3.24': {} + + '@platforma-open/milaboratories.runenv-python-3@1.11.6': dependencies: - '@platforma-open/milaboratories.runenv-python-3.12.10': 1.3.6 - '@platforma-open/milaboratories.runenv-python-3.12.10-atls': 1.2.4 - '@platforma-open/milaboratories.runenv-python-3.12.10-h5ad': 1.1.4 + '@platforma-open/milaboratories.runenv-python-3.12.10': 1.3.24 + '@platforma-open/milaboratories.runenv-python-3.12.10-atls': 1.2.7 + '@platforma-open/milaboratories.runenv-python-3.12.10-clustering': 0.1.1 + '@platforma-open/milaboratories.runenv-python-3.12.10-h5ad': 1.1.5 + '@platforma-open/milaboratories.runenv-python-3.12.10-humanness': 0.2.0 '@platforma-open/milaboratories.runenv-python-3.12.10-parapred': 1.1.0 - '@platforma-open/milaboratories.runenv-python-3.12.10-rapids': 1.4.4 - '@platforma-open/milaboratories.runenv-python-3.12.10-sccoda': 1.3.5 + '@platforma-open/milaboratories.runenv-python-3.12.10-pgen': 0.2.0 + '@platforma-open/milaboratories.runenv-python-3.12.10-rapids': 1.7.2 + '@platforma-open/milaboratories.runenv-python-3.12.10-sccoda': 1.3.6 + '@platforma-open/milaboratories.runenv-python-3.12.10-scientific-slim': 1.1.0 + '@platforma-open/milaboratories.runenv-python-3.12.10-torch-cuda': 0.2.0 '@platforma-open/milaboratories.samples-and-data.model@1.11.2': dependencies: - '@platforma-sdk/model': 1.80.8 + '@platforma-sdk/model': 1.81.1 zod: 3.23.8 '@platforma-open/milaboratories.samples-and-data.model@2.5.3': @@ -6760,9 +8488,9 @@ snapshots: dependencies: '@milaboratories/pl-model-common': 1.23.0 - '@platforma-open/milaboratories.software-ptabler.schema@1.15.20': + '@platforma-open/milaboratories.software-ptabler.schema@1.15.21': dependencies: - '@milaboratories/pl-model-common': 1.47.2 + '@milaboratories/pl-model-common': 1.47.3 '@platforma-open/milaboratories.software-ptabler@1.14.0': {} @@ -6829,19 +8557,44 @@ snapshots: '@platforma-open/milaboratories.software-small-binaries.mnz-client': 1.6.5 '@platforma-open/milaboratories.software-small-binaries.table-converter': 1.3.5 - '@platforma-sdk/block-tools@2.12.8(@types/node@25.3.2)': + '@platforma-sdk/block-tools@2.13.0(@types/node@25.3.2)': + dependencies: + '@aws-sdk/client-ecr-public': 3.859.0 + '@aws-sdk/client-s3': 3.859.0 + '@inquirer/prompts': 7.10.1(@types/node@25.3.2) + '@milaboratories/pl-http': 1.2.4 + '@milaboratories/pl-model-backend': 1.4.20 + '@milaboratories/pl-model-common': 1.47.3 + '@milaboratories/pl-model-middle-layer': 1.31.0 + '@milaboratories/resolve-helper': 1.1.3 + '@milaboratories/ts-helpers': 1.8.6 + '@platforma-sdk/blocks-deps-updater': 2.2.0 + '@platforma-sdk/package-builder-lib': 1.3.0 + canonicalize: 2.1.0 + commander: 15.0.0 + lru-cache: 11.2.4 + mime-types: 2.1.35 + tar: 7.4.3 + undici: 7.16.0 + yaml: 2.8.1 + zod: 3.25.76 + transitivePeerDependencies: + - '@types/node' + - aws-crt + + '@platforma-sdk/block-tools@2.14.0(@types/node@25.3.2)': dependencies: '@aws-sdk/client-ecr-public': 3.859.0 '@aws-sdk/client-s3': 3.859.0 '@inquirer/prompts': 7.10.1(@types/node@25.3.2) '@milaboratories/pl-http': 1.2.4 - '@milaboratories/pl-model-backend': 1.4.17 - '@milaboratories/pl-model-common': 1.47.2 - '@milaboratories/pl-model-middle-layer': 1.30.14 + '@milaboratories/pl-model-backend': 1.4.21 + '@milaboratories/pl-model-common': 1.48.0 + '@milaboratories/pl-model-middle-layer': 1.32.0 '@milaboratories/resolve-helper': 1.1.3 '@milaboratories/ts-helpers': 1.8.6 '@platforma-sdk/blocks-deps-updater': 2.2.0 - '@platforma-sdk/package-builder-lib': 1.2.1 + '@platforma-sdk/package-builder-lib': 1.3.0 canonicalize: 2.1.0 commander: 15.0.0 lru-cache: 11.2.4 @@ -6868,13 +8621,13 @@ snapshots: utility-types: 3.11.0 zod: 3.23.8 - '@platforma-sdk/model@1.80.8': + '@platforma-sdk/model@1.81.1': dependencies: '@milaboratories/helpers': 1.14.5 '@milaboratories/pl-error-like': 1.12.10 - '@milaboratories/pl-model-common': 1.47.2 - '@milaboratories/pl-model-middle-layer': 1.30.14 - '@milaboratories/ptabler-expression-js': 1.2.36 + '@milaboratories/pl-model-common': 1.47.3 + '@milaboratories/pl-model-middle-layer': 1.31.0 + '@milaboratories/ptabler-expression-js': 1.2.37 canonicalize: 2.1.0 es-toolkit: 1.42.0 fast-json-patch: 3.1.1 @@ -6882,7 +8635,7 @@ snapshots: utility-types: 3.11.0 zod: 3.25.76 - '@platforma-sdk/package-builder-lib@1.2.1': + '@platforma-sdk/package-builder-lib@1.3.0': dependencies: '@aws-sdk/client-s3': 3.859.0 '@aws-sdk/lib-storage': 3.859.0(@aws-sdk/client-s3@3.859.0) @@ -6895,24 +8648,24 @@ snapshots: transitivePeerDependencies: - aws-crt - '@platforma-sdk/tengo-builder@4.0.19': + '@platforma-sdk/tengo-builder@4.0.22': dependencies: - '@milaboratories/pl-model-backend': 1.4.17 + '@milaboratories/pl-model-backend': 1.4.20 '@milaboratories/resolve-helper': 1.1.3 '@milaboratories/tengo-tester': 1.6.4 '@milaboratories/ts-helpers': 1.8.6 commander: 15.0.0 winston: 3.17.0 - '@platforma-sdk/test@1.80.9(@bytecodealliance/preview2-shim@0.17.8)(@types/node@25.3.2)(vite@7.2.7(@types/node@25.3.2)(lightningcss@1.32.0)(yaml@2.8.1))': + '@platforma-sdk/test@1.81.3(@bytecodealliance/preview2-shim@0.17.8)(@types/node@25.3.2)(vite@8.0.8(@types/node@25.3.2)(yaml@2.8.1))': dependencies: '@milaboratories/computable': 2.9.8 - '@milaboratories/pl-client': 3.14.3 - '@milaboratories/pl-middle-layer': 1.66.8(@bytecodealliance/preview2-shim@0.17.8)(@types/node@25.3.2) - '@milaboratories/pl-tree': 1.13.2 - '@platforma-sdk/model': 1.80.8 - '@vitest/coverage-istanbul': 4.1.4(vitest@4.0.18(@types/node@25.3.2)(lightningcss@1.32.0)(yaml@2.8.1)) - vitest: 4.1.4(@types/node@25.3.2)(@vitest/coverage-istanbul@4.1.4)(vite@7.2.7(@types/node@25.3.2)(lightningcss@1.32.0)(yaml@2.8.1)) + '@milaboratories/pl-client': 3.14.6 + '@milaboratories/pl-middle-layer': 1.66.19(@bytecodealliance/preview2-shim@0.17.8)(@types/node@25.3.2) + '@milaboratories/pl-tree': 1.13.6 + '@platforma-sdk/model': 1.81.1 + '@vitest/coverage-istanbul': 4.1.4(vitest@4.1.10) + vitest: 4.1.10(@types/node@25.3.2)(@vitest/coverage-istanbul@4.1.4)(vite@8.0.8(@types/node@25.3.2)(yaml@2.8.1)) transitivePeerDependencies: - '@bytecodealliance/preview2-shim' - '@edge-runtime/vm' @@ -6931,40 +8684,13 @@ snapshots: - supports-color - vite - '@platforma-sdk/test@1.80.9(@bytecodealliance/preview2-shim@0.17.8)(@types/node@25.3.2)(vite@8.0.8(@types/node@25.3.2)(yaml@2.8.1))': + '@platforma-sdk/ui-vue@1.81.1(@bytecodealliance/preview2-shim@0.17.8)(typescript@5.6.3)': dependencies: - '@milaboratories/computable': 2.9.8 - '@milaboratories/pl-client': 3.14.3 - '@milaboratories/pl-middle-layer': 1.66.8(@bytecodealliance/preview2-shim@0.17.8)(@types/node@25.3.2) - '@milaboratories/pl-tree': 1.13.2 - '@platforma-sdk/model': 1.80.8 - '@vitest/coverage-istanbul': 4.1.4(vitest@4.1.4) - vitest: 4.1.4(@types/node@25.3.2)(@vitest/coverage-istanbul@4.1.4)(vite@8.0.8(@types/node@25.3.2)(yaml@2.8.1)) - transitivePeerDependencies: - - '@bytecodealliance/preview2-shim' - - '@edge-runtime/vm' - - '@opentelemetry/api' - - '@types/node' - - '@vitest/browser-playwright' - - '@vitest/browser-preview' - - '@vitest/browser-webdriverio' - - '@vitest/coverage-v8' - - '@vitest/ui' - - aws-crt - - encoding - - happy-dom - - jsdom - - msw - - supports-color - - vite - - '@platforma-sdk/ui-vue@1.80.9(@bytecodealliance/preview2-shim@0.17.8)(typescript@5.6.3)': - dependencies: - '@milaboratories/columns-collection-driver': 0.2.2 - '@milaboratories/pf-spec-driver': 1.4.23(@bytecodealliance/preview2-shim@0.17.8) - '@milaboratories/pl-model-common': 1.47.2 - '@milaboratories/uikit': 2.15.17(typescript@5.6.3) - '@platforma-sdk/model': 1.80.8 + '@milaboratories/columns-collection-driver': 0.2.3 + '@milaboratories/pf-spec-driver': 1.5.0(@bytecodealliance/preview2-shim@0.17.8) + '@milaboratories/pl-model-common': 1.47.3 + '@milaboratories/uikit': 2.15.23(typescript@5.6.3) + '@platforma-sdk/model': 1.81.1 '@types/d3-format': 3.0.4 '@types/node': 24.5.2 '@types/semver': 7.7.0 @@ -7010,9 +8736,9 @@ snapshots: '@platforma-open/milaboratories.software-ptexter': 1.2.4 '@platforma-open/milaboratories.software-small-binaries': 2.1.1 - '@protobuf-ts/grpc-transport@2.11.1(@grpc/grpc-js@1.13.4)': + '@protobuf-ts/grpc-transport@2.11.1(@grpc/grpc-js@1.14.4)': dependencies: - '@grpc/grpc-js': 1.13.4 + '@grpc/grpc-js': 1.14.4 '@protobuf-ts/runtime': 2.11.1 '@protobuf-ts/runtime-rpc': 2.11.1 @@ -7039,24 +8765,21 @@ snapshots: '@protobufjs/base64@1.1.2': {} - '@protobufjs/codegen@2.0.4': {} + '@protobufjs/codegen@2.0.5': {} - '@protobufjs/eventemitter@1.1.0': {} + '@protobufjs/eventemitter@1.1.1': {} - '@protobufjs/fetch@1.1.0': + '@protobufjs/fetch@1.1.1': dependencies: '@protobufjs/aspromise': 1.1.2 - '@protobufjs/inquire': 1.1.0 '@protobufjs/float@1.0.2': {} - '@protobufjs/inquire@1.1.0': {} - '@protobufjs/path@1.1.2': {} '@protobufjs/pool@1.1.0': {} - '@protobufjs/utf8@1.1.0': {} + '@protobufjs/utf8@1.1.2': {} '@rolldown/binding-android-arm64@1.0.0-rc.15': optional: true @@ -7543,75 +9266,2499 @@ snapshots: '@smithy/util-defaults-mode-browser@4.0.27': dependencies: - '@smithy/property-provider': 4.0.5 - '@smithy/smithy-client': 4.5.0 - '@smithy/types': 4.3.2 - bowser: 2.11.0 - tslib: 2.8.1 + '@smithy/property-provider': 4.0.5 + '@smithy/smithy-client': 4.5.0 + '@smithy/types': 4.3.2 + bowser: 2.11.0 + tslib: 2.8.1 + + '@smithy/util-defaults-mode-node@4.0.27': + dependencies: + '@smithy/config-resolver': 4.1.5 + '@smithy/credential-provider-imds': 4.0.7 + '@smithy/node-config-provider': 4.1.4 + '@smithy/property-provider': 4.0.5 + '@smithy/smithy-client': 4.5.0 + '@smithy/types': 4.3.2 + tslib: 2.8.1 + + '@smithy/util-endpoints@3.0.7': + dependencies: + '@smithy/node-config-provider': 4.1.4 + '@smithy/types': 4.3.2 + tslib: 2.8.1 + + '@smithy/util-hex-encoding@4.0.0': + dependencies: + tslib: 2.8.1 + + '@smithy/util-middleware@4.0.5': + dependencies: + '@smithy/types': 4.3.2 + tslib: 2.8.1 + + '@smithy/util-retry@4.0.7': + dependencies: + '@smithy/service-error-classification': 4.0.7 + '@smithy/types': 4.3.2 + tslib: 2.8.1 + + '@smithy/util-stream@4.2.4': + dependencies: + '@smithy/fetch-http-handler': 5.1.1 + '@smithy/node-http-handler': 4.1.1 + '@smithy/types': 4.3.2 + '@smithy/util-base64': 4.0.0 + '@smithy/util-buffer-from': 4.0.0 + '@smithy/util-hex-encoding': 4.0.0 + '@smithy/util-utf8': 4.0.0 + tslib: 2.8.1 + + '@smithy/util-uri-escape@4.0.0': + dependencies: + tslib: 2.8.1 + + '@smithy/util-utf8@2.3.0': + dependencies: + '@smithy/util-buffer-from': 2.2.0 + tslib: 2.8.1 + + '@smithy/util-utf8@4.0.0': + dependencies: + '@smithy/util-buffer-from': 4.0.0 + tslib: 2.8.1 + + '@smithy/util-waiter@4.0.7': + dependencies: + '@smithy/abort-controller': 4.0.5 + '@smithy/types': 4.3.2 + tslib: 2.8.1 + + '@standard-schema/spec@1.1.0': {} + + '@stdlib/array-base-accessor-getter@0.2.3': {} + + '@stdlib/array-base-accessor-setter@0.2.3': {} + + '@stdlib/array-base-arraylike2object@0.2.2': + dependencies: + '@stdlib/array-base-accessor-getter': 0.2.3 + '@stdlib/array-base-accessor-setter': 0.2.3 + '@stdlib/array-base-assert-is-accessor-array': 0.2.3 + '@stdlib/array-base-getter': 0.2.3 + '@stdlib/array-base-setter': 0.2.3 + '@stdlib/array-dtype': 0.3.1 + transitivePeerDependencies: + - supports-color + + '@stdlib/array-base-assert-contains@0.2.3': + dependencies: + '@stdlib/array-base-accessor-getter': 0.2.3 + '@stdlib/array-base-assert-is-accessor-array': 0.2.3 + '@stdlib/array-base-getter': 0.2.3 + '@stdlib/array-dtype': 0.3.1 + '@stdlib/assert-is-collection': 0.2.3 + '@stdlib/error-tools-fmtprodmsg': 0.2.3 + '@stdlib/string-format': 0.2.3 + '@stdlib/utils-define-nonenumerable-read-only-property': 0.2.3 + transitivePeerDependencies: + - supports-color + + '@stdlib/array-base-assert-is-accessor-array@0.2.3': {} + + '@stdlib/array-base-assert-is-booleanarray@0.0.3': {} + + '@stdlib/array-base-assert-is-complex-typed-array@0.1.3': {} + + '@stdlib/array-base-assert-is-complex128array@0.2.3': {} + + '@stdlib/array-base-assert-is-complex64array@0.2.3': {} + + '@stdlib/array-base-filled@0.2.3': {} + + '@stdlib/array-base-getter@0.2.3': {} + + '@stdlib/array-base-incrspace@0.2.3': + dependencies: + '@stdlib/math-base-special-ceil': 0.2.3 + transitivePeerDependencies: + - supports-color + + '@stdlib/array-base-index-of-same-value@0.1.1': + dependencies: + '@stdlib/array-base-assert-is-booleanarray': 0.0.3 + '@stdlib/array-base-assert-is-complex-typed-array': 0.1.3 + '@stdlib/array-base-resolve-getter': 0.2.3 + '@stdlib/assert-is-accessor-array': 0.2.3 + '@stdlib/assert-is-boolean': 0.2.3 + '@stdlib/assert-is-complex-like': 0.2.4 + '@stdlib/assert-is-same-value': 0.2.3 + '@stdlib/complex-float64-imag': 0.1.2 + '@stdlib/complex-float64-real': 0.1.2 + '@stdlib/strided-base-reinterpret-boolean': 0.0.3 + '@stdlib/strided-base-reinterpret-complex': 0.1.3 + transitivePeerDependencies: + - supports-color + + '@stdlib/array-base-resolve-getter@0.2.3': + dependencies: + '@stdlib/array-base-accessor-getter': 0.2.3 + '@stdlib/array-base-assert-is-accessor-array': 0.2.3 + '@stdlib/array-base-getter': 0.2.3 + '@stdlib/array-dtype': 0.3.1 + transitivePeerDependencies: + - supports-color + + '@stdlib/array-base-setter@0.2.3': {} + + '@stdlib/array-base-zeros@0.2.3': + dependencies: + '@stdlib/array-base-filled': 0.2.3 + + '@stdlib/array-bool@0.1.2': + dependencies: + '@stdlib/array-base-accessor-getter': 0.2.3 + '@stdlib/array-base-getter': 0.2.3 + '@stdlib/array-uint8': 0.2.3 + '@stdlib/assert-has-iterator-symbol-support': 0.2.3 + '@stdlib/assert-is-arraybuffer': 0.2.3 + '@stdlib/assert-is-boolean': 0.2.3 + '@stdlib/assert-is-collection': 0.2.3 + '@stdlib/assert-is-function': 0.2.3 + '@stdlib/assert-is-integer': 0.2.3 + '@stdlib/assert-is-nonnegative-integer': 0.2.3 + '@stdlib/assert-is-object': 0.2.3 + '@stdlib/assert-is-string': 0.2.3 + '@stdlib/assert-is-string-array': 0.2.3 + '@stdlib/boolean-ctor': 0.2.3 + '@stdlib/error-tools-fmtprodmsg': 0.2.3 + '@stdlib/math-base-special-floor': 0.2.4 + '@stdlib/string-format': 0.2.3 + '@stdlib/symbol-iterator': 0.2.3 + '@stdlib/utils-define-nonenumerable-read-only-accessor': 0.2.4 + '@stdlib/utils-define-nonenumerable-read-only-property': 0.2.3 + transitivePeerDependencies: + - supports-color + + '@stdlib/array-complex128@0.3.2': + dependencies: + '@stdlib/array-base-accessor-getter': 0.2.3 + '@stdlib/array-base-assert-is-complex128array': 0.2.3 + '@stdlib/array-base-assert-is-complex64array': 0.2.3 + '@stdlib/array-base-getter': 0.2.3 + '@stdlib/array-float64': 0.2.3 + '@stdlib/assert-has-iterator-symbol-support': 0.2.3 + '@stdlib/assert-is-array': 0.2.3 + '@stdlib/assert-is-array-like-object': 0.2.3 + '@stdlib/assert-is-arraybuffer': 0.2.3 + '@stdlib/assert-is-collection': 0.2.3 + '@stdlib/assert-is-complex-like': 0.2.4 + '@stdlib/assert-is-function': 0.2.3 + '@stdlib/assert-is-nonnegative-integer': 0.2.3 + '@stdlib/assert-is-object': 0.2.3 + '@stdlib/assert-is-string': 0.2.3 + '@stdlib/assert-is-string-array': 0.2.3 + '@stdlib/complex-float64-ctor': 0.1.2 + '@stdlib/complex-float64-imag': 0.1.2 + '@stdlib/complex-float64-real': 0.1.2 + '@stdlib/error-tools-fmtprodmsg': 0.2.3 + '@stdlib/math-base-assert-is-even': 0.2.5 + '@stdlib/math-base-assert-is-integer': 0.2.7 + '@stdlib/math-base-special-floor': 0.2.4 + '@stdlib/strided-base-reinterpret-complex128': 0.2.3 + '@stdlib/strided-base-reinterpret-complex64': 0.2.2 + '@stdlib/string-format': 0.2.3 + '@stdlib/symbol-iterator': 0.2.3 + '@stdlib/utils-define-nonenumerable-read-only-accessor': 0.2.4 + '@stdlib/utils-define-nonenumerable-read-only-property': 0.2.3 + transitivePeerDependencies: + - supports-color + + '@stdlib/array-complex64@0.3.2': + dependencies: + '@stdlib/array-base-accessor-getter': 0.2.3 + '@stdlib/array-base-assert-is-complex128array': 0.2.3 + '@stdlib/array-base-assert-is-complex64array': 0.2.3 + '@stdlib/array-base-getter': 0.2.3 + '@stdlib/array-float32': 0.2.3 + '@stdlib/assert-has-iterator-symbol-support': 0.2.3 + '@stdlib/assert-is-array': 0.2.3 + '@stdlib/assert-is-array-like-object': 0.2.3 + '@stdlib/assert-is-arraybuffer': 0.2.3 + '@stdlib/assert-is-collection': 0.2.3 + '@stdlib/assert-is-complex-like': 0.2.4 + '@stdlib/assert-is-function': 0.2.3 + '@stdlib/assert-is-nonnegative-integer': 0.2.3 + '@stdlib/assert-is-object': 0.2.3 + '@stdlib/assert-is-string': 0.2.3 + '@stdlib/assert-is-string-array': 0.2.3 + '@stdlib/complex-float32-ctor': 0.1.1 + '@stdlib/complex-float32-imag': 0.1.2 + '@stdlib/complex-float32-real': 0.1.2 + '@stdlib/error-tools-fmtprodmsg': 0.2.3 + '@stdlib/math-base-assert-is-even': 0.2.5 + '@stdlib/math-base-assert-is-integer': 0.2.7 + '@stdlib/math-base-special-floor': 0.2.4 + '@stdlib/strided-base-reinterpret-complex128': 0.2.3 + '@stdlib/strided-base-reinterpret-complex64': 0.2.2 + '@stdlib/string-format': 0.2.3 + '@stdlib/symbol-iterator': 0.2.3 + '@stdlib/utils-define-nonenumerable-read-only-accessor': 0.2.4 + '@stdlib/utils-define-nonenumerable-read-only-property': 0.2.3 + transitivePeerDependencies: + - supports-color + + '@stdlib/array-dtype@0.3.1': + dependencies: + '@stdlib/array-bool': 0.1.2 + '@stdlib/array-complex128': 0.3.2 + '@stdlib/array-complex64': 0.3.2 + '@stdlib/array-float32': 0.2.3 + '@stdlib/array-float64': 0.2.3 + '@stdlib/array-int16': 0.2.3 + '@stdlib/array-int32': 0.2.3 + '@stdlib/array-int8': 0.2.3 + '@stdlib/array-uint16': 0.2.3 + '@stdlib/array-uint32': 0.2.3 + '@stdlib/array-uint8': 0.2.3 + '@stdlib/array-uint8c': 0.2.3 + '@stdlib/assert-is-array': 0.2.3 + '@stdlib/assert-is-buffer': 0.2.3 + '@stdlib/utils-constructor-name': 0.2.3 + transitivePeerDependencies: + - supports-color + + '@stdlib/array-float32@0.2.3': + dependencies: + '@stdlib/assert-has-float32array-support': 0.2.3 + + '@stdlib/array-float64@0.2.3': + dependencies: + '@stdlib/assert-has-float64array-support': 0.2.3 + + '@stdlib/array-int16@0.2.3': + dependencies: + '@stdlib/assert-has-int16array-support': 0.2.3 + + '@stdlib/array-int32@0.2.3': + dependencies: + '@stdlib/assert-has-int32array-support': 0.2.3 + + '@stdlib/array-int8@0.2.3': + dependencies: + '@stdlib/assert-has-int8array-support': 0.2.3 + + '@stdlib/array-uint16@0.2.3': + dependencies: + '@stdlib/assert-has-uint16array-support': 0.2.3 + + '@stdlib/array-uint32@0.2.3': + dependencies: + '@stdlib/assert-has-uint32array-support': 0.2.3 + + '@stdlib/array-uint8@0.2.3': + dependencies: + '@stdlib/assert-has-uint8array-support': 0.2.3 + + '@stdlib/array-uint8c@0.2.3': + dependencies: + '@stdlib/assert-has-uint8clampedarray-support': 0.2.3 + + '@stdlib/assert-contains@0.3.1': + dependencies: + '@stdlib/array-base-index-of-same-value': 0.1.1 + '@stdlib/assert-is-collection': 0.2.3 + '@stdlib/assert-is-integer': 0.2.3 + '@stdlib/assert-is-string': 0.2.3 + '@stdlib/error-tools-fmtprodmsg': 0.2.3 + '@stdlib/string-format': 0.2.3 + transitivePeerDependencies: + - supports-color + + '@stdlib/assert-has-float32array-support@0.2.3': + dependencies: + '@stdlib/assert-is-float32array': 0.2.3 + '@stdlib/constants-float64-pinf': 0.2.3 + + '@stdlib/assert-has-float64array-support@0.2.3': + dependencies: + '@stdlib/assert-is-float64array': 0.2.3 + + '@stdlib/assert-has-generator-support@0.2.3': + dependencies: + '@stdlib/utils-eval': 0.2.3 + + '@stdlib/assert-has-int16array-support@0.2.3': + dependencies: + '@stdlib/assert-is-int16array': 0.2.3 + '@stdlib/constants-int16-max': 0.2.3 + '@stdlib/constants-int16-min': 0.2.3 + + '@stdlib/assert-has-int32array-support@0.2.3': + dependencies: + '@stdlib/assert-is-int32array': 0.2.3 + '@stdlib/constants-int32-max': 0.3.1 + '@stdlib/constants-int32-min': 0.2.3 + + '@stdlib/assert-has-int8array-support@0.2.3': + dependencies: + '@stdlib/assert-is-int8array': 0.2.3 + '@stdlib/constants-int8-max': 0.2.3 + '@stdlib/constants-int8-min': 0.2.3 + + '@stdlib/assert-has-iterator-symbol-support@0.2.3': + dependencies: + '@stdlib/assert-has-own-property': 0.2.3 + '@stdlib/symbol-ctor': 0.2.3 + + '@stdlib/assert-has-own-property@0.2.3': {} + + '@stdlib/assert-has-symbol-support@0.2.3': {} + + '@stdlib/assert-has-to-primitive-symbol-support@0.1.1': + dependencies: + '@stdlib/assert-has-own-property': 0.2.3 + '@stdlib/symbol-ctor': 0.2.3 + + '@stdlib/assert-has-tostringtag-support@0.2.3': + dependencies: + '@stdlib/assert-has-symbol-support': 0.2.3 + + '@stdlib/assert-has-uint16array-support@0.2.3': + dependencies: + '@stdlib/assert-is-uint16array': 0.2.3 + '@stdlib/constants-uint16-max': 0.2.3 + + '@stdlib/assert-has-uint32array-support@0.2.3': + dependencies: + '@stdlib/assert-is-uint32array': 0.2.3 + '@stdlib/constants-uint32-max': 0.2.3 + + '@stdlib/assert-has-uint8array-support@0.2.3': + dependencies: + '@stdlib/assert-is-uint8array': 0.2.3 + '@stdlib/constants-uint8-max': 0.2.3 + + '@stdlib/assert-has-uint8clampedarray-support@0.2.3': + dependencies: + '@stdlib/assert-is-uint8clampedarray': 0.2.3 + + '@stdlib/assert-is-accessor-array@0.2.3': + dependencies: + '@stdlib/assert-is-collection': 0.2.3 + '@stdlib/assert-is-function': 0.2.3 + transitivePeerDependencies: + - supports-color + + '@stdlib/assert-is-arguments@0.2.3': + dependencies: + '@stdlib/assert-has-own-property': 0.2.3 + '@stdlib/assert-is-array': 0.2.3 + '@stdlib/assert-is-enumerable-property': 0.2.3 + '@stdlib/constants-uint32-max': 0.2.3 + '@stdlib/math-base-assert-is-integer': 0.2.7 + '@stdlib/utils-native-class': 0.2.3 + transitivePeerDependencies: + - supports-color + + '@stdlib/assert-is-array-like-object@0.2.3': + dependencies: + '@stdlib/constants-array-max-array-length': 0.2.3 + '@stdlib/math-base-assert-is-integer': 0.2.7 + transitivePeerDependencies: + - supports-color + + '@stdlib/assert-is-array-like@0.2.3': + dependencies: + '@stdlib/constants-array-max-array-length': 0.2.3 + '@stdlib/math-base-assert-is-integer': 0.2.7 + transitivePeerDependencies: + - supports-color + + '@stdlib/assert-is-array@0.2.3': + dependencies: + '@stdlib/utils-native-class': 0.2.3 + + '@stdlib/assert-is-arraybuffer@0.2.3': + dependencies: + '@stdlib/utils-native-class': 0.2.3 + + '@stdlib/assert-is-big-endian@0.2.3': + dependencies: + '@stdlib/array-uint16': 0.2.3 + '@stdlib/array-uint8': 0.2.3 + + '@stdlib/assert-is-boolean@0.2.3': + dependencies: + '@stdlib/assert-has-tostringtag-support': 0.2.3 + '@stdlib/boolean-ctor': 0.2.3 + '@stdlib/utils-define-nonenumerable-read-only-property': 0.2.3 + '@stdlib/utils-native-class': 0.2.3 + + '@stdlib/assert-is-buffer@0.2.3': + dependencies: + '@stdlib/assert-is-object-like': 0.2.3 + + '@stdlib/assert-is-collection@0.2.3': + dependencies: + '@stdlib/constants-array-max-typed-array-length': 0.2.3 + '@stdlib/math-base-assert-is-integer': 0.2.7 + transitivePeerDependencies: + - supports-color + + '@stdlib/assert-is-complex-like@0.2.4': + dependencies: + '@stdlib/complex-float32-ctor': 0.1.1 + '@stdlib/complex-float64-ctor': 0.1.2 + + '@stdlib/assert-is-enumerable-property@0.2.3': + dependencies: + '@stdlib/assert-is-integer': 0.2.3 + '@stdlib/assert-is-nan': 0.2.3 + '@stdlib/assert-is-string': 0.2.3 + transitivePeerDependencies: + - supports-color + + '@stdlib/assert-is-float32array@0.2.3': + dependencies: + '@stdlib/utils-native-class': 0.2.3 + + '@stdlib/assert-is-float64array@0.2.3': + dependencies: + '@stdlib/utils-native-class': 0.2.3 + + '@stdlib/assert-is-function@0.2.3': + dependencies: + '@stdlib/utils-type-of': 0.2.3 + + '@stdlib/assert-is-int16array@0.2.3': + dependencies: + '@stdlib/utils-native-class': 0.2.3 + + '@stdlib/assert-is-int32array@0.2.3': + dependencies: + '@stdlib/utils-native-class': 0.2.3 + + '@stdlib/assert-is-int8array@0.2.3': + dependencies: + '@stdlib/utils-native-class': 0.2.3 + + '@stdlib/assert-is-integer@0.2.3': + dependencies: + '@stdlib/assert-is-number': 0.2.3 + '@stdlib/constants-float64-ninf': 0.2.3 + '@stdlib/constants-float64-pinf': 0.2.3 + '@stdlib/math-base-assert-is-integer': 0.2.7 + '@stdlib/utils-define-nonenumerable-read-only-property': 0.2.3 + transitivePeerDependencies: + - supports-color + + '@stdlib/assert-is-little-endian@0.2.3': + dependencies: + '@stdlib/array-uint16': 0.2.3 + '@stdlib/array-uint8': 0.2.3 + + '@stdlib/assert-is-nan@0.2.3': + dependencies: + '@stdlib/assert-is-number': 0.2.3 + '@stdlib/math-base-assert-is-nan': 0.2.3 + '@stdlib/utils-define-nonenumerable-read-only-property': 0.2.3 + transitivePeerDependencies: + - supports-color + + '@stdlib/assert-is-nonnegative-integer@0.2.3': + dependencies: + '@stdlib/assert-is-integer': 0.2.3 + '@stdlib/utils-define-nonenumerable-read-only-property': 0.2.3 + transitivePeerDependencies: + - supports-color + + '@stdlib/assert-is-number-array@0.2.3': + dependencies: + '@stdlib/assert-is-number': 0.2.3 + '@stdlib/assert-tools-array-like-function': 0.2.3 + '@stdlib/utils-define-nonenumerable-read-only-property': 0.2.3 + transitivePeerDependencies: + - supports-color + + '@stdlib/assert-is-number@0.2.3': + dependencies: + '@stdlib/assert-has-tostringtag-support': 0.2.3 + '@stdlib/number-ctor': 0.2.3 + '@stdlib/utils-define-nonenumerable-read-only-property': 0.2.3 + '@stdlib/utils-native-class': 0.2.3 + + '@stdlib/assert-is-object-like@0.2.3': + dependencies: + '@stdlib/assert-tools-array-function': 0.2.3 + '@stdlib/utils-define-nonenumerable-read-only-property': 0.2.3 + + '@stdlib/assert-is-object@0.2.3': + dependencies: + '@stdlib/assert-is-array': 0.2.3 + + '@stdlib/assert-is-plain-object@0.2.3': + dependencies: + '@stdlib/assert-has-own-property': 0.2.3 + '@stdlib/assert-is-function': 0.2.3 + '@stdlib/assert-is-object': 0.2.3 + '@stdlib/utils-get-prototype-of': 0.2.3 + '@stdlib/utils-native-class': 0.2.3 + + '@stdlib/assert-is-positive-integer@0.2.3': + dependencies: + '@stdlib/assert-is-integer': 0.2.3 + '@stdlib/utils-define-nonenumerable-read-only-property': 0.2.3 + transitivePeerDependencies: + - supports-color + + '@stdlib/assert-is-regexp@0.2.3': + dependencies: + '@stdlib/assert-has-tostringtag-support': 0.2.3 + '@stdlib/utils-native-class': 0.2.3 + + '@stdlib/assert-is-same-value@0.2.3': + dependencies: + '@stdlib/assert-is-complex-like': 0.2.4 + '@stdlib/complex-float64-base-assert-is-same-value': 0.0.3 + transitivePeerDependencies: + - supports-color + + '@stdlib/assert-is-string-array@0.2.3': + dependencies: + '@stdlib/assert-is-string': 0.2.3 + '@stdlib/assert-tools-array-function': 0.2.3 + '@stdlib/utils-define-nonenumerable-read-only-property': 0.2.3 + + '@stdlib/assert-is-string@0.2.3': + dependencies: + '@stdlib/assert-has-tostringtag-support': 0.2.3 + '@stdlib/utils-define-nonenumerable-read-only-property': 0.2.3 + '@stdlib/utils-native-class': 0.2.3 + + '@stdlib/assert-is-typed-array-like@0.2.3': + dependencies: + '@stdlib/assert-is-nonnegative-integer': 0.2.3 + '@stdlib/constants-array-max-typed-array-length': 0.2.3 + transitivePeerDependencies: + - supports-color + + '@stdlib/assert-is-uint16array@0.2.3': + dependencies: + '@stdlib/utils-native-class': 0.2.3 + + '@stdlib/assert-is-uint32array@0.2.3': + dependencies: + '@stdlib/utils-native-class': 0.2.3 + + '@stdlib/assert-is-uint8array@0.2.3': + dependencies: + '@stdlib/utils-native-class': 0.2.3 + + '@stdlib/assert-is-uint8clampedarray@0.2.3': + dependencies: + '@stdlib/utils-native-class': 0.2.3 + + '@stdlib/assert-napi-equal-typedarray-types@0.2.3': + dependencies: + '@stdlib/assert-napi-status-ok': 0.2.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/assert-napi-equal-types@0.2.3': + dependencies: + '@stdlib/assert-napi-status-ok': 0.2.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/assert-napi-is-type@0.2.3': + dependencies: + '@stdlib/assert-napi-equal-types': 0.2.3 + '@stdlib/assert-napi-status-ok': 0.2.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/assert-napi-is-typedarray@0.2.3': + dependencies: + '@stdlib/assert-napi-status-ok': 0.2.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/assert-napi-status-ok@0.2.3': {} + + '@stdlib/assert-tools-array-function@0.2.3': + dependencies: + '@stdlib/assert-is-array': 0.2.3 + '@stdlib/error-tools-fmtprodmsg': 0.2.3 + '@stdlib/string-format': 0.2.3 + + '@stdlib/assert-tools-array-like-function@0.2.3': + dependencies: + '@stdlib/assert-is-array-like': 0.2.3 + '@stdlib/error-tools-fmtprodmsg': 0.2.3 + '@stdlib/string-format': 0.2.3 + transitivePeerDependencies: + - supports-color + + '@stdlib/blas-base-gcopy@0.2.3': + dependencies: + '@stdlib/array-base-arraylike2object': 0.2.2 + '@stdlib/utils-define-nonenumerable-read-only-property': 0.2.3 + transitivePeerDependencies: + - supports-color + + '@stdlib/blas-ext-base-gapxsumpw@0.3.1': + dependencies: + '@stdlib/blas-ext-base-gsumpw': 0.3.1 + '@stdlib/strided-base-stride2offset': 0.1.1 + '@stdlib/utils-define-nonenumerable-read-only-property': 0.2.3 + transitivePeerDependencies: + - supports-color + + '@stdlib/blas-ext-base-gsumpw@0.3.1': + dependencies: + '@stdlib/array-base-arraylike2object': 0.2.2 + '@stdlib/math-base-special-floor': 0.2.4 + '@stdlib/strided-base-stride2offset': 0.1.1 + '@stdlib/utils-define-nonenumerable-read-only-property': 0.2.3 + transitivePeerDependencies: + - supports-color + + '@stdlib/boolean-ctor@0.2.3': {} + + '@stdlib/complex-float32-ctor@0.1.1': + dependencies: + '@stdlib/assert-is-number': 0.2.3 + '@stdlib/error-tools-fmtprodmsg': 0.2.3 + '@stdlib/number-float64-base-to-float32': 0.2.3 + '@stdlib/string-format': 0.2.3 + '@stdlib/utils-define-nonenumerable-read-only-property': 0.2.3 + '@stdlib/utils-define-read-only-property': 0.2.3 + + '@stdlib/complex-float32-imag@0.1.2': + dependencies: + '@stdlib/complex-float32-ctor': 0.1.1 + + '@stdlib/complex-float32-real@0.1.2': + dependencies: + '@stdlib/complex-float32-ctor': 0.1.1 + + '@stdlib/complex-float32-reim@0.1.4': + dependencies: + '@stdlib/array-float32': 0.2.3 + '@stdlib/complex-float32-ctor': 0.1.1 + + '@stdlib/complex-float64-base-assert-is-same-value@0.0.3': + dependencies: + '@stdlib/complex-float64-ctor': 0.1.2 + '@stdlib/complex-float64-reim': 0.1.4 + '@stdlib/number-float64-base-assert-is-same-value': 0.2.2 + transitivePeerDependencies: + - supports-color + + '@stdlib/complex-float64-ctor@0.1.2': + dependencies: + '@stdlib/assert-is-number': 0.2.3 + '@stdlib/complex-float32-ctor': 0.1.1 + '@stdlib/error-tools-fmtprodmsg': 0.2.3 + '@stdlib/string-format': 0.2.3 + '@stdlib/utils-define-nonenumerable-read-only-property': 0.2.3 + '@stdlib/utils-define-read-only-property': 0.2.3 + + '@stdlib/complex-float64-imag@0.1.2': + dependencies: + '@stdlib/complex-float64-ctor': 0.1.2 + + '@stdlib/complex-float64-real@0.1.2': + dependencies: + '@stdlib/complex-float64-ctor': 0.1.2 + + '@stdlib/complex-float64-reim@0.1.4': + dependencies: + '@stdlib/array-float64': 0.2.3 + '@stdlib/complex-float64-ctor': 0.1.2 + + '@stdlib/constants-array-max-array-length@0.2.3': {} + + '@stdlib/constants-array-max-typed-array-length@0.2.3': {} + + '@stdlib/constants-float16-eps@0.2.3': {} + + '@stdlib/constants-float16-exponent-bias@0.3.1': {} + + '@stdlib/constants-float16-exponent-mask@0.1.1': {} + + '@stdlib/constants-float16-max@0.2.3': {} + + '@stdlib/constants-float16-num-significand-bits@0.0.2': {} + + '@stdlib/constants-float16-sign-mask@0.1.1': {} + + '@stdlib/constants-float16-significand-mask@0.1.1': {} + + '@stdlib/constants-float16-smallest-normal@0.2.3': {} + + '@stdlib/constants-float32-abs-mask@0.2.3': {} + + '@stdlib/constants-float32-eps@0.2.3': + dependencies: + '@stdlib/number-float64-base-to-float32': 0.2.3 + + '@stdlib/constants-float32-exponent-bias@0.2.3': {} + + '@stdlib/constants-float32-exponent-mask@0.2.3': {} + + '@stdlib/constants-float32-max@0.2.3': {} + + '@stdlib/constants-float32-ninf@0.2.3': + dependencies: + '@stdlib/array-float32': 0.2.3 + '@stdlib/array-uint32': 0.2.3 + + '@stdlib/constants-float32-num-significand-bits@0.1.1': {} + + '@stdlib/constants-float32-pinf@0.2.3': + dependencies: + '@stdlib/array-float32': 0.2.3 + '@stdlib/array-uint32': 0.2.3 + + '@stdlib/constants-float32-sign-mask@0.2.3': {} + + '@stdlib/constants-float32-significand-mask@0.2.4': {} + + '@stdlib/constants-float32-smallest-normal@0.2.3': {} + + '@stdlib/constants-float64-e@0.2.3': {} + + '@stdlib/constants-float64-eps@0.2.3': {} + + '@stdlib/constants-float64-eulergamma@0.2.3': {} + + '@stdlib/constants-float64-exponent-bias@0.2.3': {} + + '@stdlib/constants-float64-fourth-pi@0.2.3': {} + + '@stdlib/constants-float64-gamma-lanczos-g@0.2.3': {} + + '@stdlib/constants-float64-half-ln-two@0.2.3': {} + + '@stdlib/constants-float64-half-pi@0.2.3': {} + + '@stdlib/constants-float64-high-word-abs-mask@0.2.3': {} + + '@stdlib/constants-float64-high-word-exponent-mask@0.2.3': {} + + '@stdlib/constants-float64-high-word-sign-mask@0.2.1': {} + + '@stdlib/constants-float64-high-word-significand-mask@0.2.3': {} + + '@stdlib/constants-float64-ln-sqrt-two-pi@0.2.3': {} + + '@stdlib/constants-float64-ln-two@0.2.3': {} + + '@stdlib/constants-float64-max-base10-exponent@0.2.3': {} + + '@stdlib/constants-float64-max-base2-exponent-subnormal@0.2.1': {} + + '@stdlib/constants-float64-max-base2-exponent@0.2.3': {} + + '@stdlib/constants-float64-max-ln@0.2.3': {} + + '@stdlib/constants-float64-max-nth-factorial@0.1.1': {} + + '@stdlib/constants-float64-max-safe-integer@0.2.3': {} + + '@stdlib/constants-float64-max@0.2.3': {} + + '@stdlib/constants-float64-min-base10-exponent-subnormal@0.2.1': {} + + '@stdlib/constants-float64-min-base10-exponent@0.2.3': {} + + '@stdlib/constants-float64-min-base2-exponent-subnormal@0.2.1': {} + + '@stdlib/constants-float64-min-base2-exponent@0.2.3': {} + + '@stdlib/constants-float64-min-ln@0.2.3': {} + + '@stdlib/constants-float64-ninf@0.2.3': + dependencies: + '@stdlib/number-ctor': 0.2.3 + + '@stdlib/constants-float64-num-high-word-significand-bits@0.1.1': {} + + '@stdlib/constants-float64-pi@0.2.3': {} + + '@stdlib/constants-float64-pinf@0.2.3': {} + + '@stdlib/constants-float64-smallest-normal@0.2.3': {} + + '@stdlib/constants-float64-smallest-subnormal@0.2.3': {} + + '@stdlib/constants-float64-sqrt-eps@0.2.3': {} + + '@stdlib/constants-float64-sqrt-two-pi@0.2.3': {} + + '@stdlib/constants-float64-sqrt-two@0.2.3': {} + + '@stdlib/constants-float64-two-pi@0.2.3': {} + + '@stdlib/constants-int16-max@0.2.3': {} + + '@stdlib/constants-int16-min@0.2.3': {} + + '@stdlib/constants-int32-max@0.3.1': {} + + '@stdlib/constants-int32-min@0.2.3': {} + + '@stdlib/constants-int8-max@0.2.3': {} + + '@stdlib/constants-int8-min@0.2.3': {} + + '@stdlib/constants-uint16-max@0.2.3': {} + + '@stdlib/constants-uint32-max@0.2.3': {} + + '@stdlib/constants-uint8-max@0.2.3': {} + + '@stdlib/error-tools-fmtprodmsg@0.2.3': {} + + '@stdlib/fs-exists@0.2.3': + dependencies: + '@stdlib/utils-define-nonenumerable-read-only-property': 0.2.3 + + '@stdlib/fs-resolve-parent-path@0.2.3': + dependencies: + '@stdlib/assert-has-own-property': 0.2.3 + '@stdlib/assert-is-function': 0.2.3 + '@stdlib/assert-is-plain-object': 0.2.3 + '@stdlib/assert-is-string': 0.2.3 + '@stdlib/error-tools-fmtprodmsg': 0.2.3 + '@stdlib/fs-exists': 0.2.3 + '@stdlib/process-cwd': 0.2.3 + '@stdlib/string-format': 0.2.3 + '@stdlib/utils-define-nonenumerable-read-only-property': 0.2.3 + + '@stdlib/function-ctor@0.2.3': {} + + '@stdlib/math-base-assert-is-even@0.2.5': + dependencies: + '@stdlib/math-base-assert-is-integer': 0.2.7 + '@stdlib/napi-argv': 0.2.3 + '@stdlib/napi-argv-double': 0.2.2 + '@stdlib/napi-create-int32': 0.0.3 + '@stdlib/napi-export': 0.3.1 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-assert-is-finite@0.2.3': + dependencies: + '@stdlib/constants-float64-ninf': 0.2.3 + '@stdlib/constants-float64-pinf': 0.2.3 + '@stdlib/napi-argv': 0.2.3 + '@stdlib/napi-argv-double': 0.2.2 + '@stdlib/napi-create-int32': 0.0.3 + '@stdlib/napi-export': 0.3.1 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-assert-is-finitef@0.2.3': + dependencies: + '@stdlib/constants-float32-ninf': 0.2.3 + '@stdlib/constants-float32-pinf': 0.2.3 + '@stdlib/napi-argv': 0.2.3 + '@stdlib/napi-argv-float': 0.2.3 + '@stdlib/napi-create-int32': 0.0.3 + '@stdlib/napi-export': 0.3.1 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-assert-is-infinite@0.2.3': + dependencies: + '@stdlib/constants-float64-ninf': 0.2.3 + '@stdlib/constants-float64-pinf': 0.2.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-assert-is-integer@0.2.7': + dependencies: + '@stdlib/math-base-special-floor': 0.2.4 + '@stdlib/napi-argv': 0.2.3 + '@stdlib/napi-argv-double': 0.2.2 + '@stdlib/napi-create-int32': 0.0.3 + '@stdlib/napi-export': 0.3.1 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-assert-is-nan@0.2.3': + dependencies: + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-assert-is-nanf@0.2.3': + dependencies: + '@stdlib/napi-argv': 0.2.3 + '@stdlib/napi-argv-float': 0.2.3 + '@stdlib/napi-create-int32': 0.0.3 + '@stdlib/napi-export': 0.3.1 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-assert-is-negative-zero@0.2.3': + dependencies: + '@stdlib/constants-float64-ninf': 0.2.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-assert-is-odd@0.3.2': + dependencies: + '@stdlib/math-base-assert-is-even': 0.2.5 + '@stdlib/napi-argv': 0.2.3 + '@stdlib/napi-argv-double': 0.2.2 + '@stdlib/napi-create-int32': 0.0.3 + '@stdlib/napi-export': 0.3.1 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-assert-is-positive-integer@0.3.2': + dependencies: + '@stdlib/math-base-special-floor': 0.2.4 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-assert-is-positive-zero@0.2.3': + dependencies: + '@stdlib/constants-float64-pinf': 0.2.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-napi-binary@0.3.3': + dependencies: + '@stdlib/complex-float32-ctor': 0.1.1 + '@stdlib/complex-float32-reim': 0.1.4 + '@stdlib/complex-float64-ctor': 0.1.2 + '@stdlib/complex-float64-reim': 0.1.4 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-napi-ternary@0.3.2': + dependencies: + '@stdlib/complex-float32-ctor': 0.1.1 + '@stdlib/complex-float32-reim': 0.1.4 + '@stdlib/complex-float64-ctor': 0.1.2 + '@stdlib/complex-float64-reim': 0.1.4 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-napi-unary@0.2.7': + dependencies: + '@stdlib/complex-float32-ctor': 0.1.1 + '@stdlib/complex-float32-reim': 0.1.4 + '@stdlib/complex-float64-ctor': 0.1.2 + '@stdlib/complex-float64-reim': 0.1.4 + '@stdlib/number-float16-base-to-float64': 0.1.2 + '@stdlib/number-float16-ctor': 0.1.2 + '@stdlib/number-float64-base-to-float16': 0.1.2 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-special-abs@0.2.3': + dependencies: + '@stdlib/constants-float64-high-word-abs-mask': 0.2.3 + '@stdlib/math-base-napi-unary': 0.2.7 + '@stdlib/number-float64-base-to-words': 0.2.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-special-absf@0.2.3': + dependencies: + '@stdlib/constants-float32-abs-mask': 0.2.3 + '@stdlib/math-base-napi-unary': 0.2.7 + '@stdlib/number-float32-base-to-word': 0.2.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-special-acos@0.2.4': + dependencies: + '@stdlib/constants-float64-fourth-pi': 0.2.3 + '@stdlib/math-base-assert-is-nan': 0.2.3 + '@stdlib/math-base-napi-unary': 0.2.7 + '@stdlib/math-base-special-asin': 0.2.4 + '@stdlib/math-base-special-sqrt': 0.2.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-special-asin@0.2.4': + dependencies: + '@stdlib/constants-float64-fourth-pi': 0.2.3 + '@stdlib/math-base-assert-is-nan': 0.2.3 + '@stdlib/math-base-napi-unary': 0.2.7 + '@stdlib/math-base-special-sqrt': 0.2.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-special-beta@0.3.1': + dependencies: + '@stdlib/constants-float64-e': 0.2.3 + '@stdlib/constants-float64-eps': 0.2.3 + '@stdlib/math-base-assert-is-nan': 0.2.3 + '@stdlib/math-base-napi-binary': 0.3.3 + '@stdlib/math-base-special-abs': 0.2.3 + '@stdlib/math-base-special-exp': 0.2.5 + '@stdlib/math-base-special-log1p': 0.2.4 + '@stdlib/math-base-special-pow': 0.3.1 + '@stdlib/math-base-special-sqrt': 0.2.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-special-betainc@0.2.3': + dependencies: + '@stdlib/math-base-special-kernel-betainc': 0.2.3 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-special-binomcoef@0.3.1': + dependencies: + '@stdlib/constants-float64-max-safe-integer': 0.2.3 + '@stdlib/constants-float64-pinf': 0.2.3 + '@stdlib/math-base-assert-is-integer': 0.2.7 + '@stdlib/math-base-assert-is-nan': 0.2.3 + '@stdlib/math-base-assert-is-odd': 0.3.2 + '@stdlib/math-base-napi-binary': 0.3.3 + '@stdlib/math-base-special-floor': 0.2.4 + '@stdlib/math-base-special-gcd': 0.3.2 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-special-ceil@0.2.3': + dependencies: + '@stdlib/math-base-napi-unary': 0.2.7 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-special-copysign@0.2.3': + dependencies: + '@stdlib/constants-float64-high-word-abs-mask': 0.2.3 + '@stdlib/constants-float64-high-word-sign-mask': 0.2.1 + '@stdlib/math-base-napi-binary': 0.3.3 + '@stdlib/number-float64-base-from-words': 0.2.3 + '@stdlib/number-float64-base-get-high-word': 0.2.3 + '@stdlib/number-float64-base-to-words': 0.2.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-special-cos@0.3.1': + dependencies: + '@stdlib/constants-float64-high-word-abs-mask': 0.2.3 + '@stdlib/constants-float64-high-word-exponent-mask': 0.2.3 + '@stdlib/math-base-napi-unary': 0.2.7 + '@stdlib/math-base-special-kernel-cos': 0.2.4 + '@stdlib/math-base-special-kernel-sin': 0.2.4 + '@stdlib/math-base-special-rempio2': 0.3.1 + '@stdlib/number-float64-base-get-high-word': 0.2.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-special-erfc@0.2.5': + dependencies: + '@stdlib/constants-float64-ninf': 0.2.3 + '@stdlib/constants-float64-pinf': 0.2.3 + '@stdlib/math-base-assert-is-nan': 0.2.3 + '@stdlib/math-base-napi-unary': 0.2.7 + '@stdlib/math-base-special-exp': 0.2.5 + '@stdlib/number-float64-base-set-low-word': 0.2.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-special-erfcinv@0.2.4': + dependencies: + '@stdlib/constants-float64-ninf': 0.2.3 + '@stdlib/constants-float64-pinf': 0.2.3 + '@stdlib/math-base-assert-is-nan': 0.2.3 + '@stdlib/math-base-napi-unary': 0.2.7 + '@stdlib/math-base-special-ln': 0.2.5 + '@stdlib/math-base-special-sqrt': 0.2.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-special-exp@0.2.5': + dependencies: + '@stdlib/constants-float64-ninf': 0.2.3 + '@stdlib/constants-float64-pinf': 0.2.3 + '@stdlib/math-base-assert-is-nan': 0.2.3 + '@stdlib/math-base-napi-unary': 0.2.7 + '@stdlib/math-base-special-ldexp': 0.2.5 + '@stdlib/math-base-special-trunc': 0.2.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-special-expm1@0.2.4': + dependencies: + '@stdlib/constants-float64-exponent-bias': 0.2.3 + '@stdlib/constants-float64-half-ln-two': 0.2.3 + '@stdlib/constants-float64-ninf': 0.2.3 + '@stdlib/constants-float64-pinf': 0.2.3 + '@stdlib/math-base-assert-is-nan': 0.2.3 + '@stdlib/math-base-napi-unary': 0.2.7 + '@stdlib/number-float64-base-from-words': 0.2.3 + '@stdlib/number-float64-base-get-high-word': 0.2.3 + '@stdlib/number-float64-base-set-high-word': 0.2.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-special-factorial@0.3.2': + dependencies: + '@stdlib/constants-float64-max-nth-factorial': 0.1.1 + '@stdlib/constants-float64-pinf': 0.2.3 + '@stdlib/math-base-assert-is-integer': 0.2.7 + '@stdlib/math-base-assert-is-nan': 0.2.3 + '@stdlib/math-base-napi-unary': 0.2.7 + '@stdlib/math-base-special-gamma': 0.3.1 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-special-floor@0.2.4': + dependencies: + '@stdlib/math-base-napi-unary': 0.2.7 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-special-fmod@0.1.1': + dependencies: + '@stdlib/constants-float64-exponent-bias': 0.2.3 + '@stdlib/constants-float64-high-word-abs-mask': 0.2.3 + '@stdlib/constants-float64-high-word-exponent-mask': 0.2.3 + '@stdlib/constants-float64-high-word-sign-mask': 0.2.1 + '@stdlib/constants-float64-high-word-significand-mask': 0.2.3 + '@stdlib/constants-float64-min-base2-exponent': 0.2.3 + '@stdlib/math-base-napi-binary': 0.3.3 + '@stdlib/number-float64-base-from-words': 0.2.3 + '@stdlib/number-float64-base-to-words': 0.2.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-special-gamma-delta-ratio@0.3.1': + dependencies: + '@stdlib/constants-float64-e': 0.2.3 + '@stdlib/constants-float64-eps': 0.2.3 + '@stdlib/constants-float64-gamma-lanczos-g': 0.2.3 + '@stdlib/constants-float64-max-nth-factorial': 0.1.1 + '@stdlib/math-base-napi-binary': 0.3.3 + '@stdlib/math-base-special-abs': 0.2.3 + '@stdlib/math-base-special-exp': 0.2.5 + '@stdlib/math-base-special-factorial': 0.3.2 + '@stdlib/math-base-special-floor': 0.2.4 + '@stdlib/math-base-special-gamma': 0.3.1 + '@stdlib/math-base-special-gamma-lanczos-sum': 0.3.2 + '@stdlib/math-base-special-log1p': 0.2.4 + '@stdlib/math-base-special-pow': 0.3.1 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-special-gamma-lanczos-sum-expg-scaled@0.2.2': + dependencies: + '@stdlib/math-base-napi-unary': 0.2.7 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-special-gamma-lanczos-sum@0.3.2': + dependencies: + '@stdlib/math-base-napi-unary': 0.2.7 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-special-gamma1pm1@0.3.1': + dependencies: + '@stdlib/constants-float64-eps': 0.2.3 + '@stdlib/math-base-assert-is-nan': 0.2.3 + '@stdlib/math-base-napi-unary': 0.2.7 + '@stdlib/math-base-special-expm1': 0.2.4 + '@stdlib/math-base-special-gamma': 0.3.1 + '@stdlib/math-base-special-ln': 0.2.5 + '@stdlib/math-base-special-log1p': 0.2.4 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-special-gamma@0.3.1': + dependencies: + '@stdlib/constants-float64-eulergamma': 0.2.3 + '@stdlib/constants-float64-ninf': 0.2.3 + '@stdlib/constants-float64-pi': 0.2.3 + '@stdlib/constants-float64-pinf': 0.2.3 + '@stdlib/constants-float64-sqrt-two-pi': 0.2.3 + '@stdlib/math-base-assert-is-integer': 0.2.7 + '@stdlib/math-base-assert-is-nan': 0.2.3 + '@stdlib/math-base-assert-is-negative-zero': 0.2.3 + '@stdlib/math-base-napi-unary': 0.2.7 + '@stdlib/math-base-special-abs': 0.2.3 + '@stdlib/math-base-special-exp': 0.2.5 + '@stdlib/math-base-special-floor': 0.2.4 + '@stdlib/math-base-special-pow': 0.3.1 + '@stdlib/math-base-special-sin': 0.3.1 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-special-gammainc@0.3.1': + dependencies: + '@stdlib/constants-float32-smallest-normal': 0.2.3 + '@stdlib/constants-float64-e': 0.2.3 + '@stdlib/constants-float64-eps': 0.2.3 + '@stdlib/constants-float64-gamma-lanczos-g': 0.2.3 + '@stdlib/constants-float64-max': 0.2.3 + '@stdlib/constants-float64-max-ln': 0.2.3 + '@stdlib/constants-float64-max-nth-factorial': 0.1.1 + '@stdlib/constants-float64-min-ln': 0.2.3 + '@stdlib/constants-float64-pi': 0.2.3 + '@stdlib/constants-float64-pinf': 0.2.3 + '@stdlib/constants-float64-sqrt-eps': 0.2.3 + '@stdlib/constants-float64-sqrt-two-pi': 0.2.3 + '@stdlib/constants-float64-two-pi': 0.2.3 + '@stdlib/math-base-assert-is-nan': 0.2.3 + '@stdlib/math-base-special-abs': 0.2.3 + '@stdlib/math-base-special-erfc': 0.2.5 + '@stdlib/math-base-special-exp': 0.2.5 + '@stdlib/math-base-special-floor': 0.2.4 + '@stdlib/math-base-special-gamma': 0.3.1 + '@stdlib/math-base-special-gamma-lanczos-sum-expg-scaled': 0.2.2 + '@stdlib/math-base-special-gamma1pm1': 0.3.1 + '@stdlib/math-base-special-gammaln': 0.3.1 + '@stdlib/math-base-special-ln': 0.2.5 + '@stdlib/math-base-special-log1pmx': 0.2.4 + '@stdlib/math-base-special-max': 0.3.1 + '@stdlib/math-base-special-min': 0.2.4 + '@stdlib/math-base-special-pow': 0.3.1 + '@stdlib/math-base-special-powm1': 0.3.2 + '@stdlib/math-base-special-sqrt': 0.2.3 + '@stdlib/math-base-tools-continued-fraction': 0.2.3 + '@stdlib/math-base-tools-evalpoly': 0.2.3 + '@stdlib/math-base-tools-sum-series': 0.2.3 + '@stdlib/napi-argv': 0.2.3 + '@stdlib/napi-argv-bool': 0.1.1 + '@stdlib/napi-argv-double': 0.2.2 + '@stdlib/napi-create-double': 0.0.3 + '@stdlib/napi-export': 0.3.1 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-special-gammaincinv@0.2.3': + dependencies: + '@stdlib/constants-float32-max': 0.2.3 + '@stdlib/constants-float32-smallest-normal': 0.2.3 + '@stdlib/constants-float64-ln-sqrt-two-pi': 0.2.3 + '@stdlib/constants-float64-pinf': 0.2.3 + '@stdlib/constants-float64-sqrt-two-pi': 0.2.3 + '@stdlib/constants-float64-two-pi': 0.2.3 + '@stdlib/math-base-assert-is-nan': 0.2.3 + '@stdlib/math-base-special-abs': 0.2.3 + '@stdlib/math-base-special-erfcinv': 0.2.4 + '@stdlib/math-base-special-exp': 0.2.5 + '@stdlib/math-base-special-gamma': 0.3.1 + '@stdlib/math-base-special-gammainc': 0.3.1 + '@stdlib/math-base-special-gammaln': 0.3.1 + '@stdlib/math-base-special-ln': 0.2.5 + '@stdlib/math-base-special-min': 0.2.4 + '@stdlib/math-base-special-pow': 0.3.1 + '@stdlib/math-base-special-sqrt': 0.2.3 + '@stdlib/math-base-tools-evalpoly': 0.2.3 + debug: 2.6.9 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-special-gammaln@0.3.1': + dependencies: + '@stdlib/constants-float64-pi': 0.2.3 + '@stdlib/constants-float64-pinf': 0.2.3 + '@stdlib/math-base-assert-is-infinite': 0.2.3 + '@stdlib/math-base-assert-is-nan': 0.2.3 + '@stdlib/math-base-napi-unary': 0.2.7 + '@stdlib/math-base-special-abs': 0.2.3 + '@stdlib/math-base-special-ln': 0.2.5 + '@stdlib/math-base-special-sinpi': 0.3.1 + '@stdlib/math-base-special-trunc': 0.2.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-special-gcd@0.3.2': + dependencies: + '@stdlib/constants-float64-ninf': 0.2.3 + '@stdlib/constants-float64-pinf': 0.2.3 + '@stdlib/constants-int32-max': 0.3.1 + '@stdlib/math-base-assert-is-integer': 0.2.7 + '@stdlib/math-base-assert-is-nan': 0.2.3 + '@stdlib/math-base-napi-binary': 0.3.3 + '@stdlib/math-base-special-fmod': 0.1.1 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-special-kernel-betainc@0.2.3': + dependencies: + '@stdlib/constants-float64-e': 0.2.3 + '@stdlib/constants-float64-eps': 0.2.3 + '@stdlib/constants-float64-gamma-lanczos-g': 0.2.3 + '@stdlib/constants-float64-half-pi': 0.2.3 + '@stdlib/constants-float64-max': 0.2.3 + '@stdlib/constants-float64-max-ln': 0.2.3 + '@stdlib/constants-float64-min-ln': 0.2.3 + '@stdlib/constants-float64-pi': 0.2.3 + '@stdlib/constants-float64-smallest-normal': 0.2.3 + '@stdlib/constants-int32-max': 0.3.1 + '@stdlib/math-base-assert-is-nan': 0.2.3 + '@stdlib/math-base-special-abs': 0.2.3 + '@stdlib/math-base-special-asin': 0.2.4 + '@stdlib/math-base-special-beta': 0.3.1 + '@stdlib/math-base-special-binomcoef': 0.3.1 + '@stdlib/math-base-special-exp': 0.2.5 + '@stdlib/math-base-special-expm1': 0.2.4 + '@stdlib/math-base-special-factorial': 0.3.2 + '@stdlib/math-base-special-floor': 0.2.4 + '@stdlib/math-base-special-gamma': 0.3.1 + '@stdlib/math-base-special-gamma-delta-ratio': 0.3.1 + '@stdlib/math-base-special-gamma-lanczos-sum-expg-scaled': 0.2.2 + '@stdlib/math-base-special-gammainc': 0.3.1 + '@stdlib/math-base-special-gammaln': 0.3.1 + '@stdlib/math-base-special-ln': 0.2.5 + '@stdlib/math-base-special-log1p': 0.2.4 + '@stdlib/math-base-special-max': 0.3.1 + '@stdlib/math-base-special-maxabs': 0.3.1 + '@stdlib/math-base-special-min': 0.2.4 + '@stdlib/math-base-special-minabs': 0.2.4 + '@stdlib/math-base-special-pow': 0.3.1 + '@stdlib/math-base-special-sqrt': 0.2.3 + '@stdlib/math-base-tools-continued-fraction': 0.2.3 + '@stdlib/math-base-tools-sum-series': 0.2.3 + '@stdlib/utils-define-nonenumerable-read-only-property': 0.2.3 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-special-kernel-betaincinv@0.2.3': + dependencies: + '@stdlib/constants-float64-eps': 0.2.3 + '@stdlib/constants-float64-half-pi': 0.2.3 + '@stdlib/constants-float64-max': 0.2.3 + '@stdlib/constants-float64-pi': 0.2.3 + '@stdlib/constants-float64-smallest-normal': 0.2.3 + '@stdlib/constants-float64-smallest-subnormal': 0.2.3 + '@stdlib/constants-float64-sqrt-two': 0.2.3 + '@stdlib/math-base-special-abs': 0.2.3 + '@stdlib/math-base-special-acos': 0.2.4 + '@stdlib/math-base-special-asin': 0.2.4 + '@stdlib/math-base-special-beta': 0.3.1 + '@stdlib/math-base-special-betainc': 0.2.3 + '@stdlib/math-base-special-cos': 0.3.1 + '@stdlib/math-base-special-erfcinv': 0.2.4 + '@stdlib/math-base-special-exp': 0.2.5 + '@stdlib/math-base-special-expm1': 0.2.4 + '@stdlib/math-base-special-floor': 0.2.4 + '@stdlib/math-base-special-gamma-delta-ratio': 0.3.1 + '@stdlib/math-base-special-gammaincinv': 0.2.3 + '@stdlib/math-base-special-kernel-betainc': 0.2.3 + '@stdlib/math-base-special-ldexp': 0.2.5 + '@stdlib/math-base-special-ln': 0.2.5 + '@stdlib/math-base-special-log1p': 0.2.4 + '@stdlib/math-base-special-max': 0.3.1 + '@stdlib/math-base-special-min': 0.2.4 + '@stdlib/math-base-special-pow': 0.3.1 + '@stdlib/math-base-special-round': 0.3.1 + '@stdlib/math-base-special-signum': 0.2.3 + '@stdlib/math-base-special-sin': 0.3.1 + '@stdlib/math-base-special-sqrt': 0.2.3 + '@stdlib/math-base-tools-evalpoly': 0.2.3 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-special-kernel-cos@0.2.4': + dependencies: + '@stdlib/math-base-napi-binary': 0.3.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-special-kernel-sin@0.2.4': + dependencies: + '@stdlib/math-base-napi-binary': 0.3.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-special-ldexp@0.2.5': + dependencies: + '@stdlib/constants-float64-exponent-bias': 0.2.3 + '@stdlib/constants-float64-max-base2-exponent': 0.2.3 + '@stdlib/constants-float64-max-base2-exponent-subnormal': 0.2.1 + '@stdlib/constants-float64-min-base2-exponent-subnormal': 0.2.1 + '@stdlib/constants-float64-ninf': 0.2.3 + '@stdlib/constants-float64-pinf': 0.2.3 + '@stdlib/math-base-assert-is-infinite': 0.2.3 + '@stdlib/math-base-assert-is-nan': 0.2.3 + '@stdlib/math-base-special-copysign': 0.2.3 + '@stdlib/number-float64-base-exponent': 0.2.3 + '@stdlib/number-float64-base-from-words': 0.2.3 + '@stdlib/number-float64-base-normalize': 0.2.4 + '@stdlib/number-float64-base-to-words': 0.2.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-special-ln@0.2.5': + dependencies: + '@stdlib/constants-float64-exponent-bias': 0.2.3 + '@stdlib/constants-float64-ninf': 0.2.3 + '@stdlib/math-base-assert-is-nan': 0.2.3 + '@stdlib/math-base-napi-unary': 0.2.7 + '@stdlib/number-float64-base-get-high-word': 0.2.3 + '@stdlib/number-float64-base-set-high-word': 0.2.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-special-log1p@0.2.4': + dependencies: + '@stdlib/constants-float64-exponent-bias': 0.2.3 + '@stdlib/constants-float64-ninf': 0.2.3 + '@stdlib/constants-float64-pinf': 0.2.3 + '@stdlib/math-base-assert-is-nan': 0.2.3 + '@stdlib/math-base-napi-unary': 0.2.7 + '@stdlib/number-float64-base-get-high-word': 0.2.3 + '@stdlib/number-float64-base-set-high-word': 0.2.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-special-log1pmx@0.2.4': + dependencies: + '@stdlib/constants-float64-eps': 0.2.3 + '@stdlib/math-base-napi-unary': 0.2.7 + '@stdlib/math-base-special-abs': 0.2.3 + '@stdlib/math-base-special-ln': 0.2.5 + '@stdlib/math-base-special-log1p': 0.2.4 + '@stdlib/math-base-tools-sum-series': 0.2.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-special-max@0.3.1': + dependencies: + '@stdlib/constants-float64-pinf': 0.2.3 + '@stdlib/math-base-assert-is-nan': 0.2.3 + '@stdlib/math-base-assert-is-positive-zero': 0.2.3 + '@stdlib/math-base-napi-binary': 0.3.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-special-maxabs@0.3.1': + dependencies: + '@stdlib/math-base-napi-binary': 0.3.3 + '@stdlib/math-base-special-abs': 0.2.3 + '@stdlib/math-base-special-max': 0.3.1 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-special-min@0.2.4': + dependencies: + '@stdlib/constants-float64-ninf': 0.2.3 + '@stdlib/math-base-assert-is-nan': 0.2.3 + '@stdlib/math-base-assert-is-negative-zero': 0.2.3 + '@stdlib/math-base-napi-binary': 0.3.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-special-minabs@0.2.4': + dependencies: + '@stdlib/math-base-napi-binary': 0.3.3 + '@stdlib/math-base-special-abs': 0.2.3 + '@stdlib/math-base-special-min': 0.2.4 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-special-pow@0.3.1': + dependencies: + '@stdlib/constants-float64-exponent-bias': 0.2.3 + '@stdlib/constants-float64-high-word-abs-mask': 0.2.3 + '@stdlib/constants-float64-high-word-significand-mask': 0.2.3 + '@stdlib/constants-float64-ln-two': 0.2.3 + '@stdlib/constants-float64-ninf': 0.2.3 + '@stdlib/constants-float64-num-high-word-significand-bits': 0.1.1 + '@stdlib/constants-float64-pinf': 0.2.3 + '@stdlib/math-base-assert-is-infinite': 0.2.3 + '@stdlib/math-base-assert-is-integer': 0.2.7 + '@stdlib/math-base-assert-is-nan': 0.2.3 + '@stdlib/math-base-assert-is-odd': 0.3.2 + '@stdlib/math-base-napi-binary': 0.3.3 + '@stdlib/math-base-special-abs': 0.2.3 + '@stdlib/math-base-special-copysign': 0.2.3 + '@stdlib/math-base-special-ldexp': 0.2.5 + '@stdlib/math-base-special-sqrt': 0.2.3 + '@stdlib/number-float64-base-get-high-word': 0.2.3 + '@stdlib/number-float64-base-set-high-word': 0.2.3 + '@stdlib/number-float64-base-set-low-word': 0.2.3 + '@stdlib/number-float64-base-to-words': 0.2.3 + '@stdlib/number-uint32-base-to-int32': 0.2.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-special-powm1@0.3.2': + dependencies: + '@stdlib/math-base-assert-is-infinite': 0.2.3 + '@stdlib/math-base-assert-is-nan': 0.2.3 + '@stdlib/math-base-napi-binary': 0.3.3 + '@stdlib/math-base-special-abs': 0.2.3 + '@stdlib/math-base-special-expm1': 0.2.4 + '@stdlib/math-base-special-fmod': 0.1.1 + '@stdlib/math-base-special-ln': 0.2.5 + '@stdlib/math-base-special-pow': 0.3.1 + '@stdlib/math-base-special-trunc': 0.2.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-special-rempio2@0.3.1': + dependencies: + '@stdlib/array-base-zeros': 0.2.3 + '@stdlib/constants-float64-high-word-abs-mask': 0.2.3 + '@stdlib/constants-float64-high-word-exponent-mask': 0.2.3 + '@stdlib/constants-float64-high-word-significand-mask': 0.2.3 + '@stdlib/math-base-special-floor': 0.2.4 + '@stdlib/math-base-special-ldexp': 0.2.5 + '@stdlib/math-base-special-round': 0.3.1 + '@stdlib/napi-argv': 0.2.3 + '@stdlib/napi-argv-double': 0.2.2 + '@stdlib/napi-argv-float64array': 0.2.3 + '@stdlib/napi-create-double': 0.0.3 + '@stdlib/napi-export': 0.3.1 + '@stdlib/number-float64-base-from-words': 0.2.3 + '@stdlib/number-float64-base-get-high-word': 0.2.3 + '@stdlib/number-float64-base-get-low-word': 0.2.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-special-round@0.3.1': + dependencies: + '@stdlib/math-base-assert-is-nan': 0.2.3 + '@stdlib/math-base-assert-is-negative-zero': 0.2.3 + '@stdlib/math-base-napi-unary': 0.2.7 + '@stdlib/math-base-special-floor': 0.2.4 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-special-roundn@0.2.3': + dependencies: + '@stdlib/constants-float64-max-base10-exponent': 0.2.3 + '@stdlib/constants-float64-max-safe-integer': 0.2.3 + '@stdlib/constants-float64-min-base10-exponent': 0.2.3 + '@stdlib/constants-float64-min-base10-exponent-subnormal': 0.2.1 + '@stdlib/math-base-assert-is-infinite': 0.2.3 + '@stdlib/math-base-assert-is-nan': 0.2.3 + '@stdlib/math-base-napi-binary': 0.3.3 + '@stdlib/math-base-special-abs': 0.2.3 + '@stdlib/math-base-special-pow': 0.3.1 + '@stdlib/math-base-special-round': 0.3.1 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-special-signum@0.2.3': + dependencies: + '@stdlib/math-base-assert-is-nan': 0.2.3 + '@stdlib/math-base-napi-unary': 0.2.7 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-special-sin@0.3.1': + dependencies: + '@stdlib/constants-float64-high-word-abs-mask': 0.2.3 + '@stdlib/constants-float64-high-word-exponent-mask': 0.2.3 + '@stdlib/math-base-napi-unary': 0.2.7 + '@stdlib/math-base-special-kernel-cos': 0.2.4 + '@stdlib/math-base-special-kernel-sin': 0.2.4 + '@stdlib/math-base-special-rempio2': 0.3.1 + '@stdlib/number-float64-base-get-high-word': 0.2.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-special-sinpi@0.3.1': + dependencies: + '@stdlib/constants-float64-pi': 0.2.3 + '@stdlib/math-base-assert-is-infinite': 0.2.3 + '@stdlib/math-base-assert-is-nan': 0.2.3 + '@stdlib/math-base-napi-unary': 0.2.7 + '@stdlib/math-base-special-abs': 0.2.3 + '@stdlib/math-base-special-copysign': 0.2.3 + '@stdlib/math-base-special-cos': 0.3.1 + '@stdlib/math-base-special-fmod': 0.1.1 + '@stdlib/math-base-special-sin': 0.3.1 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-special-sqrt@0.2.3': + dependencies: + '@stdlib/math-base-napi-unary': 0.2.7 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-special-trunc@0.2.3': + dependencies: + '@stdlib/math-base-napi-unary': 0.2.7 + '@stdlib/math-base-special-ceil': 0.2.3 + '@stdlib/math-base-special-floor': 0.2.4 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-tools-continued-fraction@0.2.3': + dependencies: + '@stdlib/assert-has-generator-support': 0.2.3 + '@stdlib/constants-float32-smallest-normal': 0.2.3 + '@stdlib/constants-float64-eps': 0.2.3 + '@stdlib/math-base-special-abs': 0.2.3 + transitivePeerDependencies: + - supports-color + + '@stdlib/math-base-tools-evalpoly@0.2.3': + dependencies: + '@stdlib/function-ctor': 0.2.3 + '@stdlib/utils-define-nonenumerable-read-only-property': 0.2.3 + + '@stdlib/math-base-tools-sum-series@0.2.3': + dependencies: + '@stdlib/assert-has-generator-support': 0.2.3 + '@stdlib/constants-float64-eps': 0.2.3 + '@stdlib/math-base-special-abs': 0.2.3 + transitivePeerDependencies: + - supports-color + + '@stdlib/napi-argv-bool@0.1.1': + dependencies: + '@stdlib/assert-napi-is-type': 0.2.3 + '@stdlib/assert-napi-status-ok': 0.2.3 + '@stdlib/napi-argv': 0.2.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/napi-argv-double@0.2.2': + dependencies: + '@stdlib/assert-napi-is-type': 0.2.3 + '@stdlib/assert-napi-status-ok': 0.2.3 + '@stdlib/napi-argv': 0.2.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/napi-argv-float64array@0.2.3': + dependencies: + '@stdlib/assert-napi-equal-typedarray-types': 0.2.3 + '@stdlib/assert-napi-is-typedarray': 0.2.3 + '@stdlib/assert-napi-status-ok': 0.2.3 + '@stdlib/napi-argv': 0.2.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/napi-argv-float@0.2.3': + dependencies: + '@stdlib/assert-napi-is-type': 0.2.3 + '@stdlib/assert-napi-status-ok': 0.2.3 + '@stdlib/napi-argv': 0.2.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/napi-argv-int32@0.2.3': + dependencies: + '@stdlib/assert-napi-is-type': 0.2.3 + '@stdlib/assert-napi-status-ok': 0.2.3 + '@stdlib/napi-argv': 0.2.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/napi-argv@0.2.3': + dependencies: + '@stdlib/assert-napi-status-ok': 0.2.3 + + '@stdlib/napi-create-double@0.0.3': + dependencies: + '@stdlib/assert-napi-status-ok': 0.2.3 + '@stdlib/napi-argv': 0.2.3 + '@stdlib/napi-argv-double': 0.2.2 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/napi-create-int32@0.0.3': + dependencies: + '@stdlib/assert-napi-status-ok': 0.2.3 + '@stdlib/napi-argv': 0.2.3 + '@stdlib/napi-argv-int32': 0.2.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/napi-export@0.3.1': {} + + '@stdlib/number-ctor@0.2.3': {} + + '@stdlib/number-float16-base-to-float32@0.1.2': + dependencies: + '@stdlib/constants-float16-exponent-bias': 0.3.1 + '@stdlib/constants-float16-exponent-mask': 0.1.1 + '@stdlib/constants-float16-num-significand-bits': 0.0.2 + '@stdlib/constants-float16-sign-mask': 0.1.1 + '@stdlib/constants-float16-significand-mask': 0.1.1 + '@stdlib/constants-float32-exponent-bias': 0.2.3 + '@stdlib/constants-float32-exponent-mask': 0.2.3 + '@stdlib/constants-float32-num-significand-bits': 0.1.1 + '@stdlib/number-float16-ctor': 0.1.2 + '@stdlib/number-float32-base-to-float16': 0.1.1 + '@stdlib/number-float64-base-to-float16': 0.1.2 + '@stdlib/number-float64-base-to-float32': 0.2.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/number-float16-base-to-float64@0.1.2': + dependencies: + '@stdlib/number-float16-base-to-float32': 0.1.2 + '@stdlib/number-float16-ctor': 0.1.2 + '@stdlib/number-float64-base-to-float16': 0.1.2 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/number-float16-ctor@0.1.2': + dependencies: + '@stdlib/assert-has-to-primitive-symbol-support': 0.1.1 + '@stdlib/assert-is-number': 0.2.3 + '@stdlib/error-tools-fmtprodmsg': 0.2.3 + '@stdlib/number-float64-base-to-float16': 0.1.2 + '@stdlib/string-format': 0.2.3 + '@stdlib/symbol-to-primitive': 0.1.2 + '@stdlib/utils-define-nonenumerable-read-only-property': 0.2.3 + '@stdlib/utils-define-read-only-property': 0.2.3 + transitivePeerDependencies: + - supports-color + + '@stdlib/number-float32-base-exponent@0.2.4': + dependencies: + '@stdlib/constants-float32-exponent-bias': 0.2.3 + '@stdlib/constants-float32-exponent-mask': 0.2.3 + '@stdlib/number-float32-base-to-word': 0.2.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/number-float32-base-to-float16@0.1.1': + dependencies: + '@stdlib/constants-float16-eps': 0.2.3 + '@stdlib/constants-float16-max': 0.2.3 + '@stdlib/constants-float16-smallest-normal': 0.2.3 + '@stdlib/constants-float32-eps': 0.2.3 + '@stdlib/constants-float32-exponent-mask': 0.2.3 + '@stdlib/constants-float32-num-significand-bits': 0.1.1 + '@stdlib/constants-float32-pinf': 0.2.3 + '@stdlib/constants-float32-sign-mask': 0.2.3 + '@stdlib/constants-float32-significand-mask': 0.2.4 + '@stdlib/math-base-assert-is-finitef': 0.2.3 + '@stdlib/math-base-assert-is-nanf': 0.2.3 + '@stdlib/math-base-special-absf': 0.2.3 + '@stdlib/number-float16-ctor': 0.1.2 + '@stdlib/number-float32-base-exponent': 0.2.4 + '@stdlib/number-float64-base-to-float32': 0.2.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/number-float32-base-to-word@0.2.3': + dependencies: + '@stdlib/array-float32': 0.2.3 + '@stdlib/array-uint32': 0.2.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/number-float64-base-assert-is-same-value@0.2.2': + dependencies: + '@stdlib/napi-argv': 0.2.3 + '@stdlib/napi-argv-double': 0.2.2 + '@stdlib/napi-export': 0.3.1 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/number-float64-base-exponent@0.2.3': + dependencies: + '@stdlib/constants-float64-exponent-bias': 0.2.3 + '@stdlib/constants-float64-high-word-exponent-mask': 0.2.3 + '@stdlib/number-float64-base-get-high-word': 0.2.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/number-float64-base-from-words@0.2.3': + dependencies: + '@stdlib/array-float64': 0.2.3 + '@stdlib/array-uint32': 0.2.3 + '@stdlib/assert-is-little-endian': 0.2.3 + '@stdlib/number-float64-base-to-words': 0.2.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/number-float64-base-get-high-word@0.2.3': + dependencies: + '@stdlib/array-float64': 0.2.3 + '@stdlib/array-uint32': 0.2.3 + '@stdlib/assert-is-little-endian': 0.2.3 + '@stdlib/number-float64-base-to-words': 0.2.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/number-float64-base-get-low-word@0.2.3': + dependencies: + '@stdlib/array-float64': 0.2.3 + '@stdlib/array-uint32': 0.2.3 + '@stdlib/assert-is-little-endian': 0.2.3 + '@stdlib/number-float64-base-to-words': 0.2.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/number-float64-base-normalize@0.2.4': + dependencies: + '@stdlib/constants-float64-smallest-normal': 0.2.3 + '@stdlib/math-base-assert-is-infinite': 0.2.3 + '@stdlib/math-base-assert-is-nan': 0.2.3 + '@stdlib/math-base-special-abs': 0.2.3 + '@stdlib/utils-define-nonenumerable-read-only-property': 0.2.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/number-float64-base-set-high-word@0.2.3': + dependencies: + '@stdlib/array-float64': 0.2.3 + '@stdlib/array-uint32': 0.2.3 + '@stdlib/assert-is-little-endian': 0.2.3 + '@stdlib/number-float64-base-to-words': 0.2.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/number-float64-base-set-low-word@0.2.3': + dependencies: + '@stdlib/array-float64': 0.2.3 + '@stdlib/array-uint32': 0.2.3 + '@stdlib/assert-is-little-endian': 0.2.3 + '@stdlib/number-float64-base-to-words': 0.2.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/number-float64-base-to-float16@0.1.2': + dependencies: + '@stdlib/constants-float16-eps': 0.2.3 + '@stdlib/constants-float16-max': 0.2.3 + '@stdlib/constants-float16-smallest-normal': 0.2.3 + '@stdlib/constants-float64-eps': 0.2.3 + '@stdlib/constants-float64-pinf': 0.2.3 + '@stdlib/math-base-assert-is-finite': 0.2.3 + '@stdlib/math-base-special-abs': 0.2.3 + '@stdlib/number-float16-ctor': 0.1.2 + '@stdlib/number-float32-base-to-float16': 0.1.1 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/number-float64-base-to-float32@0.2.3': + dependencies: + '@stdlib/array-float32': 0.2.3 + + '@stdlib/number-float64-base-to-words@0.2.3': + dependencies: + '@stdlib/array-float64': 0.2.3 + '@stdlib/array-uint32': 0.2.3 + '@stdlib/assert-is-little-endian': 0.2.3 + '@stdlib/os-byte-order': 0.2.3 + '@stdlib/os-float-word-order': 0.2.3 + '@stdlib/utils-define-nonenumerable-read-only-property': 0.2.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/number-uint32-base-to-int32@0.2.3': {} + + '@stdlib/object-ctor@0.2.2': {} + + '@stdlib/os-byte-order@0.2.3': + dependencies: + '@stdlib/assert-is-big-endian': 0.2.3 + '@stdlib/assert-is-little-endian': 0.2.3 + + '@stdlib/os-float-word-order@0.2.3': + dependencies: + '@stdlib/os-byte-order': 0.2.3 + + '@stdlib/process-cwd@0.2.3': {} + + '@stdlib/regexp-extended-length-path@0.2.3': + dependencies: + '@stdlib/utils-define-nonenumerable-read-only-property': 0.2.3 + + '@stdlib/regexp-function-name@0.2.3': + dependencies: + '@stdlib/utils-define-nonenumerable-read-only-property': 0.2.3 + + '@stdlib/stats-anova1@0.2.3': + dependencies: + '@stdlib/assert-has-own-property': 0.2.3 + '@stdlib/assert-is-array': 0.2.3 + '@stdlib/assert-is-boolean': 0.2.3 + '@stdlib/assert-is-nan': 0.2.3 + '@stdlib/assert-is-number': 0.2.3 + '@stdlib/assert-is-number-array': 0.2.3 + '@stdlib/assert-is-plain-object': 0.2.3 + '@stdlib/assert-is-positive-integer': 0.2.3 + '@stdlib/assert-is-typed-array-like': 0.2.3 + '@stdlib/error-tools-fmtprodmsg': 0.2.3 + '@stdlib/math-base-special-max': 0.3.1 + '@stdlib/math-base-special-roundn': 0.2.3 + '@stdlib/stats-base-dists-f-cdf': 0.2.3 + '@stdlib/stats-incr-stdev': 0.2.3 + '@stdlib/string-format': 0.2.3 + '@stdlib/string-repeat': 0.2.3 + '@stdlib/utils-define-read-only-property': 0.2.3 + transitivePeerDependencies: + - supports-color + + '@stdlib/stats-base-dists-chisquare-cdf@0.3.1': + dependencies: + '@stdlib/math-base-napi-binary': 0.3.3 + '@stdlib/stats-base-dists-gamma-cdf': 0.3.1 + '@stdlib/utils-define-nonenumerable-read-only-property': 0.2.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/stats-base-dists-degenerate-cdf@0.3.1': + dependencies: + '@stdlib/math-base-assert-is-nan': 0.2.3 + '@stdlib/math-base-napi-binary': 0.3.3 + '@stdlib/utils-constant-function': 0.2.3 + '@stdlib/utils-define-nonenumerable-read-only-property': 0.2.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/stats-base-dists-f-cdf@0.2.3': + dependencies: + '@stdlib/constants-float64-pinf': 0.2.3 + '@stdlib/math-base-assert-is-nan': 0.2.3 + '@stdlib/math-base-special-betainc': 0.2.3 + '@stdlib/utils-constant-function': 0.2.3 + '@stdlib/utils-define-nonenumerable-read-only-property': 0.2.3 + transitivePeerDependencies: + - supports-color + + '@stdlib/stats-base-dists-gamma-cdf@0.3.1': + dependencies: + '@stdlib/constants-float64-pinf': 0.2.3 + '@stdlib/math-base-assert-is-nan': 0.2.3 + '@stdlib/math-base-napi-ternary': 0.3.2 + '@stdlib/math-base-special-gammainc': 0.3.1 + '@stdlib/stats-base-dists-degenerate-cdf': 0.3.1 + '@stdlib/utils-constant-function': 0.2.3 + '@stdlib/utils-define-nonenumerable-read-only-property': 0.2.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/stats-base-dists-normal-cdf@0.3.1': + dependencies: + '@stdlib/math-base-assert-is-nan': 0.2.3 + '@stdlib/math-base-napi-ternary': 0.3.2 + '@stdlib/math-base-special-erfc': 0.2.5 + '@stdlib/math-base-special-sqrt': 0.2.3 + '@stdlib/stats-base-dists-degenerate-cdf': 0.3.1 + '@stdlib/utils-constant-function': 0.2.3 + '@stdlib/utils-define-nonenumerable-read-only-property': 0.2.3 + '@stdlib/utils-library-manifest': 0.2.4 + transitivePeerDependencies: + - supports-color + + '@stdlib/stats-base-dists-signrank-cdf@0.2.3': + dependencies: + '@stdlib/constants-float64-ln-two': 0.2.3 + '@stdlib/math-base-assert-is-finite': 0.2.3 + '@stdlib/math-base-assert-is-nan': 0.2.3 + '@stdlib/math-base-assert-is-positive-integer': 0.3.2 + '@stdlib/math-base-special-exp': 0.2.5 + '@stdlib/math-base-special-round': 0.3.1 + '@stdlib/utils-constant-function': 0.2.3 + '@stdlib/utils-define-nonenumerable-read-only-property': 0.2.3 + '@stdlib/utils-memoize': 0.2.3 + transitivePeerDependencies: + - supports-color + + '@stdlib/stats-base-dists-t-cdf@0.2.3': + dependencies: + '@stdlib/math-base-assert-is-nan': 0.2.3 + '@stdlib/math-base-special-betainc': 0.2.3 + '@stdlib/math-base-special-pow': 0.3.1 + '@stdlib/utils-constant-function': 0.2.3 + '@stdlib/utils-define-nonenumerable-read-only-property': 0.2.3 + transitivePeerDependencies: + - supports-color + + '@stdlib/stats-base-dists-t-quantile@0.2.3': + dependencies: + '@stdlib/math-base-assert-is-nan': 0.2.3 + '@stdlib/math-base-special-kernel-betaincinv': 0.2.3 + '@stdlib/math-base-special-signum': 0.2.3 + '@stdlib/math-base-special-sqrt': 0.2.3 + '@stdlib/utils-constant-function': 0.2.3 + '@stdlib/utils-define-nonenumerable-read-only-property': 0.2.3 + transitivePeerDependencies: + - supports-color + + '@stdlib/stats-incr-stdev@0.2.3': + dependencies: + '@stdlib/assert-is-number': 0.2.3 + '@stdlib/error-tools-fmtprodmsg': 0.2.3 + '@stdlib/math-base-assert-is-nan': 0.2.3 + '@stdlib/math-base-special-sqrt': 0.2.3 + '@stdlib/string-format': 0.2.3 + transitivePeerDependencies: + - supports-color + + '@stdlib/stats-kruskal-test@0.2.3': + dependencies: + '@stdlib/array-base-incrspace': 0.2.3 + '@stdlib/assert-has-own-property': 0.2.3 + '@stdlib/assert-is-array': 0.2.3 + '@stdlib/assert-is-boolean': 0.2.3 + '@stdlib/assert-is-collection': 0.2.3 + '@stdlib/assert-is-nan': 0.2.3 + '@stdlib/assert-is-number': 0.2.3 + '@stdlib/assert-is-plain-object': 0.2.3 + '@stdlib/assert-is-positive-integer': 0.2.3 + '@stdlib/error-tools-fmtprodmsg': 0.2.3 + '@stdlib/math-base-special-pow': 0.3.1 + '@stdlib/math-base-special-roundn': 0.2.3 + '@stdlib/stats-base-dists-chisquare-cdf': 0.3.1 + '@stdlib/stats-ranks': 0.2.3 + '@stdlib/string-format': 0.2.3 + '@stdlib/utils-count-by': 0.2.3 + '@stdlib/utils-define-read-only-property': 0.2.3 + '@stdlib/utils-identity-function': 0.2.3 + '@stdlib/utils-keys': 0.2.3 + transitivePeerDependencies: + - supports-color + + '@stdlib/stats-ranks@0.2.3': + dependencies: + '@stdlib/array-base-assert-contains': 0.2.3 + '@stdlib/array-base-filled': 0.2.3 + '@stdlib/array-base-zeros': 0.2.3 + '@stdlib/assert-contains': 0.3.1 + '@stdlib/assert-has-own-property': 0.2.3 + '@stdlib/assert-is-array': 0.2.3 + '@stdlib/assert-is-collection': 0.2.3 + '@stdlib/assert-is-object': 0.2.3 + '@stdlib/assert-is-string': 0.2.3 + '@stdlib/error-tools-fmtprodmsg': 0.2.3 + '@stdlib/string-format': 0.2.3 + transitivePeerDependencies: + - supports-color + + '@stdlib/stats-strided-mean@0.1.1': + dependencies: + '@stdlib/stats-strided-meanpn': 0.1.1 + '@stdlib/utils-define-nonenumerable-read-only-property': 0.2.3 + transitivePeerDependencies: + - supports-color + + '@stdlib/stats-strided-meanpn@0.1.1': + dependencies: + '@stdlib/array-base-arraylike2object': 0.2.2 + '@stdlib/blas-ext-base-gapxsumpw': 0.3.1 + '@stdlib/blas-ext-base-gsumpw': 0.3.1 + '@stdlib/strided-base-stride2offset': 0.1.1 + '@stdlib/utils-define-nonenumerable-read-only-property': 0.2.3 + transitivePeerDependencies: + - supports-color + + '@stdlib/stats-strided-variance@0.1.1': + dependencies: + '@stdlib/stats-strided-variancepn': 0.1.1 + '@stdlib/utils-define-nonenumerable-read-only-property': 0.2.3 + transitivePeerDependencies: + - supports-color + + '@stdlib/stats-strided-variancepn@0.1.1': + dependencies: + '@stdlib/array-base-arraylike2object': 0.2.2 + '@stdlib/blas-ext-base-gsumpw': 0.3.1 + '@stdlib/strided-base-stride2offset': 0.1.1 + '@stdlib/utils-define-nonenumerable-read-only-property': 0.2.3 + transitivePeerDependencies: + - supports-color + + '@stdlib/stats-ttest2@0.2.3': + dependencies: + '@stdlib/assert-has-own-property': 0.2.3 + '@stdlib/assert-is-boolean': 0.2.3 + '@stdlib/assert-is-nan': 0.2.3 + '@stdlib/assert-is-number': 0.2.3 + '@stdlib/assert-is-number-array': 0.2.3 + '@stdlib/assert-is-plain-object': 0.2.3 + '@stdlib/assert-is-positive-integer': 0.2.3 + '@stdlib/assert-is-string': 0.2.3 + '@stdlib/assert-is-typed-array-like': 0.2.3 + '@stdlib/constants-float64-ninf': 0.2.3 + '@stdlib/constants-float64-pinf': 0.2.3 + '@stdlib/error-tools-fmtprodmsg': 0.2.3 + '@stdlib/math-base-special-abs': 0.2.3 + '@stdlib/math-base-special-pow': 0.3.1 + '@stdlib/math-base-special-roundn': 0.2.3 + '@stdlib/math-base-special-sqrt': 0.2.3 + '@stdlib/stats-base-dists-t-cdf': 0.2.3 + '@stdlib/stats-base-dists-t-quantile': 0.2.3 + '@stdlib/stats-strided-mean': 0.1.1 + '@stdlib/stats-strided-variance': 0.1.1 + '@stdlib/string-format': 0.2.3 + '@stdlib/utils-define-read-only-property': 0.2.3 + transitivePeerDependencies: + - supports-color + + '@stdlib/stats-ttest@0.2.3': + dependencies: + '@stdlib/array-float64': 0.2.3 + '@stdlib/assert-contains': 0.3.1 + '@stdlib/assert-has-own-property': 0.2.3 + '@stdlib/assert-is-boolean': 0.2.3 + '@stdlib/assert-is-number': 0.2.3 + '@stdlib/assert-is-number-array': 0.2.3 + '@stdlib/assert-is-plain-object': 0.2.3 + '@stdlib/assert-is-positive-integer': 0.2.3 + '@stdlib/assert-is-string': 0.2.3 + '@stdlib/assert-is-typed-array-like': 0.2.3 + '@stdlib/blas-base-gcopy': 0.2.3 + '@stdlib/constants-float64-ninf': 0.2.3 + '@stdlib/constants-float64-pinf': 0.2.3 + '@stdlib/error-tools-fmtprodmsg': 0.2.3 + '@stdlib/math-base-assert-is-nan': 0.2.3 + '@stdlib/math-base-special-abs': 0.2.3 + '@stdlib/math-base-special-roundn': 0.2.3 + '@stdlib/math-base-special-sqrt': 0.2.3 + '@stdlib/stats-base-dists-t-cdf': 0.2.3 + '@stdlib/stats-base-dists-t-quantile': 0.2.3 + '@stdlib/stats-strided-mean': 0.1.1 + '@stdlib/stats-strided-variance': 0.1.1 + '@stdlib/string-format': 0.2.3 + '@stdlib/utils-define-read-only-property': 0.2.3 + transitivePeerDependencies: + - supports-color + + '@stdlib/stats-wilcoxon@0.2.3': + dependencies: + '@stdlib/array-base-assert-contains': 0.2.3 + '@stdlib/array-float64': 0.2.3 + '@stdlib/assert-has-own-property': 0.2.3 + '@stdlib/assert-is-boolean': 0.2.3 + '@stdlib/assert-is-nan': 0.2.3 + '@stdlib/assert-is-number': 0.2.3 + '@stdlib/assert-is-number-array': 0.2.3 + '@stdlib/assert-is-plain-object': 0.2.3 + '@stdlib/assert-is-positive-integer': 0.2.3 + '@stdlib/assert-is-string': 0.2.3 + '@stdlib/assert-is-typed-array-like': 0.2.3 + '@stdlib/error-tools-fmtprodmsg': 0.2.3 + '@stdlib/math-base-special-abs': 0.2.3 + '@stdlib/math-base-special-roundn': 0.2.3 + '@stdlib/math-base-special-signum': 0.2.3 + '@stdlib/math-base-special-sqrt': 0.2.3 + '@stdlib/stats-base-dists-normal-cdf': 0.3.1 + '@stdlib/stats-base-dists-signrank-cdf': 0.2.3 + '@stdlib/stats-ranks': 0.2.3 + '@stdlib/string-format': 0.2.3 + '@stdlib/utils-define-read-only-property': 0.2.3 + '@stdlib/utils-tabulate': 0.2.3 + transitivePeerDependencies: + - supports-color + + '@stdlib/strided-base-reinterpret-boolean@0.0.3': + dependencies: + '@stdlib/array-uint8': 0.2.3 + + '@stdlib/strided-base-reinterpret-complex128@0.2.3': + dependencies: + '@stdlib/array-float64': 0.2.3 + + '@stdlib/strided-base-reinterpret-complex64@0.2.2': + dependencies: + '@stdlib/array-float32': 0.2.3 + + '@stdlib/strided-base-reinterpret-complex@0.1.3': + dependencies: + '@stdlib/array-base-assert-is-complex128array': 0.2.3 + '@stdlib/array-base-assert-is-complex64array': 0.2.3 + '@stdlib/error-tools-fmtprodmsg': 0.2.3 + '@stdlib/strided-base-reinterpret-complex128': 0.2.3 + '@stdlib/strided-base-reinterpret-complex64': 0.2.2 + '@stdlib/string-format': 0.2.3 + + '@stdlib/strided-base-stride2offset@0.1.1': {} + + '@stdlib/string-base-format-interpolate@0.2.4': {} + + '@stdlib/string-base-format-tokenize@0.2.4': {} + + '@stdlib/string-base-lowercase@0.4.1': {} + + '@stdlib/string-base-repeat@0.2.3': {} + + '@stdlib/string-base-replace@0.2.3': {} + + '@stdlib/string-format@0.2.3': + dependencies: + '@stdlib/string-base-format-interpolate': 0.2.4 + '@stdlib/string-base-format-tokenize': 0.2.4 + + '@stdlib/string-repeat@0.2.3': + dependencies: + '@stdlib/assert-is-nonnegative-integer': 0.2.3 + '@stdlib/assert-is-string': 0.2.3 + '@stdlib/error-tools-fmtprodmsg': 0.2.3 + '@stdlib/string-base-repeat': 0.2.3 + '@stdlib/string-format': 0.2.3 + transitivePeerDependencies: + - supports-color + + '@stdlib/string-replace@0.2.3': + dependencies: + '@stdlib/assert-is-function': 0.2.3 + '@stdlib/assert-is-regexp': 0.2.3 + '@stdlib/assert-is-string': 0.2.3 + '@stdlib/error-tools-fmtprodmsg': 0.2.3 + '@stdlib/string-base-replace': 0.2.3 + '@stdlib/string-format': 0.2.3 + '@stdlib/utils-escape-regexp-string': 0.2.3 + + '@stdlib/symbol-ctor@0.2.3': {} + + '@stdlib/symbol-iterator@0.2.3': + dependencies: + '@stdlib/assert-has-iterator-symbol-support': 0.2.3 + + '@stdlib/symbol-to-primitive@0.1.2': + dependencies: + '@stdlib/assert-has-to-primitive-symbol-support': 0.1.1 + '@stdlib/symbol-ctor': 0.2.3 + + '@stdlib/types@0.4.3': {} + + '@stdlib/utils-constant-function@0.2.3': {} + + '@stdlib/utils-constructor-name@0.2.3': + dependencies: + '@stdlib/assert-is-buffer': 0.2.3 + '@stdlib/regexp-function-name': 0.2.3 + '@stdlib/utils-native-class': 0.2.3 + + '@stdlib/utils-convert-path@0.2.3': + dependencies: + '@stdlib/assert-is-string': 0.2.3 + '@stdlib/error-tools-fmtprodmsg': 0.2.3 + '@stdlib/regexp-extended-length-path': 0.2.3 + '@stdlib/string-base-lowercase': 0.4.1 + '@stdlib/string-format': 0.2.3 + '@stdlib/string-replace': 0.2.3 + + '@stdlib/utils-count-by@0.2.3': + dependencies: + '@stdlib/assert-has-own-property': 0.2.3 + '@stdlib/assert-is-collection': 0.2.3 + '@stdlib/assert-is-function': 0.2.3 + '@stdlib/assert-is-plain-object': 0.2.3 + '@stdlib/error-tools-fmtprodmsg': 0.2.3 + '@stdlib/string-format': 0.2.3 + transitivePeerDependencies: + - supports-color + + '@stdlib/utils-define-nonenumerable-read-only-accessor@0.2.4': + dependencies: + '@stdlib/types': 0.4.3 + '@stdlib/utils-define-property': 0.2.5 + + '@stdlib/utils-define-nonenumerable-read-only-property@0.2.3': + dependencies: + '@stdlib/types': 0.4.3 + '@stdlib/utils-define-property': 0.2.5 + + '@stdlib/utils-define-property@0.2.5': + dependencies: + '@stdlib/error-tools-fmtprodmsg': 0.2.3 + '@stdlib/string-format': 0.2.3 + + '@stdlib/utils-define-read-only-property@0.2.3': + dependencies: + '@stdlib/utils-define-property': 0.2.5 - '@smithy/util-defaults-mode-node@4.0.27': + '@stdlib/utils-escape-regexp-string@0.2.3': dependencies: - '@smithy/config-resolver': 4.1.5 - '@smithy/credential-provider-imds': 4.0.7 - '@smithy/node-config-provider': 4.1.4 - '@smithy/property-provider': 4.0.5 - '@smithy/smithy-client': 4.5.0 - '@smithy/types': 4.3.2 - tslib: 2.8.1 + '@stdlib/assert-is-string': 0.2.3 + '@stdlib/error-tools-fmtprodmsg': 0.2.3 + '@stdlib/string-format': 0.2.3 - '@smithy/util-endpoints@3.0.7': + '@stdlib/utils-eval@0.2.3': {} + + '@stdlib/utils-get-prototype-of@0.2.3': dependencies: - '@smithy/node-config-provider': 4.1.4 - '@smithy/types': 4.3.2 - tslib: 2.8.1 + '@stdlib/assert-is-function': 0.2.3 + '@stdlib/object-ctor': 0.2.2 + '@stdlib/utils-native-class': 0.2.3 - '@smithy/util-hex-encoding@4.0.0': + '@stdlib/utils-global@0.2.3': dependencies: - tslib: 2.8.1 + '@stdlib/assert-is-boolean': 0.2.3 + '@stdlib/error-tools-fmtprodmsg': 0.2.3 + '@stdlib/string-format': 0.2.3 - '@smithy/util-middleware@4.0.5': + '@stdlib/utils-identity-function@0.2.3': {} + + '@stdlib/utils-index-of@0.2.3': dependencies: - '@smithy/types': 4.3.2 - tslib: 2.8.1 + '@stdlib/assert-is-collection': 0.2.3 + '@stdlib/assert-is-integer': 0.2.3 + '@stdlib/assert-is-nan': 0.2.3 + '@stdlib/assert-is-string': 0.2.3 + '@stdlib/error-tools-fmtprodmsg': 0.2.3 + '@stdlib/string-format': 0.2.3 + transitivePeerDependencies: + - supports-color - '@smithy/util-retry@4.0.7': + '@stdlib/utils-keys@0.2.3': dependencies: - '@smithy/service-error-classification': 4.0.7 - '@smithy/types': 4.3.2 - tslib: 2.8.1 + '@stdlib/assert-has-own-property': 0.2.3 + '@stdlib/assert-is-arguments': 0.2.3 + '@stdlib/assert-is-enumerable-property': 0.2.3 + '@stdlib/assert-is-object-like': 0.2.3 + '@stdlib/utils-index-of': 0.2.3 + '@stdlib/utils-noop': 0.2.3 + '@stdlib/utils-type-of': 0.2.3 + transitivePeerDependencies: + - supports-color - '@smithy/util-stream@4.2.4': + '@stdlib/utils-library-manifest@0.2.4': dependencies: - '@smithy/fetch-http-handler': 5.1.1 - '@smithy/node-http-handler': 4.1.1 - '@smithy/types': 4.3.2 - '@smithy/util-base64': 4.0.0 - '@smithy/util-buffer-from': 4.0.0 - '@smithy/util-hex-encoding': 4.0.0 - '@smithy/util-utf8': 4.0.0 - tslib: 2.8.1 + '@stdlib/fs-resolve-parent-path': 0.2.3 + '@stdlib/utils-convert-path': 0.2.3 + debug: 2.6.9 + resolve: 1.22.10 + transitivePeerDependencies: + - supports-color - '@smithy/util-uri-escape@4.0.0': + '@stdlib/utils-memoize@0.2.3': dependencies: - tslib: 2.8.1 + '@stdlib/assert-has-own-property': 0.2.3 + '@stdlib/assert-is-function': 0.2.3 + '@stdlib/error-tools-fmtprodmsg': 0.2.3 + '@stdlib/string-format': 0.2.3 + '@stdlib/utils-define-nonenumerable-read-only-property': 0.2.3 + '@stdlib/utils-identity-function': 0.2.3 - '@smithy/util-utf8@2.3.0': + '@stdlib/utils-native-class@0.2.3': dependencies: - '@smithy/util-buffer-from': 2.2.0 - tslib: 2.8.1 + '@stdlib/assert-has-own-property': 0.2.3 + '@stdlib/assert-has-tostringtag-support': 0.2.3 + '@stdlib/symbol-ctor': 0.2.3 - '@smithy/util-utf8@4.0.0': + '@stdlib/utils-noop@0.2.3': {} + + '@stdlib/utils-tabulate@0.2.3': dependencies: - '@smithy/util-buffer-from': 4.0.0 - tslib: 2.8.1 + '@stdlib/assert-is-collection': 0.2.3 + '@stdlib/error-tools-fmtprodmsg': 0.2.3 + '@stdlib/string-format': 0.2.3 + '@stdlib/utils-index-of': 0.2.3 + transitivePeerDependencies: + - supports-color - '@smithy/util-waiter@4.0.7': + '@stdlib/utils-type-of@0.2.3': dependencies: - '@smithy/abort-controller': 4.0.5 - '@smithy/types': 4.3.2 - tslib: 2.8.1 + '@stdlib/utils-constructor-name': 0.2.3 + '@stdlib/utils-global': 0.2.3 - '@standard-schema/spec@1.1.0': {} + '@turbo/darwin-64@2.10.10': + optional: true + + '@turbo/darwin-arm64@2.10.10': + optional: true + + '@turbo/linux-64@2.10.10': + optional: true + + '@turbo/linux-arm64@2.10.10': + optional: true + + '@turbo/windows-64@2.10.10': + optional: true + + '@turbo/windows-arm64@2.10.10': + optional: true '@tybys/wasm-util@0.10.3': dependencies: @@ -7633,6 +11780,8 @@ snapshots: '@types/d3-format@3.0.4': {} + '@types/d3-hierarchy@3.1.7': {} + '@types/d3-scale@4.0.9': dependencies: '@types/d3-time': 3.0.4 @@ -7693,35 +11842,19 @@ snapshots: vite: 8.0.8(@types/node@25.3.2)(yaml@2.8.1) vue: 3.5.24(typescript@5.6.3) - '@vitejs/plugin-vue@6.0.6(vite@8.0.8(@types/node@25.3.2)(yaml@2.8.1))(vue@3.5.25(typescript@5.6.3))': + '@vitejs/plugin-vue@6.0.6(vite@8.0.8(@types/node@25.3.2)(yaml@2.8.1))(vue@3.5.41(typescript@5.6.3))': dependencies: '@rolldown/pluginutils': 1.0.0-rc.13 vite: 8.0.8(@types/node@25.3.2)(yaml@2.8.1) - vue: 3.5.25(typescript@5.6.3) + vue: 3.5.41(typescript@5.6.3) - '@vitejs/plugin-vue@6.0.6(vite@8.0.8(@types/node@25.3.2)(yaml@2.8.1))(vue@3.5.25(typescript@5.9.3))': + '@vitejs/plugin-vue@6.0.6(vite@8.0.8(@types/node@25.3.2)(yaml@2.8.1))(vue@3.5.41(typescript@5.9.3))': dependencies: '@rolldown/pluginutils': 1.0.0-rc.13 vite: 8.0.8(@types/node@25.3.2)(yaml@2.8.1) - vue: 3.5.25(typescript@5.9.3) - - '@vitest/coverage-istanbul@4.1.4(vitest@4.0.18(@types/node@25.3.2)(lightningcss@1.32.0)(yaml@2.8.1))': - dependencies: - '@babel/core': 7.29.0 - '@istanbuljs/schema': 0.1.3 - '@jridgewell/gen-mapping': 0.3.13 - '@jridgewell/trace-mapping': 0.3.31 - istanbul-lib-coverage: 3.2.2 - istanbul-lib-report: 3.0.1 - istanbul-reports: 3.2.0 - magicast: 0.5.2 - obug: 2.1.3 - tinyrainbow: 3.1.0 - vitest: 4.0.18(@types/node@25.3.2)(lightningcss@1.32.0)(yaml@2.8.1) - transitivePeerDependencies: - - supports-color + vue: 3.5.41(typescript@5.9.3) - '@vitest/coverage-istanbul@4.1.4(vitest@4.1.4)': + '@vitest/coverage-istanbul@4.1.4(vitest@4.1.10)': dependencies: '@babel/core': 7.29.0 '@istanbuljs/schema': 0.1.3 @@ -7733,95 +11866,48 @@ snapshots: magicast: 0.5.2 obug: 2.1.3 tinyrainbow: 3.1.0 - vitest: 4.1.4(@types/node@25.3.2)(@vitest/coverage-istanbul@4.1.4)(vite@7.2.7(@types/node@25.3.2)(lightningcss@1.32.0)(yaml@2.8.1)) + vitest: 4.1.10(@types/node@25.3.2)(@vitest/coverage-istanbul@4.1.4)(vite@8.0.8(@types/node@25.3.2)(yaml@2.8.1)) transitivePeerDependencies: - supports-color - '@vitest/expect@4.0.18': - dependencies: - '@standard-schema/spec': 1.1.0 - '@types/chai': 5.2.3 - '@vitest/spy': 4.0.18 - '@vitest/utils': 4.0.18 - chai: 6.2.2 - tinyrainbow: 3.1.0 - - '@vitest/expect@4.1.4': + '@vitest/expect@4.1.10': dependencies: '@standard-schema/spec': 1.1.0 '@types/chai': 5.2.3 - '@vitest/spy': 4.1.4 - '@vitest/utils': 4.1.4 + '@vitest/spy': 4.1.10 + '@vitest/utils': 4.1.10 chai: 6.2.2 tinyrainbow: 3.1.0 - '@vitest/mocker@4.0.18(vite@7.2.7(@types/node@25.3.2)(lightningcss@1.32.0)(yaml@2.8.1))': - dependencies: - '@vitest/spy': 4.0.18 - estree-walker: 3.0.3 - magic-string: 0.30.21 - optionalDependencies: - vite: 7.2.7(@types/node@25.3.2)(lightningcss@1.32.0)(yaml@2.8.1) - - '@vitest/mocker@4.1.4(vite@7.2.7(@types/node@25.3.2)(lightningcss@1.32.0)(yaml@2.8.1))': - dependencies: - '@vitest/spy': 4.1.4 - estree-walker: 3.0.3 - magic-string: 0.30.21 - optionalDependencies: - vite: 7.2.7(@types/node@25.3.2)(lightningcss@1.32.0)(yaml@2.8.1) - - '@vitest/mocker@4.1.4(vite@8.0.8(@types/node@25.3.2)(yaml@2.8.1))': + '@vitest/mocker@4.1.10(vite@8.0.8(@types/node@25.3.2)(yaml@2.8.1))': dependencies: - '@vitest/spy': 4.1.4 + '@vitest/spy': 4.1.10 estree-walker: 3.0.3 magic-string: 0.30.21 optionalDependencies: vite: 8.0.8(@types/node@25.3.2)(yaml@2.8.1) - '@vitest/pretty-format@4.0.18': - dependencies: - tinyrainbow: 3.1.0 - - '@vitest/pretty-format@4.1.4': + '@vitest/pretty-format@4.1.10': dependencies: tinyrainbow: 3.1.0 - '@vitest/runner@4.0.18': - dependencies: - '@vitest/utils': 4.0.18 - pathe: 2.0.3 - - '@vitest/runner@4.1.4': + '@vitest/runner@4.1.10': dependencies: - '@vitest/utils': 4.1.4 - pathe: 2.0.3 - - '@vitest/snapshot@4.0.18': - dependencies: - '@vitest/pretty-format': 4.0.18 - magic-string: 0.30.21 + '@vitest/utils': 4.1.10 pathe: 2.0.3 - '@vitest/snapshot@4.1.4': + '@vitest/snapshot@4.1.10': dependencies: - '@vitest/pretty-format': 4.1.4 - '@vitest/utils': 4.1.4 + '@vitest/pretty-format': 4.1.10 + '@vitest/utils': 4.1.10 magic-string: 0.30.21 pathe: 2.0.3 - '@vitest/spy@4.0.18': {} + '@vitest/spy@4.1.10': {} - '@vitest/spy@4.1.4': {} - - '@vitest/utils@4.0.18': - dependencies: - '@vitest/pretty-format': 4.0.18 - tinyrainbow: 3.1.0 - - '@vitest/utils@4.1.4': + '@vitest/utils@4.1.10': dependencies: - '@vitest/pretty-format': 4.1.4 + '@vitest/pretty-format': 4.1.10 convert-source-map: 2.0.0 tinyrainbow: 3.1.0 @@ -7839,17 +11925,17 @@ snapshots: '@vue/compiler-core@3.5.24': dependencies: - '@babel/parser': 7.29.0 + '@babel/parser': 7.29.8 '@vue/shared': 3.5.24 entities: 4.5.0 estree-walker: 2.0.2 source-map-js: 1.2.1 - '@vue/compiler-core@3.5.25': + '@vue/compiler-core@3.5.41': dependencies: - '@babel/parser': 7.29.0 - '@vue/shared': 3.5.25 - entities: 4.5.0 + '@babel/parser': 7.29.8 + '@vue/shared': 3.5.41 + entities: 7.0.1 estree-walker: 2.0.2 source-map-js: 1.2.1 @@ -7858,33 +11944,33 @@ snapshots: '@vue/compiler-core': 3.5.24 '@vue/shared': 3.5.24 - '@vue/compiler-dom@3.5.25': + '@vue/compiler-dom@3.5.41': dependencies: - '@vue/compiler-core': 3.5.25 - '@vue/shared': 3.5.25 + '@vue/compiler-core': 3.5.41 + '@vue/shared': 3.5.41 '@vue/compiler-sfc@3.5.24': dependencies: - '@babel/parser': 7.29.0 + '@babel/parser': 7.29.8 '@vue/compiler-core': 3.5.24 '@vue/compiler-dom': 3.5.24 '@vue/compiler-ssr': 3.5.24 '@vue/shared': 3.5.24 estree-walker: 2.0.2 magic-string: 0.30.21 - postcss: 8.5.10 + postcss: 8.5.26 source-map-js: 1.2.1 - '@vue/compiler-sfc@3.5.25': + '@vue/compiler-sfc@3.5.41': dependencies: - '@babel/parser': 7.29.0 - '@vue/compiler-core': 3.5.25 - '@vue/compiler-dom': 3.5.25 - '@vue/compiler-ssr': 3.5.25 - '@vue/shared': 3.5.25 + '@babel/parser': 7.29.8 + '@vue/compiler-core': 3.5.41 + '@vue/compiler-dom': 3.5.41 + '@vue/compiler-ssr': 3.5.41 + '@vue/shared': 3.5.41 estree-walker: 2.0.2 magic-string: 0.30.21 - postcss: 8.5.10 + postcss: 8.5.26 source-map-js: 1.2.1 '@vue/compiler-ssr@3.5.24': @@ -7892,10 +11978,10 @@ snapshots: '@vue/compiler-dom': 3.5.24 '@vue/shared': 3.5.24 - '@vue/compiler-ssr@3.5.25': + '@vue/compiler-ssr@3.5.41': dependencies: - '@vue/compiler-dom': 3.5.25 - '@vue/shared': 3.5.25 + '@vue/compiler-dom': 3.5.41 + '@vue/shared': 3.5.41 '@vue/compiler-vue2@2.7.16': dependencies: @@ -7905,9 +11991,9 @@ snapshots: '@vue/language-core@2.2.0(typescript@5.9.3)': dependencies: '@volar/language-core': 2.4.28 - '@vue/compiler-dom': 3.5.25 + '@vue/compiler-dom': 3.5.41 '@vue/compiler-vue2': 2.7.16 - '@vue/shared': 3.5.25 + '@vue/shared': 3.5.41 alien-signals: 0.4.14 minimatch: 9.0.5 muggle-string: 0.4.1 @@ -7918,8 +12004,8 @@ snapshots: '@vue/language-core@3.3.5': dependencies: '@volar/language-core': 2.4.28 - '@vue/compiler-dom': 3.5.25 - '@vue/shared': 3.5.25 + '@vue/compiler-dom': 3.5.41 + '@vue/shared': 3.5.41 alien-signals: 3.2.1 muggle-string: 0.4.1 path-browserify: 1.0.1 @@ -7929,33 +12015,33 @@ snapshots: dependencies: '@vue/shared': 3.5.24 - '@vue/reactivity@3.5.25': + '@vue/reactivity@3.5.41': dependencies: - '@vue/shared': 3.5.25 + '@vue/shared': 3.5.41 '@vue/runtime-core@3.5.24': dependencies: '@vue/reactivity': 3.5.24 '@vue/shared': 3.5.24 - '@vue/runtime-core@3.5.25': + '@vue/runtime-core@3.5.41': dependencies: - '@vue/reactivity': 3.5.25 - '@vue/shared': 3.5.25 + '@vue/reactivity': 3.5.41 + '@vue/shared': 3.5.41 '@vue/runtime-dom@3.5.24': dependencies: '@vue/reactivity': 3.5.24 '@vue/runtime-core': 3.5.24 '@vue/shared': 3.5.24 - csstype: 3.1.3 + csstype: 3.2.3 - '@vue/runtime-dom@3.5.25': + '@vue/runtime-dom@3.5.41': dependencies: - '@vue/reactivity': 3.5.25 - '@vue/runtime-core': 3.5.25 - '@vue/shared': 3.5.25 - csstype: 3.1.3 + '@vue/reactivity': 3.5.41 + '@vue/runtime-core': 3.5.41 + '@vue/shared': 3.5.41 + csstype: 3.2.3 '@vue/server-renderer@3.5.24(vue@3.5.24(typescript@5.6.3))': dependencies: @@ -7963,21 +12049,15 @@ snapshots: '@vue/shared': 3.5.24 vue: 3.5.24(typescript@5.6.3) - '@vue/server-renderer@3.5.25(vue@3.5.25(typescript@5.6.3))': - dependencies: - '@vue/compiler-ssr': 3.5.25 - '@vue/shared': 3.5.25 - vue: 3.5.25(typescript@5.6.3) - - '@vue/server-renderer@3.5.25(vue@3.5.25(typescript@5.9.3))': + '@vue/server-renderer@3.5.41': dependencies: - '@vue/compiler-ssr': 3.5.25 - '@vue/shared': 3.5.25 - vue: 3.5.25(typescript@5.9.3) + '@vue/compiler-ssr': 3.5.41 + '@vue/runtime-dom': 3.5.41 + '@vue/shared': 3.5.41 '@vue/shared@3.5.24': {} - '@vue/shared@3.5.25': {} + '@vue/shared@3.5.41': {} '@vue/test-utils@2.4.6': dependencies: @@ -7991,6 +12071,13 @@ snapshots: '@vueuse/shared': 13.8.0(vue@3.5.24(typescript@5.6.3)) vue: 3.5.24(typescript@5.6.3) + '@vueuse/core@13.8.0(vue@3.5.41(typescript@5.6.3))': + dependencies: + '@types/web-bluetooth': 0.0.21 + '@vueuse/metadata': 13.8.0 + '@vueuse/shared': 13.8.0(vue@3.5.41(typescript@5.6.3)) + vue: 3.5.41(typescript@5.6.3) + '@vueuse/integrations@13.8.0(sortablejs@1.15.6)(vue@3.5.24(typescript@5.6.3))': dependencies: '@vueuse/core': 13.8.0(vue@3.5.24(typescript@5.6.3)) @@ -8005,6 +12092,10 @@ snapshots: dependencies: vue: 3.5.24(typescript@5.6.3) + '@vueuse/shared@13.8.0(vue@3.5.41(typescript@5.6.3))': + dependencies: + vue: 3.5.41(typescript@5.6.3) + '@zip.js/zip.js@2.8.11': {} abbrev@2.0.0: {} @@ -8038,6 +12129,8 @@ snapshots: ag-charts-locale@12.1.2: optional: true + ag-charts-types@10.3.9: {} + ag-charts-types@12.1.2: {} ag-grid-community@34.1.2: @@ -8056,6 +12149,11 @@ snapshots: ag-grid-community: 34.1.2 vue: 3.5.24(typescript@5.6.3) + ag-grid-vue3@34.1.2(vue@3.5.41(typescript@5.6.3)): + dependencies: + ag-grid-community: 34.1.2 + vue: 3.5.41(typescript@5.6.3) + agent-base@7.1.3: {} ajv-draft-04@1.0.0(ajv@8.18.0): @@ -8311,6 +12409,8 @@ snapshots: color: 3.2.1 text-hex: 1.0.0 + comlink@4.4.2: {} + commander@10.0.1: {} commander@14.0.3: {} @@ -8373,7 +12473,9 @@ snapshots: shebang-command: 2.0.0 which: 2.0.2 - csstype@3.1.3: {} + csstype@3.2.3: {} + + csv-parse@6.2.1: {} d3-array@3.2.4: dependencies: @@ -8383,12 +12485,36 @@ snapshots: d3-color@3.1.0: {} + d3-dispatch@3.0.1: {} + + d3-drag@3.0.0: + dependencies: + d3-dispatch: 3.0.1 + d3-selection: 3.0.0 + + d3-ease@3.0.1: {} + d3-format@3.1.0: {} + d3-hierarchy@3.1.2: {} + d3-interpolate@3.0.1: dependencies: d3-color: 3.1.0 + d3-path@3.1.0: {} + + d3-polygon@3.0.1: {} + + d3-quadtree@3.0.1: {} + + d3-random@3.0.1: {} + + d3-scale-chromatic@3.1.0: + dependencies: + d3-color: 3.1.0 + d3-interpolate: 3.0.1 + d3-scale@4.0.2: dependencies: d3-array: 3.2.4 @@ -8399,6 +12525,10 @@ snapshots: d3-selection@3.0.0: {} + d3-shape@3.2.0: + dependencies: + d3-path: 3.1.0 + d3-time-format@4.1.0: dependencies: d3-time: 3.1.0 @@ -8407,8 +12537,31 @@ snapshots: dependencies: d3-array: 3.2.4 + d3-timer@3.0.1: {} + + d3-transition@3.0.1(d3-selection@3.0.0): + dependencies: + d3-color: 3.1.0 + d3-dispatch: 3.0.1 + d3-ease: 3.0.1 + d3-interpolate: 3.0.1 + d3-selection: 3.0.0 + d3-timer: 3.0.1 + + d3-zoom@3.0.0: + dependencies: + d3-dispatch: 3.0.1 + d3-drag: 3.0.0 + d3-interpolate: 3.0.1 + d3-selection: 3.0.0 + d3-transition: 3.0.1(d3-selection@3.0.0) + de-indent@1.0.2: {} + debug@2.6.9: + dependencies: + ms: 2.0.0 + debug@4.4.0: dependencies: ms: 2.1.3 @@ -8499,41 +12652,14 @@ snapshots: entities@4.5.0: {} + entities@7.0.1: {} + es-module-lexer@1.7.0: {} es-module-lexer@2.0.0: {} es-toolkit@1.42.0: {} - esbuild@0.25.12: - optionalDependencies: - '@esbuild/aix-ppc64': 0.25.12 - '@esbuild/android-arm': 0.25.12 - '@esbuild/android-arm64': 0.25.12 - '@esbuild/android-x64': 0.25.12 - '@esbuild/darwin-arm64': 0.25.12 - '@esbuild/darwin-x64': 0.25.12 - '@esbuild/freebsd-arm64': 0.25.12 - '@esbuild/freebsd-x64': 0.25.12 - '@esbuild/linux-arm': 0.25.12 - '@esbuild/linux-arm64': 0.25.12 - '@esbuild/linux-ia32': 0.25.12 - '@esbuild/linux-loong64': 0.25.12 - '@esbuild/linux-mips64el': 0.25.12 - '@esbuild/linux-ppc64': 0.25.12 - '@esbuild/linux-riscv64': 0.25.12 - '@esbuild/linux-s390x': 0.25.12 - '@esbuild/linux-x64': 0.25.12 - '@esbuild/netbsd-arm64': 0.25.12 - '@esbuild/netbsd-x64': 0.25.12 - '@esbuild/openbsd-arm64': 0.25.12 - '@esbuild/openbsd-x64': 0.25.12 - '@esbuild/openharmony-arm64': 0.25.12 - '@esbuild/sunos-x64': 0.25.12 - '@esbuild/win32-arm64': 0.25.12 - '@esbuild/win32-ia32': 0.25.12 - '@esbuild/win32-x64': 0.25.12 - escalade@3.2.0: {} esprima@4.0.1: {} @@ -8594,10 +12720,6 @@ snapshots: dependencies: pend: 1.2.0 - fdir@6.5.0(picomatch@4.0.3): - optionalDependencies: - picomatch: 4.0.3 - fdir@6.5.0(picomatch@4.0.4): optionalDependencies: picomatch: 4.0.4 @@ -8858,6 +12980,8 @@ snapshots: optionalDependencies: graceful-fs: 4.2.11 + kdbush@4.1.0: {} + kolorist@1.8.0: {} kuler@2.0.0: {} @@ -9029,6 +13153,8 @@ snapshots: mri@1.2.0: {} + ms@2.0.0: {} + ms@2.1.3: {} muggle-string@0.4.1: {} @@ -9040,6 +13166,8 @@ snapshots: nanoid@3.3.11: {} + nanoid@3.3.18: {} + napi-build-utils@2.0.0: {} nice-try@1.0.5: {} @@ -9070,8 +13198,6 @@ snapshots: object-assign@4.1.1: {} - obug@2.1.1: {} - obug@2.1.3: {} once@1.4.0: @@ -9209,8 +13335,6 @@ snapshots: picomatch@2.3.1: {} - picomatch@4.0.3: {} - picomatch@4.0.4: {} pify@2.3.0: {} @@ -9252,6 +13376,12 @@ snapshots: picocolors: 1.1.1 source-map-js: 1.2.1 + postcss@8.5.26: + dependencies: + nanoid: 3.3.18 + picocolors: 1.1.1 + source-map-js: 1.2.1 + prebuild-install@7.1.3: dependencies: detect-libc: 2.0.3 @@ -9275,18 +13405,17 @@ snapshots: proto-list@1.2.4: {} - protobufjs@7.4.0: + protobufjs@7.6.5: dependencies: '@protobufjs/aspromise': 1.1.2 '@protobufjs/base64': 1.1.2 - '@protobufjs/codegen': 2.0.4 - '@protobufjs/eventemitter': 1.1.0 - '@protobufjs/fetch': 1.1.0 + '@protobufjs/codegen': 2.0.5 + '@protobufjs/eventemitter': 1.1.1 + '@protobufjs/fetch': 1.1.1 '@protobufjs/float': 1.0.2 - '@protobufjs/inquire': 1.1.0 '@protobufjs/path': 1.1.2 '@protobufjs/pool': 1.1.0 - '@protobufjs/utf8': 1.1.0 + '@protobufjs/utf8': 1.1.2 '@types/node': 25.3.2 long: 5.3.2 @@ -9307,17 +13436,23 @@ snapshots: queue-tick@1.0.1: {} - quickjs-emscripten-core@0.31.0: + quickjs-emscripten-core@0.32.0: + dependencies: + '@jitl/quickjs-ffi-types': 0.32.0 + + quickjs-emscripten@0.32.0: dependencies: - '@jitl/quickjs-ffi-types': 0.31.0 + '@jitl/quickjs-wasmfile-debug-asyncify': 0.32.0 + '@jitl/quickjs-wasmfile-debug-sync': 0.32.0 + '@jitl/quickjs-wasmfile-release-asyncify': 0.32.0 + '@jitl/quickjs-wasmfile-release-sync': 0.32.0 + quickjs-emscripten-core: 0.32.0 + + quickselect@3.0.0: {} - quickjs-emscripten@0.31.0: + rbush@4.0.1: dependencies: - '@jitl/quickjs-wasmfile-debug-asyncify': 0.31.0 - '@jitl/quickjs-wasmfile-debug-sync': 0.31.0 - '@jitl/quickjs-wasmfile-release-asyncify': 0.31.0 - '@jitl/quickjs-wasmfile-release-sync': 0.31.0 - quickjs-emscripten-core: 0.31.0 + quickselect: 3.0.0 rc@1.2.8: dependencies: @@ -9326,6 +13461,13 @@ snapshots: minimist: 1.2.8 strip-json-comments: 2.0.1 + react-dom@19.2.8(react@19.2.8): + dependencies: + react: 19.2.8 + scheduler: 0.27.0 + + react@19.2.8: {} + read-yaml-file@1.1.0: dependencies: graceful-fs: 4.2.11 @@ -9505,6 +13647,8 @@ snapshots: safer-buffer@2.1.2: {} + scheduler@0.27.0: {} + seek-bzip@1.0.6: dependencies: commander: 2.20.3 @@ -9595,8 +13739,6 @@ snapshots: stackback@0.0.2: {} - std-env@3.10.0: {} - std-env@4.1.0: {} stream-browserify@3.0.0: @@ -9728,8 +13870,8 @@ snapshots: tinyglobby@0.2.15: dependencies: - fdir: 6.5.0(picomatch@4.0.3) - picomatch: 4.0.3 + fdir: 6.5.0(picomatch@4.0.4) + picomatch: 4.0.4 tinypool@2.1.0: {} @@ -9757,32 +13899,14 @@ snapshots: dependencies: safe-buffer: 5.2.1 - turbo-darwin-64@2.8.11: - optional: true - - turbo-darwin-arm64@2.8.11: - optional: true - - turbo-linux-64@2.8.11: - optional: true - - turbo-linux-arm64@2.8.11: - optional: true - - turbo-windows-64@2.8.11: - optional: true - - turbo-windows-arm64@2.8.11: - optional: true - - turbo@2.8.11: + turbo@2.10.10: optionalDependencies: - turbo-darwin-64: 2.8.11 - turbo-darwin-arm64: 2.8.11 - turbo-linux-64: 2.8.11 - turbo-linux-arm64: 2.8.11 - turbo-windows-64: 2.8.11 - turbo-windows-arm64: 2.8.11 + '@turbo/darwin-64': 2.10.10 + '@turbo/darwin-arm64': 2.10.10 + '@turbo/linux-64': 2.10.10 + '@turbo/linux-arm64': 2.10.10 + '@turbo/windows-64': 2.10.10 + '@turbo/windows-arm64': 2.10.10 tweetnacl@0.14.5: {} @@ -9872,20 +13996,6 @@ snapshots: picocolors: 1.1.1 vite: 8.0.8(@types/node@25.3.2)(yaml@2.8.1) - vite@7.2.7(@types/node@25.3.2)(lightningcss@1.32.0)(yaml@2.8.1): - dependencies: - esbuild: 0.25.12 - fdir: 6.5.0(picomatch@4.0.4) - picomatch: 4.0.4 - postcss: 8.5.10 - rollup: 4.53.3 - tinyglobby: 0.2.15 - optionalDependencies: - '@types/node': 25.3.2 - fsevents: 2.3.3 - lightningcss: 1.32.0 - yaml: 2.8.1 - vite@8.0.8(@types/node@25.3.2)(yaml@2.8.1): dependencies: lightningcss: 1.32.0 @@ -9898,80 +14008,15 @@ snapshots: fsevents: 2.3.3 yaml: 2.8.1 - vitest@4.0.18(@types/node@25.3.2)(lightningcss@1.32.0)(yaml@2.8.1): - dependencies: - '@vitest/expect': 4.0.18 - '@vitest/mocker': 4.0.18(vite@7.2.7(@types/node@25.3.2)(lightningcss@1.32.0)(yaml@2.8.1)) - '@vitest/pretty-format': 4.0.18 - '@vitest/runner': 4.0.18 - '@vitest/snapshot': 4.0.18 - '@vitest/spy': 4.0.18 - '@vitest/utils': 4.0.18 - es-module-lexer: 1.7.0 - expect-type: 1.3.0 - magic-string: 0.30.21 - obug: 2.1.1 - pathe: 2.0.3 - picomatch: 4.0.4 - std-env: 3.10.0 - tinybench: 2.9.0 - tinyexec: 1.0.2 - tinyglobby: 0.2.15 - tinyrainbow: 3.1.0 - vite: 7.2.7(@types/node@25.3.2)(lightningcss@1.32.0)(yaml@2.8.1) - why-is-node-running: 2.3.0 - optionalDependencies: - '@types/node': 25.3.2 - transitivePeerDependencies: - - jiti - - less - - lightningcss - - msw - - sass - - sass-embedded - - stylus - - sugarss - - terser - - tsx - - yaml - - vitest@4.1.4(@types/node@25.3.2)(@vitest/coverage-istanbul@4.1.4)(vite@7.2.7(@types/node@25.3.2)(lightningcss@1.32.0)(yaml@2.8.1)): - dependencies: - '@vitest/expect': 4.1.4 - '@vitest/mocker': 4.1.4(vite@7.2.7(@types/node@25.3.2)(lightningcss@1.32.0)(yaml@2.8.1)) - '@vitest/pretty-format': 4.1.4 - '@vitest/runner': 4.1.4 - '@vitest/snapshot': 4.1.4 - '@vitest/spy': 4.1.4 - '@vitest/utils': 4.1.4 - es-module-lexer: 2.0.0 - expect-type: 1.3.0 - magic-string: 0.30.21 - obug: 2.1.3 - pathe: 2.0.3 - picomatch: 4.0.4 - std-env: 4.1.0 - tinybench: 2.9.0 - tinyexec: 1.0.2 - tinyglobby: 0.2.15 - tinyrainbow: 3.1.0 - vite: 7.2.7(@types/node@25.3.2)(lightningcss@1.32.0)(yaml@2.8.1) - why-is-node-running: 2.3.0 - optionalDependencies: - '@types/node': 25.3.2 - '@vitest/coverage-istanbul': 4.1.4(vitest@4.0.18(@types/node@25.3.2)(lightningcss@1.32.0)(yaml@2.8.1)) - transitivePeerDependencies: - - msw - - vitest@4.1.4(@types/node@25.3.2)(@vitest/coverage-istanbul@4.1.4)(vite@8.0.8(@types/node@25.3.2)(yaml@2.8.1)): + vitest@4.1.10(@types/node@25.3.2)(@vitest/coverage-istanbul@4.1.4)(vite@8.0.8(@types/node@25.3.2)(yaml@2.8.1)): dependencies: - '@vitest/expect': 4.1.4 - '@vitest/mocker': 4.1.4(vite@8.0.8(@types/node@25.3.2)(yaml@2.8.1)) - '@vitest/pretty-format': 4.1.4 - '@vitest/runner': 4.1.4 - '@vitest/snapshot': 4.1.4 - '@vitest/spy': 4.1.4 - '@vitest/utils': 4.1.4 + '@vitest/expect': 4.1.10 + '@vitest/mocker': 4.1.10(vite@8.0.8(@types/node@25.3.2)(yaml@2.8.1)) + '@vitest/pretty-format': 4.1.10 + '@vitest/runner': 4.1.10 + '@vitest/snapshot': 4.1.10 + '@vitest/spy': 4.1.10 + '@vitest/utils': 4.1.10 es-module-lexer: 2.0.0 expect-type: 1.3.0 magic-string: 0.30.21 @@ -9987,7 +14032,7 @@ snapshots: why-is-node-running: 2.3.0 optionalDependencies: '@types/node': 25.3.2 - '@vitest/coverage-istanbul': 4.1.4(vitest@4.1.4) + '@vitest/coverage-istanbul': 4.1.4(vitest@4.1.10) transitivePeerDependencies: - msw @@ -10011,23 +14056,23 @@ snapshots: optionalDependencies: typescript: 5.6.3 - vue@3.5.25(typescript@5.6.3): + vue@3.5.41(typescript@5.6.3): dependencies: - '@vue/compiler-dom': 3.5.25 - '@vue/compiler-sfc': 3.5.25 - '@vue/runtime-dom': 3.5.25 - '@vue/server-renderer': 3.5.25(vue@3.5.25(typescript@5.6.3)) - '@vue/shared': 3.5.25 + '@vue/compiler-dom': 3.5.41 + '@vue/compiler-sfc': 3.5.41 + '@vue/runtime-dom': 3.5.41 + '@vue/server-renderer': 3.5.41 + '@vue/shared': 3.5.41 optionalDependencies: typescript: 5.6.3 - vue@3.5.25(typescript@5.9.3): + vue@3.5.41(typescript@5.9.3): dependencies: - '@vue/compiler-dom': 3.5.25 - '@vue/compiler-sfc': 3.5.25 - '@vue/runtime-dom': 3.5.25 - '@vue/server-renderer': 3.5.25(vue@3.5.25(typescript@5.9.3)) - '@vue/shared': 3.5.25 + '@vue/compiler-dom': 3.5.41 + '@vue/compiler-sfc': 3.5.41 + '@vue/runtime-dom': 3.5.41 + '@vue/server-renderer': 3.5.41 + '@vue/shared': 3.5.41 optionalDependencies: typescript: 5.9.3 diff --git a/pnpm-workspace.yaml b/pnpm-workspace.yaml index 3a8102f..f853f5c 100644 --- a/pnpm-workspace.yaml +++ b/pnpm-workspace.yaml @@ -8,19 +8,20 @@ packages: catalog: "@milaboratories/helpers": 1.14.5 - "@milaboratories/ts-builder": 1.6.1 - "@milaboratories/ts-configs": 1.3.1 + "@milaboratories/graph-maker": 1.4.8 + "@milaboratories/ts-builder": 1.6.2 + "@milaboratories/ts-configs": 1.4.0 typescript: ~5.9.3 "@platforma-sdk/workflow-tengo": 6.8.2 - "@platforma-sdk/block-tools": 2.12.8 - "@platforma-sdk/model": 1.80.8 - "@platforma-sdk/ui-vue": 1.80.9 - "@platforma-sdk/test": 1.80.9 - "@platforma-sdk/tengo-builder": 4.0.19 - "@platforma-sdk/package-builder": 3.14.2 + "@platforma-sdk/block-tools": 2.14.0 + "@platforma-sdk/model": 1.81.1 + "@platforma-sdk/ui-vue": 1.81.1 + "@platforma-sdk/test": 1.81.3 + "@platforma-sdk/tengo-builder": 4.0.22 + "@platforma-sdk/package-builder": 3.15.0 "@platforma-sdk/blocks-deps-updater": 2.2.0 - "@platforma-open/milaboratories.runenv-python-3": 1.7.8 + "@platforma-open/milaboratories.runenv-python-3": 1.11.6 "@platforma-open/milaboratories.software-mitool": 2.3.1-131-main # blocks used in integration tests. SND pinned EXACT to the last V1 (modelAPIVersion 1) release: @@ -29,11 +30,12 @@ catalog: "@platforma-open/milaboratories.samples-and-data": 1.13.3 "@platforma-open/milaboratories.samples-and-data.model": 1.11.2 - "turbo": 2.8.11 + "turbo": 2.10.10 "shx": 0.4.0 "@changesets/cli": 2.29.8 vue: 3.5.24 + "csv-parse": ^6.1.0 "ag-grid-enterprise": &ag-grid ~34.1.2 "ag-grid-vue3": *ag-grid - vitest: ~4.0.18 + vitest: ~4.1.10 diff --git a/software/.coverage b/software/.coverage new file mode 100644 index 0000000..e4e5a17 Binary files /dev/null and b/software/.coverage differ diff --git a/software/conftest.py b/software/conftest.py index 20dac62..6c3afa7 100644 --- a/software/conftest.py +++ b/software/conftest.py @@ -1,7 +1,7 @@ """Shared fixtures for the Feature Integration software tests. -A conftest.py at the software root is auto-applied to every test below it (fixtures are consumed by -name, never via ``import conftest``). The committed test bed lives at +A conftest.py at the software root is auto-applied to every test below it, and fixtures are consumed by +name rather than via ``import conftest``. The committed test bed lives at software/test-data/fixtures/per-cell-metrics/: a small mitool tag-stat TSV plus a tag->feature CSV. """ diff --git a/software/per-cell-metrics/package.json b/software/per-cell-metrics/package.json index 23efb8f..072993c 100644 --- a/software/per-cell-metrics/package.json +++ b/software/per-cell-metrics/package.json @@ -54,7 +54,7 @@ ] } }, - "emit-csv-meta": { + "feature-properties": { "binary": { "artifact": { "type": "python", @@ -68,11 +68,11 @@ }, "cmd": [ "python", - "{pkg}/emit_csv_meta.py" + "{pkg}/emit_feature_properties.py" ] } }, - "feature-properties": { + "qc-report": { "binary": { "artifact": { "type": "python", @@ -86,11 +86,11 @@ }, "cmd": [ "python", - "{pkg}/emit_feature_properties.py" + "{pkg}/qc_report.py" ] } }, - "qc-report": { + "emit-verdicts": { "binary": { "artifact": { "type": "python", @@ -104,7 +104,7 @@ }, "cmd": [ "python", - "{pkg}/qc_report.py" + "{pkg}/emit_verdicts.py" ] } }, diff --git a/software/per-cell-metrics/src/combine.py b/software/per-cell-metrics/src/combine.py new file mode 100644 index 0000000..96e48e5 --- /dev/null +++ b/software/per-cell-metrics/src/combine.py @@ -0,0 +1,540 @@ +"""Reducing a set's cells to the set's verdict, identity by identity. + +Cells of one set are replicates of one measurement, so where they differ at an +identity the difference is error and the modal answer is the best available +reading. The vote is per identity: a single winning antigen would collapse a set +that bound several. + +The row set is the identity universe, never the offered subset. A set's verdict at +an identity the panel never offered is NEVER_ASKED, and that comes only from the +offered map, never from a row's absence. `offered` is keyed by sample, matching +`silent_tally`'s `offered_by_sample`, because staining is a property of the sample +rather than of the clonotype grouping built on it. A set's offered set is therefore +the union over its member samples. + +A cell asked about an identity and showing no reading in `states` is silent, not +absent from the vote. `silent_tally` supplies the silent contribution. A silent +admissible cell always resolves NOT_BOUND (see `specificity_score` in verdict.py), +so silent cells vote not bound and silent inadmissible cells vote nowhere. That is +what keeps a set whose every cell failed to bind reading NOT_BOUND rather than +UNRELIABLE or NEVER_ASKED. +""" + +from __future__ import annotations + +from enum import Enum + +import polars as pl +from verdict import Admissibility, State, UnreliableReason, _admissibility_reason, silent_tally + +# Both limits default permissively, because the failure they would prevent is visible +# and the failure they would cause is not. Requiring two voting cells would silently +# discard every singleton, which many clonotypes in a run are. +DEFAULT_MIN_VOTERS = 1 +DEFAULT_MIN_AGREEMENT = None + +SETTLED = (State.BOUND.value, State.NOT_BOUND.value) + + +class SetUnreliableReason(str, Enum): + """Why a set's verdict at one identity could not be settled, or why it was never + asked. `verdict.UnreliableReason` answers "why can't this cell be compared". This + answers "why can't this set's cells, taken together, produce a verdict" -- a + different grain, kept in its own enum. + + NEVER_OFFERED is the reason recorded on a NEVER_ASKED row. It is not a reliability + problem, but the same column carries it, so a reader always finds a reason wherever + the state is not BOUND or NOT_BOUND. + + NO_COMPARATOR reuses the cell-level concept for the same fact observed over a whole + set: no settled vote exists because every asked cell individually had no + comparator. ALL_CELLS_GATED is reported only when every asked cell was gated with + no other reason mixed in. A mix of gated and comparator-less cells reports as + NO_COMPARATOR, since a gate excluding part of a set is not why the rest failed. + + TIE and BELOW_AGREEMENT_FLOOR both leave the identity UNRELIABLE and look alike + from the counts, but call for different action. A TIE has no majority to trust: the + settled cells split evenly, which may be real heterogeneity, and no parameter moves + it. A BELOW_AGREEMENT_FLOOR set formed a majority and was refused only because + `min_agreement` was raised above it -- the fix is to lower that floor or gather more + cells, not to suspect the biology. `min_agreement` defaults to off, so this reason + appears only because someone raised it. + """ + + NEVER_OFFERED = "never-offered" + NO_COMPARATOR = "no-comparator" + ALL_CELLS_GATED = "all-cells-gated" + TIE = "tie" + BELOW_AGREEMENT_FLOOR = "below-agreement-floor" + TOO_FEW_VOTERS = "too-few-voters" + + +def _dominant_reason( + asked_keys: list[tuple[str, str]], identity: str, admissibility: Admissibility +) -> SetUnreliableReason: + """The one reason that explains why none of `asked_keys` settled. + + Called only when the set has zero settled votes for the identity, which happens only + when every asked cell is individually inadmissible: an admissible cell always + settles, directly or, if silent, through `silent_tally`'s proof that a silent + admissible cell resolves NOT_BOUND. So every key here has a real, non-None + cell-level reason and this only picks among three. + + The assertion is not a formality. Without it, an admissible key slipping in -- the + caller's vote-counting and its admissibility disagreeing about which cells were + asked -- reads a `None` reason, misses the gated check, and falls through to + NO_COMPARATOR, reporting a missing comparator for a cell whose comparator is fine. + """ + # Takes the identity because one rung's comparator depends on it. A set can fail to + # settle one identity and settle every other. + reasons = {_admissibility_reason(k, identity, admissibility) for k in asked_keys} + # Raised rather than asserted: stripped under -O this does not crash, it falls through + # to NO_COMPARATOR and reports a missing comparator for a cell whose comparator is + # fine -- the exact wrong answer this check exists to turn into a loud failure. + if None in reasons: + raise ValueError( + f"an admissible cell reached _dominant_reason among {asked_keys!r}: this is only called " + "when cellsAnswered is 0, which should be possible only when every asked cell is " + "individually inadmissible -- a None reason here means the caller's vote count and " + "admissibility disagree about which cells were actually asked" + ) + if reasons == {UnreliableReason.GATED}: + return SetUnreliableReason.ALL_CELLS_GATED + return SetUnreliableReason.NO_COMPARATOR + + +def _majority(counts: dict[str, int]) -> tuple[str, int, bool]: + """The leading state, its count, and whether more than one state is tied for it.""" + top = max(counts.values()) + leaders = sorted(state for state, n in counts.items() if n == top) + return leaders[0], top, len(leaders) > 1 + + +def combine_cells( + states: pl.DataFrame, + universe: set[str], + offered: dict[str, set[str]], + cells_by_set: dict[str, list[tuple[str, str]]], + admissibility: Admissibility, + min_voters: int = DEFAULT_MIN_VOTERS, + min_agreement: float | None = DEFAULT_MIN_AGREEMENT, +) -> pl.DataFrame: + """One row per (set, identity) over the whole universe. + + `states` is `read_states`' output directly: one row per (cell, identity) that got an + explicit reading. A cell asked about an identity and absent here is silent for it, + not unasked. There is deliberately no setId column in that shape -- which set a row + belongs to is decided once, below, by looking the cell up in `cells_by_set`, never + by trusting a column that could disagree. A row for a cell no set lists is dropped, + exactly as `silent_tally` drops such cells. + + `offered` is keyed by sample: a set's offered identities are the union over its + member samples, so a set spanning two panels reads as offered whatever either + offered, while `cellsCouldAnswer` still counts only members whose OWN sample offered + that identity. + + `cells_by_set` gives each set's full membership, including cells with no row in + `states` -- the silent cells, which vote through `silent_tally`. It must be disjoint: + a cell key may repeat within one set's list with no effect, but must not appear + under two set ids, which is asserted below rather than surfacing later as a + `silent_tally` precondition failure pointing at the wrong function. + """ + group_by_cell: dict[tuple[str, str], str] = {} + for set_id, members in cells_by_set.items(): + for key in members: + owner = group_by_cell.get(key) + # Raised rather than asserted: under -O an assert vanishes, and this one + # vanishing does not crash -- it double-counts the cell into two sets and + # reports tallies that are simply wrong. + if owner is not None and owner != set_id: + raise ValueError( + f"cell {key!r} appears in both set {owner!r} and set {set_id!r} in cells_by_set: " + "a cell must belong to exactly one set" + ) + group_by_cell[key] = set_id + + cells_frame = pl.DataFrame(list(group_by_cell), orient="row", schema={"sampleId": pl.String, "cellId": pl.String}) + tally = silent_tally(states, cells_frame, offered, admissibility, group_by_cell=group_by_cell, group_column="setId") + silent_by_pair = {(row["setId"], row["identity"]): row for row in tally.iter_rows(named=True)} + + settled = states.filter(pl.col("state").is_in(SETTLED)) + explicit_counts: dict[tuple[str, str], dict[str, int]] = {} + for sample_id, cell_id, identity, state in zip( + settled["sampleId"].to_list(), + settled["cellId"].to_list(), + settled["identity"].to_list(), + settled["state"].to_list(), + strict=True, + ): + set_id = group_by_cell.get((sample_id, cell_id)) + if set_id is None: + # In no set's membership list: the same drop `silent_tally` applies, so a + # vote is never counted for a cell nobody asked to vote. + continue + if identity not in offered.get(sample_id, frozenset()): + # This cell's own sample never offered the identity, so its reading is not a + # vote. The denominator counts only members whose own sample offered it, so + # counting this would mix two populations. + continue + bucket = explicit_counts.setdefault((set_id, identity), {}) + bucket[state] = bucket.get(state, 0) + 1 + + rows = [] + for set_id in sorted(cells_by_set): + members = cells_by_set[set_id] + offered_for_set = set().union(*(offered.get(sample, set()) for sample, _ in members)) if members else set() + + for identity in sorted(universe): + if identity not in offered_for_set: + rows.append( + { + "setId": set_id, + "identity": identity, + "state": State.NEVER_ASKED.value, + "cellsCouldAnswer": 0, + "cellsAnswered": 0, + # No tally exists for a position never put to this clonotype. 0 + # is the honest count -- a null would ride into the punch value + # as an empty field. Same for cellsNotBound. + "cellsBound": 0, + "cellsNotBound": 0, + "agreement": None, + "unreliableReason": SetUnreliableReason.NEVER_OFFERED.value, + } + ) + continue + + # Guaranteed present: `identity` is in `offered_for_set` only because at + # least one member's own sample offers it, which is exactly when + # silent_tally emits a row for (set_id, identity). + silent_row = silent_by_pair[(set_id, identity)] + could = silent_row["asked"] + + counts = dict(explicit_counts.get((set_id, identity), {})) + counts[State.NOT_BOUND.value] = counts.get(State.NOT_BOUND.value, 0) + silent_row["silentNotBound"] + answered = sum(counts.values()) + + if answered == 0: + asked_keys = [key for key in members if identity in offered.get(key[0], set())] + reason = _dominant_reason(asked_keys, identity, admissibility) + rows.append( + { + "setId": set_id, + "identity": identity, + "state": State.UNRELIABLE.value, + "cellsCouldAnswer": could, + "cellsAnswered": 0, + "cellsBound": 0, + "cellsNotBound": 0, + "agreement": None, + "unreliableReason": reason.value, + } + ) + continue + + if answered < min_voters: + rows.append( + { + "setId": set_id, + "identity": identity, + "state": State.UNRELIABLE.value, + "cellsCouldAnswer": could, + "cellsAnswered": answered, + "cellsBound": counts.get(State.BOUND.value, 0), + "cellsNotBound": counts.get(State.NOT_BOUND.value, 0), + "agreement": None, + "unreliableReason": SetUnreliableReason.TOO_FEW_VOTERS.value, + } + ) + continue + + top_state, top_count, tied = _majority(counts) + agreement = top_count / answered + + # A tie has no majority to trust: the settled cells split evenly and nothing + # says which side to believe. A narrow majority below the agreement floor + # has one, refused only because the operator raised that floor. Different + # action, so different reasons. + if tied: + rows.append( + { + "setId": set_id, + "identity": identity, + "state": State.UNRELIABLE.value, + "cellsCouldAnswer": could, + "cellsAnswered": answered, + "cellsBound": counts.get(State.BOUND.value, 0), + "cellsNotBound": counts.get(State.NOT_BOUND.value, 0), + "agreement": agreement, + "unreliableReason": SetUnreliableReason.TIE.value, + } + ) + continue + + if min_agreement is not None and agreement < min_agreement: + rows.append( + { + "setId": set_id, + "identity": identity, + "state": State.UNRELIABLE.value, + "cellsCouldAnswer": could, + "cellsAnswered": answered, + "cellsBound": counts.get(State.BOUND.value, 0), + "cellsNotBound": counts.get(State.NOT_BOUND.value, 0), + "agreement": agreement, + "unreliableReason": SetUnreliableReason.BELOW_AGREEMENT_FLOOR.value, + } + ) + continue + + rows.append( + { + "setId": set_id, + "identity": identity, + "state": top_state, + "cellsCouldAnswer": could, + "cellsAnswered": answered, + # Read from the tally, never from `agreement`. `agreement` is + # top_count/answered -- the MAJORITY's share -- and the majority is + # not always bound, so deriving a bound count from it reports the + # wrong state's cells wherever the verdict is not bound. + "cellsBound": counts.get(State.BOUND.value, 0), + # Same reasoning as cellsBound. SETTLED holds only BOUND and + # NOT_BOUND, so this could be cellsAnswered - cellsBound, but reading + # the tally directly does not lean on which state was top_state. + "cellsNotBound": counts.get(State.NOT_BOUND.value, 0), + "agreement": agreement, + "unreliableReason": None, + } + ) + + return pl.DataFrame( + rows, + schema={ + "setId": pl.String, + "identity": pl.String, + "state": pl.String, + "cellsCouldAnswer": pl.Int64, + "cellsAnswered": pl.Int64, + "cellsBound": pl.Int64, + "cellsNotBound": pl.Int64, + "agreement": pl.Float64, + "unreliableReason": pl.String, + }, + ).sort(["setId", "identity"]) + + +def attach_competitor_notes(verdicts: pl.DataFrame, contending: list[set[str]]) -> pl.DataFrame: + """Name the bound competitor beside a not-bound reading. Change nothing else. + + A negative beside a bound competitor and one beside nothing are different evidence, + and the counts cannot tell them apart. The verdict reports what could have caused + the reading and leaves the call to the reader. The state stays *not bound*, because + the doubt travels beside it. Only a settled NOT_BOUND row is eligible: an UNRELIABLE + or NEVER_ASKED row made no comparison, so it has no negative to sit beside. + + `wasCompeted` exists so a statement can test the note. Without a predicate, a + condition naming the off-target passes on the state alone, the doubt is lost, and + *not bound* becomes a claim the run never earned. It is an explicit "true"/"false" + string on every row -- the convention this project uses for a boolean becoming a + p-column value -- never null, because a filter for the absence of contention must + match on the flag alone. + + Which identities contend is chosen when the repertoire is annotated and is never + inferred from the counts: contention is a property of the design. An identity may + sit in more than one declared group, and the note then names the union of bound + competitors across every group containing it, since each group is an independent + claim. Names are joined in sorted order, so the same data always produces the same + string -- which matters once this column is content-addressed as a p-column. + """ + if not contending: + return verdicts.with_columns( + pl.lit(None, dtype=pl.String).alias("competedWith"), + pl.lit("false", dtype=pl.String).alias("wasCompeted"), + ) + + bound_by_set: dict[str, set[str]] = {} + for row in verdicts.filter(pl.col("state") == State.BOUND.value).iter_rows(named=True): + bound_by_set.setdefault(row["setId"], set()).add(row["identity"]) + + notes, flags = [], [] + for row in verdicts.iter_rows(named=True): + note = None + if row["state"] == State.NOT_BOUND.value: + bound_here = bound_by_set.get(row["setId"], set()) + rivals = { + other + for group in contending + if row["identity"] in group + for other in group & bound_here + if other != row["identity"] + } + if rivals: + note = ", ".join(sorted(rivals)) + notes.append(note) + flags.append("true" if note else "false") + + return verdicts.with_columns( + pl.Series("competedWith", notes, dtype=pl.String), + pl.Series("wasCompeted", flags, dtype=pl.String), + ) + + +def set_counts(verdicts: pl.DataFrame) -> pl.DataFrame: + """Per set: bound, offered, settled, unsettled -- in identities. + + The denominator is the identities the set was offered and whose reading settled, not + the size of the panel. A clonotype from a sample carrying only eight of ten, which + bound all eight, covered everything it was asked. Reported as eight of ten it looks + like a clone with two failures. + + An offered position that did not settle leaves the count and is reported beside it. + Voiding the count instead is what the four-state model literally implies, but on a + large panel a single bad reading would then destroy every count in the run. + + `offeredCount` always equals `settledCount + unsettledCount`, since UNRELIABLE is + the only offered-but-unsettled state. A set asked nothing reports all four as zero, + so a consumer computing a rate must guard the division itself. A set that is + entirely UNRELIABLE reports `boundCount=0, settledCount=0, unsettledCount=N` -- read + that as nothing settled, never as a failure to bind N identities. + + `verdicts` is read at its existing (setId, identity) grain, one row per identity + regardless of how many tags fed it, so counting rows counts identities, never tags. + """ + return ( + verdicts.group_by("setId") + .agg( + (pl.col("state") == State.BOUND.value).sum().alias("boundCount"), + (pl.col("state") != State.NEVER_ASKED.value).sum().alias("offeredCount"), + pl.col("state").is_in(SETTLED).sum().alias("settledCount"), + (pl.col("state") == State.UNRELIABLE.value).sum().alias("unsettledCount"), + ) + .sort("setId") + ) + + +def self_disagreement( + states: pl.DataFrame, + universe: set[str], + offered: dict[str, set[str]], + cells_by_set: dict[str, list[tuple[str, str]]], + admissibility: Admissibility, +) -> pl.DataFrame: + """How often a tag's cells contradict the rest of their own set. + + A clonotype is one receptor with one specificity, so where two of a set's evaluable + cells read differently at one position, at least one reading is wrong. That makes + this the cheapest quality signal available: no threshold and no external reference, + because the contradiction comes from the data disagreeing with itself. + + **The figure pools CELLS rather than scoring sets.** For one key: every set with two + or more evaluable cells contributes all of them to `cellsCompared`, and the cells in + the minority of their own set to `minorityCells`. The rate is the second over the + first. + + Pooling needs no small-set cutoff. A per-set share does: a share over three cells + takes only four values and would otherwise set the figure, and excluding those sets + makes the counted population differ from key to key. Pooling's own weakness is that + one very large set can set the number, and that cancels -- the same set sets it for + every tag, so a tag standing clear still stands clear. + + Two states cap the rate at half, the minority being the smaller side by definition. + + A set's evaluable cells at a position are every admissible cell that settled there, + explicit row or silent. A silent admissible cell always resolves not bound, so it is + as evaluable as an explicit row. An inadmissible cell never votes. The silent count + comes from `silent_tally`, keyed by set rather than by sample, never recomputed here. + + Measured at the TAG and nowhere else, and read as a comparison rather than a rate: a + tag standing clear of the other tags in the same panel is misbehaving whatever its + absolute value. The identity-level figure is not carried, because it has nothing to + compare against and cannot separate a faulty reagent from a panel of weak binders. + + Diagnostic only. It rests on comparing each tag against the baseline separately, + which no verdict is built from, so it is evidence about a reagent, never about an + answer. + + `states` carries `key` (the tag), sampleId, cellId and state -- `combine_cells`' + sparse shape with `identity` renamed and no setId column. `offered` and `universe` + are at that same grain, since a key with every cell silent has no explicit row + anywhere in `states`. + + A key no set could compare reports a NULL rate rather than zero, which would read as + agreement. + """ + group_by_cell: dict[tuple[str, str], str] = {} + for set_id, members in cells_by_set.items(): + for cell_key in members: + owner = group_by_cell.get(cell_key) + # Raised rather than asserted: under -O an assert vanishes, and this one + # vanishing does not crash -- it double-counts the cell into two sets and + # reports tallies that are simply wrong. + if owner is not None and owner != set_id: + raise ValueError( + f"cell {cell_key!r} appears in both set {owner!r} and set {set_id!r} in cells_by_set: " + "a cell must belong to exactly one set" + ) + group_by_cell[cell_key] = set_id + + cells_frame = pl.DataFrame(list(group_by_cell), orient="row", schema={"sampleId": pl.String, "cellId": pl.String}) + observed_for_tally = states.select("sampleId", "cellId", pl.col("key").alias("identity")) + tally = silent_tally( + observed_for_tally, cells_frame, offered, admissibility, group_by_cell=group_by_cell, group_column="setId" + ) + + settled = states.filter(pl.col("state").is_in(SETTLED)) + explicit_counts: dict[tuple[str, str], dict[str, int]] = {} + for sample_id, cell_id, key, state in zip( + settled["sampleId"].to_list(), + settled["cellId"].to_list(), + settled["key"].to_list(), + settled["state"].to_list(), + strict=True, + ): + set_id = group_by_cell.get((sample_id, cell_id)) + if set_id is None: + # Same drop `combine_cells` applies: a vote is never counted for a cell no + # set's membership list names. + continue + if key not in offered.get(sample_id, frozenset()): + # This cell's own sample never offered the identity, so its reading is not a + # vote. The denominator counts only members whose own sample offered it, so + # counting this would mix two populations. + continue + bucket = explicit_counts.setdefault((set_id, key), {}) + bucket[state] = bucket.get(state, 0) + 1 + + minority_cells: dict[str, int] = {} + cells_compared: dict[str, int] = {} + for row in tally.iter_rows(named=True): + set_id, key = row["setId"], row["identity"] + counts = dict(explicit_counts.get((set_id, key), {})) + counts[State.NOT_BOUND.value] = counts.get(State.NOT_BOUND.value, 0) + row["silentNotBound"] + evaluable = sum(counts.values()) + if evaluable < 2: + # One cell has no minority of its own set to sit in. Left out of BOTH counts, + # so a key whose every set is a singleton reports nothing to compare rather + # than a rate of zero, which would read as agreement. + continue + cells_compared[key] = cells_compared.get(key, 0) + evaluable + # SETTLED holds exactly BOUND and NOT_BOUND, so the majority is one of two numbers + # and every other evaluable cell is in the minority. The zero-valued entry the + # silent add above can create contributes to neither term. + minority_cells[key] = minority_cells.get(key, 0) + (evaluable - max(counts.values())) + + return ( + pl.DataFrame({"key": sorted(universe)}) + .with_columns( + pl.col("key").replace_strict(cells_compared, default=0, return_dtype=pl.Int64).alias("cellsCompared"), + pl.col("key").replace_strict(minority_cells, default=0, return_dtype=pl.Int64).alias("minorityCells"), + ) + .with_columns( + pl.when(pl.col("cellsCompared") > 0) + .then(pl.col("minorityCells") / pl.col("cellsCompared")) + .otherwise(None) + .alias("disagreementRate"), + pl.lit("tag").alias("level"), + pl.lit("true").alias("diagnosticOnly"), + ) + .sort("key") + ) diff --git a/software/per-cell-metrics/src/emit_csv_meta.py b/software/per-cell-metrics/src/emit_csv_meta.py deleted file mode 100644 index 7e6825c..0000000 --- a/software/per-cell-metrics/src/emit_csv_meta.py +++ /dev/null @@ -1,59 +0,0 @@ -"""Emit the tag->feature CSV's column headers and the distinct values of each column as one JSON. - -A single staging pre-step, replacing the former emit-columns + emit-features pair. -The block's barcode-sequence / feature-name column dropdowns read ``columns``; the negative-control -dropdown reads ``valuesByColumn[]`` for whichever column the user maps to the -feature-name role. Emitting the distinct values of EVERY column up front (the tag->feature CSV is a -small feature panel) lets the control dropdown populate the instant the feature column is picked — -with no second staging exec and no staging rerun; the model just indexes the already-emitted map. - -Header order is preserved so the column dropdowns read top-to-bottom like the file; per-column values -are deduplicated and sorted so the output is deterministic (canonical). Stdlib only — trivial and fast. -""" - -import argparse -import csv -import json - - -def main() -> None: - p = argparse.ArgumentParser() - p.add_argument("tag_feature_csv", help="tag->feature CSV") - p.add_argument("output", help="output JSON file ({columns, valuesByColumn})") - args = p.parse_args() - - with open(args.tag_feature_csv, newline="") as fh: - reader = csv.reader(fh) - header = next(reader, None) - if not header: - raise SystemExit(f"no header row found in {args.tag_feature_csv}") - # Preserve header order, drop blank header cells; keep each named column's row index so we can - # collect its values by position (DictReader would lose order and collapse duplicate headers). - columns = [h.strip() for h in header if h.strip()] - col_index = {h.strip(): i for i, h in enumerate(header) if h.strip()} - values: dict[str, set[str]] = {c: set() for c in columns} - # Total data rows (header excluded). The model compares a column's DISTINCT-value count against - # this to detect a barcode that appears on more than one row — the case per_cell_metrics.py guards - # (a duplicated barcode fans the join). Fully-blank rows (e.g. a trailing newline) are not counted, - # so they never look like a duplicate. - row_count = 0 - for row in reader: - if not any(cell.strip() for cell in row): - continue - row_count += 1 - for c in columns: - i = col_index[c] - if i < len(row) and row[i].strip(): - values[c].add(row[i].strip()) - - meta = { - "columns": columns, - "valuesByColumn": {c: sorted(values[c]) for c in columns}, - "rowCount": row_count, - } - with open(args.output, "w") as out: - json.dump(meta, out) - - -if __name__ == "__main__": - main() diff --git a/software/per-cell-metrics/src/emit_feature_properties.py b/software/per-cell-metrics/src/emit_feature_properties.py index 53816c7..c503bf2 100644 --- a/software/per-cell-metrics/src/emit_feature_properties.py +++ b/software/per-cell-metrics/src/emit_feature_properties.py @@ -1,25 +1,25 @@ """Emit per-feature properties from the tag->feature CSV's EXTRA columns. -The user maps two columns of the tag CSV to roles: the feature-barcode sequence (the join key / -panel) and the feature name (the antigen identity). Every OTHER column is an arbitrary per-feature -property — antigen type, species of origin (human / cyno), pool, etc. This step imports each such -column generically (no hardcoded schema) so the workflow can surface it as a p-column keyed on the -feature axis, and the property then rides that axis into VDJ Multiomic Integration's per-feature -outputs and lead selection (see spec A-0026). - -Two outputs (stdlib only -- a trivial, fast pre-step like emit_csv_meta / emit_panel): - * ``_feature_properties.csv`` -- one row per distinct feature; columns are ``feature`` - (the feature-name column, always renamed to ``feature`` so the downstream xsv import keys it on - the feature axis) followed by every extra column under its own header. Rows sorted by feature. - * ``_feature_property_meta.json`` -- ``{columns, valuesByColumn}``: the property column - names in file (header) order, and each property's distinct non-empty values (sorted). The - workflow reads this to build one import column per property (and each column's discreteValues). - -Deduplication: a feature barcode can map many-to-one onto a feature, so a feature appears on several -CSV rows. Properties are intrinsic to the feature, so they are expected to agree across those rows; -the FIRST non-empty value (in file order) wins and a differing later value is reported to stderr but -not treated as fatal -- Feature Integration imports values as given and does not validate them -(A-0026). The output is fully deterministic for a given input file. +The user maps two columns to roles: the feature-barcode sequence (the join key) and the +feature name (the antigen identity). Every OTHER column is an arbitrary per-feature +property -- antigen type, species of origin, pool. This step imports each generically, +with no hardcoded schema, so the workflow can surface it as a p-column keyed on the +feature axis. The property then rides that axis into VDJ Multiomic Integration's +per-feature outputs and lead selection (spec A-0026). + +Two outputs, stdlib only: + * ``_feature_properties.csv`` -- one row per distinct feature. Columns are + ``feature`` (always renamed, so the downstream xsv import keys it on the feature + axis) followed by every extra column under its own header. Sorted by feature. + * ``_feature_property_meta.json`` -- ``{columns, valuesByColumn}``: property + names in header order, and each property's distinct non-empty values, sorted. The + workflow reads this to build one import column per property. + +Deduplication: a barcode can map many-to-one onto a feature, so a feature appears on +several rows. Properties are intrinsic to the feature and are expected to agree. The +FIRST non-empty value in file order wins, and a differing later value is reported to +stderr but is not fatal -- Feature Integration imports values as given and does not +validate them (A-0026). Output is fully deterministic for a given input file. """ import argparse @@ -29,9 +29,9 @@ def _read_header(reader: "csv._reader") -> tuple[list[str], dict[str, int]]: - """Return (ordered named columns, name -> first column index). Blank header cells are dropped; - the first index wins for a duplicated header (read by position so order is preserved -- DictReader - would collapse duplicates and lose order, mirroring emit_csv_meta).""" + """Return (ordered named columns, name -> first column index). Blank header cells are + dropped, and the first index wins for a duplicated header. Read by position so order is + preserved -- DictReader would collapse duplicates and lose order.""" header = next(reader, None) if not header: raise SystemExit("no header row found in the tag->feature CSV") @@ -51,12 +51,12 @@ def parse_properties( feature_col: str, property_cols: list[str], ) -> tuple[dict[str, dict[str, str]], dict[str, set[str]]]: - """Collapse raw CSV rows into per-feature property values + each property's distinct value set. + """Collapse raw CSV rows into per-feature property values and each property's value set. - ``by_feature[feature][property]`` = the first non-empty value seen for that (feature, property), - in file order. ``values[property]`` = the set of distinct non-empty values. A later row that - disagrees with an already-recorded non-empty value is reported to stderr (kept, not overwritten). - Pure and unit-tested; the CLI wires it to CSV I/O. + ``by_feature[feature][property]`` is the first non-empty value seen for that pair, in file + order. ``values[property]`` is the set of distinct non-empty values. A later row that + disagrees with a recorded non-empty value is reported to stderr and kept, not overwritten. + Pure and unit-tested. The CLI wires it to CSV I/O. """ by_feature: dict[str, dict[str, str]] = {} values: dict[str, set[str]] = {c: set() for c in property_cols} @@ -103,6 +103,15 @@ def main() -> None: default="", help="optional sample column for sample-aware mapping (a role column, excluded from properties)", ) + p.add_argument( + "--control-feature", + action="append", + default=[], + help="a negative-control feature name (from the block's control-feature dropdown). Repeat the flag " + "for each control. Emitted as a dedicated per-feature marker so downstream can remove the controls " + "from its antigen metrics. Repeated rather than comma-joined because a feature name may contain a " + "comma.", + ) p.add_argument("--output-prefix", default="result") args = p.parse_args() @@ -114,8 +123,8 @@ def main() -> None: raise SystemExit(f"{role}={name!r} not found in the tag->feature CSV (columns: {ordered})") rows = list(reader) - # Role columns are excluded; every remaining named column is a per-feature property. The sample - # column (sample-aware mapping) is a role too, so it is excluded when set -- it is not a property. + # Role columns are excluded, and every remaining named column is a property. The sample + # column is a role too, so it is excluded when set. roles = {args.csv_barcode_col, args.csv_feature_col} if args.sample_col: roles.add(args.sample_col) @@ -129,9 +138,9 @@ def main() -> None: file=sys.stderr, ) - # Wide per-feature CSV: feature + one column per property, one row per feature (sorted), missing - # values blank. The feature-name column is always emitted as 'feature' so the downstream xsv import - # keys it on pl7.app/feature/featureId regardless of the source header. + # Wide per-feature CSV: feature plus one column per property, one row per feature, sorted, + # missing values blank. The feature-name column is always emitted as 'feature', so the xsv + # import keys it on pl7.app/feature/featureId whatever the source header. with open(f"{args.output_prefix}_feature_properties.csv", "w", newline="") as out: w = csv.writer(out) w.writerow(["feature"] + property_cols) @@ -139,12 +148,36 @@ def main() -> None: props = by_feature[feature] w.writerow([feature] + [props.get(c, "") for c in property_cols]) - # Meta: property column names (header order) + each property's distinct sorted values. The workflow - # builds one import column per name and puts the values into that column's discreteValues. + # Meta: property names in header order plus each property's distinct sorted values. The + # workflow builds one import column per name and puts the values into its discreteValues. meta = {"columns": property_cols, "valuesByColumn": {c: sorted(values[c]) for c in property_cols}} with open(f"{args.output_prefix}_feature_property_meta.json", "w") as out: json.dump(meta, out) + # Negative-control marker. Emit each chosen control feature as a row of a (feature, value) CSV with + # value "true", so the workflow surfaces it as a pl7.app/feature/negativeControl column and VDJ + # Multiomic Integration removes those controls ENTIRELY from its antigen metrics -- restriction index, + # breadth, per-antigen fractions, dominant call. An off-target, by contrast, stays in the metrics. + # Header-only when no control is designated. Names are emitted verbatim, trimmed. + # + # SEVERAL controls are allowed: being a control is a property of the tag, and a panel may carry more + # than one, where being the reference that supplies the baseline is a job given to exactly one of them. + # This file marks controls and never nominates a reference, so it takes as many as the panel has. + # Duplicates are dropped and the given order kept, because the marker is a set and a stable file is + # easier to diff. + seen: set[str] = set() + controls = [] + for raw in args.control_feature: + name = raw.strip() + if name and name not in seen: + seen.add(name) + controls.append(name) + with open(f"{args.output_prefix}_negative_control.csv", "w", newline="") as out: + w = csv.writer(out) + w.writerow(["feature", "value"]) + for name in controls: + w.writerow([name, "true"]) + if __name__ == "__main__": main() diff --git a/software/per-cell-metrics/src/emit_panel.py b/software/per-cell-metrics/src/emit_panel.py index bd9c553..a95b022 100644 --- a/software/per-cell-metrics/src/emit_panel.py +++ b/software/per-cell-metrics/src/emit_panel.py @@ -1,11 +1,10 @@ """Emit the feature-barcode panel as a plain sequence list for mitool refine-tags whitelist correction (``-t FEATURE#file:panel.txt``). -The panel is the user-provided tag->feature CSV's tag column: the -authoritative set of feature barcodes. We write one barcode per line, deduplicated and -sorted, so the output is deterministic (canonical) and the workflow's pure-template dedup -stays stable. Only the standard library is used -- no polars -- so this stays a trivial, -fast pre-step. +The panel is the tag column of the user's tag->feature CSV: the authoritative set of +feature barcodes. One barcode per line, deduplicated and sorted, so the output is +canonical and the workflow's pure-template dedup stays stable. Standard library only, so +this stays a trivial, fast pre-step. """ import argparse @@ -25,7 +24,10 @@ def main() -> None: raise SystemExit( f"column {args.tag_col!r} not found in {args.tag_feature_csv} (columns: {reader.fieldnames})" ) - seqs = {row[args.tag_col].strip() for row in reader if row.get(args.tag_col, "").strip()} + # `or ""` rather than a get() default: a short row's missing columns are + # present-and-None in DictReader's output, not absent, so the default never fires + # and .strip() met None. One malformed line took the whole run down. + seqs = {seq for seq in ((row.get(args.tag_col) or "").strip() for row in reader) if seq} if not seqs: raise SystemExit(f"no feature barcodes found in column {args.tag_col!r}") diff --git a/software/per-cell-metrics/src/emit_verdicts.py b/software/per-cell-metrics/src/emit_verdicts.py new file mode 100644 index 0000000..dd9f6c0 --- /dev/null +++ b/software/per-cell-metrics/src/emit_verdicts.py @@ -0,0 +1,1494 @@ +"""The entrypoint: counts, a panel and a cell list become a four-state verdict. + +Composes the reading in one order, and the order is load-bearing at every step. The +floor works on the raw per-(cell, tag) counts. A cell's reference reading is taken from +the floored frame. Tags combine into an identity by the highest of their counts. The +identity's count is read against that cell's own reference. A set's cells combine by +majority. Reversing any pair changes the answer: flooring after combining would floor +one reading where two were taken, and taking the reference before the floor would +compare against a number the floor has already been applied to elsewhere. + +**The grid of every cell against every identity is never built.** A silent cell -- asked +about an identity and showing no reading for it -- scores `specificity_score(0, r)`, at +most ~0.0422 and falling as the reference rises, so it settles *not bound* unless the +cell itself cannot be compared. `silent_tally` counts those positions analytically, +because on a realistic panel the grid is 11-20x the sparse input and a pMHC panel does +not fit at all. Two consequences are enforced here rather than downstream: a `--cutoff` +at or below that ~0.0422 bound is refused, and verdict.py's row-per-position reference +implementation is never called from production, which the test suite asserts by checking +this file does not name it. + +`offered` is keyed by SAMPLE throughout and is never regrouped by set. Staining is done +per sample, so a set spanning two samples was offered whatever either panel offered, and +`combine_cells` takes that union itself. Keying the map by set instead makes every lookup +miss, reads every offered set as empty, and raises nothing. + +One `Admissibility` bundle is built and handed to `read_states`, `combine_cells` and +`self_disagreement` alike, so they cannot be given different reference dicts and then +disagree about which cells "cannot be compared" -- which surfaces as a silent-position +count that is wrong or negative rather than as an error. + +Every frame is sorted before it is written. `combine_tags_to_identities` groups without +maintaining order, so an unsorted frame varies run to run. A p-column's identity is its +content, and an unstable byte order costs every downstream node its dedup. +""" + +from __future__ import annotations + +import argparse +import functools +import json +import sys + +import numpy as np +import polars as pl +from combine import ( + DEFAULT_MIN_AGREEMENT, + DEFAULT_MIN_VOTERS, + attach_competitor_notes, + combine_cells, + self_disagreement, + set_counts, +) +from frame_io import ( + _json_arg, + _read_columns, + _read_counts, + _read_raw_feature_counts, + _write_sorted, + undeclared_feature_counts, +) +from identity_tables import ( + IDENTITY_KEY_COLUMN, + IDENTITY_SUMMARY_MAX_IDENTITIES, + CellKey, + _build_grouping, + _cells_by_set, + _declared_by_sample, + _grouping_columns, + _identity_labels, + _identity_properties, + _linker_frame, + _panel_id, + _pivot_cell_punch, + _pivot_identity_summary, + count_by_set, +) +from panel import ( + ANY_SAMPLE, + consistent_properties, + default_grouping, + identity_universe, + offered_identities, + panel_read_mismatch, + property_columns, + read_panel, +) +from qc_measures import ( + DEFAULT_LINES, + Coverage, + Line, + antigen_count_deciles, + deciles_of, + per_antigen_measures, + reads_per_cell, + sibling_disagreement, + status_for, + usable_read_fraction, +) +from qc_rows import ( + _BACKGROUND_SCHEMA, + _DECILE_SCHEMA, + _REAGENT_SCHEMA, + _SAMPLE_DECILE_SCHEMA, + _UNDECLARED_BARCODE_SCHEMA, + ROLLUP, + QcRow, + _add, + _decile_rows, + _fitted_background, + _median, + _number, + _qc_frame, + _sample_decile_rows, + _score_spread, + _sticky_measure, + sample_report_rows, + sample_summary_rows, +) +from tag_distribution import ( + DEFAULT_DISTRIBUTION_MIN_CELLS, + TagFits, + fit_tag_probabilities_by_pair, +) +from verdict import ( + BOUND_CUTOFF, + DEFAULT_FLOOR, + DEFAULT_PANEL_MIN_MEMBERS, + Admissibility, + Reference, + ReferenceChoice, + apply_floor, + cell_admissibility_reason, + cells_reading_nothing, + combine_tags_to_identities, + gate_cells, + read_states, + reference_by_cell, + specificity_score, +) + + +def _identity_probabilities(fits, grouping) -> dict[tuple[str, str, str], float]: + """Per (sample, cell, identity), the highest probability among the identity's own tags. + + `tags-combine-by-the-highest` fixes the combination: an identity's reading in a cell is + the highest of its tags and never their sum, because tags differ in uptake and a sum + would need the baseline scaled to match. The same rule applies to a probability -- the + identity is bound in that cell where any one of its tags says so, which is also how + `what-plays-the-baseline` reads a population rung's identity. + + A (tag, sample) pair that established nothing contributes nothing, so an identity all of + whose tags missed carries no key and reads *unreliable* rather than a low probability. + """ + out: dict[tuple[str, str, str], float] = {} + for row in fits.probabilities.iter_rows(named=True): + identity = grouping.get((row["tag"], row["sampleId"])) or grouping.get((row["tag"], ANY_SAMPLE)) + if identity is None: + continue + key = (row["sampleId"], row["cellId"], identity) + p = float(row["pBound"]) + if p > out.get(key, -1.0): + out[key] = p + return out + + +def _cell_keyed_reference(counts, reference_tags, source, analysed_cells, panel_size, args) -> Reference: + """The comparator for the rungs keyed by cell: a declared reagent, or the panel's own readings. + + Raw counts, never floored. Each rung computes its baseline from its own source, and + the minimum acts on the identity's reading -- the numerator -- never on the comparator. + The floored frame would make the panel rung's median a mixture of raw values (reference + tags, which the minimum exempts) and floored ones. The declared rung is unaffected + either way, since a reference tag's reading is already exempt. + """ + return reference_by_cell( + counts, + reference_tags, + source, + cells=analysed_cells, + panel_size=panel_size, + min_members=args.panel_min_members, + ) + + +# A silent cell's count is zero, and a zero count's best possible score is +# specificity_score(0, 0). At or below it, the analytic silent count and the +# row-per-position reference part company over a silent admissible cell, quietly: one +# calls it bound, the other not bound, and nothing raises. `silent_tally` states that +# refusing such a cutoff belongs to the CLI. +ANALYTIC_CUTOFF_BOUND = float(specificity_score(0, 0)) + + +# Long on purpose and not decomposed. This is one composition taken in the one order the +# reading has, and splitting it into stages would put that order in the call sites rather +# than in the code a reader follows top to bottom. +def main() -> None: + p = argparse.ArgumentParser(description="Read antigen counts into a four-state binding verdict per set.") + p.add_argument( + "counts_csv", help="sparse per-(sampleId, cellId, tag) UMI counts, with an optional totalWeight column" + ) + p.add_argument("panel_csv", help="the panel file: which tags each sample was stained with") + p.add_argument("--linker", default=None, help="cell -> clonotype set CSV (sampleId, cellId, setId)") + p.add_argument("--cells", default=None, help="the cell list (sampleId, cellId); overrides the linker's cells") + p.add_argument("--barcode-col", default="tag", help="panel column holding the barcode sequence") + p.add_argument("--feature-col", default="feature", help="panel column holding the antigen name") + p.add_argument("--sample-col", default="", help="panel column holding the sample; empty declares one panel for all") + p.add_argument("--role-column", default="", help="panel column declaring each tag's role") + p.add_argument("--reference-values", default="", help="comma-separated role values marking a comparator tag") + p.add_argument( + "--reference-source", + required=True, + # Derived from the enum, never restated. A hard-coded list lets the CLI reject a new + # rung that every layer above it accepts. + choices=[choice.value for choice in ReferenceChoice], + help=( + "which comparator to ask for; the run may serve 'none' instead, never a different one. " + "Required: nothing here picks a rung for a scientist who did not" + ), + ) + p.add_argument("--panel-min-members", type=int, default=DEFAULT_PANEL_MIN_MEMBERS) + p.add_argument( + "--distribution-min-cells", + type=int, + default=DEFAULT_DISTRIBUTION_MIN_CELLS, + help="cells a sample needs before a tag's own distribution may serve as its baseline", + ) + p.add_argument("--floor", type=int, default=DEFAULT_FLOOR, help="zero every non-comparator reading below this") + p.add_argument( + "--cutoff", type=float, default=BOUND_CUTOFF, help="specificity score at or above which a cell binds" + ) + p.add_argument("--min-voters", type=int, default=DEFAULT_MIN_VOTERS) + p.add_argument("--min-agreement", type=float, default=DEFAULT_MIN_AGREEMENT) + p.add_argument("--gate-threshold", type=int, default=None, help="set aside cells whose comparator reads this high") + p.add_argument("--grouping", default=None, help="JSON: {'by':'tag'} or {'by':'property','column':...}") + p.add_argument("--contending", default=None, help="JSON: groups of identities that contend, as a list of lists") + # Accepted and not yet read. The capture rollup was its only reader, and only the sample + # carries an aggregated status now. It stays declared because the capture axis ships on the + # QC columns for the same reason: adding an axis to a released column changes that column's + # identity, where adding a value does not. + p.add_argument("--capture-map", default=None, help="JSON: sampleId -> captureId (accepted, not yet read)") + p.add_argument( + "--sample-labels", + default=None, + help="JSON: sampleId -> the label the panel file writes for it, when the two differ", + ) + p.add_argument( + "--qc-summary", default=None, help="per-sample read QC CSV (sampleId, readsTotal, readsMatched, ...)" + ) + p.add_argument( + "--raw-feature-counts", + default=None, + help="gathered pre-refine tag-stat -t FEATURE table (sampleId, FEATURE, totalWeight)", + ) + # The four inherited lines (`315-where-the-lines-come-from`), each with a shipped default. + # Restated here rather than left to qc_measures.DEFAULT_LINES, so the value that scored a + # run is always on the command line. `error` is omitted for readsPerCell: the field + # published one boundary, so depth warns and never alerts. + default_lines = DEFAULT_LINES + p.add_argument("--cell-barcode-valid-warn", type=float, default=default_lines["cellBarcodeValidFraction"].warn) + p.add_argument("--cell-barcode-valid-error", type=float, default=default_lines["cellBarcodeValidFraction"].error) + p.add_argument("--reads-per-cell-warn", type=float, default=default_lines["readsPerCell"].warn) + p.add_argument("--aggregate-barcode-warn", type=float, default=default_lines["aggregateBarcodeFraction"].warn) + p.add_argument("--aggregate-barcode-error", type=float, default=default_lines["aggregateBarcodeFraction"].error) + p.add_argument("--undeclared-barcode-warn", type=float, default=default_lines["undeclaredBarcodeShare"].warn) + p.add_argument("--undeclared-barcode-error", type=float, default=default_lines["undeclaredBarcodeShare"].error) + p.add_argument("--usable-read-warn", type=float, default=default_lines["usableReadFraction"].warn) + p.add_argument("--usable-read-error", type=float, default=default_lines["usableReadFraction"].error) + p.add_argument("--output-prefix", default="result") + args = p.parse_args() + + # Every line an operator may move, none invented: a measurement absent from this dict + # carries no status, whatever DEFAULT_LINES says elsewhere. + lines: dict[str, Line] = { + "cellBarcodeValidFraction": Line(warn=args.cell_barcode_valid_warn, error=args.cell_barcode_valid_error), + "readsPerCell": Line(warn=args.reads_per_cell_warn), + "aggregateBarcodeFraction": Line(warn=args.aggregate_barcode_warn, error=args.aggregate_barcode_error), + "undeclaredBarcodeShare": Line(warn=args.undeclared_barcode_warn, error=args.undeclared_barcode_error), + "usableReadFraction": Line(warn=args.usable_read_warn, error=args.usable_read_error), + } + add = functools.partial(_add, lines=lines) + + if args.cutoff <= ANALYTIC_CUTOFF_BOUND: + raise SystemExit( + f"--cutoff must be strictly above {ANALYTIC_CUTOFF_BOUND:.4f}, the best score a zero count can reach. " + f"At or below it a cell that was asked and read nothing settles one way when counted and the other " + f"when written out, with no error raised. Got {args.cutoff}." + ) + + prefix = args.output_prefix + roles = {"barcode": args.barcode_col, "feature": args.feature_col} + if args.sample_col: + roles["sample"] = args.sample_col + panel, dropped_lines = read_panel(args.panel_csv, roles) + + # The panel file names samples the way a scientist does, "donor01", while the counts, the + # linker and every axis this run emits are keyed by the platform's sampleId. Nothing else + # bridges the two namespaces. Unbridged, `offered` ends up keyed by labels, no sample that + # exists is offered anything, and every verdict comes back *never asked* -- the correct + # answer to a question nobody asked, so it raises nothing. Translation happens HERE, once, + # before the panel is used for anything. A value the map does not mention is left alone + # rather than dropped, because a panel row naming a sample this run does not have is a real + # mismatch and should reach the mismatch report. + label_of_sample: dict[str, str] = _json_arg(args.sample_labels, "--sample-labels") or {} + if label_of_sample and "sample" in panel.columns: + by_label: dict[str, str] = {} + for sample_id, label in sorted(label_of_sample.items()): + if label in by_label: + raise SystemExit( + f"--sample-labels gives label {label!r} to both {by_label[label]!r} and " + f"{sample_id!r}; a panel row naming it cannot be resolved to one sample" + ) + by_label[label] = sample_id + panel = panel.with_columns(pl.col("sample").replace(by_label)) + + prop_cols = property_columns(panel) + properties, inconsistent = consistent_properties(panel, prop_cols) + # Kept, not merely reported: the values a tag disagreed about are what a fallback identity + # is labelled with, and `properties` holds only what a tag agreed on. + disagreed_by_column: dict[str, dict[str, list[str]]] = {} + for tag, column, values in inconsistent: + disagreed_by_column.setdefault(column, {})[tag] = sorted(values) + for tag, column, values in inconsistent: + print( + f"[emit-verdicts] tag {tag!r} declares {column!r} as {values}; it carries no agreed value", file=sys.stderr + ) + + # The reference designation is read through `consistent_properties`, which strips the value + # and drops any property a tag's rows disagree about. A per-sample comparator designation is + # therefore discarded rather than honoured, which is what `apply_floor` documents. + reference_values = {v.strip() for v in args.reference_values.split(",") if v.strip()} + reference_tags: set[str] = set() + # The column is checked whenever one is named, never only when values are named with it. + # Gating the check on `reference_values` leaves the worse half silent: a role column the + # panel does not declare designates no tag, and the baseline falls back to the panel's own + # readings without a word -- a different number reported as the requested one. + if args.role_column and args.role_column not in prop_cols: + raise SystemExit(f"--role-column {args.role_column!r} is not a panel column; columns are {prop_cols}") + if args.role_column and reference_values: + reference_tags = {t for t, props in properties.items() if props.get(args.role_column) in reference_values} + + grouping_rule = _json_arg(args.grouping, "--grouping") + grouping, grouping_id, ungrouped_tags, grouped_on = _build_grouping( + grouping_rule, panel, properties, reference_tags + ) + universe = identity_universe(panel, grouping) + by_tag_grouping = default_grouping(panel, reference_tags) + tag_universe = identity_universe(panel, by_tag_grouping) + + # Validated as a list of lists before it is read as one. A flat `["AgA","AgB"]` is valid + # JSON, and `set("AgA")` is a set of CHARACTERS, so the run completes, no competitor note + # fires, every `wasCompeted` reads false, and the run record states a contention that was + # never tested -- a silent wrong answer, and the shape a hand-driven run reaches for. + contending_raw = _json_arg(args.contending, "--contending") or [] + if not isinstance(contending_raw, list): + raise SystemExit(f"--contending must be a JSON list of lists of identities; got {contending_raw!r}") + for group in contending_raw: + if not isinstance(group, list) or not all(isinstance(member, str) for member in group): + raise SystemExit( + f"--contending must be a JSON list of LISTS of identities; {group!r} is not a list of " + 'strings. A flat list such as ["AgA","AgB"] declares no group -- it reads each name as ' + "its own set of characters." + ) + if len(group) < 2: + raise SystemExit( + f"--contending group {group!r} has fewer than two members; an identity cannot contend " + "with itself, and a group of one tests nothing." + ) + contending = [set(group) for group in contending_raw] + + counts = _read_counts(args.counts_csv) + + # The cell list is an input, never derived from the antigen readings: nothing in the counts + # separates a cell from a droplet that held none. `--cells` wins over the linker where both + # arrive, because a list from gene expression covers cells whose receptor never assembled. + linker = ( + _read_columns(args.linker, ("sampleId", "cellId", "setId"), "linker file") + if args.linker + else pl.DataFrame(schema={"sampleId": pl.String, "cellId": pl.String, "setId": pl.String}) + ) + cells_by_set = _cells_by_set(linker) + linker_cells = {key for keys in cells_by_set.values() for key in keys} + if args.cells: + listed = _read_columns(args.cells, ("sampleId", "cellId"), "cell list") + cell_list = set(listed.iter_rows()) + cell_list_source = "cell list" + elif args.linker: + cell_list = linker_cells + cell_list_source = "clonotype linker" + else: + # No list arrived, and one is NOT derived from the counts. Nothing in the antigen + # readings separates a cell from a droplet that held none, so the observed barcodes are + # not a cell list: in droplet data they outnumber the cells by one to two orders of + # magnitude, because ambient material lands on most barcodes. Standing them in would be + # worse than approximate, since `readsPerCell` divides by this and a healthy library + # would read undersequenced and alert. Every barcode is still analysed and every count + # still emitted. What is withheld is the claim that these barcodes are cells: + # `inCellList` is unknown rather than true, and the measurements needing a cell list + # read *not evaluated*. + cell_list = None + cell_list_source = "none" + + # `cell_list is None` means no list arrived, which differs from a list that arrived empty: + # the first cannot answer "is this barcode a cell", the second answers "no". `listed` + # collapses both for the set arithmetic below, where either way there are no barcodes to + # add. + listed = cell_list if cell_list is not None else set() + + observed_cells = set(counts.select("sampleId", "cellId").unique().iter_rows()) + # Barcodes outside the cell list stay in the frame, labelled. One dropped here is + # indistinguishable afterwards from one that never existed, and its antigen counts are real + # whatever the list says about it. + analysed_cells = sorted(listed | observed_cells | linker_cells) + + panel_samples = {s for s in panel["sample"].to_list() if s != ANY_SAMPLE} + samples = sorted( + {s for s, _ in observed_cells} | {s for s, _ in listed} | {s for s, _ in linker_cells} | panel_samples + ) + + # The floor is applied per sample, so the counters it returns land in each sample's own QC + # row. A cell key carries its sample, so partitioning is exact on both counters and the run + # totals are their sums. There is no second implementation of the rule to drift from this + # one. + floor_stats: dict[str, dict[str, int]] = {} + parts = [] + for sample in samples: + floored_part = apply_floor( + counts.filter(pl.col("sampleId") == sample), + args.floor, + reference_tags, + ) + parts.append(floored_part.counts) + floor_stats[sample] = floored_part.stats + floored = pl.concat(parts) if parts else counts + readings_floored = sum(s["readingsFloored"] for s in floor_stats.values()) + cells_emptied = sum(s["cellsEmptied"] for s in floor_stats.values()) + + # One panel size, read once and passed to both. Deriving it separately for the default + # choice and for the resolution would let the two disagree about whether the panel is large + # enough to serve as its own comparator. + panel_size = int(panel["tag"].n_unique()) + + # No default and no derivation. The rung is the scientist's choice, and a run that carried + # none is a configuration error rather than a run to guess at. argparse refuses it above, + # so this never sees an empty value. + source = ReferenceChoice[args.reference_source.upper()] + tag_fits: TagFits | None = None + # Set only by the one rung whose conditions the settings cannot answer. The other two + # refuse in `served_source` before any of this runs. + no_baseline_reason: str | None = None + if source is ReferenceChoice.DISTRIBUTION: + # Keyed by (sample, identity) and never by cell: this rung fits one distribution per + # tag across a sample's cells, so its answer is the same number for every cell of a + # sample and a different one for every identity. `reference_by_cell` has nothing to + # return for it. Fitted over the RAW counts and the FULL cell universe -- the cells that + # read nothing, and the cells a gate will later set aside. That second part is + # `baseline-over-all-returned-cells`, which is also why the fit runs before + # `gate_cells` below. + tag_fits = fit_tag_probabilities_by_pair(counts, analysed_cells, panel, args.distribution_min_cells) + probabilities = _identity_probabilities(tag_fits, grouping) + # A run where no tag fitted anywhere established no baseline. This is the one refusal + # that cannot be caught from the settings: whether a sample holds three hundred cells + # whose counts admit a two-component fit is a property of the data. So the run FINISHES, + # says so, and draws no punchcard -- rather than answering every position *unreliable*, + # which is honest and useless, or crashing after doing the work. + reference = Reference({}, ReferenceChoice.DISTRIBUTION) + by_identity = None + if not probabilities: + no_baseline_reason = ( + "no baseline could be established: the tag-distribution rung was selected and no tag's " + f"counts admitted a two-component fit in any sample, against the " + f"{args.distribution_min_cells} cells this rung requires. Whether a sample can support " + "this rung is a property of the data rather than of the settings, so it could not be " + "caught before the run. The run's quality measurements are below; no verdicts were read." + ) + probabilities = None + tag_probabilities = _identity_probabilities(tag_fits, by_tag_grouping) or None + else: + by_identity = None + probabilities = None + tag_probabilities = None + reference = _cell_keyed_reference(counts, reference_tags, source, analysed_cells, panel_size, args) + + gated, cells_high_reference = gate_cells(reference.by_cell, args.gate_threshold) + if reference.served is ReferenceChoice.DISTRIBUTION: + # No per-cell comparator exists to read a gate against, so the gate sets nothing aside + # and the exposure count is not a measurement this run made. None, never 0: a zero would + # report a run with no high background rather than one where the question does not + # arise. + cells_high_reference = None + + # Built once and handed to every consumer. Two bundles built from two reference dicts do + # not raise. They disagree about which cells cannot be compared, and the silent-position + # count comes out wrong or negative. + admissibility = Admissibility(reference.by_cell, gated, by_identity, probabilities) + + non_reference = floored.filter(~pl.col("tag").is_in(list(reference_tags))) if reference_tags else floored + identities = combine_tags_to_identities(non_reference, grouping) + states = read_states(identities, admissibility, args.cutoff) + + # The per-tag reading is diagnostic only: it compares each tag against the reference + # separately, and no verdict is built from it. The measurement set carries it at both levels + # always, so where the chosen grouping is not the per-tag one it is read a second time. + if grouping == by_tag_grouping: + tag_states = states + else: + # A second bundle, because the per-tag read asks about different identities. Where the + # comparator is keyed by identity, the bundle built for the chosen grouping answers + # about identities this read never mentions. + tag_admissibility = ( + Admissibility(reference.by_cell, gated, None, tag_probabilities) + if (by_identity is not None or tag_probabilities is not None) + else admissibility + ) + tag_states = read_states( + combine_tags_to_identities(non_reference, by_tag_grouping), tag_admissibility, args.cutoff + ) + + # Which (sample, tag) pairs the reads actually carry, from the RAW counts. Never from + # `floored`: a count the minimum zeroed is a reading that happened and failed, and settles + # *not bound*, while a tag with no reads at all is a question nobody put. Reading the + # floored frame here would turn a dead reagent into a confident clean negative on every + # clonotype in the run. + seen_pairs = { + (row["sampleId"], row["tag"]) for row in counts.select("sampleId", "tag").unique().iter_rows(named=True) + } + offered_by_sample = {s: offered_identities(panel, grouping, [s], seen_pairs) for s in samples} + tag_offered_by_sample = {s: offered_identities(panel, by_tag_grouping, [s], seen_pairs) for s in samples} + + def _answers(frame: pl.DataFrame) -> pl.DataFrame: + """The frame, or its headers alone where the run established no baseline. + + A run with no baseline read no verdicts, so the frames carrying answers carry no rows. + They keep their schemas, because every reader still needs to find its columns, and a + missing file reads as a stage that crashed rather than one that finished and said why. + + Emitting the answers instead would fill every position with *unreliable*: honest and + useless, costing what a real run costs and looking like a result at a glance. + + The STRUCTURAL frames are written in full either way -- which tags feed which identity, + what each sample was offered, the panel and identity labels. Those describe the run + rather than answering it, and a reader working out why no baseline could be established + needs them. + """ + return frame.clear() if no_baseline_reason else frame + + verdicts = attach_competitor_notes( + combine_cells( + states, + universe, + offered_by_sample, + cells_by_set, + admissibility, + args.min_voters, + args.min_agreement, + ), + contending, + ) + _write_sorted(_answers(verdicts), f"{prefix}_verdicts.csv", ["setId", "identity"]) + # The set's own cell count, joined on rather than computed inside `set_counts`, which is a + # pure reading of the verdicts frame at its (setId, identity) grain where a cell count does + # not live. It is the set's cells, NOT its answering cells: that number varies by identity + # and travels with the verdict as support. This one is a property of the clonotype, which is + # why `the-explore-readout` puts it beside the name instead of in every position. + per_set_cells = pl.DataFrame( + [(set_id, len(cells)) for set_id, cells in sorted(cells_by_set.items())], + orient="row", + schema={"setId": pl.String, "cellCount": pl.Int64}, + ) + # Set-aside cells PER CLONOTYPE, never per run. `the-explore-readout` states them once for + # the clonotype, and the run-level total in the run meta answers a different question that + # cannot be split back apart. `gated` holds (sampleId, cellId) keys and `cells_by_set` maps a + # set to its members, so this is a membership count over cells already read. + per_set_gated = pl.DataFrame( + list(count_by_set(cells_by_set, gated).items()), + orient="row", + schema={"setId": pl.String, "cellsSetAside": pl.Int64}, + ) + # Cells that read nothing at all, PER CLONOTYPE. Carried beside the clonotype's cell count + # rather than at every identity, because a cell with nothing left is empty at every identity + # and repeating the subtraction per position would report a per-identity failure that did not + # happen. It separates a negative resting on cells that read something from one resting on + # cells that read nothing, and changes no verdict. + per_set_empty = pl.DataFrame( + list(count_by_set(cells_by_set, cells_reading_nothing(floored, linker_cells)).items()), + orient="row", + schema={"setId": pl.String, "cellsReadingNothing": pl.Int64}, + ) + counts_frame = ( + set_counts(verdicts) + .join(per_set_cells, on="setId", how="left") + # Filled rather than asserted, unlike cellCount below: with no gate declared `gated` is + # empty, so every set legitimately has nothing set aside and 0 is the true answer. + .join(per_set_gated, on="setId", how="left") + # Filled for the same reason, and it bites harder here: this column ships off by + # default, so the reader who turns it on is the one asking the question, and a null + # would answer it with a blank where zero is the truth. + .join(per_set_empty, on="setId", how="left") + .with_columns(pl.col("cellsSetAside").fill_null(0), pl.col("cellsReadingNothing").fill_null(0)) + ) + # Every set comes FROM the linker, so every set has cells. Asserted rather than filled with + # zero: a set with no cells is a contradiction, and writing 0 would report it as a real, + # empty clonotype. + missing = counts_frame.filter(pl.col("cellCount").is_null())["setId"].to_list() + if missing: + raise SystemExit(f"sets carry verdicts but no cells, which cannot happen: {missing[:8]}") + _write_sorted(_answers(counts_frame), f"{prefix}_set_counts.csv", ["setId"]) + + summary, punch, summary_emitted = _pivot_identity_summary(verdicts, universe) + _write_sorted(_answers(summary), f"{prefix}_identity_summary.csv", ["setId"]) + _write_sorted(_answers(punch), f"{prefix}_identity_punch.csv", ["setId"]) + + cell_punch, cell_punch_emitted = _pivot_cell_punch(states, cells_by_set, offered_by_sample, admissibility, universe) + _write_sorted(_answers(cell_punch), f"{prefix}_cell_punch.csv", ["setId", "sampleId", "cellId"]) + + # The sparse per-tag counts and the per-cell scalars together carry every per-cell state, at + # a small fraction of the dense grid's size. They stay inside the block: reading the same + # experiment under another grouping is another execution rather than a re-derivation a reader + # performs, and the grouping enters after the counting. + # + # That argument used to end "so the dense grid is never exported", and it no longer holds: + # `_pivot_cell_punch` above exports it, because a readout showing one clonotype's cells + # against the panel cannot be assembled from a sparse frame by a grid. What survives is the + # SIZE, which is why that function carries a cell gate and reports whether it emitted. + # + # With no list, membership is unknown rather than false: a barcode nobody classified is not a + # barcode classified as "not a cell". + unlisted_reads = "false" if cell_list is not None else "unknown" + in_list = pl.DataFrame( + [(s, c, "true") for s, c in sorted(listed)], + orient="row", + schema={"sampleId": pl.String, "cellId": pl.String, "inCellList": pl.String}, + ) + + def _admissibility(key: CellKey) -> str: + reason = cell_admissibility_reason(key, admissibility) + return "admissible" if reason is None else reason.value + + # Admissibility is built HERE, in the same row as its own cell, and not attached to a later + # frame as a positional column. Polars does not promise a left frame's row order survives a + # join (`maintain_order` defaults to "none"), so a positional attach after the joins below can + # give cells each other's labels -- and `_write_sorted` then sorts the file, which hides it + # rather than repairing it. + reference_frame = pl.DataFrame( + [(s, c, reference.by_cell.get((s, c)), _admissibility((s, c))) for s, c in analysed_cells], + orient="row", + schema={ + "sampleId": pl.String, + "cellId": pl.String, + "referenceCount": pl.Int64, + "admissibility": pl.String, + }, + ) + cell_counts = ( + non_reference.join(reference_frame, on=["sampleId", "cellId"], how="left") + .join(in_list, on=["sampleId", "cellId"], how="left") + .with_columns(pl.col("inCellList").fill_null(unlisted_reads)) + .select(["sampleId", "cellId", "tag", "umiCount", "referenceCount", "inCellList"]) + ) + _write_sorted(cell_counts, f"{prefix}_cell_counts.csv", ["sampleId", "cellId", "tag"]) + + cell_scalars = ( + reference_frame.join(in_list, on=["sampleId", "cellId"], how="left") + .with_columns(pl.col("inCellList").fill_null(unlisted_reads)) + .select(["sampleId", "cellId", "referenceCount", "admissibility", "inCellList"]) + ) + _write_sorted(_answers(cell_scalars), f"{prefix}_cell_scalars.csv", ["sampleId", "cellId"]) + + # Both frames are pure key sets -- what a sample was offered, and which identity a tag feeds + # -- and each carries a constant value column so it can become a p-column at all. A frame of + # key columns alone imports as nothing: columns are built from value columns, so a key-only + # file yields no column and the fact it records never leaves the block. + offered_frame = pl.DataFrame( + [(sample, identity, "true") for sample in samples for identity in sorted(offered_by_sample[sample])], + orient="row", + schema={"sampleId": pl.String, "identity": pl.String, "offered": pl.String}, + ) + _write_sorted(offered_frame, f"{prefix}_offered.csv", ["sampleId", "identity"]) + + # The value column is named "1" and holds 1, matching the cell-linker convention already used + # for linker columns elsewhere in the platform. + # + # Deliberately NOT keyed by sample. The reason is the join, not the declaration. This linker + # puts a tag-keyed figure beside an identity-keyed verdict, and neither side carries a sample: + # verdicts are (set, identity) over clonotypes that span samples, and the per-tag figures are + # run-level. A sample axis here is an axis no participating table has. It does not sharpen the + # join -- it makes the join malformed, and `createPlDataTableV3` label discovery then rejects + # the spec frame. + # + # Under (tag, sample) grouping one tag can feed several identities, so this frame is + # many-to-many with one row per pair. Distinct rows matter: two tags of one identity would + # otherwise emit the same key twice, and duplicate axis keys break a grid silently. + linker_frame = _linker_frame(grouping) + _write_sorted(linker_frame, f"{prefix}_tag_identity.csv", ["tag", "identity"]) + + # Only disagreements in the column that SUPPLIES the label matter: a tag that disagrees about + # some other property still carries an ordinary name. Which column that is depends on the rule + # -- a property grouping labels by the value it grouped on, while the per-tag grouping borrows + # the feature name. Passing the grouping column either way made every per-tag run look up "", + # so a barcode two samples named differently fell through to its raw 15-mer with the conflict + # shown nowhere a reader would look. + # + # Under a property grouping on ONE column, that column supplies the rescue. Under several there + # is no single such column, so the feature column supplies it: a pair that fell back has no + # combination at all, and what a reader needs then is what the reagent is called. + grouping_columns = ( + _grouping_columns(grouping_rule, property_columns(panel)) + if (isinstance(grouping_rule, dict) and grouping_rule.get("by") == "property") + else [] + ) + label_column = grouping_columns[0] if len(grouping_columns) == 1 else args.feature_col + # Bound once and passed to both readers below. `_identity_labels` joins these names into the + # label a reader sees. The run record carries the same names apart so the readout can say WHY a + # label is joined. Deriving the second from the first -- splitting the label back on " / " -- + # would guess wrong for a reagent whose own name contains a slash. + label_disagreements = disagreed_by_column.get(label_column or "", {}) + labels = _identity_labels( + grouping, + properties, + args.feature_col, + grouping_id, + label_disagreements, + ) + identity_labels = pl.DataFrame( + [(identity, labels.get(identity, identity)) for identity in sorted(universe)], + orient="row", + schema={"identity": pl.String, "label": pl.String}, + ) + _write_sorted(identity_labels, f"{prefix}_identity_labels.csv", ["identity"]) + + # The declarations, keyed the same way the verdicts are. Wide -- one column per property -- + # because the workflow turns each into its own p-column with the property name in the DOMAIN, + # which is what makes two properties two distinct columns rather than one a reader unstacks. + # + # A property no identity agreed on is left out rather than exported empty: an all-blank + # filterable column offers a reader a filter with nothing to filter by. The surviving names are + # recorded in the run meta, because the workflow builds one spec per column and the headers are + # panel data, unknown until this runs. + exportable = [c for c in prop_cols if c != IDENTITY_KEY_COLUMN] + if len(exportable) < len(prop_cols): + print( + f"[emit-verdicts] panel column {IDENTITY_KEY_COLUMN!r} is the key of the identity-property " + "table and cannot also be one of its properties; it is left out of that export and reaches " + "no consumer. Rename it in the panel file to have it travel with the verdicts.", + file=sys.stderr, + ) + identity_properties = _identity_properties(grouping, properties, exportable, grouped_on, disagreed_by_column) + property_values = { + column: sorted({held[column] for held in identity_properties.values() if column in held}) + for column in exportable + } + emitted_properties = [c for c in exportable if property_values[c]] + identity_property_frame = pl.DataFrame( + [ + tuple([identity] + [identity_properties.get(identity, {}).get(c, "") for c in emitted_properties]) + for identity in sorted(universe) + ], + orient="row", + schema={IDENTITY_KEY_COLUMN: pl.String} | {c: pl.String for c in emitted_properties}, + ) + _write_sorted(identity_property_frame, f"{prefix}_identity_properties.csv", ["identity"]) + + declared = _declared_by_sample(panel, samples) + panel_of_sample = {sample: _panel_id(tags) for sample, tags in declared.items()} + tags_of_panel: dict[str, frozenset[str]] = {panel_of_sample[s]: declared[s] for s in samples} + samples_of_panel: dict[str, list[str]] = {} + for sample in samples: + samples_of_panel.setdefault(panel_of_sample[sample], []).append(sample) + + # Named for a reader, so the sample is shown under the label the panel file used rather than + # the sampleId it was translated to. The KEY is the sampleId, because a key has to join. + panel_labels = pl.DataFrame( + [ + ( + panel_id, + f"{len(tags_of_panel[panel_id])} tags: " + + ", ".join(label_of_sample.get(s, s) for s in samples_of_panel[panel_id]), + ) + for panel_id in sorted(tags_of_panel) + ], + orient="row", + schema={"panelId": pl.String, "label": pl.String}, + ) + _write_sorted(panel_labels, f"{prefix}_panel_labels.csv", ["panelId"]) + + sample_panel = pl.DataFrame( + [(sample, panel_of_sample[sample]) for sample in samples], + orient="row", + schema={"sampleId": pl.String, "panelId": pl.String}, + ) + _write_sorted(sample_panel, f"{prefix}_sample_panel.csv", ["sampleId"]) + + # Both directions of the panel-versus-reads check, re-keyed onto the panel: a per-tag failure + # is a property of the declared tag set rather than of any one sample carrying it. The samples + # reporting it travel in the row. + # + # `seen` is drawn from the counts, whose feature barcodes were already snapped onto the panel by + # refine-tags. So `seen` is a subset of the declared set here, and only the declared-never-seen + # direction can produce a row. Reporting an undeclared barcode needs a pre-correction source. + seen = counts.select("sampleId", "tag").unique() + unknown_panel = _panel_id(frozenset()) + mismatch_rows: dict[tuple[str, str, str], set[str]] = {} + for row in panel_read_mismatch(panel, seen).iter_rows(named=True): + # In the unkeyed case every row comes back under "*", which is not a sample id: the + # declaration really is global, so it reports against every sample in the run. + affected = samples if row["sample"] == ANY_SAMPLE else [row["sample"]] + for sample in affected: + key = (panel_of_sample.get(sample, unknown_panel), row["tag"], row["direction"]) + mismatch_rows.setdefault(key, set()).add(sample) + mismatch = pl.DataFrame( + [(panel_id, tag, direction, ", ".join(sorted(s))) for (panel_id, tag, direction), s in mismatch_rows.items()], + orient="row", + schema={"panelId": pl.String, "tag": pl.String, "direction": pl.String, "samples": pl.String}, + ) + _write_sorted(mismatch, f"{prefix}_panel_mismatch.csv", ["panelId", "direction", "tag"]) + + # ---- the quality measurements ------------------------------------------------- + + def _disagreement_rates(samples_here: list[str]) -> dict[str, float | None]: + """The per-tag self-disagreement rate over one panel's samples only. + + Scoped per panel rather than over the run, because the row carrying this figure is keyed + `(tag, panelId)`. A run-global rate on a panel's row says something the panel did not do: a + reagent declared in panels P and Q but misbehaving only in Q's samples shows the same + inflated rate on P's row, pointing a reader at the wrong panel and the wrong remedy. + + Measured at the tag and nowhere else. The identity-level figure has nothing to compare + against, so it cannot separate a faulty reagent from a panel full of weak binders. + + The cell sets are restricted too, not only the states. A set spanning two panels' samples + would otherwise bring its other panel's cells into this panel's evaluable count, and + self-disagreement is precisely a count of a set's cells contradicting each other. + """ + here = set(samples_here) + sets_here = {set_id: [key for key in members if key[0] in here] for set_id, members in cells_by_set.items()} + sets_here = {set_id: members for set_id, members in sets_here.items() if members} + by_tag = self_disagreement( + tag_states.filter(pl.col("sampleId").is_in(samples_here)).select( + "sampleId", "cellId", pl.col("identity").alias("key"), "state" + ), + tag_universe, + {s: tag_offered_by_sample[s] for s in samples_here}, + sets_here, + admissibility, + ) + return dict(zip(by_tag["key"].to_list(), by_tag["disagreementRate"].to_list(), strict=True)) + + read_qc: dict[str, dict] = {} + if args.qc_summary: + for row in pl.read_csv(args.qc_summary, infer_schema_length=0).iter_rows(named=True): + read_qc[str(row.get("sampleId", "")).strip()] = row + + # Why a read-level figure has no number, per source. The summary is one row per sample built + # by `qc_report.py`: it leaves `panelAssignedFraction` and `cellBarcodeValidFraction` blank + # where the refine-tags report is absent or unreadable, carries no step for that tag, or the + # step read no input. `readsTotal` comes from the parse report and is blank only when no + # summary reached this run at all. + NO_READ_QC = "no read QC summary row reached this sample" + NO_REFINE_STEP = "no refine-tags report was produced, or it supplied no %s step with input reads" + NO_READS_TO_DIVIDE = "this sample's read QC reports no reads, so the share has no denominator" + + # The pre-refine pass Task 5 added: one FEATURE tag-stat row per sequence the reads carried, + # before refine-tags snaps each one onto the panel. Without this file the table below stays + # the ordinary empty case rather than raising, because a run wired without it has made no + # claim about undeclared barcodes -- it has not checked, which is a different fact from + # having checked and found none. + raw_feature_counts = _read_raw_feature_counts(args.raw_feature_counts) if args.raw_feature_counts else None + undeclared_barcode_rows: list[dict] = [] + sample_decile_rows: list[dict] = [] + + # `totalWeight` reaches `counts` only from a gather step built after this column existed + # (see `_read_counts`). Checked once, not per sample: its presence is a property of the + # file, not of any one sample's rows within it. + has_total_weight = "totalWeight" in counts.columns + + rows: list[QcRow] = [] + sample_coverage: dict[str, Coverage] = {} + sample_report: dict[str, dict] = {} + for sample in samples: + first = len(rows) + sample_counts = counts.filter(pl.col("sampleId") == sample) + listed_here = [key for key in sorted(listed) if key[0] == sample] if cell_list is not None else None + qc = read_qc.get(sample, {}) + + reads_matched = _number(qc, "readsMatched") + matched_detail = "" if reads_matched is None else f"readsMatched={int(reads_matched)}" + add(rows, "sample", sample, "readsTotal", _number(qc, "readsTotal"), matched_detail, reason=NO_READ_QC) + # `qc_report.py` computes this from the tag-stat TSV directly, the same required input + # `readsTotal` reads from the parse report -- a missing figure means no read-QC row + # reached this sample at all, same cause and reason as that row. + add(rows, "sample", sample, "cellsDetected", _number(qc, "cellsDetected"), reason=NO_READ_QC) + add( + rows, + "sample", + sample, + "panelAssignedFraction", + _number(qc, "panelAssignedFraction"), + reason=NO_READ_QC if not qc else NO_REFINE_STEP % "FEATURE", + ) + + # `usable_read_fraction` takes the cell IDs alone: `sample_counts` is already scoped to + # this sample, so the sampleId half of each `listed_here` key would only be checked + # against itself. + cell_ids_here = [key[1] for key in listed_here] if listed_here is not None else None + reads_total = _number(qc, "readsTotal") + if has_total_weight: + usable_value, usable_detail = usable_read_fraction( + sample_counts, "cellId", cell_ids_here, int(reads_total) if reads_total is not None else None + ) + else: + usable_value, usable_detail = None, "the counts file carries no totalWeight column" + # `usable_read_fraction` returns one string for both roles, and they are not the same role: + # a detail rides alongside a number, a reason stands in place of one. Passing it as the + # detail only when a number came back keeps QcRow's invariant. + add( + rows, + "sample", + sample, + "usableReadFraction", + usable_value, + "" if usable_value is None else usable_detail, + reason=usable_detail, + ) + + # 330's undeclared-barcode table: keyed by sequence, never by (panel, tag), because an + # undeclared barcode has no row in the panel to sit beside. Read on the PRE-refine pass, + # where a sequence the panel never declared can still be seen -- `counts` above has + # already been snapped onto the panel by refine-tags, which is why that path cannot + # produce this direction (see result_panel_mismatch.csv's "undeclared-in-panel" rows). + if raw_feature_counts is not None: + sample_raw = raw_feature_counts.filter(pl.col("sampleId") == sample).select("FEATURE", "totalWeight") + undeclared, undeclared_share = undeclared_feature_counts(sample_raw, declared[sample]) + # The share is the SAMPLE's, computed once and carried on every one of that sample's + # rows. 310: the status is the barcode's and never the sample's, so it is written + # here rather than added to `rows` / `sample_report_rows`. + undeclared_status = status_for("undeclaredBarcodeShare", undeclared_share, lines) + for undeclared_row in undeclared.iter_rows(named=True): + undeclared_barcode_rows.append( + { + "sampleId": sample, + "tag": undeclared_row["tag"], + "totalWeight": int(undeclared_row["totalWeight"]), + "readShare": undeclared_share, + "status": None if undeclared_status is None else undeclared_status.value, + } + ) + add( + rows, + "sample", + sample, + "cellBarcodeValidFraction", + _number(qc, "cellBarcodeValidFraction"), + reason=NO_READ_QC if not qc else NO_REFINE_STEP % "CELL", + ) + # The denominator is the cell list, never the barcodes the reads happened to touch: the + # five-thousand recommendation is per called cell, and in droplet data observed barcodes + # run one to two orders of magnitude higher, so dividing by them would alert on a healthy + # run. No cell list means no denominator, so depth is *not evaluated*. + depth = ( + reads_per_cell(int(reads_matched), len(listed_here)) + if reads_matched is not None and listed_here is not None + else None + ) + detail = f"cellsInList={len(listed_here)}" if listed_here is not None else "no cell list supplied" + # Three cases, not two. `reads_per_cell` returns no number for an EMPTY cell list as + # well as for an absent one, and a sample with no listed cell is the zero-cells finding + # rather than a missing read count -- which is the input a reader would otherwise go fix. + depth_reason = ( + "no cell list supplied, so depth has no denominator" + if listed_here is None + else "no cell of this sample is in the cell list, so depth has no denominator" + if not listed_here + else "no read count reached this sample, so depth has no numerator" + ) + add(rows, "sample", sample, "readsPerCell", depth, detail, reason=depth_reason) + + deciles = antigen_count_deciles(sample_counts) + sample_decile_rows += _sample_decile_rows(sample, deciles) + decile_detail = "|".join( + f"{d}:{'' if v is None else round(v, 3)}" for d, v in zip(deciles["decile"], deciles["value"], strict=True) + ) + middle = deciles.filter(pl.col("decile") == 50)["value"].to_list() + # An empty input still returns all eleven decile points, each unanswered, so a value of + # None here means this sample holds no counted reading at all. + add( + rows, + "sample", + sample, + "antigenCountDistribution", + middle[0] if middle else None, + decile_detail, + reason="no barcode in this sample holds a counted reading", + ) + # qc_report.py computes this from the tag-stat TSV and the parse report. It blanks the + # figure on two distinct conditions: no read-QC row for this sample at all, and a row + # whose readsTotal is zero, which leaves the fraction no denominator. The second is + # reachable through the empty-input path in parse_gate.py, where readsTotal is present + # and zero rather than absent. + agg_fraction = _number(qc, "aggregateBarcodeFraction") + agg_flagged = _number(qc, "aggregateBarcodesFlagged") + agg_threshold = _number(qc, "aggregateBarcodeThreshold") + agg_detail = "" if agg_fraction is None else f"barcodesFlagged={int(agg_flagged or 0)}" + if agg_threshold is not None: + agg_detail += f"|threshold={agg_threshold:.1f}" + add( + rows, + "sample", + sample, + "aggregateBarcodeFraction", + agg_fraction, + agg_detail, + reason=NO_READ_QC if not qc else NO_READS_TO_DIVIDE, + ) + + stats = floor_stats.get(sample, {"readingsFloored": 0, "cellsEmptied": 0}) + add( + rows, + "sample", + sample, + "floorRemoved", + float(stats["readingsFloored"]), + f"cellsEmptied={stats['cellsEmptied']}", + ) + + listed_totals = ( + sample_counts.join(in_list, on=["sampleId", "cellId"], how="semi") + .group_by("cellId") + .agg(pl.col("umiCount").sum().alias("total"))["total"] + .to_list() + ) + add( + rows, + "sample", + sample, + "uniqueCountsPerCell", + _median([float(v) for v in listed_totals]), + f"cellsWithAReading={len(listed_totals)}", + # `in_list` is empty whenever no list arrived, so the join yields nothing for every + # sample of such a run. Branching on the same fact `readsPerCell` branches on keeps + # the two rows from giving one run two incompatible accounts. + reason=( + "no cell list supplied, so no cell of this sample is listed" + if cell_list is None + else "no listed cell in this sample holds a counted reading" + ), + ) + + # Two forms, and the gate decides which. With a gate declared this counts the cells it + # set aside. With none there is no *high* to count, so the measurement is the spread of + # the readings themselves -- which `290-reference-two-roles` names as what a scientist + # reads in order to declare a gate. + # + # A gate makes the value a count, which is 0.0 over no readings rather than absent. The + # spread has no median over none, so only the gateless form can reach the report with no + # number, and it does so exactly when no cell of this sample carries a comparator. + here = {key: value for key, value in reference.by_cell.items() if key[0] == sample} + high_value, high_detail = _sticky_measure(here, args.gate_threshold) + add( + rows, + "sample", + sample, + "highReferenceCells", + high_value, + high_detail, + reason="no cell in this sample carries a comparator reading", + ) + + # A measurement declaring `rolls_up=False` states a reagent's condition on a sample's + # row, and `310` keeps a reagent's failure off every sample: one bad reagent marking + # twenty samples is how a sample status becomes noise. Its own row keeps its status. + # + # The report and the rollup come out of one call, so the tag a reader sees beside the + # list cannot disagree with the list. + report, coverage = sample_report_rows(sample, rows[first:]) + sample_report[sample] = { + "status": None if coverage.status is None else coverage.status.value, + "judged": coverage.judged, + "unjudged": coverage.unjudged, + "notEvaluated": coverage.not_evaluated, + "measurements": report, + } + sample_coverage[sample] = coverage + + per_sample_tag_total = { + (row["sampleId"], row["tag"]): row["total"] + for row in counts.group_by(["sampleId", "tag"]) + .agg(pl.col("umiCount").sum().alias("total")) + .iter_rows(named=True) + } + + reagent_rows: list[dict] = [] + for panel_id in sorted(tags_of_panel): + panel_samples_here = samples_of_panel[panel_id] + panel_tags = tags_of_panel[panel_id] + tag_rate = _disagreement_rates(panel_samples_here) + here_total = { + tag: float(sum(per_sample_tag_total.get((s, tag), 0) for s in panel_samples_here)) + for tag in {t for (s, t) in per_sample_tag_total if s in panel_samples_here} | set(panel_tags) + } + observed_here = {tag for tag, total in here_total.items() if total > 0} + + # A declared tag is alerting at zero reads, so every declared tag gets a row rather than + # only the ones that produced nothing: reporting only the failures leaves a reader unable + # to tell a clean panel from an unchecked one. + for tag in sorted(panel_tags): + add(rows, "tag", tag, "declaredNeverSeen", here_total[tag], "", panel_id) + for tag in sorted(observed_here - panel_tags): + add(rows, "tag", tag, "undeclaredBarcodes", here_total[tag], "", panel_id) + + # The fitted background, one row per declared tag. Fits are per (sample, tag) and this + # table is keyed (tag, panel), so the value is the MEDIAN background mean over the + # panel's samples that fitted, and the detail carries how many did and the spread. A + # mean of means would let one sample's outlier move a tag's whole row. + # + # Under a declared baseline nothing is fitted, so every row carries no value and says + # why -- the same device `readsPerCell` uses when no cell list arrived. The row is there + # either way, or a reader cannot tell "not fitted" from "never measured". + for tag in sorted(panel_tags): + add( + rows, + "tag", + tag, + "fittedBackground", + *_fitted_background(tag_fits, panel_samples_here, tag), + panel_id, + ) + + panel_states = tag_states.filter(pl.col("sampleId").is_in(panel_samples_here)).rename({"identity": "tag"}) + # RAW counts, not `floored`. Cells-with-count and the median are what the reagent + # delivered, and the minimum is what survived it. Passing the floored frame here is the + # defect 330-the-quality-readout names: a reagent putting two counts into every cell + # would read the same as one that delivered nothing. + panel_counts = counts.filter(pl.col("sampleId").is_in(panel_samples_here)) + for row in per_antigen_measures( + panel_counts, panel_states, panel_tags, panel_samples_here, reference_tags + ).iter_rows(named=True): + above = row["cellsAboveTheLine"] + # None only for a reference tag, which is held out of the verdict read. Say so rather + # than printing a zero: no cell was called bound because none was asked. + detail = ( + f"cellsWithCount={row['cellsWithCount']}" + f"|medianCountPerCell={row['medianCountPerCell']}" + f"|samplesSeenIn={row['samplesSeenIn']}/{row['samplesInPanel']}" + ) + if above is None: + detail += "|cellsAboveTheLine=none asked, this tag supplies the baseline" + add( + rows, + "tag", + row["tag"], + "perAntigen", + float(above) if above is not None else None, + detail, + panel_id, + ) + + # No line stands behind this, so it reads unjudged and its value travels beside its + # siblings for a reader to compare. A tag standing clear of the other tags in its panel is + # misbehaving whatever the absolute rate -- a real finding, but one a reader makes by + # looking. Applying a threshold would need a multiplier nobody published. Keeping the rows + # per panel is what makes the comparison the right one. + for tag in sorted(panel_tags & set(tag_rate)): + add(rows, "tag", tag, "tagDisagreement", tag_rate[tag], "", panel_id) + + # The identity -> tags map comes from `grouping`, the one place that settles which tags + # an identity carries. Scoped to this panel's samples and declarations. + siblings_of_identity: dict[str, list[str]] = {} + identity_of_tag: dict[str, str] = {} + # A barcode reused for a different antigen in different samples carries two identities and + # takes a row under each. `identity_of_tag` keeps only the last and is not usable here. + identities_of_tag: dict[str, list[str]] = {} + # The samples where a tag carried one identity. Two identities of one tag hold disjoint + # sample sets, because `grouping` gives each (tag, sample) exactly one identity. + samples_of_pair: dict[tuple[str, str], set[str]] = {} + for (tag, sample), identity in grouping.items(): + if tag not in panel_tags or (sample not in set(panel_samples_here) and sample != ANY_SAMPLE): + continue + members = siblings_of_identity.setdefault(identity, []) + if tag not in members: + members.append(tag) + identity_of_tag[tag] = identity + carried = identities_of_tag.setdefault(tag, []) + if identity not in carried: + carried.append(identity) + # ANY_SAMPLE declares the identity for every sample of the panel. + samples_of_pair.setdefault((tag, identity), set()).update( + panel_samples_here if sample == ANY_SAMPLE else [sample] + ) + sibling_rate = sibling_disagreement(panel_states, siblings_of_identity) + # A tag with no row in the panel's states held no cell here. `sibling_disagreement` + # returns the same absent rate for that as for siblings that never reached a majority, + # and the two are opposite findings: one is this reagent missing, the other is the + # siblings unable to judge it. + held_a_cell = set(panel_states["tag"].unique().to_list()) + + # No line stands behind this either, so it reads unjudged beside its siblings. A blank + # and a zero are opposite findings here, so a row with no rate says which case it is. + for tag in sorted(panel_tags & set(sibling_rate)): + rate = sibling_rate[tag] + detail = "" + if rate is None: + if len(siblings_of_identity[identity_of_tag[tag]]) < 2: + detail = "this identity carries one tag, so it has no sibling" + elif tag not in held_a_cell: + detail = "this tag holds no cell beside a sibling" + else: + detail = "no cell gave this tag's siblings a majority" + add(rows, "tag", tag, "siblingDisagreement", rate, detail, panel_id) + + # One row per (tag, identity), with every figure scoped to the samples where the tag + # carried that identity. A tag absent from `grouping` here takes one row under its own + # barcode over the whole panel, which is what the barcode grouping names it. + reference_here = set(reference_tags) + pairs_of_subset: dict[frozenset[str], list[tuple[str, str]]] = {} + for tag in sorted(panel_tags): + for identity in sorted(identities_of_tag.get(tag, [tag])): + scope = samples_of_pair.get((tag, identity), set()) & set(panel_samples_here) + pairs_of_subset.setdefault(frozenset(scope or panel_samples_here), []).append((tag, identity)) + + for scope, pairs in sorted(pairs_of_subset.items(), key=lambda kv: sorted(kv[0])): + scope_samples = sorted(scope) + scope_tags = {tag for tag, _ in pairs} + scope_states = tag_states.filter(pl.col("sampleId").is_in(scope_samples)).rename({"identity": "tag"}) + scope_measure = { + row["tag"]: row + for row in per_antigen_measures( + counts.filter(pl.col("sampleId").is_in(scope_samples)), + scope_states, + scope_tags, + scope_samples, + reference_tags, + ).iter_rows(named=True) + } + # A tag's siblings are the other tags of the identity on the row, and both rates are + # taken over the row's samples only. Within one subset a tag appears under one identity, + # so the tag-keyed rates these return are unambiguous. + scope_siblings = { + identity: siblings_of_identity[identity] for _, identity in pairs if identity in siblings_of_identity + } + scope_sibling_rate = sibling_disagreement(scope_states, scope_siblings) + scope_tag_rate = _disagreement_rates(scope_samples) + scope_held_a_cell = set(scope_states["tag"].unique().to_list()) + + for tag, identity in sorted(pairs): + measure = scope_measure.get(tag, {}) + above = measure.get("cellsAboveTheLine") + median = measure.get("medianCountPerCell") + sibling = scope_sibling_rate.get(tag) + own = scope_tag_rate.get(tag) + # Scoped to the row's own identity. `identity_of_tag` keeps one identity per tag and + # would give a reused barcode's two rows the same reason. + members = siblings_of_identity.get(identity, []) + absences = [] + if above is None: + absences.append("cellsAboveTheLine=none asked, this tag supplies the baseline") + if median is None: + absences.append("medianCountPerCell=no cell holds a count of this tag") + if sibling is None: + if tag in reference_here: + absences.append("siblingDisagreement=this tag is held out of the verdict read") + elif len(members) < 2: + absences.append("siblingDisagreement=this identity carries one tag, so it has no sibling") + elif tag not in scope_held_a_cell: + absences.append("siblingDisagreement=this tag holds no cell beside a sibling") + else: + absences.append("siblingDisagreement=no cell gave this tag's siblings a majority") + if own is None: + absences.append("selfDisagreement=no cell set held this tag under an evaluable read") + # Named beside the counts above rather than replacing them: samplesSeenIn and + # samplesInPanel are released p-columns and keep their id. Sample ids are mapped + # through `label_of_sample`, the same map that resolves the panel file's own + # names, so a raw sampleId does not reach this column where a label exists. + reagent_rows.append( + { + "panelId": panel_id, + "tag": tag, + "identity": identity, + "samplesSeenIn": int(measure.get("samplesSeenIn") or 0), + "samplesInPanel": int(measure.get("samplesInPanel") or 0), + "samplesSeenInNames": ", ".join( + label_of_sample.get(s, s) for s in measure.get("samplesSeenInNames") or [] + ), + "samplesInPanelNames": ", ".join( + label_of_sample.get(s, s) for s in measure.get("samplesInPanelNames") or [] + ), + "cellsWithCount": int(measure.get("cellsWithCount") or 0), + "cellsAboveTheLine": float(above) if above is not None else None, + "medianCountPerCell": float(median) if median is not None else None, + "siblingDisagreement": float(sibling) if sibling is not None else None, + "selfDisagreement": float(own) if own is not None else None, + "reason": "|".join(absences), + } + ) + + # One row for the whole run, and the entity is the run: 320 puts the score spread at that + # grain because the cutoff is one number for the run, so a per-sample figure would answer a + # question nobody asked. Emitted outside the sample loop, which is also what keeps it out of + # every sample's rollup. + # + # The score is re-derived from the counts `read_states` returns rather than carried out of + # it. Same function and same inputs, so the two cannot drift, and `read_states` keeps its + # refusal to emit a binding level per cell. + score_value, score_detail = _score_spread(states, reference.served) + add(rows, "run", "run", "scoreDistribution", score_value, score_detail) + + # Only the sample carries an aggregated status, over its OWN per-sample measurements. A + # per-tag failure is usually a property of the reagent across the whole run, so feeding a dead + # reagent in a panel of twenty tags into a sample status would mark every sample alerting and + # make that status noise. It does not hide: the per-tag row states the reagent finding on its + # own, keyed by the panel that has it. + for sample in samples: + coverage = sample_coverage[sample] + rows.append(QcRow("sample", sample, ROLLUP, None, "", "", coverage.status, coverage)) + + _write_sorted(_qc_frame(rows, lines), f"{prefix}_qc.csv", ["level", "entity", "panelId", "measurement"]) + + # The same sample-level measurements as the frame above, keyed by sample. Read as content and + # not as a table: the sample detail view holds one sample at a time and resolves it + # synchronously. The frame stays the artefact every other reader takes. + with open(f"{prefix}_qc_by_sample.json", "w") as out: + json.dump(sample_report, out, indent=2, sort_keys=True) + + # The across-samples table `330-the-quality-readout` asks for: one row per sample, one column + # per sample-level measurement, carrying the sample's own rolled-up status. Pivoted from + # `sample_report` rather than walked a second time, so it cannot disagree with the sample's + # own report above. + _write_sorted(sample_summary_rows(samples, sample_report, read_qc), f"{prefix}_qc_summary.csv", ["sampleId"]) + + # The three distributions `330-the-quality-readout` puts last, as plottable frames rather than + # as detail strings on a measurement row. A reader settles the cutoff and the gate by looking at + # these, so they have to be drawable: a decile encoded inside a detail string is a number nobody + # can plot. + # + # Deciles of the score and of the reference reading share one frame, keyed by which distribution + # a row belongs to. Both are taken over the whole run: the cutoff is one number for the run, and + # so is the gate, so a plot must show every cell the number will act on. `330` says exactly that + # of the reference reading, and pooling over samples is deliberate rather than a simplification. + decile_rows: list[dict] = [] + if reference.served is ReferenceChoice.DECLARED: + scored = states.filter(pl.col("unreliableReason").is_null()) + if scored.height > 0: + values = specificity_score( + scored["umiCount"].to_numpy(), + np.nan_to_num(scored["referenceCount"].cast(pl.Float64).to_numpy(), nan=0.0), + ) + decile_rows += _decile_rows("score", deciles_of(np.asarray(values, dtype=float))) + if reference.by_cell: + readings = np.asarray(list(reference.by_cell.values()), dtype=float) + decile_rows += _decile_rows("referenceReading", deciles_of(readings)) + _write_sorted( + pl.DataFrame(decile_rows, schema=_DECILE_SCHEMA), + f"{prefix}_qc_deciles.csv", + ["distribution", "decile"], + ) + + # The same shape, kept PER SAMPLE: 330 asks for the antigen-count distribution as one sample's + # own plot, not pooled with any other sample's. A separate frame and a separate column, since + # adding a sample axis to `_DECILE_SCHEMA` above would change that column's identity. + _write_sorted( + pl.DataFrame(sample_decile_rows, schema=_SAMPLE_DECILE_SCHEMA), + f"{prefix}_qc_sample_deciles.csv", + ["sampleId", "decile"], + ) + + # One row per (sample, tag) the fit scored, at the fit's own grain. Aggregating to the tag would + # hide a reagent that separated in one sample and not in another, which is the comparison a + # reader makes here. + background_rows = [ + { + "sampleId": sample, + "tag": tag, + "backgroundMean": b.mean, + "signalMean": b.signal_mean, + "backgroundWeight": b.weight, + } + for (sample, tag), b in sorted((tag_fits.backgrounds if tag_fits is not None else {}).items()) + ] + _write_sorted( + pl.DataFrame(background_rows, schema=_BACKGROUND_SCHEMA), + f"{prefix}_qc_backgrounds.csv", + ["sampleId", "tag"], + ) + + _write_sorted( + pl.DataFrame(reagent_rows, schema=_REAGENT_SCHEMA), + f"{prefix}_reagents.csv", + ["panelId", "identity", "tag"], + ) + + _write_sorted( + pl.DataFrame(undeclared_barcode_rows, schema=_UNDECLARED_BARCODE_SCHEMA), + f"{prefix}_undeclared_barcodes.csv", + ["sampleId", "tag"], + ) + + meta = { + "referenceChoice": reference.served.value, + "referenceSourceRequested": source.value, + # Whether a baseline was established, and where not, why. Only the tag-distribution rung + # can reach false: its conditions are properties of the data, so a run resting on it + # proceeds and reports afterwards. The other rungs refuse from the settings. + # + # The model reads this and draws no punchcard where it is false, showing the reason in its + # place. The answer frames are header-only in that case, so a consumer that reads them + # anyway finds no rows rather than a full grid of non-answers. + "baselineEstablished": no_baseline_reason is None, + "noBaselineReason": no_baseline_reason, + "cellListSource": cell_list_source, + "cellsInList": len(cell_list) if cell_list is not None else None, + "cellsAnalysed": len(analysed_cells), + "floor": args.floor, + "cutoff": args.cutoff, + "minVoters": args.min_voters, + "minAgreement": args.min_agreement, + "gateThreshold": args.gate_threshold, + "panelMinMembers": args.panel_min_members, + "distributionMinCells": args.distribution_min_cells, + # Per (sample, tag), and only where that rung was asked for: which tags could not be + # fitted, and why. A tag missing here fitted. The reader needs both halves to tell a panel + # that mostly worked from one that mostly did not. + "distributionUnfitted": ( + {f"{sample}/{tag}": reason for (sample, tag), reason in sorted(tag_fits.reasons.items())} + if tag_fits is not None + else {} + ), + "roleColumn": args.role_column, + "referenceValues": sorted(reference_values), + "referenceTags": sorted(reference_tags), + "grouping": grouping_rule or {"by": "tag"}, + "groupingId": grouping_id, + # The narrowing a short panel file costs, carried in the output rather than only in a log + # line: these tags were answered under a grouping that could not place them. + "tagsWithoutGroupingValue": sorted(ungrouped_tags), + "contending": [sorted(group) for group in contending], + "identityCount": len(universe), + # The identities themselves, in the order the pivot lays them out. The workflow builds one + # p-column per column of result_identity_summary.csv, and the column names are the + # identities -- panel data, unknown until this runs. A count cannot name them, so without + # this the pivoted summary imports as nothing and the only per-antigen state a + # clonotype-anchored reader can see disappears with no error. + "identities": sorted(universe), + # Read by the workflow to label the punchcard's columns. An identity whose grouping value + # was dropped is labelled with the names it did declare, so the card shows a reagent rather + # than a 15-mer. Every other identity labels itself. + "identityLabels": {identity: labels.get(identity, identity) for identity in sorted(universe)}, + # Why a label above is two names joined. Keyed exactly as `_identity_labels` keys its own + # lookup, so an entry appears for precisely the identities whose label was joined. Only + # genuine conflicts: one declared name is the ordinary case. The workflow turns each entry + # into the column's description annotation, shown as a header tooltip -- otherwise a reader + # meets two antigen names in one header with nothing saying whether the barcode was shared, + # the panel was inconsistent, or the block merged something. + "identityNameConflicts": { + identity: sorted(names) + for identity in sorted(universe) + if len(names := label_disagreements.get(identity, [])) > 1 + }, + # The declaration columns that reached result_identity_properties.csv, and the distinct + # values each carries. Both are panel data: the workflow builds one p-column per name and + # annotates it with its own value set, so without these the declarations import as nothing. + "identityProperties": emitted_properties, + "identityPropertyValues": {c: property_values[c] for c in emitted_properties}, + "identitySummaryEmitted": summary_emitted, + # False where the run was too wide or too deep for the dense per-cell grid, so the readout + # can say which of the two it was rather than showing an empty tab. + "cellPunchEmitted": cell_punch_emitted, + "cellPunchCells": len(cell_punch), + "identitySummaryLimit": IDENTITY_SUMMARY_MAX_IDENTITIES, + "readingsFloored": readings_floored, + "cellsEmptied": cells_emptied, + "cellsHighReference": cells_high_reference, + "cellsSetAside": len(gated), + # The same tally per clonotype, for the expansion, and present only when a gate was + # declared: the UI's whole condition is an absent key. Sparse -- a clonotype that lost + # nothing carries no entry, and an absent key reads as zero -- because this file is parsed + # on every render. + **( + {"cellsSetAsideBySet": {k: v for k, v in count_by_set(cells_by_set, gated).items() if v > 0}} + if args.gate_threshold + else {} + ), + "panelLinesDropped": dropped_lines, + "samples": samples, + "setCount": len(cells_by_set), + # How many DISTINCT panels the run carried. One means every sample was stained with the + # same tags, and then how many of a clonotype's cells could answer is the same at every + # identity -- its own cell count, which the grid already shows beside its name. The readout + # uses this to decide whether the per-identity figure says anything. + "samplePanelCount": len(set(panel_of_sample.values())), + } + with open(f"{prefix}_run_meta.json", "w") as out: + json.dump(meta, out, indent=2, sort_keys=True) + + +if __name__ == "__main__": + main() diff --git a/software/per-cell-metrics/src/frame_io.py b/software/per-cell-metrics/src/frame_io.py new file mode 100644 index 0000000..3e765f8 --- /dev/null +++ b/software/per-cell-metrics/src/frame_io.py @@ -0,0 +1,125 @@ +"""Reading the run's input files, and writing a frame out in a fixed row order. + +Every reader here reads as strings and strips, because these columns are join keys against +each other. A tag written " AAAA " on one side and "AAAA" on the other joins to nothing. +""" + +from __future__ import annotations + +import json +from collections.abc import Collection + +import polars as pl + + +def _write_sorted(frame: pl.DataFrame, path: str, by: list[str]) -> None: + """Write a frame in a fixed row order, header-only when it has no rows. + + Every frame reaching here is built with an explicit schema, so an empty one still + carries its columns and writes a header. A consumer meeting a header-only frame knows + the step ran and found nothing. One meeting an empty file cannot tell that from a step + that never ran. + """ + frame.sort(by).write_csv(path) + + +def _read_columns(path: str, columns: tuple[str, ...], what: str) -> pl.DataFrame: + """Read a CSV as strings, keeping the named columns and stripping them. + + Stripped because these columns are join keys against the panel, whose reader strips + `tag` and `sample` for the same reason. A tag written " AAAA " on one side and "AAAA" + on the other joins to nothing, and reports the barcode as both undeclared and never + seen. + """ + frame = pl.read_csv(path, infer_schema_length=0) + missing = [c for c in columns if c not in frame.columns] + if missing: + raise SystemExit(f"{what} {path!r} has no column(s) {missing}; columns are {frame.columns}") + return frame.select([pl.col(c).str.strip_chars().fill_null("") for c in columns]) + + +def _read_counts(path: str) -> pl.DataFrame: + """The counts frame, with umiCount as an integer, or a curated exit naming the bad value. + + `_read_columns` reads every column as a string and fills nulls with "", so a blank cell + and a decimal both survive to the cast. A bare `.cast` dies there as a raw polars + traceback naming neither the file nor the column, the one thing a reader needs. + + `totalWeight` -- the post-refine tag-stat's read-weight column, gathered by + gather-counts.tpl.tengo alongside the distinct-UMI count -- is read when the file carries + it and left off the returned frame otherwise. Its absence means the run predates this + column, not a bad file: `usable_read_fraction`'s caller checks for the column rather than + crashing on it. + """ + counts = _read_columns(path, ("sampleId", "cellId", "tag", "umiCount"), "counts file") + umi = counts["umiCount"].cast(pl.Int64, strict=False) + offenders = [raw for raw, cast in zip(counts["umiCount"], umi, strict=True) if cast is None] + if offenders: + shown = ", ".join(repr(v) for v in offenders[:5]) + raise SystemExit( + f"counts file {path!r} has {len(offenders)} umiCount value(s) that are not whole numbers: " + f"{shown}. A UMI count is a count of observations; a blank or a decimal is not one." + ) + counts = counts.with_columns(umi.alias("umiCount")) + header = pl.read_csv(path, infer_schema_length=0, n_rows=0).columns + if "totalWeight" in header: + weight_raw = _read_columns(path, ("totalWeight",), "counts file")["totalWeight"] + weight = weight_raw.cast(pl.Int64, strict=False) + weight_offenders = [raw for raw, cast in zip(weight_raw, weight, strict=True) if cast is None] + if weight_offenders: + shown = ", ".join(repr(v) for v in weight_offenders[:5]) + raise SystemExit( + f"counts file {path!r} has {len(weight_offenders)} totalWeight value(s) that are not " + f"whole numbers: {shown}" + ) + counts = counts.with_columns(weight.alias("totalWeight")) + return counts + + +def undeclared_feature_counts(raw_counts: pl.DataFrame, declared: Collection[str]) -> tuple[pl.DataFrame, float | None]: + """Undeclared FEATURE barcodes in a pre-refine tag-stat table, and their read share. + + `raw_counts`: mitool `tag-stat -t FEATURE` (no `-u`) output, columns `FEATURE` and + `totalWeight`, one row per distinct observed sequence. `declared`: one sample's panel + tag set. + + Returns the undeclared rows, renamed to `tag` and sorted by it, and the share of + every row's `totalWeight` they carry. Share is `None` over zero total weight. With + no undeclared row the frame is empty and the share is `0.0`. + """ + ordered = raw_counts.rename({"FEATURE": "tag"}).select("tag", "totalWeight").sort("tag") + undeclared = ordered.filter(~pl.col("tag").is_in(set(declared))) + total_weight = float(ordered["totalWeight"].sum()) if ordered.height else 0.0 + if total_weight <= 0: + return undeclared, None + undeclared_weight = float(undeclared["totalWeight"].sum()) if undeclared.height else 0.0 + return undeclared, undeclared_weight / total_weight + + +def _read_raw_feature_counts(path: str) -> pl.DataFrame: + """The gathered pre-refine FEATURE tag-stat table, across every sample. + + Columns `sampleId`, `FEATURE`, `totalWeight` -- the workflow's per-sample gather step + injects `sampleId` from the resource-map key the same way `_read_counts` documents for the + (cell, tag) counts. Read as strings and stripped, same join-safety reason as + `_read_columns`, then `totalWeight` cast to a whole number. + """ + frame = _read_columns(path, ("sampleId", "FEATURE", "totalWeight"), "raw feature counts file") + weight = frame["totalWeight"].cast(pl.Int64, strict=False) + offenders = [raw for raw, cast in zip(frame["totalWeight"], weight, strict=True) if cast is None] + if offenders: + shown = ", ".join(repr(v) for v in offenders[:5]) + raise SystemExit( + f"raw feature counts file {path!r} has {len(offenders)} totalWeight value(s) that are not " + f"whole numbers: {shown}" + ) + return frame.with_columns(weight.alias("totalWeight")) + + +def _json_arg(raw: str | None, flag: str): + if raw is None or not raw.strip(): + return None + try: + return json.loads(raw) + except json.JSONDecodeError as exc: + raise SystemExit(f"{flag} is not valid JSON: {exc}") from exc diff --git a/software/per-cell-metrics/src/identity_tables.py b/software/per-cell-metrics/src/identity_tables.py new file mode 100644 index 0000000..415279e --- /dev/null +++ b/software/per-cell-metrics/src/identity_tables.py @@ -0,0 +1,563 @@ +"""The identity, panel and per-cell tables, and the grouping rule they are all keyed by. + +The grouping decides what an identity IS, so everything here is downstream of it: the +tag-to-identity map, the identity labels and properties, the panel ids, and the two wide +pivots. A caller that builds one of these against a different grouping than another gets +two tables that cannot be joined. +""" + +from __future__ import annotations + +import hashlib +import sys +from collections import Counter + +import polars as pl +from panel import ( + ANY_SAMPLE, + Grouping, + default_grouping, + property_columns, +) +from verdict import ( + Admissibility, + State, + UnreliableReason, + cell_admissibility_reason, +) + +CellKey = tuple[str, str] +# One column per identity, so this bounds the pivot's WIDTH and not its length. A pMHC panel +# carries more than a thousand identities and is out of this version's scope. +IDENTITY_SUMMARY_MAX_IDENTITIES = 10_000 +# What joins several grouping columns into one identity key. A scientist may group on more +# than one column, and the identity is the distinct combination of their values, so one +# string has to carry them all. A panel value containing this separator would let two +# different combinations produce one key. That is reported rather than silently merged, and +# the run continues, because refusing the panel would reject a file that reads correctly +# under every other grouping. +GROUPING_KEY_SEPARATOR = " | " + + +def _grouping_columns(rule: dict, declared: list[str]) -> list[str]: + """The columns a property rule names, as a list. + + Accepts `columns: [...]`, and the older `column: "..."` because a project stored before + the rule took a list carries that shape. Reading both costs one function. The + alternative is a data migration over every stored project. + """ + raw = rule.get("columns") + if raw is None: + single = rule.get("column") or "" + raw = [single] if single else [] + if not isinstance(raw, list) or not all(isinstance(c, str) for c in raw): + raise SystemExit(f"--grouping columns must be a list of strings; got {raw!r}") + named = [c for c in raw if c] + if not named: + raise SystemExit("--grouping names no column; give one or more, or use {'by':'tag'}") + missing = [c for c in named if c not in declared] + if missing: + raise SystemExit(f"--grouping names {missing}, which the panel does not declare: {declared}") + return named + + +def _build_grouping( + rule: dict | None, + panel: pl.DataFrame, + properties: dict[str, dict[str, str]], + reference_tags: set[str], +) -> tuple[Grouping, str, list[str]]: + """The tag -> identity map the run reads at, and the id of the rule behind it. + + A property grouping is built from `consistent_properties`, never from the panel column. + The panel reader strips `tag` and `sample` and carries property values through exactly + as written, so reading the column directly makes " Spike " and "Spike" two identities + that no clean fixture would reveal. + + Reference tags are excluded here rather than by `identity_universe`, which takes no + reference tags and never will -- one place decides, so the two cannot drift. Leaving + them in would give the comparator an identity of its own, read by comparing it against + itself. + + A tag the grouping column says nothing about keeps its own identity rather than + vanishing. Dropping it would remove a declared reagent from the answer with nothing + downstream able to tell the panel was short. + """ + by_tag = default_grouping(panel, reference_tags) + # Type-checked before it is read as one. `--grouping '"tag"'` is valid JSON and not a + # mapping, and reaching `.get` on it raises an AttributeError instead of the usage + # message written two lines below for exactly this mistake. + if rule is not None and not isinstance(rule, dict): + raise SystemExit( + f"--grouping must be a JSON object, {{'by':'tag'}} or {{'by':'property','columns':[...]}}; got {rule!r}" + ) + if rule is None or rule.get("by") == "tag": + # The per-tag grouping groups on no column, so it declares nothing of its identities. + return by_tag, "per-tag", [], {} + if rule.get("by") != "property": + raise SystemExit(f"--grouping must be {{'by':'tag'}} or {{'by':'property','columns':[...]}}; got {rule!r}") + + columns = _grouping_columns(rule, property_columns(panel)) + + # Read PER PANEL ROW, never through `consistent_properties`. The panel declares per tag + # and sample, so a value differing between a tag's rows is a declaration -- this barcode + # carries that antigen in that sample -- not a disagreement to collapse. The tag-grain + # accessor discards exactly the information the keying exists to carry, and every reused + # barcode then falls back to standing alone under its raw sequence. Values are stripped + # here for the same reason `consistent_properties` strips them. + grouping: Grouping = {} + ungrouped_pairs: list[tuple[str, str]] = [] + # What each identity was grouped ON, recorded here because this is the one place that + # knows both the column and the row's value. `panel-file-authority` makes a grouped-on + # column a declaration of the identity, unique by construction: every member carries the + # same value, because that value is what put it there. It cannot be recovered later from + # tag-grain agreement, since a reused barcode has none. + declared: dict[str, dict[str, str]] = {} + flagged: set[str] = set() + rows = zip( + panel["tag"].to_list(), + panel["sample"].to_list(), + *(panel[c].to_list() for c in columns), + ) + for tag, sample, *raw_values in sorted(rows): + if tag in reference_tags: + continue + values = [(v or "").strip() for v in raw_values] + # ALL named columns must carry a value. A combination missing one component is not + # that combination, and supplying the absent one would invent a declaration. + if all(values): + identity = GROUPING_KEY_SEPARATOR.join(values) + grouping[(tag, sample)] = identity + declared.setdefault(identity, {}).update(dict(zip(columns, values))) + flagged.update(v for v in values if GROUPING_KEY_SEPARATOR.strip() in v) + else: + # A pair the grouping column says nothing about keeps its own identity rather + # than vanishing. Dropping it would remove a declared reagent from the answer + # with nothing downstream able to tell the panel was short. + grouping[(tag, sample)] = tag + ungrouped_pairs.append((tag, sample)) + # Reported as distinct TAGS, never pairs. The returned list is a contract: it travels in + # the run meta as `tagsWithoutGroupingValue`, and the punchcard counts it in a banner + # naming barcodes rather than (barcode, sample) pairs. + ungrouped = sorted({tag for tag, _sample in ungrouped_pairs}) + if ungrouped: + # Also returned, not only logged. A property the file does not carry narrows what can + # be answered, and that narrowing has to be visible in the output rather than in a log + # line nobody reads afterwards. These tags are answered under a grouping that could + # not place them, so a bare barcode sits among the family identities. + print( + f"[emit-verdicts] {len(ungrouped)} tag(s) carry no value for every one of {columns} and " + f"stand as their own identity: {ungrouped[:8]}", + file=sys.stderr, + ) + if flagged: + print( + f"[emit-verdicts] {len(flagged)} panel value(s) contain " + f"{GROUPING_KEY_SEPARATOR.strip()!r}, which joins grouping columns: {sorted(flagged)[:8]}. " + "Two combinations may share one identity key.", + file=sys.stderr, + ) + return grouping, "property:" + "|".join(columns), ungrouped, declared + + +def _linker_frame(grouping: Grouping) -> pl.DataFrame: + """Which identities each tag feeds -- one row per distinct (tag, identity). + + Not keyed by sample. The linker lets a tag-keyed figure sit beside an identity-keyed + verdict, and neither side carries a sample: verdicts are (set, identity) over clonotypes + that span samples, and the per-tag figures are run-level. An axis no joined table has + disambiguates nothing -- it makes the join malformed. + + Many-to-many by construction: under (tag, sample) grouping one tag can feed a different + identity in each sample. Distinct rows matter, because two tags of one identity would + otherwise emit the same key twice, and duplicate axis keys break a grid silently, + rendering one row and an ellipsis with no error. The sample component is therefore read + and discarded, ANY_SAMPLE included. + """ + rows = {(tag, identity) for (tag, _sample), identity in grouping.items()} + return pl.DataFrame( + sorted(rows), + orient="row", + schema={"tag": pl.String, "identity": pl.String}, + ).with_columns(pl.lit(1, dtype=pl.Int64).alias("1")) + + +def _identity_labels( + grouping: Grouping, + properties: dict[str, dict[str, str]], + feature_col: str, + rule_id: str, + disagreed: dict[str, list[str]] | None = None, +) -> dict[str, str]: + """A readable name per identity, never two identities under one name. + + Under a property grouping the identity is the property value, already the name a reader + recognises. Under the per-tag grouping the identity is a barcode, so the panel's feature + name stands in, and where two barcodes carry the same name the tag is appended. + + `disagreed` is the exception, and it applies to BOTH branches, because both lose a label + the same way. `consistent_properties` drops a property a tag's rows disagree about, so + the tag has no value to group on and no feature name to borrow, and stands under its raw + barcode -- the least readable thing this can produce, at the moment a reader most needs + to understand what happened. Such an identity is labelled with the names it DID declare, + joined: `SARS-TRI-S_WT / SARS-TRI-S_WT__alt1`. The reagent stays recognisable and the + conflict stays visible. + + Which column's disagreements arrive here depends on the rule, and the caller resolves + that before calling. + + The uniqueness rule applies to joined names too. Two tags can disagree about the grouping + column while declaring the SAME pair of names, joining to one string -- so the fallback + would put two identities under one label. Where any label repeats, joined or plain, the + identity is appended. + """ + if rule_id != "per-tag": + joined = {tag: " / ".join(values) for tag, values in (disagreed or {}).items() if values} + by_identity = {identity: joined.get(identity, identity) for identity in set(grouping.values())} + repeated = Counter(by_identity.values()) + return { + identity: (f"{label} ({identity})" if repeated[label] > 1 else label) + for identity, label in by_identity.items() + } + # Three rungs, in this order: the name the samples agreed on, else the names they + # disagreed about joined, else the bare barcode for a tag the panel named nowhere. The + # collision rule below already covers the joined strings. + joined = {tag: " / ".join(values) for tag, values in (disagreed or {}).items() if values} + # Over the IDENTITIES, never the grouping's keys. Under the per-tag grouping an identity + # is a tag, but the grouping is keyed by (tag, sample), so its keys are pairs. Iterating + # them looks up a tuple in `properties`, finds nothing, and drops every label back to the + # bare barcode this function exists to avoid. + names = { + tag: (properties.get(tag, {}).get(feature_col) or joined.get(tag) or tag) for tag in set(grouping.values()) + } + collisions = Counter(names.values()) + return {tag: (f"{name} ({tag})" if collisions[name] > 1 else name) for tag, name in names.items()} + + +# The key column of result_identity_properties.csv. A panel column of the same name would +# collapse into the key as the frame is built: the property would not be dropped, it would +# silently BECOME the identity. Such a column is excluded from the export and reported. +IDENTITY_KEY_COLUMN = "identity" + + +def _identity_properties( + grouping: Grouping, + properties: dict[str, dict[str, str]], + columns: list[str], + declared: dict[str, dict[str, str]], + disagreed: dict[str, dict[str, list[str]]] | None = None, +) -> dict[str, dict[str, str]]: + """Per identity, the panel declarations that hold of it. + + A declaration reaches an identity two ways, and `panel-file-authority` fixes both. + + The columns the scientist GROUPED ON arrive in `declared`, from the builder that formed + the identities. They are declarations by construction: every member carries the same + value, because that value is what put it there. They are taken rather than tested, and + they must be -- a reused barcode has no tag-grain agreement to test. + + Every OTHER column holds only where all of the identity's member tags agree. + + Whatever the panel says consistently about an identity's tags must travel with that + identity's verdicts, so a reader sees the declaration wherever the reading appears. + Without this a downstream reader sees that an identity was bound and not what the + scientist declared it to be. The rule is `consistent_properties`' own rule lifted one + grain: there it holds across a tag's ROWS, here across an identity's TAGS. A property + differing between member tags is omitted, neither blanked nor resolved to a winner. + + A tag that declares nothing does not block its neighbours. `disagreed` separates that + silence from a tag whose own rows contradict each other, which without it reaches the + test below as the empty string and is filtered out like a blank cell. On a panel with + barcode reuse that inverts the outcome: on a real sixteen-row panel, an identity whose + five member tags declared six different antigen names came back carrying ONE member's + name, because four had contradicted themselves into silence and the survivor then agreed + with nobody but itself. A member that contradicted itself is a disagreement, not a + silence, and it blocks the property. Strictly more omission and never more assertion. + + Reference tags need no exclusion: `_build_grouping` keeps them out of the grouping. + """ + # Distinct member tags per identity. The grouping is keyed (tag, sample), so one tag + # reaches an identity once per sample that declares it there. The membership test keeps a + # tag from counting twice -- a repeat would misreport how many tags an identity holds. + tags_of: dict[str, list[str]] = {} + for (tag, _sample), identity in sorted(grouping.items()): + members = tags_of.setdefault(identity, []) + if tag not in members: + members.append(tag) + + conflicted = disagreed or {} + held: dict[str, dict[str, str]] = {} + for identity, tags in tags_of.items(): + # Seeded with what the identity was grouped on. Those columns are settled, so the + # agreement test below skips them rather than re-deciding them from a grain that + # cannot answer. + agreed: dict[str, str] = dict(declared.get(identity, {})) + for column in columns: + if column in agreed: + continue + # A member that contradicted itself blocks the property. Checked BEFORE the + # values are gathered, because such a member contributes nothing to them and + # would otherwise look like one that declared nothing. + if any(tag in conflicted.get(column, {}) for tag in tags): + continue + values = {v for v in (properties.get(tag, {}).get(column, "") for tag in tags) if v} + if len(values) == 1: + agreed[column] = next(iter(values)) + held[identity] = agreed + return held + + +def _panel_id(tags: frozenset[str]) -> str: + """A stable id for a declared tag set. + + No panel file names its panel, so the id is derived from the sorted tag list and is the + same in every re-run of the same declaration. Where one panel covers every sample, the + axis takes a single value and drops out. + """ + return hashlib.sha256("\t".join(sorted(tags)).encode()).hexdigest()[:12] + + +def _declared_by_sample(panel: pl.DataFrame, samples: list[str]) -> dict[str, frozenset[str]]: + """Each sample's declared tag set, with the unkeyed panel applying to all.""" + everywhere = set(panel.filter(pl.col("sample") == ANY_SAMPLE)["tag"].to_list()) + return { + sample: frozenset(everywhere | set(panel.filter(pl.col("sample") == sample)["tag"].to_list())) + for sample in samples + } + + +def _cells_by_set(linker: pl.DataFrame) -> dict[str, list[CellKey]]: + """Set membership from the linker, each cell listed once under its set. + + `combine_cells` asserts the map is disjoint, so a cell listed under two sets fails loudly + there rather than being counted twice into a tally that counts every cell once. + """ + members: dict[str, list[CellKey]] = {} + seen: set[tuple[str, CellKey]] = set() + for sample_id, cell_id, set_id in linker.iter_rows(): + key = (sample_id, cell_id) + if (set_id, key) in seen: + continue + seen.add((set_id, key)) + members.setdefault(set_id, []).append(key) + return {set_id: sorted(keys) for set_id, keys in sorted(members.items())} + + +def count_by_set(cells_by_set: dict[str, list], population: set) -> dict[str, int]: + """How many of each clonotype's cells fall in `population`. + + Two populations use it: cells a gate set aside, and cells that read nothing at all. Both + are properties of the cell rather than of a position, since a cell that answered nothing + answered nothing at every identity. Every clonotype appears, zeros included, because a + reader must not have to tell "none of them" apart from "column missing". A caller writing + into the run record drops the zeros itself, since that file is parsed on every render. + """ + return {set_id: sum(1 for key in cells if key in population) for set_id, cells in sorted(cells_by_set.items())} + + +def _pivot_identity_summary(verdicts: pl.DataFrame, universe: set[str]) -> tuple[pl.DataFrame, pl.DataFrame, bool]: + """The per-set verdict row and its support, one column per identity in each. + + Pivoted onto the set axis alone, because the block that consumes this drops a column + carrying an axis the clonotype anchor does not have, with no error. Gated on identity + count: the pivot costs a column per identity, and a large panel would turn one artifact + into a thousand. + + The second frame is the readout's, and its cell carries everything a reader needs to ask + "why is this mark this colour": `state|answered|couldAnswer|agreement|reason|bound`. + `agreement` and `reason` are empty where they do not apply. `bound` is last because it + was appended, so a reader that destructures the first five fields positionally still + decodes a value written before it existed. + + No score, and no binding level. `binary-narrowing` forbids a reading of the antigen + counts as a level or an order from leaving this block, so the cell explains a verdict by + what it RESTS on -- how many cells could answer, how many did, how far they agreed. + + One column rather than five, for two reasons. `support-travels-with-the-reading` obliges + both counts to travel with a verdict *wherever it appears*, and a punchcard drawn from + the state pivot alone would not. And the support cannot arrive as sibling columns: a + column name here IS an antigen name from a customer's panel file, so any suffix marking a + support column is a name some panel is entitled to use, and a grid pairs cells only by + position, which no import guarantees. + + The state pivot is left as it is, because lead selection reads it and a compound value + would not filter. + """ + if len(universe) > IDENTITY_SUMMARY_MAX_IDENTITIES or verdicts.height == 0: + sets = verdicts.select("setId").unique() if verdicts.height else pl.DataFrame(schema={"setId": pl.String}) + return sets, sets, False + ordered = ["setId", *sorted(universe)] + states = verdicts.pivot(on="identity", index="setId", values="state").select(ordered) + # Every part is cast and null-filled before joining. concat_str propagates a null through + # the whole value, so one absent agreement would blank the state beside it. + punch = verdicts.with_columns( + pl.concat_str( + [ + pl.col("state"), + pl.col("cellsAnswered").cast(pl.String).fill_null(""), + pl.col("cellsCouldAnswer").cast(pl.String).fill_null(""), + pl.col("agreement").cast(pl.String).fill_null(""), + pl.col("unreliableReason").cast(pl.String).fill_null(""), + # Sixth, and APPENDED rather than inserted, so a reader that destructures + # the first five fields positionally still reads them correctly and a project + # whose last run predates this field renders unchanged. The expansion needs + # it: at each identity, how many of its cells read bound. + pl.col("cellsBound").cast(pl.String).fill_null(""), + ], + separator="|", + ).alias("punch") + ).pivot(on="identity", index="setId", values="punch") + return states, punch.select(ordered), True + + +# A run whose cell count passes this gets no per-cell punchcard. The frame below is the DENSE +# per-cell-per-identity grid the rest of this module never builds -- 11-20x the sparse input on +# a realistic panel. This bounds the row count; IDENTITY_SUMMARY_MAX_IDENTITIES bounds the width. +# +# Above the line the export is skipped and `cellPunchEmitted` in the run record carries that. +# No page reads that flag yet, so the skip is currently silent to a reader. +CELL_PUNCH_MAX_CELLS = 2_000_000 + + +def _pivot_cell_punch( + states: pl.DataFrame, + cells_by_set: dict[str, list[CellKey]], + offered_by_sample: dict[str, set[str]], + admissibility: Admissibility, + universe: set[str], +) -> tuple[pl.DataFrame, bool]: + """One row per cell, one column per identity: that cell's own reading, not its set's verdict. + + The same four states the set-level card uses, and for the same reason: a cell asked about + an identity always resolves to one of them. Three come straight from `read_states`. The + fourth is structural -- an identity no sample holding this cell offered is NEVER_ASKED -- + and it is the only way a position here is blank. + + **A cell with no row in `states` is not an absence.** It was asked and read nothing, its + count is zero, and a zero count resolves the same way every time: NOT_BOUND, unless the + cell cannot be compared, in which case UNRELIABLE. That is `silent_tally`'s rule, not + re-derived here because the deciding function, `_admissibility_reason`, is the one both + `read_states` and `silent_tally` already call. Drawing a silent cell as an empty position + would contradict the arithmetic that produced its set's verdict, where the same cell + voted. + + `setId` travels as a COLUMN rather than an axis. The readout shows one clonotype at a + time and filters on it, and a cell belongs to exactly one set. + """ + members = [(sample, cell, set_id) for set_id, keys in sorted(cells_by_set.items()) for sample, cell in keys] + ordered_identities = sorted(universe) + empty = pl.DataFrame(schema={"sampleId": pl.String, "cellId": pl.String, "setId": pl.String}) + if not members or not ordered_identities: + return empty, False + # Both gates, and the identity one is the same limit the set-level pivot uses: a column + # per identity is a p-column per identity, whichever axis the rows are on. + if len(ordered_identities) > IDENTITY_SUMMARY_MAX_IDENTITIES or len(members) > CELL_PUNCH_MAX_CELLS: + return empty, False + + member_frame = pl.DataFrame( + members, orient="row", schema={"sampleId": pl.String, "cellId": pl.String, "setId": pl.String} + ) + offered_frame = pl.DataFrame( + [(sample, identity) for sample, ids in sorted(offered_by_sample.items()) for identity in sorted(ids)], + orient="row", + schema={"sampleId": pl.String, "identity": pl.String}, + ) + # The cell's own half of the reason, the admissibility gate, is one row per member rather + # than one per member and identity, because no identity changes whether a cell was set + # aside. The other half, where a comparator is keyed by identity, is joined below as + # (sample, identity): a frame of samples by identities, thousands of rows against the + # grid's tens of millions. + reasons = pl.DataFrame( + [ + ( + sample, + cell, + (lambda r: r.value if r is not None else None)( + cell_admissibility_reason((sample, cell), admissibility) + ), + ) + for sample, cell, _ in members + ], + orient="row", + schema={"sampleId": pl.String, "cellId": pl.String, "cellReason": pl.String}, + ) + + # Joined to `offered` rather than crossed with the universe: a position no sample holding + # the cell offered must not appear at all, or the silent rule below would resolve a + # question nobody asked. Where the comparator is keyed by identity, a (sample, identity) + # with no fitted background is uncomparable for every cell of that sample, and only for + # that identity. Carried as the pairs that DID fit, so the missing ones fall out of a left + # join as nulls. + fitted = ( + pl.DataFrame( + sorted(admissibility.by_identity), + orient="row", + schema={"sampleId": pl.String, "identity": pl.String}, + ).with_columns(pl.lit(True).alias("_fitted")) + if admissibility.by_identity is not None + else None + ) + + grid = ( + member_frame.join(offered_frame, on="sampleId", how="inner") + .join(reasons, on=["sampleId", "cellId"], how="left") + .join( + states.select("sampleId", "cellId", "identity", "state", "unreliableReason"), + on=["sampleId", "cellId", "identity"], + how="left", + ) + ) + if fitted is not None: + grid = grid.join(fitted, on=["sampleId", "identity"], how="left").with_columns( + pl.when(pl.col("cellReason").is_not_null()) + .then(pl.col("cellReason")) + .when(pl.col("_fitted").is_null()) + .then(pl.lit(UnreliableReason.NO_COMPARATOR.value)) + .otherwise(None) + .alias("cellReason") + ) + + grid = grid.with_columns( + pl.when(pl.col("state").is_not_null()) + .then(pl.col("state")) + .when(pl.col("cellReason").is_not_null()) + .then(pl.lit(State.UNRELIABLE.value)) + .otherwise(pl.lit(State.NOT_BOUND.value)) + .alias("cellState"), + # The reason a POSITION is unreliable where one was recorded, and the cell's own + # reason where the position is silent. Never both: a recorded row already carries + # whichever applied. + pl.when(pl.col("unreliableReason").is_not_null()) + .then(pl.col("unreliableReason")) + .otherwise(pl.col("cellReason")) + .alias("reason"), + ) + + # How many identities this cell read BOUND, over the identities it was asked. Counted + # before the pivot, where it is one group_by, and from the resolved state, so a silent + # position counts as the not-bound it is. + bound_counts = ( + grid.group_by("sampleId", "cellId") + .agg((pl.col("cellState") == State.BOUND.value).sum().alias("boundIdentities")) + .with_columns(pl.col("boundIdentities").cast(pl.Int64)) + ) + + # `state|reason`, two fields and nothing else. The set-level punch carries six because a + # verdict rests on counts a reader needs beside it. A cell IS the evidence, so there is + # nothing to report about how much of it there was. + punch = grid.with_columns( + pl.concat_str([pl.col("cellState"), pl.col("reason").fill_null("")], separator="|").alias("punch") + ).pivot(on="identity", index=["sampleId", "cellId"], values="punch") + + # Every identity gets a column even where no cell was offered it, so the readout's columns + # are the panel rather than whatever this run happened to ask. + for identity in ordered_identities: + if identity not in punch.columns: + punch = punch.with_columns(pl.lit(None, dtype=pl.String).alias(identity)) + + return ( + punch.join(member_frame, on=["sampleId", "cellId"], how="left") + .join(bound_counts, on=["sampleId", "cellId"], how="left") + .select("sampleId", "cellId", "setId", "boundIdentities", *ordered_identities), + True, + ) diff --git a/software/per-cell-metrics/src/panel.py b/software/per-cell-metrics/src/panel.py new file mode 100644 index 0000000..1304c74 --- /dev/null +++ b/software/per-cell-metrics/src/panel.py @@ -0,0 +1,340 @@ +"""The panel file as a (tag, sample) table. + +The panel is authoritative and cannot be checked against its subject, so it is +checked against the reads in both directions instead -- per sample, because the +same barcode can carry a different antigen in a different sample's panel and a +global check would let a barcode undeclared in one sample pass on another's +declaration. + +A tag is the barcode sequence. The feature name is a declared property, not an +identity: a name only travels where every row for that tag agrees on it. + +An identity is a group of tags asked as one question. The universe is every +identity a verdict row exists at. Offered is the subset a given set of cells was +actually presented. The universe always contains offered, and that gap is where +"never asked" lives. +""" + +from __future__ import annotations + +from typing import NamedTuple + +import polars as pl + +# Stands for "every sample" where the panel carries no sample column. The unkeyed +# case is this rule with the sample component constant, not a separate rule. +ANY_SAMPLE = "*" + + +class Panel(NamedTuple): + frame: pl.DataFrame + dropped_lines: list[int] + + +def _csv_line(row_index: int) -> int: + """1-based CSV record ordinal, header counted. Not the physical line number: the + two differ when a quoted field contains a newline.""" + return row_index + 2 + + +def read_panel(csv_path: str, roles: dict[str, str]) -> Panel: + """Read the panel CSV into a (tag, sample) table. + + Returns the table and the CSV lines dropped for a blank barcode, as 1-based + record ordinals with the header counted. + + Normalisation is asymmetric on purpose: "tag" and "sample" are stripped because + they are keys, and property columns are carried through exactly as written. + consistent_properties() normalises them, so reading a property column directly + can yield " AgA " and "AgA" as two distinct values. + + Compare emit_feature_properties.py, which consolidates the same file's properties + by feature NAME with first-non-empty-wins and no sample dimension -- the + global-check failure this module exists to avoid. The two rules coexist today and + a caller must choose knowingly. + """ + raw = pl.read_csv(csv_path, infer_schema_length=0) + barcode_col, sample_col = roles["barcode"], roles.get("sample") or "" + + for name, col in (("barcode", barcode_col), ("feature", roles["feature"])): + if col not in raw.columns: + raise SystemExit(f"panel file has no {name} column {col!r}; columns are {raw.columns}") + if sample_col and sample_col not in raw.columns: + raise SystemExit(f"panel file has no sample column {sample_col!r}; columns are {raw.columns}") + + # Two roles on one column silently makes "sample" a copy of "tag": the barcode + # alias below runs first and overwrites it, so the sample expression then reads + # barcodes. Per-sample keying is gone, with no error and no duplicate to catch it. + # The name-vs-role guard below misses this too. Reachable from the UI, whose + # Sample-column dropdown is unfiltered. + bound = [("barcode", barcode_col), ("feature", roles["feature"])] + if sample_col: + bound.append(("sample", sample_col)) + seen: dict[str, str] = {} + for role, col in bound: + if col in seen: + raise SystemExit( + f"panel file roles {seen[col]!r} and {role!r} both name column {col!r}; " + "each role needs a column of its own." + ) + seen[col] = role + + # A role column may be named after the column IT ITSELF produces, and nothing + # else. emit_panel.py defaults --tag-col to "tag", so a barcode column called + # "tag" must stay legal, and alias() replaces the same-named source column rather + # than duplicating it. The exclusion cannot widen to "bound to any role": a SAMPLE + # column named "tag" is fatal, because the barcode alias runs first and "sample" + # becomes a copy of "tag". The mirror, a barcode column named "sample", is correct + # only because of alias ordering, and is refused rather than left resting on it. + reserved = set() + if "tag" in raw.columns and barcode_col != "tag": + reserved.add("tag") + if "sample" in raw.columns and sample_col != "sample": + reserved.add("sample") + if "_row" in raw.columns: # injected, so any source column of that name collides + reserved.add("_row") + if reserved: + raise SystemExit( + f"panel file uses reserved column name(s) {sorted(reserved)}; rename them. " + "'tag' and 'sample' are what this reader produces." + ) + + # fill_null is load-bearing, not defensive. Under infer_schema_length=0 a bare + # empty field parses to null while a quoted one parses to "", so the two spellings + # of blank would otherwise take different branches below. + panel = raw.with_row_index("_row").with_columns(pl.col(barcode_col).str.strip_chars().fill_null("").alias("tag")) + panel = panel.with_columns( + pl.col(sample_col).str.strip_chars().fill_null("").alias("sample") + if sample_col + else pl.lit(ANY_SAMPLE).alias("sample") + ) + + # Blank barcodes are separated FIRST, and the order is the whole point. polars + # materializes a trailing blank line as a real all-null row, so a panel whose only + # flaw is a stray newline at EOF would otherwise die on the blank-sample check + # below, telling the user to fix a sample column that is not the problem. Blank + # barcodes are returned rather than filtered away, because dropping a malformed row + # silently leaves nothing downstream able to tell the panel was short. + dropped = [_csv_line(r) for r in panel.filter(pl.col("tag") == "")["_row"]] + panel = panel.filter(pl.col("tag") != "") + + # A blank sample cell on an otherwise real row is fatal, never ANY_SAMPLE. "*" + # means the panel declares no sample dimension at all, so reading an empty cell + # that way would widen one malformed row into a claim over every sample. + if sample_col: + blank_sample = panel.filter(pl.col("sample") == "") + if blank_sample.height: + rows = ", ".join(str(_csv_line(r)) for r in blank_sample["_row"]) + raise SystemExit( + f"panel file has a blank {sample_col!r} on line(s) {rows}. Leave the column out " + "entirely to declare one panel over every sample; a blank cell is ambiguous." + ) + + # ANY_SAMPLE is what this reader writes when there is no sample column, not a + # name a caller can declare. Accepting it in an explicit sample column would + # let one row claim every sample, and a frame mixing "*" with real names is + # exactly what turns the panel-versus-reads check blind. + star = panel.filter(pl.col("sample") == ANY_SAMPLE) + if star.height: + rows = ", ".join(str(_csv_line(r)) for r in star["_row"]) + raise SystemExit( + f"panel file has the literal {ANY_SAMPLE!r} in column {sample_col!r} on line(s) " + f"{rows}. Leave the column out entirely to declare one panel over every sample; " + f"{ANY_SAMPLE!r} is what this reader writes when there is no sample column, not a " + "sample name you can use." + ) + + panel = panel.drop("_row") + + dupes = panel.group_by(["tag", "sample"]).len().filter(pl.col("len") > 1).sort(["tag", "sample"]) + if dupes.height: + offenders = ", ".join(f"{t}/{s}" for t, s in zip(dupes["tag"], dupes["sample"], strict=True)) + raise SystemExit( + f"panel file declares the same barcode twice for one sample: {offenders}. " + "Each (barcode, sample) pair must appear once." + ) + + role_cols = {barcode_col} | ({sample_col} if sample_col else set()) + kept = panel.select(["tag", "sample"] + [c for c in raw.columns if c not in role_cols]) + return Panel(kept, dropped) + + +def property_columns(panel: pl.DataFrame) -> list[str]: + return [c for c in panel.columns if c not in ("tag", "sample")] + + +def consistent_properties( + panel: pl.DataFrame, columns: list[str] +) -> tuple[dict[str, dict[str, str]], list[tuple[str, str, list[str]]]]: + """Per tag, the properties holding one value across all its rows. + + Disagreements are returned rather than dropped. With barcode reuse across panels + an inconsistent declaration is the expected case, and dropping it silently would + break the panel file's own no-silent-drop rule. + """ + props: dict[str, dict[str, str]] = {} + inconsistent: list[tuple[str, str, list[str]]] = [] + for tag, rows in panel.group_by("tag", maintain_order=True): + (name,) = tag + props[name] = {} + for col in columns: + values = sorted({v.strip() for v in rows[col].to_list() if v and v.strip()}) + if len(values) == 1: + props[name][col] = values[0] + elif len(values) > 1: + inconsistent.append((name, col, values)) + return props, inconsistent + + +# (tag, sample) -> the identity that pair belongs to. Many pairs may share one +# identity, and one tag may belong to DIFFERENT identities in different samples. +# +# Keyed by both because the panel file is: a barcode is a reagent identifier drawn +# from a small fixed pool, and the same barcode carries a different antigen wherever a +# study reuses that pool to cover more antigens than it has tags. A map keyed by tag +# alone cannot say what a reused barcode is, and collapses those declarations into a +# disagreement. +# +# A pair absent from the mapping gets no verdict row, which is why every builder must +# leave the reference tags out: the comparator has nothing to be compared against. +# identity_universe() takes no reference_tags of its own, deliberately -- one place +# decides, so the two cannot drift. +Grouping = dict[tuple[str, str], str] + + +def default_grouping(panel: pl.DataFrame, reference_tags: set[str]) -> Grouping: + """One identity per tag, over non-reference tags, at every sample declaring it. + + The identity is still the tag: under the per-tag grouping a barcode names itself in + every sample, so the pairs collapse to one identity per tag and this grouping + carries no per-sample information. It is keyed anyway so every consumer reads one + shape. + + The feature name cannot key an identity: the same barcode carries a different name + in a different sample's panel, so name-keying splits one reagent and can merge two. + The reference is a comparator and never an identity, since asking one of the + reference would compare it with itself. + """ + return { + (tag, sample): tag + for tag, sample in zip(panel["tag"].to_list(), panel["sample"].to_list()) + if tag not in reference_tags + } + + +def identity_universe(panel: pl.DataFrame, grouping: Grouping) -> set[str]: + """Every identity a question is asked at -- the row set for every set's verdicts. + + The union across every sample, because an identity declared in one sample's panel + and not another's is still an identity the run asks at. It reads *never asked* for + sets drawn from the samples that never offered it. + + Using the offered set as the row set instead makes an unoffered identity vanish + from the answer. + """ + return set(grouping.values()) + + +def offered_identities( + panel: pl.DataFrame, + grouping: Grouping, + samples: list[str], + seen: set[tuple[str, str]] | None = None, +) -> set[str]: + """Which identities a set could answer at, given the samples its cells came from. + + An identity was offered when any one of its tags was on any of those samples' + panels AND the reads for that tag came back for that sample. The `any` is + deliberate on both halves: an identity is a group of tags, that group can span + several panels, and one live tag is enough to put the question. + + The identity is read from the (tag, sample) pair itself rather than through a + dataset-wide map, so a barcode carrying one antigen here and another there is not + conflated. + + **`seen` is the set of (sample, tag) pairs the reads carry, and a tag absent from + it is dropped for that sample.** Declaring a tag is not measuring it. Zero reads is + categorical and cannot arise from biology, because ambient material reaches every + cell, so a tag that bound nothing still returns counts. Zero reads means a reagent + never added, a barcode mis-declared, or a library that failed. + + This is not the reads overruling the declaration -- the file declares what was + offered, the reads say what was measured, and those were always different + questions. What follows is only that unmeasured cells do not vote. + + Pass `seen` built from the RAW counts, before the minimum. A count the minimum + zeroed is a reading that happened and failed, which settles *not bound*. A tag with + no reads is a question nobody put. Building `seen` from the floored frame collapses + the second into the first. + + `seen=None` keeps the panel-only reading, for a caller with no counts frame. + + A sample the panel never mentions is offered nothing, so every identity reads + *never asked* for a set drawn from it. + """ + wanted = set(samples) + return { + identity + for (tag, sample), identity in grouping.items() + if (sample == ANY_SAMPLE or sample in wanted) + # A globally-declared tag is live where any wanted sample's reads carry it: the + # declaration spans samples, so the measurement question does too. + and (seen is None or ((sample, tag) in seen if sample != ANY_SAMPLE else any((s, tag) in seen for s in wanted))) + } + + +def panel_read_mismatch(panel: pl.DataFrame, seen: pl.DataFrame) -> pl.DataFrame: + """Both directions of the panel-versus-reads check, per sample. + + Neither direction can be known before the reads are processed, so by the time + either is known the reading exists. Withholding it then would turn a partial answer + into none. This reports and never raises. + + Per sample, because the same barcode can carry a different antigen in a different + sample's panel: a global check lets a barcode undeclared in one sample pass on + another sample's declaration. + + In the global case every row is keyed ANY_SAMPLE, so "*" appears beside real sample + ids without being a sampleId. For declared-never-seen that is honest, because the + claim really is global. For undeclared-in-panel it is lossy: a barcode read only in + one sample reports under "*" and which sample carried it is not recoverable. That + is accepted, because a panel with no sample dimension has no per-sample declaration + to compare against. + """ + # Neither side can place a row with no sample or no barcode, and a null is not a + # usable p-column key. The reader never emits one -- this keeps the promise true + # for a caller that builds a frame directly. + panel = panel.filter(pl.col("sample").is_not_null() & pl.col("tag").is_not_null()) + seen = seen.filter(pl.col("sampleId").is_not_null() & pl.col("tag").is_not_null()) + + rows = [] + global_panel = panel.filter(pl.col("sample") == ANY_SAMPLE) + + # All rows, not any: a frame mixing "*" with real sample names must not take the + # global branch, which would discard every named row and report a per-sample + # disagreement as agreement. The reader refuses such a frame, so this is the second + # line of defence. In such a frame "*" is then compared as a literal sample name, + # so a star row reports a disagreement against a sample called "*". That noise is + # intended: a caller who builds a refused frame gets noise rather than a silent + # pass. + if panel.height and global_panel.height == panel.height: + pairs = [(ANY_SAMPLE, set(global_panel["tag"].to_list()), set(seen["tag"].to_list()))] + else: + pairs = [ + ( + s, + set(panel.filter(pl.col("sample") == s)["tag"].to_list()), + set(seen.filter(pl.col("sampleId") == s)["tag"].to_list()), + ) + for s in sorted(set(panel["sample"].to_list()) | set(seen["sampleId"].to_list())) + ] + + for sample, declared, observed in pairs: + for tag in sorted(declared - observed): + rows.append({"sample": sample, "tag": tag, "direction": "declared-never-seen"}) + for tag in sorted(observed - declared): + rows.append({"sample": sample, "tag": tag, "direction": "undeclared-in-panel"}) + + return pl.DataFrame(rows, schema={"sample": pl.String, "tag": pl.String, "direction": pl.String}).sort( + ["sample", "direction", "tag"] + ) diff --git a/software/per-cell-metrics/src/parse_gate.py b/software/per-cell-metrics/src/parse_gate.py index 2b615da..b911176 100644 --- a/software/per-cell-metrics/src/parse_gate.py +++ b/software/per-cell-metrics/src/parse_gate.py @@ -1,18 +1,17 @@ """Parse gate for the Feature Integration block. -This gate reads the parse report and emits a ``decision.json`` the workflow branches on: -continue the mitool chain only when at least one read matched; otherwise skip -refine/tag-stat (they would crash on the empty input) and feed the empty fallbacks this -gate also writes into the unchanged downstream: - - * ``--empty-tagstat`` a header-only tag-stat TSV (the columns mitool ``tag-stat -t CELL -t - FEATURE -u UMI`` emits), which per_cell_metrics / qc_report already treat - as an empty (0-cell) sample; - * ``--empty-refine-report`` an empty ``{}`` refine report, so qc_report's panel-assigned fraction is - simply blank. - -The fallbacks are written unconditionally (they are trivial and are ignored on the matched>0 path, where -real refine/tag-stat outputs are used instead). Stdlib only -- trivial and fast. +Reads the parse report and writes ``decision.json``, which the workflow branches on. +One or more matched reads: the mitool chain continues. Zero matched reads: refine and +tag-stat are skipped, because they crash on empty input, and this gate writes the empty +outputs they would have produced: + + * ``--empty-tagstat`` header-only TSV with mitool's ``tag-stat -t CELL -t FEATURE + -u UMI`` columns. per_cell_metrics and qc_report read it as + a 0-cell sample. + * ``--empty-refine-report`` empty ``{}``. qc_report's panel-assigned fraction goes blank. + +Both fallbacks are always written. They are trivial, and the matched>0 path ignores them. +Stdlib only. """ import argparse @@ -33,14 +32,14 @@ def main() -> None: with open(args.parse_report) as fh: rep = json.load(fh) - # mitool writes {"parseReport": {"total", "matched", ...}, ...}; tolerate an unwrapped report too. + # mitool writes {"parseReport": {...}, ...}. Also accept an unwrapped report. pr = rep.get("parseReport", rep) total = int(pr.get("total", 0)) matched = int(pr.get("matched", 0)) should_continue = matched > 0 if not should_continue: - # Surfaced in the exec's stderr; the block's analysis log separately flags the zero-cell sample. + # Goes to the exec's stderr. The analysis log flags the zero-cell sample separately. print( f"[parse-gate] parse matched {matched} of {total} reads — no features will be extracted " f"for this sample; check the read geometry / tag pattern against the data", @@ -50,8 +49,8 @@ def main() -> None: with open(args.decision_out, "w") as out: json.dump({"total": total, "matched": matched, "shouldContinue": should_continue}, out) - # Header-only tag-stat fallback: exactly the columns mitool `tag-stat -t CELL -t FEATURE -u UMI` - # emits, so the downstream per_cell_metrics / qc_report see a well-formed empty table. + # Header-only fallback with mitool `tag-stat -t CELL -t FEATURE -u UMI` columns, so + # per_cell_metrics and qc_report see a well-formed empty table. header = f"{args.cell_tag}\t{args.feature_tag}\tcount\ttotalWeight\tunique_{args.umi_tag}\n" with open(args.empty_tagstat, "w") as out: out.write(header) diff --git a/software/per-cell-metrics/src/per_cell_metrics.py b/software/per-cell-metrics/src/per_cell_metrics.py index a23308e..12ea2df 100644 --- a/software/per-cell-metrics/src/per_cell_metrics.py +++ b/software/per-cell-metrics/src/per_cell_metrics.py @@ -1,12 +1,9 @@ """Per-cell feature metrics for the Feature Integration block. Collapses mitool tag-stat output into a (cell x feature) UMI matrix, then computes within-cell -fractions, the consensus feature (dominant-category rule), and an optional Cell Ranger -specificity score. - -The math functions are pure and unit-tested; the CLI wires them to CSV I/O. Every output is sorted -before writing: stable row order makes the CLI deterministic and keeps the workflow's pure-template -dedup canonical. +fractions and a per-cell summary. Math functions are pure and unit-tested; the CLI wires them to +CSV I/O. Every output is sorted before writing, which keeps the workflow's pure-template dedup +canonical. """ import argparse @@ -14,122 +11,6 @@ import sys import polars as pl -from scipy.stats import beta - -DOMINANCE_FLOOR = 0.5 # threshold is user-adjustable down to 0.5, never lower - -# Schema for the no-control specificity output only. When no negative control is set we still emit a -# header-only specificity CSV (the workflow's output set is fixed), and that frame has no source rows, -# so it needs an explicit schema. Every other output is a pure-polars transform of `counts`, which -# carries its schema through the empty case natively (an empty join writes a header-only CSV, not a -# crash). -_SPECIFICITY_SCHEMA = { - "sampleId": pl.Utf8, - "cellId": pl.Utf8, - "feature": pl.Utf8, - "specificityScore": pl.Float64, -} - - -CROSS_REACTIVE = "cross-reactive" - - -def consensus_category( - counts: dict[str, float], - threshold: float, - control: str | None = None, - offtargets: frozenset[str] = frozenset(), - label_crossreactive: bool = False, -) -> str | None: - """Dominant-category rule. - - Returns the single dominant category when it is the unique maximum AND its share of the total is - >= threshold; "ambiguous" when signal exists but no unique category passes (a spread distribution, - or an exact split at the 0.5 floor); None when there is no signal at all. ``threshold`` is clamped - up to the 0.5 floor. - - The negative ``control`` and the ``offtargets`` set are references, not callable antigens: they are - excluded from the winner candidates, so a cell dominated by control/off-target signal is "ambiguous", - never the control or an off-target. Their UMIs are still counted in ``total`` (the denominator), so - control/off-target signal SUPPRESSES antigen dominance rather than being renormalised away — a cell - swamped by them correctly fails the threshold instead of having its top on-target inflated to 100%. - - ``offtargets`` designate features whose property (e.g. Type = Off-Target / Decoy) marks them as - binders the user does not want to call. When they are supplied and ``label_crossreactive`` is set, - the overloaded "ambiguous" bucket is split: a cell whose on-target (non-excluded) signal collectively - passes the threshold but is spread across >= 2 on-target features is called "cross-reactive" (a - genuine multi-/cross-reactive binder — e.g. the same target's human + cyno variants) rather than - lumped with true noise. A cell whose on-target signal fails the threshold (off-target/control-swamped, - or a flat spread) stays "ambiguous". With no off-targets designated the rule is unchanged. - """ - threshold = max(threshold, DOMINANCE_FLOOR) - excluded = set(offtargets) - if control is not None: - excluded.add(control) - positive = {k: v for k, v in counts.items() if v > 0} - total = sum(positive.values()) - if total <= 0: - return None - candidates = {k: v for k, v in positive.items() if k not in excluded} - if not candidates: - return "ambiguous" # only control/off-target (or no) signal — no on-target to call - max_val = max(candidates.values()) - winners = [k for k, v in candidates.items() if v == max_val] - if len(winners) == 1 and (max_val / total) >= threshold: - return winners[0] - # cross-reactive: on-target signal collectively dominates but is split across >= 2 on-targets. - if label_crossreactive and len(candidates) >= 2 and (sum(candidates.values()) / total) >= threshold: - return CROSS_REACTIVE - return "ambiguous" - - -def offtarget_features( - tag_feature_csv: str, - csv_feature_col: str, - offtarget_col: str, - offtarget_values: frozenset[str], -) -> frozenset[str]: - """Feature names whose designated property (``offtarget_col``) value is in ``offtarget_values``. - - The off-target designation is property-driven: the user picks one imported per-feature property - column (e.g. ``antigen_class``) and the set of its values that mark a feature as off-target (e.g. - {"Off-Target", "Decoy"}). This reads the tag->feature CSV — which carries those property columns — - and returns the resolved set of off-target FEATURE names, so the dominant call can exclude them. - - Values are matched exactly, whitespace-trimmed but CASE-SENSITIVE (``strip()`` on both sides, no - case folding): a feature is off-target only if its ``offtarget_col`` value is byte-identical (after - trimming) to one the user selected. Real panels (e.g. B043) may carry mixed casing of one designation - — ``Off-Target`` and ``Off-target`` in a single column — so the user selects every casing they mean; - each distinct value is offered separately in the block's dropdown. Whitespace is trimmed because - leading/trailing spaces are invisible in the picker; casing is left intact because it is visible and - the user's to choose (the block never silently broadens a selection to unselected values). The - returned FEATURE names are verbatim (trimmed) from the CSV. - """ - mapping = pl.read_csv(tag_feature_csv) - if offtarget_col not in mapping.columns: - raise SystemExit( - f"--offtarget-col={offtarget_col!r} is not a column of the tag->feature CSV ({mapping.columns})" - ) - wanted_trimmed = {v.strip() for v in offtarget_values} - resolved = { - (feat or "").strip() - for feat, val in mapping.select( - pl.col(csv_feature_col).cast(pl.Utf8), - pl.col(offtarget_col).cast(pl.Utf8), - ).iter_rows() - if val is not None and val.strip() in wanted_trimmed - } - return frozenset(resolved) - - -def specificity_score(antigen_umi, control_umi): - """Cell Ranger BEAM specificity score, constants are Cell Ranger's: - (1 - betaCDF(0.925, antigenUMI + 1, controlUMI + 3)) * 100. - - Accepts scalars or numpy arrays. scipy's beta.cdf is vectorized, so the CLI passes whole columns - (the array path avoids a per-row Python loop); returns a numpy float or float array accordingly. - """ - return (1.0 - beta.cdf(0.925, antigen_umi + 1, control_umi + 3)) * 100.0 def combine_barcode_counts( @@ -139,100 +20,63 @@ def combine_barcode_counts( feature_modes: dict[str, str], min_umi: float = 1.0, ) -> dict[str, float]: - """Collapse ONE cell's per-barcode UMI counts into per-feature counts, honouring each feature's - combine mode. This is the pure rule the vectorized ``_load`` path mirrors (an oracle test pins them). - - An antigen may be read out by more than one feature barcode (e.g. a dual-labeled probe). Two modes: - - - ``"sum"`` (OR, the default): the feature's UMI is the sum of its member barcodes present in the - cell; the feature is called whenever at least one member barcode has signal. This is the historical - behaviour (barcodes sharing a feature name are summed). - - ``"all"`` (AND): the feature is called ONLY when EVERY member barcode fired — each is present with - ``umi >= min_umi`` in this cell — and its UMI is then the sum of the members. If any member is - missing or below ``min_umi`` the feature is absent for this cell (omitted, not zero), so it does not - compete for dominance, take a fraction, or get a specificity score. This expresses the LIBRA-seq / - dual-probe design where a cell is antigen-specific only when both probe barcodes fire. - - ``barcode_umi`` holds only the barcodes with signal in this cell (mitool tag-stat emits count>0 rows). - Off-panel barcodes (absent from ``barcode_to_feature``) are ignored, mirroring the inner join. - Returns ``{feature: umi}`` for the features called present in this cell. + """Collapse ONE cell's per-barcode UMI counts into per-feature counts by combine mode. + + The pure rule the vectorized ``_load`` path mirrors; an oracle test pins them together. One + antigen can be read out by several barcodes, such as a dual-labeled probe: + + - ``"sum"`` (OR, default): UMI is the sum of member barcodes present in the cell. + - ``"all"`` (AND): called only when every member fired at ``umi >= min_umi``, UMI is their + sum. Otherwise the feature is absent for the cell -- omitted, not zero -- so it takes no + share of the cell's signal. This is the LIBRA-seq dual-probe design. + + ``barcode_umi`` holds only barcodes with signal, because tag-stat emits count>0 rows. + Off-panel barcodes are ignored, mirroring the inner join. """ present: dict[str, dict[str, float]] = {} for bc, umi in barcode_umi.items(): feat = barcode_to_feature.get(bc) if feat is None: - continue # off-panel barcode — ignored, mirrors the tag->feature inner join + continue # off-panel: mirrors the tag->feature inner join present.setdefault(feat, {})[bc] = umi out: dict[str, float] = {} for feat, bc_umis in present.items(): if feature_modes.get(feat, "sum") == "all": members = feature_barcodes[feat] - # Every member must be PRESENT in this cell and clear min_umi. Testing presence explicitly - # (rather than bc_umis.get(bc, 0.0) >= min_umi) keeps AND correct at min_umi == 0, where a - # 0.0 default would otherwise let an absent barcode "fire" — matching the vectorized _load - # path, which never sees absent barcodes because they drop out of the inner join. + # Presence is tested explicitly so AND stays correct at min_umi == 0, where a 0.0 + # default would let an absent barcode fire. _load never sees absent barcodes. if all(bc in bc_umis and bc_umis[bc] >= min_umi for bc in members): out[feat] = sum(bc_umis.values()) - # else: not every member fired -> feature not called in this cell (omitted) + # Not every member fired: omit the feature for this cell. else: # "sum" / OR out[feat] = sum(bc_umis.values()) return out def with_fraction(counts: pl.DataFrame) -> pl.DataFrame: - """Add the within-cell UMI ``fraction`` (each feature's share of its cell's total; sums to 1 per - cell) to the (sampleId, cellId, feature, umiCount) long frame. An empty frame carries its schema - through. Computed once in main() and reused for both the exported fractions CSV and the per-cell - summary so the two never diverge or recompute the window.""" - return counts.with_columns( - (pl.col("umiCount") / pl.col("umiCount").sum().over(["sampleId", "cellId"])).alias("fraction") - ) + """Add the within-cell UMI ``fraction`` to the (sampleId, cellId, feature, umiCount) frame. + Each feature's share of its cell's total, summing to 1 per cell. main() computes it once and + reuses it for the exported CSV and the per-cell summary, so the two cannot diverge. -def with_specificity(frame: pl.DataFrame, control: str) -> pl.DataFrame: - """Add the per-(cell, feature) Cell Ranger specificity score vs the cell's control UMIs (0 when the - cell has no control reads). scipy beta.cdf is evaluated once over the whole column (no per-row loop). - An empty join carries the schema through. Computed once in main() and reused for both the exported - specificity CSV and the per-cell summary's max, so the two never diverge or recompute the betaCDF. - - The control itself is the reference, not a scored antigen: its own row's score is nulled, so the - control never appears as a scored feature in the exported specificity CSV (main() drops null scores) - and never drives the per-cell maxSpecificityScore (a max skips nulls).""" - ctrl = frame.filter(pl.col("feature") == control).select(["cellId", pl.col("umiCount").alias("_controlUmi")]) - joined = frame.join(ctrl, on="cellId", how="left").with_columns(pl.col("_controlUmi").fill_null(0)) - scores = specificity_score(joined["umiCount"].to_numpy(), joined["_controlUmi"].to_numpy()) - return ( - joined.with_columns(pl.Series("specificityScore", scores, dtype=pl.Float64)) - .with_columns( - pl.when(pl.col("feature") == control) - .then(pl.lit(None, dtype=pl.Float64)) - .otherwise(pl.col("specificityScore")) - .alias("specificityScore") - ) - .drop("_controlUmi") - ) + A cell whose counts are all zero divides to NaN, which reaches the exported CSV and then + crashes `per_cell_summary`'s Int64 cast. Zero is the honest reading. Real input cannot get + there, because tag-stat emits count>0 rows only, so the guard is for hand-fed CSVs and any + future counts source. An empty frame carries its schema through.""" + total = pl.col("umiCount").sum().over(["sampleId", "cellId"]) + return counts.with_columns(pl.when(total > 0).then(pl.col("umiCount") / total).otherwise(0.0).alias("fraction")) def per_cell_summary(per_cell: pl.DataFrame) -> pl.DataFrame: - """One row per (sampleId, cellId): the cell's max feature UMI count and max feature fraction - (and, when a ``specificityScore`` column is present, the max specificity score), plus a - ``featureSummary`` string that lists every feature the cell has signal for as - ``feature (fraction%, umiCount UMI)``, bullet-separated and sorted by descending fraction (dominant - feature first, feature name as tie-break). Fractions display as whole percents, with "<1%" for a - nonzero feature that rounds below 1%. - - This is a TABLE-ONLY collapse of the (cell x feature) matrix -- the per-feature abundance, - fractions, consensus, and specificity outputs (the per-cell export contract) are unaffected. - ``per_cell`` is the (sampleId, cellId, feature, umiCount) long frame ALREADY carrying the - ``fraction`` column (and ``specificityScore`` when a negative control is set) that main() computed - once for the exported CSVs -- so the per-cell maxima can never diverge from the exported columns, - and the fraction window / betaCDF are not recomputed here. An empty frame carries its schema through - to a header-only summary. - """ - has_control = "specificityScore" in per_cell.columns + """One row per (sampleId, cellId): max feature UMI count, max fraction, and a + ``featureSummary`` string of every feature with signal as ``feature (fraction%, umiCount UMI)``, + bullet-separated, largest share first, feature name as tie-break. - # Whole-percent display of the fraction, with "<1%" for a nonzero feature that rounds below 1% (so a - # real-but-tiny signal never reads as "0%"). Full-precision fractions stay in the exported columns. + A TABLE-ONLY collapse; the abundance and fractions exports are unaffected. ``per_cell`` already + carries the ``fraction`` main() computed, so the maxima cannot diverge from the exports. An + empty frame carries its schema through. + """ + # "<1%" so a real-but-tiny signal never reads as "0%". Exports keep full precision. pct = (pl.col("fraction") * 100).round(0) pct_str = ( pl.when((pct == 0) & (pl.col("umiCount") > 0)) @@ -252,15 +96,11 @@ def per_cell_summary(per_cell: pl.DataFrame) -> pl.DataFrame: pl.col("fraction").max().alias("maxFraction"), pl.col("_entry") .sort_by(["fraction", "feature"], descending=[True, False]) - # comma-separated, dominant feature first. + # comma-separated, largest share first. .str.join(", ") .alias("featureSummary"), ] - out_cols = ["sampleId", "cellId", "maxUmiCount", "maxFraction"] - if has_control: - aggs.append(pl.col("specificityScore").max().alias("maxSpecificityScore")) - out_cols.append("maxSpecificityScore") - out_cols.append("featureSummary") + out_cols = ["sampleId", "cellId", "maxUmiCount", "maxFraction", "featureSummary"] return per_cell.group_by(["sampleId", "cellId"]).agg(aggs).select(out_cols).sort(["sampleId", "cellId"]) @@ -277,25 +117,21 @@ def _load( ) -> pl.DataFrame: """Aggregated mitool ``tag-stat -u`` rows -> (cellId, feature, umiCount) long frame. - ``mitool tag-stat -t CELL -t FEATURE -u UMI`` emits one row per (cell, feature-barcode) group: - columns ``CELL FEATURE count totalWeight unique_UMI``. ``unique_UMI`` is the distinct-UMI - (molecule) count for the group -- mitool does the deduplication, so we take that column directly - rather than counting raw UMI rows ourselves. The tag->feature CSV maps the feature barcode to its - feature/antigen name; ``csv_barcode_col``/``csv_feature_col`` let the user - map arbitrary CSV header names to that barcode/feature role. Barcodes that map to the same feature - are collapsed per that feature's combine mode (``combine_col``): ``"sum"``/absent sums the - distinct-UMI counts (OR — the default), ``"all"`` emits the feature only in cells where every member - barcode fired (>= ``min_umi``; AND — see ``combine_barcode_counts``). The output column is always - named ``feature`` regardless of the source CSV's header, since downstream Xsv import depends on it. + ``tag-stat -t CELL -t FEATURE -u UMI`` emits one row per (cell, barcode) group with columns + ``CELL FEATURE count totalWeight unique_UMI``. mitool deduplicates, so ``unique_UMI`` is taken + directly as the molecule count. The tag->feature CSV maps barcode to feature name; + ``csv_barcode_col`` / ``csv_feature_col`` map arbitrary headers onto those roles. + + Barcodes sharing a feature collapse by its ``combine_col`` mode: ``"sum"`` or absent sums the + counts (OR, default), ``"all"`` emits the feature only where every member fired at + >= ``min_umi``. The output column is always named ``feature``, because Xsv import needs it. """ stat = pl.read_csv(tag_stat_tsv, separator="\t") - # A header-only tag-stat (a sample whose reads were all dropped -- e.g. every read off-panel) has no - # data rows, so polars infers every column as String. Coerce the UMI-count column to a numeric type - # up front, otherwise the downstream fraction division fails on String arithmetic. On a populated - # file the column is already integer and this cast is a no-op. + # A header-only tag-stat (every read dropped) has no data rows, so polars infers String for + # every column. Coerce here, or the fraction division fails on String arithmetic. stat = stat.with_columns(pl.col(umi_count_col).cast(pl.Int64)) - mapping = pl.read_csv(tag_feature_csv) # columns: csv_barcode_col (feature barcode), csv_feature_col - # Normalize the join key + feature name like mitool. + mapping = pl.read_csv(tag_feature_csv) # columns: csv_barcode_col, csv_feature_col + # Normalize the join key and feature name the way mitool does. mapping = mapping.with_columns( pl.col(csv_barcode_col).cast(pl.Utf8).str.strip_chars(), pl.col(csv_feature_col).cast(pl.Utf8).str.strip_chars(), @@ -308,9 +144,8 @@ def _load( f"[per-cell-metrics] tag->feature CSV: {mapping.height} rows, columns={mapping.columns}", file=sys.stderr, ) - # A barcode must appear on exactly one CSV row. A barcode repeated across rows would fan the inner - # join below out once per copy, and the group_by(...).sum() that follows would then multiply that - # barcode's molecule counts silently DOUBLING counts. + # A barcode must appear on exactly one CSV row. A repeat fans the inner join out once per copy, + # and the group_by(...).sum() below then silently DOUBLES that barcode's molecule counts. dup_barcodes = ( mapping.group_by(csv_barcode_col).agg(pl.len().alias("_n")).filter(pl.col("_n") > 1)[csv_barcode_col].to_list() ) @@ -320,9 +155,8 @@ def _load( f"(column {csv_barcode_col!r}); each feature barcode must map to exactly one feature. " f"Remove the duplicate rows: {dup_barcodes[:8]}" ) - # Per-feature combine mode + member-barcode set, parsed once from the (small) mapping. Default is - # "sum" (OR). A combine column lets a feature request "all" (AND) — see combine_barcode_counts. A - # blank cell means unset (defaults to "sum"); the non-blank rows of one feature must agree. + # Per-feature combine mode and member set, parsed once from the small mapping. Default "sum" + # (OR); "all" requests AND. A blank cell is unset. Non-blank rows of one feature must agree. feature_barcodes: dict[str, set[str]] = {} feature_modes_raw: dict[str, set[str]] = {} map_cols = [csv_barcode_col, csv_feature_col] + ([combine_col] if combine_col else []) @@ -361,10 +195,9 @@ def _load( if csv_feature_col != "feature": rename[csv_feature_col] = "feature" - # Per-feature mode + expected member-barcode count, as a frame to join onto the aggregate. n_expected - # is how many DISTINCT barcodes map to the feature; the AND gate keeps a (cell, feature) group only - # when that many member barcodes fired in the cell. With no combine column every feature is "sum", - # so the filter is a no-op and the result is identical to the historical sum-only behaviour. + # Per-feature mode and expected member count, joined onto the aggregate. n_expected counts the + # DISTINCT barcodes mapping to the feature; the AND gate keeps a group only when that many + # fired. With no combine column every feature is "sum", so the filter is a no-op. mode_df = pl.DataFrame( { csv_feature_col: list(feature_barcodes.keys()), @@ -378,11 +211,11 @@ def _load( .group_by([cell_col, csv_feature_col]) .agg( pl.col(umi_count_col).sum().alias("umiCount"), - # distinct member barcodes that fired in this cell (each tag-stat row is one barcode) + # distinct members that fired here (one tag-stat row per barcode) pl.col("_fired").sum().cast(pl.UInt32).alias("_nFired"), ) .join(mode_df, on=csv_feature_col, how="left") - # sum-mode features always survive; "all"-mode only when every member barcode fired + # sum-mode always survives. "all"-mode survives only when every member fired. .filter((pl.col("_mode") != "all") | (pl.col("_nFired") == pl.col("_nExpected"))) .select([cell_col, csv_feature_col, "umiCount"]) .rename(rename) @@ -394,6 +227,22 @@ def _load( return counts +# The live-progress contract the mitool steps already use. The workflow captures this step's stdout as +# a stream and the model scrapes lines carrying this prefix, so printing one here puts the Python step +# on the same bar as parse, refine and tag-stat. Without it the bar sat at the band floor for the whole +# of this step, which is the slowest one on a large run. +# +# The percent is the share of THIS step done when the named phase begins, so the bar advances as each +# phase starts rather than jumping at the end. No ETA: these phases are whole-frame polars operations +# with no iteration count to extrapolate from, and an invented ETA is worse than none. +_PROGRESS_PREFIX = "[==PROGRESS==]" + + +def _progress(stage: str, percent: float) -> None: + """Emit one progress line. Flushed, or it sits in the pipe buffer until the step ends.""" + print(f"{_PROGRESS_PREFIX}{stage}: {percent:.1f}%", flush=True) + + def main() -> None: p = argparse.ArgumentParser() p.add_argument("tag_stat_tsv") @@ -430,42 +279,15 @@ def main() -> None: help="minimum per-barcode distinct-UMI count for a barcode to count as 'fired' under the 'all' " "(AND) combine mode (default 1)", ) - p.add_argument("--dominance-threshold", type=float, default=0.6) - p.add_argument("--control", default=None, help="negative-control feature name") - p.add_argument( - "--offtarget-col", - default=None, - help="CSV property column (e.g. antigen_class) designating on/off-target; features whose value " - "is in --offtarget-values are excluded from the dominant call (like the control) and enable the " - "cross-reactive label", - ) - p.add_argument( - "--offtarget-values", - default=None, - help="comma-separated values of --offtarget-col that mark a feature as off-target (e.g. 'Off-Target,Decoy')", - ) p.add_argument("--output-prefix", default="result") args = p.parse_args() - # Resolve the off-target feature set from the designated property column + values. Both flags must be - # given together; features carrying an off-target value are excluded from the dominant call (as the - # control is) and turn on the cross-reactive label. Absent -> unchanged behaviour (empty set). - offtargets: frozenset[str] = frozenset() - if (args.offtarget_col is None) != (args.offtarget_values is None): - raise SystemExit("--offtarget-col and --offtarget-values must be given together") - if args.offtarget_col is not None: - wanted = frozenset(v.strip() for v in args.offtarget_values.split(",") if v.strip()) - offtargets = offtarget_features(args.tag_feature_csv, args.csv_feature_col, args.offtarget_col, wanted) - label_crossreactive = len(offtargets) > 0 - - # Guard the user-mapped CSV column names: the two roles must be distinct, and neither may - # collide with a tag-stat column. On the inner join, every tag-stat column is carried into the - # joined frame -- so a --csv-feature-col that names ANY tag-stat column (e.g. `count`, - # `totalWeight`, `unique_UMI`, or the CELL/FEATURE keys) would otherwise pass through the - # join/group silently and put the WRONG data (e.g. numeric counts) into the output `feature` - # column; a collision on the cell key also crashes group_by/rename with a raw polars - # DuplicateError. Read the real tag-stat header so we reject every collision, not just the three - # flag-named columns. + # The two mapped roles must be distinct, and neither may collide with a tag-stat column. The + # inner join carries every tag-stat column through, so a --csv-feature-col naming one of them + # (`count`, `totalWeight`, `unique_UMI`, CELL, FEATURE) would silently put the WRONG data into + # the output `feature` column; a collision on the cell key crashes with a raw polars + # DuplicateError. Read the real header, so every collision is caught, not just the flag-named + # three. with open(args.tag_stat_tsv, newline="") as fh: reserved = set(next(csv.reader(fh, delimiter="\t"), [])) if args.csv_barcode_col == args.csv_feature_col: @@ -486,6 +308,7 @@ def main() -> None: f"{name}={val!r} collides with a tag-stat column ({sorted(reserved)}); choose a different CSV column" ) + _progress("Reading counts", 0.0) counts = _load( args.tag_stat_tsv, args.tag_feature_csv, @@ -499,6 +322,7 @@ def main() -> None: ) counts = counts.with_columns(pl.lit(args.sample_id).alias("sampleId")) + _progress("Writing counts per cell", 45.0) # abundance matrix (cell x feature) UMI counts ( counts.select(["sampleId", "cellId", "feature", "umiCount"]) @@ -506,89 +330,18 @@ def main() -> None: .write_csv(f"{args.output_prefix}_abundance.csv") ) - # within-cell fractions (normalised across features per cell, sum to 1). Computed - # once here (with_fraction) and reused for the per-cell summary so the two never diverge. + _progress("Computing within-cell fractions", 65.0) + # Within-cell fractions, computed once here and reused by the per-cell summary. cf = with_fraction(counts) cf.select(["sampleId", "cellId", "feature", "fraction"]).sort(["sampleId", "cellId", "feature"]).write_csv( f"{args.output_prefix}_fractions.csv" ) - # consensus feature per cell (dominant-category rule), vectorized in polars: the - # dominant feature is the unique per-cell max whose share of the cell's total is >= the threshold - # (clamped to the 0.5 floor); otherwise "ambiguous". No-signal cells never occur here (tag-stat - # counts are all > 0), so None is never produced. Mirrors consensus_category, which the tests pin - # (and an oracle test cross-checks this vectorized path against it). - threshold = max(args.dominance_threshold, DOMINANCE_FLOOR) - # The negative control and the off-target features are references, not callable antigens: exclude - # them from the winner candidates so a control/off-target-dominated cell is "ambiguous", never the - # control or an off-target. Their UMIs stay in `_total` (the denominator, computed from the full - # `counts`), so their signal suppresses dominance rather than being renormalised away. When off- - # targets are designated, a cell whose on-target signal collectively passes the threshold but is - # spread across >= 2 on-targets is "cross-reactive". Mirrors consensus_category(control=..., off - # targets=..., label_crossreactive=...), which the oracle test pins the vectorized path against. - excluded = list(offtargets) + ([args.control] if args.control is not None else []) - antigens = counts if not excluded else counts.filter(~pl.col("feature").is_in(excluded)) - totals = counts.group_by(["sampleId", "cellId"]).agg(pl.col("umiCount").sum().alias("_total")) - tops = antigens.group_by(["sampleId", "cellId"]).agg( - pl.col("umiCount").max().alias("_max"), - (pl.col("umiCount") == pl.col("umiCount").max()).sum().alias("_nAtMax"), - pl.col("feature").sort_by("umiCount", descending=True).first().alias("_top"), - # on-target signal: sum + distinct on-target features present (for the cross-reactive branch) - pl.col("umiCount").sum().alias("_onTotal"), - pl.col("feature").n_unique().alias("_nOn"), - ) - ( - totals.join(tops, on=["sampleId", "cellId"], how="left") - .with_columns( - # _top is null for a cell whose only signal is control/off-target -> ambiguous. - pl.when( - pl.col("_top").is_not_null() - & (pl.col("_nAtMax") == 1) - & (pl.col("_max") / pl.col("_total") >= threshold) - ) - .then(pl.col("_top")) - .when( - # cross-reactive: on-target signal collectively dominates but is split across >= 2 on-targets - pl.lit(label_crossreactive) - & (pl.col("_nOn") >= 2) - & (pl.col("_onTotal") / pl.col("_total") >= threshold) - ) - .then(pl.lit(CROSS_REACTIVE)) - .otherwise(pl.lit("ambiguous")) - .alias("consensusFeature") - ) - .select(["sampleId", "cellId", "consensusFeature"]) - .sort(["sampleId", "cellId"]) - .write_csv(f"{args.output_prefix}_consensus.csv") - ) - - # optional specificity score per (cell, feature) vs the negative control. Computed - # once (with_specificity: scipy beta.cdf vectorized over the whole column) and reused for the - # per-cell summary's max. An empty join carries the schema through natively -> header-only CSV. - if args.control is not None: - summary_frame = with_specificity(cf, args.control) - ( - # The control's own row carries a null score (it is the reference, not a scored antigen) -- - # drop those so the exported specificity is antigen-only. summary_frame KEEPS the control row - # (with a null score) so the per-cell summary's umi/fraction breakdown still shows it. - summary_frame.filter(pl.col("specificityScore").is_not_null()) - .select(["sampleId", "cellId", "feature", "specificityScore"]) - .sort(["sampleId", "cellId", "feature"]) - .write_csv(f"{args.output_prefix}_specificity.csv") - ) - else: - # No negative control: still emit an (empty, header-only) specificity CSV so the workflow's - # fixed output set is satisfied. It is not imported when no control is set (main.tpl and the - # model gate the specificity column on hasControl). - pl.DataFrame(schema=_SPECIFICITY_SCHEMA).write_csv(f"{args.output_prefix}_specificity.csv") - summary_frame = cf - - # per-cell summary (table-only collapse): one row per (sampleId, cellId) with the max feature UMI - # count / fraction (/ specificity, with a control) and the "feature (fraction%, umi) | ..." string. - # summary_frame already carries fraction (+ specificityScore with a control), so nothing is - # recomputed. The maxSpecificityScore column is present only with a control, matching how main.tpl / - # the model gate the specificity import on hasControl. - per_cell_summary(summary_frame).write_csv(f"{args.output_prefix}_per_cell_summary.csv") + # Per-cell summary: one row per (sampleId, cellId) with max UMI count and fraction, plus the + # "feature (fraction%, umi) | ..." string. cf already carries fraction. + _progress("Summarising cells", 85.0) + per_cell_summary(cf).write_csv(f"{args.output_prefix}_per_cell_summary.csv") + _progress("Metrics complete", 100.0) if __name__ == "__main__": diff --git a/software/per-cell-metrics/src/qc_measures.py b/software/per-cell-metrics/src/qc_measures.py new file mode 100644 index 0000000..871e289 --- /dev/null +++ b/software/per-cell-metrics/src/qc_measures.py @@ -0,0 +1,910 @@ +"""The quality measurements a run carries. + +Every measurement carries what it counts, because the reader who meets it is not the +person who chose it: a fraction with a name and no statement of what went into the +numerator gets read as whatever the name suggests, and several of these names suggest +more than they carry. + +Where a line can be defended, it also carries what a bad value implies. Where none +can, it carries nothing -- the number and its distribution are shown and the reader +judges. None carries what to do about it, because advice depends on the run, the study +and what else is available, none of which this readout knows. + +A measurement this module cannot compute is declared anyway, with the reason, so a +reader never mistakes "nothing computed this yet" for "checked and found fine". Most of +the set is computed elsewhere and only declared here: undeclared barcodes and +declared-but-unseen tags in ``panel.py``, the floor's counts and the high-reference-cell +count in ``verdict.py``, per-tag self-disagreement in ``combine.py``, read and per-cell +totals in ``qc_report.py``. This module declares the full set and computes the rest. +""" + +from __future__ import annotations + +import math +from collections.abc import Collection +from dataclasses import dataclass +from enum import Enum +from typing import NamedTuple + +import numpy as np +import polars as pl + + +# Three values and no fourth. `310-qc-status-and-rollup` refuses a fourth and a fifth on the +# grounds that a reader meeting five words in one column reads them as a scale. The two cases +# a fourth word covered are read from the VALUE instead: a computed measurement shows its +# number, and one the run could not supply the inputs for shows the reason in place of one. +# +# So a measurement with no line behind it carries no status -- `status_for` returns None -- +# and the row is still there. The strings are the atom's own, casing included, so a reader +# checking the column against the spec finds the same words. +class Status(str, Enum): + OK = "OK" + WARN = "warn" + ALERT = "alert" + + +class Line(NamedTuple): + """Where a measurement's boundaries sit. + + Two thresholds, because all four inherited lines arrive with both and collapsing them + loses a distinction somebody calibrated. `315-where-the-lines-come-from` keeps the + field's word *error* for the second threshold while the status it produces is *alert*. + + `error` is None where only one boundary was published. A stated recommendation gives one + number, so sequencing depth warns and never alerts. + """ + + warn: float + error: float | None = None + + +class Reading(NamedTuple): + """How one measurement came back, as `roll_up` needs to count it. + + The status alone is no longer enough. Both no-status cases return None from + `status_for`, and the coverage triple still separates them, so the value has to travel + with the status -- a number means computed-but-unjudged, its absence means nothing + computed it. + """ + + status: Status | None + value: float | None + + +@dataclass(frozen=True) +class Coverage: + """A level's status, and how much of it was actually checked. + + `status` is None where nothing at this level carried one. A level with nothing judged + makes no claim, which is the same refusal one level up. + """ + + status: Status | None + judged: int + unjudged: int + not_evaluated: int + + +@dataclass(frozen=True) +class Measurement: + id: str + label: str + level: str # "sample" | "tag" | "identity" | "run" + counts: str # what went into it + implies: str | None = None # what a bad value means, where a line exists + line: str | None = None # which defence route backs `implies`, if any + deferred_reason: str | None = None # set only when nothing computes this yet + # Whether this measurement's status reaches its level's rollup. False only where the + # measurement is a property of a reagent rather than of the sample it was measured on -- + # see `310-qc-status-and-rollup`, which keeps a reagent's failure off every sample. + rolls_up: bool = True + + +MEASUREMENTS: tuple[Measurement, ...] = ( + Measurement( + "readsTotal", + "Reads total and fraction matched", + "sample", + # No line. Exactly four numbers are inherited from the field, and the matched + # share is not one of them: usable antigen-read fraction (warn below 0.20), + # undeclared-barcode fraction (warn above 0.50), aggregate-barcode read fraction + # (warn above 0.05), barcode validity (warn below 0.75). Nothing published says + # what a low matched share means, so nothing here claims to. + "Every read the parser saw, and the share matching the tag pattern.", + ), + # `qc_report._refine_kept_fraction` returns the FEATURE step's outputCount/inputCount -- + # the share of matched reads whose barcode corrects onto a panel entry. Its complement is + # the share landing in barcodes the panel never declared, which is 310's undeclared-barcode + # measurement -- but that measurement is a property of a barcode, not of a sample, and 310 + # is explicit that its status "does not become a sample's". The line that used to sit here + # now backs the undeclared-barcode table's own row (see `undeclaredBarcodeShare` in + # DEFAULT_LINES / _COMPARISON below), keyed by sequence and computed on the pre-refine + # counts, where a barcode the panel never declared can still be seen. This row keeps the + # number and carries no status: `where-the-lines-come-from` gives the field's line to the + # quantity it was published for, and this sample-grain fraction is a different quantity from + # that one only by having been corrected onto the panel first. + # + # The usable row is a different quantity: Cell Ranger `main`, + # `lib/python/cellranger/rna/report_matrix.py`, `_report_genome_agnostic_metrics`, defines + # `frac_feature_reads_usable` as conf-mapped, barcoded reads restricted to the called-cell + # partition (`cell_bcs_union`), over the whole library's read count. UMI validity is that + # source's separate `good_umi_frac` figure and is not part of this one. It is declared below + # as `usableReadFraction` and computed by `usable_read_fraction`, from the post-refine + # tag-stat's `totalWeight` and the cell list -- panel recognition is already built into that + # table, since refine-tags has dropped every FEATURE value outside the panel by the time it + # is written. + # + # The id is a value on the `measurement` axis and a p-column name in the per-sample QC + # frame. Renaming it breaks both. Only the label, the wording and the line moved. + Measurement( + "panelAssignedFraction", + "Fraction of reads in undeclared barcodes (as its complement)", + "sample", + "Reads whose corrected barcode is on the panel, over reads matched.", + ), + # Ported from Cell Ranger `main`, `lib/python/cellranger/rna/report_matrix.py`, + # `_report_genome_agnostic_metrics::frac_feature_reads_usable`: conf-mapped, barcoded + # reads restricted to the called-cell partition, over the whole library's read count. + # `usable_read_fraction` sums the post-refine tag-stat's `totalWeight` over rows whose + # cell barcode is in the cell list, divided by readsTotal. Every row of that table already + # carries a panel-recognised FEATURE value, since refine-tags drops the rest before the + # table is written, so restricting to the cell list is the only condition left to apply. + Measurement( + "usableReadFraction", + "Fraction of antigen reads usable", + "sample", + "Reads whose corrected barcode is on the panel and whose cell barcode is in the cell list, over readsTotal.", + "A low share means most of the library's reads are lost before reaching a called cell " + "with a panel-recognised barcode.", + "inherited", + ), + # The fourth inherited line, and the one `315` says the third status level exists for: the + # only one whose thresholds step the same way twice rather than putting error at total + # failure. The refine-tags report already carries the CELL step this reads, beside the + # FEATURE step above. + Measurement( + "cellBarcodeValidFraction", + "Fraction of reads whose cell barcode the chemistry could have produced", + "sample", + "Reads whose cell barcode corrects onto the chemistry's whitelist, over reads entering correction.", + "A low share means the reads carry cell barcodes this chemistry does not produce, " + "which points at the wrong whitelist or the wrong read geometry.", + "inherited", + ), + # The categorical route's first member. The alerting condition is a fact -- no cell + # barcode observed at all -- rather than a quantity with a published threshold, so `315`'s + # second line route applies rather than its first or third. Computed in `qc_report.py` as + # the count of distinct cell barcodes the tag-stat table carries, before any cell-calling + # step -- the observed-barcode count `readsPerCell` below deliberately does not divide by. + # + # `implies` names only the zero case: nothing downstream can be computed for a sample with + # no cells. Above zero nothing is claimed, because how many cells a sample should yield + # depends on the experiment and no number for that is published. + Measurement( + "cellsDetected", + "Cell barcodes detected", + "sample", + "Distinct cell barcodes in the tag-stat table, before any cell-calling step.", + "Zero cells means nothing downstream can be computed for this sample.", + "categorical", + ), + # Saturation is deliberately NOT measured. The vendor's own report carries it, and + # nothing hangs on it: a scientist cannot act on it for the run already collected, and + # whether the run was deep enough is answered by reads per cell below. A number nobody + # acts on competes for attention with numbers they do. + # + # Per *cell*, not per observed barcode. The vendor's five thousand is per-cell, and in + # droplet data the observed-barcode count exceeds the called-cell count by one to two + # orders of magnitude, because ambient antigen reads land on most barcodes -- dividing + # by it would alert on a healthy library. The cell list arrives later than this module, + # which is why the division happens in the entrypoint. + Measurement( + "readsPerCell", + "Reads per cell", + "sample", + "Reads matched, over cells in the cell list.", + "Below the vendor's recommended minimum the library is undersequenced.", + "recommended-and-observed", + ), + Measurement( + "antigenCountDistribution", + "Distribution of antigen count per barcode", + "sample", + "Deciles of the total antigen count per cell barcode.", + ), + # Ports `detect_outlier_umis_bcs` from Cell Ranger `main`, + # `lib/python/cellranger/feature/antibody/analysis.py`, called for the ANTIGEN library + # type from `cell_calling_helpers.py::remove_antibody_antigen_aggregates`. That function's + # antibody sibling (`detect_aggregate_barcodes`, cross-feature co-elevation against gene + # expression) is a different rule and is not ported. + # + # The source's own floor -- a threshold under 1000 UMIs flags nothing -- is kept unchanged. + # A shallow library can sit entirely under that floor while carrying real aggregate reads, + # in which case this measurement reports 0.0 with the computed threshold in its detail + # rather than a blank: the rule ran and found nothing past its own gate, which is a + # different fact from nothing having run at all. + # + # Divided by `readsTotal` (whole-library, pre-match), not `readsMatched`: the source + # divides flagged reads by the whole antigen library's read count before any matching + # step, and `readsTotal` is the field this module holds that corresponds to that count. + Measurement( + "aggregateBarcodeFraction", + "Fraction of reads in aggregate barcodes", + "sample", + "Reads in barcodes flagged as aggregates by the top-100 IQR rule, over readsTotal.", + "A high share means much of the run's antigen signal comes from a small number of " + "clumped droplets rather than single cells.", + "inherited", + ), + # No line, and worth spelling out because the spec looks like it supplies one. The + # field publishes 0.50, but for one aggregate library fraction. This measurement is + # per sequence at tag level, and a fraction's line does not transfer to a list of + # sequences. Given a count instead, any at-most line collapses into "alerting if a + # single undeclared barcode exists" -- a categorical predicate wearing an inherited + # number. So it ships unjudged, with its sequences and their counts. + Measurement( + "undeclaredBarcodes", + "Undeclared barcodes, and which sequences", + "tag", + "Barcodes the reads carry that the sample's panel does not declare, and which sequences they are.", + ), + # No status. A fact on the tag's row, reported for the reagent's sake rather than the + # answer's, because the answer already carries it: cells in a sample where the tag + # returned nothing do not count toward what could answer there, so the verdict reads + # *never asked*. Warning a reader off an answer that already says so would be a second + # voice on one fact. + Measurement( + "declaredNeverSeen", + "Declared tags the reads never show", + "tag", + "Tags on the sample's panel with no reads at all.", + ), + Measurement( + "floorRemoved", + "Counts removed as below the minimum, and cells left with none", + "sample", + "Readings the minimum zeroed, and cells whose every non-reference reading was removed.", + ), + Measurement( + "uniqueCountsPerCell", + "Reads and unique counts per cell", + "sample", + "Reads and distinct UMIs per cell barcode.", + ), + # Two forms, and the gate decides which. `290-reference-two-roles`: how many are high + # needs a high, and only a declared gate supplies one. With a gate the value counts the + # cells it set aside; with none it is the median of the readings and the detail carries + # their deciles, which is what a scientist reads in order to declare a gate. + # + # No line either way. Nothing published says what share of cells is too high, nor what a + # background reading of any size means -- both are read against the run's own spread. + Measurement( + "highReferenceCells", + "Sticky cells, or the spread of the readings where no gate is declared", + "sample", + "Cells whose reference reading reached the declared gate, or the spread of those readings.", + ), + # The id is a value on the `measurement` axis, so renaming it does not break the column -- + # it splits the rows. Old runs would carry one measurement name and new runs another, and a + # table holding both reads as two measurements. Keep it. + # + # Its three figures changed meaning in 330-the-quality-readout: cells-with-count is now read + # before the minimum, the median is taken over every cell holding a count rather than over + # the bound ones, and every declared tag keeps a row so a dead reagent reads as a zero + # rather than as an absence. + Measurement( + "perAntigen", + "Per antigen: cells with a count, cells called bound, and the median", + "tag", + "Per tag: cells with any count and their median count, both before the minimum, " + "and cells called bound after it.", + ), + # One figure for the run rather than per sample, because the cutoff is one number for the + # run. Only the declared rung produces a score at all: a population baseline yields a + # probability, so under it this measurement does not exist and says so. + # + # No line. `320-qc-measurement-set` carries it so a scientist can move the cutoff to where + # their own run's scores separate, and that licence is unusable unless the scores are in + # front of them. A line here would be the block placing the cutoff instead. + Measurement( + "scoreDistribution", + "Distribution of the run's scores", + "run", + "Deciles of the score over every cell and identity the declared rule scored.", + ), + # Only a population baseline fits one, so under a declared baseline this carries no value + # and says so. `330-the-quality-readout` asks for it as a plot, and it is the only way to + # see whether a tag's counts separated at all -- which a scientist reads BEFORE settling the + # baseline, so it must not depend on any cutoff. + # + # No line. Nothing published says what a background of any size means: it is read against + # the signal mean beside it and against the other tags of the panel, which is a comparison + # rather than a boundary. + Measurement( + "fittedBackground", + "Fitted background, where a population baseline served", + "tag", + "The background component's mean count, its share of cells, and the signal mean beside it.", + ), + # Self-disagreement at an IDENTITY is deliberately not measured. Keeping the tag-level + # figure while dropping this one rests on which confound cancels: marginal binding + # inflates disagreement everywhere, so comparing one tag against its siblings under the + # same cells, run and line leaves a tag that stands clear standing clear for a reason + # that is not biology. The identity-level figure has nothing to compare against, so it + # cannot separate a faulty reagent from a panel of weak binders -- cells of one clonotype + # all agree only where the reading sits clear of the line, so for anything marginal + # disagreement is near certain and the rate measures how many clonotypes sit near it. + # Whatever it would say about one clonotype is already on that verdict. + # + # No line, so it reads unjudged beside its siblings. A tag standing clear of the others + # in its panel is misbehaving whatever the absolute rate -- a real finding, but one a + # reader makes by looking. Applying a threshold would need a multiplier nobody + # published. + Measurement( + "tagDisagreement", + "Clonotype self-disagreement at a single tag", + "tag", + "Of the cells whose set had another cell to compare against, the share reading the opposite way " + "from the rest of their own set, by this tag's count alone. Two states cap it at half.", + ), + # Available only where an identity carries more than one tag. + # + # Distinct from tagDisagreement: cross-tag within one cell, not within-tag across cells. + # + # No line. Nothing published says what share is too high, so nothing here claims to. + Measurement( + "siblingDisagreement", + "Disagreement with the tags sharing its identity", + "tag", + "Of the cells whose siblings reached a majority, the share reading the opposite way from it. The " + "tag casts no vote in the majority it is judged against, and tied siblings judge nothing.", + ), + # Whether a clonotype of known specificity came back correctly is deliberately NOT + # measured. It would be the only end-to-end check of the pipeline, and nothing computes + # it because nothing declares it: no surface asks a scientist which clonotype they + # already know the answer for. What they do instead is find that clonotype in the + # readout and read its row, which the readout already supports. + # + # Doublets, read from cells positive on several antigens, are deliberately NOT measured. + # The field does not read multi-antigen positivity as a doublet estimate, and one vendor + # states outright that it should not be. + # + # A false-discovery rate is deliberately NOT measured. None exists for this assay: the + # bulk-readout relative has had one for a decade, the per-cell form has none, so no + # calibrated statement is available about how many positives in a run are spurious. + # + # The share of counts landing in droplets that held no cell is deliberately NOT measured. + # It cannot be computed against a cell list derived from recovered receptors, because + # cells whose receptor did not assemble are classified as empty and inflate the very + # quantity being measured. +) + +# The three places a line can come from, and nowhere else. `Measurement.line` names one +# or None, and it is the *only* declaration of which measurements carry a line -- the +# tables below are derived facts about the route, never a second opinion. A test asserts +# the correspondence in both directions. +# +# A comparison against the other tags in a panel is NOT a line. It yields no boundary, and +# a status derived from it would need a multiplier -- an interquartile multiple, a +# median-absolute-deviation cut -- that nobody has published, which moves the invention up +# a level rather than removing it. Such a measurement reads unjudged beside its siblings, +# where the comparison is free for a reader to make. The cost is real and accepted: a +# barcoded reagent binding something other than the receptor no longer announces itself. +# +# The categorical route now has one member: `cellsDetected`, declared above. A declared tag +# the reads never show was its earlier example until the verdict took the job: that condition +# now removes the tag's cells from what could answer, so the answer carries the finding. +LINE_ROUTES: frozenset[str] = frozenset({"inherited", "categorical", "recommended-and-observed"}) + +# The categorical route's member ids. Derived from `Measurement.line`, never a second +# declaration of which measurement stands on this route. A categorical fact carries no +# numeric threshold, so its id is deliberately absent from `DEFAULT_LINES` and `_COMPARISON` +# below -- `status_for` answers it before either table is consulted. +_CATEGORICAL: frozenset[str] = frozenset(m.id for m in MEASUREMENTS if m.line == "categorical") + +# Every line is a parameter with a shipped default, and the operator may override any of +# them. No line is invented -- where none of the three routes applies the measurement carries +# no status rather than being given a number with nothing behind it. +# +# `undeclaredBarcodeShare` backs the undeclared-barcode table's own row rather than a +# declared `Measurement`: 310 makes that status the barcode's, never a sample's, so it is +# computed and carried where the barcode rows are, in emit_verdicts.py, and reaches +# `status_for` under this id. `test_a_measurement_with_a_route_has_a_line_and_a_comparison_...` +# names it as the one exception to "every line backs a declared measurement". +# +# All four of `315`'s inherited lines are in force: `cellBarcodeValidFraction` from the start, +# `aggregateBarcodeFraction` and `undeclaredBarcodeShare` here, and `usableReadFraction` here too. +DEFAULT_LINES: dict[str, Line] = { + # Both thresholds step the same way. This is the line with a real gradient at the far end. + "cellBarcodeValidFraction": Line(warn=0.75, error=0.50), + # One published number gives one boundary, so depth warns and never alerts. + "readsPerCell": Line(warn=5_000), + # 315's published values for the aggregate-barcode read fraction: warn above 0.05, error at + # total failure (1.0). + "aggregateBarcodeFraction": Line(warn=0.05, error=1.0), + # 315's published values for the undeclared-barcode read fraction, read direct rather than + # as a complement: warn above 0.50, error at total failure (1.0). + "undeclaredBarcodeShare": Line(warn=0.5, error=1.0), + # 315's published values for the usable antigen-read fraction: warn below 0.20, error at + # total failure (0.0). + "usableReadFraction": Line(warn=0.20, error=0.0), +} + +# How each line is read. Deliberately *not* overridable: an operator moves a number, never +# a direction. +# +# at-least acceptable at or above the line, alerting strictly below +# at-most acceptable at or below the line, alerting strictly above +# alerting-at alerting where the value equals the line +# +# In every case the named value satisfies the condition it names. +# +# How each line's thresholds are read. Deliberately *not* in DEFAULT_LINES: an operator moves +# a number, never a direction. +# +# at-least OK at or above the threshold, bad strictly below +# at-most OK at or below the threshold, bad strictly above +# alerting-at bad where the value equals the threshold +# +# In every case the named value satisfies the condition it names. +# +# The two thresholds of one line are read INDEPENDENTLY, because `315-where-the-lines-come-from` +# reads them that way. Three of the four inherited lines warn on a direction and put error at +# total failure -- "at 0", "at 1.0" -- which is `alerting-at` and not a further step along the +# warn direction. Only the fourth, barcode validity, steps the same way twice: warn below 0.75, +# error below 0.50. One direction per measurement collapsed those into one, and a fraction whose +# error sits "at 0" could then never alert, since nothing is below zero. +# +# The second entry is None where the line published no error threshold. +# +# `at-most` reads both `aggregateBarcodeFraction` and `undeclaredBarcodeShare`: a high share +# is the bad direction for each, unlike barcode validity, which reads at-least. `usableReadFraction` +# also reads at-least -- a low share is the bad direction -- but its error sits at total failure +# (0.0) rather than stepping further along the warn direction, unlike barcode validity. +_COMPARISON: dict[str, tuple[str, str | None]] = { + "cellBarcodeValidFraction": ("at-least", "at-least"), + "readsPerCell": ("at-least", None), + # Error at total failure (`alerting-at` 1.0) rather than a further step past warn: `315` + # puts every share it inherits at either "at least" or "at most" with error at the + # catastrophe end, and this is one of the two upward-facing members of that set. + "aggregateBarcodeFraction": ("at-most", "alerting-at"), + "undeclaredBarcodeShare": ("at-most", "alerting-at"), + # Error at total failure (`alerting-at` 0.0), the downward-facing member of that same set. + "usableReadFraction": ("at-least", "alerting-at"), +} + + +def _breaches(value: float, threshold: float, comparison: str) -> bool: + """Whether a value falls the wrong side of one threshold.""" + if comparison == "at-least": + return value < threshold + if comparison == "at-most": + return value > threshold + return value == threshold + + +_ORDINAL = {Status.OK: 0, Status.WARN: 1, Status.ALERT: 2} + +_DEFERRED: frozenset[str] = frozenset(m.id for m in MEASUREMENTS if m.deferred_reason) + + +def is_computed(value: float | None) -> bool: + """Whether a number came back at all. + + A non-finite value counts as absent. Every `<` and `>` against NaN is False, so treating + it as a number let it fall through to the acceptable branch -- corrupt input reading + green, the one status a reader will not investigate. +inf read green too against an + at-least line, and -inf happened to alert. One rule for "not a finite number" is easier + to defend than a rule whose answer depends on the sign. + """ + return value is not None and math.isfinite(value) + + +def status_for(measurement: str, value: float | None, lines: dict[str, Line]) -> Status | None: + """How one measurement reads, given the lines in force. None where no line stands behind it. + + Three answers and no fourth. A deferred measurement, a measurement with no number, and a + measurement with no line all carry no status -- and which of those happened is read from + the value, not from a fourth word in this column. + + A deferred measurement carries none whatever it is handed: nothing computes it, so a + value reaching here is a caller's mistake and must not be laundered into a judgement + about the run. + + The categorical route is read before `lines`: its fact is not a threshold, so neither + `DEFAULT_LINES` nor `_COMPARISON` carries an entry for it. Zero alerts; any other finite + value -- the count is never negative -- reads OK and claims nothing about how many cells + the sample should have yielded, a number nobody has published. + """ + if measurement in _DEFERRED or not is_computed(value): + return None + if measurement in _CATEGORICAL: + return Status.ALERT if value == 0 else Status.OK + if measurement not in lines: + return None + line = lines[measurement] + warn_comparison, error_comparison = _COMPARISON[measurement] + # Error first, so a value past both boundaries reads alert rather than warn. Where the line + # published no error threshold the measurement warns and never alerts, whatever its value. + if line.error is not None and error_comparison is not None and _breaches(value, line.error, error_comparison): + return Status.ALERT + if _breaches(value, line.warn, warn_comparison): + return Status.WARN + return Status.OK + + +def roll_up(readings: list[Reading]) -> Coverage: + """The worst status among those that carry one, plus coverage. + + Coverage stays out of the ordinal because a status and a non-status answer different + questions. The first says whether something is wrong, the second whether anybody looked. + Ranked on one scale, an unchecked run becomes indistinguishable from a checked one. + + A level with nothing judged carries no status. Given one of OK the run would look + checked, and given one of alert a scientist would chase a problem that does not exist. + """ + judged = [r.status for r in readings if r.status is not None] + unjudged = sum(1 for r in readings if r.status is None and is_computed(r.value)) + not_evaluated = sum(1 for r in readings if r.status is None and not is_computed(r.value)) + status = max(judged, key=lambda s: _ORDINAL[s]) if judged else None + return Coverage(status, len(judged), unjudged, not_evaluated) + + +# Only the sample rolls up, so `roll_up` above is the only aggregation rule here. +# +# A panel status is gone because it overestimated what could be judged categorically: of +# the per-tag measurements one is categorical and the rest are read only as outliers +# against the other tags in the same panel, which is a comparison rather than a severity +# and cannot be rolled into one without discarding what made it a finding. A capture status +# followed the same logic -- the worst of every sample and every panel becomes the worst of +# every sample, which the samples already say. +# +# Nothing hides. A reagent finding states itself on its own per-tag row, keyed by the panel +# that has it, and a sample's own report names the measurement that set it alerting. + + +def measurement_row(m: Measurement) -> dict: + """One declared measurement, rendered for a reader who never opens this module. + + A deferred measurement renders with its reason attached and keeps its place in the set. + The difference between "checked and fine" and "never checked" is lost the moment a + deferred id simply has no row. + """ + return { + "id": m.id, + "label": m.label, + "level": m.level, + "counts": m.counts, + "implies": m.implies, + # No status, ever. A declaration is not a reading, and a deferred measurement carries + # its reason in place of a number rather than a fourth status word. + "status": None, + "reason": m.deferred_reason, + } + + +def measurement_rows() -> list[dict]: + """Every declared measurement, deferred ones included, in declaration order.""" + return [measurement_row(m) for m in MEASUREMENTS] + + +def per_antigen_measures( + counts: pl.DataFrame, + states: pl.DataFrame, + declared_tags: Collection[str], + panel_samples: Collection[str], + reference_tags: Collection[str] = (), +) -> pl.DataFrame: + """Per tag: cells with any count, cells called bound, and the median count per cell. + + Grouped by tag, not identity: a tag's own reagent behaviour is the question, and an + identity built from several tags would let one weak tag hide behind a stronger one. + + Two frames, and which one each column comes from is the whole point. + + `counts` is the RAW sparse frame, before the minimum -- one row per (sampleId, cellId, + tag) with `umiCount`, reference tags included. `states` is the tag-grain frame after the + minimum, with `tag` and `state`. + + `cellsWithCount` and `medianCountPerCell` come from `counts`, `cellsAboveTheLine` from + `states`. 330-the-quality-readout fixes that split: the first measures what the reagent + delivered and the second what survived the minimum, so a reagent putting two counts into + every cell reads as delivering something rather than as delivering nothing. A median + below the minimum is that same finding and not an error. + + The median is taken over every cell holding a count. Taken over bound cells it could only + ever print a number above the cutoff's floor, because clearing the cutoff is what bound + means, so a half-degraded reagent would show a healthy figure computed from the few cells + that scraped over. It also then depends on no threshold, which matters on a first run + where the cutoff is still being settled -- this is a page read in order to settle it. + + One row per declared tag, whether or not the reads ever show it. A dead reagent is read + as a zero under cells-with-count, and a tag with no row at all offers nothing to read. + + `samplesSeenIn` counts distinct samples carrying any count of the tag, and `samplesInPanel` + is the denominator. A tag absent from every sample reads 0. + + `panel_samples` is the panel's declared roster and supplies the seen-in denominator. It is the + roster rather than the samples present in `counts`, which omits a sample that contributed no + rows. + + `samplesInPanelNames` and `samplesSeenInNames` carry the same two groups as sample ids + rather than counts, sorted. `samplesInPanelNames` is `panel_samples` itself, so a caller + passing a subset of a run's samples for one tag has that subset named here, not the whole + run. `samplesSeenInNames` is `[]`, never null, for a tag with `samplesSeenIn == 0`. + + Reference tags keep a row and carry `cellsAboveTheLine` as None. They are held out of the + verdict read, so no state exists for them -- and a blank and a zero are opposite findings + here. Their median is the run's ambient floor, which is why they belong in this table. + """ + references = sorted(set(reference_tags)) + spine = pl.DataFrame( + {"tag": sorted(set(declared_tags) | set(references))}, + schema={"tag": pl.Utf8}, + ) + + delivered = ( + counts.filter(pl.col("umiCount") > 0) + .group_by("tag") + .agg( + pl.len().alias("cellsWithCount"), + pl.col("umiCount").median().alias("medianCountPerCell"), + pl.col("sampleId").n_unique().alias("samplesSeenIn"), + pl.col("sampleId").unique().sort().alias("samplesSeenInNames"), + ) + ) + declared_names = sorted(set(panel_samples)) + samples_in_panel = len(declared_names) + bound = states.group_by("tag").agg((pl.col("state") == "bound").sum().alias("cellsAboveTheLine")) + + is_reference = pl.col("tag").is_in(references) if references else pl.lit(False) # noqa: FBT003 + return ( + spine.join(delivered, on="tag", how="left") + .join(bound, on="tag", how="left") + .with_columns( + pl.col("cellsWithCount").fill_null(0).cast(pl.Int64), + pl.col("samplesSeenIn").fill_null(0).cast(pl.Int64), + pl.col("samplesSeenInNames").fill_null([]), + pl.lit(samples_in_panel, dtype=pl.Int64).alias("samplesInPanel"), + pl.lit(declared_names, dtype=pl.List(pl.Utf8)).alias("samplesInPanelNames"), + pl.when(is_reference) + .then(pl.lit(None, dtype=pl.Int64)) + .otherwise(pl.col("cellsAboveTheLine").fill_null(0).cast(pl.Int64)) + .alias("cellsAboveTheLine"), + ) + .sort("tag") + ) + + +def sibling_disagreement( + states: pl.DataFrame, + tags_by_identity: dict[str, list[str]], +) -> dict[str, float | None]: + """Per tag: the share of its judged cells contradicting the majority of its siblings. + + Siblings are the OTHER tags the same identity carries. + + A tag is excluded from the majority it is judged against. + + Judged within one cell, over the siblings holding an explicit row there. A tag with no + row in a cell does not vote: `states` is sparse and carries no silent cell. + + A majority is strict -- more than half the sibling votes. A cell whose siblings reach no + strict majority does not judge that tag and is not counted. Two siblings need at least + three tags on the identity, so a two-tag identity always has a majority of one. + + None, never zero, where nothing judged the tag. Three causes reach it and the caller + tells them apart: an identity carrying one tag, a tag holding no row in any cell, and a + tag whose siblings reached a majority in no cell it held. + + `states` is the tag-grain frame after the minimum, with `sampleId`, `cellId`, `tag` and + `state`. + """ + rates: dict[str, float | None] = {} + for tags in tags_by_identity.values(): + members = sorted(set(tags)) + if len(members) < 2: + for tag in members: + rates[tag] = None + continue + + here = states.filter(pl.col("tag").is_in(members)).select("sampleId", "cellId", "tag", "state") + for tag in members: + mine = here.filter(pl.col("tag") == tag).select("sampleId", "cellId", "state") + counted = here.filter(pl.col("tag") != tag).group_by("sampleId", "cellId", "state").agg(pl.len().alias("n")) + totals = counted.group_by("sampleId", "cellId").agg(pl.col("n").sum().alias("total")) + # Filtered on the strict majority rather than on the largest count, so at most + # one state survives per cell and no row ordering decides which. A cell with no + # sibling row, or with its siblings tied, produces no row here and drops out. + majority = ( + counted.join(totals, on=["sampleId", "cellId"], how="inner") + .filter(pl.col("n") * 2 > pl.col("total")) + .select("sampleId", "cellId", pl.col("state").alias("majority")) + ) + judged = mine.join(majority, on=["sampleId", "cellId"], how="inner") + if judged.height == 0: + rates[tag] = None + continue + rates[tag] = judged.filter(pl.col("state") != pl.col("majority")).height / judged.height + return rates + + +def reads_per_cell(reads_matched: int, cells_in_list: int) -> float | None: + """Reads matched, over cells in the cell list. + + The denominator is the **cell list**, not the barcodes the reads happened to touch. The + vendor's five-thousand recommendation is per called cell, and in droplet data the + observed-barcode count runs one to two orders of magnitude higher, because ambient + antigen reads land on most barcodes. Dividing by observed barcodes would make a healthy + library alert -- worse than not judging depth at all, since a status that fires on good + runs teaches a reader to ignore it. + + `reads_matched` already exists in the per-sample QC row. The cell list arrives with gene + expression or with the receptors, so the caller supplies its size. Deliberately not + `cellsDetected` from that row -- that is the observed-barcode count warned against here. + + None when the cell list is empty. A rate over no cells is not a small number, it is no + number, and None keeps that distinct from a computed rate that happens to be zero. + """ + if cells_in_list <= 0: + return None + return reads_matched / cells_in_list + + +def usable_read_fraction( + tag_stat: pl.DataFrame, + cell_col: str, + listed_cells: Collection[str] | None, + reads_total: int | None, +) -> tuple[float | None, str]: + """Reads landing on a called cell, recognised against the panel, over `reads_total`. + + Ports Cell Ranger's `frac_feature_reads_usable` (Cell Ranger `main`, + `lib/python/cellranger/rna/report_matrix.py`, `_report_genome_agnostic_metrics`): + conf-mapped, barcoded reads restricted to the called-cell partition, over the whole + library's read count. UMI validity is that source's separate `good_umi_frac` figure and + is not part of this one. + + `tag_stat` is the post-refine tag-stat table, one row per (cell, feature barcode) + surviving refine-tags -- every FEATURE value outside the panel is already gone by + construction, which is the recognition condition. `totalWeight` is its read-weight + column. Restricting `cell_col` to `listed_cells` is the called-cell condition; + `listed_cells` is the sample's own cell list, not the barcodes the reads happened to + touch. + + Returns `(None, reason)` where `listed_cells` is None -- the called-cell condition + cannot be evaluated with no cell list -- or where `reads_total` is absent or zero, since + the fraction then has no denominator. An empty (non-None) cell list still returns 0.0: + no read landing on a called cell is a real finding, not a missing input. + """ + if listed_cells is None: + return None, "no cell list supplied, so the called-cell condition cannot be evaluated" + if not reads_total: + return None, "no total read count to divide by" + usable = float(tag_stat.filter(pl.col(cell_col).is_in(list(listed_cells)))["totalWeight"].sum()) + return usable / reads_total, f"cellsInList={len(listed_cells)}" + + +# Cell Ranger's own constants for the ANTIGEN branch of `detect_outlier_umis_bcs` +# (`lib/python/cellranger/feature/antibody/analysis.py`): a 3x interquartile multiplier over +# the top 100 barcodes by count, and a 1000-UMI floor below which nothing is flagged. +AGGREGATE_BARCODE_IQR_MULTIPLIER: float = 3.0 +AGGREGATE_BARCODE_MIN_THRESHOLD: float = 1000.0 +AGGREGATE_BARCODE_TOP_N: int = 100 + + +def detect_aggregate_barcodes( + per_barcode: pl.DataFrame, + multiplier: float = AGGREGATE_BARCODE_IQR_MULTIPLIER, + min_umi_threshold: float = AGGREGATE_BARCODE_MIN_THRESHOLD, + top_n: int = AGGREGATE_BARCODE_TOP_N, +) -> tuple[frozenset[str], float | None]: + """Barcodes whose antigen UMI count is an outlier, and the threshold that decided it. + + Ports `detect_outlier_umis_bcs` (Cell Ranger `main`, + `lib/python/cellranger/feature/antibody/analysis.py`), called for the ANTIGEN library type + from `cell_calling_helpers.py::remove_antibody_antigen_aggregates`. + + `per_barcode` has one row per observed barcode, columns `barcode` and `umiCount` -- the + whole whitelist-corrected barcode universe, not the cell list. q1 and q3 are taken over + the top `top_n` barcodes by count (or however many exist, below `top_n`), and a flagged + barcode must be IN that top slice: one outside it is never flagged however large its + count, which is a property of the source and not a choice made here. + + Returns an empty set and the computed threshold where the threshold falls under + `min_umi_threshold`, the source's own floor. Returns an empty set and `None` where + `per_barcode` holds no row at all, since no quantile exists over nothing. + """ + if per_barcode.height == 0: + return frozenset(), None + top = per_barcode.sort("umiCount", descending=True).head(top_n) + counts = top["umiCount"].to_numpy().astype(float) + q1 = float(np.quantile(counts, 0.25)) + q3 = float(np.quantile(counts, 0.75)) + threshold = q3 + (q3 - q1) * multiplier + if threshold < min_umi_threshold: + return frozenset(), threshold + flagged = top.filter(pl.col("umiCount") >= threshold)["barcode"].to_list() + return frozenset(flagged), threshold + + +def aggregate_barcode_fraction( + per_barcode: pl.DataFrame, + reads_total: int | None, + multiplier: float = AGGREGATE_BARCODE_IQR_MULTIPLIER, + min_umi_threshold: float = AGGREGATE_BARCODE_MIN_THRESHOLD, + top_n: int = AGGREGATE_BARCODE_TOP_N, +) -> tuple[float | None, str]: + """Reads in barcodes `detect_aggregate_barcodes` flags, over `reads_total`. + + `per_barcode` carries `barcode`, `umiCount` (what detection runs on) and `readCount` + (what the fraction's numerator sums) -- the source's ANTIGEN-branch numerator is reads, + not UMIs, for the flagged barcodes. + + `reads_total` is the whole-library, pre-match read count (mitool's parse-report `total`), + matching the source's undivided-by-matching denominator. Returns `(None, reason)` where + `reads_total` is absent or zero, since a fraction has no denominator there. + + Otherwise always returns a number, never a bare blank: where the floor in + `detect_aggregate_barcodes` suppresses every flag, the fraction is 0.0 and the detail + states the computed threshold and the floor, so a zero here is never silent. + """ + if not reads_total: + return None, "no total read count to divide by" + flagged, threshold = detect_aggregate_barcodes( + per_barcode.select("barcode", "umiCount"), multiplier, min_umi_threshold, top_n + ) + tested = min(per_barcode.height, top_n) + if threshold is None: + detail = "no antigen barcode observed in this sample" + elif threshold < min_umi_threshold: + detail = ( + f"barcodesTested={tested}|threshold={threshold:.1f} " + f"(below the {min_umi_threshold:.0f}-UMI floor, no barcode flagged)" + ) + else: + detail = f"barcodesTested={tested}|threshold={threshold:.1f}|barcodesFlagged={len(flagged)}" + flagged_reads = per_barcode.filter(pl.col("barcode").is_in(flagged))["readCount"].sum() if flagged else 0 + return flagged_reads / reads_total, detail + + +# The extremes are included alongside the interior deciles so the distribution's edges are +# visible, not only its middle: eleven points, 0 through 100 by 10. +DECILE_POINTS: tuple[int, ...] = tuple(range(0, 101, 10)) + + +def deciles_of(values: np.ndarray) -> pl.DataFrame: + """The eleven decile points of `values`, or eleven unanswered points where there are none. + + Split out of `antigen_count_deciles` so a second spread reports the same shape. An empty + input still returns all eleven rows with a null value: no observations is eleven declared, + unanswered points, never an empty frame. + """ + if values.size == 0: + return pl.DataFrame( + {"decile": list(DECILE_POINTS), "value": [None] * len(DECILE_POINTS)}, + schema={"decile": pl.Int64, "value": pl.Float64}, + ) + return pl.DataFrame( + {"decile": list(DECILE_POINTS), "value": [float(np.quantile(values, p / 100)) for p in DECILE_POINTS]} + ) + + +def antigen_count_deciles(counts: pl.DataFrame) -> pl.DataFrame: + """Deciles of the total antigen count per cell barcode. + + `counts` is the sparse per-(cell, tag) frame -- one row per observed reading, columns + sampleId, cellId, umiCount -- taken before flooring or identity-combining. A cell's + total sums every tag it shows any reading for. A cell with no row contributes no total, + since crediting it zero would read as a reading rather than the absence it is. + + Returns one row per decile point, columns `decile` and `value`. An empty input still + returns all eleven rows with `value` null: no cells observed is eleven declared, + unanswered points, never an empty frame -- the same "declared, not absent" rule the + deferred measurements follow. + """ + if counts.height == 0: + return deciles_of(np.empty(0)) + totals = counts.group_by(["sampleId", "cellId"]).agg(pl.col("umiCount").sum().alias("total"))["total"].to_numpy() + return deciles_of(totals) diff --git a/software/per-cell-metrics/src/qc_report.py b/software/per-cell-metrics/src/qc_report.py index aede0b4..ee35914 100644 --- a/software/per-cell-metrics/src/qc_report.py +++ b/software/per-cell-metrics/src/qc_report.py @@ -1,10 +1,11 @@ """Per-sample QC summary for the Feature Integration block. -One row per sample: read-level metrics from mitool's parse JSON report (parseReport.total/.matched), -cell/feature/UMI metrics from the tag-stat TSV, and the panel-assigned fraction from the refine-tags -JSON report (the FEATURE correction step's outputCount / inputCount — the fraction of reads kept after -correcting the feature barcode against the panel whitelist). panelAssignedFraction is left blank only -when no refine report is available. Stdlib + polars only. +One row per sample: read-level metrics from mitool's parse JSON report +(parseReport.total/.matched), cell/feature/UMI metrics from the tag-stat TSV, the +panel-assigned fraction from the refine-tags JSON report, and the aggregate-barcode read +fraction (`qc_measures.detect_aggregate_barcodes`) computed from the tag-stat TSV's +per-barcode UMI and read totals. panelAssignedFraction is left blank only when no refine +report is available. Stdlib and polars only. """ import argparse @@ -13,6 +14,12 @@ import sys import polars as pl +from qc_measures import ( + AGGREGATE_BARCODE_IQR_MULTIPLIER, + AGGREGATE_BARCODE_MIN_THRESHOLD, + AGGREGATE_BARCODE_TOP_N, + detect_aggregate_barcodes, +) FIELDNAMES = [ "sampleId", @@ -24,6 +31,10 @@ "totalUniqueUmis", "medianUmisPerCell", "panelAssignedFraction", + "cellBarcodeValidFraction", + "aggregateBarcodeFraction", + "aggregateBarcodesFlagged", + "aggregateBarcodeThreshold", ] @@ -34,17 +45,16 @@ def _parse_report(path: str) -> tuple[int, int]: return int(pr.get("total", 0)), int(pr.get("matched", 0)) -def _refine_assigned_fraction(path: str | None, feature_tag: str = "FEATURE") -> float | None: - """Panel-assigned fraction from the refine-tags JSON report. +def _refine_kept_fraction(path: str | None, tag_name: str = "FEATURE") -> float | None: + """The share of reads one refine-tags step kept, as ``outputCount / inputCount``. - The feature refine step corrects each feature barcode against the panel whitelist and drops reads - whose barcode is not within correction distance of any panel entry. The panel-assigned fraction is - that step's ``outputCount / inputCount`` — the fraction of reads entering feature correction that - were kept (assigned to a panel feature). ``feature_tag`` is the mitool tag name to match against the - report's per-step ``tagName``. + Each step corrects one tag's barcode against a whitelist and drops reads whose barcode is + not within correction distance of any entry. Which whitelist depends on the tag: the + FEATURE step corrects against the panel, the CELL step against the chemistry's barcodes. + ``tag_name`` is the mitool tag to match against the report's ``tagName``. - Returns None (blank in the CSV) when the report is absent/unreadable, carries no matching step, or - that step has zero input reads, so QC never crashes on a missing or edge-case report. + Returns None, blank in the CSV, when the report is absent or unreadable, carries no + matching step, or that step has zero input reads. QC never crashes on an edge-case report. """ if not path: return None @@ -55,22 +65,57 @@ def _refine_assigned_fraction(path: str | None, feature_tag: str = "FEATURE") -> return None steps = rep.get("steps", []) for step in steps: - if step.get("tagName") == feature_tag: + if step.get("tagName") == tag_name: input_count = step.get("inputCount", 0) if not input_count: return None return step.get("outputCount", 0) / input_count - # A report with steps but none matching the feature tag means the report schema or tag naming - # drifted from what we expect — surface it rather than silently blanking the metric for every sample. + # A report with steps but none matching the feature tag means the schema or tag naming + # drifted. Surface it rather than silently blanking the metric for every sample. if steps: print( - f"[qc-report] refine report has no {feature_tag!r} step " + f"[qc-report] refine report has no {tag_name!r} step " f"(saw tags {[s.get('tagName') for s in steps]}); panel-assigned fraction left blank", file=sys.stderr, ) return None +def _aggregate_barcode_metrics( + stat: pl.DataFrame, + cell_col: str, + umi_col: str, + count_col: str, + reads_total: int, + multiplier: float = AGGREGATE_BARCODE_IQR_MULTIPLIER, + min_umi_threshold: float = AGGREGATE_BARCODE_MIN_THRESHOLD, + top_n: int = AGGREGATE_BARCODE_TOP_N, +) -> tuple[float | None, int, float | None]: + """Fraction of reads_total sitting in barcodes `detect_aggregate_barcodes` flags. + + Per-barcode UMI and read totals from the whole whitelist-corrected barcode universe + (`stat`, not the cell list) feed `detect_aggregate_barcodes`. Returns `(None, 0, None)` + only where `reads_total` is falsy, since a fraction has no denominator there; otherwise + the fraction is always a number, 0.0 where nothing is flagged, so a run that checked and + found no aggregate reads is never indistinguishable from one that never checked. + """ + if not reads_total: + return None, 0, None + per_barcode = ( + stat.group_by(cell_col) + .agg( + pl.col(umi_col).sum().alias("umiCount"), + pl.col(count_col).sum().alias("readCount"), + ) + .rename({cell_col: "barcode"}) + ) + flagged, threshold = detect_aggregate_barcodes( + per_barcode.select("barcode", "umiCount"), multiplier, min_umi_threshold, top_n + ) + flagged_reads = per_barcode.filter(pl.col("barcode").is_in(flagged))["readCount"].sum() if flagged else 0 + return flagged_reads / reads_total, len(flagged), threshold + + def main() -> None: p = argparse.ArgumentParser() p.add_argument("tag_stat_tsv") @@ -80,23 +125,45 @@ def main() -> None: p.add_argument("--cell-col", default="CELL") p.add_argument("--feature-col", default="FEATURE") p.add_argument("--umi-col", default="unique_UMI") + p.add_argument("--count-col", default="count") + # The aggregate-barcode detection knobs. Defaults mirror qc_measures.py's own constants. + # Moving any of them changes which barcodes `detect_aggregate_barcodes` flags. + p.add_argument("--aggregate-iqr-multiplier", type=float, default=AGGREGATE_BARCODE_IQR_MULTIPLIER) + p.add_argument("--aggregate-min-umi-threshold", type=float, default=AGGREGATE_BARCODE_MIN_THRESHOLD) + p.add_argument("--aggregate-top-n", type=int, default=AGGREGATE_BARCODE_TOP_N) p.add_argument("--output", default="result_qc.csv") args = p.parse_args() total, matched = _parse_report(args.parse_report) stat = pl.read_csv(args.tag_stat_tsv, separator="\t") - # A header-only tag-stat (a sample whose reads were all dropped -- e.g. every read off-panel) has no - # data rows, so polars infers every column as String. Coerce the UMI-count column to a numeric type - # up front, otherwise .sum()/.median() below raise on String arithmetic. On a populated file the - # column is already integer and this cast is a no-op. Mirrors per_cell_metrics._load. - stat = stat.with_columns(pl.col(args.umi_col).cast(pl.Int64)) + # A header-only tag-stat (every read dropped) has no data rows, so polars infers String + # for every column. Coerce here, or .sum()/.median() below raise on String arithmetic. + # Mirrors per_cell_metrics._load. + stat = stat.with_columns( + pl.col(args.umi_col).cast(pl.Int64), + pl.col(args.count_col).cast(pl.Int64), + ) cells = int(stat[args.cell_col].n_unique()) features = int(stat[args.feature_col].n_unique()) total_umis = int(stat[args.umi_col].sum()) per_cell = stat.group_by(args.cell_col).agg(pl.col(args.umi_col).sum().alias("u")) median_umis = float(per_cell["u"].median()) if per_cell.height else 0.0 - assigned = _refine_assigned_fraction(args.refine_report, args.feature_col) + assigned = _refine_kept_fraction(args.refine_report, args.feature_col) + # The same report's CELL step. It corrects each cell barcode against the chemistry's + # whitelist rather than against the panel, so its kept share is the share of reads whose + # barcode the chemistry could have produced. + cell_valid = _refine_kept_fraction(args.refine_report, args.cell_col) + agg_fraction, agg_flagged, agg_threshold = _aggregate_barcode_metrics( + stat, + args.cell_col, + args.umi_col, + args.count_col, + total, + args.aggregate_iqr_multiplier, + args.aggregate_min_umi_threshold, + args.aggregate_top_n, + ) row = { "sampleId": args.sample_id, @@ -108,14 +175,18 @@ def main() -> None: "totalUniqueUmis": total_umis, "medianUmisPerCell": median_umis, "panelAssignedFraction": "" if assigned is None else assigned, + "cellBarcodeValidFraction": "" if cell_valid is None else cell_valid, + "aggregateBarcodeFraction": "" if agg_fraction is None else agg_fraction, + "aggregateBarcodesFlagged": agg_flagged, + "aggregateBarcodeThreshold": "" if agg_threshold is None else agg_threshold, } with open(args.output, "w", newline="") as out: w = csv.DictWriter(out, fieldnames=FIELDNAMES) w.writeheader() w.writerow(row) - # Also emit the row as JSON so the model can read per-sample QC (getDataAsJson) to build the block's - # live "Analysis logs" — the per-sample completed count (heartbeat) and the run-level summary. + # Also emit the row as JSON so the model can read per-sample QC (getDataAsJson) to build + # the block's live "Analysis logs": the per-sample completed count and the run summary. with open("result_qc.json", "w") as jf: json.dump(row, jf) diff --git a/software/per-cell-metrics/src/qc_rows.py b/software/per-cell-metrics/src/qc_rows.py new file mode 100644 index 0000000..3da45ff --- /dev/null +++ b/software/per-cell-metrics/src/qc_rows.py @@ -0,0 +1,455 @@ +"""The QC measurement set as rows, and the two report shapes read off them. + +`QcRow` is the one carrier: every measurement enters through `_add`, which is what attaches +the line's verdict. `_qc_frame` is the long form keyed (level, entity, measurement). +`sample_report_rows` and `sample_summary_rows` are the two narrower reads over the same rows, +and neither recomputes a status. + +A row with no number always carries a reason, and the reason is not the detail: a detail rides +alongside a number, a reason stands in place of one. +""" + +from __future__ import annotations + +from collections.abc import Collection +from typing import NamedTuple + +import numpy as np +import polars as pl +from qc_measures import ( + DEFAULT_LINES, + MEASUREMENTS, + Coverage, + Line, + Measurement, + Reading, + Status, + deciles_of, + is_computed, + roll_up, + status_for, +) +from tag_distribution import TagFits +from verdict import ReferenceChoice, specificity_score + +# A rollup is reported in the same frame as the measurements it aggregates, as a row whose +# measurement is the rollup itself. A measurement is an axis value here, so a level's +# summary costs a row rather than a column. +ROLLUP = "rollup" +ROLLUP_COUNTS = "The worst status among this level's measurements, and how much of it was checked." +# The rollup has no declaration to borrow a readable name from, and a row reading `rollup` +# beside rows reading `readsPerCell` leaves the reader guessing which is which. +ROLLUP_LABEL = "Worst status at this level" + +MEASUREMENT_BY_ID = {m.id: m for m in MEASUREMENTS} + + +class QcRow(NamedTuple): + """One measurement at one level entity, before its declaration is attached. + + `status` and `coverage` are both carried because a measurement's own status is not + recoverable from a coverage triple: `roll_up` reports *not evaluated* for a level with + nothing judgeable in it, so a row computed and left unjudged would come back saying + nobody looked. The triple says how much was checked. The status says whether what was + checked is wrong. + + `panel_id` is set on tag-level and identity-level rows and left empty on the rest. A + panel carries the worst status among those measurements, so those rows have to say + which panel they belong to. + + `reason` is set only where `value` is None, and it is separate from `detail`: a detail + is carried alongside a number, a reason stands in place of one. + """ + + level: str + entity: str + measurement: str + value: float | None + detail: str + panel_id: str + status: Status | None + coverage: Coverage + reason: str = "" + + +def _leaf(level, entity, measurement, value, detail, panel_id, status: Status | None, reason: str = "") -> QcRow: + """One measurement's row: its own status, and the coverage of that one status. + + The triple comes from `roll_up`, so a leaf and a rollup are counted by one rule, and the + row keeps the status `roll_up` would have flattened. + """ + reading = Reading(status, value) + return QcRow(level, entity, measurement, value, detail, panel_id, status, roll_up([reading]), reason) + + +def _qc_frame(rows: list[QcRow], lines: dict[str, Line] = DEFAULT_LINES) -> pl.DataFrame: + """The measurement set as a frame keyed (level, entity, measurement). + + Every declared measurement keeps its place whether or not this run could compute it, and + a measurement nothing computed reads *not evaluated* with its reason rather than being + absent. A reader must never mistake "nothing computed this yet" for "checked and found + fine". A field with nothing in it is written null rather than as an empty string: polars + quotes an empty string to keep it apart from a null, and a quoted empty cell is a value a + downstream import would carry as one. + + `lineWarn` and `lineAlert` are read from `lines`, the same dict `status_for` was given to + produce `row.status` -- so a reader who sees `warn` can see the threshold it warned + against. `route` is read from `Measurement.line` alone, never a second declaration of which + measurement stands on a line. A measurement with no route gets three null fields; the + categorical route has a route but no numeric threshold, so it gets a route and two nulls. + """ + built = [] + for row in rows: + declared = MEASUREMENT_BY_ID.get(row.measurement) + route = None if declared is None else declared.line + line = None if declared is None else lines.get(row.measurement) + built.append( + { + "level": row.level, + "entity": row.entity, + "panelId": row.panel_id, # "" not None: this is an AXIS key, and a null is not a usable one + "measurement": row.measurement, + # The readable name, carried beside the id rather than instead of it. The id + # is a p-column axis value and must stay stable. The label is what a reader + # who never opened this module sees. + "label": ROLLUP_LABEL if declared is None else declared.label, + "value": row.value, + "detail": row.detail or None, + # Null where no line stands behind the measurement. The reason is read from the + # value, which is where a reader looks next anyway. + "status": None if row.status is None else row.status.value, + "judged": row.coverage.judged, + "unjudged": row.coverage.unjudged, + "notEvaluated": row.coverage.not_evaluated, + "counts": ROLLUP_COUNTS if declared is None else declared.counts, + "implies": None if declared is None else declared.implies, + "lineWarn": None if line is None else line.warn, + "lineAlert": None if line is None else line.error, + "route": route, + # Why this row has no number. The declaration wins: a deferred measurement's + # reason is the same on every run, and a call site cannot restate it. Any other + # row carries its own. Same precedence as `sample_report_rows`. + "reason": (None if declared is None else declared.deferred_reason) or row.reason or None, + } + ) + return pl.DataFrame( + built, + schema={ + "level": pl.String, + "entity": pl.String, + "panelId": pl.String, + "measurement": pl.String, + "label": pl.String, + "value": pl.Float64, + "detail": pl.String, + "status": pl.String, + "judged": pl.Int64, + "unjudged": pl.Int64, + "notEvaluated": pl.Int64, + "counts": pl.String, + "implies": pl.String, + "lineWarn": pl.Float64, + "lineAlert": pl.Float64, + "route": pl.String, + "reason": pl.String, + }, + ) + + +def _add( + rows: list[QcRow], + level: str, + entity: str, + measurement: str, + value, + detail: str = "", + panel_id: str = "", + reason: str = "", + lines: dict[str, Line] = DEFAULT_LINES, +): + """Append one measurement row, taking its status from the lines in force. + + Every declared measurement goes through here. One with no line in force carries no + status, which is honest rather than a refusal: it was computed, no line stands behind it, + so its number is shown and nothing is claimed. + + `lines` defaults to the shipped set; a caller threading an operator override passes its + own dict, which then also governs what `_qc_frame` renders as `lineWarn` / `lineAlert`. + + `reason` belongs on a row with no number and is ignored on any other. A value that is not + a finite number is not a number the caller's reason describes, so it takes its own. + """ + rows.append( + _leaf( + level, + entity, + measurement, + value, + detail, + panel_id, + status_for(measurement, value, lines), + "" if is_computed(value) else (reason if value is None else NOT_A_NUMBER_REASON), + ) + ) + + +# The two standing reasons, for the cases no call site accounts for. +# +# A deferred measurement carries its declaration's reason and needs neither of these. Of the +# rest, every call site that can go valueless states its own, so `UNSUPPLIED_REASON` covers only +# a declared measurement with no call site at all. +UNSUPPLIED_REASON = "nothing in this run supplied a value for this measurement" +# A number arrived and was not finite. Distinct from the above, which is nothing arriving: the +# caller's reason describes an input that is missing and would misname this. +NOT_A_NUMBER_REASON = "this run computed a value for this measurement that is not a finite number" + + +def sample_report_rows(sample: str, rows: list[QcRow]) -> tuple[list[dict], Coverage]: + """One sample's report: every sample-level measurement, and the rollup over them. + + The walk is over `MEASUREMENTS` rather than over `rows`, so the report is the declared set: + a measurement this run never reached takes its place carrying a reason. + + A value that is not a finite number counts as no value, by `is_computed`, which is the rule + the coverage triple counts by. The entry's account of itself and the triple cannot disagree. + + A measurement declaring `rolls_up=False` is listed with its own status and left out of the + rollup. The entry carries `rollsUp`, which is what separates a status shown here from the + status the tag beside the list carries. + """ + by_id = {r.measurement: r for r in rows if r.level == "sample" and r.entity == sample} + entries: list[dict] = [] + readings: list[Reading] = [] + for m in MEASUREMENTS: + if m.level != "sample": + continue + row = by_id.get(m.id) + raw = None if row is None else row.value + value = raw if is_computed(raw) else None + status = None if row is None or value is None else row.status + reason = None + if value is None: + # Declaration first, so a call site cannot restate a deferred measurement's reason. + # Then the row's own, which `_add` has already replaced where a non-finite number + # arrived. `UNSUPPLIED_REASON` is left for a declared measurement with no call site. + reason = m.deferred_reason or (row.reason if row is not None else "") or UNSUPPLIED_REASON + entries.append( + { + "id": m.id, + "label": m.label, + "value": value, + "detail": (row.detail if row is not None else "") or None, + "reason": reason, + "status": None if status is None else status.value, + "counts": m.counts, + "implies": m.implies, + "rollsUp": m.rolls_up, + } + ) + if m.rolls_up: + readings.append(Reading(status, value)) + return entries, roll_up(readings) + + +_DECILE_SCHEMA = {"distribution": pl.String, "decile": pl.Int64, "value": pl.Float64} +# Deciles of the total antigen count per cell barcode, kept PER SAMPLE rather than pooled into +# `_DECILE_SCHEMA`: `330-the-quality-readout` reads this shape as one sample's own plot, and +# pooling it would answer a different question. A separate schema mints a separate p-column +# rather than adding a sample axis to the existing one. +_SAMPLE_DECILE_SCHEMA = {"sampleId": pl.String, "decile": pl.Int64, "value": pl.Float64} +_BACKGROUND_SCHEMA = { + "sampleId": pl.String, + "tag": pl.String, + "backgroundMean": pl.Float64, + "signalMean": pl.Float64, + "backgroundWeight": pl.Float64, +} + +# One row per (panelId, tag, identity). Per-tag figures repeat across a tag's identities: the frame +# is not a summary. `reason` names each figure that has no value and why, pipe-separated, and is +# empty when every figure has one. +_REAGENT_SCHEMA = { + "panelId": pl.String, + "tag": pl.String, + "identity": pl.String, + "samplesSeenIn": pl.Int64, + "samplesInPanel": pl.Int64, + "samplesSeenInNames": pl.String, + "samplesInPanelNames": pl.String, + "cellsWithCount": pl.Int64, + "cellsAboveTheLine": pl.Float64, + "medianCountPerCell": pl.Float64, + "siblingDisagreement": pl.Float64, + "selfDisagreement": pl.Float64, + "reason": pl.String, +} + +# One row per (sampleId, tag) the pre-refine pass saw and the sample's panel does not declare. +# `readShare` and `status` are the SAMPLE's undeclared-read share, repeated on every one of that +# sample's rows -- 330's "the field publishes a line for the share of a sample's reads that land +# in barcodes nobody declared", read together with 310's "its status is the barcode's, and it +# does not become a sample's": the status is computed at the sample and carried on the barcode's +# own row, never on the sample's. Usually there are no rows for a sample at all, which is the +# outcome the field wants. +_UNDECLARED_BARCODE_SCHEMA = { + "sampleId": pl.String, + "tag": pl.String, + "totalWeight": pl.Int64, + "readShare": pl.Float64, + "status": pl.String, +} + + +def _decile_rows(distribution: str, deciles: pl.DataFrame) -> list[dict]: + """One distribution's decile points as rows. A point with no value contributes none.""" + return [ + {"distribution": distribution, "decile": int(d), "value": float(v)} + for d, v in zip(deciles["decile"], deciles["value"], strict=True) + if v is not None + ] + + +def _sample_decile_rows(sample: str, deciles: pl.DataFrame) -> list[dict]: + """One sample's antigen-count decile points as rows. A point with no value contributes none. + + A sample with no counted reading gets all-null points from `antigen_count_deciles`, so this + returns no rows for it -- a sample absent from the frame, not a flat line at zero. + """ + return [ + {"sampleId": sample, "decile": int(d), "value": float(v)} + for d, v in zip(deciles["decile"], deciles["value"], strict=True) + if v is not None + ] + + +def _sticky_measure(readings: dict[tuple[str, str], int], gate: int | None) -> tuple[float | None, str]: + """One sample's sticky exposure, in whichever form the gate allows. + + A declared gate supplies a *high*, so the measurement is a count of the cells at or above + it. With no gate there is no high, and a count against a line nobody drew would assert a + boundary; the spread of the readings goes out instead, which is what a scientist reads in + order to place one. The value is then the median, matching the other spreads here. + """ + comparator_detail = f"cellsWithAComparator={len(readings)}" + if gate is not None: + return float(sum(1 for v in readings.values() if v >= gate)), f"{comparator_detail}|gate={gate}" + deciles = deciles_of(np.asarray(list(readings.values()), dtype=float)) + points = "|".join( + f"{d}:{'' if v is None else round(v, 3)}" for d, v in zip(deciles["decile"], deciles["value"], strict=True) + ) + middle = deciles.filter(pl.col("decile") == 50)["value"].to_list() + return (middle[0] if middle else None), f"{comparator_detail}|noGateDeclared|{points}" + + +def _score_spread(states: pl.DataFrame, served: ReferenceChoice) -> tuple[float | None, str]: + """The run's scores as deciles, or why there are none. + + Only the declared rung scores. A population baseline yields a probability, which is not on + the same scale and cannot be pooled with a score, so under it this does not exist. + + Cells carrying an `unreliableReason` are left out. A cell with no comparator or one a gate + set aside still has a number here -- the score is computed before the state is called -- but + it answers a comparison that never happened. + """ + if served is not ReferenceChoice.DECLARED: + return None, f"the {served.value} baseline yields no score, so a run resting on it has no spread" + scored = states.filter(pl.col("unreliableReason").is_null()) + if scored.height == 0: + return None, "no cell was scored" + values = specificity_score( + scored["umiCount"].to_numpy(), + np.nan_to_num(scored["referenceCount"].cast(pl.Float64).to_numpy(), nan=0.0), + ) + deciles = deciles_of(np.asarray(values, dtype=float)) + detail = "|".join( + f"{d}:{'' if v is None else round(v, 3)}" for d, v in zip(deciles["decile"], deciles["value"], strict=True) + ) + middle = deciles.filter(pl.col("decile") == 50)["value"].to_list() + return (middle[0] if middle else None), detail + + +def _fitted_background(tag_fits: TagFits | None, samples: Collection[str], tag: str) -> tuple[float | None, str]: + """One tag's fitted background across a panel's samples, as a value and its detail.""" + if tag_fits is None: + return None, "no population baseline served this run, so nothing was fitted" + fitted = [tag_fits.backgrounds[(s, tag)] for s in sorted(samples) if (s, tag) in tag_fits.backgrounds] + missed = [s for s in sorted(samples) if (s, tag) in tag_fits.reasons] + if not fitted: + why = tag_fits.reasons.get((missed[0], tag), "") if missed else "this tag was not fitted in any sample" + return None, f"fitted in no sample of this panel: {why}" + means = [b.mean for b in fitted] + detail = "|".join( + [ + f"samplesFitted={len(fitted)}", + f"samplesUnfitted={len(missed)}", + f"backgroundRange={min(means):.4g}..{max(means):.4g}", + f"medianSignalMean={_median([b.signal_mean for b in fitted]):.4g}", + f"medianBackgroundWeight={_median([b.weight for b in fitted]):.4g}", + ] + ) + return _median(means), detail + + +def _median(values: list[float]) -> float | None: + return float(pl.Series(values).median()) if values else None + + +def _number(row: dict, column: str) -> float | None: + """One field of a read-QC row, as a float, or None where it is absent or blank.""" + raw = row.get(column) + if raw is None or str(raw).strip() == "": + return None + return float(raw) + + +# The sample-level measurements, in MEASUREMENTS' own declaration order -- the same walk +# `sample_report_rows` makes, so the wide pivot below reads off one declared set. +_SAMPLE_MEASUREMENTS: tuple[Measurement, ...] = tuple(m for m in MEASUREMENTS if m.level == "sample") + +# Read-QC figures with no declared measurement behind them. `readsTotal`, `panelAssignedFraction`, +# `cellBarcodeValidFraction` and `cellsDetected` are declared measurements already carrying these +# same mitool figures (see `_add` calls above), so they are excluded here and read from the pivot +# instead -- one column per figure, not two agreeing ones under two names. +_MITOOL_ONLY_COLUMNS: tuple[str, ...] = ( + "readsMatched", + "matchedFraction", + "featuresDetected", + "totalUniqueUmis", + "medianUmisPerCell", +) + + +def sample_summary_rows( + samples: list[str], + sample_report: dict[str, dict], + read_qc: dict[str, dict], +) -> pl.DataFrame: + """The across-samples QC table: one row per sample, one column per sample-level measurement. + + Pivots `sample_report` -- the same dict `main` writes to `result_qc_by_sample.json` -- rather + than walking `MEASUREMENTS` a second time, so this table and a sample's own report cannot + disagree about a value or a status. `status` is `sample_report`'s own rollup, from `roll_up`, + never recomputed here. + + Every id in `samples` gets a row, a sample absent from `sample_report` included: its + measurement columns and its status come back null, which reads as nothing having rolled up + rather than as a passing sample. + """ + built = [] + for sample in samples: + report = sample_report.get(sample, {}) + entries = {e["id"]: e["value"] for e in report.get("measurements", [])} + qc = read_qc.get(sample, {}) + row = {"sampleId": sample, "status": report.get("status")} + for col in _MITOOL_ONLY_COLUMNS: + row[col] = _number(qc, col) + for m in _SAMPLE_MEASUREMENTS: + row[m.id] = entries.get(m.id) + built.append(row) + schema = { + "sampleId": pl.String, + "status": pl.String, + **{col: pl.Float64 for col in _MITOOL_ONLY_COLUMNS}, + **{m.id: pl.Float64 for m in _SAMPLE_MEASUREMENTS}, + } + return pl.DataFrame(built, schema=schema) diff --git a/software/per-cell-metrics/src/tag_distribution.py b/software/per-cell-metrics/src/tag_distribution.py new file mode 100644 index 0000000..40a730a --- /dev/null +++ b/software/per-cell-metrics/src/tag_distribution.py @@ -0,0 +1,412 @@ +"""Rung 3: a tag's own count distribution across a sample's cells, split in two. + +The ladder in `what-plays-the-baseline` puts this third: *"that tag's own distribution +across the sample's cells, split into two components, where the sample holds at least +300 cells and the counts actually separate."* It is the only rung with orthogonal +validation behind it and the only one validated at these panel sizes, and it serves a +run with no declared comparator once the panel rung's member condition rules that out. + +**Where the method comes from.** The study the 300-cell figure comes from computes a +kernel density estimate of log2 counts and takes *"the local minimum that optimally +separates the KDE into two populations"*. That paper applies it to cell hashes, not +antigen barcodes -- its own antigen rule is the empty-droplet one, the rung above this, +which needs gene expression this block does not receive. The same paper records why a +Gaussian mixture or k-means is not used: both degrade when the two populations are +unequal in size, which is the antigen case exactly. + +**This returns a comparator, not a classification.** The published use of the split +point is a threshold -- above it, positive. Nothing downstream thresholds. It scores a +reading against a comparator count. So the split point identifies which readings are +background, and the comparator is the middle of those. Reporting the split point itself +would put a classification boundary in a slot every other rung fills with a reading. + +**No normalization.** The paper normalizes by each cell's UMI total. Every reading here +is a raw integer UMI count, and a normalized comparator would be the only non-count in +the pipeline. The cost is that a cell sequenced twice as deeply contributes a reading +twice as large. The split is taken across cells, so this widens both components rather +than shifting one. + +Both gates are gates, not settings: below 300 cells, or with no separation demonstrated, +the baseline this rung would produce is wrong rather than conservative, and the rung +reports itself unavailable. +""" + +from __future__ import annotations + +from typing import NamedTuple + +import numpy as np +import polars as pl +from panel import ANY_SAMPLE +from scipy.special import logsumexp +from scipy.stats import nbinom + +CELL_KEY = ("sampleId", "cellId") + +# Both from the study the method comes from, and both gate rather than tune. +DEFAULT_DISTRIBUTION_MIN_CELLS = 300 + + +class Background(NamedTuple): + """A fitted background, as a reader needs to see it. + + The two means together are the finding. A background alone says nothing about whether the + counts separated, and `330-the-quality-readout` wants this precisely so a scientist can see + whether they did before choosing a baseline. + + `weight` is the share of cells the background component holds. A tag that bound nothing is + split anyway -- the method assumes two components exist -- and shows itself here as two + means sitting almost on top of each other rather than as a refusal. + """ + + mean: float + signal_mean: float + weight: float + + +class TagFit(NamedTuple): + """One tag's per-cell probability of binding in one sample, or why there is none. + + `probabilities` is None exactly when `reason` is not None, and a caller must branch on + that rather than defaulting: a tag whose counts did not separate established NOTHING for + its cells, which is not the same fact as every cell reading a low probability. + + Aligned to the counts the fit was given, one entry per cell in the sample. + + `background` travels with the probabilities and is None on exactly the same condition. The + fit was taken either way, and discarding its parameters left the one number a reader needs + in order to judge the fit inside the function that made it. + """ + + probabilities: np.ndarray | None + reason: str | None + cells: int + background: Background | None = None + + +# The prose a reader sees. Nothing branches on these strings. +TOO_FEW_CELLS = "the sample holds too few cells for a distribution to be fitted" +NO_SEPARATION = "this tag's counts do not separate into two populations" + + +# The share of the highest counts dropped before the fit. `what-plays-the-baseline` fixes it: +# the rule drops "the counts above the 99th percentile" so that a handful of very high +# readings cannot drag the signal component's mean up and pull the boundary with it. The +# dropped cells still get a probability -- they are the most bound cells in the sample, and +# withholding an answer for them would be the opposite of what the trim is for. +_UPPER_TRIM_PERCENTILE = 99.0 + +# EM stops when the mean log-likelihood moves less than this, or after this many rounds. +# Neither number is published. They are convergence controls rather than parameters of the +# method: a run that hit the iteration cap has not been given a different rule, only a +# slightly less settled one. +_EM_TOLERANCE = 1e-6 +_EM_MAX_ROUNDS = 200 + +# The dispersion a component falls back to where its variance does not exceed its mean. The +# negative binomial has no such shape -- that is the Poisson boundary -- so the fit uses a +# large size, which IS Poisson in the limit, rather than failing. +_POISSON_LIMIT_SIZE = 1e6 + +# The smallest mean a component may hold. A component fitted entirely on zeros has a mean of +# zero, and a negative binomial with a mean of zero puts all of its mass on zero -- so every +# non-zero count becomes impossible under it and the fit collapses. A mostly-silent tag is +# the ordinary case here, not an edge one, so the mean is held just above zero instead: a +# spike at zero that still admits the rest of the data. +_MIN_COMPONENT_MEAN = 1e-6 + + +def _nb_size(mean: float, variance: float) -> float: + """The negative binomial's size from a mean and a variance, by moments. + + var = mean + mean^2 / size, so size = mean^2 / (var - mean). Where the variance does not + exceed the mean the distribution is Poisson or narrower and no finite size fits it, so the + Poisson limit stands in. + """ + if mean <= 0.0 or variance <= mean: + return _POISSON_LIMIT_SIZE + return float(mean * mean / (variance - mean)) + + +def _nb_logpmf(counts: np.ndarray, mean: float, size: float) -> np.ndarray: + """The negative binomial log pmf, parameterised by mean and size rather than by p. + + LOG, not the density itself, and that is load-bearing. A binder's count sits far in the + tail of the background component, and far enough that the density underflows to exactly + zero in double precision. Both components then call the count impossible, the + normalisation divides by zero, and the cell most obviously bound in the sample is the one + the fit cannot answer for. In log space the same count is a large negative number and the + comparison between components still holds. + """ + return nbinom.logpmf(counts, size, size / (size + mean)) + + +def _responsibilities(counts: np.ndarray, means: np.ndarray, sizes: np.ndarray, weights: np.ndarray): + """Each count's posterior probability per component, and the mean log-likelihood. + + Softmax over log weights plus log densities, so a count in either tail normalises without + underflowing. Returns None where a log density is not finite, which happens only for a + degenerate component rather than for an extreme count. + """ + logs = np.vstack([np.log(weights[k]) + _nb_logpmf(counts, means[k], sizes[k]) for k in (0, 1)]) + totals = logsumexp(logs, axis=0) + # A single -inf is fine and expected: it says one component calls that count impossible, + # which is what a two-component fit is for. Only a count BOTH reject has no answer, and + # that is a degenerate fit rather than an extreme reading. + if not np.all(np.isfinite(totals)): + return None + return np.exp(logs - totals), float(np.mean(totals)) + + +class _Mixture(NamedTuple): + """A fitted two-component negative binomial, with the signal component identified.""" + + means: np.ndarray + sizes: np.ndarray + weights: np.ndarray + signal: int + + +def _fit_two_component_nb(counts: np.ndarray) -> _Mixture | None: + """A two-component negative binomial fitted to `counts`, or None where none exists. + + `what-plays-the-baseline` fixes the method: fit a two-component negative binomial + mixture and label the higher-median component the signal one. Scoring a cell against the + fit is `_signal_probability`, kept separate because the fit is taken over the trimmed + counts and every cell is scored, including the trimmed ones. + + Returns None where no two-component fit exists -- every count identical, or the two + components converging onto one another. A tag with no binders has one population, and the + honest answer is that this rung established nothing for it rather than a boundary drawn + through the middle of a single mode. + + EM with the dispersions re-estimated by moments each round. The dispersions are not free + parameters of the likelihood here: solving for them jointly needs a numerical root per + component per round, and the moment estimate is stable on integer counts where that is + not. A round that produces a degenerate component ends the fit rather than continuing + from it. + """ + x = counts.astype(float) + if np.unique(x).size < 2: + return None + + # Split at the median to start. Two components initialised on the same statistics never + # separate, and the median is the one split point that is always available. + pivot = float(np.median(x)) + low, high = x[x <= pivot], x[x > pivot] + if low.size == 0 or high.size == 0: + # A median equal to the maximum puts everything in one half. Fall back to splitting + # at the mean, which differs from the median exactly when the counts are skewed -- + # which is this case. + pivot = float(np.mean(x)) + low, high = x[x <= pivot], x[x > pivot] + if low.size == 0 or high.size == 0: + return None + + means = np.maximum(np.array([float(low.mean()), float(high.mean())]), _MIN_COMPONENT_MEAN) + sizes = np.array([_nb_size(means[0], float(low.var())), _nb_size(means[1], float(high.var()))]) + weights = np.array([low.size / x.size, high.size / x.size]) + + previous = -np.inf + for _ in range(_EM_MAX_ROUNDS): + # E step. + step = _responsibilities(x, means, sizes, weights) + if step is None: + return None + responsibilities, loglik = step + + # M step. A component that loses all its mass has collapsed, and continuing from it + # fits one population while reporting two. + mass = responsibilities.sum(axis=1) + if np.any(mass <= 0.0): + return None + weights = mass / x.size + for k in (0, 1): + r = responsibilities[k] + means[k] = max(float((r * x).sum() / mass[k]), _MIN_COMPONENT_MEAN) + variance = float((r * (x - means[k]) ** 2).sum() / mass[k]) + sizes[k] = _nb_size(means[k], variance) + + # The two components have converged onto each other, so there is one population. + if means[0] == means[1]: + return None + + if abs(loglik - previous) < _EM_TOLERANCE: + break + previous = loglik + + # "Label the higher-median component the signal one." The medians of the fitted + # components are ordered by their means, which the negative binomial's shape guarantees. + return _Mixture(means.copy(), sizes.copy(), weights.copy(), int(np.argmax(means))) + + +def _signal_probability(counts: np.ndarray, fit: _Mixture) -> np.ndarray | None: + """Each count's probability of belonging to the fit's signal component. + + Separate from the fit so that a cell trimmed out of the fit still gets an answer. Those + cells are the most bound in the sample, and withholding a probability for them would be + the opposite of what the trim is for. + """ + step = _responsibilities(counts.astype(float), fit.means, fit.sizes, fit.weights) + if step is None: + return None + return step[0][fit.signal] + + +def fit_tag_probabilities( + counts: np.ndarray, + min_cells: int = DEFAULT_DISTRIBUTION_MIN_CELLS, +) -> TagFit: + """One tag's counts across one sample's cells, as a probability of binding per cell. + + `counts` is one entry per cell in the sample, INCLUDING cells that read nothing, which + enter as zeros, and including cells an admissibility gate will later set aside. + `baseline-over-all-returned-cells` requires the second: a population narrowed by the gate + would be narrowed by the baseline's own consequences, and excluding the highest readings + lowers the baseline, which excludes more cells, which lowers it again. Passing only + observed readings breaks the first the same way -- the background is where most cells sit, + and the cells that read nothing are most of it. + + The count of cells, not of readings, is what the 300 gates. + + The returned probabilities are aligned to `counts`. The trim above the 99th percentile + keeps those cells and excludes them from the FIT alone, so every cell still gets an + answer. + """ + n = int(counts.size) + if n < min_cells: + return TagFit(None, TOO_FEW_CELLS, n) + + # Fit without the top tail, then score every cell against what was fitted. + ceiling = float(np.percentile(counts.astype(float), _UPPER_TRIM_PERCENTILE)) + fitted_on = counts[counts.astype(float) <= ceiling] + if fitted_on.size == 0: + return TagFit(None, NO_SEPARATION, n) + + fit = _fit_two_component_nb(fitted_on) + if fit is None: + return TagFit(None, NO_SEPARATION, n) + probabilities = _signal_probability(counts, fit) + if probabilities is None: + return TagFit(None, NO_SEPARATION, n) + background_index = 1 - fit.signal + background = Background( + mean=float(fit.means[background_index]), + signal_mean=float(fit.means[fit.signal]), + weight=float(fit.weights[background_index]), + ) + return TagFit(probabilities, None, n, background) + + +class TagFits(NamedTuple): + """Every (sample, tag) fit, as a frame of per-cell probabilities plus the misses. + + `probabilities` carries one row per (sample, cell, tag) the fit scored, with `pBound` the + probability that cell's count belongs to the signal component. A pair that established + nothing contributes no rows and appears in `reasons` instead. A caller must branch on + absence rather than defaulting: a tag that established nothing said NOTHING about its + cells, which is not the same fact as every cell reading a low probability. + """ + + probabilities: pl.DataFrame + reasons: dict[tuple[str, str], str] + # One entry per pair that fitted, on the same condition as a contribution to + # `probabilities`. A pair in `reasons` is absent here. + backgrounds: dict[tuple[str, str], Background] = {} + + +_PROB_SCHEMA = {"sampleId": pl.String, "cellId": pl.String, "tag": pl.String, "pBound": pl.Float64} + + +def fit_tag_probabilities_by_pair( + counts: pl.DataFrame, + cells: list[tuple[str, str]], + panel: pl.DataFrame, + min_cells: int = DEFAULT_DISTRIBUTION_MIN_CELLS, +) -> TagFits: + """One fit per (sample, tag) the panel declares, scored per cell. + + Per tag and before any grouping, which `what-plays-the-baseline` requires of every + fitting rung: a background fitted per identity would depend on the grouping, so changing + a grouping would change the background and a regrouping would stop being a re-reading of + unchanged counts. + + `counts` is the RAW frame -- not floored, not densified. `cells` is the analysed cell + universe, and it is the population every fit is taken over: a cell that read nothing for + a tag enters that tag's fit as a zero, and cells an admissibility gate will later set + aside are still in it. `baseline-over-all-returned-cells` requires the second, and the + first is what makes the background a background rather than the shape of whatever + happened to be observed. + + A tag the panel declared but the reads never showed is fitted over all zeros. Every count + is then identical, so no two-component fit exists and the pair establishes nothing -- the + honest answer, and also the QC finding. + + Fits are local: every sample is fitted on its own cells, so two samples' probabilities do + not share a currency. That is the ladder's property, not an artefact. + """ + universe = pl.DataFrame(cells, orient="row", schema={"sampleId": pl.String, "cellId": pl.String}) + # Checked rather than deduplicated. A duplicate would add one zero to the population it + # duplicates -- a background estimate over a population nobody chose, small, plausible, + # and invisible in the output. + if universe.height != universe.unique().height: + raise ValueError("the cell universe holds duplicated cells; every fit's population would be wrong") + + # Scoped to the cell universe first, so a barcode outside the analysis never enters a + # background it was not part of. + scoped = counts.join(universe, on=CELL_KEY, how="semi") + # The same reasoning as the universe check, one level down. A tag read twice in one cell + # contributes twice to its own background and displaces a zero. + if scoped.height != scoped.select(*CELL_KEY, "tag").unique().height: + raise ValueError("the counts frame holds duplicated readings; every fit's population would be wrong") + + by_sample = {s: f.select("cellId") for (s,), f in universe.group_by("sampleId")} + frames: list[pl.DataFrame] = [] + reasons: dict[tuple[str, str], str] = {} + backgrounds: dict[tuple[str, str], Background] = {} + for sample, tag in _declared_pairs(panel, sorted(by_sample)): + sample_cells = by_sample.get(sample) + if sample_cells is None: + continue + # Every cell in the sample, with an unobserved reading as the zero it is. The join + # keeps the frame's own order, so the probabilities come back aligned to these cells. + dense = sample_cells.join( + scoped.filter((pl.col("sampleId") == sample) & (pl.col("tag") == tag)).select("cellId", "umiCount"), + on="cellId", + how="left", + ).with_columns(pl.col("umiCount").fill_null(0)) + + fit = fit_tag_probabilities(dense["umiCount"].to_numpy(), min_cells) + if fit.probabilities is None: + reasons[(sample, tag)] = fit.reason or NO_SEPARATION + continue + if fit.background is not None: + backgrounds[(sample, tag)] = fit.background + frames.append( + dense.select("cellId") + .with_columns( + pl.lit(sample).alias("sampleId"), + pl.lit(tag).alias("tag"), + pl.Series("pBound", fit.probabilities, dtype=pl.Float64), + ) + .select(*_PROB_SCHEMA) + ) + + probabilities = pl.concat(frames) if frames else pl.DataFrame(schema=_PROB_SCHEMA) + return TagFits(probabilities, reasons, backgrounds) + + +_EMPTY = np.zeros(0, dtype=np.int64) + + +def _declared_pairs(panel: pl.DataFrame, samples: list[str]) -> list[tuple[str, str]]: + """Every (sample, tag) the panel declares, with a global declaration expanded. + + A panel with no sample column declares one row per tag under ANY_SAMPLE, meaning every + sample was stained with it. The fit is per sample regardless, because the population it + is taken over is a sample's cells. + """ + pairs = {(sample, tag) for tag, sample in panel.select("tag", "sample").iter_rows() if sample != ANY_SAMPLE} + global_tags = sorted({tag for tag, sample in panel.select("tag", "sample").iter_rows() if sample == ANY_SAMPLE}) + pairs |= {(sample, tag) for sample in samples for tag in global_tags} + return sorted(pairs) diff --git a/software/per-cell-metrics/src/verdict.py b/software/per-cell-metrics/src/verdict.py new file mode 100644 index 0000000..2741389 --- /dev/null +++ b/software/per-cell-metrics/src/verdict.py @@ -0,0 +1,786 @@ +"""Turning a cell's counts into states. + +Four steps in production, and the order is load-bearing: + + 1. the floor, on the raw count, per cell and per tag + 2. tags combine into an identity by the highest of their counts + 3. the identity's count is read against that cell's own reference reading + 4. the comparison becomes one of the four states + +tag-stat emits only observed pairs, so a cell asked about an identity and silent +produces no row. That absence is not evidence of nothing: an antigen every cell +failed to bind must read *not bound*, not vanish as though nobody offered it. +Production counts those positions analytically in `silent_tally`. `densify` builds +the full grid and is only the test oracle. It never runs in the block. + +The cell key is (sampleId, cellId) throughout: barcodes are bare 16-mers shared +across samples. + +Compare `min_umi` in per_cell_metrics.py, which resolves the other way -- below it a +feature is omitted rather than zeroed. A floored reading still answers "not bound". +An omitted one leaves nothing to answer with. + +Three frame shapes after step 2. The sparse per-tag frame and the per-identity frame +are keyed by CELL_KEY. `silent_tally` returns one keyed coarser, by (group, +identity), where group defaults to sampleId. +""" + +from __future__ import annotations + +from enum import Enum +from typing import NamedTuple + +import numpy as np +import polars as pl +from panel import ANY_SAMPLE, Grouping +from scipy.stats import beta + +CELL_KEY = ("sampleId", "cellId") + +# Uncalibrated: a declared default the scientist can move, not a fitted line. +DEFAULT_FLOOR = 4 + + +class Floored(NamedTuple): + counts: pl.DataFrame + stats: dict[str, int] + + +def apply_floor(counts: pl.DataFrame, floor: int, reference_tags: set[str]) -> Floored: + """Zero every (cell, tag) count below `floor`, except the comparator's. + + A floored count reads *not bound*, never *unreliable*: a count that small is not + distinguishable from none. + + Reference tags are ALWAYS exempt, and there is no switch. `minimum-count-before-any- + reference` puts it as a rule rather than a preference: the minimum asks whether a count + is evidence of binding, a tag declared to be bound by nothing is never evidence of + binding, so the question does not arise for it. A cell's reference reading enters the + comparison as it came back, and a small one is the measurement rather than noise. + + `reference_tags` is global by design: a tag is a comparator in every sample or in + none. The panel's (tag, sample) keying carries what a tag IS, not its role. + + Returns the floored counts and {"readingsFloored", "cellsEmptied"} for this + sample's QC row. Both assume the SPARSE frame, where every row is an observed + reading. Never densify first: manufactured rows would inflate readingsFloored and + count every unbound cell as emptied. + """ + # Not an optimisation: falling through would count a cell whose only reading is + # already zero as "emptied", when the floor removed nothing. + if floor <= 0: + return Floored(counts, {"readingsFloored": 0, "cellsEmptied": 0}) + + # is_in yields null for a null tag, so a null-tag row would escape both the floor + # and the emptied population. The panel reader never emits one -- this is a note + # for anyone feeding in an unvalidated frame. + is_ref = pl.col("tag").is_in(list(reference_tags)) if reference_tags else pl.lit(False) + exempt = is_ref + below = (pl.col("umiCount") < floor) & ~exempt + + readings_floored = int(counts.select(below.sum()).item()) + out = counts.with_columns( + pl.when(below).then(pl.lit(0, dtype=pl.Int64)).otherwise(pl.col("umiCount")).alias("umiCount") + ) + + # "Emptied" must follow the same switch. With the comparator exempt it is scoped to + # non-reference readings: a cell holding only the comparator never had evidence of + # binding to remove. With the comparator subject to the minimum it IS removable, so + # such a cell has been emptied. Scoping one way while flooring the other reports a + # cell as keeping evidence it lost, or losing evidence it never had. had_evidence + # deliberately does not filter on umiCount > 0 -- that is the sparse-frame + # assumption, not an oversight to "symmetrise". + counted = ~exempt + had_evidence = counts.filter(counted).select(CELL_KEY).unique() + kept_evidence = out.filter(counted & (pl.col("umiCount") > 0)).select(CELL_KEY).unique() + cells_emptied = had_evidence.join(kept_evidence, on=CELL_KEY, how="anti").height + + return Floored(out, {"readingsFloored": readings_floored, "cellsEmptied": cells_emptied}) + + +def cells_reading_nothing(floored: pl.DataFrame, cells: set[tuple[str, str]]) -> set[tuple[str, str]]: + """Of `cells`, the ones left with no count on any tag once the minimum has run. + + Not `cellsEmptied` renamed. That counter is scoped to readings the minimum could + remove, so an exempt comparator is invisible to it. This population is every tag, + comparator included, and that inclusion is the whole discriminator: a cell whose + antigen tags all fell below the minimum while its comparator survived took up + reagent, none of it antigen -- a real negative and a real vote. Only a cell with + nothing anywhere read nothing. So it reads the FLOORED frame and ignores which tag + is which. `--minimum-applies-to-baseline` moves this number, which is most of what + that switch is for. + + `cells` is passed in because the frame is sparse both ways: a cell with no row read + nothing on every tag and belongs here, and a cell outside the universe does not. + Passing the clonotypes' own membership keeps this count from exceeding the + clonotype's cell count. It changes no verdict, and must not -- these cells vote + *not bound*, and dropping them would shrink the denominator and make verdicts more + positive. + """ + reading = set(floored.filter(pl.col("umiCount") > 0).select(CELL_KEY).unique().rows()) + return {key for key in cells if key not in reading} + + +# Shipped defaults. Each is a visible parameter rather than a constant, because +# nothing published sets any of them and a hard-coded line would pretend to a basis +# nobody has. The panel minimum GATES rather than tunes: it comes from one preprint +# whose own panels held fifty and a hundred members, and nothing validates it lower. +# Below it, comparing a count against a handful of antigens is not a background +# estimate, so the baseline is wrong rather than conservative. Keep it above the +# fifteen-tag cap of an antibody kit, so a panel declaring no comparator falls to the +# per-tag distribution rung rather than standing in as its own background. +DEFAULT_PANEL_MIN_MEMBERS = 25 + + +class ReferenceChoice(str, Enum): + """Which comparator served. Two runs served differently do not compare. + + EMPTY_DROPLETS is deliberately absent: it needs gene expression and an + empty-droplet population this block does not receive, so declaring it would put a + crashing option in the dropdown. + + DISTRIBUTION is handled outside this module. The other two are keyed by cell and + built by `reference_by_cell`. DISTRIBUTION is keyed by (sample, identity), and its + conditions -- enough cells, counts that separate -- cannot be checked before the fit + runs, so `reference_by_cell` refuses it loudly rather than inventing an answer. + """ + + # Machine tokens, never prose, identical to the model's `ReferenceSource` union. + # These cross three boundaries -- run-meta JSON, a p-column DOMAIN, a UI branch -- + # so prose here would make rewording a sentence a breaking change. Display wording + # lives in the model's `referenceSources` output. `UnreliableReason` does the + # opposite, since its value IS the prose and nothing branches on it. DECLARED reads + # against ONE tag. A panel declaring several is refused, not combined. + DECLARED = "declared" + PANEL = "panel" + DISTRIBUTION = "distribution" + + +# Nothing here derives a default rung, and nothing may. A baseline nobody chose is a +# methodology nobody knows they used, and two runs of one experiment would be answered +# by different rules. --reference-source is required. A default would be the trap: the +# workflow omits --reference-source whenever the model's value is empty, so such a +# function would silently become the live rule -- deriving in the layer furthest from +# the reader. `served_source` below never picks a rung. It only reports that the one +# asked for cannot serve. + + +def served_source( + source: ReferenceChoice, + reference_tags: set[str], + panel_size: int, + min_members: int, +) -> ReferenceChoice: + """The source asked for, or a refusal naming the condition that failed. + + A baseline is required and a run without one does not happen. There is no bottom + rung answering everything *unreliable*: a full punchcard of non-answers costs what + a real run costs and looks like a result. + + Both conditions are properties of the SETTINGS -- whether a reference tag is + declared, and how many tags the panel carries -- so they are caught before anything + is read. The model refuses the same two in `args()`. This is the backstop for a + hand-driven run. + + The third rung cannot be checked here: whether a sample holds enough cells whose + counts separate is a property of the DATA, so the run proceeds and the caller + reports afterwards that no baseline could be established. + """ + if source is ReferenceChoice.DECLARED and not reference_tags: + raise SystemExit( + "the declared-baseline rung was selected and this panel declares no baseline tag. " + "Mark a tag as the baseline in the panel's role column, or select a different baseline " + "source. A run with no baseline produces no verdicts, which is what this block is for." + ) + if source is ReferenceChoice.PANEL and panel_size < min_members: + raise SystemExit( + f"the other-tags-in-the-cell rung was selected and this panel carries {panel_size} tags, " + f"below the {min_members} that rung needs. Below it the baseline is not conservative but " + "wrong, so the condition is a gate rather than a preference. Select a different baseline " + "source; an antibody panel cannot reach this one, its kits capping at fifteen tags." + ) + return source + + +class Reference(NamedTuple): + by_cell: dict[tuple[str, str], int] + served: ReferenceChoice + + +def reference_by_cell( + counts: pl.DataFrame, + reference_tags: set[str], + source: ReferenceChoice, + cells: list[tuple[str, str]] | None = None, + panel_size: int = 0, + min_members: int = DEFAULT_PANEL_MIN_MEMBERS, +) -> Reference: + """The reference reading per cell, and which source actually served. + + `source` is supplied, never inferred, and `served` always equals it -- there is no + rung below to fall to. `by_cell` holds a key for every analysed cell, zero where + that cell showed none of the comparator. A reader still switches on `served`, not + on key presence: `by_cell.get(key, 0)` would read "not in the analysis" as "the + comparator read zero". + + `cells`, where given, is authoritative both ways: the result holds exactly those + cells, zero-filled. Omit it and the universe comes from the counts frame, so a cell + that was asked and read nothing goes missing rather than zero. + + Receives the RAW, sparse per-tag frame -- before the minimum and before + densification. Each rung computes its baseline from its own source, and the minimum + acts on the identity's reading, never the comparator. Fed the floored frame, the + PANEL median would mix raw reference values with floored antigen ones. On a + densified frame, manufactured zeros would drag that median toward zero for every + cell. `reference_tags` is NOT excluded from the PANEL median: that comparator is + the cell's own readings, and a declared comparator is one of them. + """ + # Raises where the rung asked for cannot serve from the settings alone. No + # fall-through, because there is no bottom rung. + served = served_source(source, reference_tags, panel_size, min_members) + + all_cells = ( + # Deduplicated: a cell with several tag readings would otherwise be revisited + # once per reading by the zero-fill loop below, for no effect. + {(s, c) for s, c in zip(counts["sampleId"].to_list(), counts["cellId"].to_list(), strict=True)} + if cells is None + else cells + ) + # Semi join on the cell list, before either branch aggregates, so a cell outside + # the analysis is dropped before its rows are combined. + scoped = ( + counts + if cells is None + else counts.join( + pl.DataFrame(cells, orient="row", schema={"sampleId": pl.String, "cellId": pl.String}), + on=CELL_KEY, + how="semi", + ) + ) + + if served is ReferenceChoice.DECLARED: + if len(reference_tags) > 1: + # Scope of this version: no reference tag or one, never several. + # + # `280-baseline-scope` forbids combining references ACROSS scope groups, and + # requires replicates WITHIN one group to combine by taking the highest. Where + # no declared property separates them the whole panel is one group, so two + # undifferentiated references are replicates. This block has no scope + # construct, so it cannot tell the two cases apart, and it refuses rather than + # picking one. + named = ", ".join(sorted(reference_tags)) + raise SystemExit( + f"the panel declares {len(reference_tags)} baseline tags ({named}), and this version of " + "the block reads counts against one baseline tag or none. Reading against several needs " + "a panel column that says which antigens each one belongs to, which this version does " + "not have. Mark one tag as the baseline, or clear the role values and choose a different " + "baseline source." + ) + # An aggregator over a single tag. `group_by` still runs, so a duplicated + # reading cannot produce two rows. + rows = ( + scoped.filter(pl.col("tag").is_in(list(reference_tags))) + .group_by(CELL_KEY) + .agg(pl.col("umiCount").max().alias("ref")) + ) + elif served is ReferenceChoice.PANEL: + # cast(Int64) truncates rather than rounds, keeping the comparator an integer + # UMI count like every other reading here. + rows = scoped.group_by(CELL_KEY).agg(pl.col("umiCount").median().cast(pl.Int64).alias("ref")) + else: + # Reachable only if ReferenceChoice gains a member with no branch here, most + # plausibly EMPTY_DROPLETS. That is a missing implementation, not a fact about + # this run, so it must not read as "unavailable this time". + raise SystemExit(f"no comparator implementation for reference source {served.value!r}") + + ref = {(s, c): v for s, c, v in zip(rows["sampleId"], rows["cellId"], rows["ref"], strict=True)} + # The tag was offered. A cell showing none of it read zero, not nothing. + for key in all_cells: + ref.setdefault(key, 0) + return Reference(ref, served) + + +def gate_cells( + reference: dict[tuple[str, str], int], + threshold: int | None, +) -> tuple[set[tuple[str, str]], int | None]: + """Which cells a declared gate sets aside, and how many read high. + + ONE threshold does both jobs. `290-reference-two-roles` allows no second line: *how + many are high* needs a high, and only a declared gate supplies one. So the cells set + aside and the cells counted high are the same cells, by construction. + + The gate defaults off, and then the count is None rather than zero. There is no + boundary to count against, so the run's exposure is reported as the SPREAD of the + readings instead -- which is what a scientist reads in order to declare a gate. A + count against a line nobody drew would assert a boundary and answer a question + nobody asked. + + A sticky cell left in returns as a confident *not bound*, the collapse the four-state + model prevents, so the exposure is surfaced either way -- in one form or the other. + """ + if threshold is None: + return set(), None + gated = {k for k, v in reference.items() if v >= threshold} + return gated, len(gated) + + +# The cutoff and the three beta constants are the dominant tool's, inherited rather +# than justified: nothing published argues any of the four. They ship as the default so +# a run's numbers reconcile with what a scientist already has. +BETA_X, BETA_A_OFFSET, BETA_B_OFFSET = 0.925, 1, 3 +BOUND_CUTOFF = 75.0 + +# The population rung's own call, and not this block's to move. `what-plays-the-baseline` +# fixes it: under a fitted distribution "a cell reads *bound* at 0.9 or above", where 0.9 is +# the probability its count belongs to the signal component. `count-becomes-a-state` is +# explicit that the score above does not apply here -- each baseline brings its own rule, and +# a run selects one baseline, so exactly one rule calls the state. +# +# NOT A SETTING, and it must not become one. It comes from the literature -- "what every +# published result using this method reports at" -- so there is nothing here for a customer to +# decide. Offering it as a dial would only produce runs that cannot be compared against the +# work the method came from. +DISTRIBUTION_BOUND_PROBABILITY = 0.9 + + +class State(str, Enum): + """The four states a verdict takes. There is no fifth. + + NEVER_ASKED means the experiment did not put the identity to those cells. + UNRELIABLE means it did and the data cannot settle it. Neither is a kind of + NOT_BOUND, and collapsing either into it claims what the data does not support. + """ + + BOUND = "bound" + NOT_BOUND = "not bound" + NEVER_ASKED = "never asked" + UNRELIABLE = "unreliable" + + +class UnreliableReason(str, Enum): + """Why a cell's comparison could not be made. The value is the prose a reader sees, + the member is what code compares against, so wording can change safely.""" + + GATED = "cell set aside by the admissibility gate" + NO_COMPARATOR = "no comparator for this cell" + + +def combine_tags_to_identities(counts: pl.DataFrame, grouping: Grouping) -> pl.DataFrame: + """An identity's reading in a cell is the highest of its tags' counts. + + Resolved through the cell's OWN sample. The grouping is keyed (tag, sample) because + the panel file is, so a barcode reused across panels contributes to the antigen its + own sample declared. A cell belongs to exactly one sample, which makes "the highest + of its tags' counts" well defined under reuse. + + Counts are never added, and summing is not offered. Requiring every tag to clear + was measured and is the worst option available. Summing would need the reference + scaled to a summed identity, assuming each tag picks up background at the + reference's rate -- tags differ by an amount nobody has measured. + """ + star = {tag: identity for (tag, sample), identity in grouping.items() if sample == ANY_SAMPLE} + keyed = [(tag, sample, identity) for (tag, sample), identity in grouping.items() if sample != ANY_SAMPLE] + mapped = counts.join( + pl.DataFrame( + keyed, + orient="row", + schema={"tag": pl.String, "sampleId": pl.String, "identity": pl.String}, + ), + on=["tag", "sampleId"], + how="left", + ) + if star: + # A panel with no sample dimension declares one mapping over every sample, so + # star rows fill where the keyed join found nothing. Checked second, so an + # explicit per-sample declaration always wins. + mapped = mapped.with_columns(pl.col("identity").fill_null(pl.col("tag").replace_strict(star, default=None))) + mapped = mapped.filter(pl.col("identity").is_not_null()) + return mapped.group_by([*CELL_KEY, "identity"]).agg(pl.col("umiCount").max().alias("umiCount")) + + +def densify(identities: pl.DataFrame, cells: pl.DataFrame, offered_by_sample: dict[str, set[str]]) -> pl.DataFrame: + """Every cell against every identity its sample offered, zeros filled in. + + Without this, an antigen every cell failed to bind produces no rows and its failure + is indistinguishable from a reagent nobody offered. Epitope mapping turns on that + distinction, where *not bound* is the finding. + + The reference implementation, and it must never run in the block: on a realistic + run this grid is 11-20x the sparse input and does not fit a large panel at all. + Production uses `silent_tally`, and this is the oracle it is tested against. + """ + # Guard on the assembled blocks, never on offered_by_sample. A map whose every + # value is empty -- a sample stained with nothing -- is non-empty itself but + # contributes no block, and concat of an empty list raises. + blocks = [ + cells.filter(pl.col("sampleId") == sample).join(pl.DataFrame({"identity": sorted(offered)}), how="cross") + for sample, offered in sorted(offered_by_sample.items()) + if offered + ] + grid = ( + pl.concat(blocks, how="vertical") + if blocks + else cells.head(0).with_columns(pl.lit(None, pl.String).alias("identity")) + ) + + return grid.join(identities, on=[*CELL_KEY, "identity"], how="left").with_columns( + pl.col("umiCount").fill_null(0).cast(pl.Int64) + ) + + +def specificity_score(antigen_count, reference_count): + """How specifically the antigen count exceeds the reference: 0-100. + + At antigen_count = 0 this is ~0.0422 at reference_count = 0, and falls for every + larger reference_count. That is the module's central claim: `silent_tally` relies + on a silent admissible cell never scoring BOUND, which lets its state be known with + no row written. It holds only for a `cutoff` strictly above 0.0422 -- at or below, + `silent_tally` and the `densify` oracle part company with no error raised here. + Refusing such a cutoff is the CLI's job. + """ + a = np.asarray(antigen_count, dtype=float) + BETA_A_OFFSET + b = np.asarray(reference_count, dtype=float) + BETA_B_OFFSET + return (1.0 - beta.cdf(BETA_X, a, b)) * 100.0 + + +class Admissibility(NamedTuple): + """The pair `read_states` and `silent_tally` must share to agree on what "cannot be + compared" means for a cell. + + Sharing `_admissibility_reason` makes them agree on the *rule*, not on the + *arguments*. Passing one bundle to both makes disagreement impossible by + construction. The disagreement to fear: `read_states` given a reference restricted + to observed cells while `silent_tally` gets the full one, which sends + `silentUnreliable` wrong or negative. + """ + + reference: dict[tuple[str, str], int] + gated: set[tuple[str, str]] + by_identity: dict[tuple[str, str], int] | None = None + # The population rung's comparator is not a count. It is a probability per (sample, cell, + # identity) that the cell's reading belongs to the signal component, and the state is read + # from it directly. Set means it is the whole comparator, and `reference` and `by_identity` + # are both empty. A missing key means the fit established nothing for that position. + probabilities: dict[tuple[str, str, str], float] | None = None + + +def _admissibility_reason(key: tuple[str, str], identity: str, admissibility: Admissibility) -> UnreliableReason | None: + """Why this comparison cannot be made, or None if it can be. + + Takes an identity because one rung's comparator depends on it. Cell-keyed rungs -- + the declared reagent, the panel's own readings -- answer the same for every + identity. The per-tag distribution rung fits per (sample, tag), so a tag whose + counts did not separate leaves only the identities built from it uncomparable. + + `by_identity` distinguishes the two: set means it is the whole comparator and + `reference` is empty; None means the comparator is keyed by cell. Never merge them. + Membership is tested, never `get(..., 0)` -- a missing key means no comparator + existed, and defaulting to 0 reads as "served and found nothing". + + A LOW comparator reading is not a reason. No published line separates thin from + usable, so there is no thin-reference branch: the comparison runs, and every cell's + reference reading is emitted so a reader can see what a verdict rested on. + """ + reference, gated, by_identity, probabilities = admissibility + if key in gated: + return UnreliableReason.GATED + if probabilities is not None: + return None if (key[0], key[1], identity) in probabilities else UnreliableReason.NO_COMPARATOR + if by_identity is not None: + return None if (key[0], identity) in by_identity else UnreliableReason.NO_COMPARATOR + if key not in reference: + return UnreliableReason.NO_COMPARATOR + return None + + +def cell_admissibility_reason(key: tuple[str, str], admissibility: Admissibility) -> UnreliableReason | None: + """The part of the reason belonging to the CELL, whatever it was asked about. + + Needed by every output keyed by cell rather than by position: the per-cell scalars, + the punchcard's silent-position fallback, the set-level reason. Where the comparator + is keyed by cell this is the whole reason. Where it is keyed by identity it is only + the gate, because a cell whose identity has no fitted background is a fine cell + asked an unanswerable question -- calling the CELL uncomparable would misreport + every identity, including the ones that fitted. + """ + _reference, gated, by_identity, probabilities = admissibility + if key in gated: + return UnreliableReason.GATED + if by_identity is None and probabilities is None and key not in admissibility.reference: + return UnreliableReason.NO_COMPARATOR + return None + + +def _comparator(key: tuple[str, str], identity: str, admissibility: Admissibility) -> int | None: + """The reading this comparison is made against, or None where none served. + + None under the population rung, always. That rung's comparator is a fitted distribution + rather than a reading, so there is no count to emit -- and a number here would read as a + comparator that served, which is the one thing null is for. + """ + reference, _gated, by_identity, probabilities = admissibility + if probabilities is not None: + return None + if by_identity is not None: + return by_identity.get((key[0], identity)) + return reference.get(key) + + +def read_states(identities: pl.DataFrame, admissibility: Admissibility, cutoff: float) -> pl.DataFrame: + """Give every (cell, identity) row a state. + + Two routes to UNRELIABLE, both recorded in `unreliableReason`: the cell has no + comparator, or a gate set it aside. Gated cells stay in the frame -- dropping them + makes a set whose every cell was set aside read *never asked*. The gate is checked + first, because a cell it set aside was not measured at all. + + Emits umiCount and referenceCount, never the score: re-derivation under a new + grouping needs the counts, and no binding level may leave the block. + `referenceCount` is nullable, and null is not 0 -- null means no comparator served, + 0 means one served and read nothing. A downstream `fill_null(0)` collapses "not + measured" into "measured as zero". + + A cell in `identities` but absent from `silent_tally`'s cell list still gets a row + here, since this function takes no cell list, and `silent_tally` drops it. + """ + keys = list(zip(identities["sampleId"].to_list(), identities["cellId"].to_list(), strict=True)) + idents = identities["identity"].to_list() + reasons = [_admissibility_reason(k, i, admissibility) for k, i in zip(keys, idents, strict=True)] + refs = [_comparator(k, i, admissibility) for k, i in zip(keys, idents, strict=True)] + + df = identities.with_columns( + pl.Series("referenceCount", refs, dtype=pl.Int64), + pl.Series("unreliableReason", [r.value if r is not None else None for r in reasons], dtype=pl.String), + ) + + # Each baseline brings its own rule, and the selected baseline decides which one runs. A + # fitted population hands back a probability per position, and the state is read from it + # at the rung's own line. The score below is the declared reagent's rule and does not + # apply here: substituting a fitted background into it would produce a number the method + # it came from never defines. + if admissibility.probabilities is not None: + called = [admissibility.probabilities.get((k[0], k[1], i)) for k, i in zip(keys, idents, strict=True)] + df = df.with_columns(pl.Series("_pBound", called, dtype=pl.Float64)).with_columns( + pl.when(pl.col("unreliableReason").is_not_null()) + .then(pl.lit(State.UNRELIABLE.value)) + .when(pl.col("_pBound") >= DISTRIBUTION_BOUND_PROBABILITY) + .then(pl.lit(State.BOUND.value)) + .otherwise(pl.lit(State.NOT_BOUND.value)) + .alias("state") + ) + return df.select([*CELL_KEY, "identity", "umiCount", "referenceCount", "state", "unreliableReason"]) + + scored = specificity_score( + df["umiCount"].to_numpy(), + np.nan_to_num(df["referenceCount"].cast(pl.Float64).to_numpy(), nan=0.0), + ) + + df = df.with_columns(pl.Series("_score", scored, dtype=pl.Float64)).with_columns( + pl.when(pl.col("unreliableReason").is_not_null()) + .then(pl.lit(State.UNRELIABLE.value)) + .when(pl.col("_score") >= cutoff) + .then(pl.lit(State.BOUND.value)) + .otherwise(pl.lit(State.NOT_BOUND.value)) + .alias("state") + ) + + return df.select([*CELL_KEY, "identity", "umiCount", "referenceCount", "state", "unreliableReason"]) + + +def silent_tally( + observed: pl.DataFrame, + cells: pl.DataFrame, + offered_by_sample: dict[str, set[str]], + admissibility: Admissibility, + group_by_cell: dict[tuple[str, str], str] | None = None, + group_column: str = "sampleId", +) -> pl.DataFrame: + """Per (group, identity): how many asked cells were never observed, and how they resolve. + + The sparse path -- silent positions are counted, never materialized. `densify` then + `read_states` is the reference this must agree with, kept only for tests. + + A silently admissible cell's count is 0, and specificity_score(0, r) is ~0.0422 at + r = 0 and smaller beyond. So a silent cell resolves to NOT_BOUND unless the cell + itself cannot be compared -- gated or lacking a comparator -- a per-cell fact + independent of which identity was silent. That holds only for a `cutoff` strictly + above ~0.0422; at or below, the dense oracle can call such a cell BOUND while this + reports NOT_BOUND, silently. Refusing such a cutoff is the CLI's job. Above the + bound, three cheap terms replace a materialized row per silent cell: + + asked = cells of the group, for every identity offered to one of its members + observed = the (cell, identity) rows read_states already produced + silentUnreliable = inadmissible cells the group counts toward that identity - + inadmissible cells among the observed + silentNotBound = asked - observed - silentUnreliable + + `group_by_cell` maps a cell key to the unit the tally reports per. It defaults to + None, grouping by the cell's own sampleId -- the only grouping under which every + member shares one offered set, which is what lets `asked` and `total_inadmissible` + be computed once per group. A group spanning samples with different offered sets + has no such guarantee, so both terms move inside the identity loop. + + `offered_by_sample` is never regrouped: staining is done per sample. Every cell key + in `cells` needs an entry in `group_by_cell` where one is given. `group_column` + names the key column in the returned frame, "sampleId" by default. + + Precondition, unchecked by types: `cells` unique on the cell key, `observed` unique + on (cell, identity). A duplicated `cells` row is harmless and deduplicated below. A + duplicated `observed` row is double-counted against totals that count the cell + once, which can drive `silentUnreliable` negative. The check below makes that loud. + """ + # A duplicated cells row must not count twice -- a legitimate no-op to guard + # against, unlike a duplicated observed row, which is a contract violation. + keys = list(dict.fromkeys(zip(cells["sampleId"].to_list(), cells["cellId"].to_list(), strict=True))) + cell_keys = set(keys) + + def inadmissible(key: tuple[str, str], identity: str) -> bool: + return _admissibility_reason(key, identity, admissibility) is not None + + obs_keys = list(zip(observed["sampleId"].to_list(), observed["cellId"].to_list(), strict=True)) + obs_identity = observed["identity"].to_list() + + rows: list[tuple[str, str, int, int, int, int]] = [] + + if group_by_cell is None: + # Sample-keyed path. One accumulating pass, never one loop per sample: scanning + # all of `keys` per sample is O(groups x cells), harmless at 24 samples but + # quadratic once a wider key groups thousands of sets. + asked_count: dict[str, int] = {} + gated_count: dict[str, int] = {} + no_comparator_count: dict[str, int] = {} + for k in keys: + sample = k[0] + asked_count[sample] = asked_count.get(sample, 0) + 1 + if k in admissibility.gated: + gated_count[sample] = gated_count.get(sample, 0) + 1 + elif admissibility.by_identity is None and k not in admissibility.reference: + # Cell-keyed comparators only. Where the comparator is keyed by + # identity this is not a property of the cell, and the loop below + # computes it per identity. + no_comparator_count[sample] = no_comparator_count.get(sample, 0) + 1 + + observed_count: dict[tuple[str, str], int] = {} + observed_inadmissible_count: dict[tuple[str, str], int] = {} + for k, ident in zip(obs_keys, obs_identity, strict=True): + if k not in cell_keys: + # In `identities` but absent from `cells`: dropped rather than counted + # against a cell universe that never named it. + continue + if ident not in offered_by_sample.get(k[0], frozenset()): + # Read, but this cell's OWN sample never offered the identity. `asked` + # counts only members whose own sample offered it, so counting this + # would draw numerator and denominator from two populations -- it + # displaces a silent cell's real vote with one never asked for. + continue + pair = (k[0], ident) + observed_count[pair] = observed_count.get(pair, 0) + 1 + if inadmissible(k, ident): + observed_inadmissible_count[pair] = observed_inadmissible_count.get(pair, 0) + 1 + + for sample, offered in sorted(offered_by_sample.items()): + # `asked` hoists out of the identity loop because a sample offers the same + # identities to all its cells. The inadmissible term hoists only for + # cell-keyed comparators. Keyed by identity, a tag whose counts did not + # separate takes out every cell of the sample for the identities built from + # it and none of the others, so it is computed inside the loop from the same + # two counters. That keeps this O(samples x identities), not O(cells x + # identities). + asked = asked_count.get(sample, 0) + gated_here = gated_count.get(sample, 0) + uncomparable_here = no_comparator_count.get(sample, 0) + for identity in sorted(offered): + pair = (sample, identity) + if admissibility.by_identity is None: + total_inadmissible = gated_here + uncomparable_here + elif pair in admissibility.by_identity: + total_inadmissible = gated_here + else: + # No comparator for this identity anywhere in the sample, so every + # cell is unreliable against it -- gated ones included, and already + # counted here once. + total_inadmissible = asked + observed_n = observed_count.get(pair, 0) + observed_inadmissible_n = observed_inadmissible_count.get(pair, 0) + silent_unreliable = total_inadmissible - observed_inadmissible_n + silent_not_bound = asked - observed_n - silent_unreliable + # Raised, not asserted: stripped under -O these terms stay negative and are + # summed into the tallies, so the run reports fewer silent positions than it + # has, with nothing to show it. + if asked < 0 or silent_unreliable < 0 or silent_not_bound < 0: + raise ValueError( + f"negative silent term for {sample!r}/{identity!r} " + f"(asked={asked}, silentUnreliable={silent_unreliable}, silentNotBound={silent_not_bound}): " + "cells or observed violated the uniqueness precondition documented above" + ) + rows.append((sample, identity, asked, observed_n, silent_unreliable, silent_not_bound)) + else: + # Group-keyed path. A group can mix samples with different offered sets, so + # neither term hoists above the identity loop. Each group is walked once, member + # by member, checking that member's OWN sample's offered set. One pass produces + # every identity's `asked` and `total_inadmissible`. A single count computed + # before the loop would silently apply to an identity some members never saw. + keys_by_group: dict[str, list[tuple[str, str]]] = {} + for k in keys: + keys_by_group.setdefault(group_by_cell[k], []).append(k) + + observed_count = {} + observed_inadmissible_count = {} + for k, ident in zip(obs_keys, obs_identity, strict=True): + if k not in cell_keys: + continue + if ident not in offered_by_sample.get(k[0], frozenset()): + # Read, but this cell's OWN sample never offered the identity. `asked` + # counts only members whose own sample offered it, so counting this + # would draw numerator and denominator from two populations -- it + # displaces a silent cell's real vote with one never asked for. + continue + pair = (group_by_cell[k], ident) + observed_count[pair] = observed_count.get(pair, 0) + 1 + if inadmissible(k, ident): + observed_inadmissible_count[pair] = observed_inadmissible_count.get(pair, 0) + 1 + + for group in sorted(keys_by_group): + asked_by_identity: dict[str, int] = {} + inadmissible_by_identity: dict[str, int] = {} + for k in keys_by_group[group]: + # Per identity rather than once per member: with an identity-keyed + # comparator, whether this member can be compared depends on which + # identity is asked about. + for identity in offered_by_sample.get(k[0], set()): + asked_by_identity[identity] = asked_by_identity.get(identity, 0) + 1 + if inadmissible(k, identity): + inadmissible_by_identity[identity] = inadmissible_by_identity.get(identity, 0) + 1 + + for identity in sorted(asked_by_identity): + asked = asked_by_identity[identity] + total_inadmissible = inadmissible_by_identity.get(identity, 0) + pair = (group, identity) + observed_n = observed_count.get(pair, 0) + observed_inadmissible_n = observed_inadmissible_count.get(pair, 0) + silent_unreliable = total_inadmissible - observed_inadmissible_n + silent_not_bound = asked - observed_n - silent_unreliable + # Raised, not asserted: stripped under -O these terms stay negative and are + # summed into the tallies, so the run reports fewer silent positions than it + # has, with nothing to show it. + if asked < 0 or silent_unreliable < 0 or silent_not_bound < 0: + raise ValueError( + f"negative silent term for {group!r}/{identity!r} " + f"(asked={asked}, silentUnreliable={silent_unreliable}, silentNotBound={silent_not_bound}): " + "cells or observed violated the uniqueness precondition documented above" + ) + rows.append((group, identity, asked, observed_n, silent_unreliable, silent_not_bound)) + + return pl.DataFrame( + rows, + orient="row", + schema={ + group_column: pl.String, + "identity": pl.String, + "asked": pl.Int64, + "observed": pl.Int64, + "silentUnreliable": pl.Int64, + "silentNotBound": pl.Int64, + }, + ) diff --git a/software/per-cell-metrics/test/test_acceptance_scenarios.py b/software/per-cell-metrics/test/test_acceptance_scenarios.py new file mode 100644 index 0000000..632df54 --- /dev/null +++ b/software/per-cell-metrics/test/test_acceptance_scenarios.py @@ -0,0 +1,511 @@ +"""The spec's acceptance scenarios, each driven from files through the CLI. + +Every scenario writes a counts CSV, a panel CSV and a linker CSV, runs `emit_verdicts.py` as a +subprocess, and asserts on the CSVs it wrote. Nothing here builds a per-cell state frame, calls +`read_states`, or reaches into a module. An earlier revision of these scenarios did exactly that +and passed while the pipeline read a mutant whose cells all failed to bind as *never asked*. A +scenario that constructs the states it then reads tests its own assertion, not the reading. + +**Absence in the counts file means two different things, and which one a bed carries decides the +state it must produce.** A tag the SAMPLE's reads never carry is a reagent that produced nothing: +it removes its cells from what could answer, and the position reads *never asked*. A tag the +sample did measure that a particular CELL read nothing for is a reading that happened and failed, +and that cell votes *not bound*. So a bed testing a failure to bind gives the tag ambient counts, +and only the dead-reagent bed leaves a tag out entirely. Getting this backwards is how a dead +reagent becomes a confident clean negative on every clonotype in the run. + +Four numbers are load-bearing in the beds below, so they are stated once here rather than +rediscovered by whoever next changes a count. + +*The cutoff is 75 and the score is a beta function, not a ratio.* Against a reference of 6, a +count of 500 scores 100 and binds. Counts of 50 and 60 score 3.1 and 7.2 and read *not bound* -- +large-looking counts that cannot reach the cutoff. Against a reference of 20 a count of 500 still +scores 99.85. Check the score before asserting a state. `specificity_score(count, reference)` in +verdict.py answers directly. + +*The floor is 4.* Any antigen reading of 1-3 is zeroed before anything else runs, so background +counts here sit at 5 or above. A floored reading can also drag a panel-derived comparator to zero, +against which every surviving count scores near 100, turning a whole run *bound* for a reason that +has nothing to do with the scenario. + +*A cell with no comparator reading is inadmissible and votes nowhere.* Every bed declares a +comparator tag whose role value matches `--reference-values`, and gives every one of its cells a +count for it. + +Tags are the identities under the default per-tag grouping, so they are named for the part they +play (`TARGET`, `OFF1`) rather than written as barcode sequences. The pipeline treats a tag as an +opaque string. +""" + +import json +import subprocess +import sys +from pathlib import Path + +import polars as pl +import pytest + +SRC = Path(__file__).resolve().parents[1] / "src" + + +def _run(cwd, *args): + return subprocess.run( + [sys.executable, str(SRC / "emit_verdicts.py"), *map(str, args)], cwd=cwd, capture_output=True, text=True + ) + + +BASE = [ + "counts.csv", + "panel.csv", + "--linker", + "linker.csv", + "--barcode-col", + "Sequence", + "--feature-col", + "Name", + "--sample-col", + "Samples", + "--role-column", + "Type", + "--reference-values", + "Control", + # Stated, because the CLI requires it: nothing below the model picks a rung for a scientist who + # did not. This bed declares a comparator tag, which is the rung every scenario here is read under. + "--reference-source", + "declared", + "--output-prefix", + "result", +] + +# A comparator reading every cell shares. Above the floor of 4 so it survives +# it, and well below the high-reference observation line of 100 so nothing in +# these beds is flagged for background it does not have. +COMPARATOR = 6 + +# Clears the cutoff of 75 against a comparator of 6: the score is 100. +BINDING = 500 + +# Survives the floor of 4 and scores 0.0 against a comparator of 6. A reading +# that is present and settles *not bound*, as distinct from a cell that was +# asked and produced no row at all. +BACKGROUND = 5 + + +def _verdicts(bed): + # Read without schema inference throughout. `unreliableReason` is null on a settled row, and + # polars would otherwise infer the counts back into integers and the reason column's nulls into + # something a test cannot tell from an empty string. + return pl.read_csv(bed / "result_verdicts.csv", infer_schema_length=0) + + +def _row(bed, set_id, identity): + got = _verdicts(bed).filter((pl.col("setId") == set_id) & (pl.col("identity") == identity)) + assert got.height == 1, f"expected exactly one ({set_id}, {identity}) row, got {got.height}" + return got.row(0, named=True) + + +# --------------------------------------------------------------------------- +# Epitope loss: the finding is a failure, and the failure comes from silence. +# --------------------------------------------------------------------------- + + +# Ambient. A reagent that is present and bound nothing still returns counts, +# because ambient material reaches every droplet -- so a mutant the clone failed +# to bind reads LOW, not empty. 2 sits under the floor of 4, so the reading is +# zeroed and the position settles *not bound*, which is what this scenario is +# about. It is deliberately not 0 rows: zero rows means a reagent that never +# worked, which is a different finding with a different state. +AMBIENT = 2 + + +@pytest.fixture +def epitope_bed(tmp_path): + """One clonotype against an unmutated antigen and four point mutants. + + The fourth mutant carries **ambient counts only** -- present in the reads, + below the floor. That is the bed: the clone failed to bind M4, and a failure + to bind is not the reagent going missing. Ambient material reaches every + droplet, so a live reagent nothing bound still returns counts, and the + reading has to settle *not bound* from a count the floor zeroed. + + Giving M4 no rows at all would make this the dead-reagent bed below, where + the same absence means the question was never put and the state is *never + asked*. The two are told apart by whether the tag appears in the sample's + reads, never by how low its counts are. + """ + (tmp_path / "panel.csv").write_text( + "Samples,Name,Sequence,Type\n" + "S1,AgWT,WT,Target\n" + "S1,AgM1,M1,Target\n" + "S1,AgM2,M2,Target\n" + "S1,AgM3,M3,Target\n" + "S1,AgM4,M4,Target\n" + "S1,Ctrl,CTRL,Control\n" + ) + cells = ("c1", "c2", "c3", "c4") + rows = ["sampleId,cellId,tag,umiCount"] + for cell in cells: + rows.append(f"S1,{cell},CTRL,{COMPARATOR}") + # The clone binds the unmutated antigen and the first three mutants. The epitope it grabs + # survives those substitutions. + for tag in ("WT", "M1", "M2", "M3"): + rows.append(f"S1,{cell},{tag},{BINDING}") + # M4 is on the panel, in the reads, and bound by nothing. + rows.append(f"S1,{cell},M4,{AMBIENT}") + (tmp_path / "counts.csv").write_text("\n".join(rows) + "\n") + (tmp_path / "linker.csv").write_text("sampleId,cellId,setId\n" + "".join(f"S1,{cell},K1\n" for cell in cells)) + return tmp_path + + +@pytest.fixture +def dead_reagent_bed(tmp_path): + """The same panel, with M4's reagent having produced nothing at all. + + M4 has **no rows in the counts file for any cell of the sample**. Zero reads + is categorical and cannot arise from biology: ambient reagent reaches every + cell, so a tag that bound nothing still returns counts. What zero reads means + is a reagent never added, a barcode mis-declared, or a library that failed -- + and none of those put the question the panel file says was put. + """ + (tmp_path / "panel.csv").write_text( + "Samples,Name,Sequence,Type\nS1,AgWT,WT,Target\nS1,AgM4,M4,Target\nS1,Ctrl,CTRL,Control\n" + ) + cells = ("c1", "c2", "c3", "c4") + rows = ["sampleId,cellId,tag,umiCount"] + for cell in cells: + rows.append(f"S1,{cell},CTRL,{COMPARATOR}") + rows.append(f"S1,{cell},WT,{BINDING}") + # M4 absent for every cell: the reagent produced nothing. + (tmp_path / "counts.csv").write_text("\n".join(rows) + "\n") + (tmp_path / "linker.csv").write_text("sampleId,cellId,setId\n" + "".join(f"S1,{cell},K1\n" for cell in cells)) + return tmp_path + + +def test_the_mutant_no_cell_bound_reads_not_bound_not_never_asked(epitope_bed): + # The scientist's statement is "binds the unmutated antigen and fails on the fourth mutant", so + # *not bound* is the finding and the run must produce it from silence. Reading M4 as *never + # asked* is the failure this scenario exists to catch: it turns the finding into a gap, and the + # clonotype whose whole value is that failure goes back as unsettled. + r = _run(epitope_bed, *BASE) + assert r.returncode == 0, r.stderr + + wt = _row(epitope_bed, "K1", "WT") + assert wt["state"] == "bound" + + m4 = _row(epitope_bed, "K1", "M4") + assert m4["state"] == "not bound", "the cells were offered M4 and were silent; silence is a failure to bind" + assert m4["unreliableReason"] is None, "a settled reading carries no reason for not settling" + + # Every one of the four cells was offered M4 and every one of them voted. + # A reading resting on the four silences is what makes the failure a + # finding rather than an absence of data. + assert (int(m4["cellsCouldAnswer"]), int(m4["cellsAnswered"])) == (4, 4) + + +def test_a_live_mutant_nothing_bound_is_still_reported_as_seen(epitope_bed): + # The quality row and the verdict say different things and neither substitutes for the other. + # Here the reagent worked, because it returned ambient counts in every cell. So the + # panel-versus-reads check must NOT report it as a tag the reads never show, and the verdict is + # the clone's failure. + assert _run(epitope_bed, *BASE).returncode == 0 + qc = pl.read_csv(epitope_bed / "result_qc.csv", infer_schema_length=0) + never_seen = qc.filter((pl.col("measurement") == "declaredNeverSeen") & (pl.col("entity") == "M4")) + assert never_seen.height == 1 + assert float(never_seen.row(0, named=True)["value"]) > 0.0, "M4 returned reads, so it was seen" + assert _row(epitope_bed, "K1", "M4")["state"] == "not bound" + + +def test_a_dead_reagent_reads_never_asked_not_a_confident_negative(dead_reagent_bed): + # The headline failure, arriving by the one route the states were not watching. The antigen was + # declared, so *never asked* does not fire from the panel. Zero counts fall below the minimum, so + # every cell settles *not bound*. The result is a confident clean negative on every clone in the + # run, which is exactly the claim that qualifies a lead. + # + # A tag the reads never show removes its cells from what could answer. That is not the reads + # overruling the file: the file declares what was offered, the reads say which cells were + # actually measured, and those were always different questions. No line is drawn and no threshold + # is chosen, and a real negative cannot trigger it, since a real negative still has reads. + r = _run(dead_reagent_bed, *BASE) + assert r.returncode == 0, r.stderr + + assert _row(dead_reagent_bed, "K1", "WT")["state"] == "bound", "the live antigen still answers" + + m4 = _row(dead_reagent_bed, "K1", "M4") + assert m4["state"] == "never asked", "zero reads means nobody could answer, not that nobody bound" + assert int(m4["cellsCouldAnswer"]) == 0, "cells in a sample where the tag returned nothing do not vote" + + # And the reagent finding is still stated on its own row, for the reagent's + # sake rather than the answer's. + qc = pl.read_csv(dead_reagent_bed / "result_qc.csv", infer_schema_length=0) + never_seen = qc.filter((pl.col("measurement") == "declaredNeverSeen") & (pl.col("entity") == "M4")) + assert never_seen.height == 1 + assert float(never_seen.row(0, named=True)["value"]) == 0.0 + + +# --------------------------------------------------------------------------- +# Unasked off-target: one clonotype left unsettled, one positively disqualified. +# --------------------------------------------------------------------------- + + +@pytest.fixture +def off_target_bed(tmp_path): + """Two clonotypes under "binds the target and nothing on the off-target list". + + The list is OFF1, OFF2, OFF3. Neither sample's panel carries all three, and + the two samples omit different ones, which is what puts the two clonotypes + on opposite sides of the statement. + + KA comes from S1, whose panel omits OFF3: KA has a clean reading on the + off-targets it was asked about and an unsettled position on the one it was + not. KB comes from S2, whose panel omits OFF2 -- but KB **binds** OFF1, + which S2 did ask. The run disqualified KB on a position it settled, and the + unasked one changes nothing about that. + """ + (tmp_path / "panel.csv").write_text( + "Samples,Name,Sequence,Type\n" + "S1,AgTarget,TARGET,Target\n" + "S1,AgOff1,OFF1,Target\n" + "S1,AgOff2,OFF2,Target\n" + "S1,Ctrl,CTRL,Control\n" + "S2,AgTarget,TARGET,Target\n" + "S2,AgOff1,OFF1,Target\n" + "S2,AgOff3,OFF3,Target\n" + "S2,Ctrl,CTRL,Control\n" + ) + rows = ["sampleId,cellId,tag,umiCount"] + for cell in ("a1", "a2", "a3"): + rows.append(f"S1,{cell},CTRL,{COMPARATOR}") + rows.append(f"S1,{cell},TARGET,{BINDING}") + # Two routes to *not bound* in one clonotype, so the clean off-target list does not rest on + # either route alone. OFF1 reads low in every cell. OFF2 reads ambient in a1 only and is + # silent in a2 and a3 -- a tag the SAMPLE measured, which some of its cells read nothing for. + # That per-cell silence is a reading that happened and failed. + # + # OFF2 must appear in at least one of S1's cells. A tag absent from the whole sample is a + # reagent that produced nothing, which removes its cells from what could answer and reads + # *never asked* -- the dead reagent bed above, and a different finding from this one. + rows.append(f"S1,{cell},OFF1,{BACKGROUND}") + if cell == "a1": + rows.append(f"S1,{cell},OFF2,{AMBIENT}") + for cell in ("b1", "b2", "b3"): + rows.append(f"S2,{cell},CTRL,{COMPARATOR}") + rows.append(f"S2,{cell},TARGET,{BINDING}") + rows.append(f"S2,{cell},OFF1,{BINDING}") + rows.append(f"S2,{cell},OFF3,{BACKGROUND}") + (tmp_path / "counts.csv").write_text("\n".join(rows) + "\n") + (tmp_path / "linker.csv").write_text( + "sampleId,cellId,setId\n" + + "".join(f"S1,{cell},KA\n" for cell in ("a1", "a2", "a3")) + + "".join(f"S2,{cell},KB\n" for cell in ("b1", "b2", "b3")) + ) + return tmp_path + + +def test_an_off_target_the_panel_omitted_is_present_and_reads_never_asked(off_target_bed): + # The row has to exist. Dropping it discards a lead for a question nobody + # asked. Keeping it as anything settled asserts a clean off-target the run + # never produced. Present, in a state that says the statement could not be + # settled, naming the position responsible. + r = _run(off_target_bed, *BASE) + assert r.returncode == 0, r.stderr + + unasked = _row(off_target_bed, "KA", "OFF3") + assert unasked["state"] == "never asked" + assert unasked["unreliableReason"] == "never-offered" + assert int(unasked["cellsCouldAnswer"]) == 0 # no cell of KA was ever offered OFF3 + + # And the positions S1 did ask are settled, so the clonotype is unsettled + # by exactly one position rather than by a bed that says nothing. + assert _row(off_target_bed, "KA", "TARGET")["state"] == "bound" + assert _row(off_target_bed, "KA", "OFF1")["state"] == "not bound" + assert _row(off_target_bed, "KA", "OFF2")["state"] == "not bound" + + +def test_a_bound_off_target_survives_beside_an_unasked_one(off_target_bed): + # The other half of the check. The obvious way to satisfy the scenario above is to let any + # unsettled position make the whole statement unsettled. That sends a demonstrated off-target + # binder back as a maybe, silently, which is the direction that costs money. KB's bound OFF1 must + # reach the output so a downstream statement can fail KB on it. + r = _run(off_target_bed, *BASE) + assert r.returncode == 0, r.stderr + + assert _row(off_target_bed, "KB", "OFF2")["state"] == "never asked" + + bound_off_target = _row(off_target_bed, "KB", "OFF1") + assert bound_off_target["state"] == "bound" + assert int(bound_off_target["cellsAnswered"]) == 3 + + # The same identity settled in opposite directions for the two clonotypes, + # each from its own cells. A pipeline that folded the unasked position into + # the whole statement would make these two look alike. + assert _row(off_target_bed, "KA", "OFF1")["state"] == "not bound" + + # `set_counts` is what a ranked list is built from, so KB's disqualifying + # bind has to be countable there too: target plus off-target, two bound. + counts = pl.read_csv(off_target_bed / "result_set_counts.csv", infer_schema_length=0) + assert int(counts.filter(pl.col("setId") == "KB").row(0, named=True)["boundCount"]) == 2 + + +# --------------------------------------------------------------------------- +# Support travels with the reading. +# --------------------------------------------------------------------------- + + +@pytest.fixture +def support_bed(tmp_path): + """One clonotype spanning two samples whose panels share nothing but the comparator. + + Forty of the clone's cells sit in S1, which offered AGA and not AGB. Three sit in S2, which + offered AGB and not AGA. Both positions bind, so the states are identical and the only thing + separating a reading resting on forty cells from one resting on three is the support carried + beside it. + """ + (tmp_path / "panel.csv").write_text( + "Samples,Name,Sequence,Type\nS1,AgA,AGA,Target\nS1,Ctrl,CTRL,Control\nS2,AgB,AGB,Target\nS2,Ctrl,CTRL,Control\n" + ) + deep = [f"d{i:02d}" for i in range(40)] + thin = ["t1", "t2", "t3"] + rows = ["sampleId,cellId,tag,umiCount"] + for cell in deep: + rows.append(f"S1,{cell},CTRL,{COMPARATOR}") + rows.append(f"S1,{cell},AGA,{BINDING}") + for cell in thin: + rows.append(f"S2,{cell},CTRL,{COMPARATOR}") + rows.append(f"S2,{cell},AGB,{BINDING}") + (tmp_path / "counts.csv").write_text("\n".join(rows) + "\n") + (tmp_path / "linker.csv").write_text( + "sampleId,cellId,setId\n" + + "".join(f"S1,{cell},K1\n" for cell in deep) + + "".join(f"S2,{cell},K1\n" for cell in thin) + ) + return tmp_path + + +def test_a_reading_on_forty_cells_and_one_on_three_are_distinguishable(support_bed): + # Cells of one clonotype are replicates of one measurement, so how many could answer is how much + # confidence the reading deserves. Both positions here read *bound*, so a row carrying only the + # state makes a decision taken on three cells indistinguishable from one taken on forty -- inside + # a single clonotype's row set, which is where the two really do differ. + r = _run(support_bed, *BASE) + assert r.returncode == 0, r.stderr + + deep = _row(support_bed, "K1", "AGA") + thin = _row(support_bed, "K1", "AGB") + assert deep["state"] == thin["state"] == "bound" + + assert (int(deep["cellsCouldAnswer"]), int(deep["cellsAnswered"])) == (40, 40) + assert (int(thin["cellsCouldAnswer"]), int(thin["cellsAnswered"])) == (3, 3) + assert int(deep["cellsAnswered"]) != int(thin["cellsAnswered"]) + + # Agreement travels the same way: both readings are unanimous, and a + # reader has the figure rather than having to infer it from the states. + assert float(deep["agreement"]) == 1.0 and float(thin["agreement"]) == 1.0 + + +# --------------------------------------------------------------------------- +# Every cell set aside: the question was put and the data cannot settle it. +# --------------------------------------------------------------------------- + + +@pytest.fixture +def gated_bed(tmp_path): + """A clonotype whose every cell sits in high comparator background. + + The comparator reads 20 in every cell, so a gate at 10 sets all three aside. The antigen count of + 500 scores 99.85 against a comparator of 20 and binds outright with the gate off, which is what + makes the *unreliable* reading the gate's doing rather than an absence of signal. Nothing injects + an `unreliable` row. The state is reached by running the same bed twice, once through the gate and + once past it. + """ + (tmp_path / "panel.csv").write_text("Samples,Name,Sequence,Type\nS1,AgA,AGA,Target\nS1,Ctrl,CTRL,Control\n") + cells = ("c1", "c2", "c3") + rows = ["sampleId,cellId,tag,umiCount"] + for cell in cells: + # 20 leaves these cells admissible until the gate is what sets them aside. It is also below + # the high-reference observation line of 100, so the bed is not also exercising that + # measurement. + rows.append(f"S1,{cell},CTRL,20") + rows.append(f"S1,{cell},AGA,{BINDING}") + (tmp_path / "counts.csv").write_text("\n".join(rows) + "\n") + (tmp_path / "linker.csv").write_text("sampleId,cellId,setId\n" + "".join(f"S1,{cell},K1\n" for cell in cells)) + return tmp_path + + +def test_a_set_whose_every_cell_was_gated_reads_unreliable_and_never_not_bound(gated_bed): + # The cells were dropped because their readings could not be trusted, so nothing about the + # receptor was established. *Not bound* would assert a clean reading the run never produced, in + # the direction that costs money. *Never asked* would claim the experiment did not put the + # question, which it did. + r = _run(gated_bed, *BASE, "--gate-threshold", "10") + assert r.returncode == 0, r.stderr + + meta = json.loads((gated_bed / "result_run_meta.json").read_text()) + assert meta["cellsSetAside"] == 3, "the gate has to be what removed them" + + gated = _row(gated_bed, "K1", "AGA") + assert gated["state"] == "unreliable" + assert gated["unreliableReason"] == "all-cells-gated" + # The question was put to three cells and none of them could answer it. + # Reporting zero on both would lose the distinction the state carries. + assert (int(gated["cellsCouldAnswer"]), int(gated["cellsAnswered"])) == (3, 0) + + # Same files, gate off: the identity binds outright. Without this the test + # would pass just as well over a bed with no signal in it, where the + # *unreliable* reading says nothing about the gate. + assert _run(gated_bed, *BASE).returncode == 0 + assert _row(gated_bed, "K1", "AGA")["state"] == "bound" + + +def test_420_an_unasked_off_target_is_reachable_only_because_the_panel_is_keyed_by_sample(): + """`420-unasked-off-target`, at the grain the keying decides. + + A clonotype whose cells came from a sample whose panel omitted an off-target must read *never + asked* there, neither satisfied nor violated. That state is only reachable because what a sample + offered is worked out per sample (`242`), and under a reused panel it is only CORRECT because the + identity a barcode carries is read from that sample's own declaration. + + The second half is the other side of `420`: the identity is still in the universe, so the unasked + position has a row to sit in rather than vanishing from the answer (`205`). + """ + from emit_verdicts import _build_grouping + from panel import identity_universe, offered_identities + + panel = pl.DataFrame( + { + "tag": ["T1", "T2", "T1", "T2", "T3"], + "sample": ["s1", "s1", "s2", "s2", "s2"], + "Identity": ["target", "offA", "target", "offA", "offB"], + } + ) + grouping, rule_id, ungrouped, _declared = _build_grouping( + {"by": "property", "column": "Identity"}, panel, properties={}, reference_tags=set() + ) + assert rule_id == "property:Identity" + assert ungrouped == [] + + # s1's panel omits offB entirely, so a set drawn only from s1 was never asked there. + assert offered_identities(panel, grouping, ["s1"]) == {"target", "offA"} + assert offered_identities(panel, grouping, ["s2"]) == {"target", "offA", "offB"} + + # ...and offB is still in the universe, so that position exists to hold *never asked*. + assert identity_universe(panel, grouping) == {"target", "offA", "offB"} + + +def test_a_reused_barcode_is_read_as_what_its_own_sample_declared(): + """`260-panel-file-authority@3.0` — the case the keying exists for. + + One barcode drawn from a small fixed pool carries a different antigen in each sample, which is + how a study covers more antigens than it has tags. Each sample's cells must be read against the + antigen that sample declared, and the identity universe holds both. + """ + from emit_verdicts import _build_grouping + from panel import identity_universe + + panel = pl.DataFrame({"tag": ["T1", "T1"], "sample": ["s1", "s2"], "Identity": ["antigenA", "antigenB"]}) + grouping, _, ungrouped, _declared = _build_grouping( + {"by": "property", "column": "Identity"}, panel, properties={}, reference_tags=set() + ) + assert grouping == {("T1", "s1"): "antigenA", ("T1", "s2"): "antigenB"} + assert identity_universe(panel, grouping) == {"antigenA", "antigenB"} + # Nothing fell back: neither declaration is a "disagreement" under a per-sample panel. + assert ungrouped == [] diff --git a/software/per-cell-metrics/test/test_combine.py b/software/per-cell-metrics/test/test_combine.py new file mode 100644 index 0000000..ccab84e --- /dev/null +++ b/software/per-cell-metrics/test/test_combine.py @@ -0,0 +1,782 @@ +import random + +import polars as pl +import pytest +from combine import ( + DEFAULT_MIN_VOTERS, + SetUnreliableReason, + attach_competitor_notes, + combine_cells, + self_disagreement, + set_counts, +) +from verdict import Admissibility, State, combine_tags_to_identities, gate_cells, read_states + +B, N, U, NA = (State.BOUND.value, State.NOT_BOUND.value, State.UNRELIABLE.value, State.NEVER_ASKED.value) + + +# No setId column: `combine_cells` derives which set a row belongs to from +# `cells_by_set` alone, matching `read_states`' actual output shape. +_STATES_SCHEMA = { + "sampleId": pl.String, + "cellId": pl.String, + "identity": pl.String, + "state": pl.String, +} + + +def _states(rows): + return pl.DataFrame(rows, orient="row", schema=_STATES_SCHEMA) + + +def _row(out, identity): + return out.filter(pl.col("identity") == identity).row(0, named=True) + + +# A permissive admissibility used by every test whose cells all have an explicit row in `states`. No +# cell is silent, so asked == observed for every identity and the silent terms are 0 regardless of +# what this holds. +_NEUTRAL = Admissibility({}, set()) + + +def test_majority_wins(): + df = _states([("S1", "c1", "A", B), ("S1", "c2", "A", B), ("S1", "c3", "A", N)]) + cells_by_set = {"s1": [("S1", "c1"), ("S1", "c2"), ("S1", "c3")]} + out = combine_cells(df, universe={"A"}, offered={"S1": {"A"}}, cells_by_set=cells_by_set, admissibility=_NEUTRAL) + r = _row(out, "A") + assert r["state"] == B and r["cellsAnswered"] == 3 and r["agreement"] == 2 / 3 + + +def test_vote_is_per_identity_so_a_set_can_bind_several(): + df = _states([("S1", "c1", i, B) for i in ("A", "C")]) + cells_by_set = {"s1": [("S1", "c1")]} + out = combine_cells( + df, universe={"A", "C"}, offered={"S1": {"A", "C"}}, cells_by_set=cells_by_set, admissibility=_NEUTRAL + ).sort("identity") + assert out["state"].to_list() == [B, B] + + +def test_a_tie_cannot_be_settled(): + df = _states([("S1", "c1", "A", B), ("S1", "c2", "A", N)]) + cells_by_set = {"s1": [("S1", "c1"), ("S1", "c2")]} + out = combine_cells(df, {"A"}, {"S1": {"A"}}, cells_by_set, _NEUTRAL) + r = _row(out, "A") + assert r["state"] == U + assert r["unreliableReason"] == SetUnreliableReason.TIE.value + + +def test_a_three_way_split_that_ties_at_the_top_is_also_unreliable(): + # Not just the minimal 1-vs-1 tie: three cells settle bound, three settle + # not bound. The tie check must compare the leading counts, not special- + # case a count of one. + df = _states([("S1", f"b{i}", "A", B) for i in range(3)] + [("S1", f"n{i}", "A", N) for i in range(3)]) + cells_by_set = {"s1": [("S1", f"b{i}") for i in range(3)] + [("S1", f"n{i}") for i in range(3)]} + r = _row(combine_cells(df, {"A"}, {"S1": {"A"}}, cells_by_set, _NEUTRAL), "A") + assert r["state"] == U and r["cellsAnswered"] == 6 + assert r["unreliableReason"] == SetUnreliableReason.TIE.value + + +def test_never_asked_comes_only_from_not_being_offered(): + # Z is in the universe and NOT offered -> never asked. + df = _states([("S1", "c1", "A", B)]) + cells_by_set = {"s1": [("S1", "c1")]} + out = combine_cells( + df, universe={"A", "Z"}, offered={"S1": {"A"}}, cells_by_set=cells_by_set, admissibility=_NEUTRAL + ) + r = _row(out, "Z") + assert r["state"] == NA + assert r["cellsCouldAnswer"] == 0 + assert r["unreliableReason"] == SetUnreliableReason.NEVER_OFFERED.value + + +def test_an_offered_identity_nobody_bound_is_not_bound_not_never_asked(): + # Explicit rows, every one not-bound: offered, everybody read zero, so + # the verdict is not bound, never never-asked. + df = _states([("S1", "c1", "A", N), ("S1", "c2", "A", N)]) + cells_by_set = {"s1": [("S1", "c1"), ("S1", "c2")]} + r = _row(combine_cells(df, {"A"}, {"S1": {"A"}}, cells_by_set, _NEUTRAL), "A") + assert r["state"] == N and r["state"] != NA + + +def test_silent_cells_vote_an_antigen_every_cell_failed_still_reads_not_bound(): + # The defect this reduction exists to avoid: five cells asked about A, none has a row in `states` + # at all, because tag-stat never observed a reading for any of them, and all five are admissible. + # Silent admissible cells resolve not bound, so the set must read not bound with all five voting. + # Never unreliable, which is what happens if silent cells are simply excluded from the tally, and + # never never-asked. + df = _states([]) + members = [("S1", f"c{i}") for i in range(5)] + cells_by_set = {"s1": members} + admissibility = Admissibility({k: 5 for k in members}, set()) + r = _row(combine_cells(df, {"A"}, {"S1": {"A"}}, cells_by_set, admissibility), "A") + assert r["state"] == N + assert r["cellsAnswered"] == 5 + assert r["cellsCouldAnswer"] == 5 + assert r["agreement"] == 1.0 + + +def test_unsettled_cells_do_not_vote_but_do_count_as_could_answer(): + df = _states([("S1", "c1", "A", B), ("S1", "c2", "A", U), ("S1", "c3", "A", U)]) + cells_by_set = {"s1": [("S1", "c1"), ("S1", "c2"), ("S1", "c3")]} + r = _row(combine_cells(df, {"A"}, {"S1": {"A"}}, cells_by_set, _NEUTRAL), "A") + assert r["state"] == B and r["cellsAnswered"] == 1 and r["cellsCouldAnswer"] == 3 + + +def test_a_verdict_may_rest_on_one_cell_and_says_so(): + assert DEFAULT_MIN_VOTERS == 1 + df = _states([("S1", "c1", "A", B)]) + cells_by_set = {"s1": [("S1", "c1")]} + r = _row(combine_cells(df, {"A"}, {"S1": {"A"}}, cells_by_set, _NEUTRAL), "A") + assert r["state"] == B and r["cellsAnswered"] == 1 + + +def test_below_min_voters_is_unreliable_when_raised(): + df = _states([("S1", "c1", "A", B)]) + cells_by_set = {"s1": [("S1", "c1")]} + r = _row(combine_cells(df, {"A"}, {"S1": {"A"}}, cells_by_set, _NEUTRAL, min_voters=2), "A") + assert r["state"] == U + assert r["unreliableReason"] == SetUnreliableReason.TOO_FEW_VOTERS.value + + +def test_exactly_min_voters_settles(): + # The named value satisfies the condition it names, as elsewhere in this + # project: two settled votes with min_voters=2 must settle, not fail. + df = _states([("S1", "c1", "A", B), ("S1", "c2", "A", B)]) + cells_by_set = {"s1": [("S1", "c1"), ("S1", "c2")]} + r = _row(combine_cells(df, {"A"}, {"S1": {"A"}}, cells_by_set, _NEUTRAL, min_voters=2), "A") + assert r["state"] == B and r["cellsAnswered"] == 2 + + +def test_narrow_majority_stands_and_reports_how_narrow(): + df = _states([("S1", f"c{i}", "A", B) for i in range(6)] + [("S1", f"d{i}", "A", N) for i in range(5)]) + cells_by_set = {"s1": [("S1", f"c{i}") for i in range(6)] + [("S1", f"d{i}") for i in range(5)]} + r = _row(combine_cells(df, {"A"}, {"S1": {"A"}}, cells_by_set, _NEUTRAL), "A") + assert r["state"] == B and r["agreement"] == 6 / 11 + + +def test_exactly_min_agreement_settles_when_raised(): + # 3 bound, 1 not bound -> agreement 0.75. Raising min_agreement to + # exactly 0.75 must still settle: the boundary belongs to the pass side. + df = _states([("S1", f"b{i}", "A", B) for i in range(3)] + [("S1", "n0", "A", N)]) + cells_by_set = {"s1": [("S1", f"b{i}") for i in range(3)] + [("S1", "n0")]} + r = _row(combine_cells(df, {"A"}, {"S1": {"A"}}, cells_by_set, _NEUTRAL, min_agreement=0.75), "A") + assert r["state"] == B and r["agreement"] == 0.75 + + +def test_just_below_min_agreement_is_below_agreement_floor_not_tie(): + # A real majority exists here (3 of 4) -- it is refused only because the + # operator raised min_agreement above it. That is a different reason + # than a tie, which has no majority to refuse. + df = _states([("S1", f"b{i}", "A", B) for i in range(3)] + [("S1", "n0", "A", N)]) + cells_by_set = {"s1": [("S1", f"b{i}") for i in range(3)] + [("S1", "n0")]} + r = _row(combine_cells(df, {"A"}, {"S1": {"A"}}, cells_by_set, _NEUTRAL, min_agreement=0.76), "A") + assert r["state"] == U + assert r["unreliableReason"] == SetUnreliableReason.BELOW_AGREEMENT_FLOOR.value + + +def test_a_genuine_tie_still_reads_tie_even_when_min_agreement_would_also_fail_it(): + # A fixture-coincidence trap: a tie's agreement is exactly 0.5, so any min_agreement above 0.5 + # would ALSO fail it, and a fixture where both conditions hold cannot tell which branch produced + # the answer. Raise min_agreement to 0.6 on the same 1-vs-1 tie from test_a_tie_cannot_be_settled + # and confirm the reason is still TIE, not BELOW_AGREEMENT_FLOOR. The tie check must run and win + # regardless of where the floor sits. + df = _states([("S1", "c1", "A", B), ("S1", "c2", "A", N)]) + cells_by_set = {"s1": [("S1", "c1"), ("S1", "c2")]} + r = _row(combine_cells(df, {"A"}, {"S1": {"A"}}, cells_by_set, _NEUTRAL, min_agreement=0.6), "A") + assert r["state"] == U + assert r["unreliableReason"] == SetUnreliableReason.TIE.value + + +def test_set_with_every_cell_set_aside_is_unreliable_through_the_real_pipeline(): + # Driven through read_states, not fed a synthetic UNRELIABLE row. A gate excludes both of this + # set's cells, read_states produces the real UNRELIABLE rows from that, and combine_cells must + # still resolve the set to unreliable with reason all-cells-gated -- derived from the cells' own + # UnreliableReason.GATED, not hard-coded. + counts = pl.DataFrame( + [("S1", "c1", "TAG", 500), ("S1", "c2", "TAG", 500)], + orient="row", + schema={"sampleId": pl.String, "cellId": pl.String, "tag": pl.String, "umiCount": pl.Int64}, + ) + identities = combine_tags_to_identities(counts, {("TAG", "S1"): "A"}) + reference = {("S1", "c1"): 900, ("S1", "c2"): 900} + gated, _ = gate_cells(reference, threshold=800) + admissibility = Admissibility(reference, gated) + per_cell = read_states(identities, admissibility, cutoff=75.0) + + # No setId to attach: which set these rows belong to comes from + # cells_by_set below, not from a column on states. + states = per_cell.select("sampleId", "cellId", "identity", "state") + cells_by_set = {"s1": [("S1", "c1"), ("S1", "c2")]} + r = _row(combine_cells(states, {"A"}, {"S1": {"A"}}, cells_by_set, admissibility), "A") + assert r["state"] == U and r["cellsCouldAnswer"] == 2 and r["cellsAnswered"] == 0 + assert r["unreliableReason"] == SetUnreliableReason.ALL_CELLS_GATED.value + + +def test_all_cells_gated_is_not_reported_when_the_reason_mix_is_not_unanimous(): + # One cell gated, one with no comparator at all: the set-wide reason is + # not "all cells gated" (it is not true) but the comparator failure that + # is present, per _dominant_reason's documented priority. + df = _states([]) + members = [("S1", "c1"), ("S1", "c2")] + cells_by_set = {"s1": members} + admissibility = Admissibility({("S1", "c1"): 900}, {("S1", "c1")}) # c2 has no comparator entry + r = _row(combine_cells(df, {"A"}, {"S1": {"A"}}, cells_by_set, admissibility), "A") + assert r["state"] == U + assert r["unreliableReason"] == SetUnreliableReason.NO_COMPARATOR.value + + +def test_cellscouldanswer_is_not_a_row_count(): + # THE defect this reduction exists to fix. 40 cells. Only 3 have a row in + # `states`, the other 37 are silent and admissible. cellsCouldAnswer must + # reflect all 40 cells asked (their sample offered A), never the 3 rows. + explicit = [("S1", "c0", "A", B), ("S1", "c1", "A", B), ("S1", "c2", "A", N)] + df = _states(explicit) + members = [("S1", "c0"), ("S1", "c1"), ("S1", "c2")] + [("S1", f"s{i}") for i in range(37)] + cells_by_set = {"s1": members} + admissibility = Admissibility({k: 5 for k in members}, set()) + r = _row(combine_cells(df, {"A"}, {"S1": {"A"}}, cells_by_set, admissibility), "A") + assert r["cellsCouldAnswer"] == 40 # not 3 + assert r["cellsAnswered"] == 40 # 2 explicit bound + 1 explicit not-bound + 37 silent not-bound + assert r["state"] == N # 38 not-bound votes beat 2 bound + + +def test_a_set_spanning_two_panels_counts_only_the_asked_cells_and_does_not_inflate_silent_unreliable(): + # S1 offers A, S2 offers B (not A). The set holds cells from both. For + # identity A: cellsCouldAnswer must count only S1's cells, and S2's gated + # cell -- which never offered A -- must not inflate silentUnreliable at A. + df = _states([]) + members = [("S1", "c1"), ("S1", "c2"), ("S2", "c3"), ("S2", "c4")] + cells_by_set = {"s1": members} + # S1's cells are admissible. S2's c3 is gated, and c4 has a normal reference. + reference = {("S1", "c1"): 5, ("S1", "c2"): 5, ("S2", "c3"): 900, ("S2", "c4"): 5} + admissibility = Admissibility(reference, {("S2", "c3")}) + out = combine_cells(df, {"A", "B"}, {"S1": {"A"}, "S2": {"B"}}, cells_by_set, admissibility) + + row_a = _row(out, "A") + assert row_a["cellsCouldAnswer"] == 2 # only S1's two cells, not all four + assert row_a["state"] == N # both S1 cells silent and admissible -> not bound + assert row_a["cellsAnswered"] == 2 + + row_b = _row(out, "B") + assert row_b["cellsCouldAnswer"] == 2 # only S2's two cells + # S2's gated cell counts against B (which S2 offers), and its silent + # not-bound cell (c4) settles: one voter, one vote, not bound. + assert row_b["cellsAnswered"] == 1 + assert row_b["state"] == N + + +def test_a_row_for_a_cell_no_set_lists_is_ignored(): + # A stray row for a cell absent from every set's membership must not + # vote: cellsAnswered must never exceed cellsCouldAnswer. Before the fix, + # a stray row like this counted toward the set it happened to name in a + # setId column. There is no such column now, only cells_by_set, and this + # cell is not in it. + df = _states([("S1", "c1", "A", B), ("S1", "stray", "A", B)]) + cells_by_set = {"s1": [("S1", "c1")]} + r = _row(combine_cells(df, {"A"}, {"S1": {"A"}}, cells_by_set, _NEUTRAL), "A") + assert r["cellsCouldAnswer"] == 1 + assert r["cellsAnswered"] == 1 + assert r["cellsAnswered"] <= r["cellsCouldAnswer"] + + +def test_a_cell_in_two_sets_fails_naming_cells_by_set(): + # A cell listed under two different set ids is a malformed cells_by_set, + # not a silent_tally precondition violation: the failure must name the + # thing that is actually wrong. + cells_by_set = {"s1": [("S1", "c1")], "s2": [("S1", "c1")]} + df = _states([]) + # ValueError rather than AssertionError, and the type is the point: an `assert` is stripped + # under -O, and this guard stripped does not crash -- it returns a wrong answer. Pinning the + # type here is what keeps it from quietly becoming strippable again. + with pytest.raises(ValueError, match="cells_by_set"): + combine_cells(df, {"A"}, {"S1": {"A"}}, cells_by_set, _NEUTRAL) + + +def test_dominant_reason_raises_rather_than_falling_through_to_thin_comparator(): + # A malformed but constructible input: `states` claims this cell is + # UNRELIABLE while `admissibility` says it is fine -- a real comparator, + # not gated, not thin. That contradiction is what lets an admissible key + # reach _dominant_reason and fall through to NO_COMPARATOR. It must raise + # instead of reporting a comparator problem for a cell whose comparator is + # fine. + df = _states([("S1", "c1", "A", U)]) + cells_by_set = {"s1": [("S1", "c1")]} + admissibility = Admissibility({("S1", "c1"): 10}, set()) + # ValueError rather than AssertionError, and the type is the point: an `assert` is stripped + # under -O, and this guard stripped does not crash -- it returns a wrong answer. Pinning the + # type here is what keeps it from quietly becoming strippable again. + with pytest.raises(ValueError): + combine_cells(df, {"A"}, {"S1": {"A"}}, cells_by_set, admissibility) + + +def _verdicts(rows): + return pl.DataFrame(rows, orient="row", schema={"setId": pl.String, "identity": pl.String, "state": pl.String}) + + +def _competitor_row(out, identity): + return out.filter(pl.col("identity") == identity).row(0, named=True) + + +def test_negative_beside_a_bound_competitor_names_it(): + out = attach_competitor_notes(_verdicts([("s1", "A", B), ("s1", "C", N)]), [{"A", "C"}]) + r = _competitor_row(out, "C") + assert r["competedWith"] == "A" and r["state"] == N + + +def test_a_statement_can_test_the_note(): + out = attach_competitor_notes(_verdicts([("s1", "A", B), ("s1", "C", N)]), [{"A", "C"}]) + assert _competitor_row(out, "C")["wasCompeted"] == "true" + assert _competitor_row(out, "A")["wasCompeted"] == "false" + + +def test_no_note_where_no_competitor_was_bound(): + out = attach_competitor_notes(_verdicts([("s1", "A", N), ("s1", "C", N)]), [{"A", "C"}]) + assert _competitor_row(out, "C")["competedWith"] is None + + +def test_no_note_on_a_bound_identity(): + out = attach_competitor_notes(_verdicts([("s1", "A", B), ("s1", "C", B)]), [{"A", "C"}]) + assert out["competedWith"].to_list() == [None, None] + + +def test_no_note_without_a_declared_group(): + out = attach_competitor_notes(_verdicts([("s1", "A", B), ("s1", "C", N)]), []) + assert out["competedWith"].to_list() == [None, None] + + +def test_notes_do_not_leak_across_sets(): + out = attach_competitor_notes(_verdicts([("s1", "A", B), ("s2", "C", N)]), [{"A", "C"}]) + assert _competitor_row(out.filter(pl.col("setId") == "s2"), "C")["competedWith"] is None + + +def test_several_bound_competitors_are_all_named(): + out = attach_competitor_notes(_verdicts([("s1", "A", B), ("s1", "B", B), ("s1", "C", N)]), [{"A", "B", "C"}]) + assert _competitor_row(out, "C")["competedWith"] == "A, B" + + +def test_was_competed_is_the_string_false_never_null_with_no_declared_groups(): + # wasCompeted is the predicate a downstream statement filters on. With no contending groups at + # all, every row's flag must still be the literal string "false". A null here would make + # "wasCompeted == false" fail to match the exact rows the flag exists to describe. + out = attach_competitor_notes(_verdicts([("s1", "A", B), ("s1", "C", N)]), []) + assert out["wasCompeted"].to_list() == ["false", "false"] + assert out["wasCompeted"].dtype == pl.String + + +def test_was_competed_is_the_string_false_never_null_with_declared_groups_present(): + # Same requirement, but with a declared group in play and a row that simply has no bound rival: + # the flag column must not switch to null just because contention was possible elsewhere in the + # frame. + out = attach_competitor_notes(_verdicts([("s1", "A", N), ("s1", "C", N)]), [{"A", "C"}]) + assert out["wasCompeted"].to_list() == ["false", "false"] + + +def test_no_note_on_an_unreliable_reading(): + # An UNRELIABLE identity made no settled comparison, so it has no + # negative for a competitor to sit beside -- naming one would assert a + # comparison this run never made. + out = attach_competitor_notes(_verdicts([("s1", "A", B), ("s1", "C", U)]), [{"A", "C"}]) + r = _competitor_row(out, "C") + assert r["competedWith"] is None + assert r["wasCompeted"] == "false" + + +def test_no_note_on_a_never_asked_reading(): + out = attach_competitor_notes(_verdicts([("s1", "A", B), ("s1", "C", NA)]), [{"A", "C"}]) + r = _competitor_row(out, "C") + assert r["competedWith"] is None + assert r["wasCompeted"] == "false" + + +def test_overlapping_declared_groups_union_their_bound_competitors(): + # C sits in two declared groups, {A, C} and {C, D}, with A and D each + # bound in only one of them. The note names both: the union of bound + # competitors across every group that contains the identity, not just + # the first matching group. + out = attach_competitor_notes( + _verdicts([("s1", "A", B), ("s1", "D", B), ("s1", "C", N)]), + [{"A", "C"}, {"C", "D"}], + ) + assert _competitor_row(out, "C")["competedWith"] == "A, D" + + +def test_competitor_names_are_joined_in_sorted_order(): + # Three bound rivals whose declared-group and bound-set iteration order + # is not alphabetical. Only a sorted join reliably reads "Bee, Mango, + # Zebra" run after run. A byte-stable column depends on this. + out = attach_competitor_notes( + _verdicts([("s1", "Zebra", B), ("s1", "Mango", B), ("s1", "Bee", B), ("s1", "C", N)]), + [{"Zebra", "Mango", "Bee", "C"}], + ) + assert _competitor_row(out, "C")["competedWith"] == "Bee, Mango, Zebra" + + +def _v(rows): + return pl.DataFrame(rows, orient="row", schema={"setId": pl.String, "identity": pl.String, "state": pl.String}) + + +def test_denominator_is_offered_and_settled(): + v = _v([("s1", f"i{i}", B) for i in range(8)] + [("s1", "i8", U), ("s1", "i9", NA)]) + r = set_counts(v).row(0, named=True) + assert r["boundCount"] == 8 + assert r["settledCount"] == 8 # i8 unsettled, i9 never asked + assert r["offeredCount"] == 9 # never-asked is not offered + assert r["unsettledCount"] == 1 + + +def test_not_bound_is_settled_and_in_the_denominator(): + v = _v([("s1", "a", B), ("s1", "b", N)]) + r = set_counts(v).row(0, named=True) + assert r["boundCount"] == 1 and r["settledCount"] == 2 and r["unsettledCount"] == 0 + + +def test_never_asked_is_outside_the_denominator(): + v = _v([("s1", "a", B), ("s1", "b", NA)]) + r = set_counts(v).row(0, named=True) + assert r["offeredCount"] == 1 and r["settledCount"] == 1 + + +def test_counts_are_in_identities_not_tags(): + # One identity carried on two tags is one row here, so it counts once. + v = _v([("s1", "family", B)]) + assert set_counts(v).row(0, named=True)["boundCount"] == 1 + + +def test_each_set_counted_separately(): + v = _v([("s1", "a", B), ("s2", "a", N)]) + out = set_counts(v).sort("setId") + assert out["boundCount"].to_list() == [1, 0] + + +def test_offered_equals_settled_plus_unsettled_with_all_four_states_present(): + # A fixture carrying BOUND, NOT_BOUND, UNRELIABLE, and NEVER_ASKED at + # once, so the arithmetic relationship is pinned rather than incidentally + # true because some state never appeared. A predicate that counts the + # wrong states (say offeredCount including NEVER_ASKED, or settledCount + # including UNRELIABLE) passes every test above that uses only two or + # three states. This one does not let that slip through. + v = _v([("s1", "a", B), ("s1", "b", N), ("s1", "c", U), ("s1", "d", NA)]) + r = set_counts(v).row(0, named=True) + assert r["offeredCount"] == r["settledCount"] + r["unsettledCount"] + assert r["boundCount"] <= r["settledCount"] + assert r["boundCount"] == 1 + assert r["settledCount"] == 2 + assert r["unsettledCount"] == 1 + assert r["offeredCount"] == 3 + + +def test_a_set_asked_nothing_reports_all_zero_and_a_reader_must_guard_the_divide(): + # Every position NEVER_ASKED: offeredCount is 0, so a downstream reader + # computing boundCount / offeredCount would divide by zero. This pins + # what the row emits -- all zeros -- rather than leaving the shape + # undocumented. The guard against the zero is the caller's job, since + # this function cannot produce a rate for a set that was asked nothing. + v = _v([("s1", "a", NA), ("s1", "b", NA)]) + r = set_counts(v).row(0, named=True) + assert r["boundCount"] == 0 + assert r["offeredCount"] == 0 + assert r["settledCount"] == 0 + assert r["unsettledCount"] == 0 + + +def test_a_set_entirely_unreliable_reads_as_nothing_settled_not_as_a_bind_failure(): + # All positions UNRELIABLE: boundCount=0, settledCount=0, unsettledCount=N. This is the shape a + # fully-gated or comparator-less set produces, and it is the one most likely to be misread + # downstream as "bound none of N". The honest reading is that nothing settled, since no comparison + # was ever made. + v = _v([("s1", "a", U), ("s1", "b", U), ("s1", "c", U)]) + r = set_counts(v).row(0, named=True) + assert r["boundCount"] == 0 + assert r["settledCount"] == 0 + assert r["unsettledCount"] == 3 + assert r["offeredCount"] == 3 + + +def test_output_row_order_is_deterministic_regardless_of_input_row_order(): + # This becomes a p-column, so it must be byte-stable: the same verdicts + # fed in several shuffled row orders must produce one identical output, + # including row order, not merely equal counts. + rows = ( + [("s3", "a", B), ("s3", "b", N)] + + [("s1", "a", B), ("s1", "b", U), ("s1", "c", NA)] + + [("s2", "a", N), ("s2", "b", N)] + ) + baseline = set_counts(_v(rows)) + + rng = random.Random(1234) + for _ in range(5): + shuffled = list(rows) + rng.shuffle(shuffled) + out = set_counts(_v(shuffled)) + assert out.equals(baseline) + + +# self_disagreement's states frame is keyed by `key`, an identity or a tag according to `level`, and +# never by `identity`. It is the same sparse per-cell shape `combine_cells` reads, minus a setId +# column: set membership comes only from `cells_by_set`, matching that function's own rule. +_KEY_STATES_SCHEMA = {"sampleId": pl.String, "cellId": pl.String, "key": pl.String, "state": pl.String} + + +def _key_states(rows): + return pl.DataFrame(rows, orient="row", schema=_KEY_STATES_SCHEMA) + + +def _row_for_key(out, key): + return out.filter(pl.col("key") == key).row(0, named=True) + + +def test_a_set_whose_cells_agree_does_not_disagree(): + states = _key_states([("S1", "c1", "A", B), ("S1", "c2", "A", B)]) + cells_by_set = {"s1": [("S1", "c1"), ("S1", "c2")]} + out = self_disagreement(states, {"A"}, {"S1": {"A"}}, cells_by_set, _NEUTRAL) + r = _row_for_key(out, "A") + assert r["cellsCompared"] == 2 + assert r["minorityCells"] == 0 + assert r["disagreementRate"] == 0.0 + + +def test_a_set_whose_cells_differ_disagrees(): + states = _key_states([("S1", "c1", "A", B), ("S1", "c2", "A", N)]) + cells_by_set = {"s1": [("S1", "c1"), ("S1", "c2")]} + out = self_disagreement(states, {"A"}, {"S1": {"A"}}, cells_by_set, _NEUTRAL) + # Two cells splitting one against one: one of them is the minority. Two + # states cap the rate at half, so an even split is the worst attainable. + assert _row_for_key(out, "A")["disagreementRate"] == 0.5 + + +def test_singletons_do_not_contribute(): + states = _key_states([("S1", "c1", "A", B)]) + cells_by_set = {"s1": [("S1", "c1")]} + out = self_disagreement(states, {"A"}, {"S1": {"A"}}, cells_by_set, _NEUTRAL) + r = _row_for_key(out, "A") + assert r["cellsCompared"] == 0 + assert r["disagreementRate"] is None, "nothing to compare says so rather than reading zero" + + +def test_unsettled_cells_are_not_evaluable(): + # c2's row is UNRELIABLE, not silent: it has an explicit row and so is + # not asked through `silent_tally`, but UNRELIABLE never counts as a + # settled vote either. One evaluable cell remains -- a singleton -- so + # the position does not contribute. + states = _key_states([("S1", "c1", "A", B), ("S1", "c2", "A", U)]) + cells_by_set = {"s1": [("S1", "c1"), ("S1", "c2")]} + out = self_disagreement(states, {"A"}, {"S1": {"A"}}, cells_by_set, _NEUTRAL) + assert _row_for_key(out, "A")["cellsCompared"] == 0 + + +def test_pooled_disagreement_counts_cells_not_sets(): + # Two sets at one tag. s1 splits 4 bound and 1 not bound, so one cell sits in + # the minority. s2 splits 1 and 1, so one cell does too. Pooled: 2 minority + # cells over 7 compared. A per-set share would read 2 of 2 -- every set + # disagrees -- which says nothing about how much of the tag is affected. + states = _key_states( + [("S1", f"a{i}", "T1", B) for i in range(4)] + + [("S1", "a4", "T1", N), ("S1", "b1", "T1", B), ("S1", "b2", "T1", N)] + ) + cells_by_set = { + "s1": [("S1", f"a{i}") for i in range(5)], + "s2": [("S1", "b1"), ("S1", "b2")], + } + out = self_disagreement(states, {"T1"}, {"S1": {"T1"}}, cells_by_set, _NEUTRAL) + r = _row_for_key(out, "T1") + assert (r["minorityCells"], r["cellsCompared"]) == (2, 7) + assert r["disagreementRate"] == pytest.approx(2 / 7) + + +def test_pooled_disagreement_caps_at_half(): + # A minority is the smaller side by definition, and there are two states, so + # an even split is the worst value attainable. Nothing can exceed one half. + states = _key_states([("S1", f"c{i}", "T1", B) for i in range(3)] + [("S1", f"d{i}", "T1", N) for i in range(3)]) + cells_by_set = {"s1": [("S1", f"c{i}") for i in range(3)] + [("S1", f"d{i}") for i in range(3)]} + out = self_disagreement(states, {"T1"}, {"S1": {"T1"}}, cells_by_set, _NEUTRAL) + assert _row_for_key(out, "T1")["disagreementRate"] == pytest.approx(0.5) + + +def test_a_tag_no_set_could_compare_says_so_rather_than_reading_zero(): + # Every set is a singleton, so nothing can be compared. Zero would read as + # agreement, which is the opposite of what is known. + states = _key_states([("S1", "c1", "T1", B), ("S1", "c2", "T1", B)]) + cells_by_set = {"s1": [("S1", "c1")], "s2": [("S1", "c2")]} + out = self_disagreement(states, {"T1"}, {"S1": {"T1"}}, cells_by_set, _NEUTRAL) + r = _row_for_key(out, "T1") + assert (r["minorityCells"], r["cellsCompared"]) == (0, 0) + assert r["disagreementRate"] is None + + +def test_the_figure_is_measured_at_the_tag_and_marked_diagnostic_only(): + # One level only. The identity-level figure has nothing to compare against, + # so it cannot separate a faulty reagent from a panel of weak binders. + states = _key_states([("S1", "c1", "AAAA", B)]) + cells_by_set = {"s1": [("S1", "c1")]} + tag = self_disagreement(states, {"AAAA"}, {"S1": {"AAAA"}}, cells_by_set, _NEUTRAL) + assert tag.height == 1 + row = tag.row(0, named=True) + assert row["level"] == "tag" + assert row["diagnosticOnly"] == "true" + + +def test_a_set_too_small_to_compare_is_left_out_of_both_counts(): + # s1 contributes both its cells and one minority cell. s2 is a singleton at + # A: it has no minority of its own, so it enters neither term. Pooling needs + # no cutoff for that -- the two-cell condition does the whole job. + states = _key_states([("S1", "c1", "A", B), ("S1", "c2", "A", N), ("S1", "c3", "A", B)]) + cells_by_set = {"s1": [("S1", "c1"), ("S1", "c2")], "s2": [("S1", "c3")]} + out = self_disagreement(states, {"A"}, {"S1": {"A"}}, cells_by_set, _NEUTRAL) + r = _row_for_key(out, "A") + assert (r["minorityCells"], r["cellsCompared"]) == (1, 2) + assert r["disagreementRate"] == 0.5 + + +def test_silent_cells_flip_agreement_into_disagreement(): + # THE defect this generalisation exists to fix: a set with 2 observed bound cells and 38 silent, + # admissible not-bound cells. Counting rows on the sparse frame sees only the 2 bound rows and + # calls this agreement. The 38 silent cells are settled not-bound votes, and the set actually + # disagrees as badly as it is possible to. + members = [("S1", "c0"), ("S1", "c1")] + [("S1", f"s{i}") for i in range(38)] + states = _key_states([("S1", "c0", "A", B), ("S1", "c1", "A", B)]) + cells_by_set = {"s1": members} + admissibility = Admissibility({k: 5 for k in members}, set()) + out = self_disagreement(states, {"A"}, {"S1": {"A"}}, cells_by_set, admissibility) + r = _row_for_key(out, "A") + # 40 evaluable cells: 38 silent not-bound and 2 observed bound. The majority + # is the 38, so the 2 bound cells are the minority. Counting rows on the + # sparse frame instead would see only those 2, call them a set that agrees, + # and report nothing to compare at all. + assert (r["minorityCells"], r["cellsCompared"]) == (2, 40) + assert r["disagreementRate"] == pytest.approx(0.05) + + +def test_one_observed_positive_among_many_silent_negatives_is_evaluable(): + # A single explicit row is a singleton by row count alone, but 19 silent, + # admissible cells settle not-bound alongside it: 20 evaluable cells, not + # a discarded singleton. + members = [("S1", "c0")] + [("S1", f"s{i}") for i in range(19)] + states = _key_states([("S1", "c0", "A", B)]) + cells_by_set = {"s1": members} + admissibility = Admissibility({k: 5 for k in members}, set()) + out = self_disagreement(states, {"A"}, {"S1": {"A"}}, cells_by_set, admissibility) + r = _row_for_key(out, "A") + # 20 evaluable cells, the one bound cell being the minority of its own set. + assert (r["minorityCells"], r["cellsCompared"]) == (1, 20) + assert r["disagreementRate"] == pytest.approx(0.05) + + +def test_all_silent_not_bound_cells_agree(): + # The mirror of the defect test: every cell of the set is silent and + # admissible, so every one settles not-bound. All evaluable cells give + # the same settled state, so the set agrees with itself -- this must not + # be over-corrected into calling every silent set a disagreement. + members = [("S1", f"s{i}") for i in range(5)] + states = _key_states([]) + cells_by_set = {"s1": members} + admissibility = Admissibility({k: 5 for k in members}, set()) + out = self_disagreement(states, {"A"}, {"S1": {"A"}}, cells_by_set, admissibility) + r = _row_for_key(out, "A") + assert (r["minorityCells"], r["cellsCompared"]) == (0, 5) + assert r["disagreementRate"] == 0.0 + + +def test_self_disagreement_output_is_deterministic_regardless_of_input_row_order(): + # This becomes a p-column, so it must be byte-stable across row orders. + rows = [ + ("S1", "c0", "A", B), + ("S1", "c1", "A", N), + ("S1", "d0", "B", B), + ("S1", "d1", "B", B), + ("S2", "e0", "A", N), + ] + cells_by_set = { + "s1": [("S1", "c0"), ("S1", "c1")], + "s2": [("S1", "d0"), ("S1", "d1")], + "s3": [("S2", "e0")], + } + universe = {"A", "B"} + offered = {"S1": {"A", "B"}, "S2": {"A", "B"}} + baseline = self_disagreement(_key_states(rows), universe, offered, cells_by_set, _NEUTRAL) + + rng = random.Random(2026) + for _ in range(5): + shuffled = list(rows) + rng.shuffle(shuffled) + out = self_disagreement(_key_states(shuffled), universe, offered, cells_by_set, _NEUTRAL) + assert out.equals(baseline) + + +def test_a_position_reports_its_bound_cells_even_when_the_majority_is_not_bound(): + # 206's expansion shows "at each identity, how many of its cells read bound". That is neither the + # majority nor cellsAnswered: a 3-of-10 minority is exactly what a reader opens the expansion to + # see, and it has to survive a "not bound" verdict on the same row. Deriving it from + # `agreement * cellsAnswered` would give 7 here -- the majority's share, of the wrong state. + df = _states([("S1", f"c{i}", "A", B) for i in range(3)] + [("S1", f"c{i}", "A", N) for i in range(3, 10)]) + cells_by_set = {"s1": [("S1", f"c{i}") for i in range(10)]} + out = combine_cells(df, {"A"}, {"S1": {"A"}}, cells_by_set, _NEUTRAL) + r = _row(out, "A") + assert r["state"] == N + assert r["cellsAnswered"] == 10 + assert r["cellsBound"] == 3 + + +def test_a_never_asked_position_reports_no_bound_cells(): + # No tally exists for a position the experiment never put to the clonotype, and 0 is the honest + # count rather than a null the punch value would render as an unreadable field. + df = _states([("S1", "c1", "A", B)]) + out = combine_cells(df, {"A", "Z"}, {"S1": {"A"}}, {"s1": [("S1", "c1")]}, _NEUTRAL) + r = _row(out, "Z") + assert r["state"] == NA + assert r["cellsBound"] == 0 + + +def test_cells_not_bound_completes_cells_answered_on_every_row(): + # cellsNotBound is not a free-standing count. SETTLED holds only BOUND and NOT_BOUND (see the + # module docstring's four-state model), so it is the other half of the same pair cellsBound already + # reports. This checks the pairing across every shape combine_cells produces -- a settled majority + # either way, a tie, a floor refusal, too few voters, a position with no tally at all, and a + # position never offered -- over every row of each frame, not one identity picked out by `_row`. + def _assert_invariant(out: pl.DataFrame) -> None: + for row in out.iter_rows(named=True): + assert row["cellsBound"] + row["cellsNotBound"] == row["cellsAnswered"], row + + # Settled bound, settled not-bound, and a tie, plus Z which nobody offers -> never-asked. + df = _states( + [("S1", "b1", "A", B), ("S1", "b2", "A", B), ("S1", "n1", "A", N)] + + [("S1", "x1", "B", N), ("S1", "x2", "B", N)] + + [("S1", "t1", "C", B), ("S1", "t2", "C", N)] + ) + cells_by_set = { + "sBound": [("S1", "b1"), ("S1", "b2"), ("S1", "n1")], + "sNotBound": [("S1", "x1"), ("S1", "x2")], + "sTie": [("S1", "t1"), ("S1", "t2")], + } + out = combine_cells( + df, + universe={"A", "B", "C", "Z"}, + offered={"S1": {"A", "B", "C"}}, + cells_by_set=cells_by_set, + admissibility=_NEUTRAL, + ) + _assert_invariant(out) + z = _row(out.filter(pl.col("setId") == "sBound"), "Z") + assert z["state"] == NA and z["cellsAnswered"] == 0 + + # Too few voters: one voter, min_voters raised to 2. + out_few = combine_cells( + _states([("S1", "c1", "A", B)]), {"A"}, {"S1": {"A"}}, {"s1": [("S1", "c1")]}, _NEUTRAL, min_voters=2 + ) + _assert_invariant(out_few) + + # Below the agreement floor: a real 3-of-4 majority, refused only because the floor was raised + # above it. + out_floor = combine_cells( + _states([("S1", f"b{i}", "A", B) for i in range(3)] + [("S1", "n0", "A", N)]), + {"A"}, + {"S1": {"A"}}, + {"s1": [("S1", f"b{i}") for i in range(3)] + [("S1", "n0")]}, + _NEUTRAL, + min_agreement=0.76, + ) + _assert_invariant(out_floor) + + # No tally at all: both of the set's cells individually inadmissible (gated). + admissibility = Admissibility({("S1", "c1"): 900, ("S1", "c2"): 900}, {("S1", "c1"), ("S1", "c2")}) + out_gated = combine_cells(_states([]), {"A"}, {"S1": {"A"}}, {"s1": [("S1", "c1"), ("S1", "c2")]}, admissibility) + _assert_invariant(out_gated) diff --git a/software/per-cell-metrics/test/test_emit_csv_meta.py b/software/per-cell-metrics/test/test_emit_csv_meta.py deleted file mode 100644 index 57d3466..0000000 --- a/software/per-cell-metrics/test/test_emit_csv_meta.py +++ /dev/null @@ -1,69 +0,0 @@ -"""Tests for emit_csv_meta.py — CSV headers + per-column distinct values for the D4 dropdowns.""" - -import json -import pathlib -import subprocess -import sys - -SRC = pathlib.Path(__file__).parents[1] / "src" / "emit_csv_meta.py" - - -def _run(tmp_path, text): - csv = tmp_path / "tags.csv" - csv.write_text(text) - out = tmp_path / "meta.json" - subprocess.run([sys.executable, str(SRC), str(csv), str(out)], check=True) - return json.loads(out.read_text()) - - -def test_columns_in_header_order(tmp_path): - meta = _run(tmp_path, "barcode,antigen,pool\nAAAA,AgX,p1\nCCCC,AgY,p1\n") - assert meta["columns"] == ["barcode", "antigen", "pool"] - - -def test_values_by_column_deduped_and_sorted(tmp_path): - # Each column's distinct values, sorted — the control dropdown reads valuesByColumn[]. - meta = _run(tmp_path, "barcode,antigen,pool\nAAAA,AgY,p1\nCCCC,AgX,p1\nGGGG,AgX,p2\n") - assert meta["valuesByColumn"]["antigen"] == ["AgX", "AgY"] # deduped (AgX twice), sorted - assert meta["valuesByColumn"]["pool"] == ["p1", "p2"] - assert meta["valuesByColumn"]["barcode"] == ["AAAA", "CCCC", "GGGG"] - - -def test_row_count_counts_data_rows(tmp_path): - # rowCount is the number of data rows (header excluded) — the model compares it against a column's - # distinct-value count to detect a barcode mapped on more than one row. - meta = _run(tmp_path, "barcode,antigen\nAAAA,AgX\nCCCC,AgY\nGGGG,AgZ\n") - assert meta["rowCount"] == 3 - # No duplicate barcodes: distinct barcode count equals rowCount. - assert len(meta["valuesByColumn"]["barcode"]) == meta["rowCount"] - - -def test_row_count_ignores_trailing_blank_rows(tmp_path): - meta = _run(tmp_path, "barcode,antigen\nAAAA,AgX\n\n,\n") - assert meta["rowCount"] == 1 - - -def test_row_count_exceeds_distinct_when_barcode_duplicated(tmp_path): - # Same barcode on two rows (sample-specific mapping): distinct barcode count < rowCount. - meta = _run(tmp_path, "barcode,antigen\nAAAA,AgX\nAAAA,AgY\n") - assert meta["rowCount"] == 2 - assert len(meta["valuesByColumn"]["barcode"]) < meta["rowCount"] - - -def test_blank_cells_ignored(tmp_path): - meta = _run(tmp_path, "barcode,antigen\nAAAA,AgX\nCCCC,\n") - assert meta["valuesByColumn"]["antigen"] == ["AgX"] # the empty cell is not a value - - -def test_header_only_csv_emits_empty_value_lists(tmp_path): - meta = _run(tmp_path, "barcode,antigen\n") - assert meta["columns"] == ["barcode", "antigen"] - assert meta["valuesByColumn"] == {"barcode": [], "antigen": []} - - -def test_empty_or_headerless_csv_errors(tmp_path): - csv = tmp_path / "empty.csv" - csv.write_text("") - out = tmp_path / "meta.json" - r = subprocess.run([sys.executable, str(SRC), str(csv), str(out)]) - assert r.returncode != 0 diff --git a/software/per-cell-metrics/test/test_emit_feature_properties.py b/software/per-cell-metrics/test/test_emit_feature_properties.py index b93203b..91dfaf2 100644 --- a/software/per-cell-metrics/test/test_emit_feature_properties.py +++ b/software/per-cell-metrics/test/test_emit_feature_properties.py @@ -57,7 +57,7 @@ def test_no_extra_columns_yields_empty_meta(tmp_path): def test_many_barcodes_one_feature_deduped(tmp_path): - # A feature reached by several barcodes appears once; its (consistent) property is carried through. + # A feature reached by several barcodes appears once. Its (consistent) property is carried through. meta, rows = _run(tmp_path, "tag,feature,species\nAAAA,AGX,human\nTTTT,AGX,human\nCCCC,BGX,cyno\n") assert meta["columns"] == ["species"] assert rows[1:] == [["AGX", "human"], ["BGX", "cyno"]] @@ -76,7 +76,7 @@ def test_sample_column_excluded(tmp_path): def test_custom_role_column_names(tmp_path): - # Roles are configurable: whichever columns the user maps are excluded; the rest pass through. + # Roles are configurable: whichever columns the user maps are excluded. The rest pass through. meta, _ = _run( tmp_path, "barcode,antigen,pool\nAAAA,AgX,p1\nCCCC,AgY,p2\n", @@ -104,6 +104,60 @@ def test_missing_role_column_errors(tmp_path): assert r.returncode != 0 +def _control_rows(tmp_path): + with open(tmp_path / "r_negative_control.csv", newline="") as fh: + return list(csv.reader(fh)) + + +def test_control_feature_marker_emitted(tmp_path): + # --control-feature marks that feature "true" in the dedicated negative-control marker CSV, so the + # workflow surfaces it on the feature axis for VDJ Multiomic Integration to exclude from its metrics. + _run(tmp_path, "tag,feature\nAAAA,AGX\nGGGG,CTRL\n", "--control-feature", "CTRL") + assert _control_rows(tmp_path) == [["feature", "value"], ["CTRL", "true"]] + + +def test_no_control_feature_marker_header_only(tmp_path): + # No control designated -> marker CSV is header-only (no feature marked as the control). + _run(tmp_path, "tag,feature\nAAAA,AGX\nGGGG,BGX\n") + assert _control_rows(tmp_path) == [["feature", "value"]] + + +def test_several_controls_are_all_marked(tmp_path): + # A panel may carry several controls: being a control is a property of the tag, where supplying the + # baseline is a job given to exactly one of them. This file marks controls and nominates nothing, so + # every one given is marked. Repeated flags, in the order given. + _run( + tmp_path, + "tag,feature\nAAAA,AGX\nGGGG,CTRL1\nCCCC,CTRL2\n", + "--control-feature", + "CTRL2", + "--control-feature", + "CTRL1", + ) + assert _control_rows(tmp_path) == [["feature", "value"], ["CTRL2", "true"], ["CTRL1", "true"]] + + +def test_a_control_name_may_contain_a_comma(tmp_path): + # Repeated flags rather than one comma-joined value, so a feature name carrying a comma survives. A + # comma-joined encoding would split this name into two features that do not exist. + _run(tmp_path, 'tag,feature\nAAAA,AGX\nGGGG,"CTRL, batch 2"\n', "--control-feature", "CTRL, batch 2") + assert _control_rows(tmp_path) == [["feature", "value"], ["CTRL, batch 2", "true"]] + + +def test_a_repeated_control_is_marked_once(tmp_path): + # The marker is a set. A duplicate would emit two rows for one feature, and the import would then carry + # the same feature twice on an axis that keys on it. + _run( + tmp_path, + "tag,feature\nAAAA,AGX\nGGGG,CTRL\n", + "--control-feature", + "CTRL", + "--control-feature", + "CTRL", + ) + assert _control_rows(tmp_path) == [["feature", "value"], ["CTRL", "true"]] + + def _rows(text): reader = csv.reader(io.StringIO(text)) header = next(reader) diff --git a/software/per-cell-metrics/test/test_emit_panel.py b/software/per-cell-metrics/test/test_emit_panel.py new file mode 100644 index 0000000..b38b811 --- /dev/null +++ b/software/per-cell-metrics/test/test_emit_panel.py @@ -0,0 +1,94 @@ +"""Behavioral tests for emit_panel.py (Feature Integration software). + +Writes the panel's barcode column out as a plain one-per-line list for mitool's refine-tags whitelist +(`-t FEATURE#file:panel.txt`). Three properties of that output are contract rather than convenience, +and each has a test below: deduplicated, sorted, and one barcode per line. mitool matches against this +file verbatim, and the workflow's pure-template dedup rests on the bytes being stable. So an unsorted +or duplicated file is not cosmetically wrong -- it changes a resource handle and silently costs every +downstream node its cache. + +Run through the CLI like the other tool tests. This is a subprocess entry point, and a caller reaching +past it into `main()` would not exercise the argparse and SystemExit behavior that is most of what the +file does. +""" + +import pathlib +import subprocess +import sys + +SRC = pathlib.Path(__file__).parents[1] / "src" / "emit_panel.py" + + +def _run(tmp_path, csv_text, *args, expect_failure=False): + """Run the tool over `csv_text`, returning the output file's text. + + Asserts success by default: a tool that exits non-zero has written nothing, + so a test that goes on to assert emptiness of the output would pass for the + wrong reason. + """ + src_csv = tmp_path / "tags.csv" + src_csv.write_text(csv_text) + out = tmp_path / "panel.txt" + r = subprocess.run( + [sys.executable, str(SRC), str(src_csv), str(out), *args], + capture_output=True, + text=True, + ) + if expect_failure: + assert r.returncode != 0, f"expected failure, got 0. stdout={r.stdout!r}" + return r.stderr + assert r.returncode == 0, f"exited {r.returncode}. stderr={r.stderr!r}" + return out.read_text() + + +def test_dedupes_sorts_and_writes_one_barcode_per_line(tmp_path): + # The whole contract in one assertion, on exact bytes: AAAA appears twice in the input under two + # different antigen names and must appear once here, and CCCC must follow it rather than lead. + # Asserting the set of lines instead would pass on an unsorted file, which is the case that breaks + # dedup downstream rather than anything a reader would notice. + text = _run(tmp_path, "tag,feature\nCCCC,x\nAAAA,y\nAAAA,z\n") + assert text == "AAAA\nCCCC\n" + + +def test_tag_col_selects_a_renamed_column(tmp_path): + # The panel file's barcode column is whatever the user pointed the block at, + # so the default name is a default and not an assumption. + text = _run(tmp_path, "Sequence,Name\nGGGG,x\nTTTT,y\n", "--tag-col", "Sequence") + assert text == "GGGG\nTTTT\n" + + +def test_missing_column_names_the_column_it_wanted(tmp_path): + stderr = _run(tmp_path, "sequence,feature\nAAAA,x\n", expect_failure=True) + assert "tag" in stderr + + +def test_header_only_input_is_refused(tmp_path): + # An empty whitelist is not an empty correction -- mitool given a whitelist + # of nothing corrects every barcode to nothing. Failing here is the point. + stderr = _run(tmp_path, "tag,feature\n", expect_failure=True) + assert "no feature barcodes" in stderr + + +def test_all_blank_tag_cells_are_refused(tmp_path): + stderr = _run(tmp_path, "tag,feature\n ,x\n,y\n", expect_failure=True) + assert "no feature barcodes" in stderr + + +def test_padded_cells_are_trimmed_and_blank_cells_skipped(tmp_path): + # The barcode is a join key against the counts, whose reader strips for the same reason: " AAAA " + # and "AAAA" are one barcode, and a whitelist carrying the padded form matches nothing. + text = _run(tmp_path, "tag,feature\n AAAA ,x\n,y\nCCCC,z\n") + assert text == "AAAA\nCCCC\n" + + +def test_a_ragged_short_row_is_skipped_rather_than_crashing(tmp_path): + # csv.DictReader fills a short row's missing keys with None, so the tag cell is present-and-None + # rather than absent. `row.get(col, "")` returns None there, because the key exists and the default + # never fires, and .strip() raised AttributeError -- taking the whole run down on one malformed + # line of a user-supplied CSV. + # + # The barcode column is deliberately SECOND here. A short row only truncates the trailing columns, + # so with the barcode first the missing key is the other column and nothing reads it. The first + # version of this test put it first, passed against the unfixed code, and proved nothing. + text = _run(tmp_path, "feature,tag\nx,AAAA\ny\nz,GGGG\n", "--tag-col", "tag") + assert text == "AAAA\nGGGG\n" diff --git a/software/per-cell-metrics/test/test_emit_verdicts.py b/software/per-cell-metrics/test/test_emit_verdicts.py new file mode 100644 index 0000000..d40b6c9 --- /dev/null +++ b/software/per-cell-metrics/test/test_emit_verdicts.py @@ -0,0 +1,3451 @@ +import json +import shutil +import subprocess +import sys +from pathlib import Path + +import polars as pl +import pytest +import qc_rows +from emit_verdicts import _build_grouping, _identity_properties, _linker_frame, undeclared_feature_counts +from panel import ANY_SAMPLE, consistent_properties, property_columns +from qc_measures import DEFAULT_LINES, MEASUREMENTS, Line, Measurement +from verdict import DEFAULT_PANEL_MIN_MEMBERS, ReferenceChoice + +SRC = Path(__file__).resolve().parents[1] / "src" + + +def _run(cwd, *args, expect_failure=False): + """Run the CLI, asserting it succeeded unless the caller wants a failure. + + Success is asserted HERE rather than left to each test, because a crashed run is + invisible to most assertions in this file: the tool writes into `cwd`, so a run that + dies before writing leaves the PREVIOUS run's files in place and every read of them + still succeeds. `test_output_is_byte_stable_across_runs` was the live case -- it + compares two runs' bytes and asserted neither returncode, so a second invocation + crashing on startup compared the first run's files against themselves and passed. + + stderr rides along in the message because a bare `assert returncode == 0` tells you + the run died and not why. + """ + r = subprocess.run( + [sys.executable, str(SRC / "emit_verdicts.py"), *map(str, args)], cwd=cwd, capture_output=True, text=True + ) + if expect_failure: + assert r.returncode != 0, f"expected a non-zero exit, got 0. stdout={r.stdout!r}" + else: + assert r.returncode == 0, f"exited {r.returncode}. stderr={r.stderr!r}" + return r + + +BASE = [ + "counts.csv", + "panel.csv", + "--linker", + "linker.csv", + "--barcode-col", + "Sequence", + "--feature-col", + "Name", + "--sample-col", + "Samples", + "--role-column", + "Type", + "--reference-values", + "Control", + # Stated, because the CLI requires it and nothing below the model picks a rung. This bed declares a + # comparator tag, so the declared rung is the one it is about. A test wanting a different rung passes + # its own --reference-source, which argparse takes as the later value. + "--reference-source", + "declared", + "--output-prefix", + "result", +] + + +@pytest.fixture +def bed(tmp_path): + # The antigen counts clear the shipped cutoff of 75 against a reference of 6: + # specificity_score(500, 6) and specificity_score(600, 6) are both 100, while a + # silent cell scores specificity_score(0, 6), which is ~7.5e-09. Counts of 50 and + # 60 score 3.1 and 7.2 and would read *not bound*, which is a fact about the beta + # score rather than about this pipeline. + (tmp_path / "counts.csv").write_text( + "sampleId,cellId,tag,umiCount\nS1,c1,AAAA,500\nS1,c1,CTRL,6\nS1,c2,AAAA,600\nS1,c2,CTRL,6\nS1,c3,CTRL,6\n" + ) # c3 was asked about AAAA and read nothing + (tmp_path / "panel.csv").write_text("Samples,Name,Sequence,Type\nS1,AgA,AAAA,Target\nS1,Ctrl,CTRL,Control\n") + (tmp_path / "linker.csv").write_text("sampleId,cellId,setId\nS1,c1,K1\nS1,c2,K1\nS1,c3,K1\n") + return tmp_path + + +def test_writes_every_artifact(bed): + r = _run(bed, *BASE) + assert r.returncode == 0, r.stderr + for name in ( + "result_verdicts.csv", + "result_set_counts.csv", + "result_cell_counts.csv", + "result_cell_scalars.csv", + "result_offered.csv", + "result_identity_labels.csv", + "result_identity_properties.csv", + "result_panel_mismatch.csv", + "result_undeclared_barcodes.csv", + "result_run_meta.json", + ): + assert (bed / name).exists(), name + + +def test_a_silent_cell_votes_not_bound(bed): + # c3 has no AAAA row in the counts. It was offered AAAA, so it must vote. + _run(bed, *BASE) + v = pl.read_csv(bed / "result_verdicts.csv") + r = v.filter(pl.col("identity") == "AAAA").row(0, named=True) + assert r["cellsAnswered"] == 3 # not 2 + assert r["state"] == "bound" # 2 of 3 + + +def test_the_reference_tag_gets_no_verdict(bed): + _run(bed, *BASE) + v = pl.read_csv(bed / "result_verdicts.csv") + assert "CTRL" not in v["identity"].to_list() + + +def test_cell_counts_carry_the_re_derivation_material(bed): + _run(bed, *BASE) + c = pl.read_csv(bed / "result_cell_counts.csv") + assert {"sampleId", "cellId", "tag", "umiCount", "referenceCount", "inCellList"} <= set(c.columns) + + +def test_no_score_leaves_the_block(bed): + _run(bed, *BASE) + for f in ("result_cell_scalars.csv", "result_verdicts.csv", "result_cell_counts.csv"): + assert "score" not in pl.read_csv(bed / f).columns + + +def test_run_meta_records_every_choice(bed): + _run(bed, *BASE) + m = json.loads((bed / "result_run_meta.json").read_text()) + for key in ( + "referenceChoice", + "cellListSource", + "floor", + "cutoff", + "minVoters", + "gateThreshold", + "panelMinMembers", + "grouping", + "contending", + "readingsFloored", + "cellsEmptied", + "cellsHighReference", + ): + assert key in m, key + + +def test_none_is_no_longer_a_selectable_rung(bed): + # There is no bottom rung. "none" used to select one -- no baseline exists, every + # verdict reads unreliable -- and a baseline is now required, so the value is not a + # choice a caller can make. Refused by argparse rather than accepted and quietly + # reinterpreted as some other rung. + r = _run(bed, *BASE, "--reference-source", "none", expect_failure=True) + assert r.returncode != 0 + assert "none" in r.stderr + + +def test_a_tag_the_grouping_could_not_place_is_named_in_the_output(bed): + # A property the panel file does not carry narrows what can be answered, and the + # narrowing has to be visible where the answers are. Such a tag keeps its own + # identity rather than vanishing, so a bare barcode sits among the family identities + # -- inferable from the labels, but only this says why it is there. + (bed / "panel.csv").write_text( + "Samples,Name,Sequence,Type,Family\n" + "S1,AgA,AAAA,Target,Spike\n" + "S1,AgB,CCCC,Target,\n" + "S1,Ctrl,CTRL,Control,Reference\n" + ) + (bed / "counts.csv").write_text((bed / "counts.csv").read_text() + "S1,c1,CCCC,40\nS1,c2,CCCC,40\nS1,c3,CCCC,40\n") + r = _run(bed, *BASE, "--grouping", json.dumps({"by": "property", "column": "Family"})) + assert r.returncode == 0, r.stderr + + meta = json.loads((bed / "result_run_meta.json").read_text()) + assert meta["tagsWithoutGroupingValue"] == ["CCCC"] + + identities = set(pl.read_csv(bed / "result_verdicts.csv", infer_schema_length=0)["identity"].to_list()) + assert identities == {"Spike", "CCCC"}, "the unplaceable tag keeps its own identity rather than vanishing" + + +def test_a_tag_grouping_reports_no_unplaceable_tags(bed): + # The default grouping places every tag by construction, so the field is present and + # empty rather than absent: a reader must be able to tell "none" from "not checked". + _run(bed, *BASE) + meta = json.loads((bed / "result_run_meta.json").read_text()) + assert meta["tagsWithoutGroupingValue"] == [] + + +def test_a_tag_noisy_in_one_panel_reads_clean_on_the_panel_it_was_clean_in(bed): + # Two samples declaring different tag sets are two panels, and both declare T00. T00's clonotype splits + # itself in S2 and reads steady in S1. A run-global disagreement rate puts S2's noise on S1's row as + # well, so a reader comparing S1's tags against each other is handed a fault that belongs to S2 -- and + # is sent to re-prepare the wrong panel. The rows are keyed `(tag, panelId)`, so the figure on them has + # to be that panel's. No status is involved: the comparison is the reader's to make. + shared = [f"T{i:02d}" for i in range(5)] + (bed / "panel.csv").write_text( + "Samples,Name,Sequence,Type\n" + + "".join(f"S1,Ag{i},{t},Target\n" for i, t in enumerate(shared)) + + "S1,Ctrl,CTRL,Control\n" + # S2 declares the same five plus one more, which is what makes it a different panel. + + "".join(f"S2,Ag{i},{t},Target\n" for i, t in enumerate(shared)) + + "S2,AgX,TXXX,Target\n" + + "S2,Ctrl,CTRL,Control\n" + ) + rows = ["sampleId,cellId,tag,umiCount"] + # S1: every tag reads the same in every cell, so nothing disagrees with itself. + for cell in ("a1", "a2", "a3", "a4"): + rows.append(f"S1,{cell},CTRL,6") + rows.extend(f"S1,{cell},{t},5" for t in shared) + # S2: T00 splits its clonotype two against two. + for cell in ("b1", "b2", "b3", "b4"): + rows.append(f"S2,{cell},CTRL,6") + rows.append(f"S2,{cell},{shared[0]},{500 if cell in ('b1', 'b2') else 5}") + rows.extend(f"S2,{cell},{t},5" for t in shared[1:]) + rows.append(f"S2,{cell},TXXX,5") + (bed / "counts.csv").write_text("\n".join(rows) + "\n") + (bed / "linker.csv").write_text( + "sampleId,cellId,setId\n" + + "".join(f"S1,a{i},K1\n" for i in range(1, 5)) + + "".join(f"S2,b{i},K2\n" for i in range(1, 5)) + ) + + _run(bed, *BASE) + qc = pl.read_csv(bed / "result_qc.csv", infer_schema_length=0) + t00 = qc.filter((pl.col("measurement") == "tagDisagreement") & (pl.col("entity") == shared[0])) + assert t00.height == 2, "T00 is declared by both panels, so it carries a row in each" + + by_panel = {r["panelId"]: r for r in t00.iter_rows(named=True)} + rates = {p: float(r["value"]) for p, r in by_panel.items()} + assert len(set(rates.values())) == 2, f"one rate on both panels means the run-global figure: {rates}" + + clean = min(rates, key=lambda p: rates[p]) + noisy = max(rates, key=lambda p: rates[p]) + assert rates[clean] == 0.0, "the panel whose cells never disagreed reads zero" + # Two cells of four sit in the minority of their own set, pooled over the cells of sets + # that had something to compare: 2 of 4. + assert rates[noisy] == pytest.approx(0.5) + # Neither row carries a status. A comparison against the other tags in a panel is not + # a line, so no status can be computed from it, and the value travels instead for a + # reader to compare. + assert by_panel[clean]["status"] is None + assert by_panel[noisy]["status"] is None + + +def test_the_key_only_frames_carry_a_value_column_so_they_can_become_columns(bed): + # A p-column is built from a CSV's *value* columns, so a file of key columns alone + # imports as nothing at all -- silently, since the file exists and is well formed. + # What a sample was offered, and which identity a tag feeds, would never leave the + # block. + _run(bed, *BASE) + offered = pl.read_csv(bed / "result_offered.csv", infer_schema_length=0) + assert offered.columns == ["sampleId", "identity", "offered"] + assert set(offered["offered"].to_list()) == {"true"} + + linker = pl.read_csv(bed / "result_tag_identity.csv", infer_schema_length=0) + # (tag, identity). The linker has no sample axis, because neither side of its join has one. A third + # axis would make the join malformed. Label discovery then refuses to build a spec frame, and the + # punchcard renders no columns. + assert linker.columns == ["tag", "identity", "1"] + assert linker.height == linker.unique().height, "duplicate axis keys break a grid silently" + assert set(linker["1"].to_list()) == {"1"} + + +def test_asking_for_a_rung_that_cannot_serve_drops_to_none_and_never_to_another_rung(bed): + # There is no cascade, and its absence is the point. This bed declares no comparator tag and carries a + # panel large enough to stand in for one, exactly the shape a cascade would rescue: ask for the declared + # rung and it drops to *none*, leaving every verdict unreliable, rather than quietly serving the panel. + # + # A baseline nobody chose is a methodology nobody knows they used. The scientist gets the rung they asked + # for or nothing, and the run says which. + # + # Twenty-six tags, against a shipped minimum of twenty-five, so the panel rung IS serviceable here and + # the second half of the test proves it -- otherwise "dropped to none" would prove nothing. + tags = [f"T{i:02d}" for i in range(26)] + (bed / "panel.csv").write_text( + "Samples,Name,Sequence,Type\n" + "".join(f"S1,Ag{i},{t},Target\n" for i, t in enumerate(tags)) + ) + # Background counts sit *above* the shipped floor of 4. At 3 they would be floored to + # zero and the panel median would be 0, so every count that cleared the floor would + # score near 100 against it and read *bound* for a reason that has nothing to do with + # which comparator was chosen -- hiding the very thing this test exists to check. + rows = ["sampleId,cellId,tag,umiCount"] + for cell in ("c1", "c2", "c3"): + rows.append(f"S1,{cell},{tags[0]},900") + rows.extend(f"S1,{cell},{t},10" for t in tags[1:]) + (bed / "counts.csv").write_text("\n".join(rows) + "\n") + + # BASE asks for the declared rung, and this panel marks no tag as a comparator. A + # baseline is required, so the run is refused rather than answered with a punchcard of + # non-answers. Whether a reference tag is declared is a property of the settings, so it + # is caught before anything is read. + r = _run(bed, *BASE, expect_failure=True) + assert r.returncode != 0 + assert "declares no baseline tag" in r.stderr + assert not (bed / "result_verdicts.csv").exists(), "a refused run writes no verdicts" + + # The same counts, asked the other way: the panel rung serves them perfectly well. Which is what makes + # the drop above a choice the scientist made rather than a limit of the data. + r = _run(bed, *BASE, "--reference-source", "panel") + assert r.returncode == 0, r.stderr + meta = json.loads((bed / "result_run_meta.json").read_text()) + assert meta["referenceChoice"] == ReferenceChoice.PANEL.value + states = set(pl.read_csv(bed / "result_verdicts.csv", infer_schema_length=0)["state"].to_list()) + assert states != {"unreliable"} + + +def test_a_panel_too_small_to_serve_still_falls_to_no_comparator(bed): + # The founding three-antigen case: too small to stand in as its own comparator, so the + # third rung is right there and must not be skipped. + (bed / "panel.csv").write_text( + "Samples,Name,Sequence,Type\nS1,AgA,AAAA,Target\nS1,AgB,BBBB,Target\nS1,AgC,CCCC,Target\n" + ) + # Refused, and the message names the rung the scientist should reach for. + r = _run(bed, *BASE, expect_failure=True) + assert r.returncode != 0 + assert "declares no baseline tag" in r.stderr + + +def test_zero_cells_detected_alerts(bed): + # 315's categorical route: the alerting condition is a fact -- no cell barcode observed at + # all -- not a quantity with a published threshold. + (bed / "qc.csv").write_text( + "sampleId,readsTotal,readsMatched,matchedFraction,cellsDetected," + "featuresDetected,totalUniqueUmis,medianUmisPerCell,panelAssignedFraction\n" + "S1,20000,18000,0.9,0,2,1200,300,0.82\n" + ) + r = _run(bed, *BASE, "--qc-summary", "qc.csv") + assert r.returncode == 0, r.stderr + qc = pl.read_csv(bed / "result_qc.csv", infer_schema_length=0) + row = qc.filter(pl.col("measurement") == "cellsDetected").row(0, named=True) + assert row["value"] == "0.0" + assert row["status"] == "alert" + + +def test_a_positive_cell_count_reads_ok_and_claims_nothing_about_yield(bed): + # Above zero, nothing here says the yield was good -- how many cells a sample should + # yield depends on the experiment, and no number for that is published. + (bed / "qc.csv").write_text( + "sampleId,readsTotal,readsMatched,matchedFraction,cellsDetected," + "featuresDetected,totalUniqueUmis,medianUmisPerCell,panelAssignedFraction\n" + "S1,20000,18000,0.9,1,2,1200,300,0.82\n" + ) + r = _run(bed, *BASE, "--qc-summary", "qc.csv") + assert r.returncode == 0, r.stderr + qc = pl.read_csv(bed / "result_qc.csv", infer_schema_length=0) + row = qc.filter(pl.col("measurement") == "cellsDetected").row(0, named=True) + assert row["value"] == "1.0" + assert row["status"] == "OK" + + (bed / "qc.csv").write_text( + "sampleId,readsTotal,readsMatched,matchedFraction,cellsDetected," + "featuresDetected,totalUniqueUmis,medianUmisPerCell,panelAssignedFraction\n" + "S1,20000,18000,0.9,50000,2,1200,300,0.82\n" + ) + r = _run(bed, *BASE, "--qc-summary", "qc.csv") + assert r.returncode == 0, r.stderr + qc = pl.read_csv(bed / "result_qc.csv", infer_schema_length=0) + row = qc.filter(pl.col("measurement") == "cellsDetected").row(0, named=True) + assert row["value"] == "50000.0" + assert row["status"] == "OK" + + +def test_no_cell_list_leaves_membership_unknown_and_depth_unevaluated(bed): + # Which barcodes held a cell is an input. Nothing in the antigen readings separates a + # cell from an empty droplet, so with neither list input the observed barcodes must NOT + # stand in: they outnumber cells by one to two orders of magnitude, and `readsPerCell` + # divides by this, so a healthy library would read undersequenced. + no_linker = [a for a in BASE if a not in ("--linker", "linker.csv")] + (bed / "qc.csv").write_text( + "sampleId,readsTotal,readsMatched,matchedFraction,cellsDetected," + "featuresDetected,totalUniqueUmis,medianUmisPerCell,panelAssignedFraction\n" + "S1,20000,18000,0.9,3,2,1200,300,0.82\n" + ) + r = _run(bed, *no_linker, "--qc-summary", "qc.csv") + assert r.returncode == 0, r.stderr + + meta = json.loads((bed / "result_run_meta.json").read_text()) + assert meta["cellListSource"] == "none" + assert meta["cellsInList"] is None + + qc = pl.read_csv(bed / "result_qc.csv", infer_schema_length=0) + depth = qc.filter(pl.col("measurement") == "readsPerCell").row(0, named=True) + # No cell list, so no rate. No number means no status, and the row is still there. + assert depth["status"] is None + assert depth["value"] is None + + counts = pl.read_csv(bed / "result_cell_counts.csv", infer_schema_length=0) + assert set(counts["inCellList"].to_list()) == {"unknown"}, "unclassified is not the same as classified 'no'" + + +def test_only_the_sample_rolls_up(bed): + # A panel status assumed its per-tag measurements would mostly carry statuses. They do + # not: one is categorical and the rest are read only as outliers against the other tags + # in the same panel. A capture status was then the worst of every sample and every + # panel, which becomes the worst of every sample -- a statement that only repeats what + # is beside it. + _run(bed, *BASE) + qc = pl.read_csv(bed / "result_qc.csv", infer_schema_length=0) + rollups = qc.filter(pl.col("measurement") == "rollup") + levels = set(rollups["level"].to_list()) + assert levels == {"sample"}, f"only the sample rolls up, got {sorted(levels)}" + + def _triple(level): + r = rollups.filter(pl.col("level") == level) + return [int(r[c].cast(pl.Int64).sum()) for c in ("judged", "unjudged", "notEvaluated")] + + assert sum(_triple("sample")) > 0, "the sample rollup still counts what was checked" + + +def test_panel_assigned_fraction_keeps_its_value_and_carries_no_status(bed): + # Atom 310: the undeclared-barcode line is the barcode's, and it never becomes a sample's. + # This sample-grain measurement keeps its number and is never judged, so it never reaches + # the sample's rollup either. + (bed / "qc.csv").write_text( + "sampleId,readsTotal,readsMatched,matchedFraction,cellsDetected," + "featuresDetected,totalUniqueUmis,medianUmisPerCell,panelAssignedFraction," + "cellBarcodeValidFraction\n" + "S1,20000,18000,0.9,4,2,1200,300,0.82,0.91\n" + ) + r = _run(bed, *BASE, "--qc-summary", "qc.csv") + assert r.returncode == 0, r.stderr + qc = pl.read_csv(bed / "result_qc.csv", infer_schema_length=0) + + row = qc.filter(pl.col("measurement") == "panelAssignedFraction").row(0, named=True) + assert row["value"] == "0.82" + assert row["status"] is None + # Computed but never judged: it counts as unjudged, never judged, and never not-evaluated. + assert (row["judged"], row["unjudged"], row["notEvaluated"]) == ("0", "1", "0") + + sample = row["level"], row["entity"] + rollup = qc.filter( + (pl.col("measurement") == "rollup") & (pl.col("level") == sample[0]) & (pl.col("entity") == sample[1]) + ).row(0, named=True) + judged_in_rollup = int(rollup["judged"]) + others = qc.filter( + (pl.col("level") == "sample") + & (pl.col("entity") == sample[1]) + & (~pl.col("measurement").is_in(["rollup", "panelAssignedFraction"])) + & (pl.col("status").is_not_null()) + ).height + assert judged_in_rollup == others, "the rollup counts every judged sample measurement except this one" + + +def test_per_tag_measurements_survive_the_removed_panel_rollup(bed): + # This task removes aggregation, not measurement. A reagent finding still lands on its + # own row, keyed by the panel that has it, which is what keeps a bad reagent + # discoverable with no panel status above it. + _run(bed, *BASE) + qc = pl.read_csv(bed / "result_qc.csv", infer_schema_length=0) + tags = qc.filter((pl.col("level") == "tag") & (pl.col("measurement") != "rollup")) + assert tags.height > 0, "per-tag measurement rows are untouched by the rollup removal" + assert all(p != "" for p in tags["panelId"].to_list()), "each still names its panel" + + +def test_contending_groups_reach_the_note(bed): + (bed / "panel.csv").write_text((bed / "panel.csv").read_text() + "S1,AgB,CCCC,Target\n") + (bed / "counts.csv").write_text((bed / "counts.csv").read_text() + "S1,c1,CCCC,1\nS1,c2,CCCC,1\nS1,c3,CCCC,1\n") + _run(bed, *BASE, "--contending", json.dumps([["AAAA", "CCCC"]])) + # Read without schema inference: the flag is a literal "true"/"false" string, which is + # what a boolean p-column value has to be here, and polars would otherwise infer the + # column back into a Boolean and hide whether the file carries the string at all. + v = pl.read_csv(bed / "result_verdicts.csv", infer_schema_length=0) + r = v.filter(pl.col("identity") == "CCCC").row(0, named=True) + assert r["competedWith"] == "AAAA" and r["wasCompeted"] == "true" + + +def test_barcode_outside_the_cell_list_is_labelled_not_dropped(bed): + (bed / "counts.csv").write_text((bed / "counts.csv").read_text() + "S1,zzz,AAAA,99\n") + _run(bed, *BASE) + # Without schema inference, for the reason given in the contending test. + c = pl.read_csv(bed / "result_cell_counts.csv", infer_schema_length=0) + assert "zzz" in c["cellId"].to_list() + assert c.filter(pl.col("cellId") == "zzz").row(0, named=True)["inCellList"] == "false" + + +def _raw_feature_counts(rows): + return pl.DataFrame(rows, orient="row", schema={"FEATURE": pl.String, "totalWeight": pl.Int64}) + + +def test_undeclared_feature_counts_are_exactly_those_outside_the_declared_set(): + raw = _raw_feature_counts([("AAAA", 10), ("BBBB", 5), ("CCCC", 3)]) + undeclared, share = undeclared_feature_counts(raw, {"AAAA"}) + assert set(undeclared["tag"].to_list()) == {"BBBB", "CCCC"} + assert share == pytest.approx((5 + 3) / (10 + 5 + 3)) + + +def test_no_undeclared_barcode_is_the_ordinary_empty_case(): + raw = _raw_feature_counts([("AAAA", 10), ("BBBB", 5)]) + undeclared, share = undeclared_feature_counts(raw, {"AAAA", "BBBB"}) + assert undeclared.height == 0 + assert share == 0.0 + + +def test_zero_total_weight_reports_no_share_rather_than_zero(): + raw = _raw_feature_counts([]) + undeclared, share = undeclared_feature_counts(raw, {"AAAA"}) + assert undeclared.height == 0 + assert share is None + + +def test_the_declared_set_is_read_per_sample_not_pooled_across_samples(): + raw = _raw_feature_counts([("AAAA", 10), ("BBBB", 5), ("CCCC", 3)]) + undeclared_s1, share_s1 = undeclared_feature_counts(raw, {"AAAA"}) + undeclared_s2, share_s2 = undeclared_feature_counts(raw, {"AAAA", "BBBB", "CCCC"}) + assert set(undeclared_s1["tag"].to_list()) == {"BBBB", "CCCC"} + assert undeclared_s2.height == 0 + assert share_s1 == pytest.approx((5 + 3) / (10 + 5 + 3)) + assert share_s2 == 0.0 + + +def test_undeclared_barcode_is_reported_and_does_not_stop_the_reading(bed): + (bed / "counts.csv").write_text((bed / "counts.csv").read_text() + "S1,c1,TTTT,99\n") + r = _run(bed, *BASE) + assert r.returncode == 0 + m = pl.read_csv(bed / "result_panel_mismatch.csv") + assert "TTTT" in m.filter(pl.col("direction") == "undeclared-in-panel")["tag"].to_list() + assert pl.read_csv(bed / "result_verdicts.csv").height > 0 + + +# --- the undeclared-barcode table: keyed by sequence, carrying the field's own status ------ +# +# 330: barcodes the reads carried that no panel declares get their own table, keyed by +# sequence, and it is the one thing on that surface that carries a status -- the share of a +# sample's reads landing in undeclared barcodes. 310: that status is the barcode's, and it +# never becomes a sample's. + + +def _write_raw_feature_counts(bed, rows): + lines = ["sampleId,FEATURE,totalWeight"] + [f"{s},{f},{w}" for s, f, w in rows] + (bed / "raw_feature_counts.csv").write_text("\n".join(lines) + "\n") + + +def test_undeclared_barcode_table_is_empty_without_the_raw_feature_counts_input(bed): + # No pre-refine pass reached this run, so the table reads as the ordinary empty case -- + # usually empty is the outcome to want -- rather than as an error. + _run(bed, *BASE) + t = pl.read_csv(bed / "result_undeclared_barcodes.csv") + assert t.height == 0 + assert set(t.columns) == {"sampleId", "tag", "totalWeight", "readShare", "status"} + + +def test_undeclared_barcode_table_is_keyed_by_sequence_with_the_samples_share(bed): + # The panel declares AAAA and CTRL for S1 (bed's panel.csv). ZZZZ is neither. + _write_raw_feature_counts(bed, [("S1", "AAAA", 40), ("S1", "CTRL", 10), ("S1", "ZZZZ", 10)]) + _run(bed, *BASE, "--raw-feature-counts", "raw_feature_counts.csv") + t = pl.read_csv(bed / "result_undeclared_barcodes.csv") + assert t["tag"].to_list() == ["ZZZZ"] + row = t.row(0, named=True) + assert row["totalWeight"] == 10 + assert row["readShare"] == pytest.approx(10 / 60) + assert row["status"] == "OK" # 10/60 is well under the 0.50 warn line + + +def test_undeclared_barcode_share_warns_above_half(bed): + _write_raw_feature_counts(bed, [("S1", "AAAA", 10), ("S1", "ZZZZ", 15)]) + _run(bed, *BASE, "--raw-feature-counts", "raw_feature_counts.csv") + t = pl.read_csv(bed / "result_undeclared_barcodes.csv") + assert t.row(0, named=True)["status"] == "warn" + + +def test_undeclared_barcode_share_alerts_when_every_read_is_undeclared(bed): + _write_raw_feature_counts(bed, [("S1", "ZZZZ", 5)]) + _run(bed, *BASE, "--raw-feature-counts", "raw_feature_counts.csv") + t = pl.read_csv(bed / "result_undeclared_barcodes.csv") + assert t.row(0, named=True)["status"] == "alert" + + +def test_undeclared_barcode_table_stays_empty_when_the_panel_covers_every_sequence(bed): + # Every barcode the pre-refine pass saw is declared -- the outcome the field wants -- so the + # table carries no row, not a claim that something failed. + _write_raw_feature_counts(bed, [("S1", "AAAA", 40), ("S1", "CTRL", 10)]) + _run(bed, *BASE, "--raw-feature-counts", "raw_feature_counts.csv") + t = pl.read_csv(bed / "result_undeclared_barcodes.csv") + assert t.height == 0 + + +def test_undeclared_barcode_share_never_reaches_the_sample_report_or_rollup(bed): + # 310: the status is the barcode's, and it does not become a sample's. It must not appear + # among the sample's own measurements or feed the sample's rolled-up status. + _write_raw_feature_counts(bed, [("S1", "ZZZZ", 5)]) + r = _run(bed, *BASE, "--raw-feature-counts", "raw_feature_counts.csv") + assert r.returncode == 0, r.stderr + report = json.loads((bed / "result_qc_by_sample.json").read_text()) + ids = {m["id"] for m in report["S1"]["measurements"]} + assert "undeclaredBarcodeShare" not in ids + + +def test_empty_join_writes_headers(bed): + (bed / "linker.csv").write_text("sampleId,cellId,setId\nS1,zzz,K9\n") + r = _run(bed, *BASE) + assert r.returncode == 0, r.stderr + assert {"setId", "identity", "state"} <= set(pl.read_csv(bed / "result_verdicts.csv").columns) + + +def test_a_cutoff_at_the_analytic_floor_is_refused(bed): + # At or below specificity_score(0, 0) the analytic tally and the dense oracle disagree + # about a silent admissible cell with no error raised. `silent_tally` states that + # refusing such a cutoff is this CLI's job. Tested *at* the bound, not merely either + # side of it: the refusal is "at or below", and 0.04/0.05 alone cannot tell that from + # "below". + from verdict import specificity_score + + bound = float(specificity_score(0, 0)) + + r = _run(bed, *BASE, "--cutoff", "0.04", expect_failure=True) + assert "0.042" in (r.stderr + r.stdout) + _run(bed, *BASE, "--cutoff", repr(bound), expect_failure=True) # the bound itself is refused + _run(bed, *BASE, "--cutoff", repr(bound * 1.001)) + _run(bed, *BASE, "--cutoff", "0.05") + + +def test_rows_are_sorted_on_a_bed_wide_enough_for_order_to_show(bed): + # The default bed has one set and one identity, where sorted and unsorted are the same + # frame and a missing sort is invisible. Three identities declared in descending order + # across two sets make the two differ. + (bed / "panel.csv").write_text( + "Samples,Name,Sequence,Type\nS1,AgZ,ZZZZ,Target\nS1,AgM,MMMM,Target\nS1,AgA,AAAA,Target\nS1,Ctrl,CTRL,Control\n" + ) + rows = ["sampleId,cellId,tag,umiCount"] + for cell in ("c1", "c2", "c3"): + rows.append(f"S1,{cell},CTRL,6") + for tag in ("ZZZZ", "MMMM", "AAAA"): + rows.append(f"S1,{cell},{tag},500") + (bed / "counts.csv").write_text("\n".join(rows) + "\n") + (bed / "linker.csv").write_text("sampleId,cellId,setId\nS1,c1,K2\nS1,c2,K1\nS1,c3,K1\n") + + r = _run(bed, *BASE) + assert r.returncode == 0, r.stderr + verdicts = pl.read_csv(bed / "result_verdicts.csv", infer_schema_length=0) + assert verdicts.height == 6, "two sets by three identities" + + for name, keys in ( + ("result_verdicts.csv", ["setId", "identity"]), + ("result_cell_counts.csv", ["sampleId", "cellId", "tag"]), + ("result_offered.csv", ["sampleId", "identity"]), + ("result_tag_identity.csv", ["tag", "identity"]), + ): + frame = pl.read_csv(bed / name, infer_schema_length=0) + assert frame.height > 1, f"{name} is too small for order to mean anything" + assert frame.equals(frame.sort(keys)), name + + +def test_run_meta_records_the_comparator_served_not_the_one_requested(bed): + # `served_source` degrades to `none` where it cannot honour a request -- here a panel + # comparator is asked for and the panel is far too small to stand in as one. Recording + # the request instead would claim a comparator the run never had, and two runs compared + # against different things would look like two runs compared against the same thing. + r = _run(bed, *BASE, "--reference-source", "panel", "--panel-min-members", "50", expect_failure=True) + assert r.returncode != 0 + # The message names the condition and the number, so a scientist reading it knows which + # of the two halves to change. + assert "below the 50 that rung needs" in r.stderr + assert not (bed / "result_verdicts.csv").exists(), "nothing is written for a rung that cannot serve" + + +def test_sequencing_depth_divides_by_the_cell_list_not_by_observed_barcodes(bed): + # The vendor's five thousand is per called cell. Observed barcodes exceed called cells + # by one to two orders of magnitude in droplet data, because ambient reads land on most + # barcodes, so dividing by them would let a badly undersequenced run read acceptable. + # The bed makes the two differ: four barcodes carry counts, three are in the cell list. + (bed / "counts.csv").write_text((bed / "counts.csv").read_text() + "S1,zzz,AAAA,7\n") + (bed / "qc.csv").write_text( + "sampleId,readsTotal,readsMatched,matchedFraction,cellsDetected," + "featuresDetected,totalUniqueUmis,medianUmisPerCell,panelAssignedFraction\n" + "S1,20000,18000,0.9,4,2,1200,300,0.82\n" + ) + r = _run(bed, *BASE, "--qc-summary", "qc.csv") + assert r.returncode == 0, r.stderr + + qc = pl.read_csv(bed / "result_qc.csv", infer_schema_length=0) + depth = qc.filter(pl.col("measurement") == "readsPerCell").row(0, named=True) + # 18000 / 3 listed cells = 6000, which clears the 5000 line. + # 18000 / 4 observed barcodes = 4500, which would not. + assert float(depth["value"]) == pytest.approx(6000.0) + assert depth["status"] == "OK" + + +# Every module the entrypoint reaches, not just the entrypoint file. The check is on source +# text, so a helper moved out of `emit_verdicts.py` leaves it behind unless it is named here. +ENTRYPOINT_MODULES = ("emit_verdicts.py", "frame_io.py", "identity_tables.py", "qc_rows.py") + + +def test_the_dense_oracle_is_not_reachable_from_the_entrypoint(): + # The dense oracle exists to check the analytic tally in tests. On a realistic panel + # the grid it builds is 11-20x the sparse input, so a production caller is a memory + # failure waiting for a big panel. + for module in ENTRYPOINT_MODULES: + assert "densify" not in (SRC / module).read_text(), module + + +def test_property_grouping_normalises_and_excludes_the_reference(bed): + # The stray whitespace is the point: `read_panel` normalises tag and sample and leaves + # properties alone, so a builder reading the column directly makes " Spike " and "Spike" + # two identities. Built on `consistent_properties` it makes one. + (bed / "panel.csv").write_text( + "Samples,Name,Sequence,Type,Family\n" + "S1,AgA,AAAA,Target, Spike \n" + "S1,AgB,CCCC,Target,Spike\n" + "S1,Ctrl,CTRL,Control,Reference\n" + ) + (bed / "counts.csv").write_text((bed / "counts.csv").read_text() + "S1,c1,CCCC,40\nS1,c2,CCCC,40\nS1,c3,CCCC,1\n") + r = _run(bed, *BASE, "--grouping", json.dumps({"by": "property", "column": "Family"})) + assert r.returncode == 0, r.stderr + identities = set(pl.read_csv(bed / "result_verdicts.csv")["identity"].to_list()) + assert identities == {"Spike"} # one identity, not " Spike " and "Spike" + assert "Reference" not in identities # the comparator is never a candidate + + +def test_the_declarations_that_hold_of_an_identity_travel_with_it(bed): + # `panel-file-authority`: whatever the panel says consistently about an identity's tags travels with + # that identity's verdicts. The bed puts every case in one run -- a property both member tags agree on, + # one they disagree about, and one that agrees for a single-tag identity while disagreeing for the + # merged one -- because a property tested alone cannot show that the disagreement is dropped per + # identity rather than per column. + (bed / "panel.csv").write_text( + "Samples,Name,Sequence,Type,Family,Species,Carrier\n" + "S1,AgA,AAAA,Target,Spike,Human,Biotin\n" + "S1,AgB,CCCC,Target,Spike,Human,Streptavidin\n" + "S1,AgC,GGGG,Target,Nuc,Human,Biotin\n" + "S1,Ctrl,CTRL,Control,Reference,Cyno,Avidin\n" + ) + (bed / "counts.csv").write_text((bed / "counts.csv").read_text() + "S1,c1,CCCC,40\nS1,c1,GGGG,40\n") + _run(bed, *BASE, "--grouping", json.dumps({"by": "property", "column": "Family"})) + + held = { + row["identity"]: row + for row in pl.read_csv(bed / "result_identity_properties.csv", infer_schema_length=0).iter_rows(named=True) + } + assert set(held) == {"Spike", "Nuc"}, "one row per identity, and the comparator is not an identity" + + # Agreed across both member tags, so it holds of the merged identity. + assert held["Spike"]["Species"] == "Human" + assert held["Spike"]["Type"] == "Target" + assert held["Spike"]["Family"] == "Spike" + # Disagreed between the member tags, so it holds of nothing: neither tag's value wins. + assert held["Spike"]["Carrier"] == "" + assert held["Spike"]["Name"] == "" + # The same two columns still hold for the identity whose single tag settles them, which is what makes + # the omission above about the identity rather than about the column. + assert held["Nuc"]["Carrier"] == "Biotin" + assert held["Nuc"]["Name"] == "AgC" + + # The reference tag declares Cyno and Avidin and is no identity, so neither value can reach the export: + # a declaration travelling from a tag that gets no verdict would describe nothing. + meta = json.loads((bed / "result_run_meta.json").read_text()) + assert meta["identityPropertyValues"]["Species"] == ["Human"] + assert meta["identityPropertyValues"]["Type"] == ["Target"] + assert "Avidin" not in meta["identityPropertyValues"]["Carrier"] + # The workflow builds one spec per name in this list, so a name here that is not a column of the CSV, + # or the reverse, is an import of nothing. + columns = pl.read_csv(bed / "result_identity_properties.csv", infer_schema_length=0).columns + assert meta["identityProperties"] == [c for c in columns if c != "identity"] + + +def test_a_property_no_identity_agreed_on_is_left_out_rather_than_exported_blank(bed): + # One identity, and it disagrees about Carrier. An all-blank column would offer a reader a filter with + # no values in it, so the column does not ship at all. + (bed / "panel.csv").write_text( + "Samples,Name,Sequence,Type,Family,Carrier\n" + "S1,AgA,AAAA,Target,Spike,Biotin\n" + "S1,AgB,CCCC,Target,Spike,Streptavidin\n" + "S1,Ctrl,CTRL,Control,Reference,Avidin\n" + ) + (bed / "counts.csv").write_text((bed / "counts.csv").read_text() + "S1,c1,CCCC,40\n") + _run(bed, *BASE, "--grouping", json.dumps({"by": "property", "column": "Family"})) + + columns = pl.read_csv(bed / "result_identity_properties.csv", infer_schema_length=0).columns + assert "Carrier" not in columns + assert "Family" in columns, "a column that does hold still ships" + meta = json.loads((bed / "result_run_meta.json").read_text()) + assert "Carrier" not in meta["identityProperties"] + + +def test_two_identities_that_disagree_the_same_way_do_not_share_a_label(bed): + # Both barcodes carry Family=Spike in S1 and Family=Nuc in S2. Under `panel-file-authority@3.0` the + # panel declares per tag AND sample, so that is not a disagreement to fall back from -- it is two + # declarations, and each barcode joins the family its own sample named. The identities are the two + # families, and neither barcode stands alone under its raw sequence. + # + # Do not invert this back to a fallback where each barcode stands alone under its raw sequence, labelled + # with its declared values joined ("Nuc / Spike"). That shape is only forced by a dataset-wide map, + # which cannot hold both declarations at once. The panel is read per tag AND sample. + (bed / "panel.csv").write_text( + "Samples,Name,Sequence,Type,Family\n" + "S1,AgA,AAAA,Target,Spike\n" + "S2,AgA,AAAA,Target,Nuc\n" + "S1,AgB,CCCC,Target,Spike\n" + "S2,AgB,CCCC,Target,Nuc\n" + "S1,Ctrl,CTRL,Control,Reference\n" + "S2,Ctrl,CTRL,Control,Reference\n" + ) + (bed / "counts.csv").write_text( + "sampleId,cellId,tag,umiCount\nS1,c1,AAAA,500\nS1,c1,CTRL,6\nS2,c2,CCCC,500\nS2,c2,CTRL,6\n" + ) + (bed / "linker.csv").write_text("sampleId,cellId,setId\nS1,c1,K1\nS2,c2,K2\n") + _run(bed, *BASE, "--grouping", json.dumps({"by": "property", "column": "Family"})) + + labels = dict(pl.read_csv(bed / "result_identity_labels.csv", infer_schema_length=0).iter_rows()) + assert set(labels) == {"Spike", "Nuc"}, "each barcode joins the family its own sample declared" + assert len(set(labels.values())) == 2, f"two identities under one label: {labels}" + # A property grouping makes the identity the property's value, which is already the name a reader + # recognises, so no barcode is appended and no name is joined. + assert labels == {"Spike": "Spike", "Nuc": "Nuc"} + + meta = json.loads((bed / "result_run_meta.json").read_text()) + assert meta["tagsWithoutGroupingValue"] == [], "nothing fell back: every pair carried a value" + + +def test_a_flat_contending_list_is_refused_rather_than_read_as_characters(bed): + # `["AgA","AgB"]` is valid JSON and the shape a hand-driven run reaches for first. Read as groups it + # makes `set("AgA")` -- a set of CHARACTERS -- so the run completes, no competitor note ever fires, + # every `wasCompeted` reads false, and the run record states a contention nothing tested. A silent + # wrong answer is the worst outcome available here, so the flag is refused instead. + r = _run(bed, *BASE, "--contending", json.dumps(["AgA", "AgB"]), expect_failure=True) + assert "--contending" in r.stderr + + # A group of one tests nothing: an identity cannot contend with itself. + r = _run(bed, *BASE, "--contending", json.dumps([["AgA"]]), expect_failure=True) + assert "fewer than two members" in r.stderr + + # The valid shape still runs, so the guard rejects the mistake rather than the feature. + _run(bed, *BASE, "--contending", json.dumps([["AgA", "AgB"]])) + + +def test_a_non_object_grouping_gets_the_usage_message_not_an_attribute_error(bed): + # `--grouping '"tag"'` parses as JSON and is not a mapping. Reaching `.get` on it raises an + # AttributeError -- a stack trace about a str, where a usage message for this exact mistake is already + # written two lines away. + r = _run(bed, *BASE, "--grouping", json.dumps("tag"), expect_failure=True) + assert "--grouping must be" in r.stderr + assert "AttributeError" not in r.stderr + + +def test_a_non_integer_umi_count_names_the_file_and_the_column(bed): + # A blank or a decimal dies in the cast as a raw polars traceback naming neither the file nor the + # column -- the two things a reader needs. This module's convention is that a bad input exits with a + # message about the input. + (bed / "counts.csv").write_text("sampleId,cellId,tag,umiCount\nS1,c1,AAAA,\nS1,c2,AAAA,3.5\n") + r = _run(bed, *BASE, expect_failure=True) + assert "counts.csv" in r.stderr + assert "umiCount" in r.stderr + assert "Traceback" not in r.stderr + + +DECLARED_FLAGS = ( + "--linker", + "--cells", + "--barcode-col", + "--feature-col", + "--sample-col", + "--role-column", + "--reference-values", + "--reference-source", + "--panel-min-members", + "--floor", + "--cutoff", + "--min-voters", + "--min-agreement", + "--gate-threshold", + "--grouping", + "--contending", + "--capture-map", + "--output-prefix", +) + + +def test_every_declared_flag_is_reachable_from_the_command_line(bed): + # Every parameter of the reading is threaded from the workflow, so a parameter that + # exists only as a module default is one a scientist cannot move. The help text is the + # cheapest place the whole set is visible at once. + help_text = _run(bed, "--help").stdout + for flag in DECLARED_FLAGS: + assert flag in help_text, flag + + +def test_output_is_byte_stable_across_runs(bed): + # `combine_tags_to_identities` groups without maintaining order, so an unsorted frame + # varies run to run. A p-column's identity is content addressed, so an unstable byte + # order silently costs every downstream node its dedup. + _run(bed, *BASE) + first = {p.name: p.read_bytes() for p in bed.glob("result_*")} + _run(bed, *BASE) + second = {p.name: p.read_bytes() for p in bed.glob("result_*")} + assert first == second + + # Repeating the run is not enough on its own: polars groups deterministically for one + # input, so an unsorted frame reproduces itself byte for byte and this passes while the + # sort is missing. Sortedness itself is asserted in + # `test_rows_are_sorted_on_a_bed_wide_enough_for_order_to_show`, which needs a bed this + # one is too narrow to provide. + + +def test_a_computed_but_unjudged_measurement_is_not_reported_as_unchecked(bed): + # Both no-status cases now leave the status column empty, so the distinction they carry has + # to survive somewhere else: the VALUE and the coverage triple beside it. "Computed, and no + # line stands behind it" and "nothing computed this" are the one pair the status set exists + # to keep apart, and collapsing them reads "nothing was wrong" as "nobody looked". + _run(bed, *BASE) + qc = pl.read_csv(bed / "result_qc.csv", infer_schema_length=0) + + # Computed, no line: a number, and the triple counts it unjudged. + floor_row = qc.filter(pl.col("measurement") == "floorRemoved").row(0, named=True) + assert floor_row["status"] is None + assert floor_row["value"] is not None + assert (floor_row["judged"], floor_row["unjudged"], floor_row["notEvaluated"]) == ("0", "1", "0") + + # Nothing computed it: no number, the reason in its place, and the triple counts it + # not-evaluated. Same empty status column, opposite finding. + deferred = qc.filter(pl.col("measurement") == "aggregateBarcodeFraction").row(0, named=True) + assert deferred["status"] is None + assert deferred["value"] is None + assert (deferred["judged"], deferred["unjudged"], deferred["notEvaluated"]) == ("0", "0", "1") + assert deferred["reason"] # a deferred measurement says why nothing computed it + + +def test_a_capture_map_is_accepted_and_changes_no_row(bed): + # The capture rollup was the only reader of this map, and only the sample carries an + # aggregated status now. The argument stays accepted because the capture axis ships on + # the QC columns: adding an axis to a released column changes that column's identity, + # where adding a value does not. So supplying a map must not fail, and must not put a row + # anywhere either. + _run(bed, *BASE, "--capture-map", json.dumps({"S1": "C1"})) + qc = pl.read_csv(bed / "result_qc.csv", infer_schema_length=0) + assert "capture" not in set(qc["level"].to_list()) + assert "C1" not in set(qc["entity"].to_list()) + + +def test_a_cell_list_of_its_own_overrides_the_linker_and_is_recorded(bed): + # The cell list is an input. The linker only says which set a cell belongs to. A list + # from gene expression covers cells whose receptor never assembled, which the linker + # structurally cannot, so which list a figure was computed against has to travel with the + # run. + (bed / "cells.csv").write_text("sampleId,cellId\nS1,c1\nS1,c2\n") + r = _run(bed, *BASE, "--cells", "cells.csv") + assert r.returncode == 0, r.stderr + meta = json.loads((bed / "result_run_meta.json").read_text()) + assert meta["cellListSource"] == "cell list" and meta["cellsInList"] == 2 + scalars = pl.read_csv(bed / "result_cell_scalars.csv", infer_schema_length=0) + assert scalars.filter(pl.col("cellId") == "c3").row(0, named=True)["inCellList"] == "false" + + +def test_a_gate_sets_cells_aside_and_says_how_many(bed): + r = _run(bed, *BASE, "--gate-threshold", "5") + assert r.returncode == 0, r.stderr + meta = json.loads((bed / "result_run_meta.json").read_text()) + assert meta["cellsSetAside"] == 3 # every one of c1, c2 and c3 reads the comparator at 6 + v = pl.read_csv(bed / "result_verdicts.csv") + assert v.filter(pl.col("identity") == "AAAA").row(0, named=True)["state"] == "unreliable" + + +def test_the_floor_runs_before_tags_combine(bed): + # Order is visible in the count: the floor works on the sparse per-tag frame, so two + # readings of one identity in one cell are two floored readings. Combining first would + # take the highest and floor one. + (bed / "panel.csv").write_text( + "Samples,Name,Sequence,Type,Family\n" + "S1,AgA,AAAA,Target,Spike\n" + "S1,AgB,CCCC,Target,Spike\n" + "S1,Ctrl,CTRL,Control,Reference\n" + ) + (bed / "counts.csv").write_text("sampleId,cellId,tag,umiCount\nS1,c1,AAAA,1\nS1,c1,CCCC,1\nS1,c1,CTRL,6\n") + (bed / "linker.csv").write_text("sampleId,cellId,setId\nS1,c1,K1\n") + r = _run(bed, *BASE, "--grouping", json.dumps({"by": "property", "column": "Family"})) + assert r.returncode == 0, r.stderr + meta = json.loads((bed / "result_run_meta.json").read_text()) + assert meta["readingsFloored"] == 2 + assert meta["cellsEmptied"] == 1 + v = pl.read_csv(bed / "result_verdicts.csv") + assert v.filter(pl.col("identity") == "Spike").row(0, named=True)["state"] == "not bound" + + +# ---- the committed fixture bed --------------------------------------------------------------- +# +# Every test above writes its own three-line bed, which keeps each one readable and keeps none of them +# realistic: a run whose panel is one size, whose comparator is one tag and whose cells all come from one +# sample cannot show what happens when four samples were stained differently. The committed bed at +# software/test-data/fixtures/verdicts/ carries the awkward panel shapes at once -- panels of differing +# size, barcodes recurring under different names, one antigen on two barcodes, one comparator and two, and +# a barcode declared on one sample and read on another. + +VERDICT_BED = Path(__file__).resolve().parents[2] / "test-data" / "fixtures" / "verdicts" +VERDICT_BED_FILES = ( + "counts.csv", + "linker.csv", + "panel.csv", + "panel_with_reference.csv", + "panel_multi_reference.csv", + "panel_narrow.csv", + "panel_wide.csv", +) + +NAME_GROUPING = ("--grouping", json.dumps({"by": "property", "column": "Name"})) + + +@pytest.fixture +def wide_bed(tmp_path): + """The committed bed, copied so a run's output files never land in the repository.""" + for name in VERDICT_BED_FILES: + source = VERDICT_BED / name + if not source.exists(): + pytest.fail(f"committed bed missing at {source}; regenerate it with generate.py", pytrace=False) + shutil.copy(source, tmp_path / name) + return tmp_path + + +def _bed_args(panel_csv, *extra): + # The bed's column names are the ones BASE already names, so only the panel file varies across the three + # shapes: no comparator, one comparator, two. + # The gate is on for every bed run: it is what sets c11 aside and so what keeps *unreliable* reachable in + # the bed. 100 is above every other comparator here (6, and 60 on the two-control panel) and below c11's + # 400, so exactly one cell is set aside. + return ["counts.csv", panel_csv, *BASE[2:], "--gate-threshold", "100", *extra] + + +def _bed_shape(bed): + """The handles these tests need, recovered from the bed by the role each barcode plays. + + Derived rather than written down because the sequences come from a seeded RNG. A bed regenerated under a + different seed still has four barcodes carrying two antigen names, one antigen carried on two barcodes + and one barcode declared by a single sample and read only in another. Spelling the sequences out here + would tie every assertion below to the seed instead of to the shape. + """ + panel = pl.read_csv(bed / "panel_multi_reference.csv", infer_schema_length=0) + counts = pl.read_csv(bed / "counts.csv", infer_schema_length=0) + linker = pl.read_csv(bed / "linker.csv", infer_schema_length=0) + + read_in: dict[str, set[str]] = {} + for sample, tag in counts.select("sampleId", "tag").iter_rows(): + read_in.setdefault(tag, set()).add(sample) + + names: dict[str, set[str]] = {} + declared_in: dict[str, set[str]] = {} + offered: dict[str, set[str]] = {} + for row in panel.filter(pl.col("Type") != "Control").iter_rows(named=True): + names.setdefault(row["Sequence"], set()).add(row["Name"]) + declared_in.setdefault(row["Sequence"], set()).add(row["Samples"]) + # `offered` means what a cell could ANSWER at, which needs the declaration AND the + # reads. A tag the sample's reads never carry was declared and never measured, so its + # cells do not count toward that identity. The key is kept either way so a sample that + # measured nothing still appears. + offered.setdefault(row["Samples"], set()) + if row["Samples"] in read_in.get(row["Sequence"], set()): + offered[row["Samples"]].add(row["Sequence"]) + + tags_of_name: dict[str, set[str]] = {} + for tag, tag_names in names.items(): + for name in tag_names: + tags_of_name.setdefault(name, set()).add(tag) + + sets_of_sample: dict[str, set[str]] = {} + samples_of_set: dict[str, set[str]] = {} + for sample, set_id in linker.select("sampleId", "setId").iter_rows(): + sets_of_sample.setdefault(sample, set()).add(set_id) + samples_of_set.setdefault(set_id, set()).add(sample) + + # Declared by exactly one sample and read in none of that sample's cells: the only arrangement in which + # both directions of the panel-versus-reads check fire on the same barcode at once. + cross = [t for t, samples in declared_in.items() if len(samples) == 1 and not samples & read_in.get(t, set())] + shared = [(name, sorted(tags)) for name, tags in tags_of_name.items() if len(tags) > 1] + short_sample = min(offered, key=lambda s: (len(offered[s]), s)) + spanning = sorted(s for s, samples in samples_of_set.items() if len(samples) > 1) + + return { + "antigens": set(names), + "names": set(tags_of_name), + "renamed": {t for t, tag_names in names.items() if len(tag_names) > 1}, + "shared": shared, + "cross": cross, + "read_in": read_in, + "declared_in": declared_in, + "offered": offered, + "short_sample": short_sample, + "sets_of_sample": sets_of_sample, + "spanning": spanning, + } + + +def _states(bed): + v = pl.read_csv(bed / "result_verdicts.csv", infer_schema_length=0) + return {(r["setId"], r["identity"]): r["state"] for r in v.iter_rows(named=True)} + + +def _only_set(shape, sample): + sets = shape["sets_of_sample"][sample] + assert len(sets) == 1, f"the bed must draw one set from {sample} for this assertion to be about that set" + return next(iter(sets)) + + +def _samples_of(shape, set_id): + return {s for s, sets in shape["sets_of_sample"].items() if set_id in sets} + + +def test_the_bed_keys_identity_by_barcode_where_the_names_would_split(wide_bed): + shape = _bed_shape(wide_bed) + assert len(shape["renamed"]) >= 4, "the bed must carry the case that makes name keying wrong" + + r = _run(wide_bed, *_bed_args("panel_with_reference.csv")) + assert r.returncode == 0, r.stderr + identities = set(pl.read_csv(wide_bed / "result_verdicts.csv", infer_schema_length=0)["identity"].to_list()) + assert identities == shape["antigens"], "one identity per declared barcode, whatever it was named" + + # The two keyings do not even agree on how many questions the run asks: keying on the name would split + # each renamed barcode in two and fuse the antigen carried on two barcodes into one. + assert len(shape["names"]) > len(shape["antigens"]) + + # A label is not an identity, and two identities under one label are two rows a reader cannot tell apart, + # so where two barcodes share a name the label has to carry the barcode as well. + labels = dict(pl.read_csv(wide_bed / "result_identity_labels.csv", infer_schema_length=0).iter_rows()) + assert set(labels) == shape["antigens"] + assert len(set(labels.values())) == len(labels) + + # And from the other side: asked to group by the name, the run PLACES the renamed barcodes, one identity + # per name the panel declared. Nothing is left unplaced. + # + # A barcode named differently in two samples does NOT have "no one name that holds", and must not be + # reported in `tagsWithoutGroupingValue` or left standing alone under its raw sequence. Under + # `panel-file-authority@3.0` those are two declarations -- one reagent identifier carrying a different + # antigen in each sample -- so each is placed under the name its own sample gave it. + r = _run(wide_bed, *_bed_args("panel_with_reference.csv", *NAME_GROUPING)) + assert r.returncode == 0, r.stderr + meta = json.loads((wide_bed / "result_run_meta.json").read_text()) + assert meta["tagsWithoutGroupingValue"] == [], "a renamed barcode is placed, not left unplaceable" + named = set(pl.read_csv(wide_bed / "result_verdicts.csv", infer_schema_length=0)["identity"].to_list()) + assert named == shape["names"], "one identity per declared name, across every sample" + + +def test_the_bed_reaches_all_four_states_in_one_run(wide_bed): + # A bed that cannot reach a state tests nothing about it. All four come from one run here: bound from + # counts of 500 and 5000 against a comparator of 6, not bound from counts of 8, never asked from the + # three-tag panel, and unreliable from the one cell the admissibility gate sets aside. + r = _run(wide_bed, *_bed_args("panel_with_reference.csv")) + assert r.returncode == 0, r.stderr + v = pl.read_csv(wide_bed / "result_verdicts.csv", infer_schema_length=0) + assert set(v["state"].to_list()) == {"bound", "not bound", "never asked", "unreliable"} + + +def test_the_short_panel_is_where_never_asked_appears(wide_bed): + shape = _bed_shape(wide_bed) + short = shape["short_sample"] + assert len(shape["offered"][short]) < len(shape["antigens"]), "the bed needs panels of differing size" + + r = _run(wide_bed, *_bed_args("panel_with_reference.csv")) + assert r.returncode == 0, r.stderr + states = _states(wide_bed) + + unasked = {i for (s, i), state in states.items() if s == _only_set(shape, short) and state == "never asked"} + assert unasked == shape["antigens"] - shape["offered"][short] + assert unasked + + # A set spanning two samples could answer wherever either sample both declared and measured a tag, so a + # panel gap in one sample closes where the other covers it. + # + # One barcode is offered nowhere and cannot close: the cross declaration, declared by a single sample and + # read only in another. The sample that declared it measured nothing for it, and the sample that measured + # it never declared it, so neither side put the question. That barcode is the bed's deliberate hole -- it + # exists so both directions of the panel-versus-reads check fire on one tag -- and it is exactly what the + # spanning set still reads never asked at. + spanning = shape["spanning"] + assert spanning, "the bed needs one set drawn from two samples" + covered = set().union(*(shape["offered"][s] for s in _samples_of(shape, spanning[0]))) + assert covered == shape["antigens"] - set(shape["cross"]), ( + "the spanning set's samples must cover the universe apart from the cross declaration" + ) + unasked_spanning = {i for (s, i), state in states.items() if s == spanning[0] and state == "never asked"} + assert unasked_spanning == set(shape["cross"]) + + +def test_the_panel_mismatch_fires_per_sample_in_both_directions(wide_bed): + shape = _bed_shape(wide_bed) + assert len(shape["cross"]) == 1, "the bed carries exactly one barcode declared here and read there" + tag = shape["cross"][0] + declaring = next(iter(shape["declared_in"][tag])) + reading = sorted(shape["read_in"][tag]) + + # Read against the two-comparator panel, the only one here declaring every barcode the counts carry: on + # the others the undeclared comparator adds rows and the table is no longer a clean statement about this + # one barcode. + # + # On the panel rung, because this version refuses two declared comparators and this test is about the + # PANEL FILE rather than about the comparator. The minimum is lowered to the panel it has, for the same + # reason: neither number is the subject here. + size = pl.read_csv(wide_bed / "panel_multi_reference.csv", infer_schema_length=0)["Sequence"].n_unique() + on_panel_rung = ["--reference-source", "panel", "--panel-min-members", str(size)] + r = _run(wide_bed, *_bed_args("panel_multi_reference.csv", *on_panel_rung)) + assert r.returncode == 0, r.stderr + + m = pl.read_csv(wide_bed / "result_panel_mismatch.csv", infer_schema_length=0) + rows = {(row["tag"], row["direction"]): row["samples"] for row in m.iter_rows(named=True)} + assert m.height == 2, f"only the cross declaration should mismatch; got {m.to_dicts()}" + assert rows[(tag, "declared-never-seen")] == declaring + assert rows[(tag, "undeclared-in-panel")] == ", ".join(reading) + + # A global check would have cancelled these two against each other. The verdicts show why that matters: + # the sample that read the barcode never declared it, so its set reads never asked while a real count of + # 500 sits in the counts file -- the verdict follows the panel. + states = _states(wide_bed) + assert states[(_only_set(shape, reading[0]), tag)] == "never asked" + + # And the sample that DECLARED it read nothing for it, so its cells leave that identity's denominator + # too: zero reads across a whole sample is a reagent that produced nothing, and none of the things that + # causes -- a reagent never added, a barcode mis-declared, a failed library -- put the question the file + # says was put. Both sides read never asked, for opposite reasons: one sample was never offered the + # barcode, the other was offered it and nobody measured it. + # + # This is NOT the per-cell rule. A cell that read nothing for a tag its sample DID measure votes not + # bound, which is a reading that happened and failed. `test_a_silent_cell_votes_not_bound` pins that, and + # the off-target acceptance bed exercises both routes in one clonotype. + assert states[(_only_set(shape, declaring), tag)] == "never asked" + + +def test_one_antigen_on_two_barcodes_is_read_by_its_highest_member(wide_bed): + shape = _bed_shape(wide_bed) + assert len(shape["shared"]) == 1, "the bed carries exactly one antigen on two barcodes" + name, (first, second) = shape["shared"][0] + spanning = shape["spanning"][0] + + # Per barcode the two cells that carry them bind opposite ones, so each barcode splits its set one to one + # and reads unreliable on the tie. + r = _run(wide_bed, *_bed_args("panel_with_reference.csv")) + assert r.returncode == 0, r.stderr + per_tag = _states(wide_bed) + assert per_tag[(spanning, first)] == "unreliable" + assert per_tag[(spanning, second)] == "unreliable" + + # Read as one antigen the two barcodes combine by the highest member, never by the sum and never by an + # arbitrary one: each cell's reading becomes 500, both cells bind, and the set is bound. Summing would + # reach the same verdict here by accident. What the highest rule buys is that a cell's answer does not + # depend on how many barcodes happened to carry the antigen. + r = _run(wide_bed, *_bed_args("panel_with_reference.csv", *NAME_GROUPING)) + assert r.returncode == 0, r.stderr + assert _states(wide_bed)[(spanning, name)] == "bound" + + +def test_two_declared_comparators_are_refused_rather_than_combined(wide_bed): + # Never take the higher of the two, and never combine comparator tags the way an identity's tags combine: + # `baseline-scope` states that references are never combined, and taking the highest is a combination. + # The atom's construct scopes each reference to a group of antigens by a declared property, and this + # version has no group-by half, so it cannot say which antigens a second comparator belongs to. It + # refuses rather than choosing a rule nobody wrote down, as the field does -- the ordinary antibody run + # rejects a second control outright. + # + # Refused loudly rather than degraded to no comparator: this is a panel a scientist fixes in a minute, + # and a silent fall to *unreliable* everywhere would not tell them how. + r = _run(wide_bed, *_bed_args("panel_multi_reference.csv"), expect_failure=True) + assert "declares 2 baseline tags" in r.stderr + assert "one baseline tag or none" in r.stderr + + # The one-comparator panel over the same counts still serves, so the refusal is about the count of + # comparators and not about anything else in the bed. + assert _run(wide_bed, *_bed_args("panel_with_reference.csv")).returncode == 0 + assert any(state == "bound" for state in _states(wide_bed).values()) + + +def test_the_bed_panel_without_a_declared_comparator_serves_as_its_own(wide_bed): + # The bed's panel is eight antigens, below the shipped minimum of twenty-five, so the rung is asked for + # explicitly here. That is not the bed falling short: an eight-antigen panel is what this block's runs + # actually carry, and the test below this one is what pins the shipped default's answer for one. What + # this test is about is what the rung DOES once it serves -- which comparator it builds, and that the + # minimum count spares only a declared one. + shape = _bed_shape(wide_bed) + stands_in = ["--reference-source", "panel", "--panel-min-members", str(len(shape["antigens"]))] + + r = _run(wide_bed, *_bed_args("panel.csv", *stands_in)) + assert r.returncode == 0, r.stderr + meta = json.loads((wide_bed / "result_run_meta.json").read_text()) + assert meta["referenceChoice"] == ReferenceChoice.PANEL.value + states = set(_states(wide_bed).values()) + assert states != {"unreliable"}, "the panel could serve as its own comparator and was not asked to" + without = meta["readingsFloored"] + + # The floor spares a comparator's reading, and only a declared comparator has one to spare. With no + # declaration c08's comparator reading of 1 is floored like any other count, so this run floors strictly + # more than the same counts read against a declared comparator. + assert _run(wide_bed, *_bed_args("panel_with_reference.csv", *stands_in)).returncode == 0 + with_declared = json.loads((wide_bed / "result_run_meta.json").read_text())["readingsFloored"] + assert without > with_declared > 0 + + +def test_a_panel_of_this_size_no_longer_stands_in_for_its_own_comparator(wide_bed): + # The shipped minimum answers the bed's own panel, with nothing asked for. Eight antigens is under + # twenty-five, so the panel cannot be its own background and the run says so rather than comparing a + # count against seven other antigens and calling the result a background estimate. + # + # This is what the minimum moving from 8 to 25 changed, and it is the whole point of the move: an + # antibody kit caps at fifteen tags, so no such panel reaches this rung. Such a run now asks for the + # tag-distribution rung instead. + shape = _bed_shape(wide_bed) + assert len(shape["antigens"]) < DEFAULT_PANEL_MIN_MEMBERS, "the bed grew past the minimum" + + r = _run(wide_bed, *_bed_args("panel.csv", "--reference-source", "panel"), expect_failure=True) + assert r.returncode != 0 + assert f"below the {DEFAULT_PANEL_MIN_MEMBERS} that rung needs" in r.stderr + # Refused before anything is read, rather than answered with a punchcard where every asked position + # reads unreliable. That output was honest and useless: it cost what a real run costs and looked like a + # result at a glance. + assert not (wide_bed / "result_verdicts.csv").exists() + + +def test_a_reading_from_a_sample_that_never_offered_it_is_not_a_vote(bed): + # The denominator counts only members whose OWN sample offered the identity. If the + # numerator does not apply the same test, the two are drawn from different populations: + # a reading from a cell that was never asked displaces a silent cell's real vote. + # Reachable whenever a sample-keyed panel meets a set spanning two samples and a tag + # declared for one sample is read in the other -- which is the undeclared-in-panel case + # this block measures on purpose, not an exotic shape. + (bed / "panel.csv").write_text( + "Samples,Name,Sequence,Type\nS1,Ctrl,CTRL,Control\nS2,Ctrl,CTRL,Control\nS2,AgX,XXXX,Target\n" + ) + (bed / "counts.csv").write_text( + "sampleId,cellId,tag,umiCount\n" + "S1,c1,CTRL,6\nS1,c1,XXXX,500\n" # S1 never offered XXXX; this is not a vote + "S2,c2,CTRL,6\nS2,c2,XXXX,500\n" # offered and bound + "S2,c3,CTRL,6\n" # offered and silent -> not bound + ) + (bed / "linker.csv").write_text("sampleId,cellId,setId\nS1,c1,K1\nS2,c2,K1\nS2,c3,K1\n") + + r = _run(bed, *BASE) + assert r.returncode == 0, r.stderr + row = ( + pl.read_csv(bed / "result_verdicts.csv", infer_schema_length=0) + .filter(pl.col("identity") == "XXXX") + .row(0, named=True) + ) + # One bound and one not-bound among the two cells that were actually asked. + assert row["state"] == "unreliable" + assert row["unreliableReason"] == "tie" + assert (row["cellsCouldAnswer"], row["cellsAnswered"]) == ("2", "2") + + +def test_every_asked_cell_reading_still_counts_when_both_samples_offered_it(bed): + # The guard above must not throw away legitimate cross-sample votes: with both samples + # offering the identity, all three cells vote as before. + (bed / "panel.csv").write_text( + "Samples,Name,Sequence,Type\n" + "S1,Ctrl,CTRL,Control\nS1,AgX,XXXX,Target\nS2,Ctrl,CTRL,Control\nS2,AgX,XXXX,Target\n" + ) + (bed / "counts.csv").write_text( + "sampleId,cellId,tag,umiCount\nS1,c1,CTRL,6\nS1,c1,XXXX,500\nS2,c2,CTRL,6\nS2,c2,XXXX,500\nS2,c3,CTRL,6\n" + ) + (bed / "linker.csv").write_text("sampleId,cellId,setId\nS1,c1,K1\nS2,c2,K1\nS2,c3,K1\n") + + r = _run(bed, *BASE) + assert r.returncode == 0, r.stderr + row = ( + pl.read_csv(bed / "result_verdicts.csv", infer_schema_length=0) + .filter(pl.col("identity") == "XXXX") + .row(0, named=True) + ) + assert (row["cellsCouldAnswer"], row["cellsAnswered"]) == ("3", "3") + assert row["state"] == "bound" # two bound against one silent + + +def test_no_qc_row_carries_a_null_panel_key(bed): + # panelId is an AXIS of the imported QC frame, and a null is not a usable p-column key. + # Sample-level and capture-level rows belong to no panel, so they carry an empty string, + # which is a key, never a null. + _run(bed, *BASE, "--capture-map", json.dumps({"S1": "C1"})) + qc = pl.read_csv(bed / "result_qc.csv", infer_schema_length=0) + assert qc["panelId"].null_count() == 0 + + # Both kinds must be present, or the assertion above proves nothing: rows that belong to + # a panel carry its id, rows that belong to none carry an empty string. + panels = set(qc["panelId"].to_list()) + assert "" in panels, "sample and capture rows belong to no panel and must carry an empty key" + assert any(p for p in panels), "tag and identity rows must carry a real panel id" + + +# --- the shape a real panel file arrives in ----------------------------------------------------- +# +# Every bed above declares a role column, so every bed above can name a comparator. A panel file observed +# in the field carries three columns and no fourth: the sample, the barcode sequence, and the antigen's +# name. There is no role column to point `--role-column` at, so the declared rung is not reachable on it at +# all and the panel's own readings have to serve. It also reuses a barcode between samples under a +# different antigen name, which is the tag-inventory reuse the per-sample keying of the panel exists for. +# +# These tests fix what that file does today. They deliberately do NOT assert that a set spanning two +# samples should carry one verdict for a barcode that names two different antigens -- that question is +# open, and a test asserting today's answer would have to be deleted to settle it. + +# Twenty-six, against a shipped minimum of twenty-five. The count is the only thing this list carries that +# the panel rung cares about. Every test below reads its members by position, so widening it changes what +# serves and nothing else. Padded to two digits so the sorted identity list this bed's assertions compare +# against is the list order. +CUSTOMER_TAGS = [f"SEQ{i:02d}" for i in range(1, 27)] + + +def _customer_bed(root, *, renamed=2, span_samples=True): + """A three-column panel: sample, sequence, antigen. No role column, no grouping column. + + `renamed` barcodes carry a different antigen name in the second sample. `span_samples` puts every cell + in one clonotype set, so the set's cells come from both panels. + """ + rows = ["Sample,Sequence,Antigen"] + for sample, offset in (("SmpA", 0), ("SmpB", 100)): + for i, tag in enumerate(CUSTOMER_TAGS): + name = f"Ag{offset + i:03d}" if i < renamed else f"Ag{i:03d}" + rows.append(f"{sample},{tag},{name}") + (root / "panel.csv").write_text("\n".join(rows) + "\n") + + # SEQ01 is strong. The rest sit at 10, above the shipped floor of 4 so nothing is floored away and the + # panel median stays a real number. A background of 3 would floor to zero, drag the median to zero, and + # make every identity unreliable for a reason unrelated to the comparator. + counts = ["sampleId,cellId,tag,umiCount"] + linker = ["sampleId,cellId,setId"] + for sample in ("SmpA", "SmpB"): + for cell in ("c1", "c2", "c3"): + counts.append(f"{sample},{cell},{CUSTOMER_TAGS[0]},900") + counts.extend(f"{sample},{cell},{t},10" for t in CUSTOMER_TAGS[1:]) + linker.append(f"{sample},{cell},{'K1' if span_samples else 'K' + sample}") + (root / "counts.csv").write_text("\n".join(counts) + "\n") + (root / "linker.csv").write_text("\n".join(linker) + "\n") + return root + + +CUSTOMER_ARGS = [ + "counts.csv", + "panel.csv", + "--linker", + "linker.csv", + "--barcode-col", + "Sequence", + "--feature-col", + "Antigen", + "--sample-col", + "Sample", + # This bed's panel carries no role column, so it has no comparator tag to declare and the rung it is + # about is the panel's own readings. + "--reference-source", + "panel", + "--output-prefix", + "result", +] + + +def test_a_panel_with_no_role_column_still_produces_verdicts(bed): + # No --role-column and no --reference-values, because the file has no column to name. The run must not + # fail and must not read unreliable throughout: nine tags clear the minimum of eight, so the panel's own + # readings serve. + _customer_bed(bed) + r = _run(bed, *CUSTOMER_ARGS) + assert r.returncode == 0, r.stderr + + meta = json.loads((bed / "result_run_meta.json").read_text()) + assert meta["referenceChoice"] == ReferenceChoice.PANEL.value + assert meta["referenceValues"] == [], "nothing can be declared without a role column" + # With no grouping column either, every barcode is its own identity. + assert meta["identities"] == CUSTOMER_TAGS + + states = pl.read_csv(bed / "result_verdicts.csv", infer_schema_length=0) + assert set(states["state"].to_list()) != {"unreliable"}, "the panel could serve and was not asked to" + # The strong barcode reads bound and the background does not, or the bed cannot tell a working + # comparator from a broken one. + by_identity = dict(zip(states["identity"].to_list(), states["state"].to_list(), strict=True)) + assert by_identity[CUSTOMER_TAGS[0]] == "bound" + assert {by_identity[t] for t in CUSTOMER_TAGS[1:]} == {"not bound"} + + +def test_a_barcode_renamed_between_samples_is_labelled_with_both_names(bed): + # A barcode the two samples name differently carries no agreed antigen name. Rather than stand under the + # raw 15-mer -- which tells a scientist nothing about what happened, at the moment they most need to know + # -- it carries the names it DID declare, joined. The reagent stays recognisable and the conflict stays + # visible. This is the per-tag grouping: the panel has no grouping column at all. + _customer_bed(bed, renamed=2) + assert _run(bed, *CUSTOMER_ARGS).returncode == 0 + + labels = pl.read_csv(bed / "result_identity_labels.csv", infer_schema_length=0) + by_identity = dict(zip(labels["identity"].to_list(), labels["label"].to_list(), strict=True)) + + # The bed names tag i "Ag00i" in the first sample and "Ag10i" in the second, for the first two of them, + # so the joined label carries both in sorted order. + for i, tag in enumerate(CUSTOMER_TAGS[:2]): + assert by_identity[tag] == f"Ag{i:03d} / Ag{100 + i:03d}", f"{tag} disagrees across samples" + # The consistently-named barcodes keep their plain antigen name. Without this the assertion above would + # also pass on a build that had started joining names for every identity. + for i, tag in enumerate(CUSTOMER_TAGS[2:], start=2): + assert by_identity[tag] == f"Ag{i:03d}", f"{tag} agrees across samples and must show its name" + + +def test_two_barcodes_disagreeing_about_the_same_pair_of_names_stay_tellable_apart(bed): + # The uniqueness promise applies to the joined labels too. Two barcodes can disagree about the SAME pair + # of names, which joins to one string -- so the rescue that exists to make a conflict readable would put + # two identities under one label, the one thing the labeller promises never to do. The per-tag path needs + # nothing added for this: its existing collision rule appends the barcode to any label that repeats, + # joined or plain. This test is what keeps that true. + _customer_bed(bed, renamed=0) + rows = ["Sample,Sequence,Antigen"] + for sample, name in (("SmpA", "Shared"), ("SmpB", "Conflict")): + # The first two barcodes carry the identical pair. The rest agree, as the bed built them. + rows.extend(f"{sample},{tag},{name}" for tag in CUSTOMER_TAGS[:2]) + rows.extend(f"{sample},{tag},Ag{i:03d}" for i, tag in enumerate(CUSTOMER_TAGS[2:], start=2)) + (bed / "panel.csv").write_text("\n".join(rows) + "\n") + + assert _run(bed, *CUSTOMER_ARGS).returncode == 0 + labels = pl.read_csv(bed / "result_identity_labels.csv", infer_schema_length=0) + by_identity = dict(zip(labels["identity"].to_list(), labels["label"].to_list(), strict=True)) + + for tag in CUSTOMER_TAGS[:2]: + assert by_identity[tag] == f"Conflict / Shared ({tag})", tag + # The point of the appending, stated as the property it protects rather than as the strings above. + assert len(set(by_identity.values())) == len(by_identity), "two identities share a label" + + +def test_the_label_fallback_is_caused_by_the_disagreement_and_nothing_else(bed): + # Same bed with the renaming removed: every barcode now agrees across both samples, so no label falls + # back. This is what makes the previous test a statement about disagreement rather than about this bed's + # barcodes. + _customer_bed(bed, renamed=0) + assert _run(bed, *CUSTOMER_ARGS).returncode == 0 + + labels = pl.read_csv(bed / "result_identity_labels.csv", infer_schema_length=0) + fell_back = [ + identity + for identity, label in zip(labels["identity"].to_list(), labels["label"].to_list(), strict=True) + if identity == label + ] + assert fell_back == [], "no barcode disagrees here, so no label should fall back" + + +# --- the two shapes, run against the committed bed ---------------------------------------------- +# +# The three tests further up use an inline bed to fix what a role-less panel does. These two run the +# committed bed's own projections of the same slots, so they can be compared against each other and +# against the four-column panels. The only thing that varies is the shape of the declaration. + +NARROW_COLS = ["--barcode-col", "Sequence", "--feature-col", "Antigen", "--sample-col", "Sample"] +WIDE_COLS = ["--barcode-col", "Sequence", "--feature-col", "Name", "--sample-col", "Samples"] + +# The seven-column bed is nine antigens, under the shipped minimum of twenty-five, so a run that wants the +# panel rung asks for it and lowers the minimum to the panel it has. Neither number is the subject of the +# tests below -- they are about the ROLE column -- but the CLI requires a rung to be named, and naming one +# that cannot serve would leave every verdict unreliable and say nothing about roles. +WIDE_PANEL_RUNG = ["--reference-source", "panel", "--panel-min-members", "9"] +WIDE_DECLARED_RUNG = ["--reference-source", "declared"] + + +def _wide_roles(bed): + """tag -> the set of Type values it is declared with, from the seven-column panel.""" + panel = pl.read_csv(bed / "panel_wide.csv", infer_schema_length=0) + roles: dict[str, set[str]] = {} + for row in panel.iter_rows(named=True): + roles.setdefault(row["Sequence"], set()).add(row["Type"]) + return roles + + +def test_the_narrow_shape_labels_every_barcode_the_samples_name_differently_with_both_names(wide_bed): + # No role column, so the panel's own readings serve and the grouping is the per-tag one. The panel is + # below the shipped minimum of twenty-five, so the rung is asked for explicitly: what this test is about + # is the LABEL a barcode gets, and it needs a run that produced verdicts to look at. + # + # A barcode two samples name differently has no agreed name, and its label must never fall through to the + # raw 15-mer, which records the conflict on stderr and shows it nowhere a reader looks. It carries the + # names it DID declare, joined, exactly as a property grouping does. + narrow_size = pl.read_csv(wide_bed / "panel_narrow.csv", infer_schema_length=0)["Sequence"].n_unique() + r = _run( + wide_bed, + "counts.csv", + "panel_narrow.csv", + "--linker", + "linker.csv", + *NARROW_COLS, + "--reference-source", + "panel", + "--panel-min-members", + str(narrow_size), + "--output-prefix", + "result", + ) + assert r.returncode == 0, r.stderr + meta = json.loads((wide_bed / "result_run_meta.json").read_text()) + assert meta["referenceChoice"] == ReferenceChoice.PANEL.value + assert meta["groupingId"] == "per-tag", "this test is about the per-tag label path" + + # Read from the narrow panel itself rather than from the bed helper: its OWN feature column is what + # supplies the label here, and deriving keeps a bed regenerated under another seed asserting the same + # shape instead of the same sequences. + narrow = pl.read_csv(wide_bed / "panel_narrow.csv", infer_schema_length=0) + declared: dict[str, set[str]] = {} + for row in narrow.iter_rows(named=True): + if row["Antigen"] and row["Antigen"].strip(): + declared.setdefault(row["Sequence"], set()).add(row["Antigen"].strip()) + renamed = {t for t, names in declared.items() if len(names) > 1} + assert renamed, "the bed must rename at least one barcode or this test asserts nothing" + + labels = dict(pl.read_csv(wide_bed / "result_identity_labels.csv", infer_schema_length=0).iter_rows()) + assert len(renamed) < len(labels), "and must not rename all of them" + for tag in renamed: + assert labels[tag] == " / ".join(sorted(declared[tag])), tag + # And nothing is left standing under a bare barcode, which is the whole point: every identity here was + # named by the panel, whether the samples agreed about the name or not. + assert {i for i, label in labels.items() if i == label} == set() + + +def test_naming_the_off_target_role_as_the_comparator_deletes_the_off_target_questions(wide_bed): + # The role column says what a member is TO THE QUESTION. The comparator is a different axis. Naming the + # off-target role as the comparator does not merely move a baseline -- reference tags are held out of the + # identity universe, so the off-targets stop being asked about at all. + # The role value that marks exactly ONE tag. This version of the block reads counts against one baseline + # tag or none, so a role value marking two is refused before it can demonstrate anything -- and what is + # demonstrated here is the ROLE axis, not how several comparators combine. + roles = _wide_roles(wide_bed) + by_value: dict[str, set[str]] = {} + for tag, values in roles.items(): + if len(values) == 1: + by_value.setdefault(next(iter(values)), set()).add(tag) + single = next(v for v, tags in sorted(by_value.items()) if len(tags) == 1 and "off-target" in v.lower()) + off_target = by_value[single] + + assert ( + _run( + wide_bed, + "counts.csv", + "panel_wide.csv", + "--linker", + "linker.csv", + *WIDE_COLS, + *WIDE_PANEL_RUNG, + "--output-prefix", + "plain", + ).returncode + == 0 + ) + asked_without = {identity for _, identity in _states_prefix(wide_bed, "plain")} + + assert ( + _run( + wide_bed, + "counts.csv", + "panel_wide.csv", + "--linker", + "linker.csv", + *WIDE_COLS, + *WIDE_DECLARED_RUNG, + "--role-column", + "Type", + "--reference-values", + single, + "--output-prefix", + "named", + ).returncode + == 0 + ) + asked_with = {identity for _, identity in _states_prefix(wide_bed, "named")} + + # Without the naming they are questions. With it they are gone. + assert off_target <= asked_without, "an off-target is an identity when nothing names it a comparator" + assert not (off_target & asked_with), "naming the role deleted the off-target questions" + assert asked_with, "and must not delete every question, or the bed says nothing about which went" + + +def test_a_role_value_differing_only_in_case_is_not_matched(wide_bed): + # The observed file held six Type values that were three roles. A tag whose role is spelled + # `Off-target` is not selected by `Off-Target`, silently. + # + # The claim is now proved by WHICH tags the run names rather than by which stay questions, and it is a + # sharper proof: `Off-Target` marks two tags, so this version refuses the panel and says exactly which + # two it found. A matcher that ignored case would have found three and said so. + roles = _wide_roles(wide_bed) + agreed = {tag: next(iter(values)) for tag, values in roles.items() if len(values) == 1} + exact = {tag for tag, value in agreed.items() if value == "Off-Target"} + variant = {tag for tag, value in agreed.items() if value != "Off-Target" and value.lower() == "off-target"} + assert len(exact) > 1, "the bed must declare more than one off-target for the refusal to fire" + assert variant, "the bed must carry a case variant of the off-target role" + + r = _run( + wide_bed, + "counts.csv", + "panel_wide.csv", + "--linker", + "linker.csv", + *WIDE_COLS, + *WIDE_DECLARED_RUNG, + "--role-column", + "Type", + "--reference-values", + "Off-Target", + "--output-prefix", + "named", + expect_failure=True, + ) + assert f"declares {len(exact)} baseline tags" in r.stderr + for tag in exact: + assert tag in r.stderr, "a tag the role value names must be in the refusal" + for tag in variant: + assert tag not in r.stderr, "the case-variant tag was matched, and it must not be" + + +def _states_prefix(bed, prefix): + v = pl.read_csv(bed / f"{prefix}_verdicts.csv", infer_schema_length=0) + return {(r["setId"], r["identity"]) for r in v.iter_rows(named=True)} + + +# --- the punchcard's pivot --------------------------------------------------------- +# +# All of these run against the COMMITTED bed rather than the small inline one, and that is +# load-bearing. On the inline bed every row has cellsAnswered == cellsCouldAnswer and the panel +# yields a single identity, so swapping the two counts and shuffling the column order are both +# invisible: mutating either passed the first version of these tests. The committed bed carries +# several identities, readings whose support is short of what could have answered, and *never +# asked* positions where couldAnswer is zero. +# +# Verified by mutation: swapping the two counts, dropping the state from the value, and changing +# the separator are each caught. Dropping the `select(ordered)` that aligns the punch pivot with +# the state pivot is NOT caught and cannot be here -- polars pivots columns in order of first +# appearance, which on this bed already equals sorted order. That alignment is enforced by +# construction rather than observed by a test. + + +def _punch_bed(bed): + r = _run(bed, *_bed_args("panel_with_reference.csv")) + assert r.returncode == 0, r.stderr + return ( + pl.read_csv(bed / "result_verdicts.csv", infer_schema_length=0), + pl.read_csv(bed / "result_identity_punch.csv", infer_schema_length=0), + ) + + +def test_punch_bed_can_tell_the_two_counts_apart(wide_bed): + # The guard on the tests below. If every row answered exactly as many cells as could have, + # swapping the two counts is undetectable and the agreement test below passes while the + # punch draws the wrong size everywhere. + verdicts, punch = _punch_bed(wide_bed) + differing = verdicts.filter(pl.col("cellsAnswered") != pl.col("cellsCouldAnswer")) + assert differing.height > 0, "bed no longer distinguishes answered from could-answer" + assert len([c for c in punch.columns if c != "setId"]) > 1, "bed no longer has several identities" + + +def test_punch_pivot_agrees_with_the_long_verdicts(wide_bed): + # The punch cell is the only place its facts meet, so this is the one check that they are the SAME facts + # the long frame carries. A pivot that dropped a field, swapped the counts, or paired a state with another + # identity's numbers would still write a well-formed file. Every field is listed here on purpose: adding + # one to the value has to break this test, or the value's shape would be free to drift from the frame it + # is built from. + verdicts, punch = _punch_bed(wide_bed) + identities = sorted(set(verdicts["identity"].to_list())) + assert punch.columns == ["setId", *identities] + + expected = { + (r["setId"], r["identity"]): "|".join( + [ + r["state"], + r["cellsAnswered"], + r["cellsCouldAnswer"], + r["agreement"] or "", + r["unreliableReason"] or "", + r["cellsBound"], + ] + ) + for r in verdicts.iter_rows(named=True) + } + for row in punch.iter_rows(named=True): + for identity in identities: + assert row[identity] == expected[(row["setId"], identity)], (row["setId"], identity) + + +def test_punch_pivot_keys_and_order_match_the_state_pivot(wide_bed): + # Both pivots are gated together and ordered together: the punchcard reads one and lead + # selection reads the other, and a reader comparing them must not meet a set or an identity + # present in one and absent from the other -- or in a different column order, which is what + # makes the two frames comparable side by side at all. + _punch_bed(wide_bed) + states = pl.read_csv(wide_bed / "result_identity_summary.csv", infer_schema_length=0) + punch = pl.read_csv(wide_bed / "result_identity_punch.csv", infer_schema_length=0) + assert states.columns == punch.columns + assert states["setId"].to_list() == punch["setId"].to_list() + + +def test_punch_state_is_the_state_the_long_frame_gives(wide_bed): + # The state is the half of the cell that carries the answer, so it is asserted on its own: + # a punch whose counts are right and whose state is another identity's would still draw a + # glyph, in the wrong colour, with nothing to catch it. + verdicts, punch = _punch_bed(wide_bed) + by_key = {(r["setId"], r["identity"]): r["state"] for r in verdicts.iter_rows(named=True)} + for row in punch.iter_rows(named=True): + for identity in [c for c in punch.columns if c != "setId"]: + assert row[identity].split("|")[0] == by_key[(row["setId"], identity)] + + +def test_cell_scalars_pairs_each_cell_with_its_own_admissibility(tmp_path): + """Every cell's admissibility must be ITS OWN, not the row next to it. + + The frame this comes from is built in one order and then joined twice before the + admissibility column is attached. Polars does not promise a left frame's row order + survives a join (`maintain_order` defaults to "none"), so a positional attach can hand + cells each other's labels -- and because the file is sorted on write, nothing downstream + can tell. The keyed assertion below is what makes the pairing observable at all: asserting + the column's PRESENCE, or the multiset of its values, passes just as happily when every + label has moved one row down. + + Two distinct labels appear, which is every label this bed can reach. Since + `count-becomes-a-state` deleted the thin-reference branch the vocabulary is `admissible`, + `cell set aside by the admissibility gate`, and `no comparator for this cell` -- and the + third is unreachable here: with a declared comparator `reference_by_cell` zero-fills every + analysed cell it read nothing for, so no cell in this bed can lack one. That reason needs a + run with no comparator at all, where it is the answer for every cell and so distinguishes + nothing. + + Three of the four cells carry one label and one carries the other, so any permutation that + moves the gated label is still caught. + """ + (tmp_path / "counts.csv").write_text( + "sampleId,cellId,tag,umiCount\n" + "S1,ok1,AAAA,500\nS1,ok1,CTRL,6\n" # comparable + "S1,ok2,AAAA,500\nS1,ok2,CTRL,6\n" # comparable + "S1,low,AAAA,500\nS1,low,CTRL,1\n" # a very low comparator -- still compared, still admissible + "S1,hi,AAAA,500\nS1,hi,CTRL,400\n" # comparator above the gate -> set aside + ) + (tmp_path / "panel.csv").write_text("Samples,Name,Sequence,Type\nS1,AgA,AAAA,Target\nS1,Ctrl,CTRL,Control\n") + (tmp_path / "linker.csv").write_text("sampleId,cellId,setId\nS1,ok1,K1\nS1,ok2,K1\nS1,low,K2\nS1,hi,K2\n") + _run(tmp_path, *BASE, "--gate-threshold", "100") + + scalars = pl.read_csv(tmp_path / "result_cell_scalars.csv", infer_schema_length=0) + by_cell = {r["cellId"]: r["admissibility"] for r in scalars.iter_rows(named=True)} + assert by_cell == { + "ok1": "admissible", + "ok2": "admissible", + "low": "admissible", + "hi": "cell set aside by the admissibility gate", + } + + +# --- the panel's sample column is written in LABELS --------------------------------- +# +# Every other bed in this file uses one string on both sides: the panel's sample value IS the +# counts' sampleId. That coincidence hid a defect that made every real run answer *never asked* +# everywhere -- the panel file a scientist uploads names samples the way they do ("donor01"), +# while counts, linker and every emitted axis are keyed by the platform's opaque sampleId. +# Nothing joined, so nothing was offered to any sample that existed, and a question nobody was +# asked is correctly answered *never asked*. +# +# The two beds below therefore differ from each other ONLY in whether the two sides share a +# namespace, which is the one variable that was never varied. + +OPAQUE = "3CXWCXJ3RU3UQD22B72OYXWL" + + +@pytest.fixture +def labelled_bed(tmp_path): + (tmp_path / "counts.csv").write_text( + "sampleId,cellId,tag,umiCount\n" + f"{OPAQUE},c1,AAAA,500\n{OPAQUE},c1,CTRL,6\n" + f"{OPAQUE},c2,AAAA,500\n{OPAQUE},c2,CTRL,6\n" + ) + (tmp_path / "panel.csv").write_text( + "Samples,Name,Sequence,Type\ndonor01,AgA,AAAA,Target\ndonor01,Ctrl,CTRL,Control\n" + ) + (tmp_path / "linker.csv").write_text(f"sampleId,cellId,setId\n{OPAQUE},c1,K1\n{OPAQUE},c2,K1\n") + return tmp_path + + +def _distinct_states(bed): + """The set of states a run produced. Deliberately not named `_states`, which already exists + in this file and returns a per-key mapping.""" + v = pl.read_csv(bed / "result_verdicts.csv", infer_schema_length=0) + return set(v["state"].to_list()) + + +def test_a_label_map_joins_the_panel_to_the_counts(labelled_bed): + # The fix: the run is told which sampleId each label belongs to, so the panel's + # declarations reach the cells they were written for. + _run(labelled_bed, *BASE, "--sample-labels", json.dumps({OPAQUE: "donor01"})) + assert _distinct_states(labelled_bed) == {"bound"} + + +def test_without_the_map_a_labelled_panel_offers_nothing(labelled_bed): + # The defect, pinned so it cannot come back silently. This is not a claim that the + # behaviour is right -- it is the observable shape of the failure, and it is the reason a + # run can look finished and be empty of answers. + _run(labelled_bed, *BASE) + assert _distinct_states(labelled_bed) == {"never asked"} + + +def test_a_panel_already_keyed_by_sample_id_is_unaffected(bed): + # The map must not become mandatory: a panel whose sample values already ARE sampleIds is + # the case every other bed here exercises, and it keeps working with no map and with an + # irrelevant one. + _run(bed, *BASE) + without = _distinct_states(bed) + _run(bed, *BASE, "--sample-labels", json.dumps({"someone-else": "unrelated"})) + assert _distinct_states(bed) == without + + +def test_a_barcode_named_differently_per_sample_becomes_one_identity_per_name(tmp_path): + """A reused barcode is placed under each name its own sample declared. + + Grouping by a property makes the identity the property's value. Under + `panel-file-authority@3.0` the panel declares per tag AND sample, so a barcode carrying one + name here and another there is not a tag that "has nothing to group on" -- it is a reagent + identifier reused to cover more antigens than the study has tags, and each declaration + places it in that sample. + + AAAA is named differently across the two samples and CCCC is not, so the same run shows both + the reuse case and the ordinary one. + + Do not invert this back. AAAA must not stand alone under its raw sequence, labelled with the + two names joined ("SpikeWT / SpikeWT__alt"), nor be reported in `tagsWithoutGroupingValue`. + That shape is forced only by a dataset-wide tag->identity map, which cannot hold two + declarations for one barcode. + """ + (tmp_path / "counts.csv").write_text( + "sampleId,cellId,tag,umiCount\n" + "S1,c1,AAAA,500\nS1,c1,CCCC,7\nS1,c1,CTRL,6\n" + "S2,c2,AAAA,500\nS2,c2,CCCC,7\nS2,c2,CTRL,6\n" + ) + (tmp_path / "panel.csv").write_text( + "Samples,Name,Sequence,Type\n" + "S1,SpikeWT,AAAA,Target\n" + "S2,SpikeWT__alt,AAAA,Target\n" # same barcode, two names -> two declarations, one per sample + "S1,Lysozyme,CCCC,Target\n" + "S2,Lysozyme,CCCC,Target\n" # agrees, so it groups normally + "S1,Ctrl,CTRL,Control\nS2,Ctrl,CTRL,Control\n" + ) + (tmp_path / "linker.csv").write_text("sampleId,cellId,setId\nS1,c1,K1\nS2,c2,K2\n") + _run(tmp_path, *BASE, *NAME_GROUPING) + + labels = dict( + pl.read_csv(tmp_path / "result_identity_labels.csv", infer_schema_length=0) + .select("identity", "label") + .iter_rows() + ) + assert set(labels) == {"SpikeWT", "SpikeWT__alt", "Lysozyme"} + assert labels["SpikeWT"] == "SpikeWT" + assert labels["SpikeWT__alt"] == "SpikeWT__alt" + assert labels["Lysozyme"] == "Lysozyme" + # No bare 15-mer anywhere: the barcode is no longer an identity under this grouping. + assert "AAAA" not in labels + + meta = json.loads((tmp_path / "result_run_meta.json").read_text()) + assert meta["identityLabels"]["SpikeWT"] == "SpikeWT" + assert meta["tagsWithoutGroupingValue"] == [], "nothing was left unplaceable" + + +# --- the exported tag -> identity linker ---------------------------------------------------- + + +def test_the_linker_carries_every_identity_a_tag_feeds_exactly_once(): + # Many-to-many by design: under (tag, sample) grouping T1 feeds A in one sample and B in another, and + # both pairs are real. Deliberately NOT keyed by sample -- the linker joins a tag-keyed figure to an + # identity-keyed verdict, and neither side has a sample axis. Verdicts are (set, identity) over clonotypes + # that span samples. The per-tag figures are run-level. An axis no joined table has makes the join + # malformed rather than more precise, and label discovery rejects it. + grouping = {("T1", "s1"): "A", ("T1", "s2"): "B", ("T2", "s1"): "A", ("T2", "s2"): "A"} + frame = _linker_frame(grouping) + rows = sorted(zip(frame["tag"].to_list(), frame["identity"].to_list())) + assert rows == [("T1", "A"), ("T1", "B"), ("T2", "A")] + # T2 feeds A in both samples and appears once. Duplicate axis keys break a grid silently: one row and an + # ellipsis, no error anywhere. + assert len(rows) == len(set(rows)) + assert set(frame["1"].to_list()) == {1} + assert "sample" not in frame.columns + + +def test_a_global_declaration_adds_no_pair_of_its_own(): + # ANY_SAMPLE feeds the same identity everywhere, so it contributes that one pair and nothing more. + frame = _linker_frame({("T1", ANY_SAMPLE): "A"}) + assert sorted(zip(frame["tag"].to_list(), frame["identity"].to_list())) == [("T1", "A")] + + +# --- a grouped-on column is a declaration by construction ------------------------------------ + + +def test_a_grouped_on_column_travels_even_when_a_member_tag_is_reused(): + # `panel-file-authority`: "The columns the scientist grouped on are declarations of it, unique by + # construction." Identity B exists only because T1 is reused with a different Identity per sample, so + # tag-grain agreement drops Identity for T1 -- and B carried no declaration of the very thing it was + # grouped on. A passed only because T2 happens to agree across its samples, which is luck. + panel = pl.DataFrame( + { + "tag": ["T1", "T1", "T2", "T2"], + "sample": ["s1", "s2", "s1", "s2"], + "Identity": ["A", "B", "A", "A"], + "Channel": ["PE", "PE", "APC", "APC"], + } + ) + cols = property_columns(panel) + props, _ = consistent_properties(panel, cols) + grouping, _, _, declared = _build_grouping( + {"by": "property", "column": "Identity"}, panel, props, reference_tags=set() + ) + held = _identity_properties(grouping, props, cols, declared) + assert held["A"]["Identity"] == "A" + assert held["B"]["Identity"] == "B" + + +def test_a_column_not_grouped_on_still_needs_agreement_across_the_identity_tags(): + panel = pl.DataFrame( + { + "tag": ["T1", "T2"], + "sample": ["s1", "s1"], + "Identity": ["A", "A"], + "Channel": ["PE", "APC"], + } + ) + cols = property_columns(panel) + props, _ = consistent_properties(panel, cols) + grouping, _, _, declared = _build_grouping( + {"by": "property", "column": "Identity"}, panel, props, reference_tags=set() + ) + held = _identity_properties(grouping, props, cols, declared) + assert held["A"]["Identity"] == "A" + assert "Channel" not in held["A"], "the two member tags disagree, so Channel does not hold" + + +def _disagreements(inconsistent): + """The (column -> tag -> values) map the production call site builds.""" + out: dict[str, dict[str, list[str]]] = {} + for tag, column, values in inconsistent: + out.setdefault(column, {})[tag] = sorted(values) + return out + + +def test_a_member_that_contradicts_itself_blocks_the_property(): + """The one that inverted a real panel, and the reason `disagreed` is threaded down at all. + + T1 declares two Channels across its samples, so it has no agreed value of its own. T2 + declares one. Before the fix T1 reached the agreement test as the empty string, was filtered + out exactly like a member whose cell was blank, and T2 then agreed with nobody but itself -- + so the identity came back carrying T2's Channel as though it held of both. + + Measured on a real sixteen-row panel grouped on its role column: an identity whose five + member tags declared six different antigen names between them came back carrying ONE + member's name, because four of the five had contradicted themselves into silence. + + A member that contradicted itself is a disagreement, not a silence. + """ + panel = pl.DataFrame( + { + "tag": ["T1", "T1", "T2"], + "sample": ["s1", "s2", "s1"], + "Identity": ["A", "A", "A"], + "Channel": ["PE", "APC", "FITC"], + } + ) + cols = property_columns(panel) + props, inconsistent = consistent_properties(panel, cols) + assert props["T1"].get("Channel") is None, "T1 must have no agreed Channel or the bed proves nothing" + grouping, _, _, declared = _build_grouping( + {"by": "property", "column": "Identity"}, panel, props, reference_tags=set() + ) + + held = _identity_properties(grouping, props, cols, declared, _disagreements(inconsistent)) + assert "Channel" not in held["A"], "T2's Channel was reported as the identity's" + + # Without the disagreements the old answer is still reachable, which is what makes this a threading fix + # rather than a rewrite of the agreement rule. + assert _identity_properties(grouping, props, cols, declared)["A"]["Channel"] == "FITC" + + +def test_a_member_that_declares_nothing_still_does_not_block_its_neighbours(): + """The other silence, and it must keep behaving as it did. + + T1 leaves the cell blank. It never declared anything to contradict, so it has no + disagreement to propagate and T2's value holds of the identity. + """ + panel = pl.DataFrame( + { + "tag": ["T1", "T2"], + "sample": ["s1", "s1"], + "Identity": ["A", "A"], + "Channel": ["", "FITC"], + } + ) + cols = property_columns(panel) + props, inconsistent = consistent_properties(panel, cols) + assert inconsistent == [], "a blank cell is not a disagreement" + grouping, _, _, declared = _build_grouping( + {"by": "property", "column": "Identity"}, panel, props, reference_tags=set() + ) + held = _identity_properties(grouping, props, cols, declared, _disagreements(inconsistent)) + assert held["A"]["Channel"] == "FITC" + + +def test_a_contradicting_member_does_not_block_the_column_it_was_grouped_on(): + """Grouped-on columns are settled by construction and stay that way. + + A tag reaches an identity because of its value in the grouping column, so that value is not + open to an agreement test -- and a reused barcode has no tag-grain agreement to test in the + first place. + """ + panel = pl.DataFrame( + { + "tag": ["T1", "T1"], + "sample": ["s1", "s2"], + "Identity": ["A", "A"], + "Channel": ["PE", "APC"], + } + ) + cols = property_columns(panel) + props, inconsistent = consistent_properties(panel, cols) + grouping, _, _, declared = _build_grouping( + {"by": "property", "column": "Identity"}, panel, props, reference_tags=set() + ) + held = _identity_properties(grouping, props, cols, declared, _disagreements(inconsistent)) + assert held["A"]["Identity"] == "A" + assert "Channel" not in held["A"] + + +def test_the_per_tag_grouping_declares_nothing_of_its_identities(): + # It groups on no column, so there is nothing to take by construction. Every property still travels by + # the agreement rule. + panel = pl.DataFrame({"tag": ["T1"], "sample": ["s1"], "Channel": ["PE"]}) + cols = property_columns(panel) + props, _ = consistent_properties(panel, cols) + grouping, rule_id, _, declared = _build_grouping(None, panel, props, reference_tags=set()) + assert rule_id == "per-tag" + assert declared == {} + held = _identity_properties(grouping, props, cols, declared) + assert held["T1"]["Channel"] == "PE" + + +# --- a grouping may name several columns ----------------------------------------------------- + + +def test_two_grouping_columns_make_the_identity_the_combination(): + # `grouping-belongs-to-the-question`: "Named antigen and concentration together, the identity is the + # pair, and the same antigen at two concentrations is two identities." + panel = pl.DataFrame( + { + "tag": ["T1", "T2", "T3"], + "sample": ["s1", "s1", "s1"], + "Antigen": ["Spike", "Spike", "Nuc"], + "Dose": ["low", "high", "low"], + } + ) + grouping, rule_id, ungrouped, declared = _build_grouping( + {"by": "property", "columns": ["Antigen", "Dose"]}, panel, {}, reference_tags=set() + ) + assert grouping[("T1", "s1")] == "Spike | low" + assert grouping[("T2", "s1")] == "Spike | high" + assert grouping[("T3", "s1")] == "Nuc | low" + assert rule_id == "property:Antigen|Dose" + assert ungrouped == [] + # Both grouped-on columns are declarations of the identity, and both travel. + assert declared["Spike | high"] == {"Antigen": "Spike", "Dose": "high"} + + +def test_the_legacy_single_column_rule_still_works(): + # A project stored before the rule took a list carries `column`. It must keep running. + panel = pl.DataFrame({"tag": ["T1"], "sample": ["s1"], "Antigen": ["Spike"]}) + grouping, rule_id, _, declared = _build_grouping( + {"by": "property", "column": "Antigen"}, panel, {}, reference_tags=set() + ) + assert grouping[("T1", "s1")] == "Spike" + assert rule_id == "property:Antigen" + assert declared["Spike"] == {"Antigen": "Spike"} + + +def test_a_blank_in_any_named_column_falls_back_to_the_tag(): + # A combination missing one component is not that combination. + panel = pl.DataFrame({"tag": ["T1"], "sample": ["s1"], "Antigen": ["Spike"], "Dose": [" "]}) + grouping, _, ungrouped, declared = _build_grouping( + {"by": "property", "columns": ["Antigen", "Dose"]}, panel, {}, reference_tags=set() + ) + assert grouping[("T1", "s1")] == "T1" + assert ungrouped == ["T1"] + assert "T1" not in declared, "a pair that fell back declares nothing" + + +def test_a_column_the_panel_does_not_declare_ends_the_run(): + panel = pl.DataFrame({"tag": ["T1"], "sample": ["s1"], "Antigen": ["Spike"]}) + with pytest.raises(SystemExit) as e: + _build_grouping({"by": "property", "columns": ["Antigen", "Nope"]}, panel, {}, reference_tags=set()) + assert "Nope" in str(e.value) + + +def test_a_role_column_the_reader_consumes_as_a_key_ends_the_run_with_no_role_values(bed): + # `Sequence` is the barcode column, so panel.py strips it before the properties are read and it is never + # a property column. Naming it as the role column exited 0 whenever no role values came with it: the check + # was gated on the values, so no tag was designated and the baseline fell back to the panel's own readings + # in silence. A different number reported as the requested one is worse than a dead run, so this is the + # half of the mistake that had to stop being quiet. + r = _run( + bed, + "counts.csv", + "panel.csv", + "--linker", + "linker.csv", + "--barcode-col", + "Sequence", + "--feature-col", + "Name", + "--sample-col", + "Samples", + "--role-column", + "Sequence", + "--reference-source", + "declared", + "--output-prefix", + "result", + expect_failure=True, + ) + assert "Sequence" in r.stderr + + +def test_a_value_carrying_the_join_separator_is_reported_and_the_run_continues(capsys): + panel = pl.DataFrame({"tag": ["T1"], "sample": ["s1"], "Antigen": ["Spike | odd"], "Dose": ["low"]}) + grouping, _, _, _ = _build_grouping( + {"by": "property", "columns": ["Antigen", "Dose"]}, panel, {}, reference_tags=set() + ) + assert grouping[("T1", "s1")] == "Spike | odd | low" + assert "may share one identity key" in capsys.readouterr().err + + +def test_the_set_counts_carry_the_clonotype_cell_count(bed): + # `the-explore-readout` puts "the clonotype's own cell count beside its name" in the grid, so the grid + # needs it as a column. It is the set's cells, not its answering cells: it does not vary by identity, + # which is why it belongs beside the name rather than in every position. + _run(bed, *BASE) + counts = pl.read_csv(bed / "result_set_counts.csv", infer_schema_length=0) + assert "cellCount" in counts.columns + assert all(int(v) >= 1 for v in counts["cellCount"].to_list()) + # And it is not the answering count: that varies by identity, this one does not. + verdicts = pl.read_csv(bed / "result_verdicts.csv", infer_schema_length=0) + by_set = dict(zip(counts["setId"].to_list(), counts["cellCount"].to_list())) + for set_id, could in zip(verdicts["setId"].to_list(), verdicts["cellsCouldAnswer"].to_list()): + assert int(could) <= int(by_set[set_id]), "a set cannot answer with more cells than it has" + + +def test_set_counts_carry_the_clonotype_s_own_set_aside_cells(bed): + # 206 states set-aside cells once for the clonotype, because a set-aside cell answers nothing at any + # identity -- repeating the subtraction at every position would imply a per-identity failure that did not + # happen. Run-level is the wrong grain for that: the expansion is about one clonotype. + # + # The bed's baseline is CTRL at 6 UMIs in every cell, so a gate of 5 sets every cell aside. That gives a + # real non-zero to assert against rather than a vacuous 0 == 0. + _run(bed, *BASE, "--gate-threshold", "5") + counts = pl.read_csv(bed / "result_set_counts.csv") + assert "cellsSetAside" in counts.columns + meta = json.loads((bed / "result_run_meta.json").read_text()) + assert counts["cellsSetAside"].sum() == meta["cellsSetAside"] + assert meta["cellsSetAside"] > 0, "the gate set nothing aside, so this proves nothing" + + +def test_set_counts_report_no_set_aside_cells_when_no_gate_is_declared(bed): + # Off is the default. The column still has to be present and zero, so a reader never has to tell "no + # gate" apart from "column missing". + _run(bed, *BASE) + counts = pl.read_csv(bed / "result_set_counts.csv") + assert counts["cellsSetAside"].to_list() == [0] * len(counts) + + +def test_run_meta_carries_set_aside_cells_per_clonotype(bed): + # 206 states set-aside cells once for the clonotype, and the expansion reads them from the run record + # rather than from a p-column: a Parquet column's values cannot be read in the model, and a set-grain + # number joined into the per-identity table would repeat down every row -- which the atom forbids, because + # it implies a per-identity failure that did not happen. + # + # The bed's baseline is CTRL at 6 UMIs in every cell, so a gate of 5 sets every cell aside and the + # assertion has a real non-zero to bite on. + _run(bed, *BASE, "--gate-threshold", "5") + meta = json.loads((bed / "result_run_meta.json").read_text()) + by_set = meta["cellsSetAsideBySet"] + assert sum(by_set.values()) == meta["cellsSetAside"] + assert meta["cellsSetAside"] > 0, "the gate set nothing aside, so this proves nothing" + # Sparse: the run record is parsed on every render, so a clonotype that lost nothing carries no entry. A + # reader takes an absent key as zero. + assert all(n > 0 for n in by_set.values()) + # The CSV keeps its own dense rendering, and the two cannot disagree -- one helper produces both. + counts = pl.read_csv(bed / "result_set_counts.csv") + dense = dict(zip(counts["setId"].to_list(), counts["cellsSetAside"].to_list())) + assert by_set == {k: v for k, v in dense.items() if v > 0} + + +def test_set_counts_carry_the_clonotype_s_cells_that_read_nothing(bed): + # `the-explore-readout` carries this per clonotype, not per identity: a cell with nothing left is empty at + # every identity, and repeating the subtraction per position would report a per-identity failure that did + # not happen. + # + # At the shipped minimum nothing in this bed falls -- 500, 600 and the comparator's 6 all clear it -- so + # the column has to be present and zero rather than absent. + _run(bed, *BASE) + counts = pl.read_csv(bed / "result_set_counts.csv") + assert "cellsReadingNothing" in counts.columns + assert counts["cellsReadingNothing"].to_list() == [0] * len(counts) + + +def test_a_cell_carrying_only_its_comparator_has_not_read_nothing(bed): + # c3 was asked about AgA and read nothing of it, while its comparator read 6. That cell took up reagent + # and none of it was antigen, which `support-travels-with-the-reading` calls a real negative and a real + # vote. A minimum of 7 removes its AgA reading -- there is none to remove -- and leaves the exempt + # comparator standing, so the cell is not empty. + _run(bed, *BASE, "--floor", "7") + counts = pl.read_csv(bed / "result_set_counts.csv") + assert counts["cellsReadingNothing"].to_list() == [0] + + +def test_cells_that_read_nothing_change_no_verdict(bed): + # `support-travels-with-the-reading` forbids both shortcuts this number invites: dropping such cells from + # the vote shrinks the denominator and turns a minority into a majority, and filtering them out of the + # cell list is the same effect by another route. Raising the minimum changes which cells are counted as + # empty and must change nothing else in the run. + _run(bed, *BASE, "--floor", "1") + low = (bed / "result_verdicts.csv").read_bytes() + low_cells = (bed / "result_cell_counts.csv").read_bytes() + _run(bed, *BASE, "--floor", "7") + assert (bed / "result_verdicts.csv").read_bytes() == low + assert (bed / "result_cell_counts.csv").read_bytes() == low_cells + + +def test_a_clonotype_never_reads_nothing_in_more_cells_than_it_has(bed): + # The universe passed to the tally is the clonotype's own membership, so this cannot be violated by + # construction -- which is exactly why it is worth pinning: a later refactor that reads the population off + # the counts frame instead would break it silently. + _run(bed, *BASE, "--floor", "7") + counts = pl.read_csv(bed / "result_set_counts.csv") + for empty, total in zip(counts["cellsReadingNothing"].to_list(), counts["cellCount"].to_list()): + assert 0 <= empty <= total + + +def test_run_meta_omits_set_aside_cells_per_clonotype_when_no_gate_is_declared(bed): + # 206 shows the count only where a gate is declared. The key is ABSENT rather than an empty object, so + # the UI branches on one thing -- was a gate declared -- and never has to tell "no gate" apart from "a + # gate that took nothing". + _run(bed, *BASE) + meta = json.loads((bed / "result_run_meta.json").read_text()) + assert "cellsSetAsideBySet" not in meta + + +@pytest.fixture +def two_set_bed(tmp_path): + # Two clonotypes whose cells read the comparator differently, so a gate can catch one and leave the other + # untouched. This is the ONLY shape that can falsify a dense map: with a single clonotype, an + # implementation that emitted every clonotype including the zeros passes every other assertion in this + # file. + # + # K1's cells read CTRL at 6, so a gate of 5 takes both. K2's read it at 2, so the same gate leaves both. + # The antigen counts clear the shipped cutoff in each case, so both clonotypes still produce verdicts and + # the run is not degenerate. + (tmp_path / "counts.csv").write_text( + "sampleId,cellId,tag,umiCount\n" + "S1,c1,AAAA,500\nS1,c1,CTRL,6\n" + "S1,c2,AAAA,600\nS1,c2,CTRL,6\n" + "S1,d1,AAAA,500\nS1,d1,CTRL,2\n" + "S1,d2,AAAA,600\nS1,d2,CTRL,2\n" + ) + (tmp_path / "panel.csv").write_text("Samples,Name,Sequence,Type\nS1,AgA,AAAA,Target\nS1,Ctrl,CTRL,Control\n") + (tmp_path / "linker.csv").write_text("sampleId,cellId,setId\nS1,c1,K1\nS1,c2,K1\nS1,d1,K2\nS1,d2,K2\n") + return tmp_path + + +def test_run_meta_omits_a_clonotype_the_gate_did_not_touch(two_set_bed): + # The sparseness claim, tested where it can fail. The run record is parsed on every model render, so a + # clonotype that lost nothing carries NO entry and a reader takes an absent key as zero. A map that + # carried `"K2": 0` would defeat that and pass every relative assertion above. + _run(two_set_bed, *BASE, "--gate-threshold", "5") + meta = json.loads((two_set_bed / "result_run_meta.json").read_text()) + # Exact, not relative: K1 has two cells and the gate takes both. + assert meta["cellsSetAsideBySet"] == {"K1": 2} + assert "K2" not in meta["cellsSetAsideBySet"], "a clonotype the gate did not touch must be absent" + # The CSV stays DENSE, which is its own contract: a reader of a table must never have to tell "no gate" + # apart from "column missing". The contrast between the two renderings is the design. + counts = pl.read_csv(two_set_bed / "result_set_counts.csv") + dense = dict(zip(counts["setId"].to_list(), counts["cellsSetAside"].to_list())) + assert dense == {"K1": 2, "K2": 0} + + +@pytest.fixture +def silent_position_bed(tmp_path): + # One clonotype, two cells, two antigens, and the shape that separates "asked and silent" from "never + # asked": c1 carries counts for both antigens, c2 carries counts for AgA only. So (c2, AgB) has no row in + # `read_states` at all, and every antigen is on the sample's panel -- which makes it a SILENT position + # rather than an unasked one. An implementation that pivots the sparse frame and stops leaves that + # position blank, and blank is reserved for never-asked. + (tmp_path / "counts.csv").write_text( + "sampleId,cellId,tag,umiCount\nS1,c1,AAAA,500\nS1,c1,BBBB,400\nS1,c1,CTRL,2\nS1,c2,AAAA,600\nS1,c2,CTRL,2\n" + ) + (tmp_path / "panel.csv").write_text( + "Samples,Name,Sequence,Type\nS1,AgA,AAAA,Target\nS1,AgB,BBBB,Target\nS1,Ctrl,CTRL,Control\n" + ) + (tmp_path / "linker.csv").write_text("sampleId,cellId,setId\nS1,c1,K1\nS1,c2,K1\n") + return tmp_path + + +def _cell_punch(bed): + frame = pl.read_csv(bed / "result_cell_punch.csv") + return {(row["sampleId"], row["cellId"]): row for row in frame.iter_rows(named=True)} + + +def test_cell_punch_gives_every_cell_a_row_and_every_identity_a_column(silent_position_bed): + _run(silent_position_bed, *BASE) + rows = _cell_punch(silent_position_bed) + assert set(rows) == {("S1", "c1"), ("S1", "c2")}, "one row per cell of the set" + # The columns are the PANEL, not what this run happened to ask. The comparator is not an identity. + for key in rows: + # Keyed by the BARCODE, not the display name: with no grouping column each tag stands alone under + # its own sequence, exactly as the set-level punch keys its columns. + assert "AAAA" in rows[key] and "BBBB" in rows[key] + assert rows[key]["setId"] == "K1", "the set travels as a column so the readout can filter on it" + + +def test_cell_punch_resolves_a_silent_position_rather_than_leaving_it_blank(silent_position_bed): + # The claim this fixture exists for. (c2, AgB) has no row in the states frame, its sample offered AgB, + # and c2 can be compared -- so it reads NOT BOUND, exactly as silent_tally counts it when it produces c2's + # contribution to K1's verdict at AgB. Blank here would contradict that arithmetic. + _run(silent_position_bed, *BASE) + rows = _cell_punch(silent_position_bed) + silent = rows[("S1", "c2")]["BBBB"] + assert silent is not None, "an asked-and-silent position must not be blank" + assert silent.split("|")[0] == "not bound", silent + # And the position that DID carry counts reads bound, so the test is not passing on a frame where + # everything is not-bound. + assert rows[("S1", "c1")]["BBBB"].split("|")[0] == "bound", rows[("S1", "c1")]["BBBB"] + + +def test_cell_punch_counts_the_identities_a_cell_read_bound(silent_position_bed): + _run(silent_position_bed, *BASE) + rows = _cell_punch(silent_position_bed) + # c1 bound both antigens. C2 bound AgA and was silent, so not bound, at AgB. + assert rows[("S1", "c1")]["boundIdentities"] == 2 + assert rows[("S1", "c2")]["boundIdentities"] == 1 + + +def test_cell_punch_marks_a_gated_cell_unreliable_at_every_identity(silent_position_bed): + # A gate reading the comparator at 2 takes both cells. A gated cell was not measured at all, so no + # position of it is bound or not bound -- and its bound count is zero rather than absent. + _run(silent_position_bed, *BASE, "--gate-threshold", "1") + rows = _cell_punch(silent_position_bed) + for key, row in rows.items(): + for identity in ("AAAA", "BBBB"): + assert row[identity].split("|")[0] == "unreliable", (key, identity, row[identity]) + assert row["boundIdentities"] == 0, key + + +def test_run_meta_says_whether_the_cell_punch_was_emitted(silent_position_bed): + _run(silent_position_bed, *BASE) + meta = json.loads((silent_position_bed / "result_run_meta.json").read_text()) + assert meta["cellPunchEmitted"] is True + assert meta["cellPunchCells"] == 2 + + +def test_the_panel_comparator_is_built_from_raw_counts(tmp_path): + """The production call site passes the raw frame, not the floored one. + + The unit test in test_verdict.py pins what the two frames produce. This pins which one + production hands over, which is where the defect actually was and which no assertion in this + file reached: every fixture bed here reads well clear of the minimum, so flooring changed no + comparator and the suite stayed green either way. + + c1's five readings are 1, 1, 2, 9, 9. Raw they median to 2. Floored at the shipped minimum + of 4 they are 0, 0, 0, 9, 9 and median to 0 -- which would push every verdict in that cell + toward *bound*, since a comparator of zero is the easiest bar there is. + """ + tags = ["AAAA", "CCCC", "GGGG", "TTTT", "ACAC"] + (tmp_path / "panel.csv").write_text( + "Sample,Antigen,Sequence\n" + "".join(f"S1,Ag{i},{t}\n" for i, t in enumerate(tags)) + ) + (tmp_path / "counts.csv").write_text( + "sampleId,cellId,tag,umiCount\n" + + "".join(f"S1,c1,{t},{n}\n" for t, n in zip(tags, [1, 1, 2, 9, 9], strict=True)) + ) + (tmp_path / "linker.csv").write_text("sampleId,cellId,setId\nS1,c1,K1\n") + + _run(tmp_path, *CUSTOMER_ARGS, "--panel-min-members", "5") + + meta = json.loads((tmp_path / "result_run_meta.json").read_text()) + assert meta["referenceChoice"] == ReferenceChoice.PANEL.value + + counts = pl.read_csv(tmp_path / "result_cell_counts.csv") + refs = set(counts.filter(pl.col("cellId") == "c1")["referenceCount"].to_list()) + assert refs == {2}, "the comparator was built from floored readings" + + +DISTRIBUTION_ARGS = [ + "counts.csv", + "panel.csv", + "--linker", + "linker.csv", + "--barcode-col", + "Sequence", + "--feature-col", + "Antigen", + "--sample-col", + "Sample", + "--reference-source", + "distribution", + "--output-prefix", + "result", +] + + +def _distribution_bed(root, n_cells=400, binder_rate=300, seed=7): + """A sample whose first tag separates and whose second does not. + + Written from a seeded generator: the rung under test is a density, and a handful of + hand-written counts has no density. The seed is fixed, so the bed is the same bytes on every + run. + + `SEPS` binds in a twentieth of the cells. `FLAT` reads the SAME count in every cell, which is + one population by construction and the shape that cannot be fitted at all. + + A flat tag is deliberately not a background-shaped one. `what-plays-the-baseline` accepts that + a tag nothing bound still fits and still calls its upper tail bound -- the method assumes two + components exist and no published test replaces the eye -- so a background-shaped tag no longer + demonstrates an unfittable one. Identical counts do, and they are still a tag the reads carry, + which keeps this apart from a dead reagent. + """ + import numpy as np + + rng = np.random.default_rng(seed) + n_binders = n_cells // 20 + sep = np.concatenate([rng.poisson(2, n_cells - n_binders), rng.poisson(binder_rate, n_binders)]) + flat = np.full(n_cells, 5) + + rows = ["sampleId,cellId,tag,umiCount"] + for i in range(n_cells): + for tag, values in (("SEPS", sep), ("FLAT", flat)): + if values[i] > 0: + rows.append(f"S1,c{i},{tag},{values[i]}") + (root / "counts.csv").write_text("\n".join(rows) + "\n") + (root / "panel.csv").write_text("Sample,Antigen,Sequence\nS1,AgSep,SEPS\nS1,AgFlat,FLAT\n") + (root / "linker.csv").write_text("sampleId,cellId,setId\n" + "".join(f"S1,c{i},K{i % 4}\n" for i in range(n_cells))) + # The cell list is what fixes the fit's population, including the cells that read nothing + # for a tag. Without it the universe is only the observed cells. + (root / "cells.csv").write_text("sampleId,cellId\n" + "".join(f"S1,c{i}\n" for i in range(n_cells))) + return root + + +def test_the_tag_distribution_rung_serves_and_says_so(tmp_path): + _distribution_bed(tmp_path) + _run(tmp_path, *DISTRIBUTION_ARGS, "--cells", "cells.csv") + + meta = json.loads((tmp_path / "result_run_meta.json").read_text()) + assert meta["referenceChoice"] == ReferenceChoice.DISTRIBUTION.value + assert meta["referenceSourceRequested"] == ReferenceChoice.DISTRIBUTION.value + assert meta["distributionMinCells"] == 300 + + +def test_the_sticky_measurement_is_a_spread_when_no_gate_is_declared(bed): + # The default, and therefore the first run every scientist sees. 290: where no threshold is + # declared there is no *high* to count, and the measurement is the distribution of those + # readings instead -- which is what a scientist reads in order to declare a gate. A count here + # would assert a boundary nobody drew. + _run(bed, *BASE) + qc = pl.read_csv(bed / "result_qc.csv", infer_schema_length=0) + row = qc.filter(pl.col("measurement") == "highReferenceCells").row(0, named=True) + + assert "noGateDeclared" in row["detail"] + assert "gate=" not in row["detail"] + # Eleven decile points ride in the detail, and the value is their median. + points = [p.split(":")[0] for p in row["detail"].split("|")[2:]] + assert points == [str(p) for p in range(0, 101, 10)] + + +def test_the_sticky_measurement_counts_the_cells_the_gate_set_aside(bed): + # With a gate declared the two jobs are one number: the cells counted high are the cells set + # aside, by construction. A second line used to let those two sets differ. + _run(bed, *BASE, "--gate-threshold", "1") + qc = pl.read_csv(bed / "result_qc.csv", infer_schema_length=0) + row = qc.filter(pl.col("measurement") == "highReferenceCells").row(0, named=True) + + assert "gate=1" in row["detail"] + assert "noGateDeclared" not in row["detail"] + meta = json.loads((bed / "result_run_meta.json").read_text()) + # Same cells, counted once. The per-sample rows sum to the run's set-aside total. + per_sample = qc.filter(pl.col("measurement") == "highReferenceCells")["value"].to_list() + assert sum(int(float(v)) for v in per_sample if v is not None) == meta["cellsSetAside"] + + +def test_no_observation_line_parameter_survives(bed): + # One threshold, not two. 060-parameter-set lists seven parameters and a sticky line is not + # among them, so a run must not accept one. + assert "--high-reference-line" not in _run(bed, "--help").stdout + meta_run = _run(bed, *BASE) + assert meta_run.returncode == 0 + assert "highReferenceLine" not in json.loads((bed / "result_run_meta.json").read_text()) + + +def test_the_distributions_are_emitted_as_plottable_frames(bed): + # 330 puts three distributions last in the readout, and a scientist settles the cutoff and the + # gate by looking at them. A decile encoded inside a measurement's detail string is a number + # nobody can plot, so they also go out as frames. + _run(bed, *BASE) + + deciles = pl.read_csv(bed / "result_qc_deciles.csv", infer_schema_length=0) + kinds = set(deciles["distribution"].to_list()) + assert kinds == {"score", "referenceReading"} + for kind in kinds: + points = deciles.filter(pl.col("distribution") == kind)["decile"].to_list() + assert [int(p) for p in points] == list(range(0, 101, 10)), kind + + # Header-only rather than absent where a run fitted no background: a consumer meeting a header + # knows the step ran and found nothing. + backgrounds = pl.read_csv(bed / "result_qc_backgrounds.csv", infer_schema_length=0) + assert backgrounds.columns == [ + "sampleId", + "tag", + "backgroundMean", + "signalMean", + "backgroundWeight", + ] + assert backgrounds.height == 0, "a declared baseline fits no background" + + +@pytest.fixture +def sample_decile_bed(tmp_path): + # Three samples: S1 and S2 each hold cells with an antigen count, at different scales so their + # decile series cannot coincide by accident. S3 is declared in the panel but carries no counted + # reading at all -- the "no antigen counts" case. + (tmp_path / "counts.csv").write_text( + "sampleId,cellId,tag,umiCount\n" + "S1,c1,AAAA,500\nS1,c1,CTRL,6\nS1,c2,AAAA,900\nS1,c2,CTRL,6\n" + "S2,c1,AAAA,50\nS2,c1,CTRL,6\nS2,c2,AAAA,80\nS2,c2,CTRL,6\n" + ) + (tmp_path / "panel.csv").write_text( + "Samples,Name,Sequence,Type\n" + "S1,AgA,AAAA,Target\nS1,Ctrl,CTRL,Control\n" + "S2,AgA,AAAA,Target\nS2,Ctrl,CTRL,Control\n" + "S3,Ctrl,CTRL,Control\n" + ) + (tmp_path / "linker.csv").write_text("sampleId,cellId,setId\nS1,c1,K1\nS1,c2,K1\nS2,c1,K2\nS2,c2,K2\n") + return tmp_path + + +def test_sample_deciles_reach_a_frame_keyed_by_sample(sample_decile_bed): + # The distribution's deciles must reach a p-frame keyed by sample, not only a decile string + # buried in the measurement's detail field. + _run(sample_decile_bed, *BASE) + deciles = pl.read_csv(sample_decile_bed / "result_qc_sample_deciles.csv", infer_schema_length=0) + assert set(deciles.columns) == {"sampleId", "decile", "value"} + assert set(deciles.filter(pl.col("sampleId") == "S1")["decile"].to_list()) == set(str(p) for p in range(0, 101, 10)) + + +def test_two_samples_carry_different_decile_series(sample_decile_bed): + # S1's cells hold 500-900 antigen counts, S2's hold 50-80. Their decile series must differ -- + # a plot showing this sample alone is the point. + _run(sample_decile_bed, *BASE) + deciles = pl.read_csv(sample_decile_bed / "result_qc_sample_deciles.csv") + s1 = deciles.filter(pl.col("sampleId") == "S1").sort("decile")["value"].to_list() + s2 = deciles.filter(pl.col("sampleId") == "S2").sort("decile")["value"].to_list() + assert s1 != s2 + assert max(s1) > max(s2) + + +def test_a_sample_with_no_antigen_counts_yields_no_decile_rows(sample_decile_bed): + # S3 is declared in the panel but no read ever carried a count for it. A flat run of zeros + # would read as a real, narrow distribution; the right answer is no rows for S3 at all, with + # the sample's own measurement carrying the reason instead. + _run(sample_decile_bed, *BASE) + deciles = pl.read_csv(sample_decile_bed / "result_qc_sample_deciles.csv", infer_schema_length=0) + assert deciles.filter(pl.col("sampleId") == "S3").height == 0 + + qc = pl.read_csv(sample_decile_bed / "result_qc.csv", infer_schema_length=0) + row = qc.filter((pl.col("entity") == "S3") & (pl.col("measurement") == "antigenCountDistribution")).row( + 0, named=True + ) + assert row["value"] is None or row["value"] == "" + assert row["reason"] + + +def test_the_fitted_backgrounds_are_emitted_at_the_fits_own_grain(tmp_path): + # One row per (sample, tag) the fit scored. Aggregating to the tag would hide a reagent that + # separated in one sample and not in another, which is the comparison a reader makes here. + _distribution_bed(tmp_path) + _run(tmp_path, *DISTRIBUTION_ARGS, "--cells", "cells.csv") + + backgrounds = pl.read_csv(tmp_path / "result_qc_backgrounds.csv") + # SEPS fits, FLAT does not, so exactly one pair contributes. + assert backgrounds.height == 1 + row = backgrounds.row(0, named=True) + assert row["tag"] == "SEPS" + assert row["backgroundMean"] < row["signalMean"] + assert 0.0 < row["backgroundWeight"] < 1.0 + + +def test_a_population_baseline_emits_no_score_deciles(tmp_path): + # No score exists under that rung, so the frame carries the reference-reading rows and nothing + # claiming to be a score. + _distribution_bed(tmp_path) + _run(tmp_path, *DISTRIBUTION_ARGS, "--cells", "cells.csv") + + deciles = pl.read_csv(tmp_path / "result_qc_deciles.csv", infer_schema_length=0) + assert "score" not in set(deciles["distribution"].to_list()) + + +def test_the_run_carries_its_score_spread(bed): + # 320 puts this at the run grain because the cutoff is one number for the run, and carries it + # so a scientist can move that cutoff to where their own scores separate. A cutoff set with no + # sight of the scores is set blind, which is what shipped until now. + _run(bed, *BASE) + qc = pl.read_csv(bed / "result_qc.csv", infer_schema_length=0) + + rows = qc.filter(pl.col("measurement") == "scoreDistribution") + assert rows.height == 1, "one figure for the run, not one per sample" + row = rows.row(0, named=True) + assert row["level"] == "run" + assert row["value"] is not None + # Eleven decile points, 0 through 100 by 10. + points = [p.split(":")[0] for p in row["detail"].split("|")] + assert points == [str(p) for p in range(0, 101, 10)] + # A score is 0 to 100, and the deciles are ordered. + values = [float(p.split(":")[1]) for p in row["detail"].split("|")] + assert values == sorted(values) + assert 0.0 <= values[0] and values[-1] <= 100.0 + # No line stands behind it: 320 carries it so a scientist places the cutoff, and a line here + # would be the block placing it instead. + assert row["status"] is None + + +def test_the_run_score_spread_stays_out_of_every_sample_rollup(bed): + # It is emitted outside the sample loop, and a sample's rollup covers its OWN measurements. + # A run figure folded into a sample would say something about that sample it does not know. + _run(bed, *BASE) + qc = pl.read_csv(bed / "result_qc.csv", infer_schema_length=0) + assert qc.filter((pl.col("measurement") == "rollup") & (pl.col("level") == "run")).height == 0 + + +def test_a_population_baseline_has_no_score_to_spread(tmp_path): + # The declared rung scores; the distribution rung yields a probability, which is not on the + # same scale and cannot be pooled with one. The row is there and says so, rather than going + # missing or printing a number from the wrong rule. + _distribution_bed(tmp_path) + _run(tmp_path, *DISTRIBUTION_ARGS, "--cells", "cells.csv") + + qc = pl.read_csv(tmp_path / "result_qc.csv", infer_schema_length=0) + row = qc.filter(pl.col("measurement") == "scoreDistribution").row(0, named=True) + assert row["value"] is None + assert "yields no score" in row["detail"] + + +def test_the_fitted_background_reaches_the_measurement_set(tmp_path): + # The fit's parameters used to die inside the function that made them, so a scientist could + # not see whether a tag's counts separated -- which 330 wants read BEFORE the baseline is + # settled. SEPS separates and FLAT does not, so one row carries a number and the other + # carries why it has none. Both rows exist: absence and non-separation are different facts. + _distribution_bed(tmp_path) + _run(tmp_path, *DISTRIBUTION_ARGS, "--cells", "cells.csv") + + qc = pl.read_csv(tmp_path / "result_qc.csv", infer_schema_length=0) + rows = {r["entity"]: r for r in qc.filter(pl.col("measurement") == "fittedBackground").iter_rows(named=True)} + assert set(rows) == {"SEPS", "FLAT"} + + seps = rows["SEPS"] + assert seps["value"] is not None + assert "samplesFitted=1" in seps["detail"] + assert "medianSignalMean=" in seps["detail"] + # The background sits below the signal it was separated from. Read together they are the + # finding: a background alone says nothing about whether the counts separated. + signal = float(seps["detail"].split("medianSignalMean=")[1].split("|")[0]) + assert float(seps["value"]) < signal + + flat = rows["FLAT"] + assert flat["value"] is None + assert "fitted in no sample" in flat["detail"] + + # No line stands behind either, so neither carries a status. + assert seps["status"] is None + assert flat["status"] is None + + +def test_a_declared_baseline_fits_no_background_and_the_rows_say_so(bed): + # Every declared measurement keeps its place. A reader must not have to tell "this run did + # not fit one" apart from "nothing here measures that" by the row being missing. + _run(bed, *BASE) + + qc = pl.read_csv(bed / "result_qc.csv", infer_schema_length=0) + rows = qc.filter(pl.col("measurement") == "fittedBackground") + assert rows.height > 0, "a declared-baseline run still carries a row for every declared tag" + assert set(rows["value"].to_list()) == {None} + assert all("no population baseline served" in d for d in rows["detail"].to_list()) + + +def test_a_tag_that_could_not_be_fitted_leaves_its_identity_alone_unreliable(tmp_path): + # The whole point of a comparator keyed by identity rather than by cell: one tag fails to + # fit and only the identities built from it lose their verdicts. Under a cell-keyed + # comparator this run would be all-or-nothing. + _distribution_bed(tmp_path) + _run(tmp_path, *DISTRIBUTION_ARGS, "--cells", "cells.csv") + + meta = json.loads((tmp_path / "result_run_meta.json").read_text()) + assert list(meta["distributionUnfitted"]) == ["S1/FLAT"], meta["distributionUnfitted"] + + # The panel declares no grouping column, so every barcode is its own identity and the + # identity names here are the barcodes. + v = pl.read_csv(tmp_path / "result_verdicts.csv", infer_schema_length=0) + states = { + identity: set(v.filter(pl.col("identity") == identity)["state"].to_list()) for identity in ("FLAT", "SEPS") + } + assert states["FLAT"] == {"unreliable"} + assert "unreliable" not in states["SEPS"], "the tag that separated must still be answerable" + + # The set-level verdict is a majority of its cells, and only a twentieth of them bind, so + # every clonotype here reads *not bound* and reads it from a comparator that served. The + # binding is visible one level down, and the bed is worth nothing unless it is there. + punch = pl.read_csv(tmp_path / "result_cell_punch.csv", infer_schema_length=0) + assert any(x.startswith("bound|") for x in punch["SEPS"].to_list() if x is not None) + + +def test_the_fit_is_per_tag_and_not_per_cell(tmp_path): + # It is fitted per (sample, tag), so whether a position can be answered turns on the TAG. A + # cell-keyed comparator would make a cell either comparable or not, and no run could then + # produce one all-unreliable column beside one with none. + _distribution_bed(tmp_path) + _run(tmp_path, *DISTRIBUTION_ARGS, "--cells", "cells.csv") + + v = pl.read_csv(tmp_path / "result_cell_punch.csv", infer_schema_length=0) + assert v.height > 0 + # Every position of the unfitted identity reads unreliable, and no position of the fitted + # one does. Under a cell-keyed comparator a cell is either comparable or not, so no run + # could produce this pair of columns. + flat = [x for x in v["FLAT"].to_list() if x is not None] + fitted = [x for x in v["SEPS"].to_list() if x is not None] + assert flat and all(x.startswith("unreliable|") for x in flat) + assert fitted and not any(x.startswith("unreliable|") for x in fitted) + + +def test_a_sample_below_the_cell_condition_finishes_and_establishes_no_baseline(tmp_path): + # 200 cells, against the three hundred this rung needs. This is the ONE refusal that cannot be caught + # from the settings: whether a sample holds enough cells whose counts separate is a property of the data, + # so the only way to learn it is to count. So the run FINISHES rather than refusing up front, and then + # says that no baseline could be established and draws no punchcard. + _distribution_bed(tmp_path, n_cells=200) + r = _run(tmp_path, *DISTRIBUTION_ARGS, "--cells", "cells.csv") + assert r.returncode == 0, r.stderr + + meta = json.loads((tmp_path / "result_run_meta.json").read_text()) + assert meta["baselineEstablished"] is False + assert "no baseline could be established" in meta["noBaselineReason"] + # The rung that was asked for is still what is recorded. It served in the sense that nothing substituted + # for it -- there is no rung below to fall to. + assert meta["referenceChoice"] == ReferenceChoice.DISTRIBUTION.value + assert meta["referenceSourceRequested"] == ReferenceChoice.DISTRIBUTION.value + + # The answer frames keep their headers and carry no rows. A reader still finds its columns, and a + # consumer that reads them anyway finds nothing rather than a grid of non-answers. + v = pl.read_csv(tmp_path / "result_verdicts.csv", infer_schema_length=0) + assert v.height == 0 + assert "state" in v.columns + + # The structural frames are written in full: they describe the run rather than answering it, and a reader + # working out why no baseline could be established needs them. + assert pl.read_csv(tmp_path / "result_tag_identity.csv", infer_schema_length=0).height > 0 + + +def test_the_gate_exposure_is_not_evaluated_where_no_cell_has_a_comparator(tmp_path): + # There is no per-cell comparator for a gate to read, so the count is not a measurement + # this run made. None, never 0 -- a zero would report a run with no high background rather + # than one where the question does not arise. + _distribution_bed(tmp_path) + _run(tmp_path, *DISTRIBUTION_ARGS, "--cells", "cells.csv") + + meta = json.loads((tmp_path / "result_run_meta.json").read_text()) + assert meta["cellsHighReference"] is None + assert meta["cellsSetAside"] == 0 + + +def test_the_minimum_never_reaches_the_comparator(wide_bed): + """The exemption, checked end to end on a bed carrying a below-minimum comparator reading. + + `minimum-count-before-any-reference` makes this a rule rather than a preference, so there is no + switch to compare against. Raising the minimum must remove antigen readings and leave every + comparator reading standing, because the minimum asks whether a count is evidence of binding and + a tag declared to be bound by nothing never is. + """ + assert ( + _run(wide_bed, *_bed_args("panel_with_reference.csv"), "--floor", "1", "--output-prefix", "low").returncode == 0 + ) + assert ( + _run(wide_bed, *_bed_args("panel_with_reference.csv"), "--floor", "7", "--output-prefix", "high").returncode + == 0 + ) + + low = pl.read_csv(wide_bed / "low_cell_scalars.csv").sort(["sampleId", "cellId"]) + high = pl.read_csv(wide_bed / "high_cell_scalars.csv").sort(["sampleId", "cellId"]) + assert low["referenceCount"].to_list() == high["referenceCount"].to_list(), ( + "a comparator reading moved when the minimum rose, so the exemption is not holding" + ) + + # The guard: without it this passes on a bed where the minimum reaches nothing, and proves only + # that nothing happened. + low_meta = json.loads((wide_bed / "low_run_meta.json").read_text()) + high_meta = json.loads((wide_bed / "high_run_meta.json").read_text()) + assert high_meta["readingsFloored"] > low_meta["readingsFloored"] + + # The switch is gone from the contract, not merely defaulted off. + assert "minimumAppliesToBaseline" not in high_meta + + +def test_a_tag_holding_no_cell_is_not_blamed_on_its_siblings(bed): + # Three tags on one identity. AAAA and BBBB agree in every cell; CCCC is declared and + # holds no row anywhere. Its siblings did reach a majority, so the row must not say they + # failed -- that sends a reader to re-prepare two working reagents instead of the missing + # one. The reason is the only thing separating the two absences: both carry no rate. + (bed / "panel.csv").write_text( + "Samples,Name,Sequence,Type,Family\n" + "S1,AgA,AAAA,Target,Fam\n" + "S1,AgB,BBBB,Target,Fam\n" + "S1,AgC,CCCC,Target,Fam\n" + "S1,Ctrl,CTRL,Control,Reference\n" + ) + (bed / "counts.csv").write_text( + "sampleId,cellId,tag,umiCount\n" + "S1,c1,AAAA,500\nS1,c1,BBBB,500\nS1,c1,CTRL,6\n" + "S1,c2,AAAA,600\nS1,c2,BBBB,600\nS1,c2,CTRL,6\n" + ) + (bed / "linker.csv").write_text("sampleId,cellId,setId\nS1,c1,K1\nS1,c2,K1\n") + + _run(bed, *BASE, "--grouping", json.dumps({"by": "property", "column": "Family"})) + qc = pl.read_csv(bed / "result_qc.csv", infer_schema_length=0) + by_tag = {r["entity"]: r for r in qc.filter(pl.col("measurement") == "siblingDisagreement").iter_rows(named=True)} + + assert by_tag["AAAA"]["value"] == "0.0" + assert by_tag["BBBB"]["value"] == "0.0" + assert by_tag["CCCC"]["value"] is None + assert by_tag["CCCC"]["detail"] == "this tag holds no cell beside a sibling" + assert by_tag["AAAA"]["detail"] is None + + +def test_a_tag_that_is_the_only_one_on_its_identity_says_so(bed): + # The shipped bed groups per tag, so AAAA's identity is AAAA and carries nothing else. + # The reason has to name the missing sibling, not the siblings' failure to agree. + _run(bed, *BASE) + qc = pl.read_csv(bed / "result_qc.csv", infer_schema_length=0) + row = qc.filter((pl.col("measurement") == "siblingDisagreement") & (pl.col("entity") == "AAAA")).row(0, named=True) + assert row["value"] is None + assert row["detail"] == "this identity carries one tag, so it has no sibling" + + +REAGENT_COLUMNS = [ + "panelId", + "tag", + "identity", + "samplesSeenIn", + "samplesInPanel", + "samplesSeenInNames", + "samplesInPanelNames", + "cellsWithCount", + "cellsAboveTheLine", + "medianCountPerCell", + "siblingDisagreement", + "selfDisagreement", + "reason", +] + + +def test_the_reagent_table_names_every_absent_figure(bed): + # 330-the-quality-readout fixes the columns and forbids a status. A blank and a zero are + # opposite findings, so a figure with no value says which case it is. + (bed / "panel.csv").write_text( + "Samples,Name,Sequence,Type\nS1,AgA,AAAA,Target\nS1,AgD,DEAD,Target\nS1,Ctrl,CTRL,Control\n" + ) + _run(bed, *BASE) + reagents = pl.read_csv(bed / "result_reagents.csv", infer_schema_length=0) + assert reagents.columns == REAGENT_COLUMNS + assert "status" not in reagents.columns + + control = reagents.filter(pl.col("tag") == "CTRL").row(0, named=True) + assert control["cellsAboveTheLine"] is None + assert "cellsAboveTheLine=none asked, this tag supplies the baseline" in control["reason"] + + # A tag no read carried: zero under the counts, and the median names its own absence rather + # than leaving a blank beside them. + dead = reagents.filter(pl.col("tag") == "DEAD").row(0, named=True) + assert int(dead["cellsWithCount"]) == 0 + assert int(dead["samplesSeenIn"]) == 0 + assert dead["medianCountPerCell"] is None + assert "medianCountPerCell=no cell holds a count of this tag" in dead["reason"] + + target = reagents.filter(pl.col("tag") == "AAAA").row(0, named=True) + assert int(target["cellsWithCount"]) == 2 + assert int(target["samplesSeenIn"]) == 1 + # One tag under this identity, so no sibling comparison exists and the row says so. + assert target["siblingDisagreement"] is None + assert "siblingDisagreement=this identity carries one tag" in target["reason"] + + +def test_a_barcode_reused_for_two_antigens_takes_a_row_under_each(tmp_path): + # One row could not name both identities, and putting the two side by side is the comparison + # the table exists for: a barcode that worked where it carried one antigen and failed where it + # carried another. Each row's figures are scoped to the samples where the tag carried that + # identity, so the pair can differ. + (tmp_path / "counts.csv").write_text( + "sampleId,cellId,tag,umiCount\n" + "S1,c1,AAAA,500\nS1,c1,CTRL,6\nS1,c2,AAAA,600\nS1,c2,CTRL,6\n" + "S2,c1,AAAA,500\nS2,c1,CTRL,6\nS2,c2,CTRL,6\n" + ) + (tmp_path / "panel.csv").write_text( + "Samples,Name,Sequence,Type\n" + "S1,AgA,AAAA,Target\nS1,Ctrl,CTRL,Control\n" + "S2,AgB,AAAA,Target\nS2,Ctrl,CTRL,Control\n" + ) + (tmp_path / "linker.csv").write_text("sampleId,cellId,setId\nS1,c1,K1\nS1,c2,K1\nS2,c1,K2\nS2,c2,K2\n") + _run(tmp_path, *BASE, "--grouping", json.dumps({"by": "property", "column": "Name"})) + + reagents = pl.read_csv(tmp_path / "result_reagents.csv", infer_schema_length=0) + rows = reagents.filter(pl.col("tag") == "AAAA") + assert sorted(rows["identity"].to_list()) == ["AgA", "AgB"] + + figures = {r["identity"]: r for r in rows.iter_rows(named=True)} + # Two cells of S1 hold the barcode, one cell of S2 does. The figures are per (tag, identity) + # and not per tag, so the two rows carry the reagent's two behaviours rather than one number + # repeated. + assert int(figures["AgA"]["cellsWithCount"]) == 2 + assert int(figures["AgB"]["cellsWithCount"]) == 1 + # The denominator is the roster for the identity, not the panel's. + assert int(figures["AgA"]["samplesInPanel"]) == 1 + assert int(figures["AgB"]["samplesInPanel"]) == 1 + + +def test_a_staged_reagent_reads_apart_from_a_dead_one(tmp_path): + # STAGE is declared on S1 and S2 only, and carries a count on both -- staged into a + # two-sample study by design. DEAD is declared on all four samples and carries a count on + # none. Declaring DEAD on every sample splits it across two panels (S1+S2 share one + # declared tag set, S3+S4 share another), so it takes one row per panel; both must read + # empty under samplesSeenInNames, and neither reads like STAGE's row. + (tmp_path / "counts.csv").write_text( + "sampleId,cellId,tag,umiCount\n" + "S1,c1,STAGE,500\nS1,c1,CTRL,6\n" + "S2,c1,STAGE,500\nS2,c1,CTRL,6\n" + "S3,c1,CTRL,6\nS4,c1,CTRL,6\n" + ) + (tmp_path / "panel.csv").write_text( + "Samples,Name,Sequence,Type\n" + "S1,AgStage,STAGE,Target\nS2,AgStage,STAGE,Target\n" + "S1,AgDead,DEAD,Target\nS2,AgDead,DEAD,Target\nS3,AgDead,DEAD,Target\nS4,AgDead,DEAD,Target\n" + "S1,Ctrl,CTRL,Control\nS2,Ctrl,CTRL,Control\nS3,Ctrl,CTRL,Control\nS4,Ctrl,CTRL,Control\n" + ) + (tmp_path / "linker.csv").write_text("sampleId,cellId,setId\nS1,c1,K1\nS2,c1,K2\nS3,c1,K3\nS4,c1,K4\n") + _run(tmp_path, *BASE) + + reagents = pl.read_csv(tmp_path / "result_reagents.csv", infer_schema_length=0) + + stage = reagents.filter(pl.col("tag") == "STAGE").row(0, named=True) + assert stage["samplesInPanelNames"] == "S1, S2" + assert stage["samplesSeenInNames"] == "S1, S2" + + dead_rows = reagents.filter(pl.col("tag") == "DEAD") + assert dead_rows.height == 2 + assert sorted(dead_rows["samplesInPanelNames"].to_list()) == ["S1, S2", "S3, S4"] + # Empty, not null: a blank and a zero are opposite findings, and no read carried this tag. + assert set(dead_rows["samplesSeenInNames"].fill_null("").to_list()) == {""} + + +def test_reagent_sample_names_come_from_the_run_labels_not_the_id(labelled_bed): + # `label_of_sample` is reachable here, so a raw sampleId never has to reach this + # user-facing column. + _run(labelled_bed, *BASE, "--sample-labels", json.dumps({OPAQUE: "donor01"})) + reagents = pl.read_csv(labelled_bed / "result_reagents.csv", infer_schema_length=0) + row = reagents.filter(pl.col("tag") == "AAAA").row(0, named=True) + assert row["samplesInPanelNames"] == "donor01" + assert row["samplesSeenInNames"] == "donor01" + + +def _sample_report(bed, sample: str = "S1") -> dict: + return json.loads((bed / "result_qc_by_sample.json").read_text())[sample] + + +def test_the_sample_report_lists_every_sample_measurement(bed): + # A measurement that did not run takes a row rather than being omitted, so a reader meets it + # instead of noticing an absence. The set is the declaration order of every sample-level + # measurement, deferred ones included. + _run(bed, *BASE) + report = _sample_report(bed) + listed = [m["id"] for m in report["measurements"]] + assert listed == [m.id for m in MEASUREMENTS if m.level == "sample"] + + # The shape both the model and the UI are typed against. A field renamed on this side reaches + # them as an undefined, which renders as a blank rather than as an error. + assert set(report) == {"status", "judged", "unjudged", "notEvaluated", "measurements"} + for row in report["measurements"]: + assert set(row) == { + "id", + "label", + "value", + "detail", + "reason", + "status", + "counts", + "implies", + "rollsUp", + } + + +def test_a_sample_measurement_with_no_value_states_why(bed): + # This bed passes no --qc-summary, so panelAssignedFraction never arrived. A blank and a zero + # are opposite findings, so the row carries the reason where its number would have been. No + # line can be applied to a value that does not exist, so it carries no status either. + _run(bed, *BASE) + row = {m["id"]: m for m in _sample_report(bed)["measurements"]}["panelAssignedFraction"] + + assert row["value"] is None + assert row["reason"], "a measurement with no value names the reason in its place" + assert row["status"] is None + + +def test_no_sample_measurement_is_blank_without_a_reason(bed): + # The invariant, over the whole set rather than one row: every entry either carries a number + # or says why it does not. Neither case is ever rendered as an absence. + _run(bed, *BASE) + for row in _sample_report(bed)["measurements"]: + if row["value"] is None: + assert row["reason"], f"{row['id']} has no value and no reason" + assert row["label"] and row["counts"], row["id"] + + +def test_a_valueless_measurement_names_the_input_that_is_actually_missing(bed): + # Truthiness is not the test. A measurement with more than one route to having no number has + # to name the one that happened, or the report sends a reader at the wrong input. This bed + # passes a linker, so a cell list exists and depth is missing its NUMERATOR, not its + # denominator -- and the row two above it says where the read counts were to come from. + _run(bed, *BASE) + rows = {m["id"]: m for m in _sample_report(bed)["measurements"]} + + assert rows["readsTotal"]["reason"] == "no read QC summary row reached this sample" + assert rows["readsPerCell"]["reason"] == "no read count reached this sample, so depth has no numerator" + assert "cellsInList=3" in rows["readsPerCell"]["detail"] + + +def test_a_run_with_no_cell_list_gives_its_two_cell_rows_one_account(tmp_path): + # `in_list` is empty both when no list arrived and when one arrived holding nothing, so these + # two rows are reachable together on every run without a linker or a cell file. They named + # different causes: one said the list was absent, the other said the sample's listed cells + # held no reading. A reader cannot act on two accounts of one fact. + (tmp_path / "counts.csv").write_text( + "sampleId,cellId,tag,umiCount\nS1,c1,AAAA,500\nS1,c1,CTRL,6\nS1,c2,AAAA,600\nS1,c2,CTRL,6\n" + ) + (tmp_path / "panel.csv").write_text("Samples,Name,Sequence,Type\nS1,AgA,AAAA,Target\nS1,Ctrl,CTRL,Control\n") + args = [a for a in BASE if a not in ("--linker", "linker.csv")] + _run(tmp_path, *args) + + rows = {m["id"]: m for m in _sample_report(tmp_path)["measurements"]} + assert rows["readsPerCell"]["reason"] == "no cell list supplied, so depth has no denominator" + assert rows["uniqueCountsPerCell"]["reason"] == "no cell list supplied, so no cell of this sample is listed" + + +def test_an_empty_cell_list_is_the_zero_cells_finding_and_not_a_missing_read_count(tmp_path): + # A cell list that arrived and holds no cell of this sample is a different finding from no + # list at all, and `reads_per_cell` returns no number for both. Naming the read count would + # point a reader at an input that is present. + (tmp_path / "counts.csv").write_text( + "sampleId,cellId,tag,umiCount\nS1,c1,AAAA,500\nS1,c1,CTRL,6\nS2,c1,AAAA,500\nS2,c1,CTRL,6\n" + ) + (tmp_path / "panel.csv").write_text( + "Samples,Name,Sequence,Type\nS1,AgA,AAAA,Target\nS1,Ctrl,CTRL,Control\n" + "S2,AgA,AAAA,Target\nS2,Ctrl,CTRL,Control\n" + ) + (tmp_path / "linker.csv").write_text("sampleId,cellId,setId\nS1,c1,K1\nS2,c1,K2\n") + # The list covers S1 only, so S2 gets a list that answers "no cell here". + (tmp_path / "cells.csv").write_text("sampleId,cellId\nS1,c1\n") + (tmp_path / "read_qc.csv").write_text("sampleId,readsTotal,readsMatched\nS1,2000,1800\nS2,2000,1800\n") + _run(tmp_path, *BASE, "--cells", "cells.csv", "--qc-summary", "read_qc.csv") + + report = json.loads((tmp_path / "result_qc_by_sample.json").read_text()) + row = {m["id"]: m for m in report["S2"]["measurements"]}["readsPerCell"] + assert row["value"] is None + assert row["reason"] == "no cell of this sample is in the cell list, so depth has no denominator" + # The read count IS present, which is what makes the missing-numerator reason false here. + assert {m["id"]: m for m in report["S2"]["measurements"]}["readsTotal"]["value"] == 2000 + + +def test_the_sample_report_carries_the_rollup_the_qc_frame_carries(bed): + # The Main grid's tag is this rollup, and the report beside it lists the measurements it was + # taken over. One number in two places would let the tag and the list disagree about one + # sample, which is the defect this file exists to keep out. + _run(bed, *BASE) + report = _sample_report(bed) + + qc = pl.read_csv(bed / "result_qc.csv", infer_schema_length=0) + rollup = qc.filter( + (pl.col("measurement") == "rollup") & (pl.col("level") == "sample") & (pl.col("entity") == "S1") + ).row(0, named=True) + + assert report["status"] == rollup["status"] + assert report["judged"] == int(rollup["judged"]) + assert report["unjudged"] == int(rollup["unjudged"]) + assert report["notEvaluated"] == int(rollup["notEvaluated"]) + # Coverage accounts for every measurement the rollup was taken over. + counted = [m for m in report["measurements"] if m["rollsUp"]] + assert report["judged"] + report["unjudged"] + report["notEvaluated"] == len(counted) + + +def test_usable_read_fraction_reads_blank_where_the_counts_file_carries_no_total_weight(bed): + # `usableReadFraction` now has a call site (`qc_measures.usable_read_fraction`, wired from + # `counts.csv`'s `totalWeight` column). `bed`'s counts.csv predates that column, so the row + # reads a stated blank rather than a value -- never `UNSUPPLIED_REASON`, since a call site + # ran and found the column absent, which is a different fact from nothing having run. + _run(bed, *BASE) + row = {m["id"]: m for m in _sample_report(bed)["measurements"]}["usableReadFraction"] + + assert row["value"] is None + assert row["status"] is None + assert row["reason"] == "the counts file carries no totalWeight column" + + +def test_usable_read_fraction_computes_a_real_value_end_to_end(tmp_path): + # c1 and c2 are in the cell list; c3 is not. Only c1 and c2's totalWeight counts toward the + # numerator, over readsTotal from --qc-summary. + (tmp_path / "counts.csv").write_text( + "sampleId,cellId,tag,umiCount,totalWeight\n" + "S1,c1,AAAA,500,80\n" + "S1,c1,CTRL,6,3\n" + "S1,c2,AAAA,600,90\n" + "S1,c2,CTRL,6,3\n" + "S1,c3,CTRL,6,3\n" + ) + (tmp_path / "panel.csv").write_text("Samples,Name,Sequence,Type\nS1,AgA,AAAA,Target\nS1,Ctrl,CTRL,Control\n") + (tmp_path / "cells.csv").write_text("sampleId,cellId\nS1,c1\nS1,c2\n") + (tmp_path / "qc.csv").write_text("sampleId,readsTotal\nS1,1000\n") + no_linker = [a for a in BASE if a not in ("--linker", "linker.csv")] + + _run(tmp_path, *no_linker, "--cells", "cells.csv", "--qc-summary", "qc.csv") + row = {m["id"]: m for m in _sample_report(tmp_path)["measurements"]}["usableReadFraction"] + + assert row["value"] == pytest.approx((80 + 3 + 90 + 3) / 1000) + assert row["detail"] == "cellsInList=2" + + +def test_usable_read_fraction_with_no_cell_list_reads_a_stated_blank(tmp_path): + # totalWeight and readsTotal both arrive; only the cell list is missing, so the reason is + # `usable_read_fraction`'s own -- the called-cell condition, and nothing else. + (tmp_path / "counts.csv").write_text( + "sampleId,cellId,tag,umiCount,totalWeight\nS1,c1,AAAA,500,80\nS1,c1,CTRL,6,3\n" + ) + (tmp_path / "panel.csv").write_text("Samples,Name,Sequence,Type\nS1,AgA,AAAA,Target\nS1,Ctrl,CTRL,Control\n") + (tmp_path / "qc.csv").write_text("sampleId,readsTotal\nS1,1000\n") + no_linker = [a for a in BASE if a not in ("--linker", "linker.csv")] + + _run(tmp_path, *no_linker, "--qc-summary", "qc.csv") + row = {m["id"]: m for m in _sample_report(tmp_path)["measurements"]}["usableReadFraction"] + + assert row["value"] is None + assert row["status"] is None + assert row["reason"] == "no cell list supplied, so the called-cell condition cannot be evaluated" + + +def test_usable_read_fraction_with_an_empty_cell_list_reads_zero(tmp_path): + # A present cells.csv with a header and no rows is a checked, empty list -- distinct from + # no list at all, so the outcome is the real finding 0.0 rather than a blank. + (tmp_path / "counts.csv").write_text( + "sampleId,cellId,tag,umiCount,totalWeight\nS1,c1,AAAA,500,80\nS1,c1,CTRL,6,3\n" + ) + (tmp_path / "panel.csv").write_text("Samples,Name,Sequence,Type\nS1,AgA,AAAA,Target\nS1,Ctrl,CTRL,Control\n") + (tmp_path / "cells.csv").write_text("sampleId,cellId\n") + (tmp_path / "qc.csv").write_text("sampleId,readsTotal\nS1,1000\n") + no_linker = [a for a in BASE if a not in ("--linker", "linker.csv")] + + _run(tmp_path, *no_linker, "--cells", "cells.csv", "--qc-summary", "qc.csv") + row = {m["id"]: m for m in _sample_report(tmp_path)["measurements"]}["usableReadFraction"] + + assert row["value"] == 0.0 + assert row["detail"] == "cellsInList=0" + + +def test_a_declared_sample_measurement_nothing_computes_still_takes_a_row(monkeypatch): + # The walk is over the DECLARATION, not over the rows a run happened to emit. Every declared + # measurement has a call site today, so an implementation iterating the rows passes every + # other test in this file byte for byte. This is the one that separates them. + extra = Measurement("neverComputed", "Never computed", "sample", "nothing computes this") + monkeypatch.setattr(qc_rows, "MEASUREMENTS", MEASUREMENTS + (extra,)) + + rows = [] + qc_rows._add(rows, "sample", "S1", "readsTotal", 10.0) + entries, coverage = qc_rows.sample_report_rows("S1", rows) + + declared = [m for m in MEASUREMENTS if m.level == "sample"] + assert len(entries) == len(declared) + 1, "one row per declared measurement, not per emitted row" + + row = {e["id"]: e for e in entries}["neverComputed"] + assert row["value"] is None + assert row["status"] is None + assert row["reason"] == qc_rows.UNSUPPLIED_REASON + # Nothing computed it, so it counts as unchecked rather than as checked and fine. + assert coverage.not_evaluated >= 1 + + +def test_qc_frame_carries_the_line_and_route_for_an_inherited_measurement(): + # cellBarcodeValidFraction is on the inherited route with a published warn/error pair. A + # reader who sees `warn` in the frame must be able to see the number it warned against. + rows = [] + qc_rows._add(rows, "sample", "S1", "cellBarcodeValidFraction", 0.6) + frame = qc_rows._qc_frame(rows).row(0, named=True) + + assert frame["lineWarn"] == pytest.approx(0.75) + assert frame["lineAlert"] == pytest.approx(0.50) + assert frame["route"] == "inherited" + + +def test_qc_frame_leaves_the_numbers_null_for_the_categorical_route(): + # cellsDetected carries a route -- its status is a fact, not a threshold -- but no numeric + # line: `route` is non-null while `lineWarn` and `lineAlert` stay null. + rows = [] + qc_rows._add(rows, "sample", "S1", "cellsDetected", 12.0) + frame = qc_rows._qc_frame(rows).row(0, named=True) + + assert frame["lineWarn"] is None + assert frame["lineAlert"] is None + assert frame["route"] == "categorical" + + +def test_qc_frame_leaves_all_three_null_where_no_line_backs_the_measurement(): + # readsTotal has no line on any route. + rows = [] + qc_rows._add(rows, "sample", "S1", "readsTotal", 1000.0) + frame = qc_rows._qc_frame(rows).row(0, named=True) + + assert frame["lineWarn"] is None + assert frame["lineAlert"] is None + assert frame["route"] is None + + +def test_qc_frame_reads_the_lines_it_was_given_not_the_shipped_default(): + # `_qc_frame` renders whatever `lines` the caller passes, the same dict `_add` used to score + # the row -- an operator override must show up here, not the shipped default. + overridden = dict(DEFAULT_LINES) + overridden["cellBarcodeValidFraction"] = Line(warn=0.9, error=0.6) + + rows = [] + qc_rows._add(rows, "sample", "S1", "cellBarcodeValidFraction", 0.8, lines=overridden) + frame = qc_rows._qc_frame(rows, lines=overridden).row(0, named=True) + + assert frame["lineWarn"] == pytest.approx(0.9) + assert frame["lineAlert"] == pytest.approx(0.6) + # And the status was scored against the override too: 0.8 is below the raised warn line. + assert frame["status"] == "warn" + + +def test_cli_flags_move_a_line_end_to_end(bed): + # 0.91 reads OK against the shipped 0.75 warn line, and alert against a raised 0.95 one. + # This is the CLI surface an operator actually reaches, not the Python function alone. + (bed / "qc.csv").write_text( + "sampleId,readsTotal,readsMatched,matchedFraction,cellsDetected," + "featuresDetected,totalUniqueUmis,medianUmisPerCell,panelAssignedFraction," + "cellBarcodeValidFraction\n" + "S1,20000,18000,0.9,4,2,1200,300,0.82,0.91\n" + ) + r = _run( + bed, *BASE, "--qc-summary", "qc.csv", "--cell-barcode-valid-warn", "0.95", "--cell-barcode-valid-error", "0.92" + ) + assert r.returncode == 0, r.stderr + qc = pl.read_csv(bed / "result_qc.csv", infer_schema_length=0) + + row = qc.filter(pl.col("measurement") == "cellBarcodeValidFraction").row(0, named=True) + assert row["status"] == "alert" + assert row["lineWarn"] == "0.95" + assert row["lineAlert"] == "0.92" + + +def test_qc_frame_rollup_row_carries_no_line_or_route(): + # The rollup measurement has no declaration at all, so all three fields are null there too. + rows = [qc_rows.QcRow("sample", "S1", qc_rows.ROLLUP, None, "", "", None, qc_rows.roll_up([]))] + frame = qc_rows._qc_frame(rows).row(0, named=True) + + assert frame["lineWarn"] is None + assert frame["lineAlert"] is None + assert frame["route"] is None + + +def test_a_value_that_is_not_a_finite_number_is_no_value_and_says_which(monkeypatch): + # A NaN is not the absence the caller's reason describes: that reason names a missing input, + # and here the input arrived. Reporting it would send a reader to fix something that is fine. + rows = [] + qc_rows._add( + rows, + "sample", + "S1", + "readsPerCell", + float("nan"), + reason="no cell list supplied, so depth has no denominator", + ) + entries, coverage = qc_rows.sample_report_rows("S1", rows) + + row = {e["id"]: e for e in entries}["readsPerCell"] + assert row["value"] is None + assert row["reason"] == qc_rows.NOT_A_NUMBER_REASON + assert row["status"] is None + # Counted the way `is_computed` counts it, so the entry and the triple cannot disagree. + assert coverage.judged == 0 + + +def test_the_wide_summary_carries_every_sample_in_the_roster_including_one_with_nothing(tmp_path): + # S2 is declared on the panel and nowhere else: no counts row, no linker row, no cell-list + # entry. `main` still puts it in the sample roster (panel ∪ counts ∪ linker ∪ cell list), so + # the wide table must still carry its row rather than dropping the sample that has nothing. + (tmp_path / "counts.csv").write_text( + "sampleId,cellId,tag,umiCount\nS1,c1,AAAA,500\nS1,c1,CTRL,6\nS1,c2,AAAA,600\nS1,c2,CTRL,6\n" + ) + (tmp_path / "panel.csv").write_text( + "Samples,Name,Sequence,Type\n" + "S1,AgA,AAAA,Target\nS1,Ctrl,CTRL,Control\n" + "S2,AgA,AAAA,Target\nS2,Ctrl,CTRL,Control\n" + ) + (tmp_path / "linker.csv").write_text("sampleId,cellId,setId\nS1,c1,K1\nS1,c2,K1\n") + _run(tmp_path, *BASE) + + summary = pl.read_csv(tmp_path / "result_qc_summary.csv") + assert sorted(summary["sampleId"].to_list()) == ["S1", "S2"] + + s2 = summary.filter(pl.col("sampleId") == "S2") + assert s2.height == 1 + # Nothing computed a value for S2, so its cells read null rather than 0 or "OK" -- a blank + # and a zero are opposite findings. + assert s2["readsTotal"].item() is None + assert s2["status"].item() is None + + +def test_the_wide_summary_carries_every_sample_level_measurement_as_a_column(bed): + _run(bed, *BASE) + summary = pl.read_csv(bed / "result_qc_summary.csv") + declared = {m.id for m in MEASUREMENTS if m.level == "sample"} + missing = declared - set(summary.columns) + assert not missing, f"sample-level measurement(s) with no column: {missing}" + # The rename ban: these two ids are p-column names AND measurement-axis values elsewhere in + # the run, so they must survive under their own name rather than a fresh one. + assert "panelAssignedFraction" in summary.columns + assert "cellBarcodeValidFraction" in summary.columns + + +def test_the_wide_summary_status_is_the_sample_rollup_and_is_not_recomputed(bed): + _run(bed, *BASE) + summary = pl.read_csv(bed / "result_qc_summary.csv") + by_sample = json.loads((bed / "result_qc_by_sample.json").read_text()) + + for sample_id, report in by_sample.items(): + row = summary.filter(pl.col("sampleId") == sample_id) + assert row.height == 1 + got = row["status"].item() + assert got == report["status"], ( + f"wide table status {got!r} for {sample_id!r} disagrees with the sample's own report " + f"{report['status']!r}; the two must read off one rollup" + ) + + +def test_a_present_read_qc_row_with_no_reads_names_the_denominator_not_the_row(bed): + # Two conditions blank aggregateBarcodeFraction: no read-QC row at all, and a row whose + # readsTotal is zero. The second is reachable through parse_gate.py's empty-input path, + # where readsTotal is present and zero. Naming the row as missing would be false -- the + # row is here, and readsTotal on it reads 0. + (bed / "qc.csv").write_text( + "sampleId,readsTotal,readsMatched,matchedFraction,cellsDetected," + "featuresDetected,totalUniqueUmis,medianUmisPerCell,panelAssignedFraction\n" + "S1,0,0,0.0,0,0,0,0,\n" + ) + r = _run(bed, *BASE, "--qc-summary", "qc.csv") + assert r.returncode == 0, r.stderr + qc = pl.read_csv(bed / "result_qc.csv", infer_schema_length=0) + row = qc.filter(pl.col("measurement") == "aggregateBarcodeFraction").row(0, named=True) + assert row["value"] in ("", None) + assert "no denominator" in row["reason"] + assert "reached this sample" not in row["reason"] + + +def test_a_valueless_usable_fraction_carries_its_reason_and_no_detail(bed): + # QcRow's invariant: a detail rides alongside a number, a reason stands in place of one. A row + # with neither a value nor a number to describe must not carry the same string twice. + (bed / "qc.csv").write_text( + "sampleId,readsTotal,readsMatched,matchedFraction,cellsDetected," + "featuresDetected,totalUniqueUmis,medianUmisPerCell,panelAssignedFraction\n" + "S1,20000,18000,0.9,3,2,1200,300,0.82\n" + ) + r = _run(bed, *BASE, "--qc-summary", "qc.csv") + assert r.returncode == 0, r.stderr + qc = pl.read_csv(bed / "result_qc.csv", infer_schema_length=0) + row = qc.filter(pl.col("measurement") == "usableReadFraction").row(0, named=True) + assert row["value"] in ("", None) + assert row["reason"] + assert row["detail"] in ("", None) diff --git a/software/per-cell-metrics/test/test_panel.py b/software/per-cell-metrics/test/test_panel.py new file mode 100644 index 0000000..f16747c --- /dev/null +++ b/software/per-cell-metrics/test/test_panel.py @@ -0,0 +1,473 @@ +import polars as pl +import pytest +from panel import ( + consistent_properties, + default_grouping, + identity_universe, + offered_identities, + panel_read_mismatch, + property_columns, + read_panel, +) + +ROLES = {"barcode": "Sequence", "feature": "Name", "sample": "Samples"} + + +def _csv(tmp_path, header=("Samples", "Name", "Sequence", "Type"), *, rows): + p = tmp_path / "panel.csv" + p.write_text("\n".join([",".join(header)] + [",".join(r) for r in rows]) + "\n") + return str(p) + + +def test_read_panel_one_row_per_tag_sample(tmp_path): + path = _csv( + tmp_path, + rows=[ + ["S1", "AgA", "AAAA", "Off-Target"], + ["S2", "AgB", "AAAA", "Off-Target"], + ["S1", "AgC", "CCCC", "Target"], + ], + ) + panel, dropped = read_panel(path, ROLES) + assert panel.height == 3 + assert set(panel.columns) == {"tag", "sample", "Name", "Type"} + assert dropped == [] + + +def test_read_panel_without_sample_column_uses_star(tmp_path): + path = _csv(tmp_path, header=("Name", "Sequence"), rows=[["AgA", "AAAA"], ["AgB", "CCCC"]]) + panel, _ = read_panel(path, {"barcode": "Sequence", "feature": "Name", "sample": ""}) + assert panel["sample"].unique().to_list() == ["*"] + + +def test_consistent_properties_keeps_agreeing_values(): + panel = pl.DataFrame( + { + "tag": ["AAAA", "AAAA"], + "sample": ["S1", "S2"], + "Name": ["AgA", "AgA"], + "Channel": ["PE", "PE"], + } + ) + props, bad = consistent_properties(panel, ["Name", "Channel"]) + assert props["AAAA"] == {"Name": "AgA", "Channel": "PE"} + assert bad == [] + + +def test_consistent_properties_drops_disagreeing_and_reports_it(): + # Same barcode, different names across two samples' panels — the real shape + # this rule exists for. Names are synthetic: this repository is public. + panel = pl.DataFrame( + { + "tag": ["AAAA", "AAAA"], + "sample": ["S1", "S2"], + "Name": ["AgA", "AgB"], + "Channel": ["APC", "APC"], + } + ) + props, bad = consistent_properties(panel, ["Name", "Channel"]) + assert props["AAAA"] == {"Channel": "APC"} + assert bad == [("AAAA", "Name", ["AgA", "AgB"])] + + +def test_consistent_properties_ignores_blanks(): + panel = pl.DataFrame({"tag": ["AAAA", "AAAA"], "sample": ["S1", "S2"], "Name": ["AgA", ""]}) + props, bad = consistent_properties(panel, ["Name"]) + assert props["AAAA"] == {"Name": "AgA"} + assert bad == [] + + +def test_duplicate_tag_sample_pair_is_fatal(tmp_path): + path = _csv(tmp_path, rows=[["S1", "AgA", "AAAA", "Target"], ["S1", "AgB", "AAAA", "Target"]]) + with pytest.raises(SystemExit) as e: + read_panel(path, ROLES) + assert "AAAA/S1" in str(e.value) + + +def test_blank_barcode_row_is_reported_not_dropped(tmp_path): + path = _csv( + tmp_path, + rows=[ + ["S1", "AgA", "AAAA", "Target"], + ["S1", "AgB", "", "Target"], + ], + ) + panel, dropped = read_panel(path, ROLES) + assert panel.height == 1 + assert dropped == [3] # CSV record ordinal, header counted (not the + # physical line, which differs if a quoted field contains a newline) + + +def test_blank_sample_cell_is_fatal(tmp_path): + path = _csv( + tmp_path, + rows=[ + ["S1", "AgA", "AAAA", "Target"], + ["", "AgB", "CCCC", "Target"], + ], + ) + with pytest.raises(SystemExit) as e: + read_panel(path, ROLES) + assert "line(s) 3." in str(e.value) + + +def test_trailing_blank_line_is_not_a_blank_sample_cell(tmp_path): + # polars materializes a trailing newline as a real all-null row. A stray + # newline at EOF is the commonest shape a panel file arrives in. It must + # not read as an ambiguous sample cell. + p = tmp_path / "panel.csv" + p.write_text("Samples,Name,Sequence,Type\nS1,AgA,AAAA,Target\n\n") + panel, dropped = read_panel(str(p), ROLES) + assert panel.height == 1 + assert dropped == [3] + + +@pytest.mark.parametrize("bad_name", ["tag", "sample"]) +def test_reserved_column_name_is_fatal(tmp_path, bad_name): + # A NON-role column named "tag"/"sample" would be overwritten by the one + # this reader produces, so it is refused rather than silently shadowed. + path = _csv(tmp_path, header=("Samples", "Name", "Sequence", bad_name), rows=[["S1", "AgA", "AAAA", "x"]]) + with pytest.raises(SystemExit) as e: + read_panel(path, ROLES) + assert f"['{bad_name}']" in str(e.value) + + +def test_role_column_may_be_named_tag(tmp_path): + # emit_panel.py in this package documents this very shape and defaults + # --tag-col to "tag". A role column cannot collide: alias() replaces the + # source column rather than duplicating it. + path = _csv(tmp_path, header=("sample", "feature", "tag"), rows=[["S1", "AgA", "AAAA"]]) + panel, dropped = read_panel(path, {"barcode": "tag", "feature": "feature", "sample": "sample"}) + assert panel.height == 1 + assert panel["tag"].to_list() == ["AAAA"] + assert panel["sample"].to_list() == ["S1"] + assert dropped == [] + + +def test_sample_role_named_tag_is_fatal(tmp_path): + # The barcode alias runs first and would overwrite this column, leaving + # "sample" a silent copy of the barcode — per-sample keying gone, and no + # duplicate raised because the pairs stay unique. Refused, not corrected. + path = _csv(tmp_path, header=("tag", "Name", "Sequence"), rows=[["S1", "AgA", "AAAA"]]) + with pytest.raises(SystemExit) as e: + read_panel(path, {"barcode": "Sequence", "feature": "Name", "sample": "tag"}) + assert "['tag']" in str(e.value) + + +def test_two_blank_barcode_rows_are_not_a_duplicate(tmp_path): + # Both rows have tag "", so they would collide as a duplicate (tag, sample) + # pair if the blank-barcode filter ran after the dupe check. + path = _csv( + tmp_path, + rows=[ + ["S1", "AgA", "AAAA", "Target"], + ["S1", "AgB", "", "Target"], + ["S1", "AgC", "", "Target"], + ], + ) + panel, dropped = read_panel(path, ROLES) + assert panel.height == 1 + assert dropped == [3, 4] + + +def test_two_roles_on_one_column_is_fatal(tmp_path): + # Two roles on one column silently makes "sample" a copy of "tag" — + # reachable from the UI today, since the Sample-column dropdown is + # unfiltered. + path = _csv(tmp_path, header=("Samples", "Name", "Sequence"), rows=[["S1", "AgA", "AAAA"]]) + with pytest.raises(SystemExit) as e: + read_panel(path, {"barcode": "Sequence", "feature": "Name", "sample": "Sequence"}) + assert "column 'Sequence'" in str(e.value) + + +def test_missing_barcode_column_is_fatal(tmp_path): + path = _csv(tmp_path, rows=[["S1", "AgA", "AAAA", "Target"]]) + with pytest.raises(SystemExit) as e: + read_panel(path, {"barcode": "NoSuchCol", "feature": "Name", "sample": "Samples"}) + assert "no barcode column 'NoSuchCol'" in str(e.value) + + +def test_missing_sample_column_is_fatal(tmp_path): + path = _csv(tmp_path, rows=[["S1", "AgA", "AAAA", "Target"]]) + with pytest.raises(SystemExit) as e: + read_panel(path, {"barcode": "Sequence", "feature": "Name", "sample": "NoSuchCol"}) + assert "no sample column 'NoSuchCol'" in str(e.value) + + +def test_literal_row_header_is_fatal(tmp_path): + path = _csv(tmp_path, header=("Samples", "Name", "Sequence", "_row"), rows=[["S1", "AgA", "AAAA", "x"]]) + with pytest.raises(SystemExit) as e: + read_panel(path, ROLES) + assert "['_row']" in str(e.value) + + +def test_feature_role_named_tag_is_fatal(tmp_path): + # Pins that the rev-8 barcode-only "tag" exemption stays narrow: a FEATURE + # role named "tag" is not covered by it. + path = _csv(tmp_path, header=("Samples", "Sequence", "tag"), rows=[["S1", "AAAA", "AgA"]]) + with pytest.raises(SystemExit) as e: + read_panel(path, {"barcode": "Sequence", "feature": "tag", "sample": "Samples"}) + assert "['tag']" in str(e.value) + + +def test_property_columns_excludes_tag_and_sample_and_preserves_order(): + # Downstream column layout depends on source order being preserved. + panel = pl.DataFrame({"Type": ["Target"], "tag": ["AAAA"], "Name": ["AgA"], "sample": ["S1"], "Channel": ["APC"]}) + assert property_columns(panel) == ["Type", "Name", "Channel"] + + +def test_barcode_is_stripped(tmp_path): + # tag equality is the join key for every later task. + path = _csv(tmp_path, rows=[["S1", "AgA", " AAAA ", "Target"]]) + panel, _ = read_panel(path, ROLES) + assert panel["tag"].to_list() == ["AAAA"] + + +def test_universe_is_every_identity_not_a_per_set_subset(): + panel = pl.DataFrame({"tag": ["AAAA", "CCCC", "GGGG"], "sample": ["S1", "S2", "S2"], "Name": ["a", "c", "g"]}) + g = {("AAAA", "S1"): "A", ("CCCC", "S2"): "C", ("GGGG", "S2"): "G"} + assert identity_universe(panel, g) == {"A", "C", "G"} + + +def test_reference_tags_never_enter_the_universe(): + panel = pl.DataFrame({"tag": ["AAAA", "CTRL"], "sample": ["S1", "S1"], "Name": ["a", "ctrl"]}) + g = default_grouping(panel, reference_tags={"CTRL"}) + assert g == {("AAAA", "S1"): "AAAA"} + assert identity_universe(panel, g) == {"AAAA"} + + +def test_offered_is_the_union_over_the_sets_samples(): + panel = pl.DataFrame({"tag": ["AAAA", "CCCC", "GGGG"], "sample": ["S1", "S2", "S2"], "Name": ["a", "c", "g"]}) + g = {("AAAA", "S1"): "A", ("CCCC", "S2"): "C", ("GGGG", "S2"): "G"} + assert offered_identities(panel, g, ["S1"]) == {"A"} + assert offered_identities(panel, g, ["S2"]) == {"C", "G"} + assert offered_identities(panel, g, ["S1", "S2"]) == {"A", "C", "G"} + + +def test_offered_needs_only_one_member_tag(): + panel = pl.DataFrame({"tag": ["AAAA", "CCCC"], "sample": ["S1", "S2"], "Name": ["a1", "a2"]}) + g = {("AAAA", "S1"): "A", ("CCCC", "S2"): "A"} + assert offered_identities(panel, g, ["S1"]) == {"A"} + assert offered_identities(panel, g, ["S2"]) == {"A"} + + +def test_star_sample_offers_everything(): + panel = pl.DataFrame({"tag": ["AAAA", "CCCC"], "sample": ["*", "*"], "Name": ["a", "c"]}) + g = {("AAAA", "*"): "A", ("CCCC", "*"): "C"} + assert offered_identities(panel, g, ["anything"]) == {"A", "C"} + + +def test_an_identity_not_offered_is_still_in_the_universe(): + # The universe is what makes a never-asked ROW exist. An earlier revision + # used `offered` as the row set, so a never-offered identity vanished + # instead of reading "never asked". + panel = pl.DataFrame({"tag": ["AAAA", "CCCC"], "sample": ["S1", "S2"], "Name": ["a", "c"]}) + g = {("AAAA", "S1"): "A", ("CCCC", "S2"): "C"} + assert "C" in identity_universe(panel, g) + assert "C" not in offered_identities(panel, g, ["S1"]) + + +def test_a_sample_absent_from_the_panel_is_offered_nothing(): + # A set whose cells came from a sample the panel never mentions was + # offered nothing, so every identity reads "never asked" for it. That is + # the honest answer, not a bug — but it is a big claim from a silent + # lookup, so it is pinned here and reported by the panel/reads check. + panel = pl.DataFrame({"tag": ["AAAA"], "sample": ["S1"], "Name": ["a"]}) + g = {("AAAA", "S1"): "A"} + assert offered_identities(panel, g, ["S9"]) == set() + + +def test_a_panel_of_only_references_has_an_empty_universe(): + panel = pl.DataFrame({"tag": ["CTRL"], "sample": ["S1"], "Name": ["ctrl"]}) + g = default_grouping(panel, reference_tags={"CTRL"}) + assert g == {} + assert identity_universe(panel, g) == set() + + +def test_no_samples_offers_nothing_but_the_star(): + # An empty sample list must not accidentally mean "all samples". + panel = pl.DataFrame({"tag": ["AAAA"], "sample": ["S1"], "Name": ["a"]}) + assert offered_identities(panel, {("AAAA", "S1"): "A"}, []) == set() + star = pl.DataFrame({"tag": ["AAAA"], "sample": ["*"], "Name": ["a"]}) + assert offered_identities(star, {("AAAA", "*"): "A"}, []) == {"A"} + + +def test_a_tag_outside_the_grouping_is_skipped_not_an_error(): + # The reference is the ordinary case of this: it is on the panel and + # deliberately absent from the grouping. + panel = pl.DataFrame({"tag": ["AAAA", "ZZZZ"], "sample": ["S1", "S1"], "Name": ["a", "z"]}) + g = {("AAAA", "S1"): "A"} + assert identity_universe(panel, g) == {"A"} + assert offered_identities(panel, g, ["S1"]) == {"A"} + + +def _counts(rows): + # House shape for a reads/counts table, matching test_verdict.py: one row + # per (sample, cell, tag) with the UMI count. panel_read_mismatch only + # needs sampleId and tag, but the table it is handed always carries all + # four columns, so the fixture does too. + return pl.DataFrame( + rows, orient="row", schema={"sampleId": pl.String, "cellId": pl.String, "tag": pl.String, "umiCount": pl.Int64} + ) + + +def test_declared_tag_never_seen_is_reported_per_sample(): + panel = pl.DataFrame({"tag": ["AAAA", "GGGG"], "sample": ["S1", "S1"], "Name": ["a", "g"]}) + seen = _counts([("S1", "c1", "AAAA", 1)]) + out = panel_read_mismatch(panel, seen) + row = out.filter(pl.col("direction") == "declared-never-seen").row(0, named=True) + assert row["tag"] == "GGGG" and row["sample"] == "S1" + + +def test_undeclared_barcode_is_reported_per_sample(): + panel = pl.DataFrame({"tag": ["AAAA"], "sample": ["S1"], "Name": ["a"]}) + seen = _counts([("S1", "c1", "AAAA", 1), ("S1", "c1", "TTTT", 1)]) + out = panel_read_mismatch(panel, seen) + row = out.filter(pl.col("direction") == "undeclared-in-panel").row(0, named=True) + assert row["tag"] == "TTTT" and row["sample"] == "S1" + + +def test_a_barcode_declared_in_another_sample_does_not_pass_silently(): + # The failure this whole check exists to prevent: AAAA is declared for S3 + # only. It is read in S1, where nothing declares it. A global check would + # let S3's declaration excuse it there too. + panel = pl.DataFrame({"tag": ["CCCC", "AAAA"], "sample": ["S1", "S3"], "Name": ["c", "a"]}) + seen = _counts([("S1", "c1", "CCCC", 1), ("S1", "c1", "AAAA", 1)]) + out = panel_read_mismatch(panel, seen) + undeclared = out.filter(pl.col("direction") == "undeclared-in-panel") + assert ("S1", "AAAA") in list(zip(undeclared["sample"], undeclared["tag"], strict=True)) + + +def test_a_star_panel_is_satisfied_by_reads_in_any_sample(): + panel = pl.DataFrame({"tag": ["AAAA"], "sample": ["*"], "Name": ["a"]}) + seen = _counts([("S1", "c1", "AAAA", 1), ("S2", "c1", "AAAA", 1)]) + assert panel_read_mismatch(panel, seen).height == 0 + + +def test_mismatch_never_raises(): + panel = pl.DataFrame({"tag": ["AAAA"], "sample": ["S1"], "Name": ["a"]}) + seen = _counts([("S9", "c1", "ZZZZ", 1)]) + rows = {(r["sample"], r["tag"], r["direction"]) for r in panel_read_mismatch(panel, seen).to_dicts()} + assert rows == { + ("S1", "AAAA", "declared-never-seen"), + ("S9", "ZZZZ", "undeclared-in-panel"), + } + + +def test_a_sample_with_reads_but_no_panel_rows_reports_every_barcode(): + # The panel does not cover this sample at all. Every barcode it read is + # undeclared — a large claim, so it must be stated rather than inferred + # from an empty result. + panel = pl.DataFrame({"tag": ["AAAA"], "sample": ["S1"], "Name": ["a"]}) + seen = _counts([("S9", "c1", "CCCC", 5)]) + out = panel_read_mismatch(panel, seen) + rows = {(r["sample"], r["tag"], r["direction"]) for r in out.to_dicts()} + assert ("S9", "CCCC", "undeclared-in-panel") in rows + + +def test_a_sample_in_the_panel_with_no_reads_reports_every_tag(): + panel = pl.DataFrame({"tag": ["AAAA"], "sample": ["S1"], "Name": ["a"]}) + seen = _counts([("S2", "c1", "AAAA", 5)]) + out = panel_read_mismatch(panel, seen) + rows = {(r["sample"], r["tag"], r["direction"]) for r in out.to_dicts()} + assert ("S1", "AAAA", "declared-never-seen") in rows + + +def test_full_agreement_reports_nothing(): + # The empty result must mean agreement, not a check that failed to run. + panel = pl.DataFrame({"tag": ["AAAA", "CCCC"], "sample": ["S1", "S1"], "Name": ["a", "c"]}) + seen = _counts([("S1", "c1", "AAAA", 5), ("S1", "c1", "CCCC", 3)]) + assert panel_read_mismatch(panel, seen).height == 0 + + +def test_empty_inputs_do_not_raise(): + panel = pl.DataFrame( + {"tag": [], "sample": [], "Name": []}, schema={"tag": pl.String, "sample": pl.String, "Name": pl.String} + ) + assert panel_read_mismatch(panel, _counts([])).height == 0 + + +def test_both_directions_can_fire_for_one_sample_at_once(): + # A sample can simultaneously declare a tag it never read and read a + # barcode it never declared. Neither direction may mask the other. + panel = pl.DataFrame({"tag": ["AAAA"], "sample": ["S1"], "Name": ["a"]}) + seen = _counts([("S1", "c1", "CCCC", 5)]) + rows = {(r["sample"], r["tag"], r["direction"]) for r in panel_read_mismatch(panel, seen).to_dicts()} + assert ("S1", "AAAA", "declared-never-seen") in rows + assert ("S1", "CCCC", "undeclared-in-panel") in rows + + +def test_a_literal_star_in_a_sample_column_is_fatal(tmp_path): + # "*" is what the reader writes when there is no sample column. Accepting it + # as a sample name lets one row claim every sample, and downstream the whole + # panel-versus-reads check goes blind. + path = _csv(tmp_path, rows=[["S1", "AgA", "AAAA", "T"], ["*", "AgB", "CCCC", "T"]]) + with pytest.raises(SystemExit) as e: + read_panel(path, ROLES) + assert "*" in str(e.value) + + +def test_a_mixed_star_and_named_panel_does_not_go_global(): + # Second line of defence: the reader refuses this frame, but a caller building one directly must + # not get an empty table for a real disagreement. The output is asserted in full, not just + # membership. A mixed frame reports every row as noise, including the star row compared as a + # literal sample name, and a future "clean that spurious row up" edit must not pass here. + panel = pl.DataFrame({"tag": ["AAAA", "CCCC"], "sample": ["*", "S1"], "Name": ["a", "c"]}) + seen = _counts([("S1", "c1", "AAAA", 5)]) + rows = {(r["sample"], r["tag"], r["direction"]) for r in panel_read_mismatch(panel, seen).to_dicts()} + assert rows == { + ("*", "AAAA", "declared-never-seen"), + ("S1", "AAAA", "undeclared-in-panel"), + ("S1", "CCCC", "declared-never-seen"), + } + + +def test_null_keys_on_either_side_do_not_raise(): + # A null tag or a null sample cannot be placed on either side of the comparison, on either input. + # The panel side gets the same guard as the reads side, not a narrower one. + panel = pl.DataFrame( + {"tag": ["AAAA", "CCCC", None], "sample": ["S1", None, "S1"], "Name": ["a", "c", "z"]}, + schema={"tag": pl.String, "sample": pl.String, "Name": pl.String}, + ) + seen = _counts([("S1", "c1", "AAAA", 5), (None, "c2", "CCCC", 3), ("S1", "c3", None, 1)]) + out = panel_read_mismatch(panel, seen) + assert out.height == 0 + assert None not in out["sample"].to_list() + assert None not in out["tag"].to_list() + + +# --- per-sample grouping (panel-file-authority@3.0) ---------------------------------------- + + +def test_default_grouping_is_keyed_by_tag_and_sample(): + # The identity is still the tag under the per-tag grouping, but the map is keyed by the pair so + # every consumer reads one shape whichever grouping is in force. + panel = pl.DataFrame({"tag": ["AAAA", "AAAA"], "sample": ["S1", "S2"], "Name": ["a", "a"]}) + g = default_grouping(panel, reference_tags=set()) + assert g == {("AAAA", "S1"): "AAAA", ("AAAA", "S2"): "AAAA"} + + +def test_identity_universe_is_the_union_across_samples(): + # One barcode carrying a different antigen in each sample yields TWO identities, not a conflict. + # This is the case panel-file-authority@3.0 keys the panel for. + panel = pl.DataFrame({"tag": ["AAAA", "AAAA"], "sample": ["S1", "S2"], "Name": ["a", "b"]}) + g = {("AAAA", "S1"): "A", ("AAAA", "S2"): "B"} + assert identity_universe(panel, g) == {"A", "B"} + + +def test_offered_reads_the_identity_the_sample_itself_declared(): + # A set drawn from S1 was offered what S1's panel said that barcode was, and nothing S2 said. + panel = pl.DataFrame({"tag": ["AAAA", "AAAA"], "sample": ["S1", "S2"], "Name": ["a", "b"]}) + g = {("AAAA", "S1"): "A", ("AAAA", "S2"): "B"} + assert offered_identities(panel, g, ["S1"]) == {"A"} + assert offered_identities(panel, g, ["S2"]) == {"B"} + assert offered_identities(panel, g, ["S1", "S2"]) == {"A", "B"} + + +def test_a_global_panel_row_is_offered_to_every_sample(): + # "*" is what the reader writes where the file declares no sample dimension at all, so it applies + # everywhere rather than naming a sample. + panel = pl.DataFrame({"tag": ["AAAA"], "sample": ["*"], "Name": ["a"]}) + g = {("AAAA", "*"): "A"} + assert offered_identities(panel, g, ["S1"]) == {"A"} + assert offered_identities(panel, g, ["anything"]) == {"A"} diff --git a/software/per-cell-metrics/test/test_parse_gate.py b/software/per-cell-metrics/test/test_parse_gate.py index f8bd067..efe8972 100644 --- a/software/per-cell-metrics/test/test_parse_gate.py +++ b/software/per-cell-metrics/test/test_parse_gate.py @@ -1,8 +1,9 @@ """Behavioral tests for parse_gate.py (Feature Integration software). The gate reads mitool's parse report and decides whether the per-sample mitool chain should continue -(>=1 read matched) or be skipped (0 matched → mitool wrote no parsed.mic), and synthesizes the empty -fallbacks the no-match branch feeds downstream. Stdlib only; run via the CLI like the other slow tests. +(>=1 read matched) or be skipped (0 matched, so mitool wrote no parsed.mic). It also synthesizes the +empty fallbacks the no-match branch feeds downstream. Stdlib only, and run via the CLI like the other +slow tests. """ import json diff --git a/software/per-cell-metrics/test/test_per_cell_metrics.py b/software/per-cell-metrics/test/test_per_cell_metrics.py index 201d6e6..a25f2fe 100644 --- a/software/per-cell-metrics/test/test_per_cell_metrics.py +++ b/software/per-cell-metrics/test/test_per_cell_metrics.py @@ -11,258 +11,20 @@ import subprocess import sys +import polars as pl import pytest from hypothesis import given from hypothesis import strategies as st from per_cell_metrics import ( - CROSS_REACTIVE, _load, combine_barcode_counts, - consensus_category, - offtarget_features, - specificity_score, + per_cell_summary, + with_fraction, ) SRC = pathlib.Path(__file__).parents[1] / "src" / "per_cell_metrics.py" -# --- dominant-category rule (spec A-0012) --- - - -def test_consensus_single_winner_above_threshold(): - # 7 of 10 -> 0.7 >= 0.6 default -> that feature - assert consensus_category({"A": 7, "B": 2, "C": 1}, 0.6) == "A" - - -def test_consensus_winner_exactly_at_threshold(): - assert consensus_category({"A": 6, "B": 4}, 0.6) == "A" - - -def test_consensus_no_winner_is_ambiguous(): - # max share 0.4 < 0.6 -> ambiguous (signal present, none passes) - assert consensus_category({"A": 4, "B": 3, "C": 3}, 0.6) == "ambiguous" - - -def test_consensus_exact_half_split_at_floor_is_ambiguous(): - # 50/50 at the 0.5 floor -> tie -> ambiguous (A-0012: "an exact split at the 0.5 floor") - assert consensus_category({"A": 5, "B": 5}, 0.5) == "ambiguous" - - -def test_consensus_threshold_clamped_to_floor(): - # request 0.4 but floor is 0.5; 0.55 share passes 0.5, unique -> winner - assert consensus_category({"A": 11, "B": 9}, 0.4) == "A" - - -def test_consensus_single_category(): - assert consensus_category({"A": 3}, 0.6) == "A" - - -def test_consensus_no_signal_is_none(): - assert consensus_category({"A": 0, "B": 0}, 0.6) is None - assert consensus_category({}, 0.6) is None - - -# --- negative control is a reference, not a callable antigen (spec A-0014) --- - - -def test_consensus_excludes_control_from_candidates(): - # The control must not win consensus even when it has the most UMIs: AGX 3 / CTRL 5 -> the top - # antigen (AGX) share is 3/8 = 0.375 < 0.6 -> ambiguous, NOT "CTRL". - assert consensus_category({"AGX": 3, "CTRL": 5}, 0.6, control="CTRL") == "ambiguous" - - -def test_consensus_control_stays_in_denominator(): - # Control UMIs remain in the denominator, so control signal suppresses (not inflates) dominance. - # AGX 7 / CTRL 2 -> 7/9 = 0.78 >= 0.6 -> AGX (control did not spuriously push it under threshold). - assert consensus_category({"AGX": 7, "CTRL": 2}, 0.6, control="CTRL") == "AGX" - # Were the control dropped from the denominator, AGX 3 / CTRL 5 would renormalise to 1.0 and wrongly - # win; keeping it in the denominator makes the control-swamped cell correctly ambiguous. - assert consensus_category({"AGX": 3, "CTRL": 5}, 0.6, control="CTRL") == "ambiguous" - - -def test_consensus_control_only_is_ambiguous(): - # A cell whose only signal is the control has no antigen candidate -> ambiguous, never the control. - assert consensus_category({"CTRL": 5}, 0.6, control="CTRL") == "ambiguous" - - -def test_consensus_no_control_arg_is_unchanged(): - # control=None (no negative control set) keeps the original rule: every feature is a candidate. - assert consensus_category({"AGX": 3, "OTHER": 5}, 0.6) == "OTHER" - - -# --- off-target-aware dominant call + cross-reactive label (spec A-0014 Type-aware direction, F2) --- - - -def test_consensus_excludes_offtargets_like_control(): - # An off-target feature is excluded from the winners exactly as the control is: OT swamps the cell, - # the single on-target's share of the total is 3/8 = 0.375 < 0.6 -> ambiguous, never "OT". - assert consensus_category({"AGX": 3, "OT": 5}, 0.6, offtargets=frozenset({"OT"})) == "ambiguous" - - -def test_consensus_offtargets_stay_in_denominator(): - # Off-target UMIs remain in the denominator (suppress, not inflate): AGX 7 / OT 2 -> 7/9 >= 0.6 -> AGX. - assert consensus_category({"AGX": 7, "OT": 2}, 0.6, offtargets=frozenset({"OT"})) == "AGX" - - -def test_consensus_crossreactive_two_ontargets_pass_together(): - # Two on-targets (same target's human+cyno) split ~50/50 with only minor off-target signal: neither - # passes alone, but the on-target set is 90% of the total across 2 features -> cross-reactive, not - # ambiguous. This is the binder F2 rescues from the overloaded "ambiguous" bucket. - assert ( - consensus_category( - {"TgtA_human": 45, "TgtA_cyno": 45, "OT": 10}, - 0.6, - offtargets=frozenset({"OT"}), - label_crossreactive=True, - ) - == CROSS_REACTIVE - ) - - -def test_consensus_crossreactive_needs_label_flag(): - # Without the label flag the same split stays "ambiguous" (backward-compatible when the feature is off). - assert ( - consensus_category({"TgtA_human": 45, "TgtA_cyno": 45, "OT": 10}, 0.6, offtargets=frozenset({"OT"})) - == "ambiguous" - ) - - -def test_consensus_offtarget_swamped_is_ambiguous_not_crossreactive(): - # On-target set collectively below threshold (off-target-dominated) -> ambiguous, never cross-reactive: - # TgtA 20 + TgtB 20 = 40 of 100 (0.4 < 0.6); OT 60 swamps. - assert ( - consensus_category( - {"TgtA": 20, "TgtB": 20, "OT": 60}, - 0.6, - offtargets=frozenset({"OT"}), - label_crossreactive=True, - ) - == "ambiguous" - ) - - -def test_consensus_crossreactive_single_ontarget_still_calls_feature(): - # A single dominant on-target still wins outright (not cross-reactive): AGX 80 / OT 20 -> AGX. - assert ( - consensus_category({"AGX": 80, "OT": 20}, 0.6, offtargets=frozenset({"OT"}), label_crossreactive=True) == "AGX" - ) - - -def test_consensus_only_offtarget_signal_is_ambiguous(): - # A cell whose only signal is off-target has no on-target candidate -> ambiguous. - assert consensus_category({"OT": 5}, 0.6, offtargets=frozenset({"OT"}), label_crossreactive=True) == "ambiguous" - - -def test_offtarget_features_resolves_from_property_column(tmp_path): - # The off-target feature set is resolved from a designated property column + its off-target values. - csv = tmp_path / "tags.csv" - csv.write_text( - "tag,feature,antigen_class\n" - "b1,TgtA,Target\n" - "b2,TgtB,Target\n" - "b3,DecoyX,Decoy\n" - "b4,OTx, Off-Target \n" # whitespace tolerated (stripped) - ) - got = offtarget_features(str(csv), "feature", "antigen_class", frozenset({"Off-Target", "Decoy"})) - assert got == frozenset({"DecoyX", "OTx"}) - - -def test_offtarget_features_bad_column_exits(tmp_path): - csv = tmp_path / "tags.csv" - csv.write_text("tag,feature\nb1,TgtA\n") - with pytest.raises(SystemExit): - offtarget_features(str(csv), "feature", "nope", frozenset({"Off-Target"})) - - -def test_offtarget_features_matching_is_case_sensitive(tmp_path): - # Matching is whitespace-trimmed but CASE-SENSITIVE: selecting "Off-Target" catches only that exact - # value (surrounding whitespace tolerated), NOT "Off-target". Real B043 panels carry both casings in - # one Type column; the user selects every casing they mean (each is offered separately in the - # dropdown). The block never silently broadens a selection to unselected casings. Names stay verbatim. - csv = tmp_path / "tags.csv" - csv.write_text( - "tag,feature,Type\n" - "b1,AgExact,Off-Target\n" # exact match - "b2,AgSpaced, Off-Target \n" # surrounding whitespace -> trimmed, still matches - "b3,AgLower,Off-target\n" # lower 't' — a DIFFERENT value, not selected - "b4,AgOn,Target\n" - ) - # Selecting only "Off-Target": the exact and whitespace-padded rows match; the lowercase one does not. - assert offtarget_features(str(csv), "feature", "Type", frozenset({"Off-Target"})) == frozenset( - {"AgExact", "AgSpaced"} - ) - # Selecting BOTH casings explicitly catches the lowercase feature too — the user opts in. - assert offtarget_features(str(csv), "feature", "Type", frozenset({"Off-Target", "Off-target"})) == frozenset( - {"AgExact", "AgSpaced", "AgLower"} - ) - - -# --- specificity score (spec A-0014, Cell Ranger betaCDF) --- - - -def test_specificity_strong_signal_high_score(): - # many antigen UMIs, no control -> high confidence (the betaCDF formula gives ~98.6 here) - s = specificity_score(antigen_umi=100, control_umi=0) - assert s > 95.0 - - -def test_specificity_no_signal_low_score(): - # no antigen reads, control present -> low confidence - s = specificity_score(antigen_umi=0, control_umi=20) - assert 0.0 <= s < 5.0 - - -def test_specificity_formula_exact(): - # Reference-oracle guard against a constant typo. Weak on its own (mirrors the impl via the same - # scipy call); the bounds + monotonicity PROPERTIES below are the real behavioral guards. - from scipy.stats import beta - - a, c = 7, 3 - expected = (1.0 - float(beta.cdf(0.925, a + 1, c + 3))) * 100.0 - assert specificity_score(a, c) == pytest.approx(expected) - - -# --- properties (invariants that hold for ALL valid inputs) --- - - -@given( - st.dictionaries(st.text(min_size=1), st.integers(min_value=0, max_value=1000), max_size=8), - st.floats(min_value=0.5, max_value=1.0), -) -def test_consensus_result_in_domain(counts, threshold): - # The result is always a key present in counts, "ambiguous", or None -- never an arbitrary string. - r = consensus_category(counts, threshold) - assert r is None or r == "ambiguous" or r in counts - - -@given( - st.dictionaries(st.text(min_size=1), st.integers(min_value=0, max_value=1000), max_size=8), - st.floats(min_value=0.5, max_value=1.0), - st.sets(st.text(min_size=1), max_size=4), -) -def test_consensus_offtarget_result_in_domain(counts, threshold, offtargets): - # With off-targets + the label on, the result is an on-target key, "cross-reactive", "ambiguous", or - # None -- and never an off-target/control key (they can never win). - r = consensus_category(counts, threshold, offtargets=frozenset(offtargets), label_crossreactive=True) - assert r is None or r in ("ambiguous", CROSS_REACTIVE) or (r in counts and r not in offtargets) - - -@given(st.integers(min_value=0, max_value=10_000), st.integers(min_value=0, max_value=10_000)) -def test_specificity_bounded_0_100(antigen, control): - # It is a confidence percentage: always within [0, 100]. - assert 0.0 <= specificity_score(antigen, control) <= 100.0 - - -@given( - st.integers(min_value=0, max_value=500), # control - st.integers(min_value=0, max_value=500), # base antigen - st.integers(min_value=1, max_value=500), # delta -) -def test_specificity_monotonic_in_antigen(control, base, delta): - # More antigen UMIs (same control) never lowers confidence. - assert specificity_score(base + delta, control) >= specificity_score(base, control) - - # --- multi-barcode antigen combine modes: sum (OR) / all (AND) --- # A dual-barcode antigen (BG505 read out by b1 + b2) alongside a single-barcode antigen (OTHER = cx). @@ -280,8 +42,8 @@ def test_combine_all_both_fire_emits_summed(): def test_combine_all_one_missing_omits_feature(): - # Only one BG505 barcode fired -> under AND the antigen is NOT called; the cell has no BG505 entry - # at all (omitted, not zero), so it never competes for dominance or takes a fraction. + # Only one BG505 barcode fired, so under AND the antigen is NOT called. The cell has no BG505 + # entry at all -- omitted, not zero -- so it never takes a fraction of that cell's signal. assert combine_barcode_counts({"b1": 5}, _B2F, _FB, {"BG505": "all"}) == {} @@ -398,34 +160,10 @@ def test_cli_writes_outputs(tagstat_tsv, tags_csv, tmp_path): check=True, cwd=tmp_path, ) - for name in ["result_abundance.csv", "result_fractions.csv", "result_consensus.csv"]: + for name in ["result_abundance.csv", "result_fractions.csv", "result_per_cell_summary.csv"]: assert (tmp_path / name).exists(), f"missing {name}" -@pytest.mark.slow -def test_cli_consensus_golden(tagstat_tsv, tags_csv, tmp_path): - # End-to-end over the committed bed: cell1 dominant on AGX, cell2 ambiguous, cell3 single-feature. - subprocess.run( - [ - sys.executable, - str(SRC), - str(tagstat_tsv), - str(tags_csv), - "--sample-id", - "s1", - "--output-prefix", - str(tmp_path / "result"), - ], - check=True, - cwd=tmp_path, - ) - with open(tmp_path / "result_consensus.csv", newline="") as f: - by_cell = {row["cellId"]: row["consensusFeature"] for row in csv.DictReader(f)} - assert by_cell["cell1"] == "AGX" - assert by_cell["cell2"] == "ambiguous" - assert by_cell["cell3"] == "AGX" - - @pytest.mark.slow def test_cli_abundance_uses_unique_umi(tagstat_tsv, tags_csv, tmp_path): # DP-2: the matrix must use mitool's deduplicated `unique_UMI` (cell1/AGX = 3 distinct UMIs), @@ -455,7 +193,7 @@ def test_cli_abundance_uses_unique_umi(tagstat_tsv, tags_csv, tmp_path): def test_cli_with_renamed_csv_columns(tagstat_tsv, tmp_path): # D4: the CSV's barcode/feature columns can be named anything -- --csv-barcode-col / # --csv-feature-col map them to the join key and output "feature" column. Barcode values - # (AAAA, CCCC) match the committed tagstat_main.tsv bed; mapped to the same AGX/BGX names the + # (AAAA, CCCC) match the committed tagstat_main.tsv bed. Mapped to the same AGX/BGX names the # golden test expects, just via a differently-named CSV. renamed_csv = tmp_path / "renamed_tags.csv" renamed_csv.write_text("barcode,antigen\nAAAA,AGX\nCCCC,BGX\n") @@ -512,27 +250,6 @@ def test_cli_rejects_colliding_feature_col(tagstat_tsv, tmp_path): assert r.returncode != 0 -@pytest.mark.slow -def test_cli_with_control_writes_specificity(tagstat_tsv, tags_csv, tmp_path): - subprocess.run( - [ - sys.executable, - str(SRC), - str(tagstat_tsv), - str(tags_csv), - "--sample-id", - "s1", - "--control", - "CTRL", - "--output-prefix", - str(tmp_path / "result"), - ], - check=True, - cwd=tmp_path, - ) - assert (tmp_path / "result_specificity.csv").exists() - - @pytest.mark.slow @pytest.mark.parametrize( "tagstat_body", @@ -544,9 +261,9 @@ def test_cli_with_control_writes_specificity(tagstat_tsv, tags_csv, tmp_path): ) def test_cli_empty_join_writes_header_only_not_crash(tags_csv, tmp_path, tagstat_body): # Regression: when no (cell, feature) pair survives the tag->feature join -- a wrong read geometry, - # or a sample with no on-panel reads -- the run must still emit all four CSVs header-only, never - # crash. --control exercises the specificity write too. (consensus and specificity are pure-polars - # transforms that carry their schema through the empty case; this guards that they stay header-only.) + # or a sample with no on-panel reads -- the run must still emit both CSVs header-only, never crash. + # abundance and fractions are pure-polars transforms that carry their schema through the empty + # case, and this guards that they stay header-only. tagstat = tmp_path / "tagstat.tsv" tagstat.write_text("CELL\tFEATURE\tcount\ttotalWeight\tunique_UMI\n" + tagstat_body) @@ -558,8 +275,6 @@ def test_cli_empty_join_writes_header_only_not_crash(tags_csv, tmp_path, tagstat str(tags_csv), "--sample-id", "s1", - "--control", - "CTRL", "--output-prefix", str(tmp_path / "result"), ], @@ -569,8 +284,6 @@ def test_cli_empty_join_writes_header_only_not_crash(tags_csv, tmp_path, tagstat for name, header in [ ("result_abundance.csv", ["sampleId", "cellId", "feature", "umiCount"]), ("result_fractions.csv", ["sampleId", "cellId", "feature", "fraction"]), - ("result_consensus.csv", ["sampleId", "cellId", "consensusFeature"]), - ("result_specificity.csv", ["sampleId", "cellId", "feature", "specificityScore"]), ]: p = tmp_path / name assert p.exists(), f"missing {name}" @@ -582,9 +295,9 @@ def test_cli_empty_join_writes_header_only_not_crash(tags_csv, tmp_path, tagstat @pytest.mark.slow def test_cli_per_cell_summary_maxima_match_exported_columns(tagstat_tsv, tags_csv, tmp_path): - # The per-cell summary's maxUmiCount / maxFraction / maxSpecificityScore are a collapse of the - # exported (cell x feature) columns -- they must equal the per-cell max of those exported CSVs, not a - # separately-recomputed value (guards the with_fraction / with_specificity single-compute refactor). + # The per-cell summary's maxUmiCount and maxFraction are a collapse of the exported + # (cell x feature) columns. They must equal the per-cell max of those exported CSVs, not a + # separately-recomputed value. This guards the with_fraction single-compute reuse. subprocess.run( [ sys.executable, @@ -593,8 +306,6 @@ def test_cli_per_cell_summary_maxima_match_exported_columns(tagstat_tsv, tags_cs str(tags_csv), "--sample-id", "s1", - "--control", - "CTRL", "--output-prefix", str(tmp_path / "result"), ], @@ -612,7 +323,6 @@ def _max_by_cell(path, value_col, cast): exp_umi = _max_by_cell(tmp_path / "result_abundance.csv", "umiCount", int) exp_frac = _max_by_cell(tmp_path / "result_fractions.csv", "fraction", float) - exp_spec = _max_by_cell(tmp_path / "result_specificity.csv", "specificityScore", float) with open(tmp_path / "result_per_cell_summary.csv", newline="") as f: summary = {r["cellId"]: r for r in csv.DictReader(f)} @@ -621,265 +331,12 @@ def _max_by_cell(path, value_col, cast): for cell, row in summary.items(): assert int(row["maxUmiCount"]) == exp_umi[cell] assert float(row["maxFraction"]) == pytest.approx(exp_frac[cell]) - assert float(row["maxSpecificityScore"]) == pytest.approx(exp_spec[cell]) - - -@pytest.mark.slow -def test_cli_consensus_matches_pure_rule(tmp_path): - # Oracle: the vectorized CLI consensus must equal the pure consensus_category rule across cases the - # committed golden bed doesn't cover (unique winner, exact tie, sub-threshold spread, single feature). - tags = tmp_path / "tags.csv" - tags.write_text("tag,feature\nAAAA,AGX\nCCCC,BGX\nGGGG,CGX\n") - tagstat = tmp_path / "tagstat.tsv" - tagstat.write_text( - "CELL\tFEATURE\tcount\ttotalWeight\tunique_UMI\n" - "cellW\tAAAA\t8\t8\t8\n" - "cellW\tCCCC\t1\t1\t1\n" - "cellW\tGGGG\t1\t1\t1\n" # AGX 8 / BGX 1 / CGX 1 -> unique winner AGX (0.8 >= 0.6) - "cellX\tAAAA\t5\t5\t5\n" - "cellX\tCCCC\t5\t5\t5\n" # AGX 5 / BGX 5 -> tie, 0.5 < 0.6 -> ambiguous - "cellY\tAAAA\t4\t4\t4\n" - "cellY\tCCCC\t3\t3\t3\n" - "cellY\tGGGG\t3\t3\t3\n" # max share 0.4 < 0.6 -> ambiguous - "cellZ\tAAAA\t6\t6\t6\n" # single feature -> AGX - ) - subprocess.run( - [ - sys.executable, - str(SRC), - str(tagstat), - str(tags), - "--sample-id", - "s1", - "--dominance-threshold", - "0.6", - "--output-prefix", - str(tmp_path / "result"), - ], - check=True, - cwd=tmp_path, - ) - with open(tmp_path / "result_consensus.csv", newline="") as f: - got = {r["cellId"]: r["consensusFeature"] for r in csv.DictReader(f)} - expected = { - "cellW": consensus_category({"AGX": 8, "BGX": 1, "CGX": 1}, 0.6), - "cellX": consensus_category({"AGX": 5, "BGX": 5}, 0.6), - "cellY": consensus_category({"AGX": 4, "BGX": 3, "CGX": 3}, 0.6), - "cellZ": consensus_category({"AGX": 6}, 0.6), - } - assert got == expected # vectorized CLI == the pure rule - # ...and the pure rule is what we think (guards against a vacuous match to a wrong rule) - assert expected == {"cellW": "AGX", "cellX": "ambiguous", "cellY": "ambiguous", "cellZ": "AGX"} - - -@pytest.mark.slow -def test_cli_specificity_matches_pure_score(tagstat_tsv, tags_csv, tmp_path): - # Oracle: the vectorized specificity column must equal the pure specificity_score per (cell, feature) - # vs the cell's control (CTRL) UMIs -- 0 when the cell has no control reads. Guards the array path - # (scipy beta.cdf over whole columns) against the scalar formula, including the fill_null(0) case. - subprocess.run( - [ - sys.executable, - str(SRC), - str(tagstat_tsv), - str(tags_csv), - "--sample-id", - "s1", - "--control", - "CTRL", - "--output-prefix", - str(tmp_path / "result"), - ], - check=True, - cwd=tmp_path, - ) - with open(tmp_path / "result_abundance.csv", newline="") as f: - umi = {(r["cellId"], r["feature"]): int(r["umiCount"]) for r in csv.DictReader(f)} - control_umi = {cell: umi.get((cell, "CTRL"), 0) for (cell, _feat) in umi} - with open(tmp_path / "result_specificity.csv", newline="") as f: - rows = list(csv.DictReader(f)) - assert rows # non-empty: the committed bed has cells and features - assert "CTRL" not in {r["feature"] for r in rows} # control is the reference, not a scored feature - for r in rows: - expected = specificity_score(umi[(r["cellId"], r["feature"])], control_umi[r["cellId"]]) - assert float(r["specificityScore"]) == pytest.approx(float(expected)) - - -@pytest.mark.slow -def test_cli_consensus_excludes_control(tmp_path): - # With --control set, the control is a reference and never a called antigen (spec A-0014): a - # control-dominated cell must be "ambiguous", not the control. Control UMIs stay in the denominator, - # so they suppress dominance rather than being renormalised away. - tags = tmp_path / "tags.csv" - tags.write_text("tag,feature\nAAAA,AGX\nGGGG,CTRL\n") - tagstat = tmp_path / "tagstat.tsv" - tagstat.write_text( - "CELL\tFEATURE\tcount\ttotalWeight\tunique_UMI\n" - "cellP\tAAAA\t3\t3\t3\n" - "cellP\tGGGG\t5\t5\t5\n" # AGX 3 / CTRL 5 -> top antigen 3/8 < 0.6 -> ambiguous (NOT CTRL) - "cellQ\tAAAA\t7\t7\t7\n" - "cellQ\tGGGG\t2\t2\t2\n" # AGX 7 / CTRL 2 -> 7/9 = 0.78 >= 0.6 -> AGX - "cellR\tGGGG\t5\t5\t5\n" # only control signal -> ambiguous - ) - subprocess.run( - [ - sys.executable, - str(SRC), - str(tagstat), - str(tags), - "--sample-id", - "s1", - "--control", - "CTRL", - "--dominance-threshold", - "0.6", - "--output-prefix", - str(tmp_path / "result"), - ], - check=True, - cwd=tmp_path, - ) - with open(tmp_path / "result_consensus.csv", newline="") as f: - got = {r["cellId"]: r["consensusFeature"] for r in csv.DictReader(f)} - assert got == {"cellP": "ambiguous", "cellQ": "AGX", "cellR": "ambiguous"} - # ...and the vectorized CLI agrees with the pure rule (guards against a vacuous match). - assert got == { - "cellP": consensus_category({"AGX": 3, "CTRL": 5}, 0.6, control="CTRL"), - "cellQ": consensus_category({"AGX": 7, "CTRL": 2}, 0.6, control="CTRL"), - "cellR": consensus_category({"CTRL": 5}, 0.6, control="CTRL"), - } - - -@pytest.mark.slow -def test_cli_consensus_offtarget_and_crossreactive(tmp_path): - # End-to-end: with an --offtarget-col/--offtarget-values designation the vectorized consensus must - # match the pure rule -- off-targets excluded from winners, and an on-target-split cell called - # cross-reactive. antigen_class is a per-feature property column of the tag CSV (A-0026 pass-through). - tags = tmp_path / "tags.csv" - tags.write_text("tag,feature,antigen_class\nAAAA,TgtA_human,Target\nCCCC,TgtA_cyno,Target\nGGGG,OTx,Off-Target\n") - tagstat = tmp_path / "tagstat.tsv" - tagstat.write_text( - "CELL\tFEATURE\tcount\ttotalWeight\tunique_UMI\n" - "cellX\tAAAA\t45\t45\t45\n" - "cellX\tCCCC\t45\t45\t45\n" - "cellX\tGGGG\t10\t10\t10\n" # human+cyno split 45/45, OT 10 -> cross-reactive - "cellY\tAAAA\t80\t80\t80\n" - "cellY\tGGGG\t20\t20\t20\n" # single on-target 80/100 -> TgtA_human - "cellZ\tAAAA\t20\t20\t20\n" - "cellZ\tCCCC\t20\t20\t20\n" - "cellZ\tGGGG\t60\t60\t60\n" # OT-swamped (on-target 40/100 < 0.6) -> ambiguous - "cellW\tGGGG\t7\t7\t7\n" # only off-target signal -> ambiguous - ) - subprocess.run( - [ - sys.executable, - str(SRC), - str(tagstat), - str(tags), - "--sample-id", - "s1", - "--dominance-threshold", - "0.6", - "--offtarget-col", - "antigen_class", - "--offtarget-values", - "Off-Target,Decoy", - "--output-prefix", - str(tmp_path / "result"), - ], - check=True, - cwd=tmp_path, - ) - with open(tmp_path / "result_consensus.csv", newline="") as f: - got = {r["cellId"]: r["consensusFeature"] for r in csv.DictReader(f)} - ot = frozenset({"OTx"}) - expected = { - "cellX": consensus_category( - {"TgtA_human": 45, "TgtA_cyno": 45, "OTx": 10}, 0.6, offtargets=ot, label_crossreactive=True - ), - "cellY": consensus_category({"TgtA_human": 80, "OTx": 20}, 0.6, offtargets=ot, label_crossreactive=True), - "cellZ": consensus_category( - {"TgtA_human": 20, "TgtA_cyno": 20, "OTx": 60}, 0.6, offtargets=ot, label_crossreactive=True - ), - "cellW": consensus_category({"OTx": 7}, 0.6, offtargets=ot, label_crossreactive=True), - } - assert got == expected # vectorized CLI == pure rule - # ...and the pure rule is what we intend (guards against a vacuous match). - assert expected == { - "cellX": CROSS_REACTIVE, - "cellY": "TgtA_human", - "cellZ": "ambiguous", - "cellW": "ambiguous", - } - - -@pytest.mark.slow -def test_cli_offtarget_flags_require_each_other(tmp_path): - # --offtarget-col without --offtarget-values (or vice versa) is a user error -> exit non-zero. - tags = tmp_path / "tags.csv" - tags.write_text("tag,feature,antigen_class\nAAAA,TgtA,Target\n") - tagstat = tmp_path / "tagstat.tsv" - tagstat.write_text("CELL\tFEATURE\tcount\ttotalWeight\tunique_UMI\ncX\tAAAA\t3\t3\t3\n") - r = subprocess.run( - [ - sys.executable, - str(SRC), - str(tagstat), - str(tags), - "--sample-id", - "s1", - "--offtarget-col", - "antigen_class", - "--output-prefix", - str(tmp_path / "result"), - ], - cwd=tmp_path, - ) - assert r.returncode != 0 - - -@pytest.mark.slow -def test_cli_control_not_scored_as_feature(tmp_path): - # The control is the specificity reference, not a scored antigen: it must not appear as a feature in - # the specificity output, and a control-heavy cell's maxSpecificityScore must be the real antigen's - # score vs the control, never the control's self-score. - tags = tmp_path / "tags.csv" - tags.write_text("tag,feature\nAAAA,AGX\nGGGG,CTRL\n") - tagstat = tmp_path / "tagstat.tsv" - tagstat.write_text( - "CELL\tFEATURE\tcount\ttotalWeight\tunique_UMI\n" - "cellP\tAAAA\t3\t3\t3\n" - "cellP\tGGGG\t9\t9\t9\n" # control-heavy cell - ) - subprocess.run( - [ - sys.executable, - str(SRC), - str(tagstat), - str(tags), - "--sample-id", - "s1", - "--control", - "CTRL", - "--output-prefix", - str(tmp_path / "result"), - ], - check=True, - cwd=tmp_path, - ) - with open(tmp_path / "result_specificity.csv", newline="") as f: - spec_features = {r["feature"] for r in csv.DictReader(f)} - assert spec_features == {"AGX"} # CTRL is not emitted as a scored feature - with open(tmp_path / "result_per_cell_summary.csv", newline="") as f: - summary = {r["cellId"]: r for r in csv.DictReader(f)} - # AGX 3 UMIs vs CTRL 9 UMIs — the antigen's score, not specificity_score(9, 9) (the control self-score) - assert float(summary["cellP"]["maxSpecificityScore"]) == pytest.approx(specificity_score(3, 9)) @pytest.mark.slow def test_cli_combine_all_gates_dual_barcode_antigen(tmp_path): # End-to-end: a dual-barcode antigen (BG505 = b1 + b2) in "all" (AND) mode is called only in cells - # where BOTH barcodes fired; a single-barcode antigen (OTHER = cx) stays OR. Mirrors the LIBRA-seq + # where BOTH barcodes fired. A single-barcode antigen (OTHER = cx) stays OR. Mirrors the LIBRA-seq # dual-probe design (a cell is BG505-specific only when both probe barcodes are present). tags = tmp_path / "tags.csv" tags.write_text("tag,feature,combine\nb1,BG505,all\nb2,BG505,all\ncx,OTHER,sum\n") @@ -887,7 +344,7 @@ def test_cli_combine_all_gates_dual_barcode_antigen(tmp_path): tagstat.write_text( "CELL\tFEATURE\tcount\ttotalWeight\tunique_UMI\n" "cellBoth\tb1\t6\t6\t6\n" - "cellBoth\tb2\t6\t6\t6\n" # both BG505 barcodes fire -> BG505 called (12), dominant + "cellBoth\tb2\t6\t6\t6\n" # both BG505 barcodes fire -> BG505 called (12) "cellOne\tb1\t9\t9\t9\n" # only b1 fired -> BG505 NOT called "cellOne\tcx\t1\t1\t1\n" # OTHER present ) @@ -912,11 +369,6 @@ def test_cli_combine_all_gates_dual_barcode_antigen(tmp_path): assert umi[("cellBoth", "BG505")] == 12 # AND: both fired -> summed assert ("cellOne", "BG505") not in umi # AND: only one fired -> omitted entirely assert umi[("cellOne", "OTHER")] == 1 - # consensus follows: cellBoth is BG505; cellOne has only OTHER present -> OTHER - with open(tmp_path / "result_consensus.csv", newline="") as f: - cons = {r["cellId"]: r["consensusFeature"] for r in csv.DictReader(f)} - assert cons["cellBoth"] == "BG505" - assert cons["cellOne"] == "OTHER" @pytest.mark.slow @@ -943,3 +395,41 @@ def test_cli_rejects_conflicting_combine_mode(tmp_path): cwd=tmp_path, ) assert r.returncode != 0 + + +def test_a_cell_whose_every_count_is_zero_gets_a_zero_share_not_a_nan(): + # 0/0 is NaN, and the NaN does not stay put. It reaches the exported fractions CSV as a float + # nothing downstream expects, and in per_cell_summary it slips past the "<1%" guard, which requires + # umiCount > 0, into a cast to Int64 that raises and takes the whole CLI down with a raw traceback. + # Real tag-stat cannot emit such a row, but this CLI is driven by hand during verification. + frame = pl.DataFrame( + { + "sampleId": ["S1", "S1"], + "cellId": ["c1", "c1"], + "feature": ["A", "B"], + "umiCount": [0, 0], + } + ) + fractions = with_fraction(frame) + assert fractions["fraction"].to_list() == [0.0, 0.0] + + # And the collapse runs rather than dying. A cell with no signal still gets its row: dropping it + # would make a cell that was measured and read nothing indistinguishable from one never measured. + summary = per_cell_summary(fractions) + assert summary.height == 1 + assert summary["maxUmiCount"].to_list() == [0] + assert summary["maxFraction"].to_list() == [0.0] + + +def test_a_cell_with_signal_is_unaffected_by_the_zero_total_guard(): + # The guard must not change the ordinary case -- a `when/otherwise` around a division is exactly the + # shape that quietly zeroes a whole column if the predicate is wrong. + frame = pl.DataFrame( + { + "sampleId": ["S1", "S1"], + "cellId": ["c1", "c1"], + "feature": ["A", "B"], + "umiCount": [3, 1], + } + ) + assert with_fraction(frame)["fraction"].to_list() == [0.75, 0.25] diff --git a/software/per-cell-metrics/test/test_qc_measures.py b/software/per-cell-metrics/test/test_qc_measures.py new file mode 100644 index 0000000..276a06c --- /dev/null +++ b/software/per-cell-metrics/test/test_qc_measures.py @@ -0,0 +1,1053 @@ +import dataclasses +import re + +import polars as pl +import pytest +from qc_measures import ( + _COMPARISON, + DEFAULT_LINES, + LINE_ROUTES, + MEASUREMENTS, + Coverage, + Line, + Measurement, + Reading, + Status, + aggregate_barcode_fraction, + antigen_count_deciles, + detect_aggregate_barcodes, + measurement_rows, + per_antigen_measures, + reads_per_cell, + roll_up, + sibling_disagreement, + status_for, + usable_read_fraction, +) + +# Every row maps one to one. This is the expected per-id level, built from the +# spec's own table rather than copied from this module, so a level typo on any id +# changes the multiset below. +EXPECTED_LEVEL_BY_ID = { + "readsTotal": "sample", + "cellsDetected": "sample", + "usableReadFraction": "sample", + "panelAssignedFraction": "sample", + "cellBarcodeValidFraction": "sample", + "readsPerCell": "sample", + "antigenCountDistribution": "sample", + "aggregateBarcodeFraction": "sample", + "undeclaredBarcodes": "tag", + "declaredNeverSeen": "tag", + "floorRemoved": "sample", + "uniqueCountsPerCell": "sample", + "highReferenceCells": "sample", + "perAntigen": "tag", + "fittedBackground": "tag", + "scoreDistribution": "run", + "tagDisagreement": "tag", + "siblingDisagreement": "tag", +} + +DEFERRED_IDS: set[str] = set() + + +def test_every_declared_id_is_expected_and_every_expected_id_is_declared(): + assert {m.id for m in MEASUREMENTS} == set(EXPECTED_LEVEL_BY_ID) + + +def test_a_comparison_is_not_a_line(): + # A comparison against siblings yields no boundary: nothing separates OK from alerting, so nothing + # can be computed. A status derived from it would need a multiplier -- a + # median-absolute-deviation cut, an interquartile multiple -- that nobody has published for this + # measurement. The invention would move up a level rather than disappear. + assert LINE_ROUTES == {"inherited", "categorical", "recommended-and-observed"} + assert {m.line for m in MEASUREMENTS if m.line} <= LINE_ROUTES + + +def test_tag_disagreement_reads_unjudged(): + # Such measurements read unjudged and are shown where the comparison is free + # to make: a column beside its siblings. The value still travels. + assert status_for("tagDisagreement", 0.24, DEFAULT_LINES) is None + + +def test_no_measurement_is_refused_by_status_for(): + # The against-the-run route was the only case `status_for` raised on. With it + # gone, every declared measurement gets an answer rather than an exception. None is + # an answer: it says no line stands behind this one. + for m in MEASUREMENTS: + answer = status_for(m.id, 0.5, DEFAULT_LINES) + assert answer is None or isinstance(answer, Status), m.id + + +def test_self_disagreement_is_measured_at_the_tag_and_nowhere_else(): + # The identity-level figure has nothing to compare against, so it cannot separate a faulty reagent + # from a panel full of weak binders. It measures how many clonotypes sit near the line, which is a + # fact about the panel rather than a fault to fix. The tag-level figure is read against the other + # tags in the same panel, under the same cells and the same line. + ids = {m.id for m in MEASUREMENTS} + assert "tagDisagreement" in ids + assert "identityDisagreement" not in ids + + +def test_saturation_and_known_answer_are_not_measured(): + # Both are stated exclusions rather than gaps. Saturation is a number nobody + # can act on for the run already collected, and depth is answered by reads + # per cell against a stated recommendation. The known-answer check needs a + # declaration no surface asks for, so building it means building that first. + ids = {m.id for m in MEASUREMENTS} + assert "sequencingSaturation" not in ids + assert "knownAnswerRecovered" not in ids + assert "readsPerCell" in ids, "the depth question is answered here instead" + + +def test_declared_levels_match_the_spec_as_a_multiset(): + # A multiset comparison, not a per-id comparison: swapping any one + # measurement's level changes how many times that level appears overall, + # so a typo trips this even without knowing which id was mistyped. + declared = sorted(m.level for m in MEASUREMENTS) + expected = sorted(EXPECTED_LEVEL_BY_ID.values()) + assert declared == expected + + +def test_every_measurement_declares_a_known_level(): + # `run` is a grain of one. 320 puts the score spread there because the cutoff is one number + # for the run, so a per-sample figure would answer a question nobody asked. + assert {m.level for m in MEASUREMENTS} <= {"sample", "tag", "identity", "run"} + + +def test_every_measurement_says_what_it_counts(): + assert all(m.counts for m in MEASUREMENTS) + + +def test_measurement_has_no_produced_today_field(): + # produced_today would answer whether the superseded tool produced this + # measurement, which reads backwards from what a reader of this block + # needs: deferred_reason is None already answers whether THIS build does. + field_names = {f.name for f in dataclasses.fields(Measurement)} + assert "produced_today" not in field_names + + +def test_an_unjudged_measurement_says_nothing_about_a_bad_value(): + for m in MEASUREMENTS: + if m.line is None: + assert m.implies is None, m.id + + +BANNED_ADVICE_PHRASES = ( + "should", + "must", + "need to", + "needs to", + "ought", + "advise", + "advice", + "we suggest", + "try ", + "consider ", + "re-run", + "rerun", + "replace", + "avoid", + "ensure", + "make sure", + "flag for", + "recommend that", + "recommend you", +) + +# A sentence opening with one of these reads as an instruction regardless of what follows. Compare +# "Replace the reagent." against "A reagent that produced nothing did not work". So this catches advice +# phrased as an imperative, which the substring list above does not: none of these words are banned +# outright, and several appear as ordinary nouns or adjectives elsewhere in the set, such as "the +# vendor's recommended minimum". +IMPERATIVE_OPENERS = { + "check", + "verify", + "replace", + "remove", + "increase", + "decrease", + "use", + "try", + "consider", + "avoid", + "ensure", + "fix", + "rerun", + "lower", + "raise", + "discard", + "exclude", + "flag", + "recheck", + "investigate", + "review", +} + + +def test_no_measurement_carries_advice(): + for m in MEASUREMENTS: + text = f"{m.counts} {m.implies or ''}" + lowered = text.lower() + # Word boundaries, not bare substrings. "try " matched inside "chemistry does", which + # would have rejected honest prose -- "industry", "poultry", "symmetry" all carry it. + assert not any(re.search(rf"\b{re.escape(phrase.strip())}\b", lowered) for phrase in BANNED_ADVICE_PHRASES), ( + m.id + ) + + for sentence in text.split("."): + first_word = sentence.strip().split(" ", 1)[0].strip(",:;").lower() + assert first_word not in IMPERATIVE_OPENERS, (m.id, sentence) + + +def test_deferred_measurements_are_declared_not_omitted(): + deferred = {m.id for m in MEASUREMENTS if m.deferred_reason} + assert deferred == DEFERRED_IDS + + +def test_deferred_measurement_reasons_are_stated(): + for m in MEASUREMENTS: + if m.id in DEFERRED_IDS: + assert m.deferred_reason, m.id + assert m.implies is None, m.id + + +def test_deferred_measurement_produces_a_row_with_its_reason_and_no_status(): + rows = measurement_rows() + # Never absent: every declared id, deferred or not, has a row. + assert {r["id"] for r in rows} == {m.id for m in MEASUREMENTS} + + by_id = {r["id"]: r for r in rows} + for deferred_id in DEFERRED_IDS: + row = by_id[deferred_id] + # A declaration is not a reading, so no row here carries a status. The reason is what + # tells a reader nothing computed this one. + assert row["status"] is None + assert row["reason"] + + +def test_usable_read_fraction_measurement_is_no_longer_deferred(): + by_id = {m.id: m for m in MEASUREMENTS} + m = by_id["usableReadFraction"] + assert m.deferred_reason is None + assert m.line == "inherited" + assert m.implies + + +def test_usable_read_fraction_line_and_comparison(): + assert DEFAULT_LINES["usableReadFraction"] == Line(warn=0.20, error=0.0) + assert _COMPARISON["usableReadFraction"] == ("at-least", "alerting-at") + + +def test_usable_read_fraction_status_boundaries(): + assert status_for("usableReadFraction", 0.19, DEFAULT_LINES) is Status.WARN + assert status_for("usableReadFraction", 0.21, DEFAULT_LINES) is Status.OK + # Error is `alerting-at` total failure (0.0), not a further step past warn. + assert status_for("usableReadFraction", 0.0, DEFAULT_LINES) is Status.ALERT + + +def _tag_stat(cells: list[str], weights: list[int]) -> pl.DataFrame: + return pl.DataFrame({"CELL": cells, "totalWeight": weights}, schema={"CELL": pl.String, "totalWeight": pl.Int64}) + + +def test_usable_read_fraction_sums_listed_cells_over_total_reads(): + tag_stat = _tag_stat(["c1", "c1", "c2", "c3"], [10, 5, 20, 100]) + fraction, detail = usable_read_fraction(tag_stat, "CELL", {"c1", "c2"}, reads_total=1000) + assert fraction == pytest.approx(0.035) # (10 + 5 + 20) / 1000 + assert detail + + +def test_usable_read_fraction_excludes_a_read_whose_cell_is_outside_the_list(): + tag_stat = _tag_stat(["c1", "c2"], [10, 990]) + fraction, _detail = usable_read_fraction(tag_stat, "CELL", {"c1"}, reads_total=1000) + assert fraction == pytest.approx(0.01) + + +def test_usable_read_fraction_with_no_cell_list_is_no_value_with_a_reason(): + # No list means the called-cell condition cannot be evaluated -- a blank, never a zero. + tag_stat = _tag_stat(["c1"], [10]) + fraction, reason = usable_read_fraction(tag_stat, "CELL", None, reads_total=1000) + assert fraction is None + assert reason + + +def test_usable_read_fraction_needs_a_reads_total_denominator(): + tag_stat = _tag_stat(["c1"], [10]) + fraction, reason = usable_read_fraction(tag_stat, "CELL", {"c1"}, reads_total=None) + assert fraction is None + assert reason + + +def test_a_computed_measurement_carries_no_status(): + rows = measurement_rows() + by_id = {r["id"]: r for r in rows} + for m in MEASUREMENTS: + if m.id not in DEFERRED_IDS: + assert by_id[m.id]["status"] is None, m.id + assert by_id[m.id]["reason"] is None, m.id + + +# --- per_antigen_measures: tag grain ----------------------------------------- + + +def _counts(tag: list[str], umi: list[int], sample: list[str] | None = None) -> pl.DataFrame: + return pl.DataFrame({"sampleId": sample or ["S1"] * len(tag), "tag": tag, "umiCount": umi}) + + +def _states( + tag: list[str], + state: list[str], + sample: list[str] | None = None, + cell: list[str] | None = None, +) -> pl.DataFrame: + # One cell by default. Sibling disagreement is judged within one cell, so tags spread + # across cells never meet. + return pl.DataFrame( + { + "sampleId": sample or ["S1"] * len(tag), + "cellId": cell or ["C1"] * len(tag), + "tag": tag, + "state": state, + } + ) + + +def test_per_antigen_measures_separates_delivered_from_bound(): + # The reagent 330-the-quality-readout names: two counts into every cell. Counted after the + # minimum it delivered nothing, which is the reading that must not come back. Counted before + # it, it delivered into every cell and none of them cleared the line. + counts = _counts(["T1"] * 3, [2, 2, 2]) + states = _states(["T1"] * 3, ["not bound"] * 3) + + out = per_antigen_measures(counts, states, ["T1"], ["S1"]).row(0, named=True) + + assert out["cellsWithCount"] == 3 + assert out["cellsAboveTheLine"] == 0 + # Below the minimum on purpose. The atom calls that the finding rather than an error. + assert out["medianCountPerCell"] == 2.0 + + +def test_per_antigen_measures_medians_over_every_cell_holding_a_count(): + # The regression this guards. Over the bound cells alone the median is 50, which is a healthy + # figure for a half-degraded reagent, computed from the two cells that scraped over. Over every + # cell holding a count it is 22, which moves with the reagent. + counts = _counts(["T1"] * 4, [2, 4, 40, 60]) + states = _states(["T1"] * 4, ["not bound", "not bound", "bound", "bound"]) + + out = per_antigen_measures(counts, states, ["T1"], ["S1"]).row(0, named=True) + + assert out["medianCountPerCell"] == 22.0 + assert out["medianCountPerCell"] != 50.0 + assert out["cellsWithCount"] == 4 + assert out["cellsAboveTheLine"] == 2 + + +def test_per_antigen_measures_keeps_a_row_for_a_tag_the_reads_never_show(): + # A dead reagent reads as a zero under cells-with-count. Grouping the observed frame gave it no + # row at all, and a row that is not there cannot be scanned against its neighbours. + counts = _counts(["T1", "T1"], [10, 20]) + states = _states(["T1", "T1"], ["bound", "bound"]) + + out = per_antigen_measures(counts, states, ["T1", "DEAD"], ["S1"]) + dead = out.filter(pl.col("tag") == "DEAD").row(0, named=True) + + assert out.height == 2 + assert dead["cellsWithCount"] == 0 + assert dead["cellsAboveTheLine"] == 0 + # No counts at all, so no median exists. Distinct from a median that computed to zero. + assert dead["medianCountPerCell"] is None + + +def test_per_antigen_measures_gives_the_reference_tag_no_bound_count(): + # The reference is held out of the verdict read, so no cell was ever asked about it. A zero + # here would read as "asked and never bound", which is the opposite finding. Its median stays, + # because that is the run's ambient floor and the reason it belongs in this table. + counts = _counts(["T1", "T1", "REF", "REF"], [10, 20, 3, 5]) + states = _states(["T1", "T1"], ["bound", "bound"]) + + out = per_antigen_measures(counts, states, ["T1"], ["S1"], reference_tags=["REF"]) + ref = out.filter(pl.col("tag") == "REF").row(0, named=True) + + assert ref["cellsAboveTheLine"] is None + assert ref["cellsWithCount"] == 2 + assert ref["medianCountPerCell"] == 4.0 + + +def test_per_antigen_measures_differs_between_tag_and_identity_grain(): + # T1 and T2 both feed one identity. As tags, T1 shows one weak cell (one + # bound of two). As the combined identity, the same cells collapse to one + # row and T1's weak showing is no longer visible on its own. + tag_counts = _counts(["T1", "T1", "T2", "T2"], [8, 1, 20, 15]) + tag_states = _states(["T1", "T1", "T2", "T2"], ["bound", "not bound", "bound", "bound"]) + # The identity each cell's highest tag reading combined into. + identity_counts = _counts(["ID1"] * 3, [20, 1, 15]) + identity_states = _states(["ID1"] * 3, ["bound", "not bound", "bound"]) + + by_tag = per_antigen_measures(tag_counts, tag_states, ["T1", "T2"], ["S1"]) + by_identity = per_antigen_measures(identity_counts, identity_states, ["ID1"], ["S1"]) + + assert by_tag.height == 2 + assert by_identity.height == 1 + assert dict(zip(by_tag["tag"], by_tag["cellsAboveTheLine"], strict=True)) == {"T1": 1, "T2": 2} + assert by_identity.row(0, named=True)["cellsAboveTheLine"] == 2 + + +# --- reads_per_cell -------------------------------------------------------- + + +def test_reads_per_cell_computes_the_rate(): + assert reads_per_cell(1000, 200) == 5.0 + + +def test_reads_per_cell_empty_cell_list_does_not_divide_by_zero(): + assert reads_per_cell(1000, 0) is None + + +# --- aggregate-barcode detection -------------------------------------------- +# +# Ports `detect_outlier_umis_bcs` from Cell Ranger `main`, +# `lib/python/cellranger/feature/antibody/analysis.py`, called for the ANTIGEN library type +# from `cell_calling_helpers.py::remove_antibody_antigen_aggregates`. + + +def _per_barcode(umi: list[int], read: list[int] | None = None, barcode: list[str] | None = None) -> pl.DataFrame: + n = len(umi) + return pl.DataFrame( + { + "barcode": barcode or [f"b{i}" for i in range(n)], + "umiCount": umi, + "readCount": read or [c * 2 for c in umi], + } + ) + + +def test_detect_aggregate_barcodes_flags_a_clear_outlier(): + # 20 barcodes spread 600..790 plus one at 5000. q1=650, q3=750, threshold=750+(750-650)*3=1050, + # which clears the 1000-UMI floor, so only the barcode at 5000 (the sole one >= 1050) is flagged. + normal = list(range(600, 600 + 20 * 10, 10)) + per_barcode = _per_barcode(normal + [5000], barcode=[f"b{i}" for i in range(20)] + ["AGG"]) + flagged, threshold = detect_aggregate_barcodes(per_barcode.select("barcode", "umiCount")) + assert flagged == frozenset({"AGG"}) + assert threshold == pytest.approx(1050.0) + + +def test_detect_aggregate_barcodes_below_the_floor_flags_nothing(): + # q1=12, q3=20, threshold=20+(20-12)*3=44 -- well under the 1000-UMI floor. The source refuses + # to flag anything under that floor, so nothing is flagged even though 400 clears 44 on its own. + per_barcode = _per_barcode([10, 12, 15, 20, 400]) + flagged, threshold = detect_aggregate_barcodes(per_barcode.select("barcode", "umiCount")) + assert flagged == frozenset() + assert threshold == pytest.approx(44.0) + + +def test_detect_aggregate_barcodes_works_under_a_hundred_barcodes(): + # Same rule, five barcodes. q1=650, q3=750, threshold=1050, clears the floor, and the one + # barcode at 9000 is the only one at or above it. + per_barcode = _per_barcode([600, 650, 700, 750, 9000], barcode=["b0", "b1", "b2", "b3", "AGG"]) + flagged, threshold = detect_aggregate_barcodes(per_barcode.select("barcode", "umiCount")) + assert flagged == frozenset({"AGG"}) + assert threshold == pytest.approx(1050.0) + + +def test_detect_aggregate_barcodes_top_n_narrows_the_quantile_slice(): + # top_n=10 admits only the ten barcodes at 5000 into the quantile calc: q1 == q3 == 5000, so + # the threshold sits at 5000 and all ten meet it. The 90 barcodes at 700 never enter the slice. + per_barcode = _per_barcode([5000] * 10 + [700] * 90) + flagged, threshold = detect_aggregate_barcodes(per_barcode.select("barcode", "umiCount"), top_n=10) + assert threshold == pytest.approx(5000.0) + assert len(flagged) == 10 + + +def test_aggregate_barcode_fraction_top_n_narrows_the_tested_slice(): + per_barcode = _per_barcode(list(range(600, 600 + 150 * 10, 10))) + _, detail_default = aggregate_barcode_fraction(per_barcode, reads_total=1000) + _, detail_50 = aggregate_barcode_fraction(per_barcode, reads_total=1000, top_n=50) + assert "barcodesTested=100" in detail_default + assert "barcodesTested=50" in detail_50 + + +def test_aggregate_barcode_fraction_divides_flagged_reads_by_reads_total(): + normal = list(range(600, 600 + 20 * 10, 10)) + per_barcode = _per_barcode( + normal + [5000], + read=[c * 2 for c in normal] + [10000], + barcode=[f"b{i}" for i in range(20)] + ["AGG"], + ) + fraction, detail = aggregate_barcode_fraction(per_barcode, reads_total=100_000) + assert fraction == pytest.approx(0.1) + assert "AGG" not in detail # the detail states counts, not which barcodes + assert "1" in detail # one barcode flagged, stated somewhere in the detail + + +def test_aggregate_barcode_fraction_below_the_floor_is_zero_with_a_stated_reason(): + per_barcode = _per_barcode([10, 12, 15, 20, 400]) + fraction, detail = aggregate_barcode_fraction(per_barcode, reads_total=1000) + assert fraction == 0.0 + assert detail # a stated reason, not a bare blank and not a silent zero + assert "floor" in detail.lower() or "1000" in detail + + +def test_aggregate_barcode_fraction_needs_a_reads_total_denominator(): + per_barcode = _per_barcode([600, 650, 700, 750, 9000]) + fraction, detail = aggregate_barcode_fraction(per_barcode, reads_total=None) + assert fraction is None + assert detail + fraction, detail = aggregate_barcode_fraction(per_barcode, reads_total=0) + assert fraction is None + assert detail + + +def test_aggregate_barcode_fraction_measurement_is_no_longer_deferred(): + by_id = {m.id: m for m in MEASUREMENTS} + m = by_id["aggregateBarcodeFraction"] + assert m.deferred_reason is None + assert m.line == "inherited" + assert m.implies + + +def test_aggregate_barcode_fraction_line_and_comparison(): + assert DEFAULT_LINES["aggregateBarcodeFraction"] == Line(warn=0.05, error=1.0) + assert _COMPARISON["aggregateBarcodeFraction"] == ("at-most", "alerting-at") + + +def test_aggregate_barcode_fraction_status_boundaries(): + assert status_for("aggregateBarcodeFraction", 0.04, DEFAULT_LINES) is Status.OK + assert status_for("aggregateBarcodeFraction", 0.06, DEFAULT_LINES) is Status.WARN + # Error is `alerting-at`, total failure, not a further step past warn: 0.99 still warns. + assert status_for("aggregateBarcodeFraction", 0.99, DEFAULT_LINES) is Status.WARN + assert status_for("aggregateBarcodeFraction", 1.0, DEFAULT_LINES) is Status.ALERT + + +# --- antigen_count_deciles ----------------------------------------------------- + + +def _cell_counts(totals: dict[str, int], sample_id: str = "S1") -> pl.DataFrame: + return pl.DataFrame( + { + "sampleId": [sample_id] * len(totals), + "cellId": list(totals.keys()), + "umiCount": list(totals.values()), + }, + schema={"sampleId": pl.String, "cellId": pl.String, "umiCount": pl.Int64}, + ) + + +def test_antigen_count_deciles_on_a_known_distribution(): + # 11 cells with totals 0, 10, ..., 100. With linear interpolation over 11 sorted points, the p-th + # percentile lands exactly on index p/10, so every decile equals its own cell's total. That is a + # fixture an off-by-one position error cannot pass unnoticed on. + counts = _cell_counts({f"c{i}": i * 10 for i in range(11)}) + out = antigen_count_deciles(counts) + assert out["decile"].to_list() == list(range(0, 101, 10)) + assert out["value"].to_list() == [float(i * 10) for i in range(11)] + + +def test_antigen_count_deciles_single_cell_sample(): + counts = _cell_counts({"c0": 42}) + out = antigen_count_deciles(counts) + assert out.height == 11 + assert all(v == 42.0 for v in out["value"].to_list()) + + +def test_antigen_count_deciles_empty_sample(): + counts = _cell_counts({}) + out = antigen_count_deciles(counts) + assert out.height == 11 + assert out["decile"].to_list() == list(range(0, 101, 10)) + assert all(v is None for v in out["value"].to_list()) + + +def test_three_statuses_and_no_fourth(): + # Atom 310 refuses a fourth and a fifth: a reader meeting five words in one column reads + # them as a scale. The two cases a fourth word covered are read from the value instead. + assert {s.value for s in Status} == {"OK", "warn", "alert"} + + +def test_a_measurement_with_no_line_carries_no_status_rather_than_a_fourth_word(): + assert status_for("antigenCountDistribution", 12, DEFAULT_LINES) is None + assert status_for("readsPerCell", None, DEFAULT_LINES) is None + + +# --- the route is the single authority ------------------------------------- +# Both directions, so neither table can grow an entry the other does not know +# about. This is what stops `DEFAULT_LINES` becoming a second declaration of +# which measurements carry a line. + + +def test_every_declared_route_is_one_of_the_three(): + assert {m.line for m in MEASUREMENTS if m.line} <= LINE_ROUTES + + +def test_a_numeric_route_has_a_line_and_a_comparison_and_nothing_else_does(): + # The two threshold routes -- inherited, recommended-and-observed -- put an absolute + # number on the measurement, so their id set is exactly DEFAULT_LINES' and _COMPARISON's. + # The categorical route publishes no threshold at all, so its member is absent from both; + # see test_the_categorical_route_now_has_one_member below. + # + # `undeclaredBarcodeShare` is the one entry in those tables with no `Measurement` behind + # it: it backs the undeclared-barcode table's own row (310), not a declared measurement. + numeric_routed = {m.id for m in MEASUREMENTS if m.line in {"inherited", "recommended-and-observed"}} + assert set(DEFAULT_LINES) - {"undeclaredBarcodeShare"} == numeric_routed + assert set(_COMPARISON) - {"undeclaredBarcodeShare"} == numeric_routed + assert "undeclaredBarcodeShare" in DEFAULT_LINES + assert "undeclaredBarcodeShare" in _COMPARISON + + +def test_the_categorical_route_now_has_one_member(): + categorical_routed = {m.id for m in MEASUREMENTS if m.line == "categorical"} + assert categorical_routed == {"cellsDetected"} + # Both directions again, this time for the categorical route: a fact carries no numeric + # threshold, so it must appear in neither table. + assert categorical_routed.isdisjoint(DEFAULT_LINES) + assert categorical_routed.isdisjoint(_COMPARISON) + + +def test_the_undeclared_barcode_line_is_read_direct_not_as_a_complement(): + # 315 publishes this line on the undeclared share itself: warn above 0.50, error at 1.0. + # The barcode table measures that share directly, so the thresholds are not mirrored. + line = DEFAULT_LINES["undeclaredBarcodeShare"] + assert (line.warn, line.error) == (0.50, 1.0) + assert status_for("undeclaredBarcodeShare", 0.60, DEFAULT_LINES) is Status.WARN + assert status_for("undeclaredBarcodeShare", 1.0, DEFAULT_LINES) is Status.ALERT + assert status_for("undeclaredBarcodeShare", 0.40, DEFAULT_LINES) is Status.OK + + +def test_panel_assigned_fraction_carries_no_line_any_more(): + # The line moved to the barcode's own row (310: "its status is the barcode's, and it does + # not become a sample's"). This measurement keeps its value and is never judged. + assert "panelAssignedFraction" not in DEFAULT_LINES + assert "panelAssignedFraction" not in _COMPARISON + assert status_for("panelAssignedFraction", 0.1, DEFAULT_LINES) is None + + +def test_barcode_validity_is_the_line_with_a_gradient_at_both_ends(): + # The one inherited line whose thresholds step the same way twice, and the reason 310 admits + # a third status level at all. The other three put error at total failure. + line = DEFAULT_LINES["cellBarcodeValidFraction"] + assert (line.warn, line.error) == (0.75, 0.50) + assert _COMPARISON["cellBarcodeValidFraction"] == ("at-least", "at-least") + assert status_for("cellBarcodeValidFraction", 0.80, DEFAULT_LINES) is Status.OK + assert status_for("cellBarcodeValidFraction", 0.75, DEFAULT_LINES) is Status.OK + assert status_for("cellBarcodeValidFraction", 0.60, DEFAULT_LINES) is Status.WARN + assert status_for("cellBarcodeValidFraction", 0.40, DEFAULT_LINES) is Status.ALERT + + +def test_no_measurement_declares_rolls_up_false(): + # The undeclared-barcode line no longer sits on a sample measurement at all -- it moved to + # the barcode's own row in the undeclared-barcode table -- so no declared measurement needs + # the rollup exemption `rolls_up=False` exists for. + assert [m.id for m in MEASUREMENTS if not m.rolls_up] == [] + + +def test_a_line_without_an_error_threshold_declares_no_error_comparison(): + # Two places say "this line has no error threshold" and they must agree, or a line would + # carry a direction for a boundary it does not have. + for measurement, line in DEFAULT_LINES.items(): + _, error_comparison = _COMPARISON[measurement] + assert (line.error is None) == (error_comparison is None), measurement + + +def test_an_unjudged_measurement_claims_nothing_about_a_bad_value(): + # Atom 315: where no line can be defended, nothing is said about what a bad + # value would mean, because nothing is known. + for m in MEASUREMENTS: + if m.line is None and m.deferred_reason is None: + assert m.implies is None, m.id + + +def test_reads_total_and_high_reference_cells_are_unjudged(): + by_id = {m.id: m for m in MEASUREMENTS} + assert by_id["readsTotal"].line is None + assert by_id["highReferenceCells"].line is None + assert status_for("readsTotal", 0.5, DEFAULT_LINES) is None + + +def test_the_invented_matched_fraction_line_is_gone(): + assert "matchedFraction" not in DEFAULT_LINES + assert "readsTotal" not in DEFAULT_LINES + + +# --- lines are parameters, and every boundary is pinned -------------------- + + +def test_depth_line_is_a_parameter_not_a_literal(): + assert DEFAULT_LINES["readsPerCell"] == Line(warn=5_000) + assert status_for("readsPerCell", 4_000, {"readsPerCell": Line(5_000)}) is Status.WARN + assert status_for("readsPerCell", 4_000, {"readsPerCell": Line(1_000)}) is Status.OK + + +def test_a_stated_recommendation_warns_and_never_alerts(): + # Atom 315: "One number gives one boundary, so it warns and never alerts." Depth has no + # published error threshold, so no value of it can reach alert -- not even zero. + assert DEFAULT_LINES["readsPerCell"].error is None + assert status_for("readsPerCell", 0, DEFAULT_LINES) is Status.WARN + assert status_for("readsPerCell", -1_000_000, DEFAULT_LINES) is Status.WARN + + +def test_two_thresholds_give_three_levels(): + # The distinction collapsing them lost. Three of the four inherited lines put error at total + # failure, so a low-but-non-zero share warns and only a wholly failed one alerts. + line = DEFAULT_LINES["undeclaredBarcodeShare"] + assert (line.warn, line.error) == (0.5, 1.0) + assert status_for("undeclaredBarcodeShare", 0.4, DEFAULT_LINES) is Status.OK + assert status_for("undeclaredBarcodeShare", 0.6, DEFAULT_LINES) is Status.WARN + assert status_for("undeclaredBarcodeShare", 1.0, DEFAULT_LINES) is Status.ALERT + + +def test_error_is_tested_before_warn(monkeypatch): + # A value past both boundaries reads alert. Tested warn-first it would read warn, and the + # worse finding would be the one that never showed. Barcode validity is the shipped line + # that steps the same way twice -- warn below 0.75, error below 0.50 -- and it is not + # computed here yet, so the case is exercised against a registered stand-in. + monkeypatch.setitem(_COMPARISON, "syntheticTwoStep", ("at-least", "at-least")) + lines = {"syntheticTwoStep": Line(warn=0.75, error=0.50)} + assert status_for("syntheticTwoStep", 0.80, lines) is Status.OK + assert status_for("syntheticTwoStep", 0.60, lines) is Status.WARN + assert status_for("syntheticTwoStep", 0.40, lines) is Status.ALERT + + +def test_at_least_is_acceptable_exactly_at_the_line(): + # Atom 315 alerts *below* the recommendation, so the recommendation itself + # is acceptable. The named value satisfies the condition it names. + assert status_for("readsPerCell", 5_000, DEFAULT_LINES) is Status.OK + assert status_for("readsPerCell", 4_999, DEFAULT_LINES) is Status.WARN + + +def test_at_most_is_acceptable_exactly_at_the_line(): + # `undeclaredBarcodeShare` reads `at-most`: the named value (the warn line itself) + # satisfies the condition it names, and only strictly above it warns. + assert status_for("undeclaredBarcodeShare", 0.5, DEFAULT_LINES) is Status.OK + assert status_for("undeclaredBarcodeShare", 0.51, DEFAULT_LINES) is Status.WARN + + +def test_the_undeclared_barcode_fraction_ships_unjudged(): + # Atom 315 lists it among the four inherited numbers and the field does + # publish 0.50 -- but for one aggregate library fraction, while this + # measurement is per sequence at tag level. A fraction's line does not + # transfer to a list of sequences, and given a count any upper bound + # collapses into "alerting if a single undeclared barcode exists". + by_id = {m.id: m for m in MEASUREMENTS} + assert by_id["undeclaredBarcodes"].line is None + assert by_id["undeclaredBarcodes"].implies is None + assert "undeclaredBarcodes" not in DEFAULT_LINES + assert status_for("undeclaredBarcodes", 0.4, DEFAULT_LINES) is None + + +def test_a_tag_the_reads_never_show_carries_no_status(): + # The verdict took this job: a tag with no reads removes its cells from what could answer, so the + # position reads *never asked* rather than a confident negative. The measurement is a fact on the + # tag's row, kept for the reagent's sake, and warning a reader off an answer that already says so + # would be a second voice on one fact. + assert status_for("declaredNeverSeen", 0, DEFAULT_LINES) is None + assert status_for("declaredNeverSeen", 1, DEFAULT_LINES) is None + assert "declaredNeverSeen" not in DEFAULT_LINES + + +def test_cells_detected_is_the_categorical_route(): + # `cellsDetected` is the categorical route's first member: the alerting condition is a + # fact (no cells at all) rather than a quantity with a published threshold. + by_id = {m.id: m for m in MEASUREMENTS} + m = by_id["cellsDetected"] + assert m.line == "categorical" + assert "categorical" in LINE_ROUTES + + +def test_cells_detected_alerts_at_zero_and_reads_ok_above_it(): + assert status_for("cellsDetected", 0, DEFAULT_LINES) is Status.ALERT + assert status_for("cellsDetected", 1, DEFAULT_LINES) is Status.OK + assert status_for("cellsDetected", 50_000, DEFAULT_LINES) is Status.OK + + +def test_cells_detected_claims_nothing_about_yield(): + # The judgement stays narrow: zero cells means nothing downstream can be computed. Above + # zero, nothing here says the yield was good -- that number is not published anywhere. + by_id = {m.id: m for m in MEASUREMENTS} + m = by_id["cellsDetected"] + assert "yield" not in (m.implies or "").lower() + assert "cellsDetected" not in DEFAULT_LINES + assert "cellsDetected" not in _COMPARISON + + +def test_no_defensible_line_means_unjudged(): + assert status_for("antigenCountDistribution", 12, DEFAULT_LINES) is None + + +def test_a_missing_value_is_not_evaluated(): + assert status_for("readsPerCell", None, DEFAULT_LINES) is None + + +# --- the three-level rollup ----------------------------------------------- + + +def test_rollup_takes_the_worst_status(): + assert roll_up([Reading(Status.OK, 1.0), Reading(Status.ALERT, 1.0)]).status is Status.ALERT + + +def test_a_rollup_returns_a_coverage(): + assert isinstance(roll_up([Reading(Status.OK, 1.0)]), Coverage) + + +def test_coverage_never_enters_the_ordinal(): + r = roll_up([Reading(Status.OK, 1.0), Reading(None, 1.0), Reading(None, None)]) + assert r.status is Status.OK + + +def test_coverage_is_reported_beside_the_status(): + # Two unjudged against one not-evaluated, deliberately unequal. With one of each, a counter that + # reported the other's total would read correctly, and the two questions "was a line defensible" + # and "did anybody look" would be silently interchangeable. + r = roll_up( + [ + Reading(Status.OK, 1.0), + Reading(Status.ALERT, 1.0), + # No status but a number came back: computed, and no line to judge it against. + Reading(None, 1.0), + Reading(None, 1.0), + # No status and no number: nothing computed it. + Reading(None, None), + ] + ) + assert (r.judged, r.unjudged, r.not_evaluated) == (2, 2, 1) + + +def test_the_two_no_status_cases_are_told_apart_by_the_value(): + # Both return None from `status_for`, so the coverage triple is the only thing keeping + # "no line to judge it against" apart from "nothing computed it". + assert roll_up([Reading(None, 0.0)]).unjudged == 1 + assert roll_up([Reading(None, 0.0)]).not_evaluated == 0 + assert roll_up([Reading(None, float("nan"))]).not_evaluated == 1 + + +def test_a_level_with_nothing_judgeable_is_not_evaluated(): + assert roll_up([Reading(None, 1.0), Reading(None, None)]).status is None + + +def test_a_level_with_no_measurements_at_all_is_not_evaluated(): + r = roll_up([]) + assert r.status is None + assert (r.judged, r.unjudged, r.not_evaluated) == (0, 0, 0) + + +def test_only_one_aggregation_rule_remains(): + # A panel status overestimated what could be judged categorically, and a + # capture status became the worst of every sample -- which the samples + # already say. `roll_up` over a sample's own measurements is what is left. + import qc_measures + + assert not hasattr(qc_measures, "roll_up_panel") + assert not hasattr(qc_measures, "roll_up_capture") + + +def test_a_dead_reagent_does_not_mark_every_sample_alerting(): + # A per-tag failure is usually a property of the reagent across the whole run + # rather than of any one sample. Fed into a sample status, one dead reagent in + # a panel of twenty tags would mark every sample alerting, which makes that + # status noise within one run. The sample rolls up its OWN measurements only. + samples = [roll_up([Reading(Status.OK, 1.0)]).status for _ in range(3)] + assert samples == [Status.OK] * 3 + + +# --- corrupt numbers must never read green ------------------------------------------- +# +# Every `<` and `>` comparison against NaN is False, so an unguarded NaN value falls through to +# `bad = False` and the measurement reads ACCEPTABLE. For QC code, corrupt-input-reads-green is the +# worst available failure mode: it is the one state a reader will not investigate. + + +def test_a_nan_value_is_not_evaluated_rather_than_acceptable(): + assert status_for("readsPerCell", float("nan"), DEFAULT_LINES) is None + + +def test_infinite_values_are_not_evaluated_rather_than_judged(): + # +inf would have read ACCEPTABLE against an at-least line, which is the green reading again. -inf + # happens to alert, so only one direction was dangerous. But neither is a measurement, and one rule + # for "not a finite number" is easier to defend than a rule that depends on the sign. + assert status_for("readsPerCell", float("inf"), DEFAULT_LINES) is None + assert status_for("readsPerCell", float("-inf"), DEFAULT_LINES) is None + + +def test_seen_in_counts_distinct_samples_not_cells(): + # Two cells in one sample must read as one sample, not two. + counts = _counts(["T1", "T1"], [5, 3], ["S1", "S1"]) + states = _states(["T1"], ["bound"]) + + row = per_antigen_measures(counts, states, ["T1"], ["S1"]).row(0, named=True) + + assert row["samplesSeenIn"] == 1 + + +def test_seen_in_is_zero_for_a_dead_tag(): + # A declared tag with no counts anywhere. Zero, never null: a blank and a zero are opposite + # findings on this table. + counts = _counts(["T1"], [5]) + states = _states(["T1"], ["bound"]) + + out = per_antigen_measures(counts, states, ["T1", "DEAD"], ["S1"]) + dead = out.filter(pl.col("tag") == "DEAD").row(0, named=True) + + assert dead["samplesSeenIn"] == 0 + assert dead["cellsWithCount"] == 0 + + +def test_seen_in_reports_the_panel_size_beside_it(): + # samplesInPanel is the denominator beside samplesSeenIn. + counts = _counts(["T1", "T1"], [5, 5], ["S1", "S2"]) + states = _states(["T1"], ["bound"]) + + row = per_antigen_measures(counts, states, ["T1"], ["S1", "S2"]).row(0, named=True) + + assert row["samplesSeenIn"] == 2 + assert row["samplesInPanel"] == 2 + + +def test_a_reference_tag_reports_its_own_seen_in(): + # Held out of the verdict read, so cellsAboveTheLine is None. The reagent still delivered it, + # and seen-in says so. + counts = _counts(["REF", "REF"], [9, 9], ["S1", "S2"]) + states = _states(["T1"], ["bound"]) + + out = per_antigen_measures(counts, states, ["T1"], ["S1", "S2"], reference_tags=["REF"]) + ref = out.filter(pl.col("tag") == "REF").row(0, named=True) + + assert ref["samplesSeenIn"] == 2 + assert ref["cellsAboveTheLine"] is None + + +def test_the_denominator_is_the_declared_roster_not_the_observed_samples(): + # S3 is in the panel and contributed no rows. samplesInPanel counts it; + # samplesSeenIn does not. + counts = _counts(["T1", "T1"], [5, 5], ["S1", "S2"]) + states = _states(["T1"], ["bound"]) + + out = per_antigen_measures(counts, states, ["T1"], ["S1", "S2", "S3"]) + row = out.filter(pl.col("tag") == "T1").row(0, named=True) + + assert row["samplesSeenIn"] == 2 + assert row["samplesInPanel"] == 3 + + +# --- seen-in names its samples, so a staged panel reads apart from a dead reagent ----------- +# +# samplesSeenIn/samplesInPanel stay as counts (ADD, do not rename). These name the same two +# groups so a reader tells "not declared here" from "declared, zero reads" without a second +# table. All three tests share a four-sample roster so the three readings sit side by side. + +FOUR_SAMPLES = ["S1", "S2", "S3", "S4"] + + +def test_declared_and_seen_names_every_sample_for_a_working_tag(): + # Declared on every sample's panel, seen on all of them: the two named lists match the + # full roster. + counts = _counts(["T1"] * 4, [5, 5, 5, 5], FOUR_SAMPLES) + states = _states(["T1"], ["bound"]) + + row = per_antigen_measures(counts, states, ["T1"], FOUR_SAMPLES).row(0, named=True) + + assert row["samplesInPanelNames"] == FOUR_SAMPLES + assert row["samplesSeenInNames"] == FOUR_SAMPLES + + +def test_a_dead_tag_is_declared_everywhere_and_seen_nowhere(): + # Declared on the full roster, seen on none. samplesSeenInNames is an empty list -- a + # zero, not an absence -- while samplesInPanelNames still names the full roster. + counts = _counts(["T1"], [5], ["S1"]) + states = _states(["T1"], ["bound"]) + + out = per_antigen_measures(counts, states, ["T1", "DEAD"], FOUR_SAMPLES) + dead = out.filter(pl.col("tag") == "DEAD").row(0, named=True) + + assert dead["samplesInPanelNames"] == FOUR_SAMPLES + assert dead["samplesSeenInNames"] == [] + + +def test_a_staged_tag_reads_apart_from_a_dead_one(): + # Declared on only two of the four-sample roster and seen on both of those -- the case + # this exists to distinguish. samplesInPanelNames names the narrower roster the caller + # declared it against, not the full one, and samplesSeenInNames matches it exactly: + # neither list is empty, which is what keeps this from reading like the dead tag above + # (declared on the full roster, seen on none). + counts = _counts(["T1", "T1"], [5, 5], ["S1", "S2"]) + states = _states(["T1"], ["bound"]) + + row = per_antigen_measures(counts, states, ["T1"], ["S1", "S2"]).row(0, named=True) + + assert row["samplesInPanelNames"] == ["S1", "S2"] + assert row["samplesSeenInNames"] == ["S1", "S2"] + assert row["samplesSeenInNames"] != [] + + +# --- sibling disagreement: a tag against the other tags of its identity ------ + + +def test_a_tag_with_no_sibling_has_no_sibling_disagreement(): + # An identity carrying one tag has no sibling. None, not 0. + states = _states(["AAAA"], ["bound"]) + out = sibling_disagreement(states, {"IDENT": ["AAAA"]}) + assert out["AAAA"] is None + + +def test_a_tag_agreeing_with_its_siblings_reports_zero(): + states = _states(["AAAA", "BBBB", "CCCC"], ["bound", "bound", "bound"]) + out = sibling_disagreement(states, {"IDENT": ["AAAA", "BBBB", "CCCC"]}) + assert out["AAAA"] == 0.0 + + +def test_a_tag_contradicting_its_siblings_reports_one(): + # Two siblings say bound, this one says not bound, in the same cell. + states = _states(["AAAA", "BBBB", "CCCC"], ["not bound", "bound", "bound"]) + out = sibling_disagreement(states, {"IDENT": ["AAAA", "BBBB", "CCCC"]}) + assert out["AAAA"] == 1.0 + + +def test_a_tag_whose_siblings_tie_is_not_judged_in_that_cell(): + # One cell, three tags on the identity, and each tag is judged against the other two. + # AAAA's siblings both say bound and convict it. BBBB's siblings are AAAA (not bound) + # and CCCC (bound), and CCCC's are AAAA (not bound) and BBBB (bound) -- both tie, so + # the only cell there is judges neither. A tie is a two-sibling case and needs three + # tags: on a two-tag identity the single sibling is the majority. + states = _states(["AAAA", "BBBB", "CCCC"], ["not bound", "bound", "bound"]) + out = sibling_disagreement(states, {"IDENT": ["AAAA", "BBBB", "CCCC"]}) + assert out["AAAA"] == 1.0 + assert out["BBBB"] is None + assert out["CCCC"] is None + + +def test_a_lone_sibling_is_its_own_majority(): + # Two tags, so neither has a tie available. Each is judged against the other. + states = _states(["AAAA", "BBBB"], ["not bound", "bound"]) + out = sibling_disagreement(states, {"IDENT": ["AAAA", "BBBB"]}) + assert out["AAAA"] == 1.0 + assert out["BBBB"] == 1.0 + + +def test_disagreement_is_judged_within_one_cell(): + # The same two readings in two cells. Neither tag ever meets a sibling, so neither + # has a comparison to report. + states = _states( + ["AAAA", "BBBB"], + ["not bound", "bound"], + cell=["C1", "C2"], + ) + out = sibling_disagreement(states, {"IDENT": ["AAAA", "BBBB"]}) + assert out["AAAA"] is None + assert out["BBBB"] is None + + +def test_the_rate_is_the_share_of_cells_that_contradict(): + states = _states( + ["AAAA", "BBBB", "CCCC"] * 2, + ["not bound", "bound", "bound", "bound", "bound", "bound"], + cell=["C1"] * 3 + ["C2"] * 3, + ) + out = sibling_disagreement(states, {"IDENT": ["AAAA", "BBBB", "CCCC"]}) + assert out["AAAA"] == 0.5 + + +def test_a_tag_holding_no_row_has_no_rate(): + # CCCC is on the identity and holds no cell. Its siblings agree everywhere, so the + # absent rate is about CCCC being missing rather than about them failing to agree. + states = _states(["AAAA", "BBBB"], ["bound", "bound"]) + out = sibling_disagreement(states, {"IDENT": ["AAAA", "BBBB", "CCCC"]}) + assert out["CCCC"] is None + assert out["AAAA"] == 0.0 diff --git a/software/per-cell-metrics/test/test_qc_report.py b/software/per-cell-metrics/test/test_qc_report.py index a342588..5c4bc41 100644 --- a/software/per-cell-metrics/test/test_qc_report.py +++ b/software/per-cell-metrics/test/test_qc_report.py @@ -16,7 +16,7 @@ def _refine_report(steps): return {"inputRecords": 0, "outputRecords": 0, "steps": steps, "filterReport": None} -def _run(tmp_path, tagstat, parse_report, refine_report=None): +def _run(tmp_path, tagstat, parse_report, refine_report=None, extra=()): args = [ sys.executable, str(SRC), @@ -36,6 +36,7 @@ def _run(tmp_path, tagstat, parse_report, refine_report=None): ] if refine_report is not None: args += ["--refine-report", str(refine_report)] + args += list(extra) subprocess.run(args, check=True, cwd=tmp_path) with open(tmp_path / "result_qc.csv", newline="") as f: return next(csv.DictReader(f)) @@ -43,7 +44,7 @@ def _run(tmp_path, tagstat, parse_report, refine_report=None): def test_qc_metrics_from_parse_report_and_tagstat(tmp_path): tagstat = tmp_path / "tagstat.tsv" - # two cells; cell1 has 2 features (3+1 UMIs), cell2 has 1 feature (4 UMIs) + # two cells. Cell1 has 2 features (3+1 UMIs), cell2 has 1 feature (4 UMIs) tagstat.write_text( "CELL\tFEATURE\tcount\ttotalWeight\tunique_UMI\n" "cell1\tAAAA\t7\t7\t3\n" @@ -117,6 +118,75 @@ def test_panel_assigned_fraction_from_feature_step(tmp_path, input_count, output assert float(row["panelAssignedFraction"]) == pytest.approx(expected) +def _tagstat_lines(umi_by_cell: dict[str, int], read_by_cell: dict[str, int]) -> str: + """One FEATURE row per cell, so group-by-CELL sums equal the given per-cell totals.""" + lines = ["CELL\tFEATURE\tcount\ttotalWeight\tunique_UMI\n"] + for cell, umi in umi_by_cell.items(): + lines.append(f"{cell}\tAAAA\t{read_by_cell[cell]}\t{read_by_cell[cell]}\t{umi}\n") + return "".join(lines) + + +def test_aggregate_barcode_fraction_flags_a_clear_outlier(tmp_path): + # 20 cells spread 600..790 antigen UMIs, one at 5000. q1=650, q3=750, threshold=1050 -- + # above the 1000-UMI floor -- so only the 5000 barcode is flagged. Its reads are 10000 of + # a 100000 total, so the fraction is 0.1. + normal = {f"c{i}": 600 + i * 10 for i in range(20)} + umi = {**normal, "agg": 5000} + reads = {c: v * 2 for c, v in umi.items()} + reads["agg"] = 10_000 + tagstat = tmp_path / "tagstat.tsv" + tagstat.write_text(_tagstat_lines(umi, reads)) + parse_report = tmp_path / "parse.json" + parse_report.write_text(json.dumps({"parseReport": {"total": 100_000, "matched": sum(reads.values())}})) + + row = _run(tmp_path, tagstat, parse_report) + assert float(row["aggregateBarcodeFraction"]) == pytest.approx(0.1) + assert int(row["aggregateBarcodesFlagged"]) == 1 + assert float(row["aggregateBarcodeThreshold"]) == pytest.approx(1050.0) + + +def test_aggregate_barcode_fraction_below_the_floor_is_zero(tmp_path): + # q1=12, q3=20, threshold=44 -- under the 1000-UMI floor, so nothing is flagged even + # though one barcode (400) clears 44 on its own. + umi = {"c0": 10, "c1": 12, "c2": 15, "c3": 20, "c4": 400} + reads = {c: v for c, v in umi.items()} + tagstat = tmp_path / "tagstat.tsv" + tagstat.write_text(_tagstat_lines(umi, reads)) + parse_report = tmp_path / "parse.json" + parse_report.write_text(json.dumps({"parseReport": {"total": 1000, "matched": sum(reads.values())}})) + + row = _run(tmp_path, tagstat, parse_report) + assert float(row["aggregateBarcodeFraction"]) == 0.0 + assert int(row["aggregateBarcodesFlagged"]) == 0 + assert float(row["aggregateBarcodeThreshold"]) == pytest.approx(44.0) + + +def test_aggregate_barcode_knobs_are_cli_flags(tmp_path): + # Same bed as the below-the-floor case, but a lowered --aggregate-min-umi-threshold lets + # the 44 threshold clear the floor, so the 400-UMI barcode is now flagged. + umi = {"c0": 10, "c1": 12, "c2": 15, "c3": 20, "c4": 400} + reads = {c: v for c, v in umi.items()} + tagstat = tmp_path / "tagstat.tsv" + tagstat.write_text(_tagstat_lines(umi, reads)) + parse_report = tmp_path / "parse.json" + parse_report.write_text(json.dumps({"parseReport": {"total": 1000, "matched": sum(reads.values())}})) + + row = _run(tmp_path, tagstat, parse_report, extra=["--aggregate-min-umi-threshold", "10"]) + assert float(row["aggregateBarcodeThreshold"]) == pytest.approx(44.0) + assert int(row["aggregateBarcodesFlagged"]) == 1 + + +def test_aggregate_barcode_fraction_survives_header_only_tagstat(tmp_path): + tagstat = tmp_path / "tagstat.tsv" + tagstat.write_text("CELL\tFEATURE\tcount\ttotalWeight\tunique_UMI\n") + parse_report = tmp_path / "parse.json" + parse_report.write_text(json.dumps({"parseReport": {"total": 1000, "matched": 0}})) + row = _run(tmp_path, tagstat, parse_report) + assert float(row["aggregateBarcodeFraction"]) == 0.0 + assert int(row["aggregateBarcodesFlagged"]) == 0 + assert row["aggregateBarcodeThreshold"] == "" + + def test_panel_assigned_fraction_blank_without_feature_step(tmp_path): # A refine report with no FEATURE step (e.g. CELL/UMI only) leaves the fraction blank rather than # reporting a wrong number. diff --git a/software/per-cell-metrics/test/test_tag_distribution.py b/software/per-cell-metrics/test/test_tag_distribution.py new file mode 100644 index 0000000..0fe5132 --- /dev/null +++ b/software/per-cell-metrics/test/test_tag_distribution.py @@ -0,0 +1,326 @@ +"""Rung 3's fit: a two-component negative binomial per (sample, tag), scored per cell. + +`what-plays-the-baseline` fixes the rule: on the raw counts, drop the counts above the 99th percentile, +fit a two-component negative binomial mixture, label the higher-median component the signal one, and +give each cell the probability that its count belongs to it. A cell reads bound at 0.9 or above. + +Every bed here is generated from a seeded generator rather than written out by hand, because the thing +under test is a distribution and a hand-written handful of counts has none. The seed is fixed, so the +beds are the same bytes on every run and on every machine. + +The population sizes are the ones a real run carries: the manual bed's presets are 2000 cells per donor, +and a binder fraction of a few percent is what this method exists to find. + +**One thing this file pins deliberately, and it looks like a bug.** A tag nothing bound still fits, and +the fit still calls some of its cells bound. `what-plays-the-baseline` says so outright -- the method +assumes two components exist, "the fit will split that single population anyway and call its upper slice +signal, inventing binders on exactly the tag that had none", and no published test replaces the eye. An +earlier implementation rejected such a tag with a separation test of its own invention. That test is +exactly what the spec refuses, so it is gone, and the run shows the fit instead of judging it. +""" + +import numpy as np +import polars as pl +import pytest +from panel import ANY_SAMPLE +from tag_distribution import ( + DEFAULT_DISTRIBUTION_MIN_CELLS, + NO_SEPARATION, + TOO_FEW_CELLS, + fit_tag_probabilities, + fit_tag_probabilities_by_pair, +) +from verdict import DISTRIBUTION_BOUND_PROBABILITY + +SEED = 7 + + +def _mixture(n_background, background_rate, n_binders, binder_rate, seed=SEED): + """A background population and a binder population, in that order.""" + rng = np.random.default_rng(seed) + return np.concatenate([rng.poisson(background_rate, n_background), rng.poisson(binder_rate, n_binders)]) + + +def _bound(probabilities): + return probabilities >= DISTRIBUTION_BOUND_PROBABILITY + + +def _panel(rows): + return pl.DataFrame(rows, orient="row", schema={"tag": pl.String, "sample": pl.String}) + + +def _counts_frame(rows): + return pl.DataFrame( + rows, + orient="row", + schema={"sampleId": pl.String, "cellId": pl.String, "tag": pl.String, "umiCount": pl.Int64}, + ) + + +def _bed(n_cells=2000, n_binders=60, sample="S1", tag="AAAA", seed=SEED): + """A sample of `n_cells` cells where `n_binders` of them bind `tag`. + + Returns the counts frame and the cell universe. Cells that read nothing are absent from the counts + frame and present in the universe, which is the shape the block actually receives. + """ + rng = np.random.default_rng(seed) + values = np.concatenate([rng.poisson(2, n_cells - n_binders), rng.poisson(300, n_binders)]) + cells = [(sample, f"c{i}") for i in range(n_cells)] + rows = [(sample, f"c{i}", tag, int(v)) for i, v in enumerate(values) if v > 0] + return _counts_frame(rows), cells + + +# --- the cell condition ------------------------------------------------------------------------- + + +def test_the_cell_condition_is_the_shipped_one(): + # 300 is the study's own bootstrapping figure, and it gates rather than tunes. + assert DEFAULT_DISTRIBUTION_MIN_CELLS == 300 + + +def test_a_sample_below_the_cell_condition_gets_no_fit(): + fit = fit_tag_probabilities(_mixture(279, 2, 20, 300)) + assert fit.probabilities is None + assert fit.reason == TOO_FEW_CELLS + + +def test_the_cell_condition_counts_cells_not_readings(): + # A mostly-silent tag over 400 cells clears the condition, though far fewer than 300 of them + # carry a reading. The population the fit is taken over is the sample's cells. + counts = _mixture(390, 0.2, 10, 200) + assert int((counts > 0).sum()) < DEFAULT_DISTRIBUTION_MIN_CELLS + assert fit_tag_probabilities(counts).reason is None + + +# --- what the fit calls ------------------------------------------------------------------------- + + +def test_a_few_percent_binder_population_is_called(): + fit = fit_tag_probabilities(_mixture(1940, 2, 60, 300)) + assert fit.reason is None + called = _bound(fit.probabilities) + assert called[1940:].all(), "every planted binder must be called" + assert not called[:1940].any(), "no background cell may be called" + + +def test_a_one_percent_binder_population_is_still_called(): + fit = fit_tag_probabilities(_mixture(1980, 2, 20, 300)) + assert fit.reason is None + assert _bound(fit.probabilities)[1980:].all() + + +def test_a_mostly_silent_background_is_handled(): + # A background that read almost nothing is the common shape, and the zeros are most of it. + fit = fit_tag_probabilities(_mixture(1900, 0.2, 100, 200)) + assert fit.reason is None + assert _bound(fit.probabilities)[1900:].all() + + +def test_an_even_split_is_handled(): + # Two populations of the same size. The published method's own paper rejects a Gaussian + # mixture because it degrades on UNEQUAL populations, so the equal case must also hold. + fit = fit_tag_probabilities(_mixture(1000, 2, 1000, 300)) + assert fit.reason is None + assert _bound(fit.probabilities)[1000:].all() + + +def test_the_probability_is_a_probability(): + fit = fit_tag_probabilities(_mixture(1940, 2, 60, 300)) + assert fit.probabilities.min() >= 0.0 + assert fit.probabilities.max() <= 1.0 + + +def test_one_probability_per_cell(): + counts = _mixture(1940, 2, 60, 300) + assert fit_tag_probabilities(counts).probabilities.size == counts.size + + +# --- what does not fit -------------------------------------------------------------------------- + + +def test_a_tag_every_cell_read_identically_does_not_fit(): + # One value is one population by construction. That is the answer rather than an error. + fit = fit_tag_probabilities(np.full(2000, 4, dtype=np.int64)) + assert fit.probabilities is None + assert fit.reason == NO_SEPARATION + + +def test_a_tag_no_cell_read_at_all_does_not_fit(): + fit = fit_tag_probabilities(np.zeros(2000, dtype=np.int64)) + assert fit.probabilities is None + assert fit.reason == NO_SEPARATION + + +def test_a_tag_nothing_bound_still_fits_and_calls_some_cells_bound(): + # DELIBERATE, and the spec says so: the method assumes two components exist, so it splits a + # single population and calls its upper slice signal. Rejecting this would be a separation test + # of our own invention, which `what-plays-the-baseline` refuses -- the run shows the fit instead. + # + # The background here is OVERDISPERSED rather than Poisson, which is what a real one is: the + # invented binders are the long tail of a single skewed population, so a Poisson bed does not + # produce them and would let this pass for the wrong reason. + # + # Pinned so that nobody restores the rejection as a bug fix. If the spec ever admits a published + # separation test, this is the test to change. + rng = np.random.default_rng(SEED) + fit = fit_tag_probabilities(rng.negative_binomial(3, 3 / (3 + 2), size=2000)) + assert fit.reason is None + assert _bound(fit.probabilities).any(), "the spec accepts invented binders on a tag that bound nothing" + + +# --- the trim ----------------------------------------------------------------------------------- + + +def test_the_trimmed_cells_still_get_a_probability(): + # The fit drops the counts above the 99th percentile so a handful of very high readings cannot + # drag the signal component's mean. Those cells are the most bound in the sample, so withholding + # an answer for them would be the opposite of what the trim is for. + counts = _mixture(1940, 2, 60, 300) + counts[-1] = 100_000 + fit = fit_tag_probabilities(counts) + assert fit.reason is None + assert fit.probabilities.size == counts.size + assert _bound(fit.probabilities)[-1], "the highest reading in the sample must read bound" + + +def test_the_fit_is_deterministic(): + counts = _mixture(1940, 2, 60, 300) + first = fit_tag_probabilities(counts).probabilities + second = fit_tag_probabilities(counts).probabilities + assert np.array_equal(first, second) + + +# --- the per-pair driver ------------------------------------------------------------------------ + + +def test_the_silent_cells_are_in_the_fit(): + # The counts frame holds only observed readings. A fit over those alone is a fit over the cells + # that read SOMETHING, which is not the background -- and on a mostly-silent tag it is barely + # any of it. + rng = np.random.default_rng(SEED) + cells = [("S1", f"c{i}") for i in range(2000)] + values = np.concatenate([rng.poisson(0.2, 1900), rng.poisson(200, 100)]) + rows = [("S1", f"c{i}", "AAAA", int(v)) for i, v in enumerate(values) if v > 0] + counts = _counts_frame(rows) + assert counts.height < 500, "the bed must be mostly silent or it proves nothing" + + fits = fit_tag_probabilities_by_pair(counts, cells, _panel([("AAAA", "S1")])) + assert fits.reasons == {} + assert fits.probabilities.height == 2000, "every cell in the sample is scored" + + +def test_the_probabilities_are_keyed_to_the_right_cells(): + # The fit is taken over an array and the answer is read back per cell, so the alignment between + # the two is the thing most easily lost. Only the planted binders may come back bound. + counts, cells = _bed(n_cells=2000, n_binders=60) + fits = fit_tag_probabilities_by_pair(counts, cells, _panel([("AAAA", "S1")])) + called = fits.probabilities.filter(pl.col("pBound") >= DISTRIBUTION_BOUND_PROBABILITY) + planted = {f"c{i}" for i in range(1940, 2000)} + assert set(called["cellId"].to_list()) == planted + + +def test_a_declared_tag_the_reads_never_showed_gets_no_fit(): + # Fitted over all zeros. One population, so no fit -- the honest answer, and the quality finding. + counts, cells = _bed() + fits = fit_tag_probabilities_by_pair(counts, cells, _panel([("AAAA", "S1"), ("DEAD", "S1")])) + assert fits.reasons == {("S1", "DEAD"): NO_SEPARATION} + assert set(fits.probabilities["tag"].unique().to_list()) == {"AAAA"} + + +def test_every_sample_is_fitted_on_its_own_cells(): + # Fits are local. Two samples staining one tag get two fits, and the one below the cell + # condition gets none -- the other is unaffected. + big_counts, big_cells = _bed(n_cells=2000, sample="S1") + small_counts, small_cells = _bed(n_cells=200, n_binders=6, sample="S2") + counts = pl.concat([big_counts, small_counts]) + fits = fit_tag_probabilities_by_pair(counts, big_cells + small_cells, _panel([("AAAA", "S1"), ("AAAA", "S2")])) + assert fits.reasons == {("S2", "AAAA"): TOO_FEW_CELLS} + assert set(fits.probabilities["sampleId"].unique().to_list()) == {"S1"} + + +def test_a_panel_with_no_sample_column_is_fitted_per_sample_anyway(): + # ANY_SAMPLE declares the tag for every sample. The population a fit is taken over is still one + # sample's cells, so there is still one fit each. + big_counts, big_cells = _bed(n_cells=2000, sample="S1") + small_counts, small_cells = _bed(n_cells=200, n_binders=6, sample="S2") + counts = pl.concat([big_counts, small_counts]) + fits = fit_tag_probabilities_by_pair(counts, big_cells + small_cells, _panel([("AAAA", ANY_SAMPLE)])) + assert fits.reasons == {("S2", "AAAA"): TOO_FEW_CELLS} + assert fits.probabilities.height == 2000 + + +def test_cells_outside_the_universe_do_not_enter_a_fit(): + # A barcode the analysis excluded carries real counts, and letting them into the background makes + # the fit a fit over a population nobody chose. + counts, cells = _bed() + intruders = _counts_frame([("S1", "x1", "AAAA", 900), ("S1", "x2", "AAAA", 900)]) + clean = fit_tag_probabilities_by_pair(counts, cells, _panel([("AAAA", "S1")])) + with_intruders = fit_tag_probabilities_by_pair(pl.concat([counts, intruders]), cells, _panel([("AAAA", "S1")])) + assert clean.probabilities.equals(with_intruders.probabilities) + + +def test_a_duplicated_cell_is_refused(): + # A duplicate adds one zero to the population it duplicates. That is a background over a + # population nobody chose -- small, plausible, and invisible in the output. + counts, cells = _bed() + with pytest.raises(ValueError, match="duplicated cells"): + fit_tag_probabilities_by_pair(counts, cells[:-1] + [cells[0]], _panel([("AAAA", "S1")])) + + +def test_a_tag_read_twice_in_one_cell_is_refused(): + # A tag read twice in one cell contributes twice to its own background and displaces a zero. + counts, cells = _bed() + doubled = pl.concat([counts, counts.head(1)]) + with pytest.raises(ValueError, match="duplicated readings"): + fit_tag_probabilities_by_pair(doubled, cells, _panel([("AAAA", "S1")])) + + +def test_a_fit_returns_the_background_it_fitted(): + # The parameters used to die inside the function that made them, so the one number a reader + # needs in order to judge a fit never left it. Two clear populations: a low background and a + # high signal, overdispersed so the negative binomial is the right model. + rng = np.random.default_rng(11) + background = rng.negative_binomial(2, 2 / (2 + 3), size=900) + signal = rng.negative_binomial(6, 6 / (6 + 90), size=300) + counts = np.concatenate([background, signal]) + + fit = fit_tag_probabilities(counts, min_cells=100) + + assert fit.probabilities is not None + assert fit.background is not None + # The background sits below the signal, which is what labelling the higher-median component + # signal means. Bounds rather than point values: the fit is stochastic. + assert fit.background.mean < fit.background.signal_mean + assert 0.0 < fit.background.weight < 1.0 + # The background holds the larger share, since three quarters of the cells are background. + assert fit.background.weight > 0.5 + + +def test_a_fit_that_established_nothing_carries_no_background(): + # `background` is None on exactly the condition `probabilities` is. A caller must branch on + # absence, and a background sitting beside a None probability would invite the other reading. + fit = fit_tag_probabilities(np.zeros(500, dtype=int), min_cells=100) + assert fit.probabilities is None + assert fit.background is None + + +def test_backgrounds_are_collected_per_sample_and_tag(): + # A pair in `reasons` contributes no background, and a pair that fitted contributes exactly + # one. The two dicts partition the declared pairs. + rng = np.random.default_rng(5) + rows = [] + for sample in ("S1", "S2"): + for cell in range(400): + rows.append((sample, f"c{cell}", "T1", int(rng.negative_binomial(2, 2 / (2 + 4))))) + counts = pl.DataFrame( + rows, orient="row", schema={"sampleId": pl.String, "cellId": pl.String, "tag": pl.String, "umiCount": pl.Int64} + ) + cells = [(s, f"c{c}") for s in ("S1", "S2") for c in range(400)] + panel = pl.DataFrame({"sample": ["S1", "S2"], "tag": ["T1", "T1"]}) + + fits = fit_tag_probabilities_by_pair(counts, cells, panel, min_cells=100) + + for key in fits.backgrounds: + assert key not in fits.reasons + for key in fits.reasons: + assert key not in fits.backgrounds diff --git a/software/per-cell-metrics/test/test_verdict.py b/software/per-cell-metrics/test/test_verdict.py new file mode 100644 index 0000000..989e0f0 --- /dev/null +++ b/software/per-cell-metrics/test/test_verdict.py @@ -0,0 +1,969 @@ +import math +import random + +import polars as pl +import pytest +from scipy.stats import beta +from verdict import ( + BOUND_CUTOFF, + DEFAULT_FLOOR, + DEFAULT_PANEL_MIN_MEMBERS, + Admissibility, + ReferenceChoice, + State, + UnreliableReason, + apply_floor, + cells_reading_nothing, + combine_tags_to_identities, + densify, + gate_cells, + read_states, + reference_by_cell, + served_source, + silent_tally, + specificity_score, +) + + +def _counts(rows): + return pl.DataFrame( + rows, orient="row", schema={"sampleId": pl.String, "cellId": pl.String, "tag": pl.String, "umiCount": pl.Int64} + ) + + +def test_default_floor_is_four(): + assert DEFAULT_FLOOR == 4 + + +def test_counts_below_the_floor_become_zero(): + df = _counts([("S1", "c1", "AAAA", 3), ("S1", "c1", "CCCC", 4)]) + out, stats = apply_floor(df, floor=4, reference_tags=set()) + assert out.sort("tag")["umiCount"].to_list() == [0, 4] + assert stats["readingsFloored"] == 1 + + +def test_floor_is_per_cell_and_tag_not_per_cell_total(): + df = _counts([("S1", "c1", "AAAA", 3), ("S1", "c1", "CCCC", 3)]) + out, stats = apply_floor(df, floor=4, reference_tags=set()) + assert out["umiCount"].to_list() == [0, 0] + assert stats["readingsFloored"] == 2 + + +def test_reference_tags_are_never_floored(): + df = _counts([("S1", "c1", "CTRL", 1), ("S1", "c1", "AAAA", 1)]) + out, _ = apply_floor(df, floor=4, reference_tags={"CTRL"}) + got = dict(zip(out["tag"].to_list(), out["umiCount"].to_list(), strict=True)) + assert got["CTRL"] == 1 # the comparator is not evidence of binding + assert got["AAAA"] == 0 + + +def test_cells_left_with_nothing_are_counted(): + df = _counts([("S1", "c1", "AAAA", 1), ("S1", "c2", "AAAA", 9)]) + _, stats = apply_floor(df, floor=4, reference_tags=set()) + assert stats["cellsEmptied"] == 1 + + +def test_the_floor_zeroes_readings_it_never_drops_rows(): + df = _counts([("S1", "c1", "AAAA", 9), ("S2", "c1", "AAAA", 9)]) + out, _ = apply_floor(df, floor=4, reference_tags=set()) + assert out.height == 2 + + +def test_floor_of_zero_removes_nothing(): + df = _counts([("S1", "c1", "AAAA", 1)]) + out, stats = apply_floor(df, floor=0, reference_tags=set()) + assert out["umiCount"].to_list() == [1] + assert stats["readingsFloored"] == 0 + + +def test_count_exactly_at_the_floor_survives(): + df = _counts([("S1", "c1", "AAAA", 4)]) + out, _ = apply_floor(df, floor=4, reference_tags=set()) + assert out["umiCount"].to_list() == [4] + + +def test_a_cell_holding_only_the_reference_is_not_emptied(): + # Its non-reference readings are absent, not zeroed. "Emptied" means the + # floor took a cell's evidence away, not that it never had any. + df = _counts([("S1", "c1", "CTRL", 1)]) + _, stats = apply_floor(df, floor=4, reference_tags={"CTRL"}) + assert stats["cellsEmptied"] == 0 + + +def test_a_cell_keeping_one_reading_is_not_emptied(): + df = _counts([("S1", "c1", "AAAA", 1), ("S1", "c1", "CCCC", 9)]) + _, stats = apply_floor(df, floor=4, reference_tags=set()) + assert stats["cellsEmptied"] == 0 + assert stats["readingsFloored"] == 1 + + +def test_the_same_cell_id_in_two_samples_empties_independently(): + # (sampleId, cellId) is the key. Keying on cellId alone would let S2's + # surviving reading rescue S1's emptied cell. + df = _counts([("S1", "c1", "AAAA", 1), ("S2", "c1", "AAAA", 9)]) + _, stats = apply_floor(df, floor=4, reference_tags=set()) + assert stats["cellsEmptied"] == 1 + + +def test_a_disabled_floor_is_a_no_op_even_for_a_zero_reading(): + # floor <= 0 returns early, and that early return is behavioural rather than an optimisation. + # Falling through would count a cell whose only reading is already 0 as "emptied", when the floor + # removed nothing. + df = _counts([("S1", "c1", "AAAA", 0)]) + out, stats = apply_floor(df, floor=0, reference_tags=set()) + assert out["umiCount"].to_list() == [0] + assert stats == {"readingsFloored": 0, "cellsEmptied": 0} + + +def test_an_empty_frame_floors_to_nothing(): + df = _counts([]) + out, stats = apply_floor(df, floor=4, reference_tags=set()) + assert out.height == 0 + assert stats == {"readingsFloored": 0, "cellsEmptied": 0} + + +def test_a_reading_that_was_already_zero_still_counts_as_evidence_lost(): + # Pins the deliberate asymmetry between had_evidence and kept_evidence. had_evidence must NOT + # filter on > 0: on the sparse frame this step is contracted to receive, every row is an observed + # reading, so a row's existence is what makes a cell one that had evidence. Adding "> 0" to + # had_evidence is a no-op on real input and silently changes this count once densified zeros + # exist, which is the reason densify runs after. + df = _counts([("S1", "c1", "AAAA", 0)]) + _, stats = apply_floor(df, floor=4, reference_tags=set()) + assert stats["cellsEmptied"] == 1 + + +def test_nothing_here_picks_a_rung(): + # `what-plays-the-baseline` requires the scientist to select among the rungs and requires that + # nothing selects for them. + # + # A tripwire rather than a permanent ban, the same shape as the empty-droplets one below: wanting + # a default again should be a deliberate act that deletes this test, not a helper that reappears + # in the layer furthest from the reader. The workflow omits --reference-source where the model's + # value is empty, so anything here that could pick a rung becomes the live rule the moment the + # model stops picking one. + import verdict + + assert not hasattr(verdict, "resolve_default_source") + + +def test_a_choice_that_cannot_serve_refuses_and_never_falls_to_another_rung(): + # `served_source` survived the removal above and is a different thing: it + # never picks a rung. There is no rung below to fall to either -- a baseline + # is required and a run without one does not happen, so it refuses and names + # the condition that failed. Both conditions are properties of the settings, + # knowable before anything is read. + with pytest.raises(SystemExit, match="declares no baseline tag"): + served_source(ReferenceChoice.DECLARED, set(), 40, 25) + with pytest.raises(SystemExit, match="below the 25 that rung needs"): + served_source(ReferenceChoice.PANEL, {"CTRL"}, 3, 25) + assert served_source(ReferenceChoice.DECLARED, {"CTRL"}, 3, 25) is ReferenceChoice.DECLARED + + +def test_there_is_no_bottom_rung(): + # A tripwire. Rung five was "no baseline exists, every verdict unreliable", + # and it is gone: a full punchcard of non-answers costs what a real run costs + # and looks like a result at a glance. + assert not hasattr(ReferenceChoice, "NONE") + + +def test_empty_droplets_is_not_offered(): + # A tripwire, not a permanent ban: the day this block genuinely receives + # gene expression and an empty-droplet population, EMPTY_DROPLETS gets + # implemented and this test is deleted, not fixed. + assert not hasattr(ReferenceChoice, "EMPTY_DROPLETS") + + +def test_several_reference_tags_are_refused_rather_than_combined(): + # Never take the highest of them: `baseline-scope` states that references are never combined, and + # taking the highest is a combination. + # + # Refused rather than given a different rule, because the atom's construct scopes each reference + # to a group of antigens by a declared property, and this version of the block has no group-by + # half. So it cannot say WHICH antigens a second comparator belongs to. It is also what the field + # does: the ordinary antibody run rejects a second control outright. + counts = _counts([("S1", "c1", "CTRL1", 3), ("S1", "c1", "CTRL2", 11)]) + with pytest.raises(SystemExit, match="declares 2 baseline tags"): + reference_by_cell(counts, {"CTRL1", "CTRL2"}, ReferenceChoice.DECLARED) + + +def test_one_reference_tag_still_serves(): + # The supported shape, and the one every antibody kit ships: one comparator + # for the whole panel. The refusal above must not catch it. + counts = _counts([("S1", "c1", "CTRL", 7)]) + ref, choice = reference_by_cell(counts, {"CTRL"}, ReferenceChoice.DECLARED) + assert choice is ReferenceChoice.DECLARED + assert ref[("S1", "c1")] == 7 + + +def test_several_reference_tags_do_not_block_a_rung_that_does_not_use_them(): + # The refusal is scoped to the rung that reads a declared tag AS the comparator. Under the panel + # rung several declared tags are just readings in the median, which is well defined however many + # there are. Refusing there would withdraw a run over a question that does not arise in it. + counts = _counts([("S1", "c1", "CTRL1", 3), ("S1", "c1", "CTRL2", 11), ("S1", "c1", "AAAA", 9)]) + _, choice = reference_by_cell(counts, {"CTRL1", "CTRL2"}, ReferenceChoice.PANEL, panel_size=25, min_members=25) + assert choice is ReferenceChoice.PANEL + + +def test_cell_missing_the_reference_tag_reads_zero(): + counts = _counts([("S1", "c1", "CTRL", 5), ("S1", "c2", "AAAA", 9)]) + ref, _ = reference_by_cell(counts, {"CTRL"}, ReferenceChoice.DECLARED) + assert ref[("S1", "c2")] == 0 + + +def test_a_declared_rung_with_no_declared_tag_refuses(): + # The refusal reaches the caller through `reference_by_cell` too, since that + # is where a hand-driven run enters. The message names the condition and what + # to change, because a scientist meeting it can fix the panel in a minute. + counts = _counts([("S1", "c1", "AAAA", 9)]) + with pytest.raises(SystemExit, match="declares no baseline tag"): + reference_by_cell(counts, set(), ReferenceChoice.DECLARED) + + +def test_shipped_defaults_are_pinned(): + # These are user-facing numbers that appear in a dropdown and change what the block produces, so + # an edit to any of them must be a deliberate, visible act -- not a silent one that only this test + # would otherwise catch. The high-reference line is not calibrated against real data. + # + # The panel minimum is different in kind: it GATES the rung rather than tuning it, and it comes + # from one preprint whose own panels held fifty and a hundred members. It was 8, which no source + # supports. At 25 the rung is out of reach of any antibody panel, since those kits cap at fifteen + # tags, and such a panel falls to the tag-distribution rung instead. + assert DEFAULT_PANEL_MIN_MEMBERS == 25 + + +def test_panel_source_serves_exactly_at_the_minimum(): + # The minimum is a floor, not a gap: a panel of exactly min_members is + # large enough. Nothing else in the suite distinguishes < from <=. + counts = _counts([("S1", "c1", "AAAA", 9), ("S1", "c1", "CCCC", 1)]) + _, choice = reference_by_cell(counts, set(), ReferenceChoice.PANEL, panel_size=5, min_members=5) + assert choice is ReferenceChoice.PANEL + + +def test_panel_source_refuses_one_below_the_minimum(): + # A gate rather than a preference: below it the baseline the rung permits is + # not conservative but wrong, so the run refuses rather than serving it. + counts = _counts([("S1", "c1", "AAAA", 9)]) + with pytest.raises(SystemExit, match="below the 5 that rung needs"): + reference_by_cell(counts, set(), ReferenceChoice.PANEL, panel_size=4, min_members=5) + + +def test_the_gate_boundary_includes_the_line_itself(): + # A reading exactly at the threshold is high: the named value satisfies the condition it names. + # This matches the floor, where a count of exactly `floor` is evidence, and the panel minimum, + # where exactly `min_members` is large enough. Both sides are pinned so that changing the + # comparison is a deliberate act. + at_line = {("S1", "c1"): 100} + just_below = {("S1", "c2"): 99} + aside_at, high_at = gate_cells(at_line, threshold=100) + aside_below, high_below = gate_cells(just_below, threshold=100) + assert aside_at == {("S1", "c1")} and high_at == 1 + assert aside_below == set() and high_below == 0 + + +def test_one_threshold_does_both_jobs(): + # 290-reference-two-roles allows one line here, not two: how many are high needs a high, and + # only a declared gate supplies one. So the cells set aside and the cells counted high are the + # SAME cells, by construction. A second observation line used to make the count independent of + # the gate, which meant counting against a boundary nobody declared. + ref = {("S1", "a"): 500, ("S1", "b"): 50, ("S1", "c"): 2000} + + aside_hi, high_hi = gate_cells(ref, threshold=1000) + assert aside_hi == {("S1", "c")} + assert high_hi == len(aside_hi) + + aside_lo, high_lo = gate_cells(ref, threshold=10) + assert aside_lo == {("S1", "a"), ("S1", "b"), ("S1", "c")} + assert high_lo == len(aside_lo) + + +def test_no_gate_counts_nothing_rather_than_counting_zero(): + # None, never 0. A zero would report a run in which no cell read high, which is a finding. With + # no gate there is no *high* to read against at all, and the caller reports the spread of the + # readings instead -- what a scientist reads in order to declare a gate. + ref = {("S1", "a"): 500, ("S1", "b"): 50, ("S1", "c"): 2000} + aside, high = gate_cells(ref, threshold=None) + assert aside == set() + assert high is None + + +def test_a_source_that_cannot_be_served_refuses_and_never_moves_sideways(): + # A rung that cannot serve refuses. It never substitutes another, which would + # silently answer a question the scientist did not ask -- and two runs served + # by different baselines are not comparable, so nothing may pick for them. + counts = _counts([("S1", "c1", "AAAA", 9), ("S1", "c1", "CCCC", 3)]) + # DECLARED with nothing declared: refused, not served by PANEL, even though a + # panel of a hundred tags sits right there and would satisfy that rung. + with pytest.raises(SystemExit, match="declares no baseline tag"): + reference_by_cell(counts, set(), ReferenceChoice.DECLARED, panel_size=100, min_members=5) + # PANEL below the minimum: refused, not served by DECLARED, even though a + # reference tag exists and would satisfy that rung. + with pytest.raises(SystemExit, match="below the 5 that rung needs"): + reference_by_cell(counts, {"CTRL"}, ReferenceChoice.PANEL, panel_size=1, min_members=5) + + +def test_the_panel_comparator_is_the_median_not_the_mean(): + # A cell with one strong binder: the mean is dragged up by it, the median + # is not. The comparator is meant to stand for the cell's background, so a + # single high reading must not raise the bar it is measured against. + counts = _counts( + [ + ("S1", "c1", "AAAA", 1), + ("S1", "c1", "CCCC", 2), + ("S1", "c1", "GGGG", 3), + ("S1", "c1", "TTTT", 200), + ] + ) + ref, choice = reference_by_cell(counts, set(), ReferenceChoice.PANEL, panel_size=8, min_members=5) + assert choice is ReferenceChoice.PANEL + assert ref[("S1", "c1")] == 2 # median of 1,2,3,200 -> 2.5 -> int 2; mean would be 51 + + +def test_the_panel_median_truncates_rather_than_rounds(): + # A median of 1.5 is the value that separates the two: truncation gives 1, and polars' + # round-half-to-even gives 2. At a median of 2.5 both give 2, so a fixture there cannot tell them + # apart. The difference matters, because a comparator of 1 and one of 2 give different scores. + # Truncation is the behaviour. This pins it. + counts = _counts( + [ + ("S1", "c1", "AAAA", 1), + ("S1", "c1", "CCCC", 1), + ("S1", "c1", "GGGG", 2), + ("S1", "c1", "TTTT", 2), + ] + ) + ref, choice = reference_by_cell(counts, set(), ReferenceChoice.PANEL, panel_size=8, min_members=5) + assert choice is ReferenceChoice.PANEL + assert ref[("S1", "c1")] == 1 + + +def test_an_explicit_empty_cell_list_means_no_cells(): + # Not "derive them from the counts frame". An empty list is a statement. + counts = _counts([("S1", "c1", "CTRL", 7)]) + ref, choice = reference_by_cell(counts, {"CTRL"}, ReferenceChoice.DECLARED, cells=[]) + assert choice is ReferenceChoice.DECLARED + assert ref == {} + + +def test_cells_outside_the_given_list_are_excluded(): + # The cell list is the analysis. A cell with a real reference reading that is not in it has a + # comparator nobody will consult, and returning it would invite a reader to treat the result as + # the cell universe. + counts = _counts([("S1", "c1", "CTRL", 7), ("S1", "c2", "CTRL", 9)]) + ref, _ = reference_by_cell(counts, {"CTRL"}, ReferenceChoice.DECLARED, cells=[("S1", "c1")]) + assert ref == {("S1", "c1"): 7} + + +def test_a_named_cell_with_no_reference_reading_is_zero_not_missing(): + # Both directions in one assertion: c2 is added at 0, c3 is excluded. + counts = _counts([("S1", "c1", "CTRL", 7), ("S1", "c3", "CTRL", 4)]) + ref, _ = reference_by_cell(counts, {"CTRL"}, ReferenceChoice.DECLARED, cells=[("S1", "c1"), ("S1", "c2")]) + assert ref == {("S1", "c1"): 7, ("S1", "c2"): 0} + + +def test_the_panel_source_also_respects_the_given_cell_list(): + # Two branches now share the cell-list rule. Only one is covered above. + counts = _counts([("S1", "c1", "AAAA", 9), ("S1", "c2", "AAAA", 3)]) + ref, choice = reference_by_cell( + counts, set(), ReferenceChoice.PANEL, cells=[("S1", "c1")], panel_size=8, min_members=5 + ) + assert choice is ReferenceChoice.PANEL + assert ref == {("S1", "c1"): 9} + + +def _ident(rows): + return pl.DataFrame( + rows, + orient="row", + schema={"sampleId": pl.String, "cellId": pl.String, "identity": pl.String, "umiCount": pl.Int64}, + ) + + +def _cells(pairs): + return pl.DataFrame(pairs, orient="row", schema={"sampleId": pl.String, "cellId": pl.String}) + + +def test_state_has_exactly_four_members(): + assert {s.value for s in State} == {"bound", "not bound", "never asked", "unreliable"} + + +def test_densify_gives_a_silent_cell_a_real_zero(): + counts = _ident([("S1", "c1", "A", 7)]) + cells = _cells([("S1", "c1")]) + out = densify(counts, cells, offered_by_sample={"S1": {"A", "B"}}).sort("identity") + assert out["identity"].to_list() == ["A", "B"] + assert out["umiCount"].to_list() == [7, 0] # B was asked and silent + + +def test_densify_does_not_invent_unoffered_identities(): + counts = _ident([("S1", "c1", "A", 7)]) + cells = _cells([("S1", "c1")]) + out = densify(counts, cells, offered_by_sample={"S1": {"A"}}) + assert out["identity"].to_list() == ["A"] + + +def test_identity_reading_is_the_highest_not_the_sum(): + df = _counts([("S1", "c1", "AAAA", 10), ("S1", "c1", "CCCC", 7)]) + out = combine_tags_to_identities(df, {("AAAA", "S1"): "A", ("CCCC", "S1"): "A"}) + assert out["umiCount"].to_list() == [10] + + +def test_combine_keeps_sample_id(): + df = _counts([("S1", "c1", "AAAA", 5), ("S2", "c1", "AAAA", 9)]) + out = combine_tags_to_identities(df, {("AAAA", "S1"): "A", ("AAAA", "S2"): "A"}) + assert out.height == 2 and set(out["sampleId"].to_list()) == {"S1", "S2"} + + +def test_specificity_score_matches_the_published_formula(): + assert math.isclose(specificity_score(10, 2), (1.0 - beta.cdf(0.925, 11, 5)) * 100.0, rel_tol=1e-12) + + +def test_cutoff_is_seventy_five(): + assert BOUND_CUTOFF == 75.0 + + +def test_high_count_against_a_quiet_reference_is_bound(): + out = read_states(_ident([("S1", "c1", "A", 200)]), Admissibility({("S1", "c1"): 0}, set()), 75.0) + assert out["state"].to_list() == [State.BOUND.value] + + +def test_zero_reads_not_bound_never_unreliable(): + out = read_states(_ident([("S1", "c1", "A", 0)]), Admissibility({("S1", "c1"): 5}, set()), 75.0) + assert out["state"].to_list() == [State.NOT_BOUND.value] + + +def test_a_very_low_comparator_is_scored_rather_than_rerouted(): + # `count-becomes-a-state` deleted the thin-reference branch rather than filling it in. No + # published line separates thin from usable, so the comparison runs and the reference reading is + # emitted for the reader to judge instead. A comparator of 1 is a real comparison, and the score + # decides it like any other. + out = read_states(_ident([("S1", "c1", "A", 50)]), Admissibility({("S1", "c1"): 1}, set()), 75.0) + assert out["state"].to_list() == [State.NOT_BOUND.value] # scores 58.4, under the cutoff + assert out["unreliableReason"].to_list() == [None] + assert out["referenceCount"].to_list() == [1], "the reader is given what the verdict rested on" + + +def test_gated_cell_is_unreliable_and_stays_in_the_frame(): + out = read_states(_ident([("S1", "c1", "A", 500)]), Admissibility({("S1", "c1"): 900}, {("S1", "c1")}), 75.0) + assert out.height == 1 + assert out["state"].to_list() == [State.UNRELIABLE.value] + + +def test_a_gated_cell_reports_the_gate_even_when_its_reference_is_very_low(): + # The gate set this cell aside AND its comparator reads 1. A very low comparator is no longer a + # reason on its own, so only the gate can be reported. But the reason is an exported column that a + # later step reads to tell a panel problem from a re-run problem, so this pins that the gate is + # what it says. A cell the gate set aside was not measured at all. + out = read_states(_ident([("S1", "c1", "A", 500)]), Admissibility({("S1", "c1"): 1}, {("S1", "c1")}), 75.0) + assert out["state"].to_list() == [State.UNRELIABLE.value] + reason = out["unreliableReason"].to_list()[0] + assert reason == UnreliableReason.GATED + + +def test_densify_handles_a_sample_stained_with_nothing(): + # A non-empty offered map whose every value is empty contributes no block. + # Guarding on the map rather than the assembled blocks raised here. + out = densify(_ident([]), _cells([("S1", "c1")]), offered_by_sample={"S1": set()}) + assert out.height == 0 + assert out.schema["identity"] == pl.String + assert out.schema["umiCount"] == pl.Int64 + + +def test_never_asked_is_not_produced_here(): + out = read_states(_ident([("S1", "c1", "A", 0)]), Admissibility({("S1", "c1"): 5}, set()), 75.0) + assert State.NEVER_ASKED.value not in out["state"].to_list() + + +def test_no_score_column_leaves_the_reading(): + out = read_states(_ident([("S1", "c1", "A", 50)]), Admissibility({("S1", "c1"): 5}, set()), 75.0) + assert "score" not in out.columns + assert {"umiCount", "referenceCount"} <= set(out.columns) + + +def test_specificity_score_stays_within_zero_and_hundred_at_sample_points(): + for a, r in [(0, 0), (5, 5), (1000, 3)]: + assert 0.0 <= specificity_score(a, r) <= 100.0 + + +def test_a_score_exactly_at_the_cutoff_is_bound(): + # The named value satisfies the condition it names, as everywhere else here. Integer counts have + # no rational preimage of a fixed cutoff like 75.0 under the beta CDF, so the exact boundary is + # built the other way round: compute a reading's own score, then feed that exact value back in as + # the cutoff. The comparison then lands on the line with no floating-point drift, and ">=" must + # call it bound. + exact = specificity_score(10, 2) + out = read_states(_ident([("S1", "c1", "A", 10)]), Admissibility({("S1", "c1"): 2}, set()), cutoff=exact) + assert out["state"].to_list() == [State.BOUND.value] + + +def test_no_line_separates_a_thin_comparator_from_a_usable_one(): + # There is no boundary left to be off-by-one about. A comparator of 1 and one + # of 2 differ only in the score they produce, and a large enough antigen count + # binds against either. + for ref in (1, 2): + low = read_states(_ident([("S1", "c1", "A", 0)]), Admissibility({("S1", "c1"): ref}, set()), 75.0) + high = read_states(_ident([("S1", "c2", "A", 500)]), Admissibility({("S1", "c2"): ref}, set()), 75.0) + assert low["unreliableReason"].to_list() == [None], f"comparator {ref} is a comparison" + assert low["state"].to_list() == [State.NOT_BOUND.value] + assert high["state"].to_list() == [State.BOUND.value], f"500 binds against a comparator of {ref}" + + +def test_no_comparator_is_unreliable_but_a_comparator_reading_zero_is_scored(): + # The two must not collapse. served=NONE, modelled here as an empty reference dict per + # reference_by_cell's contract, means no comparison existed. A comparator present and reading 0 is + # a real comparison and scores normally: a positive antigen count against a zero reference is + # This also subsumes the plain no-comparator-is-unreliable check. Nothing else in the suite needs + # a weaker, reason-blind version of this. + no_comparator = read_states(_ident([("S1", "c1", "A", 200)]), Admissibility({}, set()), 75.0) + zero_comparator = read_states(_ident([("S1", "c1", "A", 200)]), Admissibility({("S1", "c1"): 0}, set()), 75.0) + assert no_comparator["state"].to_list() == [State.UNRELIABLE.value] + assert no_comparator["unreliableReason"].to_list() == [UnreliableReason.NO_COMPARATOR] + assert zero_comparator["state"].to_list() == [State.BOUND.value] + assert zero_comparator["unreliableReason"].to_list() == [None] + + +def test_silent_admissible_cell_can_never_score_bound(): + # The fact the analytic path rests on: specificity_score(0, r) is ~0.0422 + # at r = 0 and smaller for every larger r. A silent admissible cell is + # therefore always *not bound* for any cutoff above that bound, which is + # what lets silent_tally skip materializing its row. + assert math.isclose(specificity_score(0, 0), 0.0422, abs_tol=5e-4) + scores = [specificity_score(0, r) for r in range(0, 50)] + assert scores == sorted(scores, reverse=True) + assert all(s < 0.05 for s in scores) + + +def test_duplicated_cells_rows_give_the_deduped_answer(): + # A row-count bug this project shipped once already: keys built from + # `cells` without dedup counted the duplicated c2 row as if it were a + # second cell. asked must count distinct cells (2), not rows (3), and + # silentNotBound must follow from the deduped count. + cells = _cells([("S1", "c1"), ("S1", "c2"), ("S1", "c2")]) + admissibility = Admissibility({("S1", "c1"): 5, ("S1", "c2"): 5}, set()) + observed = read_states(_ident([("S1", "c1", "A", 50)]), admissibility, 75.0) + tally = silent_tally(observed, cells, {"S1": {"A"}}, admissibility) + row = tally.row(0, named=True) + assert row["asked"] == 2 # not 3: the duplicated c2 row counts once + assert row["silentNotBound"] == 1 + + +def test_duplicated_observed_rows_are_rejected_not_silently_wrong(): + # Recorded rather than latent: without the assertion in silent_tally, this combination silently + # returned silentUnreliable == -1. A duplicated observed row for an inadmissible cell is counted + # twice against a total that counts the cell once. `observed` must be unique on (cell, identity). + # This input violates that, so the function must now refuse it loudly instead of emitting a + # negative count. + cells = _cells([("S1", "c1")]) + admissibility = Admissibility({}, set()) # no comparator for c1: inadmissible + observed = read_states(_ident([("S1", "c1", "A", 50), ("S1", "c1", "A", 50)]), admissibility, 75.0) + # ValueError rather than AssertionError, and the type is the point: an `assert` is stripped + # under -O, and this guard stripped does not crash -- it returns a wrong answer. Pinning the + # type here is what keeps it from quietly becoming strippable again. + with pytest.raises(ValueError): + silent_tally(observed, cells, {"S1": {"A"}}, admissibility) + + +def _build_silent_tally_population(seed, force_empty_sample=None): + # Shared by every check below: build a small, varied population by + # construction -- several samples, cells, identities, some cells gated, + # some with a very low reference, some with a normal one. + rng = random.Random(seed) + samples = ["S1", "S2", "S3"] + identities = ["A", "B", "C"] + gated: set[tuple[str, str]] = set() + reference: dict[tuple[str, str], int] = {} + cell_rows = [] + tag_rows = [] + offered_by_sample: dict[str, set[str]] = {} + + for sample in samples: + if sample == force_empty_sample: + offered_by_sample[sample] = set() + else: + offered_by_sample[sample] = set(rng.sample(identities, k=rng.randint(1, len(identities)))) + for i in range(6): + cell = f"c{i}" + cell_rows.append((sample, cell)) + key = (sample, cell) + # Reference reading: sometimes missing (no comparator), sometimes + # very low, sometimes ordinary. + roll = rng.random() + if roll < 0.2: + pass # no comparator for this cell + elif roll < 0.4: + reference[key] = 1 # a very low comparator, still comparable + else: + reference[key] = rng.randint(2, 20) + if rng.random() < 0.15: + gated.add(key) + # Sparse observed readings: only some (cell, identity) pairs the + # sample offered actually got a tag-stat row. + for identity in offered_by_sample[sample]: + if rng.random() < 0.5: + # Some readings must actually clear the cutoff. Against references of 2-20 the + # largest score a count of 30 can reach is about 11.8, so a population drawn only + # from 0-30 contains no bound cell at all. The oracle comparison's bound assertion + # below then reads 0 == 0 in every run, proving nothing about the claim it names. + count = rng.randint(0, 30) if rng.random() < 0.7 else rng.randint(200, 900) + tag_rows.append((sample, cell, identity, count)) + + return samples, identities, gated, reference, cell_rows, tag_rows, offered_by_sample + + +def _check_silent_tally_matches_oracle(seed, cutoff=BOUND_CUTOFF, force_empty_sample=None): + # Checks silent_tally's three cheap terms, sample-keyed (the default), + # against the dense grid built by densify and read through read_states, + # which never skips a row. + samples, identities, gated, reference, cell_rows, tag_rows, offered_by_sample = _build_silent_tally_population( + seed, force_empty_sample + ) + + cells = _cells(cell_rows) + sparse_identities = _ident(tag_rows) + admissibility = Admissibility(reference, gated) + observed = read_states(sparse_identities, admissibility, cutoff) + + dense = densify(sparse_identities, cells, offered_by_sample) + oracle = read_states(dense, admissibility, cutoff) + + tally = silent_tally(observed, cells, offered_by_sample, admissibility) + + # A tally that emits extra rows -- one for an identity a sample never + # offered -- must fail here, not just disagree on counts. + expected_row_count = sum(len(offered) for offered in offered_by_sample.values()) + assert tally.height == expected_row_count + + for sample in samples: + offered = offered_by_sample[sample] + for identity in identities: + group_filter = (pl.col("sampleId") == sample) & (pl.col("identity") == identity) + if identity not in offered: + # Never asked of this sample: no row at all, not a zero row. + assert tally.filter(group_filter).height == 0 + continue + + oracle_group = oracle.filter(group_filter) + observed_group = observed.filter(group_filter) + tally_row = tally.filter(group_filter).row(0, named=True) + + oracle_states = oracle_group["state"].to_list() + observed_states = observed_group["state"].to_list() + + # A silent admissible cell can never be observed as bound, so the + # oracle and the sparse frame must agree exactly on bound counts. + assert oracle_states.count(State.BOUND.value) == observed_states.count(State.BOUND.value) + + expected_silent_unreliable = oracle_states.count(State.UNRELIABLE.value) - observed_states.count( + State.UNRELIABLE.value + ) + expected_silent_not_bound = oracle_states.count(State.NOT_BOUND.value) - observed_states.count( + State.NOT_BOUND.value + ) + + assert tally_row["asked"] == len(oracle_states) + assert tally_row["observed"] == len(observed_states) + assert tally_row["silentUnreliable"] == expected_silent_unreliable + assert tally_row["silentNotBound"] == expected_silent_not_bound + + +@pytest.mark.parametrize( + "seed, force_empty_sample", + [ + (20260817, None), + (1, None), + (2, None), + (3, None), + (4, None), + (5, None), + (6, None), + # The generator above never draws an empty offered set on its own. Force one so the + # empty-block path in densify and the zero-row case in silent_tally are both exercised against + # the oracle, not just against each other. + (7, "S2"), + ], +) +def test_silent_tally_agrees_with_the_densify_oracle_on_small_random_inputs(seed, force_empty_sample): + _check_silent_tally_matches_oracle(seed, force_empty_sample=force_empty_sample) + + +def test_silent_tally_agrees_with_the_oracle_at_a_low_valid_cutoff(): + # 0.5 is comfortably above specificity_score(0, 0) ~= 0.0422, the bound + # named in specificity_score's and silent_tally's docstrings. This guards + # the boundary itself rather than assuming BOUND_CUTOFF=75.0 is + # representative of every cutoff the equivalence must hold for. + _check_silent_tally_matches_oracle(seed=20260817, cutoff=0.5) + + +def _check_silent_tally_matches_oracle_grouped(seed, cutoff=BOUND_CUTOFF, force_empty_sample=None): + # Same population and same dense oracle as the sample-keyed check above, but the cells are + # regrouped into sets that mix samples with different offered identities, and silent_tally is + # called with that grouping. A set's cell index (0..5) becomes its group, independent of sample, + # so every group is guaranteed to contain a member from all three samples. That is exactly the + # shape a hoisted asked/total_inadmissible would get wrong, since S1, S2 and S3 are built with + # independently random offered sets and need not agree on what a given group's identity was + # offered. + samples, identities, gated, reference, cell_rows, tag_rows, offered_by_sample = _build_silent_tally_population( + seed, force_empty_sample + ) + + cells = _cells(cell_rows) + sparse_identities = _ident(tag_rows) + admissibility = Admissibility(reference, gated) + observed = read_states(sparse_identities, admissibility, cutoff) + dense = densify(sparse_identities, cells, offered_by_sample) + oracle = read_states(dense, admissibility, cutoff) + + group_by_cell = {(sample, cell): cell for sample, cell in cell_rows} + groups = sorted({cell for _, cell in cell_rows}) + + tally = silent_tally( + observed, cells, offered_by_sample, admissibility, group_by_cell=group_by_cell, group_column="setId" + ) + + for group in groups: + members = {k for k, g in group_by_cell.items() if g == group} + offered_here = set().union(*(offered_by_sample[k[0]] for k in members)) + for identity in identities: + group_filter = pl.col("setId") == group + if identity not in offered_here: + # None of this group's members' own samples offered it: no + # row at all, not a zero row. + assert tally.filter(group_filter & (pl.col("identity") == identity)).height == 0 + continue + + def _states_for(frame): + # A plain Python filter, not a polars struct comparison. This only needs to run over + # a handful of rows in a test, and it sidesteps any doubt about how polars compares + # struct columns. + return [ + state + for sample_id, cell_id, ident, state in zip( + frame["sampleId"].to_list(), + frame["cellId"].to_list(), + frame["identity"].to_list(), + frame["state"].to_list(), + strict=True, + ) + if (sample_id, cell_id) in members and ident == identity + ] + + oracle_states = _states_for(oracle) + observed_states = _states_for(observed) + + tally_row = tally.filter(group_filter & (pl.col("identity") == identity)).row(0, named=True) + + expected_silent_unreliable = oracle_states.count(State.UNRELIABLE.value) - observed_states.count( + State.UNRELIABLE.value + ) + expected_silent_not_bound = oracle_states.count(State.NOT_BOUND.value) - observed_states.count( + State.NOT_BOUND.value + ) + + assert tally_row["asked"] == len(oracle_states) + assert tally_row["observed"] == len(observed_states) + assert tally_row["silentUnreliable"] == expected_silent_unreliable + assert tally_row["silentNotBound"] == expected_silent_not_bound + + +@pytest.mark.parametrize( + "seed, force_empty_sample", + [ + (20260817, None), + (1, None), + (2, None), + (7, "S2"), + ], +) +def test_silent_tally_agrees_with_the_oracle_when_groups_span_differing_panels(seed, force_empty_sample): + _check_silent_tally_matches_oracle_grouped(seed, force_empty_sample=force_empty_sample) + + +def test_silent_tally_group_column_is_named_by_the_caller(): + cells = _cells([("S1", "c1"), ("S2", "c1")]) + observed = _ident([]) + admissibility = Admissibility({("S1", "c1"): 5, ("S2", "c1"): 5}, set()) + tally = silent_tally( + observed, + cells, + {"S1": {"A"}, "S2": {"A"}}, + admissibility, + group_by_cell={("S1", "c1"): "G1", ("S2", "c1"): "G1"}, + group_column="setId", + ) + assert tally.columns[0] == "setId" + row = tally.row(0, named=True) + assert row["setId"] == "G1" and row["asked"] == 2 # both samples' c1 land in one group + + +def test_a_group_spanning_two_panels_does_not_inflate_silent_unreliable(): + # THE hoist bug, pinned directly: S1 offers A, S2 does not. A group holds + # one cell from each, and S2's cell is gated. A hoisted total_inadmissible + # would count S2's gated cell against identity A too, even though S2 + # never offered A -- inflating silentUnreliable for an identity that + # cell was never asked about. + cells = _cells([("S1", "c1"), ("S2", "c2")]) + observed = _ident([]) # both cells silent + admissibility = Admissibility({("S1", "c1"): 5}, {("S2", "c2")}) + tally = silent_tally( + observed, + cells, + offered_by_sample={"S1": {"A"}, "S2": {"B"}}, + admissibility=admissibility, + group_by_cell={("S1", "c1"): "G1", ("S2", "c2"): "G1"}, + group_column="setId", + ) + row_a = tally.filter(pl.col("identity") == "A").row(0, named=True) + assert row_a["asked"] == 1 # only S1's cell, not S2's + assert row_a["silentUnreliable"] == 0 # S2's gated cell must not count against A + assert row_a["silentNotBound"] == 1 # S1's silent, admissible cell votes not bound + + +def test_same_barcode_combines_into_different_identities_by_sample(): + # The cell's own sample decides which antigen its barcode counted toward. Keyed by tag alone this + # put both cells under whichever identity the dataset-wide map happened to hold. + df = _counts([("S1", "c1", "AAAA", 10), ("S2", "c2", "AAAA", 20)]) + out = combine_tags_to_identities(df, {("AAAA", "S1"): "A", ("AAAA", "S2"): "B"}).sort("sampleId") + assert out["identity"].to_list() == ["A", "B"] + assert out["umiCount"].to_list() == [10, 20] + + +def test_combine_is_still_the_max_within_one_sample(): + # tags-combine-by-the-highest is unchanged by the keying: two tags of one identity in one cell + # still combine to the highest, never the sum. + df = _counts([("S1", "c1", "AAAA", 3), ("S1", "c1", "CCCC", 7)]) + out = combine_tags_to_identities(df, {("AAAA", "S1"): "A", ("CCCC", "S1"): "A"}) + assert out["umiCount"].to_list() == [7] + + +def test_a_global_panel_entry_applies_to_every_sample_when_combining(): + df = _counts([("S1", "c1", "AAAA", 4), ("S2", "c2", "AAAA", 9)]) + out = combine_tags_to_identities(df, {("AAAA", "*"): "A"}).sort("sampleId") + assert out["identity"].to_list() == ["A", "A"] + assert out["umiCount"].to_list() == [4, 9] + + +def test_an_explicit_per_sample_declaration_beats_the_global_one(): + # A panel mixing "*" with named rows is refused by the reader. The fill order is pinned here so a + # caller building a frame directly cannot silently get the global answer for a named sample. + df = _counts([("S1", "c1", "AAAA", 5)]) + out = combine_tags_to_identities(df, {("AAAA", "*"): "GLOBAL", ("AAAA", "S1"): "MINE"}) + assert out["identity"].to_list() == ["MINE"] + + +def test_the_comparator_is_computed_on_raw_counts_not_floored_ones(tmp_path): + """The minimum count acts on the numerator only. Every rung reads its own source raw. + + The bug this pins was at the call site, not in this module: production handed `reference_by_cell` + the FLOORED frame. Two consequences, and the second is the sharper one. + + A cell whose panel readings straddle the minimum medians differently before and after: + [1, 1, 2, 9, 9] medians to 2, and the same readings floored at 4 are [0, 0, 0, 9, 9], which + medians to 0. A comparator of 0 rather than 2 moves every one of that cell's verdicts toward + *bound*. + + And the median was internally inconsistent wherever a reference tag was also present, because the + minimum exempts reference tags and floors every antigen tag. So a single median ran over a mixture + of raw and floored values. Nobody chose that. + """ + raw = _counts( + [ + ("S1", "c1", "AAAA", 1), + ("S1", "c1", "CCCC", 1), + ("S1", "c1", "GGGG", 2), + ("S1", "c1", "TTTT", 9), + ("S1", "c1", "ACAC", 9), + ] + ) + floored = apply_floor(raw, 4, set()).counts + + on_raw, _ = reference_by_cell(raw, set(), ReferenceChoice.PANEL, panel_size=5, min_members=5) + on_floored, _ = reference_by_cell(floored, set(), ReferenceChoice.PANEL, panel_size=5, min_members=5) + + assert on_raw[("S1", "c1")] == 2 + # Held so the test fails loudly if the minimum ever stops biting here — the + # two frames must actually differ, or this proves nothing. + assert on_floored[("S1", "c1")] == 0 + + +def test_the_minimum_spares_the_baseline_tag_by_default(): + # The shipped behaviour. The minimum removes what is not evidence OF BINDING, + # and the comparator is not evidence of binding. + counts = _counts([("S1", "c1", "CTRL", 2), ("S1", "c1", "AAAA", 2)]) + out, stats = apply_floor(counts, floor=4, reference_tags={"CTRL"}) + by_tag = dict(zip(out["tag"].to_list(), out["umiCount"].to_list(), strict=True)) + assert by_tag == {"CTRL": 2, "AAAA": 0} + assert stats["readingsFloored"] == 1 + + +def test_the_minimum_never_reaches_the_baseline_tag(): + # `minimum-count-before-any-reference` puts this as a rule and not a preference: the + # minimum asks whether a count is evidence of binding, and a tag declared to be bound + # by nothing never is. There is no switch. A below-minimum comparator survives, and + # the count of removed readings counts the antigen tag alone. + counts = _counts([("S1", "c1", "CTRL", 2), ("S1", "c1", "AAAA", 2)]) + out, stats = apply_floor(counts, floor=4, reference_tags={"CTRL"}) + by_tag = dict(zip(out["tag"].to_list(), out["umiCount"].to_list(), strict=True)) + assert by_tag == {"CTRL": 2, "AAAA": 0} + assert stats["readingsFloored"] == 1 + + +def test_a_cell_holding_only_an_exempt_comparator_is_not_emptied(): + # A cell holding ONLY a below-minimum comparator. The comparator is exempt, so the cell + # never had evidence of binding removed and the minimum emptied nothing. + # + # Scoping the emptied population one way while flooring the other would report a cell as + # losing evidence it never held. + counts = _counts([("S1", "c1", "CTRL", 2)]) + assert apply_floor(counts, 4, {"CTRL"}).stats["cellsEmptied"] == 0 + + +def test_the_minimum_changes_no_comparator(): + # Pinned at the source. Every rung reads its own counts raw, so the comparator is built + # from the unfloored frame and the minimum cannot reach it. That is what makes the + # exemption a statement about the numerator alone. + counts = _counts([("S1", "c1", "CTRL", 2), ("S1", "c1", "AAAA", 9)]) + apply_floor(counts, 4, {"CTRL"}) + ref, _ = reference_by_cell(counts, {"CTRL"}, ReferenceChoice.DECLARED) + assert ref[("S1", "c1")] == 2 + + +def _counts(rows): + return pl.DataFrame( + rows, orient="row", schema={"sampleId": pl.String, "cellId": pl.String, "tag": pl.String, "umiCount": pl.Int64} + ) + + +def test_a_cell_whose_comparator_survived_read_something(): + # The whole discriminator. `support-travels-with-the-reading` says a cell whose antigen tags all + # fell below the minimum while its comparator survived took up reagent and none of it was antigen: + # a real negative and a real vote, not an empty cell. `cellsEmptied` cannot see this, because with + # the comparator exempt it is scoped to the readings the minimum was allowed to remove. + counts = _counts([("S1", "c1", "AAAA", 2), ("S1", "c1", "CTRL", 6)]) + floored = apply_floor(counts, 4, {"CTRL"}).counts + assert floored.filter(pl.col("umiCount") > 0)["tag"].to_list() == ["CTRL"] + assert cells_reading_nothing(floored, {("S1", "c1")}) == set() + + +def test_a_cell_keeping_only_its_comparator_did_not_read_nothing(): + # The comparator is always exempt, so this cell keeps a reading. It did not read nothing, even + # though nothing it holds is evidence of binding. That is the accounting the exemption fixes. + counts = _counts([("S1", "c1", "AAAA", 2), ("S1", "c1", "CTRL", 3)]) + floored = apply_floor(counts, 4, {"CTRL"}).counts + assert cells_reading_nothing(floored, {("S1", "c1")}) == set() + + +def test_a_cell_with_no_row_at_all_read_nothing(): + # The frame is sparse, so a cell that read nothing anywhere produces no row rather than a row of + # zeros. Reading the population off the frame would miss exactly the cells it is looking for. + counts = _counts([("S1", "c1", "AAAA", 500)]) + floored = apply_floor(counts, 4, set()).counts + assert cells_reading_nothing(floored, {("S1", "c1"), ("S1", "c2")}) == {("S1", "c2")} + + +def test_the_universe_bounds_the_answer(): + # A cell outside the universe does not belong here however it read. This is what keeps the count + # from ever exceeding the clonotype's own cell count. + counts = _counts([("S1", "c1", "AAAA", 500)]) + floored = apply_floor(counts, 4, set()).counts + assert cells_reading_nothing(floored, set()) == set() + assert cells_reading_nothing(floored, {("S1", "c1")}) == set() diff --git a/software/test-data/fixtures/per-cell-metrics/generate.py b/software/test-data/fixtures/per-cell-metrics/generate.py index 3346184..d5c2f79 100644 --- a/software/test-data/fixtures/per-cell-metrics/generate.py +++ b/software/test-data/fixtures/per-cell-metrics/generate.py @@ -4,11 +4,11 @@ The bed mirrors the output of `mitool tag-stat -t CELL -t FEATURE -u UMI`: one row per (cell, feature-barcode) group, columns `CELL FEATURE count totalWeight unique_UMI`, ordered by the -distinct-UMI count descending. `unique_UMI` is the distinct-molecule count mitool computes; `count` -(raw read occurrences) is deliberately larger than `unique_UMI` so the tests prove the metrics read -the deduplicated `unique_UMI` column, not the raw read count. +distinct-UMI count descending. `unique_UMI` is the distinct-molecule count mitool computes. `count`, +the raw read occurrences, is deliberately larger than `unique_UMI` so the tests prove the metrics read +the deduplicated `unique_UMI` column and not the raw read count. -It is intentionally tiny and hand-designed so per-cell metrics are hand-computable (on unique_UMI): +It is intentionally tiny and hand-designed so per-cell metrics are hand-computable, on unique_UMI: cell1 -> dominant on AGX (3 vs 1 UMIs, 0.75 share) cell2 -> ambiguous (1/1/1 across AGX/BGX/CTRL) cell3 -> single feature AGX (2 UMIs) diff --git a/software/test-data/fixtures/verdicts/README.md b/software/test-data/fixtures/verdicts/README.md new file mode 100644 index 0000000..59c0673 --- /dev/null +++ b/software/test-data/fixtures/verdicts/README.md @@ -0,0 +1,114 @@ +# Verdict Fixture Bed + +The bed the binding-verdict tests run the whole CLI against. Committed rather than generated at test +time, so a test never has to run the generator, and excluded from ruff. Regenerate with +`python generate.py` — it is stdlib only and takes one fixed seed, so the files come back byte +identical. + +Entirely synthetic. This repository is public: barcodes are random ACGT strings, antigens are `AgNN`, +samples are `SNN`, and no real sequence, antigen or sample identifier appears anywhere. + +## Files + +| File | What it is | +|---|---| +| `panel.csv` | Four samples, panels of 3, 4, 4 and 5 tags. **No** comparator tag. | +| `panel_with_reference.csv` | The same panels plus **one** comparator tag (`Ctrl1`) on every sample. | +| `panel_multi_reference.csv` | The same panels plus **two** comparator tags (`Ctrl1`, `Ctrl2`) on every sample. This version of the block **refuses** it on the declared rung — see the shapes table below. It still serves on the panel rung, where a declared tag is an ordinary reading, and it is the only panel here declaring every barcode the counts carry. | +| `panel_narrow.csv` | The **three-column** shape: sample, barcode, antigen name, and no fourth column. No role column, so nothing can be named as a comparator and the panel's own readings serve. The control is an ordinary row nothing marks. | +| `panel_wide.csv` | The **seven-column** shape: sample, name, catalogue id, barcode, channel, a constant column, role. The role column declares target vs off-target and carries **no** comparator value. Includes case-variant role values. | + +### The two customer shapes, and what they are for + +Both were observed in use at one account at the same time, on two of its projects — so neither is a +legacy form of the other. They are projections of the same slots, samples and names as the three panels +above, which is what lets `counts.csv` and `linker.csv` apply to all five unchanged: the panels differ +only in the shape of the declaration. + +**Neither carries a value meaning "comparator."** In both, the negative control is one antigen the +scientist points at by name in the interface. So a run over either resolves to the panel's own readings. + +`panel_narrow.csv` reproduces, on nine tags, what the observed file does on seventeen: four barcodes +carry a different antigen name in different samples, so four identity labels fall back to the raw +barcode. Run it with `--barcode-col Sequence --feature-col Antigen --sample-col Sample` and no +`--role-column`. + +`panel_wide.csv` adds three things the narrow shape cannot show. A **catalogue id** 1:1 with the +sequence, so pointing the barcode role at the wrong one of the two joins to nothing. A **channel** +column holding four values that are three channels, one of them spelled two ways. A **constant** +column, which is a declared property carrying no information — group on it and every tag becomes one +identity. + +And its `Type` column carries **case variants**, deliberately, because the observed file held six values +that were three roles. Two failure modes, kept separate so a test can tell them apart: + +- `A0`'s slot reads `Target (Primary)` in two samples and `Target (primary)` in the other two. One + barcode, two values, so the property is **dropped for that tag entirely** and it ends up with no role. +- `A5`'s slot reads `Off-target` wherever it appears — self-consistent, so it keeps its role, but it no + longer matches the `Off-Target` written elsewhere. Selecting one value silently misses the other. + +**What running the wide panel with `--reference-values "Off-Target"` demonstrates** is why the role +column is the wrong source for a comparator: reference tags are held out of the identity universe, so +the identity count drops from nine to seven and **the off-targets stop being asked about at all**. The +question an off-target exists to pose is deleted rather than answered. +| `counts.csv` | Sparse per-(sample, cell, barcode) UMI counts for all eleven cells. | +| `linker.csv` | Cell to clonotype set: `K01`, `K02`, `K03` (spanning two samples), `K04` (a singleton). | + +Columns are `Samples,Name,Sequence,Type` in the panels, so a run reads the bed with +`--barcode-col Sequence --feature-col Name --sample-col Samples --role-column Type +--reference-values Control`. + +One `counts.csv` serves all three panels, so the two comparator barcodes are read in every run +including those whose panel does not declare them. Those readings surface as `undeclared-in-panel` +rows and are expected: `panel_multi_reference.csv` is the only panel here that declares every barcode +the counts carry, so it is the bed to use when the mismatch table itself is under test. + +## Panel shapes covered + +| Shape | How the bed carries it | +|---|---| +| Per-sample panels of differing size | 3, 4, 4 and 5 tags. `K01` is drawn from the three-tag sample alone, so five of the eight identities read *never asked*. | +| Same barcode, different names across samples | Four barcodes carry two `AgNN` names each. A fifth recurs under one name, so a test can tell "recurs" from "recurs inconsistently". | +| One panel over every sample | Not a separate file: the comparator rows are declared on all four samples, which is the unkeyed case within a keyed panel. | +| A designated negative control | `panel_with_reference.csv`. | +| **No** negative control | `panel.csv`. Eight distinct barcodes, below the shipped minimum of 25, so the panel does **not** serve as its own comparator: a run over it reads *unreliable* throughout unless the minimum is lowered on the command line, which several tests do deliberately. | +| **Several** designated controls | `panel_multi_reference.csv`. `Ctrl2` reads above `Ctrl1` in every cell. This used to be the bed for *the higher of the two serves*; that rule is gone. `baseline-scope` states that references are never combined, and taking the highest is a combination — so the run is now **refused**, naming both tags. The 60-against-6 spread is kept because it is what makes a combining rule detectable if one ever returns. | +| One antigen on several barcodes | `Ag07` is carried on two barcodes, both on the fourth sample. | +| A barcode declared in one sample, read in another | `Ag06`'s barcode is declared by the third sample only and read in the second only, so both directions of the check fire on different samples at once. | +| Free-text properties, inconsistently spelled | Not carried here. The panel has no free-text property column beyond `Name`; `test_panel.py` covers the hygiene measurement. | + +## The counts, and which threshold each one is for + +Shipped defaults in `verdict.py`: floor **4**, bound cutoff **75** on +`specificity_score`, high-reference observation line **100**. The score is a beta function and not a +ratio, so the useful values are not where intuition puts them — against a comparator of 6 a count of +8 scores 0.0001, 50 scores 3.1, 60 scores 7.2 and 500 scores 100. + +| Count | Chosen against | +|---|---| +| `8` | The *not bound* reading. Above the floor of 4, so it survives to be compared, and 0.0001 against a comparator of 6, so it is compared and fails. A count of 2 would be zeroed by the floor and read *not bound* for a different reason. | +| `500` | The *bound* reading while the comparator is 6 (score 100) — and a *not bound* reading against 60 (score 0.1). That difference is what the two-comparator panel measures. | +| `5000` | Bound against either comparator, so one binding survives on the two-comparator panel and the bed does not degenerate into all *not bound*. | +| `2` (one reading only) | Below the floor of 4, so it is zeroed and counted in `readingsFloored`. | +| `6` (`Ctrl1`) | Far below 500, so a real binding clears the cutoff. There is no lower bound to calibrate against any more: `count-becomes-a-state` deleted the thin-reference branch, so any comparator reading is compared. | +| `60` (`Ctrl2`) | Above `Ctrl1` so the highest-member rule is observable, and below the high-reference line of 100 so that measurement stays quiet. | +| `1` (`Ctrl1` in `c08`) | Below the floor of 4. The floor spares a **declared** comparator, so this reading survives to be compared and gives the same states a 6 would. Read with no declared comparator it is floored like any other count, which is what makes the exemption observable. | +| `400` (`Ctrl1` in `c11`) | Above the bed's `--gate-threshold` of 100, so `c11` is set aside by the admissibility gate. This is the bed's only per-cell source of *unreliable*: lower it and the fourth state disappears from `K04`. | + +## What each set reads, on `panel_with_reference.csv` + +| Set | Cells | Reads | +|---|---|---| +| `K01` | three cells of the three-tag sample | *bound* twice, *not bound* once, *never asked* five times. One of its cells is silent on a bound identity and votes *not bound* against two that bind it. | +| `K02` | three cells of a four-tag sample | *bound* twice, *not bound* twice, *never asked* four times. One of its readings is floored. | +| `K03` | four cells across two samples | Offered the union of two panels, so nothing in it reads *never asked*. Two identities read *unreliable* on a tie. | +| `K04` | one cell the gate sets aside | *unreliable* everywhere it was offered, *never asked* elsewhere. | + +Two readings are worth naming, because both are states an earlier revision got wrong: + +- `Ag06`'s barcode is declared by the third sample and read in none of its cells. `K03` draws from + that sample, so it was offered `Ag06`, its cells could be compared, and they read nothing — which + is *not bound*, not *never asked*. A silent cell that can be compared is a negative answer. +- The same barcode is read in the second sample, which never declared it. `K02` therefore reads + *never asked* at that identity while a real count of 500 sits in `counts.csv`. The verdict follows + the panel; the mismatch table is what makes the reading visible. diff --git a/software/test-data/fixtures/verdicts/counts.csv b/software/test-data/fixtures/verdicts/counts.csv new file mode 100644 index 0000000..2871c42 --- /dev/null +++ b/software/test-data/fixtures/verdicts/counts.csv @@ -0,0 +1,64 @@ +sampleId,cellId,tag,umiCount +S01,c01,TGTAGACGCATA,6 +S01,c01,GGGGAATTCAAT,60 +S01,c01,AGAACCCCCCTT,5000 +S01,c01,AGTTAAGAACAA,8 +S01,c01,AAGCAACAATCT,5000 +S01,c02,TGTAGACGCATA,6 +S01,c02,GGGGAATTCAAT,60 +S01,c02,AGAACCCCCCTT,5000 +S01,c02,AGTTAAGAACAA,8 +S01,c02,AAGCAACAATCT,5000 +S01,c03,TGTAGACGCATA,6 +S01,c03,GGGGAATTCAAT,60 +S01,c03,AGAACCCCCCTT,5000 +S01,c03,AGTTAAGAACAA,8 +S02,c04,TGTAGACGCATA,6 +S02,c04,GGGGAATTCAAT,60 +S02,c04,AGAACCCCCCTT,500 +S02,c04,AGTTAAGAACAA,8 +S02,c04,TCGTGGTCCTGG,500 +S02,c04,TCCGTGACTTTG,8 +S02,c04,ACCTTACGGGCT,500 +S02,c05,TGTAGACGCATA,6 +S02,c05,GGGGAATTCAAT,60 +S02,c05,AGAACCCCCCTT,500 +S02,c05,AGTTAAGAACAA,8 +S02,c05,TCGTGGTCCTGG,500 +S02,c05,TCCGTGACTTTG,8 +S02,c06,TGTAGACGCATA,6 +S02,c06,GGGGAATTCAAT,60 +S02,c06,AGAACCCCCCTT,500 +S02,c06,AGTTAAGAACAA,8 +S02,c06,TCGTGGTCCTGG,2 +S02,c06,TCCGTGACTTTG,8 +S03,c07,TGTAGACGCATA,6 +S03,c07,GGGGAATTCAAT,60 +S03,c07,AGAACCCCCCTT,500 +S03,c07,AAGCAACAATCT,500 +S03,c07,TCGTGGTCCTGG,8 +S03,c08,TGTAGACGCATA,1 +S03,c08,GGGGAATTCAAT,60 +S03,c08,AGAACCCCCCTT,500 +S03,c08,AAGCAACAATCT,500 +S03,c08,TCGTGGTCCTGG,8 +S04,c09,TGTAGACGCATA,6 +S04,c09,GGGGAATTCAAT,60 +S04,c09,AGAACCCCCCTT,500 +S04,c09,AGTTAAGAACAA,8 +S04,c09,TCCGTGACTTTG,500 +S04,c09,CTTTTTGCCGTT,500 +S04,c09,CATCTCTAGTCT,8 +S04,c10,TGTAGACGCATA,6 +S04,c10,GGGGAATTCAAT,60 +S04,c10,AGAACCCCCCTT,500 +S04,c10,AGTTAAGAACAA,8 +S04,c10,TCCGTGACTTTG,500 +S04,c10,CTTTTTGCCGTT,8 +S04,c10,CATCTCTAGTCT,500 +S04,c11,TGTAGACGCATA,400 +S04,c11,AGAACCCCCCTT,500 +S04,c11,AGTTAAGAACAA,8 +S04,c11,TCCGTGACTTTG,500 +S04,c11,CTTTTTGCCGTT,8 +S04,c11,CATCTCTAGTCT,8 diff --git a/software/test-data/fixtures/verdicts/generate.py b/software/test-data/fixtures/verdicts/generate.py new file mode 100644 index 0000000..ca14b03 --- /dev/null +++ b/software/test-data/fixtures/verdicts/generate.py @@ -0,0 +1,335 @@ +"""Regenerate the synthetic verdict fixture bed. + +Run from this directory: python generate.py + +Stdlib only, and the only random thing is the barcode alphabet soup. Every count, name, sample and set +membership below is written out by hand, because each one is load-bearing against a threshold and a +generated number would be load-bearing against nothing. The seed is passed to `random.Random` rather +than seeding the module, so the bed regenerates byte-identically and a second generator running in the +same process cannot disturb this one. + +Everything here is invented. The repository is public: no real barcode sequence, antigen name or +sample identifier may appear. Barcodes are drawn from ACGT, antigens are `AgNN`, samples are `SNN`. + +The thresholds the counts are chosen against, all shipped defaults in `verdict.py`: + + floor 4 a reading below this is zeroed before anything else runs + bound cutoff 75 on `specificity_score`, which is a beta function and not a ratio + high-reference line 100 a comparator at or above this is flagged as an observation + +Against a comparator of 6 the score is 0.0001 at a count of 8, 3.1 at 50, 7.2 at 60 and 100 at 500. +Against a comparator of 60 it is 0.1 at 500 and 100 at 5000. That is why 8 means *not bound*, 500 means +*bound* only while the comparator stays at 6, and 5000 is the count that survives the higher comparator +of the two-control panel. +""" + +import random + +SEED = 20260817 +BARCODE_LENGTH = 12 + +# Slot names, not sequences. The tests never hard-code a sequence. They recover each barcode by the role +# it plays in the panel -- two names, two barcodes under one name, declared here and read there -- so a +# regenerated bed with different sequences still exercises the same shapes. +ANTIGEN_SLOTS = ["A0", "A1", "A2", "A3", "A4", "A5", "A6", "A7"] +CONTROL_SLOTS = ["R0", "R1"] + +# (sample, antigen name, barcode slot). Panels of 3, 4, 4 and 5 tags across four samples. +# +# A0, A1, A3, A4 each recur under two different names. That is the case that makes name-keyed +# identity wrong, and the reason the pipeline keys on the barcode. +# A2 recurs under ONE name, so a test can tell "recurs" from "recurs inconsistently". +# A6 and A7 both carry Ag07: one antigen on two barcodes, read by the highest member. +# A5 is declared by S03 alone and, see COUNTS, read in S02 alone. That is the only way both +# directions of the panel-versus-reads check can be seen to run per sample rather than globally: +# a global check would let S03's declaration excuse the reading in S02. +PANEL = [ + ("S01", "Ag01", "A0"), + ("S01", "Ag02", "A1"), + ("S01", "Ag03", "A2"), + ("S02", "Ag11", "A0"), + ("S02", "Ag02", "A1"), + ("S02", "Ag04", "A3"), + ("S02", "Ag05", "A4"), + ("S03", "Ag01", "A0"), + ("S03", "Ag03", "A2"), + ("S03", "Ag14", "A3"), + ("S03", "Ag06", "A5"), + ("S04", "Ag11", "A0"), + ("S04", "Ag12", "A1"), + ("S04", "Ag15", "A4"), + ("S04", "Ag07", "A6"), + ("S04", "Ag07", "A7"), +] + +SAMPLES = ["S01", "S02", "S03", "S04"] + +# The comparator rows the two reference beds add, on every sample. A comparator declared in one sample +# and not another is discarded by `consistent_properties` rather than honoured, so a tag is a comparator +# everywhere or nowhere. +CONTROL_NAMES = {"R0": "Ctrl1", "R1": "Ctrl2"} + +# (sample, cell, barcode slot, umiCount). +# +# R0 reads 6 in every cell but c11, where it reads 400. The bed is run with --gate-threshold 100, so +# c11 is set aside by the admissibility gate and every identity its set was offered reads *unreliable*. +# That is the bed's only per-cell source of the fourth state, so lowering c11's 400 costs it. +# +# It used to read 1 instead, which fell below a thin-reference line that routed the cell to +# *unreliable*. `count-becomes-a-state` deleted that branch -- no published line separates a thin +# comparator from a usable one -- so the state now comes from the gate, which is a declared parameter. +# +# R1 reads 60, above R0 everywhere it appears, so the two-control panel's comparator is 60 and not 6. +# 60 also sits below the high-reference observation line of 100, keeping that measurement quiet. +# c11 has no R1 row, so it stays impossible to compare on the two-control panel too. +# +# 8 is the *not bound* count: above the floor of 4, so the reading survives to be compared, and +# 0.0001 against a comparator of 6, so it is compared and fails. A count of 2 would be zeroed by the +# floor and would read *not bound* for the wrong reason. +# 500 is the *bound* count against a comparator of 6 (score 100) and a *not bound* count against 60 +# (score 0.1) -- that difference is what the two-control bed measures. +# 5000 stays bound against either comparator. +COUNTS = [ + # -- S01, set K01: the three-tag panel, so five of the eight identities were never asked. + ("S01", "c01", "R0", 6), + ("S01", "c01", "R1", 60), + ("S01", "c01", "A0", 5000), + ("S01", "c01", "A1", 8), + ("S01", "c01", "A2", 5000), + ("S01", "c02", "R0", 6), + ("S01", "c02", "R1", 60), + ("S01", "c02", "A0", 5000), + ("S01", "c02", "A1", 8), + ("S01", "c02", "A2", 5000), + ("S01", "c03", "R0", 6), + ("S01", "c03", "R1", 60), + ("S01", "c03", "A0", 5000), + ("S01", "c03", "A1", 8), + # c03 has no A2 row: a cell that was asked and read nothing. It is comparable, so it votes + # *not bound* against the two cells that bound A2, and the majority still says bound. + # -- S02, set K02. + ("S02", "c04", "R0", 6), + ("S02", "c04", "R1", 60), + ("S02", "c04", "A0", 500), + ("S02", "c04", "A1", 8), + ("S02", "c04", "A3", 500), + ("S02", "c04", "A4", 8), + # A5 is read here and declared only by S03. The reading is real and the panel still says S02 was + # never asked, which is the whole reason the mismatch table has to travel with the answer. + ("S02", "c04", "A5", 500), + ("S02", "c05", "R0", 6), + ("S02", "c05", "R1", 60), + ("S02", "c05", "A0", 500), + ("S02", "c05", "A1", 8), + ("S02", "c05", "A3", 500), + ("S02", "c05", "A4", 8), + ("S02", "c06", "R0", 6), + ("S02", "c06", "R1", 60), + ("S02", "c06", "A0", 500), + ("S02", "c06", "A1", 8), + # 2 is below the floor of 4 and is zeroed, so this is the bed's floored reading and c06 votes + # *not bound* on A3 while c04 and c05 bind it. + ("S02", "c06", "A3", 2), + ("S02", "c06", "A4", 8), + # -- S03 and S04 together form set K03, so its offered set is the union of two panels and + # nothing in it reads *never asked*. + ("S03", "c07", "R0", 6), + ("S03", "c07", "R1", 60), + ("S03", "c07", "A0", 500), + ("S03", "c07", "A2", 500), + ("S03", "c07", "A3", 8), + # No A5 row in S03 at all, though S03 is the only sample that declares it: the other direction of + # the same check. Both S03 cells were offered A5 and read nothing, so K03 reads *not bound* at A5, + # not *never asked*, which is the regression this shape exists to catch. + # + # c08's comparator reads 1: below the floor of 4, but the floor spares a DECLARED comparator, so it + # survives to be compared -- 0.96 against 8, and 100 against 500, the same states a 6 gives. Read + # without a declaration it is floored like any other count, which is what makes the exemption + # observable. It used to be c11 that carried this, before c11 was raised to 400 for the gate. + ("S03", "c08", "R0", 1), + ("S03", "c08", "R1", 60), + ("S03", "c08", "A0", 500), + ("S03", "c08", "A2", 500), + ("S03", "c08", "A3", 8), + ("S04", "c09", "R0", 6), + ("S04", "c09", "R1", 60), + ("S04", "c09", "A0", 500), + ("S04", "c09", "A1", 8), + ("S04", "c09", "A4", 500), + # A6 and A7 are the two barcodes of Ag07, and the two S04 cells of K03 bind opposite ones. Read + # per barcode each splits its set one to one and reads *unreliable* on the tie. Read as one + # antigen by the highest member, both cells bind Ag07 and the set reads *bound*. + ("S04", "c09", "A6", 500), + ("S04", "c09", "A7", 8), + ("S04", "c10", "R0", 6), + ("S04", "c10", "R1", 60), + ("S04", "c10", "A0", 500), + ("S04", "c10", "A1", 8), + ("S04", "c10", "A4", 500), + ("S04", "c10", "A6", 8), + ("S04", "c10", "A7", 500), + # -- S04, set K04: one cell, comparator high enough for the gate to set it aside. + ("S04", "c11", "R0", 400), + ("S04", "c11", "A0", 500), + ("S04", "c11", "A1", 8), + ("S04", "c11", "A4", 500), + ("S04", "c11", "A6", 8), + ("S04", "c11", "A7", 8), +] + +# (sample, cell, set). K03 spans two samples on purpose; K04 is a singleton, which many real +# clonotypes are. +LINKER = [ + ("S01", "c01", "K01"), + ("S01", "c02", "K01"), + ("S01", "c03", "K01"), + ("S02", "c04", "K02"), + ("S02", "c05", "K02"), + ("S02", "c06", "K02"), + ("S03", "c07", "K03"), + ("S03", "c08", "K03"), + ("S04", "c09", "K03"), + ("S04", "c10", "K03"), + ("S04", "c11", "K04"), +] + + +def barcodes() -> dict[str, str]: + """A distinct ACGT sequence per slot, in slot order, from the fixed seed.""" + rng = random.Random(SEED) + assigned: dict[str, str] = {} + used: set[str] = set() + for slot in ANTIGEN_SLOTS + CONTROL_SLOTS: + while True: + seq = "".join(rng.choice("ACGT") for _ in range(BARCODE_LENGTH)) + if seq not in used: + break + used.add(seq) + assigned[slot] = seq + return assigned + + +def write_panel(path: str, seq: dict[str, str], controls: list[str]) -> None: + with open(path, "w") as f: + f.write("Samples,Name,Sequence,Type\n") + for sample, name, slot in PANEL: + f.write(f"{sample},{name},{seq[slot]},Target\n") + for sample in SAMPLES: + for slot in controls: + f.write(f"{sample},{CONTROL_NAMES[slot]},{seq[slot]},Control\n") + + +# --- the two shapes real panel files arrive in -------------------------------------------------- +# +# Projections of the same slots, samples and names as the panels above, so counts.csv and linker.csv +# apply to them unchanged and the three panels differ only in the shape of the declaration. Both shapes +# were observed in use at one account, at the same time, on two of its projects. +# +# NEITHER carries a value meaning "comparator", and that is the point of them. In both, the negative +# control is one antigen the scientist points at by name in the interface. So a run over either resolves +# to the panel's own readings, and `--reference-values` has nothing correct to name. The `Type` column of +# the wide shape declares what a member is TO THE QUESTION -- a target, an off-target -- which is a +# different axis from what a count is read against. Naming `Off-Target` as the comparator does not merely +# mis-set a baseline: reference tags are held out of the identity universe, so every off-target stops +# being asked about at all, and the question an off-target exists to pose is deleted. + +# A per-slot catalogue id, 1:1 with the sequence. Real files carry both, and a reader who points the +# barcode role at the catalogue id instead of the sequence joins to nothing. +CATALOGUE_IDS = {slot: f"T{100 + i:04d}" for i, slot in enumerate(ANTIGEN_SLOTS + CONTROL_SLOTS)} + +# Four distinct values, two of which are one channel spelled two ways -- so grouping on this column +# splits one channel in two. +CHANNELS = { + "A0": "PE", "A1": "PE", + "A2": "APC", "A3": "APC", + "A4": "PE Dazzle", "A5": "PE Dazzle", + "A6": "PE-Dazzle 5120", "A7": "PE-Dazzle 5120", + "R0": "APC", "R1": "APC", +} + +# One value on every row: a declared column that carries no information at all. Grouping on it puts +# every tag in one identity, which is legal and useless. +RESIDUES = "ECD protein" + +# What each member is to the question. No "Control" anywhere, deliberately. +TYPES = { + "A0": "Target (Primary)", "A1": "Target (Primary)", + "A2": "Off-Target", "A3": "Target (Secondary)", + "A4": "Target (Secondary)", "A5": "Off-Target", + "A6": "Target (Primary)", "A7": "Target (Primary)", + "R0": "Off-Target", "R1": "Off-Target", +} + +# Case variants, because the observed file carried six values that were three roles. Two failure +# modes, kept separate so a test can tell them apart: +# +# A0 reads "Target (Primary)" in S01/S03 and "Target (primary)" in S02/S04. One barcode, two +# values, so the property is dropped for that tag entirely -- it ends up with no role at all. +# +# A5 reads "Off-target" everywhere it appears. Self-consistent, so it keeps its role, but it no +# longer matches A2's "Off-Target" -- so selecting one role value silently misses the other. +LOWERCASED_IN = {"A0": {"S02", "S04"}} +ALWAYS_LOWERCASED = {"A5"} + + +def _typed(slot: str, sample: str) -> str: + role = TYPES[slot] + if slot in ALWAYS_LOWERCASED or sample in LOWERCASED_IN.get(slot, set()): + # Lowercase only the parenthesised qualifier or the word after the hyphen, which is how the + # observed variants differed -- not a blanket .lower(). + return role.replace("(P", "(p").replace("(S", "(s").replace("-Target", "-target") + return role + + +def write_panel_narrow(path: str, seq: dict[str, str]) -> None: + """Three columns and no fourth: sample, barcode, antigen name. + + The declared control is present as an ordinary antigen row, exactly as it is in the observed file + -- nothing in the table says it is the control. + """ + with open(path, "w") as f: + f.write("Sample,Sequence,Antigen\n") + for sample, name, slot in PANEL: + f.write(f"{sample},{seq[slot]},{name}\n") + for sample in SAMPLES: + f.write(f"{sample},{seq['R0']},{CONTROL_NAMES['R0']}\n") + + +def write_panel_wide(path: str, seq: dict[str, str]) -> None: + """Seven columns: sample, name, catalogue id, barcode, channel, a constant, and the role.""" + with open(path, "w") as f: + f.write("Samples,Name,Barcode,Sequence,Channel,Residues,Type\n") + for sample, name, slot in PANEL: + f.write( + f"{sample},{name},{CATALOGUE_IDS[slot]},{seq[slot]}," + f"{CHANNELS[slot]},{RESIDUES},{_typed(slot, sample)}\n" + ) + for sample in SAMPLES: + slot = "R0" + f.write( + f"{sample},{CONTROL_NAMES[slot]},{CATALOGUE_IDS[slot]},{seq[slot]}," + f"{CHANNELS[slot]},{RESIDUES},{_typed(slot, sample)}\n" + ) + + +def main() -> None: + seq = barcodes() + write_panel("panel.csv", seq, []) + write_panel("panel_with_reference.csv", seq, ["R0"]) + write_panel("panel_multi_reference.csv", seq, ["R0", "R1"]) + write_panel_narrow("panel_narrow.csv", seq) + write_panel_wide("panel_wide.csv", seq) + + with open("counts.csv", "w") as f: + f.write("sampleId,cellId,tag,umiCount\n") + for sample, cell, slot, umi in COUNTS: + f.write(f"{sample},{cell},{seq[slot]},{umi}\n") + + with open("linker.csv", "w") as f: + f.write("sampleId,cellId,setId\n") + for sample, cell, set_id in LINKER: + f.write(f"{sample},{cell},{set_id}\n") + + +if __name__ == "__main__": + main() diff --git a/software/test-data/fixtures/verdicts/linker.csv b/software/test-data/fixtures/verdicts/linker.csv new file mode 100644 index 0000000..5e7dafa --- /dev/null +++ b/software/test-data/fixtures/verdicts/linker.csv @@ -0,0 +1,12 @@ +sampleId,cellId,setId +S01,c01,K01 +S01,c02,K01 +S01,c03,K01 +S02,c04,K02 +S02,c05,K02 +S02,c06,K02 +S03,c07,K03 +S03,c08,K03 +S04,c09,K03 +S04,c10,K03 +S04,c11,K04 diff --git a/software/test-data/fixtures/verdicts/panel.csv b/software/test-data/fixtures/verdicts/panel.csv new file mode 100644 index 0000000..7df09f6 --- /dev/null +++ b/software/test-data/fixtures/verdicts/panel.csv @@ -0,0 +1,17 @@ +Samples,Name,Sequence,Type +S01,Ag01,AGAACCCCCCTT,Target +S01,Ag02,AGTTAAGAACAA,Target +S01,Ag03,AAGCAACAATCT,Target +S02,Ag11,AGAACCCCCCTT,Target +S02,Ag02,AGTTAAGAACAA,Target +S02,Ag04,TCGTGGTCCTGG,Target +S02,Ag05,TCCGTGACTTTG,Target +S03,Ag01,AGAACCCCCCTT,Target +S03,Ag03,AAGCAACAATCT,Target +S03,Ag14,TCGTGGTCCTGG,Target +S03,Ag06,ACCTTACGGGCT,Target +S04,Ag11,AGAACCCCCCTT,Target +S04,Ag12,AGTTAAGAACAA,Target +S04,Ag15,TCCGTGACTTTG,Target +S04,Ag07,CTTTTTGCCGTT,Target +S04,Ag07,CATCTCTAGTCT,Target diff --git a/software/test-data/fixtures/verdicts/panel_multi_reference.csv b/software/test-data/fixtures/verdicts/panel_multi_reference.csv new file mode 100644 index 0000000..f8a5dc4 --- /dev/null +++ b/software/test-data/fixtures/verdicts/panel_multi_reference.csv @@ -0,0 +1,25 @@ +Samples,Name,Sequence,Type +S01,Ag01,AGAACCCCCCTT,Target +S01,Ag02,AGTTAAGAACAA,Target +S01,Ag03,AAGCAACAATCT,Target +S02,Ag11,AGAACCCCCCTT,Target +S02,Ag02,AGTTAAGAACAA,Target +S02,Ag04,TCGTGGTCCTGG,Target +S02,Ag05,TCCGTGACTTTG,Target +S03,Ag01,AGAACCCCCCTT,Target +S03,Ag03,AAGCAACAATCT,Target +S03,Ag14,TCGTGGTCCTGG,Target +S03,Ag06,ACCTTACGGGCT,Target +S04,Ag11,AGAACCCCCCTT,Target +S04,Ag12,AGTTAAGAACAA,Target +S04,Ag15,TCCGTGACTTTG,Target +S04,Ag07,CTTTTTGCCGTT,Target +S04,Ag07,CATCTCTAGTCT,Target +S01,Ctrl1,TGTAGACGCATA,Control +S01,Ctrl2,GGGGAATTCAAT,Control +S02,Ctrl1,TGTAGACGCATA,Control +S02,Ctrl2,GGGGAATTCAAT,Control +S03,Ctrl1,TGTAGACGCATA,Control +S03,Ctrl2,GGGGAATTCAAT,Control +S04,Ctrl1,TGTAGACGCATA,Control +S04,Ctrl2,GGGGAATTCAAT,Control diff --git a/software/test-data/fixtures/verdicts/panel_narrow.csv b/software/test-data/fixtures/verdicts/panel_narrow.csv new file mode 100644 index 0000000..19965a2 --- /dev/null +++ b/software/test-data/fixtures/verdicts/panel_narrow.csv @@ -0,0 +1,21 @@ +Sample,Sequence,Antigen +S01,AGAACCCCCCTT,Ag01 +S01,AGTTAAGAACAA,Ag02 +S01,AAGCAACAATCT,Ag03 +S02,AGAACCCCCCTT,Ag11 +S02,AGTTAAGAACAA,Ag02 +S02,TCGTGGTCCTGG,Ag04 +S02,TCCGTGACTTTG,Ag05 +S03,AGAACCCCCCTT,Ag01 +S03,AAGCAACAATCT,Ag03 +S03,TCGTGGTCCTGG,Ag14 +S03,ACCTTACGGGCT,Ag06 +S04,AGAACCCCCCTT,Ag11 +S04,AGTTAAGAACAA,Ag12 +S04,TCCGTGACTTTG,Ag15 +S04,CTTTTTGCCGTT,Ag07 +S04,CATCTCTAGTCT,Ag07 +S01,TGTAGACGCATA,Ctrl1 +S02,TGTAGACGCATA,Ctrl1 +S03,TGTAGACGCATA,Ctrl1 +S04,TGTAGACGCATA,Ctrl1 diff --git a/software/test-data/fixtures/verdicts/panel_wide.csv b/software/test-data/fixtures/verdicts/panel_wide.csv new file mode 100644 index 0000000..608581e --- /dev/null +++ b/software/test-data/fixtures/verdicts/panel_wide.csv @@ -0,0 +1,21 @@ +Samples,Name,Barcode,Sequence,Channel,Residues,Type +S01,Ag01,T0100,AGAACCCCCCTT,PE,ECD protein,Target (Primary) +S01,Ag02,T0101,AGTTAAGAACAA,PE,ECD protein,Target (Primary) +S01,Ag03,T0102,AAGCAACAATCT,APC,ECD protein,Off-Target +S02,Ag11,T0100,AGAACCCCCCTT,PE,ECD protein,Target (primary) +S02,Ag02,T0101,AGTTAAGAACAA,PE,ECD protein,Target (Primary) +S02,Ag04,T0103,TCGTGGTCCTGG,APC,ECD protein,Target (Secondary) +S02,Ag05,T0104,TCCGTGACTTTG,PE Dazzle,ECD protein,Target (Secondary) +S03,Ag01,T0100,AGAACCCCCCTT,PE,ECD protein,Target (Primary) +S03,Ag03,T0102,AAGCAACAATCT,APC,ECD protein,Off-Target +S03,Ag14,T0103,TCGTGGTCCTGG,APC,ECD protein,Target (Secondary) +S03,Ag06,T0105,ACCTTACGGGCT,PE Dazzle,ECD protein,Off-target +S04,Ag11,T0100,AGAACCCCCCTT,PE,ECD protein,Target (primary) +S04,Ag12,T0101,AGTTAAGAACAA,PE,ECD protein,Target (Primary) +S04,Ag15,T0104,TCCGTGACTTTG,PE Dazzle,ECD protein,Target (Secondary) +S04,Ag07,T0106,CTTTTTGCCGTT,PE-Dazzle 5120,ECD protein,Target (Primary) +S04,Ag07,T0107,CATCTCTAGTCT,PE-Dazzle 5120,ECD protein,Target (Primary) +S01,Ctrl1,T0108,TGTAGACGCATA,APC,ECD protein,Off-Target +S02,Ctrl1,T0108,TGTAGACGCATA,APC,ECD protein,Off-Target +S03,Ctrl1,T0108,TGTAGACGCATA,APC,ECD protein,Off-Target +S04,Ctrl1,T0108,TGTAGACGCATA,APC,ECD protein,Off-Target diff --git a/software/test-data/fixtures/verdicts/panel_with_reference.csv b/software/test-data/fixtures/verdicts/panel_with_reference.csv new file mode 100644 index 0000000..e77f2df --- /dev/null +++ b/software/test-data/fixtures/verdicts/panel_with_reference.csv @@ -0,0 +1,21 @@ +Samples,Name,Sequence,Type +S01,Ag01,AGAACCCCCCTT,Target +S01,Ag02,AGTTAAGAACAA,Target +S01,Ag03,AAGCAACAATCT,Target +S02,Ag11,AGAACCCCCCTT,Target +S02,Ag02,AGTTAAGAACAA,Target +S02,Ag04,TCGTGGTCCTGG,Target +S02,Ag05,TCCGTGACTTTG,Target +S03,Ag01,AGAACCCCCCTT,Target +S03,Ag03,AAGCAACAATCT,Target +S03,Ag14,TCGTGGTCCTGG,Target +S03,Ag06,ACCTTACGGGCT,Target +S04,Ag11,AGAACCCCCCTT,Target +S04,Ag12,AGTTAAGAACAA,Target +S04,Ag15,TCCGTGACTTTG,Target +S04,Ag07,CTTTTTGCCGTT,Target +S04,Ag07,CATCTCTAGTCT,Target +S01,Ctrl1,TGTAGACGCATA,Control +S02,Ctrl1,TGTAGACGCATA,Control +S03,Ctrl1,TGTAGACGCATA,Control +S04,Ctrl1,TGTAGACGCATA,Control diff --git a/software/test-data/manual/README.md b/software/test-data/manual/README.md index 0dd0500..d2e02c8 100644 --- a/software/test-data/manual/README.md +++ b/software/test-data/manual/README.md @@ -375,3 +375,262 @@ real BEAM-T run) — a non-regenerable fallback pool used only if the full list that the `realistic` calibration targets. - `design-and-schemas.md` — design rationale, the join-spine axis contract, per-arm file schemas, and the biology/coherence model. + +## Driving A Whole Run From A Real Panel File + +`reshape_panel.py` (below) makes a generated run *look* like a real panel. `--real-panel` does the other +half: it takes a real panel file and generates the run **from** it, at cohort scale, so the panel's own +samples, antigen names and feature barcodes are the ones in the reads. + +```bash +python3 generate.py --real-panel /path/to/panel.csv --cells-per-sample 6000 +``` + +Nothing about the panel is baked into this repository. The file is read from wherever it lives, copied +into the run directory as `panel.csv`, and **everything under `runs/` is gitignored whatever its +extension** — so a confidential panel can drive a run without any of it being committed. The one thing to +keep out of a commit is a panel's own vocabulary: sample names, antigen names, catalogue ids, sequences +and channel values belong in the file, never in a generator or a doc. + +The panel is expected in the **wide** shape — sample, name, catalogue id, barcode, channel, a constant, +role — and four of those columns are read. Name them if they are spelled differently: + +| column | flag | default | +|---|---|---| +| sample | `--panel-sample-col` | `Samples` | +| antigen name | `--panel-name-col` | `Name` | +| barcode sequence | `--panel-seq-col` | `Sequence` | +| role | `--panel-role-col` | `Type` | + +Role values are matched on their leading word, so `Target (Primary)` and `Target (Secondary)` both read +as on-target while staying two distinct values. `--target-roles` / `--offtarget-roles` override the words. + +### What this path does that no preset does + +**Per-sample panels.** A wide panel says, per row, which sample offers which antigen. Reads are generated +per sample against that sample's own panel, so a barcode sequence reused across samples under different +antigen names is reused *in the data*, not only in the CSV. `sampleColumn` is therefore **required** in +the block — without it the duplicate-barcode guard fires, correctly. + +**No declared comparator.** A real role column names what a member is to the question and carries no value +meaning *negative control*, so this path plants none. Background lives on the panel's own members, and the +off-target members are what a reading is compared against. See the comparator note in the generated +`RUN.md`. + +**A reading-quality mix.** Every cell is planted at one of eight named tiers, and each tier is calibrated +against the block's own rule — count floor 4, specificity cutoff 75, reference thin line 2, comparator = +the **max** over the off-target members (≈5 UMIs at this background, which puts the line at ≈120 antigen +UMIs). + +| tier | share | dominant UMIs | reads as | +|---|---|---|---| +| `strong` | 28% | 500–1400 | bound, with margin | +| `good` | 20% | 150–500 | bound | +| `medium` | 16% | 60–200 | **straddles the line** — about half bound, half not | +| `weak` | 11% | 15–45 | not bound; a real reading, clear of the floor | +| `noise` | 9% | 1–4 everywhere | not bound (floored), or unreliable where the comparator is too thin | +| `crossreactive` | 6% | two targets co-dominant | bound on two identities at once | +| `offtarget` | 6% | an off-target dominates | nothing bound, comparator high — no lead | +| `gated` | 4% | comparator 400–1200 | set aside by the admissibility gate | + +Per-cell ground truth is `truth/expected-readings.tsv`: tier, dominant member, its UMI count, its role, +and the state it should reach. A tier a sample's panel cannot support — `crossreactive` where the sample +declares one target, `offtarget` where it declares none — degrades to `good`, and the truth records the +tier the cell actually got. + +**Library quality is the other axis.** `--library-quality mixed` (default) deals `clean / good / fair / +poor` across the samples; `uniform` makes every library clean; `spread` forces an OK/WARN/ALERT span. +`truth/library-quality.tsv` carries each sample's matched and panel-assigned fractions and the Quality tag +it should show. Reading tiers are about a cell's binding signal; library tiers are about whether the +library read out at all. They are independent. + +**Real read geometry.** These are TotalSeq-C style antigen-capture barcodes, whose 15 nt barcode sits +behind a 10 nt lead on Read 2, so R2 = 10 bp lead-in + 15 bp feature + tail and the block wants the +**Custom feature-barcode kit** preset with **Read 2 offset = 10** (not the BEAM-Core preset, which assumes +0). `--offset 0` generates the offset-0 geometry instead. Cell barcodes are real `737K-august-2016` +members drawn per sample *independently*, so samples share some barcodes — which is what real GEM wells do, +and what makes `(sampleId, cellId)` the load-bearing key. ~18% of reads are ambient, on off-cell barcodes; +~1.5% carry a 1 bp feature-barcode error for tag refinement to correct. + +**A coherent V(D)J arm.** One AIRR single-cell TSV per sample (`cell_id, locus, v_call, j_call, c_call, +junction, junction_aa, productive, duplicate_count`), paired IGH + IGK, every cell carrying the SAME bare +16 nt barcode the antigen FASTQ carries — so the `[sampleId, cellId]` join is 100% by construction, not by +luck. Clonotypes are built ON TOP of the antigen truth: a clonotype's cells bind the same antigen, which +is what makes the per-clonotype verdict mean anything. `truth/truth_clonotypes.csv` carries every clone's +target antigen, size, V/J/C genes and CDR3s. + +**How much clonal expansion is a measured question, and the two answers differ by 30x.** +`--clonal-profile` picks the shape; `--clonal-mean-size` and `--clonal-singleton-cell-frac` tune it. + +| source | cells per clonotype | what it is | +|---|---|---| +| real in-vivo BEAM libraries | **1.05** | 4,549 IGHeavy clonotypes over 4,773 paired cells, and 3,707 over 3,716. Essentially all singletons. | +| public 10x BEAM-T runs | 32–92 | both hold **deliberately expanded spike-ins** — one is 40% expanded anti-CMV T cells | +| public 10x BEAM-Ab run | 15 | a **transgenic monoclonal** control: one clone is 90% of the cells | + +The `immunized` profile argues from first principles that a sorted, immunised repertoire must be +expansion-heavy. That argument is sound and that real data still does not follow it, so the profile is +no longer treated as the one true shape. `--regime shallow` sets the clonal knobs to the measured +measured 1.05 (see **Regimes** below); `--regime deep` keeps the expanded shape. + +At 1.05 cells per clonotype a verdict rests on **one cell** almost everywhere, so the per-clonotype +agreement rules — minimum voting cells, minimum agreement, a split vote — go unexercised. That is a fact +about the data, not a gap in the bed: that real data does not exercise them either. Raise +`--clonal-mean-size` when you want to test those rules on purpose. + +The V(D)J arm is the cheap half of a run — a few MB of TSV against hundreds of MB of FASTQ — and the half +worth iterating on, so `--arm vdj` reshapes the repertoire over the antigen arm already on disk in +seconds, without regenerating a read: + +```bash +python3 generate.py --real-panel --arm vdj --clonal-mean-size 25 +``` + +**Sample metadata.** `samples-metadata.tsv`, keyed by `Sample` so it joins to the sampleId the three arms +share. `Donor` and `Condition` are invented — the panel names its samples and says nothing else about them +— and exist to give downstream grouping something to split on. The other two are read from the panel and +are the reason the file is worth having here: `LibraryQuality` is the tier each library was degraded to, so +grouping the QC report on it should track the Quality tag; `PanelMembers` / `PanelTargets` vary per sample +in a per-sample panel, and that variation is what makes *never asked* reachable. + +### Regimes — which library the bed stands in for + +`--regime` picks between two MEASURED calibrations. They disagree by more than an order of magnitude, and +the difference is not a detail: it decides whether the bed tests the regime real in-vivo data occupies. + +| | `deep` (default) | `shallow` | +|---|---|---| +| stands in for | the public 10x BEAM runs | **real in-vivo BEAM libraries** | +| source | 10x published output files | a production deployment, measured 2026-08-21 | +| reads per distinct UMI | ~33 (97% saturation) | **2.7–5.8** | +| UMIs per cell | median 200 | median **7** across barcodes clearing the floor | +| dominance | median 0.995, near-mono | median **0.44** | +| barcodes reported as cells | called cells only | the **raw universe** — 1.37M, median 1 UMI | +| antigen aggregates | none | **5 barcodes holding 59%** of the library | +| cells per clonotype | 25 | **1.05** | +| readings coming back bound | most | **1–3%** | + +`deep` reproduces every run made before 2026-08-21 byte for byte, and the test suite asserts that. Use it +when you need the block to reach a confident answer — a clean binder, a straddling line, a cross-reactive +call. Use `shallow` when the question is what real in-vivo data actually shows. + +```bash +python3 generate.py --real-panel --regime shallow +``` + +**Three things `shallow` does that change how the block must be configured.** + +1. **Leave the cell whitelist EMPTY.** The observed live configuration sets none, so the block consumes the raw + barcode universe and reports it as *cells detected*. A shallow run plants ~100x as many ambient + barcodes as real cells, so `cells detected` reads in the hundreds of thousands and `median UMIs / cell` + reads **1** — the numbers the scientists running it see. Setting a whitelist collapses the universe and hides + the effect. `--ambient-barcode-ratio 0` turns it off. +2. **Antigen aggregates are present and unfiltered.** Five barcodes per library hold ~59% of its UMIs, the + largest ~18% alone. Cell Ranger removes this population before cell calling; this block does not. + Recorded in `truth/aggregates.tsv` so anything they distort can be traced to them. +3. **Expect 1–3% of readings to come back bound.** That is the band that pipeline produces (1.4% and + 2.9%). The validator asserts it. A shallow run showing a clean majority of confident binders has lost + the regime, and the validator will say so. + +Because the line is out of reach for most cells at this depth, the shallow validator does not assert that +each tier reaches its promised state. It asserts the two things that still carry meaning: the bound share +sits in the measured 1–3% band, and the bound rate falls **monotonically** from `strong` to `noise`. That +catches an inverted score or a broken comparator without pretending to know where the line sits. + +Every regime-owned value is also a flag, so a regime is a starting point rather than a straitjacket: +`--reads-per-umi`, `--ambient-barcode-ratio`, `--aggregates`, `--aggregate-umi-share`, `--unpaired-frac`, +`--clonal-mean-size`, `--clonal-singleton-cell-frac`. + +### The narrow panel shape + +`--panel-shape` handles both shapes seen in production use, and defaults to auto-detecting from the +header. The **wide** shape declares a role column. The **narrow** shape — `Sample,Sequence,Antigen` — +declares none, and role is inferred from the antigen NAME instead. + +Narrow is not the exception: it is what the production in-vivo project uploads. Its antigen names +carry the role in free text, and the comparator is chosen by naming one member, which `--control-feature` +does (mirroring the block's own dropdown). A generator that assumes role lives in a column models the +wrong half of real production work. + +Role inference is deliberately conservative: a member matching none of the generic off-target words comes +back on-target, because mistaking a target for a comparator silently moves the line every reading in that +sample is judged against. + +### Verified before any backend run + +`--real-panel` validates itself. Beyond geometry and per-sample barcode checks it does two things worth +knowing about: + +- **Re-derives the planted UMI counts straight from the FASTQ pair.** Over-recovery is a hard failure (a + member reading more UMIs than were planted means the panel's members are not far enough apart); + under-recovery is bounded by the 1.5% barcode-error rate. +- **Simulates every verdict.** The block's own reading rule — floor, comparator as the max over reference + tags, `specificity = (1 − I₀.₉₂₅(count+1, ref+3))·100`, cutoff — re-implemented in the standard library + (checked against scipy to 12 digits) and run over the truth tables, then checked against what each tier + promises. A tier whose magnitudes drift out of its verdict fails here rather than surfacing later as a + puzzling run. + +``` +tier cells bound not bound unreliable bound on 2+ +strong 485 96% 0% 4% 0% +good 389 96% 0% 4% 0% +medium 288 45% 51% 4% 3% +weak 188 0% 96% 4% 0% +noise 128 0% 70% 30% 0% +crossreactive 68 94% 0% 6% 94% +offtarget 116 0% 100% 0% 0% +gated 73 0% 100% 0% 0% +whole grid: bound 16%, not bound 79%, unreliable 5% +``` + +Re-check an existing run with `--real-panel --validate-only`. + +## The Two Shapes A Real Panel File Arrives In + +`generate.py` emits one panel shape: `tag,feature,Type,Species,Class`, one panel for every sample, with +the control carrying its own `Decoy` role. Two other shapes were observed in use in production at the +same time, on two of its projects, and neither looks like that. `reshape_panel.py` rewrites a generated +run's `tags.csv` into both, **keeping every barcode unchanged** so either can be uploaded against the +same FASTQs: + +```bash +python3 generate.py tiny --arm antigen --panel-size 12 --offtarget-count 3 +python3 generate.py tiny --arm vdj --panel-size 12 +python3 reshape_panel.py runs/tiny +``` + +| File | Shape | What it exercises | +|---|---|---| +| `tags_narrow.csv` | `Sample,Sequence,Antigen` | No role column at all, so nothing can be named as the comparator and the panel's own readings serve. The control is an ordinary row nothing marks. | +| `tags_wide.csv` | `Samples,Name,Barcode,Sequence,Channel,Residues,Type` | A role column that declares target vs off-target and carries **no** comparator value; a catalogue id 1:1 with the sequence; a channel column holding four values that are three channels; a constant column; and case-variant role values. | + +Both rename a barcode between samples (`--rename`, default 2), so the same sequence carries a different +antigen name in different samples. Under the per-tag grouping the identity is the barcode, so those +identities lose their label and show a raw 15-mer. `--drop-from-later N` makes a later sample declare +fewer tags, which is what makes *never asked* reachable. + +**A panel below 8 tags cannot serve as its own comparator**, so generate at least that many +(`--panel-size 12` gives 12 + 1 control). The script warns if you are under. + +### ⚠️ Set the count floor to 1 for these two shapes + +This bed plants background at 1–3 UMIs per barcode — **253 of 432 readings in a `tiny --panel-size 12` +run sit below the shipped count floor of 4**. Neither shape declares a comparator, so the panel's own +readings have to serve, and with the floor at 4 that background is zeroed and the panel median collapses +to 0. + +**The failure this causes has inverted, and the new one is more dangerous.** It used to be caught: a +median of 0 fell below the reference thin line of 2, and every cell carrying signal read *impossible to +compare*, which looks broken and is therefore investigated. `count-becomes-a-state` deleted that branch, +so a median of 0 is now a real comparison — and any count that clears the floor, read against a +reference of 0, scores near 100 and reads **bound**. The run no longer looks broken. It looks +spectacularly successful. + +So set **Advanced → count floor = 1** in the block when uploading either shape. The two numbers are each +defensible and simply do not compose: the bed's background is calibrated to a real 5k BEAM-T library, +and the floor of 4 comes from the antibody-side lineage. A declared comparator would sidestep it, which +is exactly what neither of these shapes can supply. + +The old `--floor 4` / `--floor 1` tallies recorded here have been removed rather than updated: they were +measured against the thin-reference behaviour and no longer describe this bed. Re-measure them before +citing any number for this shape. diff --git a/software/test-data/manual/design-and-schemas.md b/software/test-data/manual/design-and-schemas.md deleted file mode 100644 index 7c12d56..0000000 --- a/software/test-data/manual/design-and-schemas.md +++ /dev/null @@ -1,190 +0,0 @@ -# Multiomics synthetic data — design, schemas, and the join contract - -Background for the manual run in `README.md`: the experiment modeled, the pipeline, the axis contract the -data must satisfy, per-arm file schemas (verified against block code), the coherence model, and the -viability tests. (Consolidates the former `multiomics-manual-test-data-report.md` scoping report + -`multiomics-generator-spec.md` build spec.) - ---- - -## 1. The experiment: BEAM-Ab - -One GEM emulsion produces **three co-registered 10x 5′ v2 libraries from the same cells**, all sharing -**one 16 nt cell barcode** (from the 5′ gel-bead list `737K-august-2016`) — the *only* multiomic linking key. - -| Library | R1 | R2 | Purpose | -|---|---|---|---| -| Gene Expression (GEX) | 16 nt CB + 10 nt UMI | cDNA (5′) | transcriptome / cell typing | -| BCR V(D)J | 16 nt CB + 10 nt UMI | V(D)J contig | paired IGH + IGK/IGL → clonotype | -| Antigen Capture (BEAM) | 16 nt CB + 10 nt UMI | 15 nt antigen barcode @ pos 0 + adapter | per-cell antigen binding | - -UMIs are independent per library; the **cell barcode string** is the shared key. Specificity score -(Cell Ranger BEAM, which `feature-integration` reproduces): `(1 − beta.cdf(0.925, antigenUMI+1, controlUMI+3)) × 100`. - -**Why the import path (not raw FASTQ + Cell Ranger + MiXCR):** only the antigen arm has no import entry -point, so only it needs synthetic FASTQ. GEX (`import-sc-rnaseq-data`) accepts a count matrix; VDJ -(`import-vdj-data`) accepts an AIRR contig table and emits the clonotype key + linker directly. Three -lightweight assets off one shared barcode population — no aligners, no references. - ---- - -## 2. Pipeline - -``` - samples-and-data - ┌─────────────────────────┼──────────────────────────────┐ - GEX arm VDJ arm Antigen arm - import-sc-rnaseq-data import-vdj-data feature-integration - │ │ │ - rna-seq/countMatrix anchor: vdj/uniqueCellCount feature/umiCount - [cellId, geneId] linker: sc/cellLinker [sampleId, cellId, featureId] - │ [sampleId, cellId, scClonotypeKey] │ - cell-type-annotation │ │ - → rna-seq/cellType │ │ - └───────────────┬───────┴─────────────────────────────────┘ - ▼ - vdj-multiomic-integration - anchor = VDJ sc-clonotype dataset; REQUIRED: feature umiCount + cellLinker; - OPTIONAL: GEX countMatrix, cellType. Inner-join on [sampleId, cellId], group by scClonotypeKey. - ▼ - antibody-tcr-lead-selection → top-N antibody leads -``` - ---- - -## 3. The canonical cell barcode (the one rule) - -The convergence join is a **silent inner-join on `[sampleId, cellId]`** — any barcode mismatch drops -cells with no error. The canonical `cellId` = the **bare 16 nt** barcode. Verified per-arm normalization: - -| Arm | barcode handling | verified at | -|---|---|---| -| Antigen (`feature-integration`) | bare 16 nt from R1; de-novo corrected (error-free input → verbatim) | tag-pattern / mitool CELL | -| VDJ (`import-vdj-data`, `airr-sc`) | `cell_id` verbatim (`cellKeyMode:"direct"`) | `formats.lib.tengo:126-146` | -| GEX (`import-sc-rnaseq-data`) | strips `-\d+$` suffix → bare 16 nt | `clean_barcode_suffix` | - -Synthetic barcodes are random 16-mers (not real `737K` members), so keep them **error-free** and leave -cell-barcode whitelist correction **off** in the run (a whitelist would drop them all). The de-novo-error -scenario is for *standalone* `feature-integration` testing only — it would split cells across arms. - ---- - -## 4. The join-spine — axes that must align (byte-identical name + domain) - -| Axis / column | valueType | Key annotations | Produced by | Consumed by | -|---|---|---|---|---| -| `pl7.app/sampleId` | String | — | samples-and-data | all | -| `pl7.app/sc/cellId` | String | `parents=[sampleId]`, no domain | all three arms | the linker; **the multiomic key** | -| `pl7.app/vdj/scClonotypeKey` | String | domain: receptor/structure/runId | import-vdj-data | integration anchor + outputs; lead-selection | -| `pl7.app/vdj/uniqueCellCount` | Int/Long | **`isAnchor:"true"`**, `isAbundance` | import-vdj-data | integration `datasetOptions` anchor | -| `pl7.app/sc/cellLinker` | Int | **`isLinkerColumn:"true"`**, axes `[sampleId, cellId, scClonotypeKey]` | import-vdj-data | integration (REQUIRED linker) | -| `pl7.app/feature/umiCount` | Int | `isAbundance` | feature-integration | integration (REQUIRED feature) | -| `pl7.app/feature/featureId` | String | — | feature-integration | integration feature axis | -| `pl7.app/rna-seq/countMatrix` | Double | axes `[sampleId, cellId, geneId]` (geneId domain `{species}`) | import-sc-rnaseq-data | integration (OPTIONAL GEX) | -| `pl7.app/rna-seq/cellType` | String | axes `[sampleId, cellId]` | cell-type-annotation | integration (OPTIONAL annotation) | - -Integration mechanism: materialize `cellLinker` → `linker.csv [sampleId, cellId, scClonotypeKey]`; write -each per-cell input to its own CSV; inner-join each to the linker on `[sampleId, cellId]`; group by -`scClonotypeKey`. Outputs reuse `scClonotypeKey` verbatim, joining back onto the VDJ clonotype table. - ---- - -## 5. Per-arm file schemas (verified against block code) - -All three upload through **one Samples & Data block** as datasets keyed by the same `sampleId`(s). - -### 5.1 Antigen — `feature-integration` (paired FASTQ) -- **R1** (`*_R1.fastq.gz`): `[16 nt cell barcode][10 nt UMI]` = 26 nt. -- **R2** (`*_R2.fastq.gz`): `[15 nt antigen barcode @ pos 0][tail]`. -- **Panel CSV** (`tag,feature`): antigens + `negative_control`; barcodes pairwise Hamming ≥ 3. -- Error-free cell barcodes in the multiomics dataset. - -### 5.2 VDJ — `import-vdj-data`, format `airr-sc` (AIRR rearrangement TSV, one row per contig) -Columns present: `cell_id`, `locus`, `v_call`, `j_call`, `c_call`, `junction`, `junction_aa`, -`productive`, `duplicate_count`. -- `cell_id` = bare 16 nt (used verbatim). `junction` = CDR3 nt (ACGT, len %3==0 for productive). -- `v_call`/`j_call` = real IMGT gene names. `duplicate_count` (Int ≥1) = UMI support → primary abundance, - gets `isAnchor:"true"` (`infer-columns-airr.lib.tengo:141`). -- Each cell has ≥1 IGH + ≥1 IGK/IGL row. Clonotype key = CDR3-nt + V + J (+ C). Cells with identical - paired rows → same `scClonotypeKey` (lead clones); unique rows → singletons. -- Consumer settings: format = "AIRR single cell", chains = IG Heavy + IG Light. - -### 5.3 GEX — `import-sc-rnaseq-data` (CSV, genes-in-rows) -- First column = **gene IDs** (real Ensembl `ENSG…`); header = **cell barcodes**; body = non-neg integer counts. -- `detect_orientation` → genes-in-rows on an all-numeric body; `check_format` passes on gene-like first col. -- Include real marker genes per class (B: MS4A1/CD79A; plasmablast: MZB1/XBP1/PRDM1) so - `cell-type-annotation` is meaningful. No mapping file needed — species/format inferred. - ---- - -## 6. Coherence model — one cell, three consistent modalities - -Each synthetic cell gets a latent identity; all three arms derive from it, so the downstream result is -assertable. - -| Cell class | VDJ | Antigen (UMIs) | GEX program | -|---|---|---|---| -| **Lead B cells** (few dominant clones) | one of clones L1..L4 (paired IGH+IGK/L) | high on-target, low control → specificity ~100 | plasmablast: MZB1/XBP1/PRDM1 high | -| **Background B cells** | many singleton clones | antigen ≈ control (low) | naive-B: MS4A1/CD79A/TCL1A | -| **Non-B contaminants** (optional) | absent from VDJ | none / control-only | T/myeloid | - -Each lead clone binds exactly one antigen. **Every lead-clone cell appears in all three arms** (survives -the inner-join). Background/contaminant cells may be partial (realistic per-arm dropout). Seeded RNG. - ---- - -## 7. Viability tests (`validate_multiomics.py`, stdlib-only) - -- **Barcode alignment** (the test): the `cellId` set each arm will produce must overlap as intended; every - lead-clone cell ∈ all three sets. -- **Per-arm schema/geometry:** antigen R1=26/R2≥15, on-panel; VDJ AIRR header + heavy/light pairing + - junction validity; GEX orientation + Ensembl IDs + non-neg + no all-zero row/col + markers elevated. -- **Join simulation** (strongest offline proof): build the linker from VDJ pairing, join the antigen - per-cell UMIs on `cellId`, group by clonotype, apply the dominant-antigen + specificity rules; assert - the per-clonotype table is non-empty and L1..L4 show their intended antigen + high specificity. -- Realistic profile: **38/38 PASS**. - ---- - -## 8. Verification status (source review, 2026-07-01) - -A source-level review of all seven blocks (not just the docs) confirmed the join spine ties together. -Resolved items (previously "assumed"): - -1. **`import-vdj-data` per-clonotype anchor carries `isAnchor` — VERIFIED.** The SC anchor - `pl7.app/vdj/uniqueCellCount` (cell count) is freshly authored with `isAnchor:"true"` at the - per-clonotype stage (`process-single-cell.tpl.tengo:191`), so aggregation can't strip it, and it - matches the integration `datasetOptions` predicate (`[sampleId, scClonotypeKey]` + isAnchor). -2. **Samples & Data `Xsv` → import dropdowns — VERIFIED.** Xsv publishes `pl7.app/sequencing/data`/File - with `pl7.app/fileExtension` (csv/tsv); import-vdj matches tsv, import-sc matches csv/tsv. -3. **cellId join key — VERIFIED byte-identical** (`pl7.app/sc/cellId`, String, no domain) across FI, - the VDJ linker, the GEX countMatrix, and cellType. The integration join is a Python inner-join on - `[sampleId, cellId]` grouped by `scClonotypeKey`; cellType's cell axis is inherited from its input. - -Remaining risks / live checks: - -- **Build from the right source (CRITICAL):** `blocks/feature-integration` (stub, no outputs) and - `blocks/vdj-multiomic-integration` (README-only) are NOT the real code — build both from their - MILAB-6496 worktrees, else the convergence feature dropdown is empty and it can't run. -- **Silent inner-join on cellId:** a barcode mismatch drops cells with no error → keep barcodes - byte-identical bare-16nt and match the cell-whitelist setting to the profile. -- **Single-receptor only:** import-vdj emits one linker/anchor per receptor (receptor domain); a - TCR+BCR dataset publishes multiple `pl7.app/sc/cellLinker` columns and the integration's `addSingle` - expects exactly one. BEAM-Ab (BCR/IG) is safe. -- **lead-selection ranking (live check):** the integration's per-clonotype outputs (`restrictionIndex`, - `breadth`, `dominantFeature`, keyed on `scClonotypeKey`, no sample axis) are spec-compatible with - lead-selection's ranking discovery, but confirm they appear in its "Rank by" dropdown on a live run. -- **Backend assets:** import-sc needs `gene-annotations-assets:homo-sapiens`; cell-type needs the - CellTypist model assets — cached automatically online, required for a strictly-offline backend. - -**Tiering:** Tier 0 = VDJ + antigen → integration (feature + linker only). Tier 1 adds GEX + annotation -(the full run in `README.md`). Tier 2 = wider ecosystem on the same cells. - ---- - -## Key code references - -- Convergence: `vdj-multiomic-integration/.../model/src/index.ts`, `.../workflow/src/{main,aggregate}.tpl.tengo`, `.../software/aggregate-clonotypes/`. -- VDJ import: `blocks/import-vdj-data/workflow/src/{process-single-cell.tpl,infer-columns-airr.lib,formats.lib}.tengo`. -- GEX import: `blocks/import-sc-rnaseq-data/workflow/src/libs/pf-counts-conv.lib.tengo`. -- Antigen: this block (`feature-integration`) + `docs/dormant-features/cell-whitelist-correction-plan.md`. diff --git a/software/test-data/manual/generate.py b/software/test-data/manual/generate.py index 81a4d06..8f2ddf9 100644 --- a/software/test-data/manual/generate.py +++ b/software/test-data/manual/generate.py @@ -2,7 +2,7 @@ """Single entry point for the manual BEAM-Ab test-data generators. Builds a full, colocated multiomic run (antigen + VDJ + GEX arms) into one folder, or an antigen-only -behavioural scenario. Standard-library only, deterministic (seeded). +behavioural scenario. Standard-library only, deterministic and seeded. # A full multiomic run (all three arms + shared panel/metadata + truth), then validate it offline: python3 generate.py realistic # 24 donors x 2000 cells x 15-antigen panel + control @@ -19,7 +19,7 @@ Output layout (everything under runs/ is gitignored): - runs// a full multiomic run — one folder, no jumping between arms + runs// a full multiomic run -- one folder, no jumping between arms antigen/ donorNN_R{1,2}.fastq.gz vdj/ donorNN.tsv gex/ donorNN.csv @@ -35,7 +35,7 @@ HERE = os.path.dirname(os.path.abspath(__file__)) sys.path.insert(0, HERE) -from lib import annotations, antigen, beam_exact, gex, panelswap, validate, vdj # noqa: E402 +from lib import annotations, antigen, beam_exact, gex, panelswap, realpanel, validate, vdj # noqa: E402 from lib import panel as panel_mod # noqa: E402 from lib.antigen import AntigenConfig # noqa: E402 @@ -50,7 +50,7 @@ "whitelist737k": dict(samples=24, cells=2000, panel_size=15, barcode_source="whitelist737k"), } -# Antigen-only scenarios. errors/offpanel/multilane/control run through antigen.build; the rest have +# Antigen-only scenarios. errors/offpanel/multilane/control run through antigen.build. The rest have # their own generators. Default scenario scale = small (tiny), overridable with --samples/etc. ANTIGEN_SCENARIOS = ["errors", "offpanel", "multilane", "control"] SPECIAL_SCENARIOS = ["degraded", "panel-swap", "multisample", "libraseq"] @@ -215,8 +215,8 @@ def main(): ap.add_argument( "--heavy-only", action="store_true", - help="emit a HEAVY-CHAIN-ONLY (IGH, no IGK) VDJ arm — the customer's VHH single-domain " - "antibody — so the heavy-only end-to-end path is reproducible; applies to the vdj and all " + help="emit a HEAVY-CHAIN-ONLY (IGH, no IGK) VDJ arm — the shape a VHH single-domain antibody " + "library produces — so the heavy-only end-to-end path is reproducible; applies to the vdj and all " "arms. Each cell keeps its shared bare-16nt cell_id. Off by default (paired IGH+IGK)", ) ap.add_argument( @@ -230,17 +230,222 @@ def main(): ap.add_argument( "--messy-metadata", action="store_true", - help="inject the real customer panel's inconsistent casing/whitespace into the EMITTED panel " + help="inject the inconsistent casing/whitespace real panels carry into the EMITTED panel " "metadata: the Type column carries a mixed-case off-target set (both 'Off-Target' and " - "'Off-target') and one antigen name gains a stray double space. Reproduces the B043 problem so a " + "'Off-target') and one antigen name gains a stray double space. Reproduces that problem so a " "mixed-case panel is available to exercise the block's case-sensitive off-target matching — the " "user must select each casing present (whitespace is trimmed, casing is not folded). Messy LABELS " "only — the barcode joins and truth tables stay coherent. Applies to the full-run panel; off by " "default (byte-identical to prior runs)", ) + ap.add_argument( + "--real-panel", + metavar="CSV", + help="build a cohort-scale run against a REAL, externally-supplied wide panel CSV " + "(sample / name / barcode / role columns) instead of a synthesized panel. The panel's own " + "samples, antigen names and feature barcodes drive the run; every cell is planted at one of " + "eight named reading tiers (strong -> noise) so good, medium and bad readings are all present " + "in stated proportions. Writes into runs/real-panel/ by default. The panel file is COPIED into " + "the run, and everything under runs/ is gitignored — a confidential panel can drive a run " + "without any of it entering the repository", + ) + ap.add_argument("--panel-sample-col", default=None, help="real-panel: the sample column (default: Samples)") + ap.add_argument("--panel-name-col", default=None, help="real-panel: the antigen-name column (default: Name)") + ap.add_argument("--panel-seq-col", default=None, help="real-panel: the barcode-sequence column (default: Sequence)") + ap.add_argument("--panel-role-col", default=None, help="real-panel: the role column (default: Type)") + ap.add_argument( + "--target-roles", + default=",".join(realpanel.DEFAULT_TARGET_ROLES), + help="real-panel: comma-separated role prefixes meaning on-target (default: target)", + ) + ap.add_argument( + "--offtarget-roles", + default=",".join(realpanel.DEFAULT_OFFTARGET_ROLES), + help="real-panel: comma-separated role prefixes meaning off-target (default: off-target,offtarget,off target)", + ) + ap.add_argument( + "--offset", + type=int, + default=10, + help="real-panel: bp of lead-in before the feature barcode in R2. 10 = the real BEAM geometry " + "(default); 0 = feature at position 0", + ) + ap.add_argument( + "--library-quality", + default="mixed", + choices=list(realpanel.QUALITY_PROFILES), + help="real-panel: how per-sample LIBRARY quality is dealt out — uniform (all clean), mixed " + "(clean/good/fair/poor, default) or spread (forces an OK/WARN/ALERT span)", + ) + ap.add_argument( + "--clonal-profile", + default=None, + choices=["immunized", "lead"], + help="real-panel: VDJ clone-size distribution — immunized (default: the shape an immunized, " + "antigen-sorted repertoire has — an expanded head holding most of the CELLS plus a singleton " + "tail) or lead (one clone holding 60%% of an antigen's cells, the small-fixture shape)", + ) + ap.add_argument( + "--clonal-mean-size", + type=float, + default=None, + help="real-panel: mean cells per clone in the EXPANDED compartment (default 25). Lower it for a " + "more diverse, less expanded repertoire; raise it for a few very large lead clones", + ) + ap.add_argument( + "--clonal-singleton-cell-frac", + type=float, + default=None, + help="real-panel: share of a group's CELLS left as one-cell clonotypes (default 0.10). These stay " + "a large share of clonotypes and a small share of cells, which is the real shape; raise it toward " + "an unsorted/naive repertoire, lower it for a heavily sorted one", + ) + ap.add_argument( + "--regime", + default="deep", + choices=list(realpanel.REGIMES), + help="real-panel: which MEASURED calibration to generate against. `deep` (default) is the " + "public 10x BEAM shape — ~33 reads per UMI, ~200 antigen UMIs per called cell, near-mono " + "dominance; it reproduces every run made before 2026-08-21 byte for byte. `shallow` is the " + "shape real in-vivo BEAM libraries measure — 2.7-5.8 reads per UMI, a median of 7 UMIs " + "across barcodes clearing the floor, dominance ~0.44, the raw barcode universe instead of " + "called cells, and unfiltered antigen aggregates. Every flag below overrides the regime it " + "came from", + ) + ap.add_argument( + "--ambient-barcode-ratio", + type=float, + default=None, + help="real-panel: size the raw BARCODE UNIVERSE at this multiple of the real cell count " + "(shallow default 100). The block applies no cell calling and the live configuration sets no " + "whitelist, so what it reports as `cells detected` is this universe — 1.37M barcodes with a " + "median of ONE UMI each. 0 keeps the old behaviour, where the ambient barcode count follows " + "from the ambient read share alone", + ) + ap.add_argument( + "--aggregates", + type=int, + default=None, + help="real-panel: number of ANTIGEN-AGGREGATE barcodes to plant (shallow default 5). Protein " + "clumps produce droplets with enormous UMI counts; Cell Ranger removes them before cell " + "calling and this block does not. 0 plants none", + ) + ap.add_argument( + "--aggregate-umi-share", + type=float, + default=None, + help="real-panel: share of the finished library's UMIs the aggregates hold (shallow default " + "0.59, measured). At that share aggregates outnumber real signal UMIs, which is why the " + "measured per-cell depth is starved despite large libraries", + ) + ap.add_argument( + "--reads-per-umi", + type=float, + default=None, + dest="dup_mean", + help="real-panel: mean reads per distinct UMI. Drives FASTQ size and nothing the block " + "concludes, since the reading rule counts UMIs. deep ~1.3, shallow 4.0 (measured 2.7-5.8)", + ) + ap.add_argument( + "--unpaired-frac", + type=float, + default=None, + help="real-panel: share of cells emitting the HEAVY chain only (shallow default 0.35). In the " + "measured libraries the clonotypes dropped for want of a pair outnumbered the paired ones", + ) + ap.add_argument( + "--baseline-tag", + default=None, + help="real-panel: name ONE tag (antigen name or 15 bp sequence) as the baseline the verdict " + "simulation reads against. The block refuses a panel declaring several, and its panel rung needs " + "at least 25 tags, so a small per-sample panel can otherwise reach no comparator at all and " + "every reading comes back unreliable. Cells in samples that do not offer the named tag still " + "have no comparator — the baseline is global by tag, the panel is per sample", + ) + ap.add_argument( + "--panel-shape", + default="auto", + choices=["auto", "wide", "narrow"], + help="real-panel: wide declares a role column; narrow declares none (sample / antigen / " + "sequence) and role is inferred from the antigen NAME. Both shapes are live in production " + "on different projects. Default auto-detects from the header", + ) + ap.add_argument( + "--control-feature", + default=None, + help="real-panel: name one member as the comparator, whatever the panel says. Mirrors the " + "block, where a user picks a control from a dropdown of antigen names — the only route a " + "narrow panel has to a declared comparator", + ) + ap.add_argument( + "--barcode-source", + default=None, + choices=["whitelist737k", "random"], + help="real-panel: cell-barcode source (default: whitelist737k, so the 737K cell whitelist " + "setting is usable and cellIds match a real VDJ producer)", + ) ap.add_argument("--out", help="override the output directory") args = ap.parse_args() + if args.real_panel: + run_dir = args.out or os.path.join(RUNS_DIR, "real-panel") + cells = args.cells_per_sample or 6000 + columns = { + k: v + for k, v in ( + ("sample", args.panel_sample_col), + ("name", args.panel_name_col), + ("sequence", args.panel_seq_col), + ("role", args.panel_role_col), + ) + if v + } + roles = tuple(r.strip().lower() for r in args.target_roles.split(",") if r.strip()) + off_roles = tuple(r.strip().lower() for r in args.offtarget_roles.split(",") if r.strip()) + print(f"=== real panel: {args.real_panel} ({cells} cells/sample) -> {run_dir} ===") + if args.validate_only: + sys.exit(0 if realpanel.validate(run_dir, columns=columns, regime=args.regime, + baseline_tag=args.baseline_tag, target_roles=roles, + offtarget_roles=off_roles) else 1) + info = realpanel.build( + run_dir, + args.real_panel, + cells_per_sample=cells, + barcode_source=args.barcode_source or "whitelist737k", + assets_dir=ASSETS_DIR, + columns=columns, + target_roles=roles, + offtarget_roles=off_roles, + offset=args.offset, + quality_profile=args.library_quality, + arm=args.arm, + regime=args.regime, + clonal_profile=args.clonal_profile, + clonal_mean_size=args.clonal_mean_size, + clonal_singleton_cell_frac=args.clonal_singleton_cell_frac, + ambient_barcode_ratio=args.ambient_barcode_ratio, + aggregates=args.aggregates, + aggregate_umi_share=args.aggregate_umi_share, + dup_mean=args.dup_mean, + unpaired_frac=args.unpaired_frac, + panel_shape=args.panel_shape, + control_feature=args.control_feature, + ) + # A V(D)J-only rebuild leaves the reads, the panel and the tiers exactly as they were, so the + # report still describes the run. Rewriting it from a partial build would only replace its read + # count with a zero. + if args.arm == "all": + realpanel.write_run_report(run_dir, info, args.real_panel, args.library_quality, + baseline_tag=args.baseline_tag) + print(f" settings -> {os.path.join(run_dir, 'RUN.md')}") + if not args.no_validate: + sample_check = info["samples"][0] if info["samples"] else None + if not realpanel.validate(run_dir, columns=columns, sample_check=sample_check, + regime=args.regime, baseline_tag=args.baseline_tag, + target_roles=roles, offtarget_roles=off_roles): + sys.exit(1) + return + if args.beam: run_dir = args.out or os.path.join(RUNS_DIR, "beam-exact") cells = args.cells_per_sample or 150 @@ -260,7 +465,7 @@ def main(): sys.exit(0 if validate.validate(run_dir) else 1) if args.scenario: - # scenarios default to a small scale for hand inspection; --samples/etc override + # scenarios default to a small scale for hand inspection. --samples and friends override it. samples = args.samples or 2 cells = args.cells_per_sample or 80 panel_size = args.panel_size or 4 diff --git a/software/test-data/manual/lib/annotations.py b/software/test-data/manual/lib/annotations.py index 0054c2e..3e5bef0 100644 --- a/software/test-data/manual/lib/annotations.py +++ b/software/test-data/manual/lib/annotations.py @@ -6,18 +6,18 @@ replacement: the same cells, labelled directly. Source of cells: the antigen arm's ground truth (`truth/expected-consensus.tsv`, columns `sample`, -`cellId`, `planted_consensus`). Reading cell ids from there guarantees the annotation's `cell_id` -values are exactly the arm-shared bare 16-mers, so the annotation joins the other arms on +`cellId`, `planted_consensus`). Reading cell ids from there guarantees the annotation's `cell_id` values +are exactly the arm-shared bare 16-mers, so the annotation joins the other arms on `[sampleId, cellId]` with no barcode drift. -Coherence with the GEX program map (lib/gex.py PROGRAMS): a cell's `planted_consensus` biases its -cell type. A binder (planted_consensus is an antigen name) or a cross-reactive cell -> a plasma-like -type; an ambiguous / non-binder cell (planted_consensus == "ambiguous") -> a naive-B type; a fixed -share of cells is reassigned to a memory-B type so the small vocabulary has all three terms. The +Coherence with the GEX program map (lib/gex.py PROGRAMS): a cell's `planted_consensus` biases its cell +type. A binder, where planted_consensus is an antigen name, or a cross-reactive cell gets a plasma-like +type. An ambiguous or non-binder cell, where planted_consensus == "ambiguous", gets a naive-B type. A +fixed share of cells is reassigned to a memory-B type so the small vocabulary has all three terms. The integer `cluster` (0-4) is coherent with the cell type. Everything is deterministic under ANNOTATION_SEED. -Output (genes-in-rows is irrelevant here — this is a plain per-cell table): +Output -- genes-in-rows is irrelevant here, because this is a plain per-cell table: annotations/.tsv columns: cell_id cell_type cluster (one file per donor/sample) Canonical downstream axis order is [pl7.app/sampleId, pl7.app/sc/cellId] (see README). diff --git a/software/test-data/manual/lib/antigen.py b/software/test-data/manual/lib/antigen.py index 6f1850d..1039d45 100644 --- a/software/test-data/manual/lib/antigen.py +++ b/software/test-data/manual/lib/antigen.py @@ -91,13 +91,14 @@ def add_ambient(rng, panel, reads, frac=0.15): def assign_features(rng, panel, nonbinder=False, crossreactive=False): """Plant the per-feature distinct-UMI counts for one cell. Returns (per_feature, consensus_label). - One dominant antigen (high), optional ambiguous second, 0-2 ambient antigens (low), control - background. `nonbinder=True` (the control scenario) plants a TRUE non-binder: every antigen at - ~control level. `crossreactive=True` plants a co-dominant pair of two ON-TARGET antigens at - ~equal UMIs (second at 0.85-1.0x the first): neither passes the dominance threshold alone but their - on-target sum does, so the block calls the cell "cross-reactive" (not "ambiguous"). Magnitudes are - calibrated to the real 5k BEAM-T library: dominant ~600 UMIs (right-skewed, low-signal tail), - near-mono dominance (median ~1.0, p10 ~0.79), tight background (~3 UMIs/cell).""" + One dominant antigen (high), an optional ambiguous second, 0-2 ambient antigens (low), and control + background. `nonbinder=True`, the control scenario, plants a TRUE non-binder: every antigen at + ~control level. `crossreactive=True` plants a co-dominant pair of two ON-TARGET antigens at ~equal + UMIs, the second at 0.85-1.0x the first. Neither passes the dominance threshold alone but their + on-target sum does, so the block calls the cell "cross-reactive" rather than "ambiguous". + Magnitudes are calibrated to the real 5k BEAM-T library: dominant ~600 UMIs (right-skewed, with a + low-signal tail), near-mono dominance (median ~1.0, p10 ~0.79), and tight background + (~3 UMIs/cell).""" antigen_names = panel.names control = panel.control_name per_feature = {} @@ -109,9 +110,10 @@ def assign_features(rng, panel, nonbinder=False, crossreactive=False): per_feature[control] = rng.randint(1, bg_hi) return per_feature, "ambiguous" if crossreactive: - # Co-dominant on-TARGET pair: both must be Type=Target (not the control, not an Off-Target), so - # the block's dominant call excludes neither and — with two on-targets sharing the signal near - # 50/50 — lands on "cross-reactive" rather than a single dominant antigen or "ambiguous". + # Co-dominant on-TARGET pair. Both must be Type=Target, so neither the control nor an + # Off-Target, which means the block's dominant call excludes neither. With two on-targets + # sharing the signal near 50/50 it lands on "cross-reactive" rather than a single dominant + # antigen or "ambiguous". on_target = [a for a in antigen_names if panel.types.get(a) == "Target"] if len(on_target) >= 2: first, second = rng.sample(on_target, 2) @@ -198,7 +200,7 @@ def build_sample(rng, panel, sample, cells, nonbinder_frac=0.0, crossreactive_fr else: # Multi-barcode antigen (only present in a --multibarcode panel; single-barcode runs # never take this branch, so they stay byte-identical). combine="all" fires EVERY - # member barcode at ~k UMIs (AND / dual-probe); combine="sum" splits the k UMIs across + # member barcode at ~k UMIs (AND / dual-probe). Combine="sum" splits the k UMIs across # the members so the per-feature sum stays ~k. Same UMI/dup shape as the single path. mode = panel.combine.get(feat, "sum") if mode == "all": @@ -313,9 +315,9 @@ def write_fastqs(outdir, sample, reads, multilane): def _messify(value, rng): - """Return a casing/whitespace variant of a panel LABEL, mimicking the real customer panel's B043 - inconsistency (e.g. "Off-Target" -> "Off-target"). Used ONLY for emitted CSV label columns under - --messy-metadata; it never touches the barcode join keys or the truth tables, so generation stays + """Return a casing or whitespace variant of a panel LABEL, mimicking the label inconsistency a real + panel carries, such as "Off-Target" -> "Off-target". Used ONLY for emitted CSV label columns under + --messy-metadata. It never touches the barcode join keys or the truth tables, so generation stays coherent while the block-facing labels carry the mess the normalization tasks must resolve.""" if rng.random() < 0.5: # case variant: lower-case the segment after the last hyphen ("Off-Target" -> "Off-target") @@ -327,11 +329,11 @@ def _messify(value, rng): def _messy_types(panel, rng): - """Per-antigen Type overrides that GUARANTEE a mixed-case off-target set (the B043 `Off-Target` vs - `Off-target` bug): the first off-target stays canonical `Off-Target`, the second is forced to - `Off-target`, any further off-targets get a seeded `_messify` variant. Targets and the control keep - their canonical Type. Returns {name: type} for the off-targets only (callers fall back to - panel.types for everything else).""" + """Per-antigen Type overrides that GUARANTEE a mixed-case off-target set, the `Off-Target` against + `Off-target` bug. The first off-target stays canonical `Off-Target`, the second is forced to + `Off-target`, and any further off-targets get a seeded `_messify` variant. Targets and the control + keep their canonical Type. Returns {name: type} for the off-targets only, and callers fall back to + panel.types for everything else.""" offtargets = [n for n in panel.names if panel.types.get(n) == "Off-Target"] override = {} for idx, n in enumerate(offtargets): @@ -345,7 +347,7 @@ def _messy_types(panel, rng): def write_metadata(shared_dir, panel, samples, multibarcode=False, messy=False): - # --messy-metadata: inject the real customer panel's inconsistent Type casing into the EMITTED tags.csv + # --messy-metadata: inject the inconsistent Type casing real panels carry into the EMITTED tags.csv # values only (the feature-name double space is injected upstream in build_panel). A dedicated # constant-seed RNG keeps the mess deterministic without perturbing the read/truth streams (which are # already generated by the time write_metadata runs). Off by default -> byte-identical to before. @@ -356,10 +358,10 @@ def type_of(name): with open(os.path.join(shared_dir, "tags.csv"), "w", newline="") as f: w = csv.writer(f) - # tag,feature stay first (backward-compatible role mapping); Type/Species/Class mirror the real - # customer panel so downstream off-target-call and species-grouping have synthetic inputs. A + # tag,feature stay first (backward-compatible role mapping); Type/Species/Class mirror a real + # panel so downstream off-target-call and species-grouping have synthetic inputs. A # --multibarcode panel adds `combine` (between feature and Type) and emits one row PER member - # barcode; single-barcode runs keep the exact prior header + one row per feature (byte-stable). + # barcode. Single-barcode runs keep the exact prior header + one row per feature (byte-stable). if multibarcode: w.writerow(["tag", "feature", "combine", "Type", "Species", "Class"]) for name, bcs in panel.features.items(): @@ -383,7 +385,7 @@ def type_of(name): w.writerow(["id", "name", "read", "pattern", "sequence", "feature_type"]) for name, bcs in panel.features.items(): for j, bc in enumerate(bcs): - # per-member id `_` when a feature has >1 barcode; bare feature name otherwise + # per-member id `_` when a feature has >1 barcode. Bare feature name otherwise # (so single-barcode feature_reference.csv is byte-identical to before). bc_id = f"{name}_{j + 1}" if len(bcs) > 1 else name w.writerow([bc_id, name, "R2", "^(BC)", bc, "Antigen Capture"]) @@ -424,11 +426,11 @@ def build( ): """Generate one antigen scenario into the given dirs. - fastq_dir - R1/R2 FASTQs (+ offpanel-barcodes.txt for the offpanel scenario) + fastq_dir - R1/R2 FASTQs, plus offpanel-barcodes.txt for the offpanel scenario shared_dir - tags.csv / feature_reference.csv / samples-metadata.tsv (the block uploads) - truth_dir - expected-abundance/consensus (+ expected-specificity for control) + truth_dir - expected-abundance/consensus, plus expected-specificity for control - For a colocated preset run these are runs//{antigen, ., truth}; for a standalone scenario + For a colocated preset run these are runs//{antigen, ., truth}. For a standalone scenario all three point at runs/scenarios//. """ use_whitelist = cfg.barcode_source == "whitelist737k" @@ -494,13 +496,13 @@ def build( def build_libraseq(cfg, out_dir): """LIBRA-seq / dual-probe fixture: one antigen (BG505) read out by TWO feature barcodes that must BOTH fire, alongside a single-barcode antigen (gp120) and a negative control. Exercises Feature - Barcode Analysis's multi-barcode combine mode 'all' (AND): cells where only one BG505 probe barcode + Barcode Analysis's multi-barcode combine mode "all" (AND): cells where only one BG505 probe barcode fires must NOT be called BG505. - Antigen-only (no VDJ/GEX arm) — FI is antigen-only, so this alone drives the per-cell antigen call. - Writes tags.csv WITH a `combine` column (BG505=all, gp120/control=sum). Read geometry is the BEAM - default (R1 = 16 bp cell + 10 bp UMI; R2 = 15 bp feature at position 0), so the block's default - preset + de-novo cell whitelist parse it directly. + Antigen-only, with no VDJ or GEX arm, because FI is antigen-only and this alone drives the per-cell + antigen call. Writes tags.csv WITH a `combine` column (BG505=all, gp120/control=sum). Read geometry + is the BEAM default -- R1 = 16 bp cell + 10 bp UMI, R2 = 15 bp feature at position 0 -- so the + block's default preset plus a de-novo cell whitelist parse it directly. """ rng = new_rng(cfg.seed) os.makedirs(out_dir, exist_ok=True) diff --git a/software/test-data/manual/lib/beam_exact.py b/software/test-data/manual/lib/beam_exact.py index 32fc972..a636ead 100644 --- a/software/test-data/manual/lib/beam_exact.py +++ b/software/test-data/manual/lib/beam_exact.py @@ -1,6 +1,6 @@ """BEAM-exact fixture: a small multiomic run whose SHAPE matches real-world BEAM antibody-barcode -libraries, so the full Feature Barcode Profiling -> VDJ Multiomic Integration -> Lead Selection chain can -be exercised FAST on a local backend instead of multi-GB reference FASTQs. +libraries, so the full Feature Barcode Profiling -> VDJ Multiomic Integration -> Lead Selection chain +can be exercised FAST on a local backend instead of multi-GB reference FASTQs. Two things distinguish this from the generic `realistic`/`multisample` presets, both modelled on production BEAM libraries: @@ -8,17 +8,18 @@ 1. R2 read geometry has a 10 bp OFFSET before the feature barcode: R1 = CELL(16) + UMI(10) R2 = OFFSET(10) + FEATURE(15) + tail (real pattern: ^N{10}(FEATURE:N{15})(R2:*)) - The generic generators put the feature at R2 position 0; the real BEAM libraries carry a 10 bp - lead-in, which is why the block runs with the `generic-fb-umi` preset, not `tenx-beam`. + The generic generators put the feature at R2 position 0. The real BEAM libraries carry a 10 bp + lead-in, which is why the block runs with the `generic-fb-umi` preset and not `tenx-beam`. 2. A genuinely SAMPLE-AWARE panel CSV (Sample,Sequence,Protein): each sample has its own antigen panel, and a couple of barcode SEQUENCES are REUSED across samples mapped to DIFFERENT proteins. - That mirrors real multi-sample BEAM panels: the same 15-mer means one antigen in one sample's panel - and a different one in another's. Without the Sample column the tag CSV then has one barcode on two rows with - different proteins, which trips Feature Barcode Profiling's duplicate-barcode guard; WITH the Sample - column the workflow filters the CSV per sample and each barcode is unique again. + That mirrors real multi-sample BEAM panels, where the same 15-mer means one antigen in one + sample's panel and a different one in another's. Without the Sample column the tag CSV then has + one barcode on two rows with different proteins, which trips Feature Barcode Profiling's + duplicate-barcode guard. WITH the Sample column the workflow filters the CSV per sample and each + barcode is unique again. -Colocated with a coherent AIRR single-cell VDJ arm (reusing lib.vdj) so clonotypes bind their sample's +Colocated with a coherent AIRR single-cell VDJ arm, reusing lib.vdj, so clonotypes bind their sample's antigens and the convergence [sampleId, cellId] join lines up. Deterministic, standard-library only. """ @@ -106,12 +107,12 @@ def build( """Generate the BEAM-exact run under run_dir: antigen FASTQs (offset-10 R2), a sample-aware tag CSV, the coherent AIRR VDJ arm, and truth tables. - The tag CSV also carries the real customer panel's Type/Species columns: the control -> Decoy; the - first `offtarget_count` antigens of each sample's panel -> Off-Target; the rest -> Target; species - alternate Human/Cyno. + The tag CSV also carries the Type/Species columns a real panel declares. The control becomes Decoy. + The first `offtarget_count` antigens of each sample's panel become Off-Target, and the rest become + Target. Species alternate Human/Cyno. With `multibarcode=True` the first non-control antigen of each sample gets a 2nd barcode under - combine="all" (AND) and the second a 2nd barcode under combine="sum"; the tag CSV gains a `combine` + combine="all" (AND) and the second a 2nd barcode under combine="sum". The tag CSV gains a `combine` column and emits one row per member barcode. The extra barcodes are drawn ONLY in this branch, so a default beam run is byte-identical to before.""" rng = new_rng(BEAM_SEED) @@ -206,7 +207,7 @@ def build( reads.append([f"{sample}_read{read_no}", cell + umi, r2, 1]) else: # Multi-barcode antigen (only in a --multibarcode run): combine="all" fires EVERY - # member at ~k (AND); combine="sum" splits k across the members. Same UMI/dup shape. + # member at ~k (AND). Combine="sum" splits k across the members. Same UMI/dup shape. if combine.get(feat, "sum") == "all": shares = [k] * len(member_bcs) else: @@ -234,7 +235,7 @@ def build( tags_csv = os.path.join(run_dir, "tags.csv") with open(tags_csv, "w", newline="") as f: w = csv.writer(f) - # Type/Species mirror the real customer panel; Class is intentionally omitted here (beam-exact + # Type/Species mirror a real panel; Class is intentionally omitted here (beam-exact # antigens are all synthetic -> uniform class), so the full-run tags.csv is the Class exemplar. # --multibarcode inserts a `combine` column after Protein (one row per member barcode). if multibarcode: @@ -251,7 +252,7 @@ def build( w.writerow(["sample", "cellId", "planted_consensus"]) w.writerows(consensus_rows) - # vdj.build needs a `feature`-column CSV for load_clear_antigens; write a plain flat view for it. + # vdj.build needs a `feature`-column CSV for load_clear_antigens. Write a plain flat view for it. vdj_tags = os.path.join(run_dir, "_vdj_tags.csv") with open(vdj_tags, "w", newline="") as f: w = csv.writer(f) diff --git a/software/test-data/manual/lib/common.py b/software/test-data/manual/lib/common.py index cbf8f2c..13adc36 100644 --- a/software/test-data/manual/lib/common.py +++ b/software/test-data/manual/lib/common.py @@ -1,7 +1,7 @@ """Shared primitives for the manual test-data generators (antigen / VDJ / GEX arms). -Standard library only, deterministic (seeded). Read geometry matches the Feature Integration -block defaults (10x 5' v2): +Standard library only, deterministic and seeded. Read geometry matches the Feature Integration block +defaults (10x 5' v2): R1 = CELL(16) + UMI(10) -> 26 bp R2 = feature barcode(15) + filler -> 25 bp (block reads first 15 bp as the feature) @@ -51,11 +51,11 @@ def gen_distinct(rng, count, length, min_dist, avoid=()): def gen_cells(rng, count): - """Distinct random 16-mer cell barcodes, O(count) via a set (the panel's gen_distinct is O(n^2) - and does not scale to tens of thousands of cells). Hamming spacing is NOT enforced: at cohort - scale a 1 bp error colliding with a *different* real barcode is astronomically unlikely - (~count * 48 / 4^16), so the `errors` scenario's clean-correction guarantee still holds, and - real barcodes are Hamming-close anyway.""" + """Distinct random 16-mer cell barcodes, O(count) via a set. The panel's gen_distinct is O(n^2) and + does not scale to tens of thousands of cells. Hamming spacing is NOT enforced: at cohort scale a + 1 bp error colliding with a *different* real barcode is astronomically unlikely + (~count * 48 / 4^16), so the `errors` scenario's clean-correction guarantee still holds, and real + barcodes are Hamming-close anyway.""" seen = set() out = [] while len(out) < count: diff --git a/software/test-data/manual/lib/gex.py b/software/test-data/manual/lib/gex.py index ca30eff..29fc67d 100644 --- a/software/test-data/manual/lib/gex.py +++ b/software/test-data/manual/lib/gex.py @@ -1,19 +1,20 @@ """GEX (gene-expression) arm generator. -Builds a synthetic single-cell count matrix per donor sharing the SAME cell barcodes as the antigen + +Builds a synthetic single-cell count matrix per donor sharing the SAME cell barcodes as the antigen and VDJ arms, so import-sc-rnaseq-data -> pl7.app/rna-seq/countMatrix feeds the OPTIONAL GEX input of vdj-multiomic-integration and cell-type-annotation (CellTypist). -Coherence: each cell's expression program follows its planted antigen class (from the antigen arm's -expected-consensus.tsv). Clear-binder cells get a plasmablast/plasma program (MZB1/XBP1/PRDM1/CD38/ -TNFRSF17 high); ambiguous cells get a naive-B program (TCL1A/IGHD/IGHM high). All cells are B lineage -(a CD19-sorted BEAM experiment). - -Format (verified against import-sc-rnaseq-data): genes-in-rows CSV — first column = real human Ensembl -IDs (`^ENSG\\d{11}$` -> species=human, gene-format=Ensembl auto-detected); header = bare-16nt cell -barcodes; body = integer counts. import-sc-rnaseq-data's detect_orientation TRANSPOSES a matrix whenever -cells outnumber genes, so build() keeps the gene count strictly above the largest per-donor cell count — -otherwise CellTypist reads barcodes as gene names and fails with "No features overlap with the model". +Coherence: each cell's expression program follows its planted antigen class, from the antigen arm's +expected-consensus.tsv. Clear-binder cells get a plasmablast/plasma program, with +MZB1/XBP1/PRDM1/CD38/TNFRSF17 high. Ambiguous cells get a naive-B program, with TCL1A/IGHD/IGHM high. +All cells are B lineage, since this is a CD19-sorted BEAM experiment. + +Format, verified against import-sc-rnaseq-data: genes-in-rows CSV. The first column is real human +Ensembl IDs (`^ENSG\\d{11}$` -> species=human, gene-format=Ensembl auto-detected), the header is +bare-16nt cell barcodes, and the body is integer counts. import-sc-rnaseq-data's detect_orientation +TRANSPOSES a matrix whenever cells outnumber genes, so build() keeps the gene count strictly above the +largest per-donor cell count. Otherwise CellTypist reads barcodes as gene names and fails with "No +features overlap with the model". """ import csv @@ -35,7 +36,7 @@ def poisson(rng, lam): - """Small counts via Knuth; gaussian approximation for large lambda.""" + """Small counts via Knuth. Gaussian approximation for large lambda.""" if lam <= 0: return 0 if lam > 30: @@ -98,7 +99,7 @@ def build_gene_table(rng, sym2ens, protein_coding, n_filler): genes.append((ens, prog, means)) if missing: print(f" WARNING: markers not found in gene map (skipped): {missing}") - # filler: real protein_coding genes, low baseline noise; per-gene mean gives clustering texture + # filler: real protein_coding genes, low baseline noise. Per-gene mean gives clustering texture pool = [e for e in protein_coding if e not in used] rng.shuffle(pool) for ens in pool[:n_filler]: diff --git a/software/test-data/manual/lib/panel.py b/software/test-data/manual/lib/panel.py index 769ffd4..ae19471 100644 --- a/software/test-data/manual/lib/panel.py +++ b/software/test-data/manual/lib/panel.py @@ -1,7 +1,7 @@ -"""The antigen/feature panel — the feature-barcode whitelist the Feature Integration block snaps -reads to. The first (up to) 4 barcodes are the REAL 10x BEAM-Ab panel from the public "2k transgenic -HEL mouse splenocytes" dataset; the rest are synthesized as distinct 15-mers (Hamming >= 3 from each -other and the anchors) so the panel scales to any size while keeping authentic anchors. +"""The antigen/feature panel -- the feature-barcode whitelist the Feature Integration block snaps reads +to. The first (up to) 4 barcodes are the REAL 10x BEAM-Ab panel from the public "2k transgenic HEL +mouse splenocytes" dataset. The rest are synthesized as distinct 15-mers, Hamming >= 3 from each other +and the anchors, so the panel scales to any size while keeping authentic anchors. https://www.10xgenomics.com/datasets/2k-transgenic-hel-mouse-splenocytes-beam-ab-2-standard """ @@ -21,9 +21,9 @@ ] CONTROL_BC = "CTATCTACCGGCTCG" -# Per-antigen "Class" vocabulary, matching the real customer panel's Class column. The real anchors -# carry a biologically meaningful class; synthetic antigens default to "synthetic"; the negative -# control is "control". +# Per-antigen "Class" vocabulary, matching the Class column a real panel carries. The real anchors carry +# a biologically meaningful class. Synthetic antigens default to "synthetic", and the negative control +# is "control". ANCHOR_CLASS = { "SARS-TRI-S_WT": "viral", "Anti-Hen_Egg_Lysozyme": "enzyme", @@ -35,13 +35,13 @@ def classify_antigens(names, offtarget_count): - """Classify an ORDERED list of non-control antigen names into the real customer panel's per-antigen - Type / Species / Class. The first `offtarget_count` -> Off-Target; the rest -> Target; species - alternate Human/Cyno; class from ANCHOR_CLASS (real anchors) else "synthetic". + """Classify an ORDERED list of non-control antigen names into the per-antigen Type / Species / Class + a real panel declares. The first `offtarget_count` become Off-Target and the rest become Target. + Species alternate Human/Cyno. Class comes from ANCHOR_CLASS for the real anchors, else "synthetic". - Single source of the classification + offtarget-count validation rule, shared by `build_panel` and - the beam-exact path so the two never drift. The negative control is NOT included — callers add it - (Decoy / "" / control). Raises SystemExit if `offtarget_count` is out of range. + Single source of the classification and the offtarget-count validation rule, shared by `build_panel` + and the beam-exact path so the two never drift. The negative control is NOT included -- callers add + it (Decoy / "" / control). Raises SystemExit if `offtarget_count` is out of range. Returns (types, species, classes), each a dict keyed by antigen name.""" if not 0 <= offtarget_count <= len(names): @@ -61,7 +61,7 @@ class Panel: negative control. Replaces the old module-level ANTIGEN_NAMES / FEATURES globals so a generator run is fully described by its arguments. - Carries the real customer panel's per-antigen metadata — `types` (Target/Off-Target/Decoy), + Carries the per-antigen metadata a real panel declares — `types` (Target/Off-Target/Decoy), `species` (Human/Cyno/""), `classes` (viral/enzyme/synthetic/control) — each a dict keyed by feature name and INCLUDING the control. They default to sensible values so `Panel(names, feats)` stays valid for any legacy construction. @@ -93,12 +93,12 @@ def barcodes(self): def _double_space_name(name): - """Return `name` with a stray INTERNAL double space, reproducing the real customer panel's - `Human OT1 Biotin`. Replaces the first separator ("_" / "-" / " ") with two spaces; if the name has - none, injects " " near the middle. Injected at the panel SOURCE (so the name is used consistently - across tags.csv, feature_reference.csv, the truth tables and the VDJ/GEX name joins) — a messy - *label*, not broken data: nothing in generation is left mismatched, and the block's - whitespace-normalization collapses it back.""" + """Return `name` with a stray INTERNAL double space, reproducing a quirk real panels carry in antigen + names. Replaces the first separator ("_" / "-" / " ") with two spaces. If the name has none, injects + " " near the middle. Injected at the panel SOURCE, so the name is used consistently across + tags.csv, feature_reference.csv, the truth tables and the VDJ/GEX name joins. A messy *label*, not + broken data: nothing in generation is left mismatched, and the block's whitespace-normalization + collapses it back.""" for sep in ("_", "-", " "): i = name.find(sep) if i != -1: @@ -108,24 +108,25 @@ def _double_space_name(name): def build_panel(panel_size, seed=common.ANTIGEN_SEED, offtarget_count=0, multibarcode=False, messy=False): - """Build a Panel of `panel_size` antigens + 1 control. The first min(panel_size, 4) antigens are - the real 10x anchors; the rest are synthetic 15-mers (Hamming >= 3 from each other and the anchors - + control). Uses an independent RNG so the panel is identical regardless of sample/cell scale. - - Per-antigen Type/Species/Class match the real customer panel shape: the control -> Decoy/control; - the first `offtarget_count` antigens -> Off-Target; the rest -> Target; species alternate - Human/Cyno; class from ANCHOR_CLASS (real anchors) else "synthetic". - - With `multibarcode=True` the first antigen gets a SECOND barcode read out under combine="all" - (AND — both members must fire) and the second antigen a second barcode under combine="sum" (the - per-barcode UMIs add up); every other antigen stays single-barcode "sum". This is the shared-path - analogue of the libraseq fixture, so the FI multi-barcode combine logic is exercisable inside a - full multiomic run. The extra barcodes come from the panel RNG and are ONLY drawn in this branch, - so a default (single-barcode) panel is byte-identical to before. - - With `messy=True` (from --messy-metadata) the first antigen NAME gets a stray double space, so the - emitted metadata reproduces the customer panel's whitespace inconsistency for the normalization - tasks. Off by default (byte-identical).""" + """Build a Panel of `panel_size` antigens plus 1 control. The first min(panel_size, 4) antigens are + the real 10x anchors. The rest are synthetic 15-mers, Hamming >= 3 from each other and the anchors + and control. Uses an independent RNG so the panel is identical regardless of sample or cell scale. + + Per-antigen Type/Species/Class match the shape a real panel carries. The control becomes + Decoy/control. The first `offtarget_count` antigens become Off-Target and the rest become Target. + Species alternate Human/Cyno, and class comes from ANCHOR_CLASS for the real anchors, else + "synthetic". + + With `multibarcode=True` the first antigen gets a SECOND barcode read out under combine="all", where + both members must fire, and the second antigen a second barcode under combine="sum", where the + per-barcode UMIs add up. Every other antigen stays single-barcode "sum". This is the shared-path + analogue of the libraseq fixture, so the FI multi-barcode combine logic is exercisable inside a full + multiomic run. The extra barcodes come from the panel RNG and are ONLY drawn in this branch, so a + default single-barcode panel is byte-identical to before. + + With `messy=True`, from --messy-metadata, the first antigen NAME gets a stray double space, so the + emitted metadata reproduces the whitespace inconsistency real panels carry for the normalization + tasks. Off by default, and byte-identical.""" if panel_size < 1: raise SystemExit("panel size must be >= 1") prng = new_rng(seed + 7) diff --git a/software/test-data/manual/lib/panelswap.py b/software/test-data/manual/lib/panelswap.py index 691a2fe..ed3d5fd 100644 --- a/software/test-data/manual/lib/panelswap.py +++ b/software/test-data/manual/lib/panelswap.py @@ -13,7 +13,8 @@ multisample Two samples with DIFFERENT binding profiles (sample1 Spike-heavy, sample2 RSV/HA-heavy), exercising the per-sample axis the next block aggregates on. -10x 5' v2 geometry: R1 = CELL(16) + UMI(10); R2 = FEATURE(15) + remainder. Deterministic, stdlib only. +10x 5' v2 geometry: R1 = CELL(16) + UMI(10), and R2 = FEATURE(15) + remainder. Deterministic, stdlib +only. """ import csv diff --git a/software/test-data/manual/lib/realpanel.py b/software/test-data/manual/lib/realpanel.py new file mode 100644 index 0000000..2260a23 --- /dev/null +++ b/software/test-data/manual/lib/realpanel.py @@ -0,0 +1,1871 @@ +"""A run built on a REAL, externally-supplied panel file, at cohort scale, with a deliberate spread of +reading quality. + +Everything the other presets synthesize -- sample names, antigen names, feature barcodes, per-antigen +roles -- this path READS from a panel CSV whose path is given on the command line. Nothing about that +panel is baked in here: this module carries column-name defaults for the wide panel shape and nothing +else, so a real and possibly confidential panel can drive a run without any of it entering the +repository. The panel is copied verbatim into the run directory, which is gitignored like every other +generated artefact. + +Three things this path does that no other preset does: + + 1. **Per-sample panels.** A wide panel declares, per row, which sample offers which antigen, so each + sample has its own panel and a barcode SEQUENCE may carry a different antigen name in a different + sample. That is not a defect to be normalised away -- it is tag-inventory reuse, and the block's + `sampleColumn` exists for it. Reads are generated per sample against that sample's own panel, so + the reuse is real in the data and not only in the CSV. + + 2. **No declared comparator.** A real panel's role column names what a member is TO THE QUESTION + (target, off-target) and carries no value meaning "negative control". So this path plants NO + control feature. Background lives on the panel's own members, which is what forces the run down + the panel-reference or no-reference path rather than the declared-reference one. + + 3. **A reading-quality mix.** Every cell is planted at one of eight named tiers, from a clean strong + binder down to pure noise, chosen so each tier lands in a KNOWN verdict state -- bound, not bound + or unreliable -- and the tier is written to the truth table. A run therefore carries good, medium + and bad readings together, in stated proportions, instead of one uniform signal strength. + +Read geometry is the real BEAM one -- R2 = 10 bp lead-in + 15 bp feature + tail -- because these are +antigen-capture barcodes read out by that chemistry. `--offset 0` puts the feature at position 0 for the +generic geometry instead. + +Deterministic and standard-library only, like the rest of this bed. +""" + +import csv +import os +import shutil +import statistics as st + +from . import vdj +from .common import ( + CELL_LEN, + FEAT_LEN, + R2_FILLER, + UMI_LEN, + gen_cells, + gen_distinct, + mutate, + new_rng, + rand_seq, + write_fastq_gz, +) + +REALPANEL_SEED = 20260820 +# Share of reads carrying a 1 bp error in the FEATURE barcode. A real library measured 1-2% of reads as +# Hamming-1 variants of the dominant barcode, which tag refinement corrects back. +SEQ_ERROR_FRAC = 0.015 +R2_TAIL = "TTAATTAATT" # neutral remainder after the feature barcode (captured by R2:* and ignored) + +# Default column names for the wide panel shape: sample, antigen name, catalogue id, barcode sequence, +# detection channel, a constant, role. Generic headers for a generic shape -- override any of them on the +# command line when a panel spells them differently. +DEFAULT_COLUMNS = { + "sample": "Samples", + "name": "Name", + "sequence": "Sequence", + "role": "Type", +} + +# Which role values mean "this member is the question" against "this member is the comparator the question +# is read against". Matched case-insensitively on the leading word, so `Target (Primary)`, +# `Target (Secondary)` and a bare `Target` all read as on-target. A panel using other words needs +# --target-roles / --offtarget-roles. +DEFAULT_TARGET_ROLES = ("target",) +DEFAULT_OFFTARGET_ROLES = ("off-target", "offtarget", "off target") + +# The NARROW panel shape: sample, antigen name, barcode sequence, and no role column at all. This is the +# shape the production in-vivo project actually uploads, so it is not an edge case. Role is carried in the +# antigen NAME instead, and the comparator is chosen by naming one member. +NARROW_COLUMNS = { + "sample": "Sample", + "name": "Antigen", + "sequence": "Sequence", +} + +# Generic role words looked for in an antigen NAME when the panel declares no role column. These are +# industry words, not any panel's own vocabulary: nothing from a real panel belongs in this file. Matched +# case-insensitively as substrings, because a name carries them mid-string ("... (high OT risk)") rather +# than as a leading token the way a role COLUMN does. +NAME_OFFTARGET_HINTS = ( + "off-target", "off target", "offtarget", "ot risk", "high ot", + "homology", "decoy", "irrelevant", "unrelated", "negative control", "neg ctrl", +) + + +# --- reading-quality tiers ------------------------------------------------------------------------ +# +# (name, weight, doc). The weights are the shares of a sample's cells. Magnitudes are the real 5k-cell +# BEAM library's: dominant ~600 distinct UMIs (p10 ~18, p90 ~1490), near-mono dominance (median ~1.0, +# p10 ~0.79), background ~3 UMIs/cell. Each tier is chosen to land in a known verdict state given the +# block's defaults (count floor 4, a specificity cutoff in the 90s), so the truth table's `tier` column +# is an expectation and not a label. +TIERS = [ + ("strong", 0.28, "a clean high-count binder on one target; background at the floor -> bound"), + ("good", 0.20, "a solid binder, an order less signal than strong -> bound"), + ("medium", 0.16, "moderate signal with a real second reading -> lands on both sides of the line"), + ("weak", 0.11, "a real reading clear of the count floor but far below the line -> not bound"), + ("noise", 0.09, "every member within a couple of UMIs of nothing -> floored, not bound"), + ("crossreactive", 0.06, "two on-target members co-dominant -> neither alone dominant"), + ("offtarget", 0.06, "the dominant reading is an OFF-target member -> a high comparator, no lead"), + ("gated", 0.04, "comparator reads as high as the target -> set aside by the admissibility gate"), +] +TIER_NAMES = [t[0] for t in TIERS] + + +# --- the two measured regimes -------------------------------------------------------------------- +# +# Two calibrations exist, both measured, and they disagree by more than an order of magnitude. Which one a +# bed should carry is a question about which library it stands in for. +# +# deep - the public 10x BEAM runs. Antigen libraries sequenced to ~97% saturation: ~33 reads per +# recovered UMI, a median of 200 antigen UMIs per called cell, near-mono dominance +# (median 0.995), and cell CALLING applied before anything is reported. +# shallow - real in-vivo BEAM libraries, measured from a production in-vivo deployment on 2026-08-21: +# 2.7-5.8 reads per distinct UMI, a median of 7 UMIs across the barcodes clearing a floor of +# 4, dominance around 0.44, no cell calling and no whitelist, so the raw barcode universe is +# what the block sees, and unfiltered antigen aggregates holding most of the library. +# +# `deep` is kept because it reproduces every run made before 2026-08-21 byte for byte. `shallow` stands in +# for real production data. Neither is right on its own: a bed carrying only `deep` tests a regime real +# in-vivo data never occupies, and one carrying only `shallow` cannot show the block reaching a confident +# answer at all. +# +# MAGNITUDES entries are the inclusive (lo, hi) UMI range a tier plants on its dominant member. +MAGNITUDES_DEEP = { + "strong": (500, 1400), + "good": (150, 500), + "medium": (60, 200), + "weak": (15, 45), + "noise": (1, 4), + "cross": (300, 1100), + "offtarget": (300, 1000), + "gated_target": (100, 400), + "gated_ref": (400, 1200), +} +# Scaled to the measured shallow distribution: among barcodes clearing a floor of 4 the per-barcode totals +# run p25 5, median 7, p75 11, p90 17, p99 201. So every tier but the top sits in single or low double +# digits, and `strong` carries the p99 tail rather than the bulk. +# +# Recalibrated 2026-08-21 against the CURRENT rule, where the comparator is a single declared baseline tag +# rather than the highest of several off-targets. That baseline usually reads 0, and against 0 the cutoff +# is reached at a count of 49 (against 5 it was ~120). So the line moved DOWN in comparator terms and UP +# relative to this regime's depth: with per-cell totals whose median is 7, only the top few percent of +# cells can reach it at all. That is not a defect to tune away -- it is why real in-vivo runs report 1.4% +# and 2.9% of cells bound, and the weights below are set so this bed lands in that band. +# +# Each tier's range is placed relative to the line at 49, so the tier NAMES describe outcomes again: +# strong clears it outright, good straddles it, and everything below it reads not bound. +MAGNITUDES_SHALLOW = { + "strong": (60, 250), + "good": (30, 70), + "medium": (12, 30), + "weak": (4, 12), + "noise": (1, 3), + "cross": (60, 150), + "offtarget": (12, 40), + "gated_target": (12, 30), + "gated_ref": (120, 400), +} + +# Tier weights. `deep` makes a clean binder the common case. `shallow` makes a sub-floor reading the +# common case, which is what 63-80% of real barcodes measured as. Weights chosen so the share of CELLS +# reaching a bound identity lands in the 1.4-2.9% band real in-vivo runs report. Only `strong` and +# `crossreactive` clear the line outright and `good` straddles it, so the bound share is roughly +# strong + half of good + crossreactive. +TIERS_SHALLOW = [ + ("strong", 0.010, "clears the line outright -> bound"), + ("good", 0.030, "straddles the line -> mostly not bound, since the baseline's own background " + "moves the line from 49 up toward 90"), + ("medium", 0.15, "a real reading well clear of the floor, far below the line -> not bound"), + ("weak", 0.30, "barely clear of the count floor -> not bound"), + ("noise", 0.452, "every member sub-floor -> floored, nothing to answer with"), + ("crossreactive", 0.008, "two members co-dominant ABOVE the line -> bound on two identities"), + ("offtarget", 0.03, "the baseline is the dominant reading -> not bound"), + ("gated", 0.02, "the comparator reads far above the target -> set aside when the gate is on"), +] + +# _background() parameters. At shallow depth background is not a floor UNDER the signal, it is COMPARABLE +# TO it: a dominant of 7 with one UMI on each of five other members gives a dominance fraction of 0.44, +# which is the measured median. That is counting noise rather than promiscuity, and reproducing it is the +# whole point of the regime. +BACKGROUND_DEEP = {"offtarget_hi": 6, "target_hi": 3, "offtarget_p": 0.85, "target_p": 0.35} +BACKGROUND_SHALLOW = {"offtarget_hi": 3, "target_hi": 3, "offtarget_p": 0.70, "target_p": 0.65} + +REGIMES = { + "deep": { + "tiers": TIERS, + "magnitudes": MAGNITUDES_DEEP, + "background": BACKGROUND_DEEP, + # None selects the original three-branch duplication draw (mean ~1.3 reads/UMI). Kept as its own + # branch, not a special case of the geometric one, so the RNG call sequence is unchanged and + # pre-2026-08-21 runs still reproduce byte for byte. + "dup_mean": None, + "ambient_frac": 0.18, + # 0 leaves the ambient barcode COUNT following from the read share alone, as it always did. + "ambient_barcode_ratio": 0.0, + "aggregates": 0, + "aggregate_umi_share": 0.0, + "clonal_profile": "immunized", + "clonal_mean_size": 25, + "clonal_singleton_cell_frac": 0.10, + "clonal_tail_cycle": None, + "unpaired_frac": 0.0, + "seq_error_frac": SEQ_ERROR_FRAC, + }, + "shallow": { + "tiers": TIERS_SHALLOW, + "magnitudes": MAGNITUDES_SHALLOW, + "background": BACKGROUND_SHALLOW, + # 2.7 and 5.8 reads per distinct UMI measured across the two measured libraries, so 4 sits + # between them. Drawn geometrically, which the capped three-branch draw cannot reach. + "dup_mean": 4.0, + "ambient_frac": 0.25, + # ~700k raw barcodes against a few thousand real cells. This is the single largest divergence from + # the old bed: the block reports "cells detected" off the raw universe, so the universe IS the QC + # number a user reads. + "ambient_barcode_ratio": 100.0, + # Five barcodes held 58.9% of one library's antigen UMIs, and the largest held 18.3% alone. + "aggregates": 5, + "aggregate_umi_share": 0.59, + # 4,549 IGHeavy clonotypes over 4,773 cells with paired chains = 1.05 cells per clonotype. + # Expressed through the existing knobs: 90% of cells left as singletons, the rest in clones + # averaging 3, which lands at ~1.07. + "clonal_profile": "immunized", + "clonal_mean_size": 2, + "clonal_singleton_cell_frac": 0.97, + "clonal_tail_cycle": vdj.TAIL_CYCLE_SPARSE, + # Clonotypes dropped for want of a pair outnumbered paired ones in both libraries. + "unpaired_frac": 0.35, + # "Fraction unrecognized antigen" measured at 4.22% on the public run, against the 1.5% this bed + # used. + "seq_error_frac": 0.042, + }, +} + + +class SamplePanel: + """One sample's panel: ordered member names, name -> 15 bp barcode, and each member's role. + + `targets` / `offtargets` split the members by the role column. Both may be empty -- a panel is + whatever the file says it is -- and every planter below degrades to the next-best tier rather than + failing when a sample cannot support the one it was asked for.""" + + def __init__(self, sample, members, target_roles, offtarget_roles): + self.sample = sample + self.names = [m["name"] for m in members] + self.barcode = {m["name"]: m["sequence"] for m in members} + self.role = {m["name"]: m["role"] for m in members} + self.rows = members # the panel's own rows, verbatim, for the run report + self.targets = [n for n in self.names if _role_in(self.role[n], target_roles)] + self.offtargets = [n for n in self.names if _role_in(self.role[n], offtarget_roles)] + + @property + def barcodes(self): + return [self.barcode[n] for n in self.names] + + +def _role_in(value, words): + """True when a role value starts with one of `words`, case- and space-insensitively. Matching on the + leading word is what lets `Target (Primary)` and `Target (Secondary)` both read as on-target while + staying two distinct values in the panel -- which is what they are, and what the block groups on.""" + v = " ".join((value or "").split()).lower() + return any(v == w or v.startswith(w) for w in words) + + +def _infer_role_from_name(name, control_feature=None): + """Role for a member the panel gave no role column for. + + A narrow panel still carries role information, in the antigen name. `control_feature`, when given, + names the one member serving as the comparator and wins outright. That mirrors the block, where the + user picks a control by name from a dropdown of antigen names. Otherwise the name is searched for the + generic off-target words in NAME_OFFTARGET_HINTS. + + A member matching nothing comes back on-target, which is the safe default: mistaking a target for a + comparator would silently move the line every reading is judged against.""" + if control_feature and name.strip().lower() == control_feature.strip().lower(): + return "Off-Target" + low = name.lower() + return "Off-Target" if any(h in low for h in NAME_OFFTARGET_HINTS) else "Target" + + +def detect_panel_shape(csv_path, columns=None): + """"wide" if the panel declares a role column, else "narrow". Reads only the header.""" + cols = dict(DEFAULT_COLUMNS) + cols.update(columns or {}) + if not os.path.exists(csv_path): + raise SystemExit(f"panel file not found: {csv_path}") + with open(csv_path, newline="") as fh: + header = csv.DictReader(fh).fieldnames or [] + return "wide" if cols["role"] in header else "narrow" + + +def load_panel(csv_path, columns=None, target_roles=DEFAULT_TARGET_ROLES, + offtarget_roles=DEFAULT_OFFTARGET_ROLES, shape="auto", control_feature=None): + """Read a panel CSV into {sample: SamplePanel}, in file order. Handles both shapes seen in use. + + WIDE declares a role column and this reads it. NARROW declares none -- sample, antigen, sequence and + nothing else -- and role is then inferred from the antigen name, or from `control_feature` where one + is named. Both shapes are live in production on different projects, so neither is the exception: a + loader that only reads a role column models the wrong half of real production work. + + Validates only what generation cannot proceed without: the named columns exist, every sequence is + 15 bp, and no sequence appears twice within one sample, which would be a genuine duplicate the + block's own guard rejects. Sequence reuse ACROSS samples is left alone -- it is the point of the + per-sample keying, not an error.""" + if shape == "auto": + shape = detect_panel_shape(csv_path, columns) + if shape not in ("wide", "narrow"): + raise SystemExit(f"unknown panel shape {shape!r}; expected wide, narrow or auto") + cols = dict(DEFAULT_COLUMNS if shape == "wide" else NARROW_COLUMNS) + cols.update(columns or {}) + if not os.path.exists(csv_path): + raise SystemExit(f"panel file not found: {csv_path}") + with open(csv_path, newline="") as fh: + reader = csv.DictReader(fh) + header = reader.fieldnames or [] + needed = ["sample", "name", "sequence"] + (["role"] if shape == "wide" else []) + missing = [cols[k] for k in needed if cols[k] not in header] + if missing: + raise SystemExit( + f"{csv_path} has no column(s) {missing}; its columns are {header}. " + "Name the right ones with --panel-sample-col / --panel-name-col / --panel-seq-col / " + "--panel-role-col." + ) + by_sample = {} + order = [] + for lineno, row in enumerate(reader, start=2): + sample = (row[cols["sample"]] or "").strip() + name = " ".join((row[cols["name"]] or "").split()) + seq = (row[cols["sequence"]] or "").strip().upper() + role = ((row[cols["role"]] or "").strip() if shape == "wide" + else _infer_role_from_name(name, control_feature)) + if not sample or not name or not seq: + raise SystemExit(f"{csv_path}:{lineno}: sample, name and sequence must all be present") + if len(seq) != FEAT_LEN: + raise SystemExit(f"{csv_path}:{lineno}: sequence {seq!r} is {len(seq)} bp, expected {FEAT_LEN}") + if sample not in by_sample: + by_sample[sample] = [] + order.append(sample) + if any(m["sequence"] == seq for m in by_sample[sample]): + raise SystemExit( + f"{csv_path}:{lineno}: sequence {seq} appears twice in sample {sample!r} — a real " + "duplicate, which the block rejects. Fix the panel." + ) + by_sample[sample].append({"name": name, "sequence": seq, "role": role, "row": row}) + if not by_sample: + raise SystemExit(f"{csv_path} has no data rows") + return {s: SamplePanel(s, by_sample[s], target_roles, offtarget_roles) for s in order} + + +def load_wide_panel(csv_path, columns=None, target_roles=DEFAULT_TARGET_ROLES, + offtarget_roles=DEFAULT_OFFTARGET_ROLES): + """Wide-shape loader. Kept as the name the tests and earlier callers use. `load_panel` is the one that + handles both shapes.""" + return load_panel(csv_path, columns, target_roles, offtarget_roles, shape="wide") + + +# --- per-cell planting --------------------------------------------------------------------------- + +def pick_tier(rng, tiers=None): + """One tier name, by weight. A single random draw per cell so the stream stays stable when the tier + table changes weight but not order.""" + tiers = tiers or TIERS + r = rng.random() + acc = 0.0 + for name, weight, _doc in tiers: + acc += weight + if r < acc: + return name + return tiers[-1][0] + + +def _background(rng, panel, exclude, offtarget_hi=6, target_hi=3, offtarget_p=0.85, target_p=0.35): + """Background readings on the members that are not the cell's dominant one. + + OFF-TARGET members are treated differently from other targets, and the difference is load-bearing. A + panel with no declared negative control is read against its off-target members, so an off-target + reading is this cell's COMPARATOR -- and a comparator below the reference thin line (2 by default) + makes the position *unreliable*, not *not bound*: the comparison could not be made. Plant the + off-targets sparsely and almost every cell in the run comes back unreliable, which says nothing about + binding and is an artefact of the bed, not a finding. + + So off-targets read in ~85% of cells at 2-`offtarget_hi` UMIs -- a comparator that can be compared + against -- while the remaining on-target members stay at true background, present in ~35% of cells at + 1-`target_hi`. The real library measured ~3 members read per cell and a background median of ~3 UMIs, + which both of these sit inside. The ~15% of cells with no off-target reading are left alone on + purpose: *unreliable for want of a comparator* is a state the block has to be able to show, and a run + with none of it cannot show it.""" + out = {} + for name in panel.names: + if name in exclude: + continue + if name in panel.offtargets: + if rng.random() < offtarget_p: + out[name] = rng.randint(2, max(2, offtarget_hi)) + elif rng.random() < target_p: + out[name] = rng.randint(1, max(1, target_hi)) + return out + + +def plant_cell(rng, panel, tier, primary_bias=0.0, mag=None, bg=None): + """Plant one cell's per-member distinct-UMI counts at `tier`. + + Returns (per_member, consensus, tier_actually_used). `consensus` is what the VDJ arm groups clonotypes + on: a member NAME when the cell has one clear dominant, else the tier word. A tier a sample's panel + cannot support -- no off-target member, fewer than two targets -- degrades to `good`, and the returned + tier says so, so the truth table never claims a tier the data does not hold. + + `primary_bias` tilts the dominant choice toward the panel's FIRST target: an antigen-sorted library is + not a uniform draw over its panel, and a real one had a single antigen at 90% of the library. + + `mag` and `bg` come from the regime (see REGIMES). They carry only MAGNITUDES. Every relative decision + -- which member is dominant, how a tier degrades, how the second reading relates to the first -- is + regime-independent and stays here. Defaulting them to the deep tables keeps the RNG call sequence + identical to what this function did before regimes existed.""" + mag = mag or MAGNITUDES_DEEP + bg = bg or BACKGROUND_DEEP + targets = panel.targets or panel.names + offtargets = panel.offtargets + + def choose_target(): + if primary_bias > 0 and targets and rng.random() < primary_bias: + return targets[0] + return rng.choice(targets) + + if tier == "noise": + # Deliberately thin, off-targets included: nothing here clears the count floor, and a cell whose + # off-target reading lands at 0 or 1 has no comparator at all. Both outcomes are real and this is + # the tier they come from, so the rest of the run does not have to carry them. + per = {n: rng.randint(*mag["noise"]) for n in panel.names if rng.random() < 0.6} + if not per: + per = {rng.choice(panel.names): rng.randint(*mag["noise"])} + return per, "noise", tier + + if tier == "crossreactive": + if len(targets) < 2: + tier = "good" + else: + a, b = rng.sample(targets, 2) + dom = rng.randint(*mag["cross"]) + per = {a: dom, b: max(1, int(dom * rng.uniform(0.85, 1.0)))} + per.update(_background(rng, panel, set(per), **bg)) + return per, "crossreactive", tier + + if tier == "offtarget": + if not offtargets: + tier = "good" + else: + dom = rng.choice(offtargets) + per = {dom: rng.randint(*mag["offtarget"])} + per.update(_background(rng, panel, {dom}, **bg)) + return per, dom, tier + + if tier == "gated": + if not offtargets: + tier = "medium" + else: + tgt = choose_target() + ref = rng.choice(offtargets) + per = {tgt: rng.randint(*mag["gated_target"]), ref: rng.randint(*mag["gated_ref"])} + per.update(_background(rng, panel, set(per), **bg)) + return per, tgt, tier + + tgt = choose_target() + if tier == "strong": + per = {tgt: rng.randint(*mag["strong"])} + per.update(_background(rng, panel, {tgt}, **bg)) + elif tier == "good": + per = {tgt: rng.randint(*mag["good"])} + per.update(_background(rng, panel, {tgt}, **bg)) + elif tier == "medium": + # Straddles the line on purpose. With the off-target members serving as the comparator, a cell's + # reference reading is the MAX over them -- about 5 UMIs at this background -- and the antigen + # count that reaches a specificity of 75 against a reference of 5 is about 120. 60-200 therefore + # lands on both sides of the line, which is the only way to see where the line is. + dom = rng.randint(*mag["medium"]) + per = {tgt: dom} + # The second reading goes on another TARGET where the panel has one. On an off-target it would + # raise this cell's own comparator and turn a near-the-line cell into a comparator-dominated one, + # which is what the `offtarget` tier is for, and mixing the two makes neither legible. + rest = [n for n in (panel.targets or panel.names) if n != tgt] + if rest: + per[rng.choice(rest)] = max(1, int(dom * rng.uniform(0.35, 0.65))) + per.update(_background(rng, panel, set(per), + **dict(bg, offtarget_hi=bg["offtarget_hi"] + 2, + target_hi=bg["target_hi"] + 3))) + elif tier == "weak": + # Clear of the count floor of 4, so every reading here is a reading and answers *not bound* -- not + # the same thing as the floored readings the `noise` tier produces, which answer nothing. + dom = rng.randint(*mag["weak"]) + per = {tgt: dom} + per.update(_background(rng, panel, {tgt}, + **dict(bg, offtarget_hi=max(4, dom // 3), target_p=0.5))) + else: + raise AssertionError(f"unhandled tier {tier!r}") + return per, tgt, tier + + +# --- read emission ------------------------------------------------------------------------------- + +def _r2(barcode, offset): + """R2 for one read. offset=10 is the real BEAM geometry, with a 10 bp lead-in before the feature. + offset=0 puts the feature at position 0, the generic feature-barcode geometry.""" + return (R2_FILLER[:offset] if offset else "") + barcode + R2_TAIL + + +def _dup_count(rng, dup_mean): + """How many reads one distinct UMI produces. + + `dup_mean is None` keeps the original capped three-branch draw (mean ~1.3 reads/UMI). It is left as + its own branch rather than a special case of the geometric one so the RNG call sequence is unchanged + and pre-regime runs reproduce byte for byte. + + Otherwise a geometric draw with mean `dup_mean`. The capped draw tops out at 3 and so cannot reach the + 2.7-5.8 reads per distinct UMI real in-vivo libraries measure. This can. Capped at 64 so a + pathological tail cannot dominate a run.""" + if dup_mean is None: + return 1 if rng.random() < 0.75 else (2 if rng.random() < 0.8 else 3) + p = 1.0 / max(1.0, dup_mean) + n = 1 + while n < 64 and rng.random() > p: + n += 1 + return n + + +def emit_cell_reads(rng, reads, sample, panel, cell, per_member, offset, seq_error_frac, read_no, + dup_mean=None): + """Append the reads one planted cell produces. Distinct UMIs per member, PCR duplication per + `dup_mean`, and `seq_error_frac` of reads carrying a 1 bp error in the FEATURE barcode -- the + Hamming-1 variants a real library shows and tag refinement corrects back.""" + for member, k in per_member.items(): + bc = panel.barcode[member] + umis = set() + while len(umis) < k: + umis.add(rand_seq(rng, UMI_LEN)) + # sorted(): set iteration order of strings varies per process (PYTHONHASHSEED), which would make + # the emitted read order non-reproducible. + for umi in sorted(umis): + dups = _dup_count(rng, dup_mean) + for _ in range(dups): + read_no += 1 + emitted = mutate(rng, bc) if seq_error_frac and rng.random() < seq_error_frac else bc + reads.append([f"{sample}_read{read_no}", cell + umi, _r2(emitted, offset), 1]) + return read_no + + +def add_ambient(rng, panel, reads, offset, frac, n_cells=0, barcode_ratio=0.0, dup_mean=None): + """Append ambient reads on OFF-cell barcodes: random 16-mers carrying on-panel features. + + Two modes, and the difference is the largest single divergence between this bed and the measured real + data. + + `barcode_ratio == 0` (the original): the ambient READ SHARE is `frac`, and the barcode count falls out + of it -- one barcode per ambient read pair, so ~18% of reads become a modest phantom population. + + `barcode_ratio > 0`: the barcode UNIVERSE is sized directly, at `n_cells * barcode_ratio` distinct + barcodes. This is the mode that matters. The block applies no cell calling and the live configuration + sets no whitelist, so what it reports as "cells detected" is the raw barcode universe: 1,374,025 of + them across two samples, with a MEDIAN of one UMI each. Every QC number downstream inherits that. A + bed whose barcodes are all real cells cannot reproduce a single one of those numbers. + + The per-barcode UMI shape is drawn to match: ~85% carry exactly one UMI, the rest a decaying tail. The + measured whole-table distribution was p50 1, p75 1, p90 16, p99 61. + + Ambient barcodes stay out of the truth tables. A whitelist drops them, de-novo correction keeps them + as phantom low-count cells, and both behaviours are worth having data for.""" + if frac <= 0 and barcode_ratio <= 0: + return 0 + bcs = panel.barcodes + base = len(reads) + if barcode_ratio > 0: + n_bc = int(n_cells * barcode_ratio) + planted_umis = 0 + for i in range(n_bc): + cell = rand_seq(rng, CELL_LEN) + # Median one UMI with a long tail. The measured whole-table shape was p50 1, p75 1-6, p90 + # 10-16, p99 53-61, so a majority of singletons is not enough on its own: the tail has to + # reach the tens or the panel median a comparator rests on comes out too clean. + k = 1 if rng.random() < 0.62 else 1 + int(rng.expovariate(1 / 9.0)) + k = min(k, 400) + planted_umis += k + for _ in range(k): + umi = rand_seq(rng, UMI_LEN) + r2 = _r2(rng.choice(bcs), offset) + for _ in range(_dup_count(rng, dup_mean)): + reads.append([f"ambient_read{base + i}", cell + umi, r2, 1]) + return planted_umis + n = int(len(reads) * frac) + for i in range(n): + cell = rand_seq(rng, CELL_LEN) + umi = rand_seq(rng, UMI_LEN) + r2 = _r2(rng.choice(bcs), offset) + for _ in range(rng.randint(1, 2)): + reads.append([f"ambient_read{base + i}", cell + umi, r2, 1]) + return n + + +# Relative UMI shares of the five aggregate barcodes measured in one 200k-barcode window of the larger +# measured library, normalised within the aggregate population. The largest held 18.3% of the WHOLE +# library's antigen UMIs on its own, and the five together held 58.9%. +AGGREGATE_PROFILE = (0.311, 0.266, 0.183, 0.168, 0.072) + + +def add_aggregates(rng, panel, reads, offset, n_aggregates, umi_share, other_umis, dup_mean=None): + """Append antigen-aggregate barcodes: a handful of droplets holding most of the library. + + Proteins clump nonspecifically during sample prep and the resulting GEMs carry enormous UMI counts. + Cell Ranger detects and removes exactly this population BEFORE cell calling. This block does not, and + until now this bed contained none -- so nothing in the bed exercised what an aggregate does to a panel + median, a comparator, or a "cells detected" count. + + `umi_share` is the share of the FINISHED library's UMIs these barcodes hold, so the count planted is + `other_umis * share / (1 - share)` where `other_umis` is every non-aggregate UMI already in the + library, signal AND ambient. Sizing it against signal alone under-plants badly once the barcode + universe is large, because the universe holds most of the non-aggregate UMIs. + + At the measured 0.59 the aggregates outnumber everything else about 1.4 to 1, which is why the + measured per-cell depth is starved even though those libraries are large. Most of the sequencing went + into five droplets. + + UMIs are spread over several panel features per barcode, because an aggregate is nonspecific: a clump + binds whatever is nearby, and a single-feature aggregate would read as an extremely confident binder + rather than as junk.""" + if n_aggregates <= 0 or umi_share <= 0 or other_umis <= 0: + return [] + total = int(other_umis * umi_share / max(1e-9, 1.0 - umi_share)) + profile = list(AGGREGATE_PROFILE[:n_aggregates]) + while len(profile) < n_aggregates: + profile.append(profile[-1] / 2) + scale = sum(profile) + planted = [] + bcs = panel.barcodes + for i, w in enumerate(profile): + cell = rand_seq(rng, CELL_LEN) + k = max(1, int(total * w / scale)) + members = rng.sample(bcs, min(len(bcs), rng.randint(2, max(2, min(5, len(bcs)))))) + planted.append((cell, k)) + for j in range(k): + umi = rand_seq(rng, UMI_LEN) + r2 = _r2(members[j % len(members)], offset) + for _ in range(_dup_count(rng, dup_mean)): + reads.append([f"aggregate_read{i}_{j}", cell + umi, r2, 1]) + return planted + + +def convert_offpanel(rng, reads, off_bcs, off_frac, offset): + """Rewrite `off_frac` of reads onto barcodes NOT in any sample's panel (Hamming >= 5 from every panel + member, so refinement drops rather than corrects them). Drives the panel-assigned fraction the block's + QC reports, and with it the Quality tag.""" + if off_frac <= 0 or not off_bcs: + return + k = min(len(reads), int(len(reads) * off_frac)) + for i in rng.sample(range(len(reads)), k): + reads[i][2] = _r2(rng.choice(off_bcs), offset) + + +def add_malformed(rng, reads, matched_frac): + """Append reads parse cannot read at all -- R1 too short for CELL+UMI, or R2 too short for the feature + -- so the matched fraction lands near `matched_frac`.""" + if matched_frac >= 1.0: + return + m = max(0.01, matched_frac) + n_bad = int(len(reads) * (1 - m) / m) + base = len(reads) + for i in range(n_bad): + if i % 2 == 0: + reads.append([f"malformed_read{base + i}", rand_seq(rng, 18), rand_seq(rng, FEAT_LEN), 1]) + else: + reads.append([f"malformed_read{base + i}", rand_seq(rng, 26), rand_seq(rng, 10), 1]) + + +# --- per-sample library quality ------------------------------------------------------------------ +# +# What reaches the block as a library, before any per-cell reading is read. (matched fraction, +# panel-assigned fraction) drive the block's Read-recovery bar and Quality tag: matched < 80% or +# panel-assigned < 50% -> WARN, panel-assigned < 25% -> ALERT. These are LIBRARY defects -- a bad prep, a +# panel that does not match the reads -- and they are a different axis from the per-cell reading tiers, +# which are about a cell's binding signal in a library that read out fine. +LIBRARY_TIERS = { + "clean": (1.00, 0.98, "OK"), + "good": (0.96, 0.94, "OK"), + "fair": (0.88, 0.78, "OK"), + "poor": (0.70, 0.45, "WARN"), + "bad": (0.55, 0.20, "ALERT"), +} +# How the tiers are dealt out across samples, cycling when there are more samples than entries. +QUALITY_PROFILES = { + "uniform": ["clean"], + "mixed": ["clean", "good", "fair", "poor"], + "spread": ["clean", "good", "poor", "bad"], +} + +# What each reading tier should come back as, given the block's defaults. A statement of intent for the +# reader of the truth table, not an assertion the generator can make on its own. +# +# The shallow table is what each tier should come back as under the CURRENT rule, at shallow depth, +# against a single declared baseline. The line is a count of 49 when the baseline reads 0 and climbs +# toward 90 as the baseline's own background rises, so `good` straddles a MOVING line and lands mostly not +# bound. +EXPECTED_STATE_SHALLOW = { + "strong": "bound — clears the line even where the baseline's own background has pushed it up", + "good": "on the line: bound where the baseline read 0, not bound where it read a few", + "medium": "not bound — a real reading, well clear of the floor, far below the line", + "weak": "not bound — barely clear of the count floor", + "noise": "floored; not bound, and nothing to answer with", + "crossreactive": "bound on TWO identities — both co-dominant readings clear the line", + "offtarget": "not bound; the baseline IS the dominant reading", + "gated": "unreliable (gate on) / not bound with a high comparator (gate off)", +} + +EXPECTED_STATE = { + "strong": "bound — hundreds of UMIs against a comparator of a few", + "good": "bound — clear of the line, without much margin", + "medium": "on the line: bound or not bound, and the tier straddles it deliberately", + "weak": "not bound — a real reading, clear of the count floor, nowhere near the line", + "noise": "not bound (every reading floored), or unreliable where the comparator reads below the thin line", + "crossreactive": "bound on TWO identities at once — neither is uniquely dominant", + "offtarget": "nothing bound and the comparator reads high (it IS the dominant one); " + "reads bound on the off-target instead when the comparator is the panel or none", + "gated": "unreliable (gate on) / not bound with a high comparator (gate off)", +} + + +def load_whitelist_cells(rng, count, assets_dir): + """Draw `count` real 737K-august-2016 cell barcodes, WITH replacement across samples (see build()). + Prefers the full 10x inclusion list, and falls back to the small harvested pool.""" + big = os.path.join(assets_dir, "737K-august-2016.txt") + small = os.path.join(assets_dir, "whitelist_cells.txt") + path = big if os.path.exists(big) else small + if not os.path.exists(path): + raise SystemExit( + f"no cell whitelist in {assets_dir}. Fetch the full 10x inclusion list:\n" + " curl -sSL -o 737K-august-2016.txt https://raw.githubusercontent.com/10XGenomics/" + "supernova/master/tenkit/lib/python/tenkit/barcodes/737K-august-2016.txt" + ) + with open(path) as fh: + pool = [ln.strip() for ln in fh if ln.strip() and not ln.startswith("#")] + if len(pool) < count: + raise SystemExit( + f"{os.path.basename(path)} holds {len(pool)} barcodes; this run needs {count} distinct ones " + "per sample. Fetch the full 737K-august-2016 list, or lower --cells-per-sample." + ) + return rng.sample(pool, count) + + +def build( + run_dir, + panel_csv, + cells_per_sample=6000, + barcode_source="whitelist737k", + assets_dir=None, + columns=None, + target_roles=DEFAULT_TARGET_ROLES, + offtarget_roles=DEFAULT_OFFTARGET_ROLES, + offset=10, + quality_profile="mixed", + regime="deep", + ambient_frac=None, + seq_error_frac=None, + primary_bias=0.35, + cell_jitter=0.25, + arm="all", + clonal_profile=None, + clonal_mean_size=None, + clonal_singleton_cell_frac=None, + ambient_barcode_ratio=None, + aggregates=None, + aggregate_umi_share=None, + unpaired_frac=None, + dup_mean=None, + panel_shape="auto", + control_feature=None, + seed=REALPANEL_SEED, +): + """Generate a cohort-scale run against the panel at `panel_csv`, into `run_dir`. + + Cell barcodes are drawn per sample INDEPENDENTLY from the whitelist, so samples share some barcodes -- + which is what real GEM wells do, and what makes (sampleId, cellId) the load-bearing key rather than + cellId alone. `cell_jitter` varies each sample's cell count so no two libraries are the same size. + + The panel file is copied into the run directory verbatim: that copy is the one the block uploads, so + what the block reads is the panel as it actually arrived, not a re-serialisation of it. + + `arm="vdj"` rebuilds ONLY the V(D)J arm, from the antigen arm's existing ground truth. The antigen arm + is the expensive half -- hundreds of megabytes of FASTQ against a few of TSV -- and the repertoire + shape is the half worth iterating on, so reshaping it should not cost a regeneration of the reads. + + `regime` selects a measured calibration (see REGIMES). Every regime-owned argument defaults to None, + meaning "take the regime's value", and passing one explicitly overrides it. `regime="deep"` with + nothing overridden reproduces every run made before regimes existed, byte for byte.""" + if regime not in REGIMES: + raise SystemExit(f"unknown regime {regime!r}; expected one of {', '.join(REGIMES)}") + R = REGIMES[regime] + tiers = R["tiers"] + tier_names = [t[0] for t in tiers] + mag, bgp = R["magnitudes"], R["background"] + expected_state = EXPECTED_STATE if regime == "deep" else EXPECTED_STATE_SHALLOW + pick = lambda given, key: R[key] if given is None else given # noqa: E731 + dup_mean = pick(dup_mean, "dup_mean") + ambient_frac = pick(ambient_frac, "ambient_frac") + seq_error_frac = pick(seq_error_frac, "seq_error_frac") + ambient_barcode_ratio = pick(ambient_barcode_ratio, "ambient_barcode_ratio") + aggregates = pick(aggregates, "aggregates") + aggregate_umi_share = pick(aggregate_umi_share, "aggregate_umi_share") + unpaired_frac = pick(unpaired_frac, "unpaired_frac") + clonal_profile = pick(clonal_profile, "clonal_profile") + clonal_mean_size = pick(clonal_mean_size, "clonal_mean_size") + clonal_singleton_cell_frac = pick(clonal_singleton_cell_frac, "clonal_singleton_cell_frac") + clonal_tail_cycle = R["clonal_tail_cycle"] + rng = new_rng(seed) + panels = load_panel(panel_csv, columns, target_roles, offtarget_roles, shape=panel_shape, + control_feature=control_feature) + samples = list(panels) + + if arm == "vdj": + return _rebuild_vdj_only(run_dir, panels, samples, clonal_profile, clonal_mean_size, + clonal_singleton_cell_frac, offset, columns, + unpaired_frac=unpaired_frac, tail_cycle=clonal_tail_cycle) + + antigen_dir = os.path.join(run_dir, "antigen") + truth_dir = os.path.join(run_dir, "truth") + for d in (antigen_dir, truth_dir): + os.makedirs(d, exist_ok=True) + + # Off-panel barcodes: far from EVERY sample's panel, so a read on one is dropped rather than corrected + # onto a real member. Independent RNG so the per-sample streams are unperturbed. + all_panel_bcs = sorted({bc for p in panels.values() for bc in p.barcodes}) + off_bcs = gen_distinct(new_rng(seed + 99), 4, FEAT_LEN, min_dist=5, avoid=all_panel_bcs) + + profile = QUALITY_PROFILES[quality_profile] + lib_tier_of = {s: profile[i % len(profile)] for i, s in enumerate(samples)} + + ab_rows, con_rows, read_rows, lib_rows = [], [], [], [] + tier_counts = {t: 0 for t in tier_names} + total_reads = 0 + agg_rows = [] + print(f"[real-panel] {len(samples)} samples from {os.path.basename(panel_csv)}, offset-{offset} R2, " + f"regime {regime}") + for sample in samples: + panel = panels[sample] + n_cells = max(1, int(cells_per_sample * (1 + rng.uniform(-cell_jitter, cell_jitter)))) + if barcode_source == "whitelist737k": + cells = load_whitelist_cells(rng, n_cells, assets_dir) + else: + cells = gen_cells(rng, n_cells) + + reads = [] + read_no = 0 + signal_umis = 0 + for cell in cells: + tier = pick_tier(rng, tiers) + per_member, consensus, tier = plant_cell(rng, panel, tier, primary_bias, mag, bgp) + tier_counts[tier] += 1 + con_rows.append((sample, cell, consensus)) + dominant = max(per_member, key=lambda m: (per_member[m], m)) + read_rows.append(( + sample, cell, tier, dominant, per_member[dominant], + panel.role.get(dominant, ""), sum(per_member.values()), + len(per_member), expected_state[tier], + )) + for member, k in per_member.items(): + ab_rows.append((sample, cell, member, k)) + signal_umis += sum(per_member.values()) + read_no = emit_cell_reads( + rng, reads, sample, panel, cell, per_member, offset, seq_error_frac, read_no, + dup_mean=dup_mean, + ) + + signal_reads = len(reads) + matched, panel_assigned, tag = LIBRARY_TIERS[lib_tier_of[sample]] + # Ambient first, then aggregates sized against everything else already in the library. The + # aggregate share is a share of the FINISHED library, and once the barcode universe is large it + # holds most of the non-aggregate UMIs -- sizing against signal alone under-plants by an order of + # magnitude. + ambient_umis = add_ambient(rng, panel, reads, offset, ambient_frac, n_cells=len(cells), + barcode_ratio=ambient_barcode_ratio, dup_mean=dup_mean) or 0 + planted_agg = add_aggregates(rng, panel, reads, offset, aggregates, aggregate_umi_share, + signal_umis + ambient_umis, dup_mean=dup_mean) + for rank, (agg_cell, agg_umis) in enumerate(planted_agg, 1): + agg_rows.append((sample, rank, agg_cell, agg_umis)) + convert_offpanel(rng, reads, off_bcs, 1 - panel_assigned, offset) + add_malformed(rng, reads, matched) + rng.shuffle(reads) + write_fastq_gz(os.path.join(antigen_dir, f"{sample}_R1.fastq.gz"), reads, 1) + write_fastq_gz(os.path.join(antigen_dir, f"{sample}_R2.fastq.gz"), reads, 2) + total_reads += len(reads) + lib_rows.append((sample, len(cells), len(panel.names), len(panel.targets), len(panel.offtargets), + signal_reads, len(reads), lib_tier_of[sample], f"{matched:.2f}", + f"{panel_assigned:.2f}", tag)) + extra = "" + if planted_agg: + extra += f", {len(planted_agg)} aggregates ({sum(k for _, k in planted_agg)} UMIs)" + if ambient_barcode_ratio > 0: + extra += f", ~{int(len(cells) * ambient_barcode_ratio)} ambient barcodes" + # `offtargets` holds whatever --offtarget-roles selected, which is the BASELINE, not necessarily a + # member named off-target. Printing it as "off-target" made a run whose baseline is its Decoy read + # as though the decoy had vanished and the off-targets had become the comparator -- exactly + # backwards. Members matching neither role are counted too rather than silently dropped. + n_other = len(panel.names) - len(panel.targets) - len(panel.offtargets) + split = f"{len(panel.targets)} target / {len(panel.offtargets)} baseline" + if n_other: + split += f" / {n_other} neither" + print(f" {sample}: {len(cells)} cells, {len(panel.names)} members ({split}), " + f"{len(reads)} reads, library {lib_tier_of[sample]} -> expect {tag}{extra}") + + _write_truth(truth_dir, ab_rows, con_rows, read_rows, lib_rows, panels) + _write_sample_metadata(run_dir, samples, panels, lib_tier_of) + uploaded_panel = os.path.join(run_dir, "panel.csv") + shutil.copyfile(panel_csv, uploaded_panel) + + if arm in ("all", "vdj"): + vdj.build( + os.path.join(truth_dir, "panel-canonical.csv"), + os.path.join(truth_dir, "expected-consensus.tsv"), + out_dir=os.path.join(run_dir, "vdj"), + truth_dir=truth_dir, + clonal_profile=clonal_profile, + mean_size=clonal_mean_size, + singleton_cell_frac=clonal_singleton_cell_frac, + unpaired_frac=unpaired_frac, + tail_cycle=clonal_tail_cycle, + ) + + n_cells_total = len(con_rows) + print(f"\n[real-panel] {len(samples)} samples, {n_cells_total} cells, {total_reads} reads -> {run_dir}") + print(" reading tiers: " + ", ".join( + f"{t}={tier_counts[t]} ({tier_counts[t] / max(1, n_cells_total):.0%})" for t in tier_names + )) + print(f" panel to upload: {uploaded_panel}") + if agg_rows: + _write_tsv(os.path.join(truth_dir, "aggregates.tsv"), + ("sample", "rank", "cellId", "umis"), agg_rows) + # The regime decides what the tier table should COME BACK as, so a later --validate-only has to be able + # to recover it. Without this, revalidating a shallow run applies the deep expectations and reports + # five failures on a run that is behaving exactly as intended. + with open(os.path.join(truth_dir, "regime.txt"), "w") as fh: + fh.write(regime + "\n") + return {"samples": samples, "cells": n_cells_total, "reads": total_reads, "tiers": tier_counts, + "panels": panels, "offset": offset, "columns": dict(DEFAULT_COLUMNS, **(columns or {})), + "regime": regime, "tier_table": tiers, "aggregates": agg_rows, + "ambient_barcode_ratio": ambient_barcode_ratio, "dup_mean": dup_mean} + + +def _rebuild_vdj_only(run_dir, panels, samples, clonal_profile, mean_size, singleton_cell_frac, + offset, columns, unpaired_frac=0.0, tail_cycle=None): + """Rebuild the V(D)J arm alone, over the antigen arm already on disk. Returns the same info dict + `build` does, read back from the truth tables rather than recomputed, so the run report stays accurate + without the reads being touched.""" + truth_dir = os.path.join(run_dir, "truth") + consensus = os.path.join(truth_dir, "expected-consensus.tsv") + if not os.path.exists(consensus): + raise SystemExit( + f"no antigen arm under {run_dir} — --arm vdj rebuilds the repertoire over an EXISTING " + "antigen arm. Generate the full run first (drop --arm)." + ) + vdj.build( + os.path.join(truth_dir, "panel-canonical.csv"), + consensus, + out_dir=os.path.join(run_dir, "vdj"), + truth_dir=truth_dir, + clonal_profile=clonal_profile, + mean_size=mean_size, + singleton_cell_frac=singleton_cell_frac, + unpaired_frac=unpaired_frac, + tail_cycle=tail_cycle, + ) + # Keyed off whatever tiers the truth table actually holds, not a fixed list: an --arm vdj rebuild runs + # over an antigen arm that may have been generated under a different regime's tier table. + tiers, cells = {}, 0 + with open(os.path.join(truth_dir, "expected-readings.tsv"), newline="") as fh: + for r in csv.DictReader(fh, delimiter="\t"): + tiers[r["tier"]] = tiers.get(r["tier"], 0) + 1 + cells += 1 + print(f"\n[real-panel] V(D)J arm only, over the existing antigen arm ({cells} cells) -> {run_dir}") + return {"samples": samples, "cells": cells, "reads": 0, "tiers": tiers, "panels": panels, + "offset": offset, "columns": dict(DEFAULT_COLUMNS, **(columns or {}))} + + +def _write_tsv(path, header, rows): + """One tab-separated truth table. Small enough not to warrant a dependency.""" + with open(path, "w", newline="") as fh: + w = csv.writer(fh, delimiter="\t") + w.writerow(list(header)) + w.writerows(rows) + + +def _write_sample_metadata(run_dir, samples, panels, lib_tier_of): + """samples-metadata.tsv -- the per-sample table Samples & Data imports as metadata, keyed by the sample + name so it joins to the same sampleId the three arms share. + + The panel names its samples and says nothing else about them, so Donor and Condition are invented + here: they exist to give downstream grouping something to split on, and a two-arm condition is the + smallest thing that does. The other two columns are NOT invented and are the reason this file is worth + having in this bed: + + LibraryQuality the tier this sample's library was degraded to. Group the QC report on it and the + Quality tag should track it, which is the one claim the library axis makes. + PanelTargets how many on-target members this sample's panel declares. It varies per sample in a + real per-sample panel, and it is what makes *never asked* reachable -- a sample that + never offered an identity cannot have answered about it. + """ + path = os.path.join(run_dir, "samples-metadata.tsv") + with open(path, "w", newline="") as fh: + w = csv.writer(fh, delimiter="\t") + w.writerow(["Sample", "Donor", "Condition", "LibraryQuality", "PanelMembers", "PanelTargets"]) + for i, s in enumerate(samples): + panel = panels[s] + w.writerow([ + s, + f"Donor {i + 1}", + "baseline" if i % 2 == 0 else "stimulated", + lib_tier_of[s], + len(panel.names), + len(panel.targets), + ]) + return path + + +def _write_truth(truth_dir, ab_rows, con_rows, read_rows, lib_rows, panels): + ab_rows.sort() + with open(os.path.join(truth_dir, "expected-abundance.tsv"), "w", newline="") as fh: + w = csv.writer(fh, delimiter="\t") + w.writerow(["sample", "cellId", "feature", "planted_distinct_umis"]) + w.writerows(ab_rows) + con_rows.sort() + with open(os.path.join(truth_dir, "expected-consensus.tsv"), "w", newline="") as fh: + w = csv.writer(fh, delimiter="\t") + w.writerow(["sample", "cellId", "planted_consensus"]) + w.writerows(con_rows) + read_rows.sort() + with open(os.path.join(truth_dir, "expected-readings.tsv"), "w", newline="") as fh: + w = csv.writer(fh, delimiter="\t") + w.writerow(["sample", "cellId", "tier", "dominantFeature", "dominantUmis", "dominantRole", + "totalUmis", "nFeatures", "expectedState"]) + w.writerows(read_rows) + with open(os.path.join(truth_dir, "library-quality.tsv"), "w", newline="") as fh: + w = csv.writer(fh, delimiter="\t") + w.writerow(["sample", "cells", "panelMembers", "targets", "offTargets", "signalReads", + "totalReads", "libraryTier", "matchedFrac", "panelAssignedFrac", "expectedQualityTag"]) + w.writerows(lib_rows) + # The flat (tag, feature) view the VDJ arm's clear-antigen lookup reads. Not a block upload -- the + # block gets the panel file verbatim -- so it lives under truth/ with the rest of the derived state. + with open(os.path.join(truth_dir, "panel-canonical.csv"), "w", newline="") as fh: + w = csv.writer(fh) + w.writerow(["Sample", "tag", "feature", "role"]) + for sample, panel in panels.items(): + for name in panel.names: + w.writerow([sample, panel.barcode[name], name, panel.role.get(name, "")]) + + +def write_run_report(run_dir, info, panel_csv, quality_profile, gate_hint=True, baseline_tag=None): + """Write RUN.md inside the run: the settings this run expects, worked out from the panel that actually + drove it. It lives in the run directory, which is gitignored, and not in the tracked README, because it + names the panel's own samples and columns and the panel is the user's, not this repository's.""" + cols = info["columns"] + panels = info["panels"] + offset = info["offset"] + # Exactly what model/src/pattern.ts assembles from the builder fields below, so the report can be + # compared against the pattern the block shows rather than paraphrasing it. + skip = "N{%d}" % offset if offset else "" + pattern = "^(CELL:N{16})(UMI:N{10})*\\^" + skip + "(FEATURE:N{15})(R2:*)" + role_values = sorted({p.role[n] for p in panels.values() for n in p.names}) + offtarget_values = sorted({p.role[n] for p in panels.values() for n in p.offtargets}) + # Distinct TAGS across the whole panel, which is what the block's panel rung counts + # (`emit_verdicts.py` reads `panel["tag"].n_unique()`), not rows and not per-sample members. + bstate = baseline_state(panels, baseline_tag) + # What the PANEL declares on its own. `--baseline-tag` narrows the simulation to one member, but the + # block's declared rung is reached by a column value, so a run read against a named tag needs the + # panel edited before the block can be pointed at it the same way. + declared = baseline_state(panels) if baseline_tag else bstate + panel_tag_count = bstate["panel_tags"] + n_baseline = len(bstate["seqs"]) + served = bstate["served"] + marked = ", ".join(f"`{v}`" for v in offtarget_values) + # Rows 15, 16 and the comparator section all turn on the same three states, so they are settled here + # rather than as three ternaries that could drift apart. + served_label = "`The panel's own readings`" if served == "panel" else "`No baseline`" + if n_baseline == 1 and len(declared["seqs"]) > 1: + row15 = (marked + f" | **edit the panel first** — {marked} marks {len(declared['seqs'])} tags " + f"here, and this run was read against `{baseline_tag}` alone |") + row16 = ("`Declared baseline tag` | valid only once one member carries the baseline value — " + "see below |") + elif n_baseline == 1: + row15 = marked + " | marks exactly one baseline tag, which is what the block reads against |" + row16 = "`Declared baseline tag` | read every count against that one member |" + elif n_baseline > 1: + row15 = (marked + f" | **leave unset** — these mark {n_baseline} distinct baseline tags, and the " + "block reads against one or none |") + row16 = served_label + " | the declared rung is refused on this panel — see below |" + else: + row15 = "*(nothing to pick)* | no value in that column marks a baseline — see below |" + row16 = served_label + " | nothing carries the baseline role — see below |" + row17 = (f"{PANEL_MIN_MEMBERS} | *default*. This panel holds {panel_tag_count} tags, so the panel's " + "own readings " + ("can serve |" if panel_tag_count >= PANEL_MIN_MEMBERS + else "cannot serve |")) + if n_baseline == 1 and len(declared["seqs"]) > 1: + lead_para = (f"This run was read against `{baseline_tag}` alone, because {marked} marks " + f"**{len(declared['seqs'])} distinct tags** and the block reads against one or " + "none. The block reaches its declared rung through a column VALUE, not by naming a " + "member, so it cannot be pointed at this panel the way this run was read — edit " + f"`panel.csv` so only `{baseline_tag}` carries a baseline value, then set it below.") + declared_bullet = ("- **Declared baseline tag** — what this run simulates, but only after the " + f"panel is edited: as shipped, {marked} marks {len(declared['seqs'])} tags " + "and the block exits rather than combining them.") + elif n_baseline == 1: + lead_para = (f"This panel declares exactly one baseline tag ({marked}), so there IS a declared " + "comparator to point at.") + declared_bullet = ("- **Declared baseline tag** — reads every count against the one member named " + f"in **Values that mark the baseline tag** ({marked}). This is the reading " + "this run is built for.") + elif n_baseline > 1: + lead_para = (f"This panel's role column marks {marked} on **{n_baseline} distinct tags**. The " + "block reads against one baseline tag or none and refuses several outright — " + "reading against several needs a panel column saying which antigens each one " + "belongs to, which the panel format has no room for. So the declared rung is " + "**refused on this panel**, not degraded, and the run is read on " + f"{served_label} instead.") + declared_bullet = ("- **Declared baseline tag** — **refused here.** It reads every count against " + f"ONE member, and {marked} marks {n_baseline}. Pointing the block at this " + "panel with that value set exits with an error rather than producing a grid.") + else: + lead_para = ("The role column names what a member is TO THE QUESTION and carries no value " + "meaning *negative control*, so there is no declared comparator to point at.") + declared_bullet = ("- **Declared baseline tag** — reads every count against the one member named " + "in **Values that mark the baseline tag**. Nothing in this panel carries that " + "role, so it cannot serve.") + # What the reader is left with, and the one lever that changes it. Stating the rung without stating + # the consequence is what let the old report read as though a run were configurable when it was not. + closing = [] + if served == "none": + closing = [ + "", + "**No rung serves this panel, so every reading comes back *unreliable*.** That is a fact " + "about the panel shape, not a defect in this run — the truth tables are still coherent and " + "the FASTQs still carry the planted signal.", + "", + "The one lever is `--baseline-tag `, which names a single member as the " + "comparator the way the block's own dropdown would. The baseline is global BY TAG while this " + "panel is per sample, so a named tag only serves the samples that offer it; cells elsewhere " + "still have no comparator. Re-check without regenerating: " + "`generate.py --real-panel --validate-only --baseline-tag `.", + ] + elif bstate["samples_offering"] and len(bstate["samples_offering"]) < len(panels): + missing = sorted(set(panels) - set(bstate["samples_offering"])) + closing = [ + "", + f"The baseline is global by tag and this panel is per sample: only " + f"{len(bstate['samples_offering'])} of {len(panels)} samples offer it. Cells in " + + ", ".join(f"`{s}`" for s in missing) + + " have no comparator and read *unreliable* whatever the dropdown says.", + ] + lines = [ + "# Run settings", + "", + f"Generated from `{os.path.basename(panel_csv)}` — {len(info['samples'])} samples, " + f"{info['cells']} cells, {info['reads']} reads, regime " + f"**{info.get('regime', 'deep')}**.", + "", + "## Upload", + "", + "| what | where |", + "| --- | --- |", + "| feature-barcode FASTQs | `antigen/*_R{1,2}.fastq.gz` (Fastq dataset) |", + "| panel | `panel.csv` (Xsv-csv), the source panel verbatim |", + "| single-cell V(D)J | `vdj/*.tsv` (Xsv-tsv, import-vdj-data format **AIRR single cell**) |", + "| sample metadata | `samples-metadata.tsv` (Samples & Data → Metadata), keyed by `Sample` |", + "", + "All arms carry the same bare-16nt cell barcode and the same sample names, so one " + "Samples & Data block mints one sampleId per sample across all of them.", + "", + "## Feature Barcode Profiling", + "", + # Named as the block shows them, and ORDERED as the settings drawer shows them, top to bottom. A + # reader configures the block by walking the drawer. A table in any other order makes them hunt for + # each control, and a control named by its internal argument cannot be found at all. + "Named and ordered as they appear in the settings drawer, top to bottom. Rows marked *default* " + "need no change — they are listed so the drawer can be read straight through.", + "", + "| # | control | set it to | why |", + "| --- | --- | --- | --- |", + "| 1 | **Feature-barcode FASTQ dataset** | the `antigen/` dataset | the reads |", + f"| 2 | **Preset** | `Custom feature-barcode kit` | the 10x BEAM preset reads the feature at " + f"offset 0; this data carries a {offset} bp lead-in, so the configurable preset is required |", + "| 3 | **Cell barcode length** | 16 | Read 1 tab of the pattern builder |", + "| 4 | **UMI length** | 10 | Read 1 tab |", + f"| 5 | **Feature barcode offset** | **{offset}** | Read 2 tab. Bases to skip before the feature " + "barcode. Wrong here sends panel-assigned to ~0% and every sample to ALERT, with no other " + "symptom |", + "| 6 | **Feature barcode length** | 15 | Read 2 tab |", + f"| — | **Tag pattern** | `{pattern}` | what the builder produces from rows 3-6. Only visible if " + "you switch the editor to write the pattern by hand |", + "| 7 | **Tag-feature CSV** | `panel.csv` | the panel, verbatim |", + f"| 8 | **Barcode sequence column** | `{cols['sequence']}` | holds the 15 bp feature barcode |", + f"| 9 | **Feature name column** | `{cols['name']}` | holds the antigen name |", + "| 10 | **Run mode** | `Full` | *default*. Preview read-limits the run |", + f"| 11 | **Sample column** | `{cols['sample']}` | " + ( + "**required** — the panel is per-sample and reuses sequences across samples; without it the " + "duplicate-barcode guard fires |" if len(panels) > 1 else + "one sample in this panel; still set it so the panel is read per sample |"), + "| 12 | **Control feature marker (output only)** | *(leave unset)* | marks a member in the " + "output; it does not change any number |", + "| 13 | **Single-cell V(D)J dataset (optional)** | the `vdj/` dataset | needed for the " + "per-clonotype rollup |", + f"| 14 | **Panel column naming the baseline tag** | `{cols['role']}` | values present: " + f"{', '.join(role_values)} |", + f"| 15 | **Values that mark the baseline tag** | {row15}", + f"| 16 | **What sets the baseline** | {row16}", + f"| 17 | *Baseline thresholds* → **Minimum panel size to serve as baseline** | {row17}", + "| 18 | *Baseline thresholds* → **High baseline reading** | 100 | *default*. Counts how many " + "cells read a high baseline; it sets nothing aside |", + "| 19 | *Baseline thresholds* → **Admissibility gate (baseline UMIs)** | *(off)* | *default*. " + "Turn it on to see the `gated` tier read *unreliable* |", + "| 20 | **Panel columns that define an identity** | *(default)* | change only to test identity " + "grouping |", + f"| 21 | **Minimum count** | {FLOOR} | *default*. Readings below it read as zero; the baseline is " + "exempt |", + f"| 22 | **Bound cutoff (0–100)** | {CUTOFF} | *default* |", + "| 23 | **Minimum voting cells** | 1 | *default*. At ~1 cell per clonotype in this regime, " + "raising it leaves most clonotypes unanswered |", + "", + "### The comparator", + "", + lead_para, + "", + "The **What sets the baseline** dropdown offers these, and they answer differently:", + "", + declared_bullet, + # The panel rung GATES on member count, so offering it where it cannot serve sends the reader to a + # choice that silently degrades to no comparator at all. + (f"- **The panel's own readings** — the cell's other readings serve as its background. Needs at " + f"least the number in **Minimum panel size to serve as baseline** ({PANEL_MIN_MEMBERS}); this " + f"panel holds {panel_tag_count} tags, so it " + + ("can serve." if panel_tag_count >= PANEL_MIN_MEMBERS else + "**cannot serve** and silently degrades to no baseline.")), + "- **Each tag's own distribution** — fits one distribution per tag across the sample's cells; " + "needs the cell count and separation set under **Baseline thresholds**.", + "- **No baseline** — every reading comes back *unreliable*.", + *closing, + "", + "### Reading tiers planted", + "", + "| tier | cells | expected state |", + "| --- | --- | --- |", + ] + for tier, _w, _doc in TIERS: + n = info["tiers"].get(tier, 0) + table = EXPECTED_STATE if info.get("regime", "deep") == "deep" else EXPECTED_STATE_SHALLOW + lines.append(f"| {tier} | {n} ({n / max(1, info['cells']):.0%}) | {table[tier]} |") + lines += [ + "", + "Per-cell ground truth is `truth/expected-readings.tsv` (one row per cell: tier, dominant " + "member, its UMIs, its role, expected state).", + "", + f"Library quality is the other axis, profile `{quality_profile}` — see " + "`truth/library-quality.tsv` for each sample's matched / panel-assigned fractions and the " + "Quality tag it should show.", + ] + if gate_hint: + lines += [ + "", + "The `gated` tier only reads as *unreliable* with the admissibility gate ON. Set the gate " + "threshold near 300 comparator UMIs to catch it; leave it off and those cells read *bound*.", + ] + if info.get("regime") == "shallow": + # aggregates holds one row PER SAMPLE per aggregate, so the per-library count is the quotient. + # Reporting the row count would say "20 barcodes" of a four-sample run that planted five each. + aggs = info.get("aggregates") or [] + per_lib = len(aggs) // max(1, len(info["samples"])) + ratio = info.get("ambient_barcode_ratio") or 0 + lines += [ + "", + "## What the shallow regime changes", + "", + "This run stands in for real in-vivo BEAM libraries rather than for a public 10x " + "BEAM run. Three things follow, and the first is the one that surprises people.", + "", + "**Expect `cells detected` to read in the hundreds of thousands, and `median UMIs / cell` " + "to read 1.** There is nothing to configure for this and nothing has gone wrong. The block " + "does no cell calling and offers no whitelist control, so what it counts as a cell is every " + f"distinct barcode it saw. This run plants about {ratio:.0f}x as many ambient barcodes as " + "real cells, each carrying a median of one UMI, which is the shape a real library has. Every " + "QC number derived from that count inherits it.", + "", + f"**Antigen aggregates are present and unfiltered.** {per_lib} barcodes per library hold " + "roughly 59% of its UMIs, the largest about 18% on its own. Cell Ranger removes this " + "population before cell calling; this block does not. They are recorded in " + "`truth/aggregates.tsv`, so anything they distort — a panel median, a comparator, a " + "*cells detected* count — can be traced back to them.", + "", + "**Expect roughly 1-3% of readings to come back bound.** A real in-vivo pipeline produces " + "1.4% and " + "2.9% across the two measured libraries, and this run is calibrated to land in that band. A " + "shallow run showing a clean majority of confident binders has lost the regime.", + ] + with open(os.path.join(run_dir, "RUN.md"), "w") as fh: + fh.write("\n".join(lines) + "\n") + + +# --- offline validation -------------------------------------------------------------------------- + +def validate(run_dir, panel_csv=None, columns=None, sample_check=None, regime=None, baseline_tag=None, + target_roles=DEFAULT_TARGET_ROLES, offtarget_roles=DEFAULT_OFFTARGET_ROLES): + """Check a generated run without a backend. Re-derives each checked sample's per-(cell, member) + distinct-UMI counts straight from the FASTQ pair and compares them to the planted truth -- the one + test that proves the reads say what the truth table claims. Also checks read geometry, per-sample + barcode uniqueness, the tier mix, and that the VDJ arm's cell ids are the antigen arm's. + + `sample_check` limits the FASTQ re-derivation to one sample. The pass is linear in reads, so on a + cohort-scale run checking every sample is slow for no extra coverage. Returns True on a clean pass.""" + import gzip + + # Prefer what the run RECORDS over what the caller guessed: a --validate-only invocation carries + # whatever --regime happened to be on the command line, which need not be the one that built it. + recorded = os.path.join(run_dir, "truth", "regime.txt") + if os.path.exists(recorded): + with open(recorded) as fh: + regime = (fh.read().strip() or None) or regime + regime = regime or "deep" + failures, checks = [], 0 + + def ok(cond, label): + nonlocal checks + checks += 1 + if not cond: + failures.append(label) + + panel_path = panel_csv or os.path.join(run_dir, "panel.csv") + # Shape-aware AND role-aware: a run generated from a narrow panel must validate against a narrow read + # of it, and one generated with custom role words must use the SAME words here. Re-reading with the + # defaults silently reclassifies every member -- a panel whose comparator is its `Decoy` row comes back + # with five off-target baselines instead of one, and the declared rung then reports itself refused on a + # run where it serves. + panels = load_panel(panel_path, columns, target_roles, offtarget_roles, shape="auto") + truth = os.path.join(run_dir, "truth") + + for sample, panel in panels.items(): + ok(len(set(panel.barcodes)) == len(panel.barcodes), f"{sample}: barcodes unique within the sample") + ok(all(len(b) == FEAT_LEN for b in panel.barcodes), f"{sample}: every barcode is {FEAT_LEN} bp") + # A 1 bp read error must not turn one member into another, or the planted counts and the + # recoverable counts are different numbers and no per-cell reading can be checked at all. + bcs = panel.barcodes + closest = min( + (sum(1 for x, y in zip(a, b) if x != y) for i, a in enumerate(bcs) for b in bcs[i + 1:]), + default=FEAT_LEN, + ) + ok(closest >= 3, f"{sample}: panel members are >= 3 bp apart (closest pair {closest} bp)") + + # tier mix present and complete + tiers = {} + with open(os.path.join(truth, "expected-readings.tsv"), newline="") as fh: + rows = list(csv.DictReader(fh, delimiter="\t")) + for r in rows: + tiers[r["tier"]] = tiers.get(r["tier"], 0) + 1 + # `offtarget` and `gated` both need a COMPARATOR member to plant a dominant on, and `plant_cell` + # degrades them when the sample offers none. A panel that declares no baseline at all -- every member + # either a target or an off-target, with no negative control -- therefore cannot carry them, and + # requiring them would fail a run that is faithfully reproducing that panel. Every other tier is + # unconditional. + needs_comparator = {"offtarget", "gated"} + any_comparator = any(pnl.offtargets for pnl in panels.values()) + for tier in TIER_NAMES: + if tier in needs_comparator and not any_comparator: + continue + ok(tiers.get(tier, 0) > 0, f"tier {tier} is present ({tiers.get(tier, 0)} cells)") + ok(len(rows) > 0, "readings truth is non-empty") + + # Each sample's panel-assigned fraction. It is the recovery bound: a degraded library has a share of + # its reads deliberately rewritten onto off-panel barcodes, so its planted UMIs are NOT all recoverable + # and a fixed bar would fail every sample the profile degrades on purpose. + panel_assigned = {} + lq = os.path.join(truth, "library-quality.tsv") + if os.path.exists(lq): + with open(lq, newline="") as fh: + for r in csv.DictReader(fh, delimiter="\t"): + panel_assigned[r["sample"]] = float(r["panelAssignedFrac"]) + + # planted abundance, indexed by (sample, cell, member) + planted = {} + with open(os.path.join(truth, "expected-abundance.tsv"), newline="") as fh: + for r in csv.DictReader(fh, delimiter="\t"): + planted[(r["sample"], r["cellId"], r["feature"])] = int(r["planted_distinct_umis"]) + + check_samples = [sample_check] if sample_check else list(panels) + for sample in check_samples: + panel = panels[sample] + by_seq = {} + for name in panel.names: + by_seq.setdefault(panel.barcode[name], name) + r1p = os.path.join(run_dir, "antigen", f"{sample}_R1.fastq.gz") + r2p = os.path.join(run_dir, "antigen", f"{sample}_R2.fastq.gz") + ok(os.path.exists(r1p) and os.path.exists(r2p), f"{sample}: FASTQ pair exists") + if not (os.path.exists(r1p) and os.path.exists(r2p)): + continue + # offset is whatever position the panel barcodes actually sit at. Read it off the first + # structurally valid read rather than trusting an argument. + seen = {} + n_reads = 0 + offsets = set() + with gzip.open(r1p, "rt") as f1, gzip.open(r2p, "rt") as f2: + while True: + h1, s1, _p1, _q1 = (f1.readline(), f1.readline(), f1.readline(), f1.readline()) + h2, s2, _p2, _q2 = (f2.readline(), f2.readline(), f2.readline(), f2.readline()) + if not h1 or not h2: + break + n_reads += 1 + ok(h1.split()[0] == h2.split()[0], f"{sample}: R1/R2 read names line up") if n_reads == 1 else None + s1, s2 = s1.strip(), s2.strip() + if len(s1) < CELL_LEN + UMI_LEN: + continue + cell, umi = s1[:CELL_LEN], s1[CELL_LEN:CELL_LEN + UMI_LEN] + for off in (0, 10): + seq = s2[off:off + FEAT_LEN] + member = by_seq.get(seq) + if member: + offsets.add(off) + seen.setdefault((cell, member), set()).add(umi) + break + ok(n_reads > 0, f"{sample}: FASTQ is non-empty ({n_reads} reads)") + ok(len(offsets) == 1, f"{sample}: one feature offset in the file (found {sorted(offsets)})") + # Every planted (cell, member) must be recoverable from the reads, and NOTHING must be recoverable + # that was not planted. Two claims, because they fail for different reasons and only one tolerates + # slack: + # + # over-recovery is bounded near zero. A member reading MORE distinct UMIs than were planted is + # either a panel whose members are close enough that a 1 bp error turns one into another, which + # the >= 3 bp check above rules out, or an AMBIENT read whose random 16-mer cell barcode happened + # to equal a real one. The second is real and unavoidable: at ~1M ambient reads against ~7k cells + # it lands about twice per sample (n_ambient * n_cells / 4^16), as a real library does. So the bar + # is a rate rather than zero, and a rate this small can only be met by ambient collision. + # + # under-recovery is expected, and how much depends on the sample's LIBRARY tier. A UMI is lost + # when every one of its reads is unrecoverable, whether it took a feature-barcode error (1.5% of + # reads) or was rewritten onto an off-panel barcode (whatever the tier's panel-assigned fraction + # leaves). Multi-read UMIs survive better than reads do, so recovery sits ABOVE the panel-assigned + # fraction, never below it and never above 1. Those two are the bound. Checked in aggregate rather + # than per pair: per pair the loss is a coin toss, and in aggregate it is the rate worth + # asserting. + n_pairs = sum(1 for key in planted if key[0] == sample) + over = [key for key, k in planted.items() + if key[0] == sample and len(seen.get((key[1], key[2]), ())) > k] + over_rate = len(over) / max(1, n_pairs) + ok(over_rate < 0.001, f"{sample}: over-recovery stays at the ambient-collision rate " + f"({len(over)}/{n_pairs} = {over_rate:.4%}, must be < 0.1%)") + tot_planted = sum(k for key, k in planted.items() if key[0] == sample) + tot_got = sum(len(seen.get((key[1], key[2]), ())) for key in planted if key[0] == sample) + frac = tot_got / max(1, tot_planted) + # The floor is the per-READ survival probability: a read survives when it was neither rewritten + # off-panel (the library tier) nor hit by a feature-barcode error (SEQ_ERROR_FRAC). A UMI can only + # do BETTER than one of its reads, because losing it needs every one of its reads to fail, so + # per-read survival is a true lower bound on UMI recovery, and a tight one. + floor_frac = panel_assigned.get(sample, 1.0) * (1 - SEQ_ERROR_FRAC) + ok(floor_frac <= frac <= 1.0, + f"{sample}: planted UMIs recovered from the FASTQs ({tot_got}/{tot_planted} = {frac:.2%}, " + f"expected between this library's per-read survival {floor_frac:.2%} and 100%)") + + # metadata must name exactly the samples the arms carry, or a grouping column silently covers only part + # of the run + meta_path = os.path.join(run_dir, "samples-metadata.tsv") + ok(os.path.exists(meta_path), "samples-metadata.tsv exists") + if os.path.exists(meta_path): + with open(meta_path, newline="") as fh: + meta = list(csv.DictReader(fh, delimiter="\t")) + ok({r["Sample"] for r in meta} == set(panels), + "metadata names exactly the panel's samples") + ok(all(int(r["PanelMembers"]) == len(panels[r["Sample"]].names) for r in meta), + "metadata's panel sizes match the panel") + + # cross-arm barcode alignment: every VDJ cell must be an antigen cell of the same sample + vdj_dir = os.path.join(run_dir, "vdj") + if os.path.isdir(vdj_dir): + antigen_cells = {} + with open(os.path.join(truth, "expected-consensus.tsv"), newline="") as fh: + for r in csv.DictReader(fh, delimiter="\t"): + antigen_cells.setdefault(r["sample"], set()).add(r["cellId"]) + for fn in sorted(os.listdir(vdj_dir)): + if not fn.endswith(".tsv"): + continue + sample = fn[:-4] + with open(os.path.join(vdj_dir, fn), newline="") as fh: + cells = {r["cell_id"] for r in csv.DictReader(fh, delimiter="\t")} + ok(cells and cells <= antigen_cells.get(sample, set()), + f"{sample}: every VDJ cell_id is an antigen cell of the same sample ({len(cells)} cells)") + + # --- the reading, simulated ------------------------------------------------------------------- + # The tiers are promises about verdicts. Predict every verdict from the truth with the block's own rule + # and check the promises, so a magnitude that drifts out of its tier fails here rather than surfacing + # as a puzzling run. + # + # Which rungs this panel can even be read on, under the CURRENT rule. A bed whose panel cannot serve a + # comparator is a fact about the panel, not a failure of the bed, and the report has to say which. + rungs = {} + for src in ("declared", "panel", "none"): + try: + rungs[src] = simulate_verdicts(run_dir, panels, source=src, baseline_tag=baseline_tag) + except BaselineRefused as exc: + rungs[src] = exc + # Report on the best rung that actually served: a declared single baseline, else the panel, else none. + chosen = None + for src in ("declared", "panel", "none"): + r = rungs[src] + if not isinstance(r, BaselineRefused) and r[3] == src: + chosen = src + break + if chosen is None: + chosen = "none" + per_tier, totals, multi, served, cells_bound, n_cells_sim = simulate_verdicts( + run_dir, panels, source="none", baseline_tag=baseline_tag) + else: + per_tier, totals, multi, served, cells_bound, n_cells_sim = rungs[chosen] + try: + gated_tier = simulate_verdicts(run_dir, panels, source=chosen, gate=300, + baseline_tag=baseline_tag)[0] + except BaselineRefused: + gated_tier = {t: {} for t in TIER_NAMES} + n_of = {t: sum(per_tier[t].values()) for t in TIER_NAMES} + + def share(table, tier, state): + """`state` as a share of the tier's cells. For bound / not bound the denominator is the cells whose + comparison could be MADE -- a cell with no off-target reading has no comparator and reads + unreliable, and folding those into the denominator makes a claim about binding depend on how many + off-target members the sample happens to declare. `unreliable` keeps the full denominator, because + that is the number being asked about.""" + full = sum(table[tier].values()) + if not full: + return 0.0 + if state == "unreliable": + return table[tier].get(state, 0) / full + comparable = full - table[tier].get("unreliable", 0) + return table[tier].get(state, 0) / comparable if comparable else 0.0 + + print(f"\n[verdict simulation] regime {regime} — every cell's DOMINANT identity, floor {FLOOR} / " + f"cutoff {CUTOFF}, no thin-reference line (the block removed it)") + for src in ("declared", "panel", "none"): + r = rungs[src] + if isinstance(r, BaselineRefused): + note = f"REFUSED — {r}" + elif r[3] != src: + # The two rungs degrade for different reasons and saying so matters: one is fixed by naming a + # baseline in the panel, the other cannot be fixed by a panel this size at all. + why = ("no tag carries the baseline role" if src == "declared" + else f"panel holds fewer than {PANEL_MIN_MEMBERS} tags " + f"({len({p.barcode[n] for p in panels.values() for n in p.names})})") + note = f"cannot serve ({why}) -> degrades to none" + else: + note = "serves" + (" <- reported below" if src == chosen else "") + print(f" rung {src:<10} {note}") + print(" bound / not bound are shares of the cells that HAD a comparator; unreliable is of all of them") + print(f" {'tier':<14}{'cells':>7}{'bound':>9}{'not bound':>11}{'unreliable':>12} bound on 2+") + for tier in TIER_NAMES: + n = n_of[tier] + print(f" {tier:<14}{n:>7}" + f"{share(per_tier, tier, 'bound'):>9.0%}" + f"{share(per_tier, tier, 'not bound'):>11.0%}" + f"{share(per_tier, tier, 'unreliable'):>12.0%}" + f"{(multi.get(tier, 0) / n if n else 0):>14.0%}") + grid = sum(totals.values()) + print(" whole grid (every cell x every identity its sample offered): " + + ", ".join(f"{k} {v} ({v / max(1, grid):.0%})" for k, v in sorted(totals.items()))) + print(f" cells reaching a bound identity: {cells_bound}/{n_cells_sim} " + f"({cells_bound / max(1, n_cells_sim):.2%}) — the metric a real run's per-cell table reports") + + grid_bound = totals.get("bound", 0) / max(1, grid) + if chosen == "none": + # No rung could serve, so EVERY reading is unreliable for want of a comparator. The correct answer + # for this panel under the current rule rather than a defect in the bed, so asserting the regime's + # bound share here would fail a run behaving exactly as the block would. + # + # It happens because the baseline is global BY TAG while a per-sample panel's comparators are per + # sample: `declared` refuses several tags, `panel` needs PANEL_MIN_MEMBERS, and a small per-sample + # panel satisfies neither. Name one tag with --baseline-tag to read on `declared`, accepting that + # cells in samples not offering that tag still have no comparator. + ok(totals.get("unreliable", 0) == grid, + f"no rung serves this panel, so every reading is unreliable ({totals.get('unreliable', 0)}" + f"/{grid})") + ok(grid_bound == 0.0, f"nothing reads bound without a comparator ({grid_bound:.1%})") + elif regime == "deep": + ok(all(totals.get(k, 0) > 0 for k in ("bound", "not bound")), + "both settled states occur in the run") + ok(share(per_tier, "strong", "bound") >= 0.98, + f"strong reads bound ({share(per_tier, 'strong', 'bound'):.0%})") + ok(share(per_tier, "good", "bound") >= 0.85, + f"good reads bound ({share(per_tier, 'good', 'bound'):.0%})") + ok(share(per_tier, "medium", "bound") >= 0.10 and share(per_tier, "medium", "not bound") >= 0.10, + f"medium straddles the line (bound {share(per_tier, 'medium', 'bound'):.0%} / " + f"not bound {share(per_tier, 'medium', 'not bound'):.0%})") + ok(share(per_tier, "weak", "not bound") >= 0.98, + f"weak reads not bound ({share(per_tier, 'weak', 'not bound'):.0%})") + ok(share(per_tier, "noise", "bound") == 0.0, + f"noise never reads bound ({share(per_tier, 'noise', 'bound'):.0%})") + ok(share(per_tier, "offtarget", "bound") <= 0.05, + f"offtarget does not read bound against itself ({share(per_tier, 'offtarget', 'bound'):.0%})") + cr_comparable = n_of["crossreactive"] - per_tier["crossreactive"].get("unreliable", 0) + cr_multi = multi.get("crossreactive", 0) / max(1, cr_comparable) + ok(cr_multi >= 0.85, + f"crossreactive binds two identities at once ({cr_multi:.0%} of its comparable cells)") + ok(share(gated_tier, "gated", "unreliable") >= 0.95, + f"gated is set aside with the gate at 300 ({share(gated_tier, 'gated', 'unreliable'):.0%})") + else: + # SHALLOW asserts something different, because at this depth the block's line is not reachable from + # most cells and asserting that it is would assert a fiction. Two things are checked instead, both + # properties of real measured output rather than of the bed. + # + # 1. The bound share of the whole grid stays in the band a real pipeline produces: 1.4% and 2.9% + # across the two measured libraries. A shallow run coming back with 20% bound has lost the + # regime, and one coming back with 0% has nothing to test against. + # 2. Signal ORDERING holds. Absolute rates are all low, so the invariant that carries meaning is + # monotonicity: a tier planted with more signal must never read bound LESS often than one planted + # with less. That catches a broken comparator or an inverted score without pretending to know + # where the line sits. + ok(all(totals.get(k, 0) > 0 for k in ("bound", "not bound")), + "both settled states occur in the run") + cell_bound = cells_bound / max(1, n_cells_sim) + ok(0.008 <= cell_bound <= 0.05, + f"cells reaching bound match the 1.4-2.9% real in-vivo runs report ({cell_bound:.2%})") + # A sparse comparator no longer produces *unreliable* -- that was the thin-reference line, and the + # block removed it. With a comparator serving and the gate off, nothing is unreliable, and a run + # that still shows some has either lost its comparator for part of the panel or been gated. + ok(totals.get("unreliable", 0) == 0, + f"a served comparator leaves nothing unreliable ({totals.get('unreliable', 0)} of {grid})") + ladder = [(t, share(per_tier, t, "bound")) for t in ("strong", "good", "medium", "weak", "noise")] + monotone = all(ladder[i][1] >= ladder[i + 1][1] - 1e-9 for i in range(len(ladder) - 1)) + ok(monotone, "bound rate falls monotonically from strong to noise (" + + " >= ".join(f"{t} {v:.0%}" for t, v in ladder) + ")") + ok(share(per_tier, "noise", "bound") == 0.0, + f"noise never reads bound ({share(per_tier, 'noise', 'bound'):.0%})") + ok(share(per_tier, "offtarget", "bound") <= 0.05, + f"offtarget does not read bound against itself ({share(per_tier, 'offtarget', 'bound'):.0%})") + + print(f"\n[validate] {checks - len(failures)}/{checks} PASS") + for f in failures: + print(f" FAIL: {f}") + return not failures + + +# --- verdict simulation -------------------------------------------------------------------------- +# +# The block's own reading rule, re-implemented over the truth tables so a run can be checked BEFORE it +# reaches a backend. Its whole value is that it fails when the planted magnitudes do not land where the +# tier names claim: a tier is a promise about a verdict, and a promise nothing checks drifts. +# +# The rule, from software/per-cell-metrics/src/verdict.py: +# 1. counts below the floor read as zero, except a comparator's; +# 2. the cell's comparator reading is the MAX over its reference tags; +# 3. score = (1 - I_0.925(count + 1, reference + 3)) * 100; +# 4. no comparator -> unreliable; comparator below the thin line -> unreliable; comparator at or above +# the gate -> unreliable; else score >= cutoff -> bound, otherwise not bound. + +FLOOR = 4 +CUTOFF = 75 +# There is no thin-reference line in the block, and this must not simulate one. A low comparator is not a +# reason to call a reading unreliable: the comparison runs, and a comparator of 0 is a real comparison that +# any count clearing the floor beats. Simulating such a line models a rule the block does not have, and +# hides the failure that matters, which is the opposite one: a run that looks broken reads as +# spectacularly successful. +# +# The panel rung GATES on how many members the panel holds instead. Below the minimum, comparing a count +# against a handful of other antigens is not a background estimate, so the rung refuses to serve. +PANEL_MIN_MEMBERS = 25 + + +class BaselineRefused(Exception): + """The panel declares more than one baseline tag, which the block refuses rather than combines. + + Not a simulation limitation -- it is a hard exit in `verdict.py`, because reading against several + baselines needs a panel column saying which antigens each one belongs to, and the panel format has no + such column. Raised here so a bed whose panel cannot be read that way says so, rather than quietly + reporting a grid the block would never produce.""" + + +def baseline_sequences(panels, baseline_tag=None): + """The distinct baseline TAGS the panel declares, as sequences. + + Keyed on the sequence, not the antigen name: one sequence can carry different names in different + samples, and the block keys on the tag. `baseline_tag` names one member outright, mirroring the + block's dropdown.""" + if baseline_tag: + return {p.barcode[n] for p in panels.values() for n in p.names + if n == baseline_tag or p.barcode[n] == baseline_tag} + return {p.barcode[n] for p in panels.values() for n in p.offtargets} + + +def baseline_state(panels, baseline_tag=None, min_members=PANEL_MIN_MEMBERS): + """Which comparator rung this panel can be read on, and why. + + Resolved once so the run report and the verdict simulation cannot disagree. The block counts distinct + baseline TAGS, not distinct role VALUES: one role value spread over five sequences is five baseline + tags, and `verdict.py` refuses more than one. Keying the report on role values instead told the reader + a panel had a declared comparator where the block would exit. + + `served` is the best rung reachable after the reader changes the dropdown, which is what `validate` + reports on -- the declared rung refusing does not fall back on its own.""" + seqs = baseline_sequences(panels, baseline_tag) + panel_tags = len({p.barcode[n] for p in panels.values() for n in p.names}) + if len(seqs) == 1: + served = "declared" + elif panel_tags >= min_members: + served = "panel" + else: + served = "none" + offering = sorted(s for s, p in panels.items() if any(p.barcode[n] in seqs for n in p.names)) + return {"seqs": seqs, "panel_tags": panel_tags, "refused": len(seqs) > 1, + "served": served, "samples_offering": offering} + + +BETA_X, BETA_A_OFFSET, BETA_B_OFFSET = 0.925, 1, 3 + + +def _betacf(a, b, x, maxit=300, eps=3e-16, fpmin=1e-300): + """Continued fraction for the incomplete beta function (modified Lentz). Standard formulation. The + block itself calls scipy, which this bed does not depend on.""" + import math + + qab, qap, qam = a + b, a + 1.0, a - 1.0 + c = 1.0 + d = 1.0 - qab * x / qap + if abs(d) < fpmin: + d = fpmin + d = 1.0 / d + h = d + for m in range(1, maxit + 1): + m2 = 2 * m + aa = m * (b - m) * x / ((qam + m2) * (a + m2)) + d = 1.0 + aa * d + if abs(d) < fpmin: + d = fpmin + c = 1.0 + aa / c + if abs(c) < fpmin: + c = fpmin + d = 1.0 / d + h *= d * c + aa = -(a + m) * (qab + m) * x / ((a + m2) * (qap + m2)) + d = 1.0 + aa * d + if abs(d) < fpmin: + d = fpmin + c = 1.0 + aa / c + if abs(c) < fpmin: + c = fpmin + d = 1.0 / d + de = d * c + h *= de + if abs(de - 1.0) < eps: + break + _ = math + return h + + +def specificity_score(count, reference): + """The block's specificity score, 0-100, in the standard library.""" + import math + + a = count + BETA_A_OFFSET + b = reference + BETA_B_OFFSET + x = BETA_X + if x <= 0: + cdf = 0.0 + elif x >= 1: + cdf = 1.0 + else: + lb = math.lgamma(a + b) - math.lgamma(a) - math.lgamma(b) + bt = math.exp(lb + a * math.log(x) + b * math.log1p(-x)) + cdf = bt * _betacf(a, b, x) / a if x < (a + 1.0) / (a + b + 2.0) else 1.0 - bt * _betacf(b, a, 1.0 - x) / b + return (1.0 - cdf) * 100.0 + + +def simulate_verdicts(run_dir, panels, floor=FLOOR, cutoff=CUTOFF, gate=None, + source="declared", baseline_tag=None, min_members=PANEL_MIN_MEMBERS): + """Predict every (cell, identity) state from the truth tables under the block's CURRENT rule. + + Returns (per_tier, totals, per_tier_multi, served). `per_tier` maps tier -> {state: n} counted over + each cell's DOMINANT identity. `totals` counts every position in the grid, including the silent ones, + since an identity a cell was offered and did not read answers *not bound* rather than nothing. + `served` is the rung that actually served -- never the one asked for unless it could serve. + + The rule, as `verdict.py` now has it: + + * The floor zeroes a non-baseline reading below it. The baseline is EXEMPT, because the floor removes + what is not evidence of binding and the baseline is not evidence of binding. + * `declared` reads against exactly ONE baseline tag. A panel declaring several is REFUSED, not + combined -- the block used to take the highest across them and no longer does. + * `panel` reads against the median of the cell's own readings, and only serves at all when the panel + holds `min_members` tags. Below that it degrades to no comparator. + * There is no thin-reference line. A comparator of 0 is a real comparison. + * A reading is unreliable only where there is NO comparator, or where the gate set the cell aside. + """ + truth = os.path.join(run_dir, "truth") + counts = {} + with open(os.path.join(truth, "expected-abundance.tsv"), newline="") as fh: + for r in csv.DictReader(fh, delimiter="\t"): + counts.setdefault((r["sample"], r["cellId"]), {})[r["feature"]] = int(r["planted_distinct_umis"]) + tier_of, dominant_of = {}, {} + with open(os.path.join(truth, "expected-readings.tsv"), newline="") as fh: + for r in csv.DictReader(fh, delimiter="\t"): + tier_of[(r["sample"], r["cellId"])] = r["tier"] + dominant_of[(r["sample"], r["cellId"])] = r["dominantFeature"] + + # panel_size the way the block computes it: distinct TAGS across the whole panel, not per sample + # (`emit_verdicts.py` reads `panel["tag"].n_unique()`), so a per-sample panel does not clear the gate + # by being counted several times. + panel_size = len({p.barcode[n] for p in panels.values() for n in p.names}) + + # Baseline tags are global: a tag is the comparator in every sample or in none. + baseline_seqs = baseline_sequences(panels, baseline_tag) + + served = source + if source == "declared": + if not baseline_seqs: + served = "none" + elif len(baseline_seqs) > 1: + raise BaselineRefused( + f"the panel declares {len(baseline_seqs)} baseline tags and the block reads against one " + "or none. Name a single tag with --baseline-tag, or read with source 'panel' or 'none'." + ) + elif source == "panel" and panel_size < min_members: + served = "none" + + per_tier = {t: {} for t in TIER_NAMES} + per_tier_multi = {t: 0 for t in TIER_NAMES} + totals = {} + cells_bound = 0 + for key, per in counts.items(): + sample, _cell = key + panel = panels[sample] + seq_of = {n: panel.barcode[n] for n in panel.names} + local_baseline = [n for n in panel.names if seq_of[n] in baseline_seqs] + + if served == "none": + reference = None + elif served == "declared": + # Offered but unread is a reading of zero. Not offered at all is no comparator. The block + # tests membership rather than defaulting to 0 for exactly this reason. + reference = max((per.get(n, 0) for n in local_baseline), default=None) if local_baseline else None + else: # panel + observed = [v for v in per.values()] + reference = int(st.median(observed)) if observed else None + + tier = tier_of.get(key, "?") + n_bound = 0 + for name in panel.names: + raw = per.get(name, 0) + count = raw if (name in local_baseline or raw >= floor) else 0 + if reference is None: + state = "unreliable" + elif gate is not None and reference >= gate: + state = "unreliable" + else: + state = "bound" if specificity_score(count, reference) >= cutoff else "not bound" + totals[state] = totals.get(state, 0) + 1 + if state == "bound": + n_bound += 1 + if name == dominant_of.get(key): + per_tier[tier][state] = per_tier[tier].get(state, 0) + 1 + if n_bound >= 1: + cells_bound += 1 + if n_bound >= 2: + per_tier_multi[tier] = per_tier_multi.get(tier, 0) + 1 + # cells_bound is the metric a real run publishes: its per-cell table carries ONE row per cell with that + # cell's MAX specificity, so a reported "1.4% above the cutoff" is 1.4% of CELLS, not of the + # (cell x identity) grid. Asserting the grid share instead compares against the wrong denominator and + # moves with panel width rather than with the reading. + return per_tier, totals, per_tier_multi, served, cells_bound, len(counts) diff --git a/software/test-data/manual/lib/validate.py b/software/test-data/manual/lib/validate.py index dde7c6e..7a0a4ae 100644 --- a/software/test-data/manual/lib/validate.py +++ b/software/test-data/manual/lib/validate.py @@ -1,16 +1,16 @@ -"""Offline viability suite for a full multiomics run (antigen + VDJ + GEX arms, colocated under one -run dir). +"""Offline viability suite for a full multiomics run (antigen + VDJ + GEX arms, colocated under one run +dir). -Proves the synthetic data will actually flow through the pipeline BEFORE any backend run — the #1 -failure is the convergence inner-join producing empty output because cell barcodes don't line up across +Proves the synthetic data will actually flow through the pipeline BEFORE any backend run. The number one +failure is the convergence inner-join producing empty output because cell barcodes do not line up across arms. Runs three things: 1. Per-arm schema/geometry checks (antigen FASTQ, VDJ AIRR-sc TSV, panel, GEX matrix). 2. Barcode alignment across arms (the load-bearing test). - 3. JOIN SIMULATION — emulate vdj-multiomic-integration end to end, offline: derive per-(cell,antigen) + 3. JOIN SIMULATION -- emulate vdj-multiomic-integration end to end, offline: derive per-(cell, antigen) distinct-UMI from the antigen FASTQ, build the cell->clonotype linker from the VDJ pairing, inner-join on cellId, group by clonotype, take the dominant antigen, and assert every - clear-antigen clonotype's dominant == the planted antigen (and output non-empty). + clear-antigen clonotype's dominant == the planted antigen, and that output is non-empty. """ import csv @@ -89,7 +89,7 @@ def _derive_antigen_umis(antigen_dir, donor, panel): def _load_vdj(vdj_dir, donor): - """Parse the AIRR-sc TSV; return per-cell clone key + schema stats.""" + """Parse the AIRR-sc TSV. Returns the per-cell clone key plus schema stats.""" path = vdj_dir / f"{donor}.tsv" required = {"cell_id", "junction", "v_call", "j_call", "duplicate_count"} per_cell = defaultdict(lambda: {"IGH": [], "IGK": []}) @@ -130,7 +130,8 @@ def _load_vdj(vdj_dir, donor): def _load_gex(gex_dir, donor): - """Parse the genes-in-rows count CSV; return cell set, per-cell plasma-marker mean, validity.""" + """Parse the genes-in-rows count CSV. Returns the cell set, the per-cell plasma-marker mean, and + validity.""" with open(gex_dir / f"{donor}.csv", newline="") as fh: rows = list(csv.reader(fh)) header = rows[0] diff --git a/software/test-data/manual/lib/vdj.py b/software/test-data/manual/lib/vdj.py index 37db999..9b2bbe6 100644 --- a/software/test-data/manual/lib/vdj.py +++ b/software/test-data/manual/lib/vdj.py @@ -1,13 +1,13 @@ """VDJ (BCR, single-cell) arm generator. -Builds a *coherent* single-cell BCR repertoire ON TOP of the antigen arm: it reads each cell's planted -dominant antigen (from the antigen arm's expected-consensus.tsv) and groups cells into clonotypes so a -clonotype's cells bind the same antigen — exactly the biology vdj-multiomic-integration surfaces -(per-clonotype antigen binding -> antibody lead selection). - -Output = one AIRR-`airr-sc` rearrangement TSV per donor (import-vdj-data, format "AIRR single cell", -cellKeyMode=direct -> cell_id used verbatim). cell_id is the SAME bare-16nt barcode the antigen FASTQ -carries — the canonical cellId the convergence inner-join lines up on. +Builds a *coherent* single-cell BCR repertoire ON TOP of the antigen arm. It reads each cell's planted +dominant antigen, from the antigen arm's expected-consensus.tsv, and groups cells into clonotypes so a +clonotype's cells bind the same antigen. That is exactly the biology vdj-multiomic-integration surfaces: +per-clonotype antigen binding, feeding antibody lead selection. + +Output is one AIRR-`airr-sc` rearrangement TSV per donor (import-vdj-data, format "AIRR single cell", +cellKeyMode=direct, so cell_id is used verbatim). cell_id is the SAME bare-16nt barcode the antigen +FASTQ carries, the canonical cellId the convergence inner-join lines up on. """ import csv @@ -68,7 +68,7 @@ def make_junction(rng, n_codons): def make_bcr(rng, heavy_only=False): """One rearrangement's sequences: paired heavy+light by default, heavy-only (IGH, no light chain - — the customer's VHH single-domain antibody) when heavy_only is set.""" + — the shape a VHH single-domain antibody library produces) when heavy_only is set.""" bcr = { "IGH": { "v": rng.choice(HEAVY_V), "j": rng.choice(HEAVY_J), "c": rng.choice(HEAVY_C), @@ -83,9 +83,104 @@ def make_bcr(rng, heavy_only=False): return bcr -def build_clones(rng, cells, clear_antigens, heavy_only=False): +def _clone_sizes(n, mean_size=25, singleton_cell_frac=0.10, alpha=0.9, tail_cycle=None): + """Clone sizes for `n` cells of one (donor, antigen) group, shaped like an IMMUNIZED, + ANTIGEN-SORTED repertoire, which is the only kind this bed's blocks are pointed at. + + HOW MUCH EXPANSION IS A MEASURED QUESTION, AND THE ANSWER DEPENDS ON THE LIBRARY. Read this before + changing a default here. + + The argument from first principles says expansion should be everywhere: immunization drives a + germinal-centre response, sorting for antigen-positive cells enriches the expanded families, and + expansion is the signal being looked for. That argument is why this function replaced an earlier + power-law version whose tail made ~97% of clonotypes singletons. + + Real in-vivo BEAM libraries do not agree with it. Characterised 2026-08-21: 4,549 IGHeavy clonotypes + over 4,773 cells with paired chains, and 3,707 over 3,716. That is about 1.05 cells per clonotype, + essentially all singletons. The public figures that supported the expanded shape came from libraries + that cannot carry the argument: two 10x BEAM-T runs holding deliberately expanded spike-in + populations, a transgenic monoclonal control, and one literature ratio formed by dividing a FILTERED + antigen-labelled clone count by a FULL cell count. + + So both shapes are real and neither is the default for every run. `mean_size`, + `singleton_cell_frac` and `tail_cycle` carry the difference, and the regime tables in realpanel.py + set them. At shallow depth a clonotype's verdict DOES rest on one cell almost everywhere, which is a + fact about the data rather than a defect in the bed. The per-clonotype agreement rules go + unexercised because that real data does not exercise them either. + + Two compartments, because a real sorted sample has both: + - an EXPANDED compartment holding `1 - singleton_cell_frac` of the cells, split into clones whose + sizes follow `i**-alpha` and average `mean_size`. A heavy head, the leads, and a graded tail. + - a SMALL-CLONE tail holding the rest: mostly one-cell clonotypes, mixed with 2-4 cell ones. The + tail is NOT all singletons on purpose. A distribution that jumps from a wall of 1s straight to + clones of 7+ has a hole in it where real data is dense, and that hole is visible the moment + anyone sorts a clonotype table by cell count. + Singletons stay a large share of CLONOTYPES and a small share of CELLS, which is what the real + distribution looks like. + + Sizes sum to exactly `n` by largest-remainder apportionment. A size list that does not account for + every cell drops or duplicates cells, and the cross-arm join then loses them silently.""" + if n <= 0: + return [] + n_single = int(round(n * singleton_cell_frac)) + n_exp = n - n_single + if n_exp < 2: + return _tail_sizes(n, tail_cycle) + + k = max(1, int(round(n_exp / max(2.0, mean_size)))) + weights = [(i + 1) ** -alpha for i in range(k)] + total_w = sum(weights) + raw = [n_exp * w / total_w for w in weights] + sizes = [int(x) for x in raw] + # Largest-remainder: hand the rounding shortfall to the clones with the biggest fractional parts, so + # the list sums to n_exp exactly without a correction loop that can stall against the min-size floor. + short = n_exp - sum(sizes) + for i in sorted(range(k), key=lambda i: -(raw[i] - sizes[i]))[:short]: + sizes[i] += 1 + # A clone in the expanded compartment holds at least 2 cells. Fold anything smaller into the head so + # the total is preserved. (A 1-cell clone belongs to the singleton tail, which is counted separately.) + runts = sum(x for x in sizes if x < 2) + sizes = [x for x in sizes if x >= 2] + if not sizes: + sizes = [n_exp] + elif runts: + sizes[0] += runts + + return sizes + _tail_sizes(n_single, tail_cycle) + + +# Sizes for the small-clone tail, by position. Mostly 1s with 2s, 3s and a 4 mixed through: 22 cells +# across 15 clonotypes, so two thirds of the tail's clonotypes are singletons and the rest fill the +# 2-4 band. Positional rather than drawn, so the tail is reproducible without touching the caller's RNG. +TAIL_CYCLE = (1, 1, 1, 2, 1, 1, 1, 3, 1, 2, 1, 1, 4, 1, 2) + +# The tail real in-vivo libraries measure: 21 cells across 20 clonotypes, so 1.05 cells per clonotype. +# TAIL_CYCLE averages 1.47 and is therefore a FLOOR no combination of `mean_size` and +# `singleton_cell_frac` can get under. That is why this exists as a separate cycle rather than as +# another parameter setting. +TAIL_CYCLE_SPARSE = (1,) * 19 + (2,) + + +def _tail_sizes(budget, cycle=None): + """Clone sizes summing to exactly `budget`, following `cycle` and truncating the last clone to fit.""" + cycle = cycle or TAIL_CYCLE + out, left, i = [], budget, 0 + while left > 0: + size = min(cycle[i % len(cycle)], left) + out.append(size) + left -= size + i += 1 + return out + + +def build_clones(rng, cells, clear_antigens, heavy_only=False, clonal_profile="lead", + mean_size=25, singleton_cell_frac=0.10, tail_cycle=None): """Group a donor's cells into clonotypes. Clear-antigen cells -> one lead clone (~60%) + singletons, - all binding that antigen (coherent). Ambiguous cells -> singleton clones (no clear target).""" + all binding that antigen (coherent). Ambiguous cells -> singleton clones (no clear target). + + `clonal_profile="immunized"` replaces the one-lead-clone split with the size distribution an + immunized, antigen-sorted repertoire has (see _clone_sizes). The default "lead" is the original + behaviour, so every existing preset stays byte-identical.""" by_antigen = {} for cell_id, consensus in cells: by_antigen.setdefault(consensus, []).append(cell_id) @@ -94,8 +189,19 @@ def build_clones(rng, cells, clear_antigens, heavy_only=False): cidx = 0 for antigen, members in sorted(by_antigen.items()): rng.shuffle(members) - is_clear = antigen in clear_antigens - if is_clear: + # "crossreactive" is not a panel feature name, so it is not in `clear_antigens`. But a + # cross-reactive clone is a real and interesting lead, and leaving those cells as singletons + # means no CLONOTYPE is ever cross-reactive with more than one cell agreeing. + is_clear = antigen in clear_antigens or antigen == "crossreactive" + if is_clear and clonal_profile == "immunized": + for size in _clone_sizes(len(members), mean_size, singleton_cell_frac, + tail_cycle=tail_cycle): + grp, members = members[:size], members[size:] + kind = "expanded" if size >= 10 else ("minor" if size > 1 else "singleton") + clones.append({"id": f"clone{cidx}", "target": antigen, "kind": kind, + "cells": grp, "bcr": make_bcr(rng, heavy_only)}) + cidx += 1 + elif is_clear: n_lead = max(1, round(DOMINANT_FRACTION * len(members))) lead, rest = members[:n_lead], members[n_lead:] clones.append({"id": f"clone{cidx}", "target": antigen, "kind": "lead", @@ -119,22 +225,36 @@ def build_clones(rng, cells, clear_antigens, heavy_only=False): return clones -def write_airr(path, clones, rng, heavy_only=False): +def write_airr(path, clones, rng, heavy_only=False, unpaired_frac=0.0): + """Write one donor's AIRR single-cell rows. + + `unpaired_frac` is the share of cells that emit their HEAVY chain only, standing in for the cells a + real run recovers one chain from. It is not cosmetic: in the two measured libraries the clonotypes + dropped for want of a pair OUTNUMBERED the paired ones, 7,732 against 4,549 and 23,127 against + 3,707, and a bed where every cell pairs perfectly never exercises the drop. Heavy is the chain kept, + because heavy is the chain a VHH library has. + + Ignored when `heavy_only` is set, since there is no pair to break.""" loci = ("IGH",) if heavy_only else ("IGH", "IGK") n_rows = 0 + n_unpaired = 0 with open(path, "w", newline="") as fh: w = csv.writer(fh, delimiter="\t") w.writerow(AIRR_HEADER) for clone in clones: for cell_id in clone["cells"]: - for locus in loci: + cell_loci = loci + if not heavy_only and unpaired_frac > 0 and rng.random() < unpaired_frac: + cell_loci = ("IGH",) + n_unpaired += 1 + for locus in cell_loci: chain = clone["bcr"][locus] w.writerow([ cell_id, locus, chain["v"], chain["j"], chain["c"], chain["junction"], chain["junction_aa"], "T", rng.randint(5, 60), ]) n_rows += 1 - return n_rows + return n_rows, n_unpaired def write_truth(path, all_clones, heavy_only=False): @@ -154,15 +274,18 @@ def write_truth(path, all_clones, heavy_only=False): h["v"], h["j"], h["c"], k["v"], k["j"], h["junction_aa"], k["junction_aa"]]) -def build(tags_csv, consensus_tsv, out_dir, truth_dir, seed=VDJ_SEED, heavy_only=False): - """Build the VDJ arm from the antigen arm's ground truth. Writes out_dir/.tsv (AIRR-sc) + +def build(tags_csv, consensus_tsv, out_dir, truth_dir, seed=VDJ_SEED, heavy_only=False, + clonal_profile="lead", mean_size=25, singleton_cell_frac=0.10, unpaired_frac=0.0, + tail_cycle=None): + """Build the VDJ arm from the antigen arm's ground truth. Writes out_dir/.tsv (AIRR-sc) plus truth_dir/truth_clonotypes.csv. The filename stem is the bare donor id so Samples & Data mints ONE - shared sampleId across all three arms — a per-library suffix would fork the donor into separate - samples and the convergence [sampleId,cellId] join would then match nothing. + shared sampleId across all three arms. A per-library suffix would fork the donor into separate + samples, and the convergence [sampleId,cellId] join would then match nothing. - heavy_only=True emits HEAVY-CHAIN-ONLY (IGH, no IGK) rearrangements — the customer's VHH - single-domain antibody — so the heavy-only end-to-end path is reproducible synthetically. Each cell - keeps the SAME bare-16nt cell_id it carries in the antigen arm (the convergence join key).""" + heavy_only=True emits HEAVY-CHAIN-ONLY rearrangements, IGH with no IGK, which is the shape a VHH + single-domain antibody library produces, so the heavy-only end-to-end path is reproducible + synthetically. Each cell keeps the SAME bare-16nt cell_id it carries in the antigen arm, the + convergence join key.""" os.makedirs(out_dir, exist_ok=True) os.makedirs(truth_dir, exist_ok=True) rng = new_rng(seed) @@ -171,14 +294,20 @@ def build(tags_csv, consensus_tsv, out_dir, truth_dir, seed=VDJ_SEED, heavy_only all_clones = {} print(f"[vdj] airr-sc arm on the antigen ground truth " - f"({len(by_donor)} donors, {len(clear)} clear antigens{', heavy-only' if heavy_only else ''}):") + f"({len(by_donor)} donors, {len(clear)} clear antigens, {clonal_profile} clonality" + f"{', heavy-only' if heavy_only else ''}):") for donor in sorted(by_donor): - clones = build_clones(rng, by_donor[donor], clear, heavy_only) + clones = build_clones(rng, by_donor[donor], clear, heavy_only, clonal_profile, + mean_size, singleton_cell_frac, tail_cycle) all_clones[donor] = clones - n_rows = write_airr(os.path.join(out_dir, f"{donor}.tsv"), clones, rng, heavy_only) + n_rows, n_unpaired = write_airr(os.path.join(out_dir, f"{donor}.tsv"), clones, rng, heavy_only, + unpaired_frac) n_cells = sum(len(c["cells"]) for c in clones) - n_lead = sum(1 for c in clones if c["kind"] == "lead") + n_lead = sum(1 for c in clones if c["kind"] in ("lead", "expanded")) + per_clone = n_cells / max(1, len(clones)) + unp = f", {n_unpaired} heavy-only" if n_unpaired else "" print(f" {donor}: {n_cells} cells, {len(clones)} clonotypes " - f"({n_lead} lead) -> {n_rows} contig rows (vdj/{donor}.tsv)") + f"({n_lead} lead, {per_clone:.2f} cells/clonotype{unp}) -> {n_rows} contig rows " + f"(vdj/{donor}.tsv)") write_truth(os.path.join(truth_dir, "truth_clonotypes.csv"), all_clones, heavy_only) print(" truth -> truth/truth_clonotypes.csv") diff --git a/software/test-data/manual/real-data-calibration.md b/software/test-data/manual/real-data-calibration.md deleted file mode 100644 index 7536fb2..0000000 --- a/software/test-data/manual/real-data-calibration.md +++ /dev/null @@ -1,85 +0,0 @@ -# Real-data calibration — synthetic BEAM vs a real 5k BEAM-T reference - -Provenance + empirical basis for the `realistic` generator profile. The raw reference data is not in the -repo (10x public download); only this write-up is tracked. - -## Source -- 10x Genomics public dataset `5k_BEAM-T_Human_A0201_B0702_PBMC_5pv2_Multiplex` (~12 GB FASTQ tar, freely downloadable from 10x). -- **BEAM-T** (pMHC-multimer / TCR) — our synthetic is **BEAM-Ab** (BCR). Only *technical shapes* are - borrowed (read geometry, barcode error, UMI depth/duplication, panel separation); the biology - (cell types, antigen semantics, clone structure) is not. -- Measured 2026-07-01: one lane (6 M reads) per library, streamed; raw reads never persisted. - -## Read geometry — confirms ours -R1 = **26 nt** (16 CB + 10 UMI), R2 = **90 nt**, all three libraries. (Our R1 matches; real R2 is 90 nt -but the block only reads the first 15 nt as the feature, so our 25 nt R2 is fine.) - -## Antigen-capture shapes — measured vs synthetic -| Metric | Real BEAM-T | Default profile | **Realistic profile** (verified) | -|---|---|---|---| -| Dominant-feature UMIs/cell | median **632** (p10 18, p90 1490) | 8–30 | median **629** (18–1081) | -| Dominance fraction | median **1.00** (p10 0.79) | ~0.75 | median **0.99** (p10 0.70) | -| Background UMIs/cell | median **3** (p90 9) | higher | median **4** | -| PCR dup (reads/UMI) | median **1.3** (p90 2.0) | 1–4 | **1.30** | -| Reads/cell (antigen) | median **860** | ~60 | **~863** | -| Features detected/cell | median 3 (p90 4) | 3–4 | ~3–4 | - -## Panel barcodes — ours are authentic -Top real R2 15-mers are the **same standard 10x Antigen-Capture barcodes we use**: `GATTGGCTACTCAAT` -(90.2%), `CGGCTCACCGCGTCT` (4.9%), `CTATCTACCGGCTCG` (1.3%) + `CATGTCTACGTTAAG` (1.1%, one we don't -have). Pairwise Hamming among panel barcodes = **min 8** (our `≥3` design floor is safe). ~1–2% of -reads are Hamming-1 variants of the dominant barcode (feature-barcode seq errors → refine-tags snaps back). - -## Cross-library barcode overlap — validates the linking design -Top-5000 cell-barcode sets: **antigen∩gex 80%, antigen∩vdj 67%, gex∩vdj 69%, all-three 67%**. The same -16 nt barcode links the three libraries — and overlap is **partial** (~67–80%, not 100%), i.e. real -per-arm dropout. Our convergence inner-join is built for exactly this (cells missing from an arm drop). - -## GEX depth — already on the right path -~900 distinct UMIs/cell/lane (≈1800 across both lanes). Our synthetic GEX totals ~1800 counts/cell — -**matches**. (Genes/cell needs alignment; literature ~1–3.5k for 5′ PBMC.) The realistic profile bumps -genes 341→~1000 for more realistic genes/cell; UMI depth is unchanged (it was correct). - -## Cell-barcode error / correction structure -440k distinct raw barcodes / 6 M reads; top-5000 = **81.5%** of reads (≈ the 5k real cells). **50% of -distinct barcodes are singletons**; **~18.5% of reads are ambient/error** (not a real cell); **only -~14% of singleton junk is Hamming-1 of a real barcode** (→ correctable), the rest is ambient. So real -junk is *ambient-dominated*, not 1-bp-error-dominated. - -## What was applied — the `realistic` profile (defaults untouched) -| Generator | Flag | Output | Change | -|---|---|---|---| -| `antigen/generate.py` | `--profile realistic` | `realistic/` | UMI depth ↑, dup ↓, dominance ↑, background ↓ | -| `multiomics/generate_vdj.py` | `--realistic` | `vdj/realistic/` | reads realistic antigen consensus | -| `multiomics/generate_gex.py` | `--realistic` | `gex/realistic/` | 1000 genes (depth already matched) | -| `multiomics/validate_multiomics.py` | `--realistic` | — | validates the realistic chain (**38/38**) | - -Build the realistic multiomics chain: -```bash -# antigen (in antigen/) -python3 generate.py --profile realistic # + optionally --scenario all -# arms (in multiomics/) -python3 generate_vdj.py --realistic && python3 generate_gex.py --realistic -python3 validate_multiomics.py --realistic # 38/38 -``` - -## Recommendations NOT auto-applied (future, if wanted) -- **Ambient/error scenario:** make the `errors` fixture ambient-dominated — add a heavy tail of random - non-cell barcodes (~18% of reads), only ~14% of them Hamming-1 of a real cell. The current `errors` - scenario over-weights clean 1-bp errors. -- **Cross-library dropout scenario:** drop ~15–30% of cells per arm so triple-overlap ≈ 67% (tests the - inner-join drop). The default keeps 100% overlap for a clean, maximal join. -- **Dynamic range:** one antigen was 90% of the whole BEAM-T library. BEAM-Ab discovery is more even, so - we keep a spread — but a "single-dominant-antigen" scenario would mirror BEAM-T. - -## Scale (samples / panel / cells) — parameterized 2026-07-02 - -The depth/dominance calibration above is orthogonal to *scale*. Sample count, antigen-panel size, and -cells-per-sample are now CLI flags (`--samples` / `--panel-size` / `--cells-per-sample`, defaults -24 / 64 / 2000), so the fixture spans a toy bed to a cohort-scale run without touching this calibration. -Targets are corroborated by the BEAM dataset-scale survey (deep-research, adversarially verified -2026-06-30): cohort high-water ~22–50 donors; verified feature ceiling = 64 (BEAM-proper alone is ~6); -a real GEM well is 2k–10k cells. The panel keeps the 4 -real 10x anchor barcodes and synthesizes the rest (15-mers, Hamming ≥ 3). The `whitelist737k` cell pool -now samples the full 10x `737K-august-2016` inclusion list (737,280 barcodes) rather than the ~800 -harvested pool, so it scales with `--samples`/`--cells-per-sample`. diff --git a/software/test-data/manual/reshape_panel.py b/software/test-data/manual/reshape_panel.py new file mode 100644 index 0000000..377125b --- /dev/null +++ b/software/test-data/manual/reshape_panel.py @@ -0,0 +1,170 @@ +#!/usr/bin/env python3 +"""Rewrite a generated run's tags.csv into the two shapes a real panel file arrives in. + + python3 reshape_panel.py runs/tiny + +Writes `tags_narrow.csv` and `tags_wide.csv` beside `tags.csv`, both carrying **the same barcodes**, so +either can be uploaded to the block against the same FASTQs. A reshaped panel with a changed barcode +joins to nothing, which is why nothing here touches the `tag` column. + +Why this exists. `generate.py` emits one panel shape: `tag,feature,Type,Species,Class`, one panel for +every sample, with the control carrying its own `Decoy` role. Two other shapes were observed in use at +one account at the same time, on two of its projects, and neither looks like that: + + narrow sample, barcode, antigen name -- and no fourth column. Nothing declares a role, so nothing can + be named as the comparator and the panel's own readings have to serve. + wide sample, name, catalogue id, barcode, channel, a constant, role. The role column declares what + a member is TO THE QUESTION (target, off-target) and carries **no** comparator value. + +In both, the negative control is one antigen the scientist points at by name in the interface. So +neither shape can reach the declared-comparator path, for two different reasons. That is the thing these +files exist to make visible in the app rather than only in a CSV. + +Both shapes rename a barcode between samples: the same sequence carries a different antigen name in +different samples, which is the tag-inventory reuse the per-sample keying of the panel exists for. Under +the per-tag grouping the identity is the barcode, so those identities lose their label and a reader +meets a raw 15-mer where every other row shows an antigen. + +Deterministic: every choice below is positional, so a rerun over the same tags.csv is byte-identical. +Stdlib only, like the rest of this bed. +""" + +import argparse +import csv +import os +import sys + +# Four values that are three channels — one of them spelled two ways, so grouping on this column splits +# one channel in two. Assigned by position, cycling. +CHANNELS = ["PE", "PE", "APC", "APC", "PE Dazzle", "PE Dazzle", "PE-Dazzle 5120", "PE-Dazzle 5120"] + +# One value on every row: a declared property carrying no information at all. Group on it and every tag +# lands in one identity, which is legal and useless. +RESIDUES = "ECD protein" + +# The control's own role is folded into the off-target set on purpose. The observed wide file had no +# value meaning "comparator" anywhere in its role column, and that is the whole point of the shape. +ROLE_OF = {"Target": "Target (Primary)", "Off-Target": "Off-Target", "Decoy": "Off-Target"} + + +def _lowercased(role: str) -> str: + """The role as the observed file also spelled it — the qualifier or the word after the hyphen.""" + return role.replace("(P", "(p").replace("(S", "(s").replace("-Target", "-target") + + +def read_tags(run_dir: str) -> list[dict]: + path = os.path.join(run_dir, "tags.csv") + if not os.path.exists(path): + sys.exit(f"no tags.csv in {run_dir} — generate a run first (python3 generate.py tiny --arm antigen)") + with open(path, newline="") as f: + rows = list(csv.DictReader(f)) + for col in ("tag", "feature"): + if not rows or col not in rows[0]: + sys.exit(f"{path} has no {col!r} column; columns are {list(rows[0]) if rows else '[]'}") + return rows + + +def read_samples(run_dir: str) -> list[str]: + """Sample names from the antigen arm's filenames, so they match Samples & Data exactly.""" + antigen = os.path.join(run_dir, "antigen") + if not os.path.isdir(antigen): + sys.exit(f"no antigen/ arm in {run_dir} — run: python3 generate.py --arm antigen") + names = sorted({f.split("_R")[0] for f in os.listdir(antigen) if f.endswith(".fastq.gz")}) + if not names: + sys.exit(f"no FASTQs in {antigen}") + return names + + +def _renamed(name: str, sample_index: int) -> str: + """A plainly different antigen name for a reused barcode in a later sample.""" + return f"{name}__alt{sample_index}" + + +def write_narrow(path: str, tags: list[dict], samples: list[str], rename: int, drop: int) -> int: + with open(path, "w", newline="") as f: + w = csv.writer(f) + w.writerow(["Sample", "Sequence", "Antigen"]) + rows = 0 + for s_i, sample in enumerate(samples): + # A later sample may declare fewer tags, which is what makes *never asked* reachable: a set + # whose cells sit only in that sample was never offered the dropped identities. + offered = tags[: len(tags) - drop] if s_i else tags + for t_i, tag in enumerate(offered): + name = _renamed(tag["feature"], s_i) if (s_i and t_i < rename) else tag["feature"] + w.writerow([sample, tag["tag"], name]) + rows += 1 + return rows + + +def write_wide(path: str, tags: list[dict], samples: list[str], rename: int, drop: int) -> int: + with open(path, "w", newline="") as f: + w = csv.writer(f) + w.writerow(["Samples", "Name", "Barcode", "Sequence", "Channel", "Residues", "Type"]) + rows = 0 + for s_i, sample in enumerate(samples): + offered = tags[: len(tags) - drop] if s_i else tags + for t_i, tag in enumerate(offered): + name = _renamed(tag["feature"], s_i) if (s_i and t_i < rename) else tag["feature"] + role = ROLE_OF.get(tag.get("Type", "Target"), "Target (Primary)") + # Two case-variant failure modes, kept apart so each can be told from the other. Tag 0 + # reads one spelling in the first sample and another in the rest, so it carries two + # values, the property is dropped for it, and it ends up with no role at all. Tag 1 + # reads the other spelling everywhere, so it keeps its role but no longer matches the + # same role written normally elsewhere. + if (t_i == 0 and s_i) or t_i == 1: + role = _lowercased(role) + w.writerow( + [ + sample, + name, + f"T{100 + t_i:04d}", + tag["tag"], + CHANNELS[t_i % len(CHANNELS)], + RESIDUES, + role, + ] + ) + rows += 1 + return rows + + +def main() -> None: + p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + p.add_argument("run_dir", help="a generated run directory, e.g. runs/tiny") + p.add_argument( + "--rename", + type=int, + default=2, + help="barcodes carrying a different antigen name in later samples (default 2; 0 disables)", + ) + p.add_argument( + "--drop-from-later", + type=int, + default=0, + help="tags a later sample does not declare, making *never asked* reachable (default 0)", + ) + args = p.parse_args() + + tags = read_tags(args.run_dir) + samples = read_samples(args.run_dir) + if args.rename > len(tags) or args.drop_from_later >= len(tags): + sys.exit(f"--rename/--drop-from-later exceed the panel's {len(tags)} tags") + + narrow = os.path.join(args.run_dir, "tags_narrow.csv") + wide = os.path.join(args.run_dir, "tags_wide.csv") + n_rows = write_narrow(narrow, tags, samples, args.rename, args.drop_from_later) + w_rows = write_wide(wide, tags, samples, args.rename, args.drop_from_later) + + kept = len(tags) - args.drop_from_later + print(f"{len(tags)} tags x {len(samples)} samples -> {samples}") + print(f" {narrow} ({n_rows} rows, 3 columns, no role column)") + print(f" {wide} ({w_rows} rows, 7 columns, role column with no comparator value)") + print(f" {args.rename} barcode(s) renamed in later samples; later samples declare {kept} of {len(tags)}") + if len(tags) < 8: + print(f" WARNING: {len(tags)} tags is below the shipped panel minimum of 8, so the panel's own") + print(" readings cannot serve and every verdict will read unreliable. Regenerate with") + print(" a larger --panel-size, or lower the minimum in the block's settings.") + + +if __name__ == "__main__": + main() diff --git a/software/test-data/manual/tests/test_panel.py b/software/test-data/manual/tests/test_panel.py index 00b3638..d967df9 100644 --- a/software/test-data/manual/tests/test_panel.py +++ b/software/test-data/manual/tests/test_panel.py @@ -1,16 +1,32 @@ -"""Panel-metadata tests: the generated tags.csv must carry the real customer panel's per-antigen -Type / Species / Class columns (alongside the backward-compatible tag,feature role mapping), the +"""Panel-metadata tests: the generated tags.csv must carry the per-antigen Type / Species / Class +columns a real panel declares, alongside the backward-compatible tag,feature role mapping. The --offtarget-count flag must designate exactly N non-control antigens as Off-Target, and out-of-range counts must be rejected on BOTH the full-run and the --beam paths.""" import csv -import importlib.util import subprocess import sys from pathlib import Path +import pytest + HERE = Path(__file__).resolve().parent.parent # software/test-data/manual +# The gex arm annotates against a human gene-annotations table that is downloaded, not committed, so a +# full multiomic run cannot be built in a clean checkout. Tests needing only the antigen arm use the +# scenario path instead and are unaffected. The ones that genuinely need a full run say so rather than +# failing with a bare subprocess error that names no cause. +GENE_ANNOTATIONS = HERE / "assets" / "homo_sapiens_gene_annotations.csv" +needs_full_run = pytest.mark.skipif( + not GENE_ANNOTATIONS.exists(), + reason=( + f"missing {GENE_ANNOTATIONS.name}; fetch it with:\n" + " curl -sSL -o /tmp/hs.zip https://bin.pl-open.science/assets/platforma-open/" + "milaboratories.gene-annotations.homo-sapiens/main/1.1.0.zip" + f" && unzip -o /tmp/hs.zip -d {GENE_ANNOTATIONS.parent}" + ), +) + def _run(*args, out): subprocess.run( @@ -28,7 +44,9 @@ def _read_csv(path): def test_panel_has_type_species_class(tmp_path): - _run("tiny", "--offtarget-count", "2", out=tmp_path) + # tags.csv comes from the antigen arm, so the self-contained scenario bed suffices — no full run, + # no gene-annotations asset. + _run("--scenario", "errors", "--offtarget-count", "2", out=tmp_path) header, rows = _read_csv(tmp_path / "tags.csv") assert {"Type", "Species", "Class"} <= set(header) types = {r["Type"] for r in rows} @@ -37,6 +55,7 @@ def test_panel_has_type_species_class(tmp_path): assert {"Human", "Cyno"} <= {r["Species"] for r in rows} +@needs_full_run def test_multibarcode_combine_column(tmp_path): _run("tiny", "--multibarcode", out=tmp_path) with (tmp_path / "tags.csv").open() as fh: @@ -51,6 +70,7 @@ def test_multibarcode_combine_column(tmp_path): assert {"all", "sum"} <= {r["combine"] for r in rows} +@needs_full_run def test_messy_metadata_variants(tmp_path): _run("tiny", "--offtarget-count", "3", "--messy-metadata", out=tmp_path) with (tmp_path / "tags.csv").open() as fh: @@ -61,7 +81,7 @@ def test_messy_metadata_variants(tmp_path): def test_beam_panel_has_type_species(tmp_path): - # beam-exact: 2 samples x panel_size antigens; --offtarget-count applies per sample. + # beam-exact: 2 samples x panel_size antigens. --offtarget-count applies per sample. _run("--beam", "--offtarget-count", "2", "--cells-per-sample", "10", out=tmp_path) header, rows = _read_csv(tmp_path / "tags.csv") assert {"Type", "Species"} <= set(header) @@ -69,29 +89,64 @@ def test_beam_panel_has_type_species(tmp_path): assert "Target" in {r["Type"] for r in rows} -def _load_consensus(): - p = HERE.parent.parent / "per-cell-metrics" / "src" / "per_cell_metrics.py" - spec = importlib.util.spec_from_file_location("pcm", p) - mod = importlib.util.module_from_spec(spec) - spec.loader.exec_module(mod) - return mod +def _read_tsv(path): + with path.open() as fh: + return list(csv.DictReader(fh, delimiter="\t")) -def test_crossreactive_two_even_antigens(): - pcm = _load_consensus() - counts = {"AgA": 50.0, "AgB": 48.0, "ctrl": 3.0} - result = pcm.consensus_category( - counts, threshold=0.6, control="ctrl", offtargets=frozenset(), label_crossreactive=True - ) - assert result == "cross-reactive" +def _top_two_ratio(counts): + """Second-largest count over the largest, or None where the cell has fewer than two features.""" + ranked = sorted(counts, reverse=True) + return None if len(ranked) < 2 else ranked[1] / ranked[0] def test_generator_plants_crossreactive(tmp_path): - _run("tiny", "--offtarget-count", "1", "--crossreactive-frac", "0.1", out=tmp_path) - consensus = list(csv.DictReader((tmp_path / "truth" / "expected-consensus.tsv").open(), delimiter="\t")) - assert any(r.get("planted_consensus") == "crossreactive" for r in consensus) - - + """A planted cross-reactive cell carries a co-dominant pair of two ON-TARGET antigens. + + This asserts the generator against its own output and reads nothing from the block. It used to check + the block's `consensus_category` instead, which no longer exists: a single dominant antigen per cell + answers a different question from the four-state verdict and was removed with it. What the bed still + owes is the planted shape, because the gex and vdj arms are both built from this truth file. + """ + # The antigen-only scenario bed rather than a full `tiny` run. A full run also builds the gex arm, + # which needs a gene-annotations asset that is downloaded rather than committed, so a full run + # cannot be generated in a clean checkout. This test needs the antigen arm alone. The scenario bed + # is self-contained and writes its truth files flat in the output directory. + _run("--scenario", "errors", "--offtarget-count", "1", "--crossreactive-frac", "0.1", out=tmp_path) + + consensus = _read_tsv(tmp_path / "expected-consensus.tsv") + crossreactive = {(r["sample"], r["cellId"]) for r in consensus if r["planted_consensus"] == "crossreactive"} + assert crossreactive, "no cross-reactive cell was planted at --crossreactive-frac 0.1" + + _, panel_rows = _read_csv(tmp_path / "tags.csv") + on_target = {r["feature"] for r in panel_rows if r["Type"] == "Target"} + + counts_by_cell = {} + for r in _read_tsv(tmp_path / "expected-abundance.tsv"): + counts_by_cell.setdefault((r["sample"], r["cellId"]), {})[r["feature"]] = int(r["planted_distinct_umis"]) + + # The pair is planted as second = first * U(0.85, 1.0), then truncated to an int, so 0.84 is the + # floor a correctly planted cell cannot fall below. + for key in crossreactive: + counts = counts_by_cell[key] + ratio = _top_two_ratio(counts.values()) + assert ratio is not None and ratio >= 0.84, f"{key} is labelled cross-reactive but its top two counts are {ratio}" + top_two = sorted(counts, key=counts.get, reverse=True)[:2] + assert set(top_two) <= on_target, f"{key}'s co-dominant pair includes a non-Target antigen: {top_two}" + + # The check above is only worth running if it can fail, and an evenness test passes vacuously on a + # bed where every cell is even. An ordinary binder plants one dominant antigen over background, so + # some non-cross-reactive cell must be visibly UNEVEN — otherwise the assertion above proves + # nothing about the label. + uneven = [ + key + for key, counts in counts_by_cell.items() + if key not in crossreactive and (_top_two_ratio(counts.values()) or 0) < 0.5 + ] + assert uneven, "every cell in the bed is co-dominant, so the cross-reactive assertion cannot discriminate" + + +@needs_full_run def test_heavy_only_airr(tmp_path): _run("tiny", "--heavy-only", out=tmp_path) tsvs = list((tmp_path / "vdj").glob("*.tsv")) @@ -104,6 +159,7 @@ def test_heavy_only_airr(tmp_path): assert loci == {"IGH"} +@needs_full_run def test_annotation_emitter(tmp_path): _run("tiny", "--with-annotations", out=tmp_path) tsvs = list((tmp_path / "annotations").glob("*.tsv")) @@ -116,7 +172,7 @@ def test_annotation_emitter(tmp_path): with (tmp_path / "truth" / "expected-consensus.tsv").open() as fh: cons = list(csv.DictReader(fh, delimiter="\t")) ann_ids = {r["cell_id"] for r in rows} - # expected-consensus.tsv keys the barcode as `cellId` (antigen arm); annotations use `cell_id`. + # expected-consensus.tsv keys the barcode as `cellId` (antigen arm). Annotations use `cell_id`. # Accept either so the join-compatibility check is meaningful against the real truth schema. cons_ids = {r.get("cell_id") or r.get("cellId") for r in cons} assert ann_ids & cons_ids @@ -140,4 +196,9 @@ def test_offtarget_count_out_of_range_errors(tmp_path): capture_output=True, text=True, ) + # Asserting only a nonzero exit would pass for any failure at all -- a missing asset, a syntax + # error, a bad path -- so it must name the rejection it is checking for. assert result.returncode != 0, f"expected nonzero exit for {extra}, got 0" + assert "--offtarget-count must be between" in (result.stderr + result.stdout), ( + f"exited nonzero for {extra}, but not because the count was out of range:\n{result.stderr}" + ) diff --git a/software/test-data/manual/tests/test_realpanel.py b/software/test-data/manual/tests/test_realpanel.py new file mode 100644 index 0000000..5c1c7b2 --- /dev/null +++ b/software/test-data/manual/tests/test_realpanel.py @@ -0,0 +1,472 @@ +"""Real-panel path tests. + +The point of `--real-panel` is that a panel supplied from outside the repository drives the run, so +these tests write their OWN synthetic wide panel and drive the generator with that. Nothing here may +carry a real panel's vocabulary -- its sample names, antigen names, catalogue ids, sequences or channel +values -- because this file is committed and a real panel is not ours to commit. + +The synthetic panel reproduces the three shapes that matter: per-sample panels of unequal size, a +sequence reused across two samples under different antigen names, and a role column that declares target +and off-target and nothing meaning *negative control*. +""" + +import csv +import subprocess +import sys +from pathlib import Path + +import pytest + +HERE = Path(__file__).resolve().parent.parent # software/test-data/manual +sys.path.insert(0, str(HERE)) + +from lib import realpanel # noqa: E402 + +# 15-mers, pairwise Hamming >= 3, invented for this test. +SEQ = { + "a": "ACGTACGTACGTACG", + "b": "TTGGCCAATTGGCCA", + "c": "GGATCCGGATCCGGA", + "d": "CCTAGGCCTAGGCCT", + "e": "AAGCTTAAGCTTAAG", + "f": "TCGATCGATCGATCG", +} + +# (Samples, Name, Barcode, Sequence, Channel, Residues, Type) +PANEL_ROWS = [ + ("grp1", "Ag Alpha", "X0001", SEQ["a"], "PE", "ECD", "Target (Primary)"), + ("grp1", "Ag Alpha Var", "X0002", SEQ["b"], "PE", "ECD", "Target (Secondary)"), + ("grp1", "Ctl One", "X0003", SEQ["c"], "APC", "ECD", "Off-Target"), + ("grp1", "Ctl Two", "X0004", SEQ["d"], "APC", "ECD", "Off-Target"), + ("grp2", "Ag Beta", "X0005", SEQ["e"], "PE", "ECD", "Target (Primary)"), + ("grp2", "Ag Gamma", "X0006", SEQ["f"], "PE", "ECD", "Target (Secondary)"), + # SEQ["c"] again, under a DIFFERENT antigen name: the cross-sample reuse the sample column exists for. + ("grp2", "Ctl Other", "X0003", SEQ["c"], "APC", "ECD", "Off-Target"), +] + + +@pytest.fixture +def panel_csv(tmp_path): + path = tmp_path / "panel.csv" + with open(path, "w", newline="") as fh: + w = csv.writer(fh) + w.writerow(["Samples", "Name", "Barcode", "Sequence", "Channel", "Residues", "Type"]) + w.writerows(PANEL_ROWS) + return path + + +def run_generator(panel_csv, out_dir, *extra): + return subprocess.run( + [sys.executable, str(HERE / "generate.py"), "--real-panel", str(panel_csv), + "--cells-per-sample", "300", "--barcode-source", "random", "--out", str(out_dir), *extra], + capture_output=True, text=True, + ) + + +# --- panel loading ------------------------------------------------------------------------------- + +def test_loads_one_panel_per_sample(panel_csv): + panels = realpanel.load_wide_panel(str(panel_csv)) + assert list(panels) == ["grp1", "grp2"], "samples keep file order" + assert len(panels["grp1"].names) == 4 + assert len(panels["grp2"].names) == 3 + + +def test_role_matched_on_the_leading_word(panel_csv): + """`Target (Primary)` and `Target (Secondary)` are two distinct values and both are on-target.""" + panels = realpanel.load_wide_panel(str(panel_csv)) + assert panels["grp1"].targets == ["Ag Alpha", "Ag Alpha Var"] + assert panels["grp1"].offtargets == ["Ctl One", "Ctl Two"] + + +def test_a_sequence_may_carry_different_names_in_different_samples(panel_csv): + panels = realpanel.load_wide_panel(str(panel_csv)) + assert panels["grp1"].barcode["Ctl One"] == panels["grp2"].barcode["Ctl Other"] + + +def test_duplicate_sequence_within_one_sample_is_rejected(tmp_path): + """The block's own guard rejects it, so the bed must not be able to produce it.""" + path = tmp_path / "dup.csv" + with open(path, "w", newline="") as fh: + w = csv.writer(fh) + w.writerow(["Samples", "Name", "Sequence", "Type"]) + w.writerow(["grp1", "One", SEQ["a"], "Target"]) + w.writerow(["grp1", "Two", SEQ["a"], "Off-Target"]) + with pytest.raises(SystemExit, match="appears twice"): + realpanel.load_wide_panel(str(path)) + + +def test_wrong_length_sequence_is_rejected(tmp_path): + path = tmp_path / "short.csv" + with open(path, "w", newline="") as fh: + w = csv.writer(fh) + w.writerow(["Samples", "Name", "Sequence", "Type"]) + w.writerow(["grp1", "One", "ACGT", "Target"]) + with pytest.raises(SystemExit, match="expected 15"): + realpanel.load_wide_panel(str(path)) + + +def test_missing_column_names_the_flags_that_fix_it(tmp_path): + path = tmp_path / "wrong.csv" + with open(path, "w", newline="") as fh: + w = csv.writer(fh) + w.writerow(["sample_id", "antigen", "bc", "role"]) + w.writerow(["grp1", "One", SEQ["a"], "Target"]) + with pytest.raises(SystemExit, match="--panel-seq-col"): + realpanel.load_wide_panel(str(path)) + + +def test_columns_are_overridable(tmp_path): + path = tmp_path / "renamed.csv" + with open(path, "w", newline="") as fh: + w = csv.writer(fh) + w.writerow(["sample_id", "antigen", "bc", "role"]) + w.writerow(["grp1", "One", SEQ["a"], "Target"]) + w.writerow(["grp1", "Two", SEQ["b"], "Off-Target"]) + panels = realpanel.load_wide_panel( + str(path), columns={"sample": "sample_id", "name": "antigen", "sequence": "bc", "role": "role"} + ) + assert panels["grp1"].targets == ["One"] + assert panels["grp1"].offtargets == ["Two"] + + +# --- the reading rule ---------------------------------------------------------------------------- + +@pytest.mark.parametrize("count,reference,expected", [ + (0, 0, 0.0421875), + (1, 1, 0.01487109375), + (60, 4, 23.5711265135), + (150, 6, 85.0165945400), + (300, 20, 63.2309112766), + (600, 50, 29.4968614580), +]) +def test_specificity_score_matches_the_block(count, reference, expected): + """The bed's standard-library score against the values scipy gives for the block's own formula. + They have to agree, or the verdict simulation predicts a different run than the block produces.""" + assert realpanel.specificity_score(count, reference) == pytest.approx(expected, rel=1e-9) + + +def test_the_line_sits_where_the_tiers_assume(tmp_path): + """`medium` is 60-200 UMIs because the line against a comparator of ~5 is ~120. If that moves, the + tier stops straddling anything and the bed silently loses its borderline cells.""" + below = realpanel.specificity_score(60, 5) + above = realpanel.specificity_score(200, 5) + assert below < realpanel.CUTOFF < above + + +# --- a whole run --------------------------------------------------------------------------------- + +def test_a_run_generates_and_validates(panel_csv, tmp_path): + out = tmp_path / "run" + res = run_generator(panel_csv, out) + assert res.returncode == 0, res.stdout + res.stderr + assert "PASS" in res.stdout + for rel in ("panel.csv", "RUN.md", "truth/expected-readings.tsv", "truth/expected-abundance.tsv", + "truth/expected-consensus.tsv", "truth/library-quality.tsv", "truth/panel-canonical.csv"): + assert (out / rel).exists(), rel + for sample in ("grp1", "grp2"): + assert (out / "antigen" / f"{sample}_R1.fastq.gz").exists() + assert (out / "antigen" / f"{sample}_R2.fastq.gz").exists() + assert (out / "vdj" / f"{sample}.tsv").exists() + + +def test_the_uploaded_panel_is_the_source_file_byte_for_byte(panel_csv, tmp_path): + """The block must read the panel as it arrived, not a re-serialisation of it.""" + out = tmp_path / "run" + assert run_generator(panel_csv, out).returncode == 0 + assert (out / "panel.csv").read_bytes() == panel_csv.read_bytes() + + +def test_every_tier_a_sample_can_support_is_present(panel_csv, tmp_path): + out = tmp_path / "run" + assert run_generator(panel_csv, out).returncode == 0 + with open(out / "truth" / "expected-readings.tsv", newline="") as fh: + tiers = {r["tier"] for r in csv.DictReader(fh, delimiter="\t")} + assert tiers == set(realpanel.TIER_NAMES), f"missing {set(realpanel.TIER_NAMES) - tiers}" + + +def test_no_negative_control_is_planted(panel_csv, tmp_path): + """A real role column declares no comparator, so the run must not invent one — otherwise the + declared-reference path is exercised and the panel-reference path never is.""" + out = tmp_path / "run" + assert run_generator(panel_csv, out).returncode == 0 + with open(out / "truth" / "expected-abundance.tsv", newline="") as fh: + features = {r["feature"] for r in csv.DictReader(fh, delimiter="\t")} + assert "negative_control" not in features + assert features <= {name for _s, name, _b, _q, _c, _r, _t in PANEL_ROWS} + + +def test_the_run_is_reproducible(panel_csv, tmp_path): + a, b = tmp_path / "a", tmp_path / "b" + assert run_generator(panel_csv, a).returncode == 0 + assert run_generator(panel_csv, b).returncode == 0 + for rel in ("antigen/grp1_R1.fastq.gz", "antigen/grp1_R2.fastq.gz", + "truth/expected-readings.tsv", "vdj/grp1.tsv"): + assert (a / rel).read_bytes() == (b / rel).read_bytes(), rel + + +def test_library_quality_profiles_change_the_expected_tag(panel_csv, tmp_path): + out = tmp_path / "run" + assert run_generator(panel_csv, out, "--library-quality", "spread").returncode == 0 + with open(out / "truth" / "library-quality.tsv", newline="") as fh: + rows = list(csv.DictReader(fh, delimiter="\t")) + assert {r["libraryTier"] for r in rows} == {"clean", "good"}, "two samples take the first two tiers" + out2 = tmp_path / "run2" + assert run_generator(panel_csv, out2, "--library-quality", "uniform").returncode == 0 + with open(out2 / "truth" / "library-quality.tsv", newline="") as fh: + assert {r["libraryTier"] for r in csv.DictReader(fh, delimiter="\t")} == {"clean"} + + +def test_offset_zero_moves_the_feature_to_the_front_of_r2(panel_csv, tmp_path): + import gzip + + out = tmp_path / "run" + assert run_generator(panel_csv, out, "--offset", "0").returncode == 0 + panels = realpanel.load_wide_panel(str(panel_csv)) + at_front = 0 + with gzip.open(out / "antigen" / "grp1_R2.fastq.gz", "rt") as fh: + for i, line in enumerate(fh): + if i >= 4000: + break + if i % 4 == 1 and line[:15] in set(panels["grp1"].barcodes): + at_front += 1 + assert at_front > 100, "with offset 0 the feature barcode sits at R2 position 0" + + +# --- sample metadata ------------------------------------------------------------------------------ + +def test_sample_metadata_names_exactly_the_panels_samples(panel_csv, tmp_path): + """Samples & Data joins metadata on the sample name. A row for a sample that is not in the run, or a + sample with no row, means a grouping column that silently covers part of the run.""" + out = tmp_path / "run" + assert run_generator(panel_csv, out).returncode == 0 + with open(out / "samples-metadata.tsv", newline="") as fh: + rows = list(csv.DictReader(fh, delimiter="\t")) + assert {r["Sample"] for r in rows} == {"grp1", "grp2"} + assert list(rows[0]) == ["Sample", "Donor", "Condition", "LibraryQuality", "PanelMembers", "PanelTargets"] + + +def test_metadata_carries_the_per_sample_panel_shape(panel_csv, tmp_path): + """PanelMembers / PanelTargets are the two metadata columns that are read from the panel rather than + invented, and the per-sample difference is what makes *never asked* reachable.""" + out = tmp_path / "run" + assert run_generator(panel_csv, out).returncode == 0 + with open(out / "samples-metadata.tsv", newline="") as fh: + by_sample = {r["Sample"]: r for r in csv.DictReader(fh, delimiter="\t")} + assert by_sample["grp1"]["PanelMembers"] == "4" + assert by_sample["grp2"]["PanelMembers"] == "3" + assert by_sample["grp1"]["PanelTargets"] == "2" + + +def test_metadata_records_the_library_tier(panel_csv, tmp_path): + out = tmp_path / "run" + assert run_generator(panel_csv, out, "--library-quality", "uniform").returncode == 0 + with open(out / "samples-metadata.tsv", newline="") as fh: + assert {r["LibraryQuality"] for r in csv.DictReader(fh, delimiter="\t")} == {"clean"} + + +# --- reshaping the repertoire without regenerating the reads --------------------------------------- + +def test_arm_vdj_rebuilds_the_repertoire_and_leaves_the_reads_alone(panel_csv, tmp_path): + out = tmp_path / "run" + assert run_generator(panel_csv, out).returncode == 0 + fastq = out / "antigen" / "grp1_R1.fastq.gz" + before_reads = fastq.read_bytes() + before_vdj = (out / "vdj" / "grp1.tsv").read_bytes() + + res = run_generator(panel_csv, out, "--arm", "vdj", "--clonal-mean-size", "60") + assert res.returncode == 0, res.stdout + res.stderr + assert fastq.read_bytes() == before_reads, "the antigen arm must not be regenerated" + assert (out / "vdj" / "grp1.tsv").read_bytes() != before_vdj, "the repertoire must change" + + +def test_arm_vdj_without_an_antigen_arm_says_so(panel_csv, tmp_path): + res = run_generator(panel_csv, tmp_path / "empty", "--arm", "vdj") + assert res.returncode != 0 + assert "rebuilds the repertoire over an EXISTING antigen arm" in res.stdout + res.stderr + + +def test_most_cells_sit_in_expanded_clones(panel_csv, tmp_path): + """The shape that matters for an antibody-discovery bed: a clonotype's verdict must usually rest on + SEVERAL cells. Asserted on the share of CELLS in clones of >= 10, not on the share of clonotypes. + Singletons are legitimately a large share of clonotypes and a small share of cells, and confusing + the two is what produced a 97%-singleton repertoire in the first place.""" + out = tmp_path / "run" + assert run_generator(panel_csv, out).returncode == 0 + with open(out / "truth" / "truth_clonotypes.csv", newline="") as fh: + sizes = [int(r["nCells"]) for r in csv.DictReader(fh)] + cells = sum(sizes) + in_expanded = sum(s for s in sizes if s >= 10) + assert in_expanded / cells > 0.5, f"only {in_expanded}/{cells} cells in clones of >= 10" + assert max(sizes) >= 20, f"largest clone is {max(sizes)} cells — no lead to find" + + +def test_clone_size_knobs_move_the_distribution(panel_csv, tmp_path): + def biggest(out): + with open(out / "truth" / "truth_clonotypes.csv", newline="") as fh: + return max(int(r["nCells"]) for r in csv.DictReader(fh)) + + diverse, expanded = tmp_path / "d", tmp_path / "e" + assert run_generator(panel_csv, diverse, "--clonal-mean-size", "4").returncode == 0 + assert run_generator(panel_csv, expanded, "--clonal-mean-size", "60").returncode == 0 + assert biggest(expanded) > biggest(diverse) + + +def test_crossreactive_cells_form_clones_not_singletons(panel_csv, tmp_path): + """A cross-reactive clonotype is a real lead. If those cells are all singletons, no clonotype is ever + cross-reactive with more than one cell agreeing, and the two-identity case is untestable.""" + out = tmp_path / "run" + assert run_generator(panel_csv, out).returncode == 0 + with open(out / "truth" / "truth_clonotypes.csv", newline="") as fh: + cr = [int(r["nCells"]) for r in csv.DictReader(fh) if r["targetAntigen"] == "crossreactive"] + assert cr, "no cross-reactive clones at all" + assert max(cr) > 1, "every cross-reactive clone is a singleton" + + +def test_clone_sizes_account_for_every_cell(): + """A size list that does not sum to n drops or duplicates cells, and the cross-arm join then loses + them silently.""" + from lib import vdj + + for n in (0, 1, 2, 7, 50, 1500, 3701): + for mean_size in (2, 5, 25, 60): + for frac in (0.0, 0.1, 0.35, 1.0): + sizes = vdj._clone_sizes(n, mean_size, frac) + assert sum(sizes) == n, (n, mean_size, frac, sum(sizes)) + assert all(s >= 1 for s in sizes) + + +# --- regimes ------------------------------------------------------------------------------------- +# +# Two measured calibrations exist and they disagree by more than an order of magnitude. `deep` is the +# public 10x BEAM shape, and `shallow` stands in for real in-vivo BEAM libraries. The contract these +# tests hold is that `deep` is unchanged by the existence of `shallow`, and that `shallow` actually lands +# in the regime it claims rather than merely running. + +NARROW_ROWS = [ + ("grp1", SEQ["a"], "Ag Alpha"), + ("grp1", SEQ["b"], "Ag Alpha Var"), + ("grp1", SEQ["c"], "Ctl One (high OT risk)"), + ("grp1", SEQ["d"], "Ctl Two homology"), + ("grp2", SEQ["e"], "Ag Beta"), + ("grp2", SEQ["f"], "Ag Gamma"), + ("grp2", SEQ["c"], "Ctl Other off-target"), +] + + +@pytest.fixture +def narrow_panel_csv(tmp_path): + """The NARROW shape: sample, sequence, antigen — no role column. Role lives in the antigen name.""" + path = tmp_path / "narrow.csv" + with open(path, "w", newline="") as fh: + w = csv.writer(fh) + w.writerow(["Sample", "Sequence", "Antigen"]) + w.writerows(NARROW_ROWS) + return path + + +def test_regime_tables_are_complete(): + """Every regime supplies every key the builder reads, and both tier tables use the same tier names. + + The tier NAMES are shared on purpose: the truth tables, the validator and RUN.md all key off them, + so a regime that renamed a tier would break those silently rather than loudly.""" + keys = set(realpanel.REGIMES["deep"]) + for name, spec in realpanel.REGIMES.items(): + assert set(spec) == keys, f"{name} has {set(spec) ^ keys} against deep" + assert abs(sum(w for _n, w, _d in spec["tiers"]) - 1.0) < 1e-9, f"{name} weights do not sum to 1" + assert [t[0] for t in spec["tiers"]] == realpanel.TIER_NAMES + for tier in ("strong", "good", "medium", "weak", "noise"): + lo, hi = spec["magnitudes"][tier] + assert 0 < lo <= hi + + +def test_shallow_magnitudes_are_ordered_below_deep(): + """Signal ordering holds within a regime, and shallow sits strictly under deep. + + Ordering is what the shallow validator asserts instead of absolute bound rates, so if the + magnitudes ever stop descending the validator's monotonicity check becomes vacuous.""" + for spec in realpanel.REGIMES.values(): + ladder = [spec["magnitudes"][t] for t in ("strong", "good", "medium", "weak", "noise")] + for (lo, _hi), (nlo, _nhi) in zip(ladder, ladder[1:]): + assert lo > nlo + for tier in ("strong", "good", "medium"): + assert realpanel.MAGNITUDES_SHALLOW[tier][1] < realpanel.MAGNITUDES_DEEP[tier][1] + + +def test_deep_regime_is_the_default(panel_csv, tmp_path): + """The default path must not acquire the shallow regime's behaviours by accident: no aggregates, no + sized barcode universe, and the original duplication draw.""" + r = realpanel.REGIMES["deep"] + assert r["aggregates"] == 0 and r["ambient_barcode_ratio"] == 0.0 + assert r["dup_mean"] is None + assert r["unpaired_frac"] == 0.0 + out = run_generator(panel_csv, tmp_path / "run", "--no-validate") + assert out.returncode == 0, out.stderr + assert not (tmp_path / "run" / "truth" / "aggregates.tsv").exists() + assert (tmp_path / "run" / "truth" / "regime.txt").read_text().strip() == "deep" + + +def test_shallow_run_lands_in_the_measured_regime(panel_csv, tmp_path): + """A shallow run reproduces the shape real in-vivo data actually shows, checked from the truth tables + rather than from the log line.""" + out = run_generator(panel_csv, tmp_path / "run", "--regime", "shallow") + assert out.returncode == 0, out.stdout + out.stderr + assert "[validate]" in out.stdout and "FAIL" not in out.stdout, out.stdout + assert (tmp_path / "run" / "truth" / "regime.txt").read_text().strip() == "shallow" + + # Aggregates: five per library, and the largest holds far more than an even split would. + aggs = list(csv.DictReader(open(tmp_path / "run" / "truth" / "aggregates.tsv"), delimiter="\t")) + per_sample = {} + for row in aggs: + per_sample.setdefault(row["sample"], []).append(int(row["umis"])) + assert per_sample, "shallow planted no aggregates" + for sample, umis in per_sample.items(): + assert len(umis) == 5, f"{sample} has {len(umis)} aggregates" + assert max(umis) > sum(umis) / len(umis), f"{sample} aggregates are evenly sized" + + # Cells per clonotype sits near the measured 1.05, which TAIL_CYCLE alone cannot reach. + sizes = [int(r["nCells"]) for r in csv.DictReader(open(tmp_path / "run" / "truth" / "truth_clonotypes.csv"))] + assert sizes + assert 1.0 <= sum(sizes) / len(sizes) <= 1.15, f"{sum(sizes) / len(sizes):.2f} cells per clonotype" + + +def test_shallow_tail_cycle_beats_the_deep_floor(): + """TAIL_CYCLE averages 1.47 cells per clonotype and is a floor no parameter setting gets under. The + sparse cycle is why the shallow regime can reach 1.05.""" + from lib import vdj + assert sum(vdj.TAIL_CYCLE) / len(vdj.TAIL_CYCLE) > 1.4 + sparse = sum(vdj.TAIL_CYCLE_SPARSE) / len(vdj.TAIL_CYCLE_SPARSE) + assert 1.0 < sparse <= 1.06, sparse + assert realpanel.REGIMES["shallow"]["clonal_tail_cycle"] is vdj.TAIL_CYCLE_SPARSE + assert realpanel.REGIMES["deep"]["clonal_tail_cycle"] is None + + +# --- the narrow panel shape ---------------------------------------------------------------------- + +def test_narrow_shape_is_detected_and_wide_still_is(panel_csv, narrow_panel_csv): + assert realpanel.detect_panel_shape(str(panel_csv)) == "wide" + assert realpanel.detect_panel_shape(str(narrow_panel_csv)) == "narrow" + + +def test_narrow_panel_infers_role_from_the_antigen_name(narrow_panel_csv): + """A narrow panel declares no role column, so role has to come from the name. Getting this wrong in + the permissive direction is the dangerous one: a target mistaken for a comparator moves the line + every reading in the sample is judged against.""" + panels = realpanel.load_panel(str(narrow_panel_csv)) + grp1 = panels["grp1"] + assert set(grp1.targets) == {"Ag Alpha", "Ag Alpha Var"} + assert set(grp1.offtargets) == {"Ctl One (high OT risk)", "Ctl Two homology"} + assert set(panels["grp2"].offtargets) == {"Ctl Other off-target"} + + +def test_control_feature_overrides_the_name(narrow_panel_csv): + """Naming a control wins over the name heuristic, mirroring the block's own dropdown.""" + panels = realpanel.load_panel(str(narrow_panel_csv), control_feature="Ag Alpha") + assert "Ag Alpha" in panels["grp1"].offtargets + assert "Ag Alpha" not in panels["grp1"].targets + + +def test_narrow_panel_drives_a_whole_run(narrow_panel_csv, tmp_path): + out = run_generator(narrow_panel_csv, tmp_path / "run", "--regime", "shallow") + assert out.returncode == 0, out.stdout + out.stderr + assert "FAIL" not in out.stdout, out.stdout diff --git a/test/src/csvMeta.test.ts b/test/src/csvMeta.test.ts new file mode 100644 index 0000000..685354b --- /dev/null +++ b/test/src/csvMeta.test.ts @@ -0,0 +1,120 @@ +import { describe, expect, it } from "vitest"; +// parseTagCsvMeta lives in the block UI package; imported directly (pure function, no Vue or driver deps). +import { parseTagCsvMeta } from "../../ui/src/csvMeta"; + +const bytes = (s: string) => new TextEncoder().encode(s); +const parse = (s: string) => parseTagCsvMeta(bytes(s)); + +// These cases were the acceptance tests for the emit_csv_meta.py entrypoint that used to do this job in +// the workflow. They are kept because they describe behaviour the block's dropdowns and its +// duplicate-mapping gate already depend on, not because two parsers now have to agree — there is only one. +describe("parseTagCsvMeta — headers", () => { + it("keeps the file's header order", () => { + const m = parse("Barcode,Name,Type\nAAA,Ag1,Target\n"); + expect(m.columns).toStrictEqual(["Barcode", "Name", "Type"]); + }); + + it("trims surrounding whitespace from headers and values", () => { + const m = parse(" Barcode , Name \nAAA , Ag1\n"); + expect(m.columns).toStrictEqual(["Barcode", "Name"]); + expect(m.valuesByColumn["Barcode"]).toStrictEqual(["AAA"]); + expect(m.valuesByColumn["Name"]).toStrictEqual(["Ag1"]); + }); + + it("drops blank header cells without shifting the columns that follow", () => { + // The trailing/middle empty heading must not make `Name` read its values from the wrong index. + const m = parse("Barcode,,Name\nAAA,junk,Ag1\n"); + expect(m.columns).toStrictEqual(["Barcode", "Name"]); + expect(m.valuesByColumn["Name"]).toStrictEqual(["Ag1"]); + }); + + it("keeps both entries for a duplicated header and reads values from the last one", () => { + const m = parse("Name,Name\nfirst,second\n"); + expect(m.columns).toStrictEqual(["Name", "Name"]); + expect(m.valuesByColumn["Name"]).toStrictEqual(["second"]); + }); +}); + +describe("parseTagCsvMeta — values", () => { + it("deduplicates and sorts each column's values", () => { + const m = parse("Name\nzebra\nalpha\nzebra\nmid\n"); + expect(m.valuesByColumn["Name"]).toStrictEqual(["alpha", "mid", "zebra"]); + }); + + it("ignores blank cells rather than collecting an empty value", () => { + const m = parse("Barcode,Name\nAAA,\nCCC,Ag2\n"); + expect(m.valuesByColumn["Name"]).toStrictEqual(["Ag2"]); + expect(m.valuesByColumn["Barcode"]).toStrictEqual(["AAA", "CCC"]); + }); + + it("tolerates a row shorter than the header", () => { + const m = parse("Barcode,Name,Type\nAAA\nCCC,Ag2,Target\n"); + expect(m.valuesByColumn["Barcode"]).toStrictEqual(["AAA", "CCC"]); + expect(m.valuesByColumn["Type"]).toStrictEqual(["Target"]); + expect(m.rowCount).toBe(2); + }); + + it("emits an empty value list for every column of a header-only file", () => { + const m = parse("Barcode,Name\n"); + expect(m.columns).toStrictEqual(["Barcode", "Name"]); + expect(m.valuesByColumn).toStrictEqual({ Barcode: [], Name: [] }); + expect(m.rowCount).toBe(0); + }); +}); + +describe("parseTagCsvMeta — rowCount", () => { + it("counts data rows and not the header", () => { + expect(parse("Barcode\nAAA\nCCC\nGGG\n").rowCount).toBe(3); + }); + + it("ignores trailing blank rows", () => { + expect(parse("Barcode\nAAA\nCCC\n\n\n").rowCount).toBe(2); + }); + + it("ignores a row whose every cell is blank, not only truly empty lines", () => { + expect(parse("Barcode,Name\nAAA,Ag1\n , \nCCC,Ag2\n").rowCount).toBe(2); + }); + + // This is the comparison the duplicate-mapping gate is built on: more rows than distinct barcodes means + // a barcode is declared more than once, which is legal only for a sample-keyed panel. + it("exceeds the distinct barcode count when a barcode is repeated", () => { + const m = parse("Barcode,Sample\nAAA,s1\nAAA,s2\nCCC,s1\n"); + expect(m.rowCount).toBe(3); + expect(m.valuesByColumn["Barcode"]).toStrictEqual(["AAA", "CCC"]); + }); +}); + +describe("parseTagCsvMeta — real-world dialects", () => { + it("reads CRLF line endings", () => { + // Verified against a real customer panel CSV, which is CRLF. + const m = parse("Barcode,Name\r\nAAA,Ag1\r\nCCC,Ag2\r\n"); + expect(m.columns).toStrictEqual(["Barcode", "Name"]); + expect(m.valuesByColumn["Name"]).toStrictEqual(["Ag1", "Ag2"]); + expect(m.rowCount).toBe(2); + }); + + it("strips a UTF-8 BOM instead of gluing it to the first header", () => { + // An Excel-exported CSV carries a BOM. Left in place it becomes part of the first header's name and + // every later match against that name fails. + const m = parse("Barcode,Name\nAAA,Ag1\n"); + expect(m.columns).toStrictEqual(["Barcode", "Name"]); + expect(m.valuesByColumn["Barcode"]).toStrictEqual(["AAA"]); + }); + + it("honours quoting: commas, newlines and doubled quotes inside a field", () => { + const m = parse('Barcode,Name\nAAA,"Ag1, variant b"\nCCC,"multi\nline"\nGGG,"say ""hi"""\n'); + // Sorted by code point, so "multi\nline" precedes 'say "hi"'. + expect(m.valuesByColumn["Name"]).toStrictEqual(["Ag1, variant b", "multi\nline", 'say "hi"']); + expect(m.rowCount).toBe(3); + }); +}); + +describe("parseTagCsvMeta — refusals", () => { + it("throws on an empty file rather than reporting no columns", () => { + expect(() => parse("")).toThrow(/empty/i); + }); + + it("throws where every heading is blank", () => { + expect(() => parse(",,\nAAA,BBB,CCC\n")).toThrow(/header/i); + }); +}); diff --git a/test/src/progress.test.ts b/test/src/progress.test.ts index 9b80924..0935913 100644 --- a/test/src/progress.test.ts +++ b/test/src/progress.test.ts @@ -1,15 +1,19 @@ import { describe, expect, it } from "vitest"; import { ProgressPrefix } from "@platforma-open/milaboratories.feature-integration.model"; -// deriveProgress lives in the block UI package; imported directly (pure function, model-only deps). +// deriveProgress lives in the block UI package, and is imported directly: a pure function with model-only +// deps. import { deriveProgress } from "../../ui/src/progress"; const S = "SAMPLE1"; -const line = (stage: string) => `${ProgressPrefix}${stage}`; +// Every case below is a stream that is still open, which is what these tests are about. `closed` is the +// other half: a finished step's last tick, which must not be replayed as a live reading. +const line = (stage: string) => ({ line: `${ProgressPrefix}${stage}`, live: true }); +const closed = (stage: string) => ({ line: `${ProgressPrefix}${stage}`, live: false }); describe("deriveProgress — label follows the live stream, not the report step", () => { it("does not flash 'Counting UMIs' while refine is still the furthest live step", () => { - // Report step already advanced to "counting", but only refine has a live line → label must stay - // on refine (the pre-fix bug flashed "Counting UMIs" here). + // The report step has advanced to "counting" while only refine has a live line, so the label must stay on + // refine. Following the report step flashes "Counting UMIs" here. const cell = deriveProgress( S, new Set(), @@ -24,7 +28,7 @@ describe("deriveProgress — label follows the live stream, not the report step" }); it("shows a Counting UMIs variant once tag-stat streams", () => { - // Now tag-stat is the furthest step with a live line → label follows it. + // tag-stat is now the furthest step with a live line, so the label follows it. const cell = deriveProgress( S, new Set(), @@ -37,8 +41,40 @@ describe("deriveProgress — label follows the live stream, not the report step" expect(cell.text).toContain("Counting UMIs"); }); + it("does not replay a finished step's last tick as a live reading", () => { + // mitool prints progress on a timer and the process finishes between ticks, so a completed parse ends + // on whatever tick landed last -- 97.8% with a one-second ETA is a normal way for it to end. Shown as + // live it says the run is nearly through a step it already finished, and offers an ETA that never + // elapses. A closed stream sits at the TOP of its band instead. + const cell = deriveProgress( + S, + new Set(), + { [S]: "parsing" }, + { + "1-parse": closed("Parsing sequences: 97.8% ETA: 00:00:01"), + }, + ); + expect(cell.percent).toBe(25); + expect(cell.text).not.toContain("97.8"); + expect(cell.suffix).toBe(""); + }); + + it("still shows a live step's own percent", () => { + // The other side of the same rule: an open stream is a reading and its figures are current. + const cell = deriveProgress( + S, + new Set(), + { [S]: "parsing" }, + { + "1-parse": line("Parsing sequences: 40.0% ETA: 00:00:30"), + }, + ); + expect(cell.text).toContain("40.0"); + expect(cell.percent).toBeLessThan(25); + }); + it("keeps the bar monotonic — never below the reported step floor", () => { - // Report floor for "counting" is 50; refine's within-band fill must not drag the bar below it. + // The report floor for "counting" is 50, and refine's within-band fill must not drag the bar below it. const cell = deriveProgress( S, new Set(), diff --git a/test/src/qcDefaults.test.ts b/test/src/qcDefaults.test.ts new file mode 100644 index 0000000..519969d --- /dev/null +++ b/test/src/qcDefaults.test.ts @@ -0,0 +1,67 @@ +import { readFileSync } from "node:fs"; +import { join } from "node:path"; +import { describe, expect, it } from "vitest"; +import { + AGGREGATE_DETECTION_DEFAULTS, + QC_LINE_DEFAULTS, +} from "@platforma-open/milaboratories.feature-integration.model"; + +// The settings fields display these numbers where the stored value is undefined, and the workflow +// substitutes its own copy on the command line. Two copies that disagree put a number on screen that no +// run was scored against, and nothing else in the block compares them. +// +// Read as text rather than imported: one source is Tengo and the other Python. +const root = join(__dirname, "..", ".."); +const tengo = readFileSync(join(root, "workflow/src/verdict-args.lib.tengo"), "utf8"); +const python = readFileSync(join(root, "software/per-cell-metrics/src/qc_measures.py"), "utf8"); + +function tengoConst(name: string): number { + const m = tengo.match(new RegExp(`^${name}\\s*:=\\s*([0-9.]+)\\s*$`, "m")); + if (!m) throw new Error(`verdict-args.lib.tengo declares no ${name}`); + return Number(m[1]); +} + +function pythonConst(name: string): number { + const m = python.match(new RegExp(`^${name}\\s*:\\s*\\w+\\s*=\\s*([0-9.]+)\\s*$`, "m")); + if (!m) throw new Error(`qc_measures.py declares no ${name}`); + return Number(m[1]); +} + +describe("QC_LINE_DEFAULTS matches verdict-args.lib.tengo", () => { + const pairs: [keyof typeof QC_LINE_DEFAULTS, string][] = [ + ["cellBarcodeValidWarn", "DEFAULT_CELL_BARCODE_VALID_WARN"], + ["cellBarcodeValidError", "DEFAULT_CELL_BARCODE_VALID_ERROR"], + ["readsPerCellWarn", "DEFAULT_READS_PER_CELL_WARN"], + ["aggregateBarcodeWarn", "DEFAULT_AGGREGATE_BARCODE_WARN"], + ["aggregateBarcodeError", "DEFAULT_AGGREGATE_BARCODE_ERROR"], + ["undeclaredBarcodeWarn", "DEFAULT_UNDECLARED_BARCODE_WARN"], + ["undeclaredBarcodeError", "DEFAULT_UNDECLARED_BARCODE_ERROR"], + ["usableReadWarn", "DEFAULT_USABLE_READ_WARN"], + ["usableReadError", "DEFAULT_USABLE_READ_ERROR"], + ]; + + it.each(pairs)("%s", (key, tengoName) => { + expect(QC_LINE_DEFAULTS[key]).toBe(tengoConst(tengoName)); + }); + + it("covers every line the tengo file declares", () => { + const declared = [ + ...tengo.matchAll( + /^(DEFAULT_(?:CELL_BARCODE_VALID|READS_PER_CELL|AGGREGATE_BARCODE|UNDECLARED_BARCODE|USABLE_READ)_[A-Z]+)\s*:=/gm, + ), + ].map((m) => m[1]); + expect(new Set(declared)).toStrictEqual(new Set(pairs.map(([, name]) => name))); + }); +}); + +describe("AGGREGATE_DETECTION_DEFAULTS matches qc_measures.py", () => { + const pairs: [keyof typeof AGGREGATE_DETECTION_DEFAULTS, string][] = [ + ["aggregateBarcodeIqrMultiplier", "AGGREGATE_BARCODE_IQR_MULTIPLIER"], + ["aggregateBarcodeMinUmiThreshold", "AGGREGATE_BARCODE_MIN_THRESHOLD"], + ["aggregateBarcodeTopN", "AGGREGATE_BARCODE_TOP_N"], + ]; + + it.each(pairs)("%s", (key, pythonName) => { + expect(AGGREGATE_DETECTION_DEFAULTS[key]).toBe(pythonConst(pythonName)); + }); +}); diff --git a/test/src/wf.test.ts b/test/src/wf.test.ts index 55961b7..7d7eaf2 100644 --- a/test/src/wf.test.ts +++ b/test/src/wf.test.ts @@ -10,15 +10,26 @@ import { FeatureIntegrationBlockPointer as myBlockSpec } from "this-block"; import type { InferBlockState, PTableHandle } from "@platforma-sdk/model"; import { createPlDataTableStateV2, wrapOutputs } from "@platforma-sdk/model"; -// Level-4 integration test (plan Task 7): a live end-to-end run emitting `pl7.app/feature/umiCount`. +// Block tests for the Feature Barcode Profiling block. // -// Upstream chain follows the proven samples-and-data FASTQ pattern (blocks/mixcr-amplicon-alignment). -// The tag->feature CSV is a direct upload (M7 resolution): set as the block arg `tagFeatureCsvHandle` -// via a local file handle; the workflow imports it with file.importFile and shares the blob across the -// per-sample bodies. +// WHAT THIS FILE COVERS. The block has two halves and this file reaches one of them: // -// Golden (decoded from test/assets/fb_small_R{1,2}.fastq.gz; geometry CELL 16 + UMI 10 on R1, feature -// 15 on R2; tags.csv: 15xG -> AGX, 15xC -> BGX): +// * the per-sample counting half, FASTQ in and per-cell UMI counts out, is exercised end to end by +// the second test below, when a backend can run it. +// * the ANTIGEN VERDICT half is not covered here at all. It needs a single-cell V(D)J dataset +// upstream to supply the clonotype sets, and the samples-and-data chain this file uses cannot +// produce one. The verdict logic is covered by the Python suite +// (software/per-cell-metrics/test/) and by the Tengo suite for the p-column specs +// (workflow/src/*.test.tengo). Neither substitutes for a live run, which is why the block is +// verified by hand against software/test-data/fixtures/verdicts/ before release. +// +// The upstream chain follows the proven samples-and-data FASTQ pattern +// (blocks/mixcr-amplicon-alignment). The tag->feature CSV is a direct upload, set as the block arg +// `tagFeatureCsvHandle` through a local file handle, and the workflow imports it with file.importFile +// and shares the blob across the per-sample bodies. +// +// Golden, decoded from test/assets/fb_small_R{1,2}.fastq.gz. Geometry is CELL 16 + UMI 10 on R1 and +// feature 15 on R2, and tags.csv maps 15xG -> AGX and 15xC -> BGX: // read0: cell ACGTACGTACGTACGT, UMI AAAAAAAAAA, feature AGX // read1: cell ACGTACGTACGTACGT, UMI AAAAAAAAAC, feature AGX // read2: cell ACGTACGTACGTACGT, UMI AAAAAAAAAG, feature BGX @@ -37,26 +48,49 @@ blockTest("empty inputs", { timeout: 20000 }, async ({ rawPrj: project, expect } project.getBlockState(blockId), 15000, )) as InferBlockState; - // With no upstream FASTQ column in the pool the option list is empty (args() throws, disabling Run, - // but outputs still resolve). - expect(stableState.outputs).toMatchObject({ fastqOptions: { ok: true, value: [] } }); + // With no upstream FASTQ column in the pool the option lists are empty. args() throws, disabling Run, + // and the outputs still resolve. + expect(stableState.outputs).toMatchObject({ + fastqOptions: { ok: true, value: [] }, + datasetOptions: { ok: true, value: [] }, + }); + + // Every output a page reads must RESOLVE on a freshly added block, before anything has run. An output + // that throws here is not a failed computation: it breaks the page that reads it at the moment the block + // is created, which is the first thing a user sees. Each verdict-branch output is guarded by its own + // undefined-until-computed path, and the guards are what this asserts. Values are deliberately not + // asserted, because `ok` with an undefined value is the correct empty-state answer for all of them, + // punchcardIdentityOptions included, which answers with an empty list before any run. + const mustResolve = [ + "perCellTable", + "qcSummaryTable", + "punchcardTable", + "punchcardIdentityOptions", + "verdictRunMeta", + "isRunning", + "started", + ] as const; + const unresolved = mustResolve.filter((name) => stableState.outputs?.[name]?.ok !== true); + expect(unresolved, "outputs that failed to resolve on an empty block").toEqual([]); }); // Level-4 end-to-end run against the published mitool (software-mitool 2.3.1-129-main, carrying the -// FEATURE tag type #86 + tag-stat -u #84). Exercises the full per-sample chain: SND FASTQ upload, -// fastqOptions, args derivation, the tag->feature CSV upload (driven by prerun.tpl), the mitool +// FEATURE tag type #86 and tag-stat -u #84). Exercises the full per-sample chain: SND FASTQ upload, +// fastqOptions, args derivation, the tag->feature CSV upload driven by prerun.tpl, the mitool // parse -> refine-tags -> tag-stat -u exec chain, the per-cell-metrics Python, and the processColumn // export emitting pl7.app/feature/umiCount. // -// SKIPPED (2026-07-06): hangs on the CI / run-platforma FS-storage backend. The tag->feature CSV is a -// DIRECT upload consumed by file.importFile (prerun.tpl + main.tpl), and raw file.importFile of a local -// handle never finalizes on that backend — so the prerun's `csvColumns` never resolves and -// awaitStableState aborts with `field_not_resolved:csvColumns`. This is a test-backend limitation, not -// a block bug: the block runs correctly against a real backend (driven live). No block e2e-tests this -// direct-upload path — every block that consumes a file.importFile handle (immune-assay-data, blast, -// makeblastdb, antibody-sequence-liabilities) ships without block tests, and the Samples & Data upstream -// chain (the one CI-working file-input pattern) cannot supply a direct CSV upload. Re-enable if the -// backend gains a working local file.importFile, or if the CSV moves to a pool column. +// SKIPPED (2026-07-06): it hangs on the CI and run-platforma FS-storage backend. The tag->feature CSV +// is a DIRECT upload consumed by file.importFile, in prerun.tpl and main.tpl, and a raw +// file.importFile of a local handle never finalizes on that backend. So the prerun's `csvColumns` +// never resolves and awaitStableState aborts with `field_not_resolved:csvColumns`. A test-backend +// limitation rather than a block bug: the block runs correctly against a real backend, driven live. +// +// No block e2e-tests this direct-upload path. Every block consuming a file.importFile handle +// (immune-assay-data, blast, makeblastdb, antibody-sequence-liabilities) ships without block tests, +// and the Samples & Data upstream chain, the one CI-working file-input pattern, cannot supply a direct +// CSV upload. Re-enable if the backend gains a working local file.importFile, or if the CSV moves to a +// pool column. blockTest.skip( "feature integration end-to-end emits per-cell umiCount", { timeout: 300000 }, @@ -118,17 +152,46 @@ blockTest.skip( // The tag->feature CSV is a direct upload (not a pool ref). Provision it as a local file handle. const csvHandle = await helpers.getLocalFileHandle("./assets/tags.csv"); - // Configure the block. update-block-data must carry EVERY BlockArgsValid field, else the backend - // reports "currentArgs not set". controlFeature is optional (no specificity score here). + // Configure the block. update-block-data must carry EVERY BlockArgsValid field, or the backend + // reports "currentArgs not set". controlFeature is optional, since there is no negative-control + // marker here, and so is datasetRef: this run has no single-cell V(D)J dataset, so it exercises the + // counting half alone. The reading's numeric parameters are required and carry the shipped + // defaults, the same values a freshly created block starts with. + // + // What a datasetless run emits BESIDES the per-cell table is deliberately not asserted. Today the + // whole antigen stage is skipped, so nothing antigen-related is produced, while the spec's + // qc-measurement set requires the eight read-and-panel measurements and the panel mismatch report + // to survive a run with no cell list, marking the rest not-evaluated. That gap is open (decision + // log O-4), and asserting today's behaviour would have to be deleted to fix it, so this test + // asserts only what both readings agree on. await project.mutateBlockStorage(fiBlockId, { operation: "update-block-data", value: { fbFastqRef: fiOutputs1.fastqOptions[0].ref, tagFeatureCsvHandle: csvHandle, - dominanceThreshold: 0.6, - presetId: "tenx-beam", // 10x 5' v2 BEAM geometry (16/10/15); pattern owned by the preset + presetId: "tenx-beam", // 10x 5' v2 BEAM geometry (16/10/15). The preset owns the pattern. + countFloor: 4, + boundCutoff: 75, + minVotingCells: 1, + // Stated, because the block chooses no baseline for anyone and `args()` refuses a run without + // one. The tag-distribution rung is the one this bed can serve: the declared rung needs a role + // column marking a baseline tag, which this panel does not carry, and the panel's own readings + // are retired. + referenceSource: "distribution", + // Still required by the data shape though the rung it gated is retired. Recorded as debt in + // .meta/milab-6496-ui-first-logic-debt-20260824.md. + panelReferenceMinMembers: 25, + distributionMinCells: 300, tableState: createPlDataTableStateV2(), qcSummaryTableState: createPlDataTableStateV2(), + punchcardTableState: createPlDataTableStateV2(), + runQualityTableState: createPlDataTableStateV2(), + scoreDistributionGraphState: { title: "Scores", template: "line" }, + referenceReadingGraphState: { title: "Reference readings", template: "line" }, + fittedBackgroundGraphState: { title: "Backgrounds", template: "dots" }, + runQualityMismatchTableState: createPlDataTableStateV2(), + reagentTableState: createPlDataTableStateV2(), + undeclaredBarcodesTableState: createPlDataTableStateV2(), } satisfies BlockData, }); @@ -145,27 +208,42 @@ blockTest.skip( const pFrameDriver = ml.driverKit.pFrameDriver; const fullHandle = tableOutput.value!.fullTableHandle as PTableHandle; - // perCellTable is COLLAPSED to one row per cell [sampleId, cellId] (DECISION 2026-07-02): the - // per-(cell,feature) matrix — (cellA,AGX)=2, (cellA,BGX)=1, (cellB,AGX)=1 — becomes 2 rows: cellA, - // cellB. (The per-feature matrix is still exported to the pool, just not in this table.) + // perCellTable is COLLAPSED to one row per cell [sampleId, cellId] (DECISION 2026-07-02). The + // per-(cell,feature) matrix — (cellA,AGX)=2, (cellA,BGX)=1, (cellB,AGX)=1 — becomes 2 rows, cellA and + // cellB. The per-feature matrix is still exported to the pool, just not in this table. const shape = await pFrameDriver.getShape(fullHandle); expect(shape.rows).toBe(2); const indices = Array.from({ length: shape.columns }, (_, i) => i); const data = await pFrameDriver.getData(fullHandle, indices); - // The only Int column is pl7.app/feature/maxUmiCount (the cell's largest per-feature UMI count): - // cellA max(2,1)=2, cellB=1. maxFraction is a Double column, so it is not counted here. Order is - // unspecified -> compare sorted. + // The only Int column is pl7.app/feature/maxUmiCount, the cell's largest per-feature UMI count: cellA + // max(2,1)=2, cellB=1. maxFraction is a Double column, so it is not counted here. Order is unspecified, + // so compare sorted. const maxUmiColumns = data.filter((c) => c.type === "Int"); expect(maxUmiColumns).toHaveLength(1); const maxUmiCounts = [...maxUmiColumns[0].data].map(Number).sort((a, b) => a - b); expect(maxUmiCounts).toEqual([1, 2]); expect(maxUmiCounts.reduce((a, b) => a + b, 0)).toBe(3); - // Consensus feature per cell: cellA dominant AGX (2 of 3 = 0.67 >= 0.6), cellB single-feature AGX. - // The consensusFeature String column is thus "AGX" for both cells. + // The cell's largest per-feature share of its own total: cellA 2/3, cellB 1/1. The one surviving + // per-cell magnitude. It says how concentrated a cell's counts were rather than which antigen it bound, + // so it is not a binding level and does not fall under the no-ordering prohibition. + const fractionColumns = data.filter((c) => c.type === "Double"); + expect(fractionColumns).toHaveLength(1); + const maxFractions = [...fractionColumns[0].data].map(Number).sort((a, b) => a - b); + expect(maxFractions[0]).toBeCloseTo(2 / 3, 5); + expect(maxFractions[1]).toBeCloseTo(1, 5); + + // No dominant-feature call, and it must never come back through this table. It answers a different + // question from the four-state verdict -- one antigen per cell, chosen by a share threshold -- and + // beside a verdict it would give a reader two disagreeing answers with no rule for which wins. + // `guardNoScore` in column-specs.lib.tengo refuses the score annotation at build time, and this is + // the same claim checked against what actually reached a table. const stringColumnValues = data.filter((c) => c.type === "String").map((c) => [...c.data]); - expect(stringColumnValues.some((vals) => vals.every((v) => v === "AGX"))).toBe(true); + expect( + stringColumnValues.some((vals) => vals.every((v) => v === "AGX")), + "a per-cell column is calling one dominant feature — consensusFeature has returned", + ).toBe(false); }, ); diff --git a/ui/package.json b/ui/package.json index 661be34..1c00c2b 100644 --- a/ui/package.json +++ b/ui/package.json @@ -11,10 +11,13 @@ "check": "ts-builder check --target block-ui" }, "dependencies": { + "@milaboratories/graph-maker": "catalog:", "@platforma-open/milaboratories.feature-integration.model": "workspace:*", + "@platforma-sdk/model": "catalog:", "@platforma-sdk/ui-vue": "catalog:", "ag-grid-enterprise": "catalog:", "ag-grid-vue3": "catalog:", + "csv-parse": "catalog:", "vue": "catalog:" }, "devDependencies": { diff --git a/ui/src/app.ts b/ui/src/app.ts index b3b6a76..3d027b4 100644 --- a/ui/src/app.ts +++ b/ui/src/app.ts @@ -1,15 +1,17 @@ import { platforma } from "@platforma-open/milaboratories.feature-integration.model"; import { defineAppV3 } from "@platforma-sdk/ui-vue"; import { watchEffect } from "vue"; +import AntigenQcPage from "./pages/AntigenQcPage.vue"; import MainPage from "./pages/MainPage.vue"; import QcSummaryPage from "./pages/QcSummaryPage.vue"; +import PunchcardPage from "./pages/PunchcardPage.vue"; import ResultsPage from "./pages/ResultsPage.vue"; export const sdkPlugin = defineAppV3(platforma, (app) => { - // Block-label pattern: mirror the model's suggestedBlockLabel (" / → ") - // into data.defaultBlockLabel, which the sidebar subtitle reads. The subtitle render context has no - // result pool, so the dataset label is resolved in the output and copied here. Guarded so it only - // writes on change (multi-client safe). + // Block-label pattern: mirror the model's suggestedBlockLabel (" / -> ") into + // data.defaultBlockLabel, which the sidebar subtitle reads. The subtitle render context has no result pool, + // so the dataset label is resolved in the output and copied here. Guarded to write only on change, which is + // multi-client safe. watchEffect(() => { const suggested = app.model.outputs.suggestedBlockLabel ?? ""; if (app.model.data.defaultBlockLabel !== suggested) { @@ -17,11 +19,11 @@ export const sdkPlugin = defineAppV3(platforma, (app) => { } }); - // A stale negative-control selection is cleared on the user gesture that invalidates it — changing the - // CSV or the feature-name column (see MainPage.vue's clearControlOnInputChange). It is deliberately NOT - // done here via a watcher on the controlOptions output: that would be the spec-facts-resync hairpin - // (output → data write; see hairpin.md / model.md), and a gesture-driven data→data write has no - // multi-client interleave and no hydration-timing risk. + // A stale negative-control selection is cleared on the user gesture that invalidates it, meaning a change + // to the CSV or the feature-name column (see MainPage.vue's clearControlOnInputChange). Never here, through + // a watcher on the controlOptions output: that is the spec-facts-resync hairpin, an output-to-data write + // (see hairpin.md and model.md). A gesture-driven data-to-data write has no multi-client interleave and no + // hydration-timing risk. return { // Drive the block spinner while the main run is executing. @@ -30,6 +32,10 @@ export const sdkPlugin = defineAppV3(platforma, (app) => { "/": () => MainPage, "/qc": () => QcSummaryPage, "/results": () => ResultsPage, + "/punchcard": () => PunchcardPage, + // The run's own quality: the verdict stage's measurements and the panel-versus-reads check. Distinct + // from "/qc", the mitool per-sample read statistics. + "/antigen-qc": () => AntigenQcPage, }, }; }); diff --git a/ui/src/clonotypeLabels.ts b/ui/src/clonotypeLabels.ts new file mode 100644 index 0000000..ed80c75 --- /dev/null +++ b/ui/src/clonotypeLabels.ts @@ -0,0 +1,111 @@ +import type { AxisId, PFrameHandle } from "@platforma-sdk/model"; +import { getColumnsFull, getSingleColumnData } from "@platforma-sdk/model"; +import { ref, watch, type ComputedRef } from "vue"; + +/** + * The clonotype's readable name (`C-ZDKEZ`) and its cell count, for the one place that cannot get them from a + * table: the expansion's title. + * + * The name is real stored data, not something this block derives. Upstream emits it as a `pl7.app/label` + * column on the clonotype axis, generated by `mixcr-clonotyping/workflow/src/clonotype-label.lib.tengo`. A + * TABLE shows it without being asked -- `PlAgDataTableV2` detects label columns and substitutes them into + * axis cells, which is why the card's `Clone Id` column reads `C-ZDKEZ` while the row event that opens the + * panel carries only the raw key. + * + * The model cannot look it up. Both columns are Parquet-stored, and the model-side label APIs + * (`deriveAxisValuesLabels`, the older `findLabels`) skip any label column that is not JSON-backed. The + * pFrame DRIVER reads Parquet, so the lookup goes through it, and `getColumnsFull` / `getSingleColumnData` + * are the SDK's own helpers for that rather than a hand-rolled query. + * + * Lifted from `blocks/3D-Structure-Based-Liabilities/ui/src/composables/useClonotypeLabels.ts`, which builds + * a detail-modal title the same way, and `3d-structure-prediction` does it too. Both take the whole column + * rather than one key, and so does this: it is one value per clonotype and it leaves the map ready for the + * next one. `getSingleColumnData` accepts filters if a large run ever makes that the wrong trade. + */ +export function useClonotypeLabels( + labelsPf: ComputedRef, + axisId: ComputedRef, +) { + const labels = ref>({}); + const cellCounts = ref>({}); + + /** + * One column's values, keyed by axis value. Returns an empty map rather than throwing: a missing title is a + * cosmetic loss, and the caller has a generic word to fall back on. + */ + async function valuesByAxisKey( + handle: PFrameHandle, + axis: AxisId, + columnName: string, + ): Promise> { + const cols = await getColumnsFull(handle, { + selectedSources: [], + strictlyCompatible: false, + names: [columnName], + }); + // One axis, and it has to be OUR axis. This frame carries other one-axis columns keyed on other things, + // and a map built from the wrong axis would attach names to keys that are not clonotypes. + const match = cols.find( + (c) => c.spec.axesSpec.length === 1 && c.spec.axesSpec[0].name === axis.name, + ); + if (match === undefined) return {}; + const { axesData, data } = await getSingleColumnData(handle, match.columnId); + const axisKeys = Object.values(axesData)[0]; + // Paired by position, so a length mismatch means these are not the same rows and every value would land + // on the wrong clonotype. Nothing is worse than a confidently wrong name. + if (axisKeys === undefined || axisKeys.length !== data.length) return {}; + const out: Record = {}; + for (let i = 0; i < axisKeys.length; i++) { + const key = axisKeys[i]; + const value = data[i]; + if (key !== null && value !== null) out[String(key)] = String(value); + } + return out; + } + + watch( + [labelsPf, axisId], + async ([handle, axis]) => { + if (handle === undefined || axis === undefined) { + labels.value = {}; + cellCounts.value = {}; + return; + } + try { + const [resolvedLabels, resolvedCounts] = await Promise.all([ + valuesByAxisKey(handle, axis, "pl7.app/label"), + valuesByAxisKey(handle, axis, "pl7.app/antigen/cellCount"), + ]); + labels.value = resolvedLabels; + cellCounts.value = resolvedCounts; + } catch (err) { + // A failed lookup costs a title, never the panel. Logged rather than swallowed: a name that never + // resolves means the driver call is wrong, and someone should be able to see that. + console.warn("clonotype label lookup failed", err); + labels.value = {}; + cellCounts.value = {}; + } + }, + { immediate: true }, + ); + + /** + * The clonotype's name and cell count, as the design writes it: `C-ZDKEZ — 4 cells`. + * + * Undefined while the lookup is in flight or if it failed, and deliberately NOT falling back to the raw + * key. That key is the string this exists to keep off the screen: it appears nowhere else in the block, so + * it names nothing and reads as a bug. A caller with no name says something generic instead. + * + * The count is optional on its own -- a name with no count still names the clonotype, so a missing count + * degrades the title rather than removing it. + */ + const resolveTitle = (key: string | number | null | undefined): string | undefined => { + if (key === null || key === undefined) return undefined; + const label = labels.value[String(key)]; + if (label === undefined) return undefined; + const cells = cellCounts.value[String(key)]; + return cells === undefined ? label : `${label} — ${cells} cells`; + }; + + return { labels, cellCounts, resolveTitle }; +} diff --git a/ui/src/components/CellPunchCell.vue b/ui/src/components/CellPunchCell.vue new file mode 100644 index 0000000..ca834e4 --- /dev/null +++ b/ui/src/components/CellPunchCell.vue @@ -0,0 +1,141 @@ + + + diff --git a/ui/src/components/PatternEditor.vue b/ui/src/components/PatternEditor.vue index 37d5f00..4fd253c 100644 --- a/ui/src/components/PatternEditor.vue +++ b/ui/src/components/PatternEditor.vue @@ -12,7 +12,6 @@ import { PlCheckbox, PlDropdown, PlNumberField, - PlSectionSeparator, PlTextField, } from "@platforma-sdk/ui-vue"; import { computed, reactive, ref, watch } from "vue"; @@ -39,8 +38,8 @@ const isUserConfigurable = computed(() => selectedPreset.value?.userConfigurable function setPresetId(id: string | undefined) { app.model.data.presetId = id; const p = getPreset(id); - // User-configurable preset: seed a starting pattern so the builder opens populated. Fixed presets own - // their pattern (args reads preset.pattern) — nothing to write into data. + // User-configurable preset: seed a starting pattern so the builder opens populated. A fixed preset owns its + // pattern, which args reads from preset.pattern, so there is nothing to write into data. if (p?.userConfigurable && !app.model.data.pattern) { app.model.data.pattern = assemblePattern(DEFAULT_PARTS); } @@ -82,16 +81,18 @@ const patternParseError = computed(() => { if (!isUserConfigurable.value) return null; const p = app.model.data.pattern; if (!p) return null; - // Loose check (same rule as the model's args): mitool does the real parsing; we only require the - // CELL/UMI/FEATURE tags + R2 capture. Extra flanks/spacers/anchors are allowed via the string field. + // Loose check, the same rule as the model's args. mitool does the real parsing, and only the CELL/UMI/FEATURE + // tags and the R2 capture are required here. An extra flank, spacer or anchor is allowed through the string + // field. return validatePattern(p); }); -// ── Bidirectional sync (data.pattern ↔ builder fields). Hairpin-safe: both directions read/write -// data.pattern (persisted data) and local refs, never a model output. ── +// ── Bidirectional sync (data.pattern <-> builder fields). Hairpin-safe: both directions read and write +// data.pattern, which is persisted data, and local refs. Never a model output. ── const lastAssembled = ref(undefined); -// data.pattern → fields (skips our own write; leaves fields untouched on an unparseable write-mode string) +// data.pattern -> fields. Skips this component's own write, and leaves the fields untouched on an +// unparseable write-mode string. watch( () => app.model.data.pattern, (pattern) => { @@ -107,7 +108,7 @@ watch( { immediate: true }, ); -// fields → data.pattern (build mode only) +// fields -> data.pattern (build mode only) function reassembleFromFields() { const assembled = preview.value; lastAssembled.value = assembled; @@ -228,10 +229,6 @@ watch(editorMode, (mode) => { - - - diff --git a/ui/src/components/PunchCell.vue b/ui/src/components/PunchCell.vue new file mode 100644 index 0000000..da5eafc --- /dev/null +++ b/ui/src/components/PunchCell.vue @@ -0,0 +1,185 @@ + + + diff --git a/ui/src/components/PunchLegend.vue b/ui/src/components/PunchLegend.vue new file mode 100644 index 0000000..f07f193 --- /dev/null +++ b/ui/src/components/PunchLegend.vue @@ -0,0 +1,113 @@ + + + diff --git a/ui/src/components/QcEntityCell.vue b/ui/src/components/QcEntityCell.vue new file mode 100644 index 0000000..bdcd310 --- /dev/null +++ b/ui/src/components/QcEntityCell.vue @@ -0,0 +1,12 @@ + + + diff --git a/ui/src/components/QcSection.vue b/ui/src/components/QcSection.vue new file mode 100644 index 0000000..b5b62cd --- /dev/null +++ b/ui/src/components/QcSection.vue @@ -0,0 +1,144 @@ + + + + + diff --git a/ui/src/components/punchMarks.ts b/ui/src/components/punchMarks.ts new file mode 100644 index 0000000..87b23d7 --- /dev/null +++ b/ui/src/components/punchMarks.ts @@ -0,0 +1,97 @@ +import type { CSSProperties } from "vue"; + +/** + * The punchcard's marks, in one place. + * + * The cell renderer and the legend both paint from this map, so the legend cannot describe a colour the card + * does not draw. Two hand-maintained copies is the ordinary way a legend goes wrong, and it is invisible + * when it does: both halves look deliberate, and only a reader comparing them closely would notice that the + * swatch and the cell disagree. + * + * Styles are inline values rather than CSS classes because these are consumed inside an ag-grid cell + * renderer, which is instantiated outside Vue's scope-id context: a scoped stylesheet emits every rule with + * a `[data-v-...]` attribute the rendered elements do not carry, so not one rule matches and the card paints + * blank. That is not hypothetical -- it shipped, and the card was reported as empty. + */ +export type PunchGlyph = "bound" | "not-bound" | "unreliable" | "unknown"; + +/** + * One diameter for every mark on the card. + * + * The card used to size bound and not-bound by how many of a clonotype's cells answered, on the reading that + * a verdict resting on three cells must not look like one resting on forty. The obligation behind that is + * `support-travels-with-the-reading`, and it is a delivery obligation: it fixes that the scientist is HANDED + * the two counts, not that a dot encode them. They are handed over twice already -- in the per-cell tooltip, + * and as columns in the clonotype expansion -- so the card is free to be a field of flat colour, which is + * what it is actually read as at panel density. + */ +export const PUNCH_DIAMETER_PX = 22; + +/** + * The legend's swatches, smaller than the card's marks on purpose. + * + * Two numbers rather than one is safe here for the reason the encoding was removed: a diameter carries no + * meaning any more, so a swatch drawn at a different size cannot misreport anything -- it is an example of a + * COLOUR, at the scale a line of text wants. While the card sized its dots by evidence this would have been + * a real hazard, because the swatch would have been read as one particular amount of support. + */ +export const PUNCH_LEGEND_DIAMETER_PX = 11; + +export const PUNCH_PAINT: Record = { + bound: { background: "#1a7f37" }, + "not-bound": { background: "#d94438" }, + unreliable: { background: "#9aa3ae" }, + // Not a verdict, but a value the renderer could not read. Marked rather than left blank, because blank + // already means "never asked" on this card. + unknown: { border: "1.5px dotted #d94438", opacity: "0.7" }, +}; + +// ONE decoder for the punch value, shared by the grid cell and the clonotype expansion. The value is a single +// `|`-joined string because a grid pairs a cell with another column's cell only by position, and no import +// guarantees that, so everything a position needs travels together. Two readers of one format would be two +// chances to disagree about it, which is why this lives here rather than in a component. +// +// state | cellsAnswered | cellsCouldAnswer | agreement | unreliableReason | cellsBound +// +// `cellsBound` is the sixth field and was appended, so a value written before it existed has five and still +// decodes. Anything that does not decode is reported as such rather than guessed at: an unreadable value must +// never pass for an answer. +export const VERDICT_STATES = ["bound", "not bound", "unreliable", "never asked"] as const; +export type VerdictState = (typeof VERDICT_STATES)[number]; + +export type Punch = + | { + kind: "read"; + state: VerdictState; + answered: number; + couldAnswer: number; + agreement?: number; + reason?: string; + bound?: number; + } + | { kind: "unparsed" }; + +export function parsePunch(raw: unknown): Punch { + if (typeof raw !== "string") return { kind: "unparsed" }; + const parts = raw.split("|"); + if (parts.length !== 5 && parts.length !== 6) return { kind: "unparsed" }; + const [state, answered, couldAnswer, agreement, reason, bound] = parts; + const known = VERDICT_STATES.find((s) => s === state); + const a = Number(answered); + const c = Number(couldAnswer); + if (known === undefined || !Number.isFinite(a) || !Number.isFinite(c)) + return { kind: "unparsed" }; + // Empty is carried as absent rather than as zero. A settled verdict has no reason, a set nobody could ask + // has no agreement, and a zero agreement would read as total disagreement. + const ag = agreement === "" ? undefined : Number(agreement); + const b = bound === undefined || bound === "" ? undefined : Number(bound); + return { + kind: "read", + state: known, + answered: a, + couldAnswer: c, + agreement: ag !== undefined && Number.isFinite(ag) ? ag : undefined, + reason: reason === "" ? undefined : reason, + bound: b !== undefined && Number.isFinite(b) ? b : undefined, + }; +} diff --git a/ui/src/csvMeta.ts b/ui/src/csvMeta.ts new file mode 100644 index 0000000..08a7495 --- /dev/null +++ b/ui/src/csvMeta.ts @@ -0,0 +1,82 @@ +import type { CsvMeta } from "@platforma-open/milaboratories.feature-integration.model"; +import { parse } from "csv-parse/browser/esm/sync"; + +/** + * Reads the tag->feature CSV's headers, each header's distinct values, and its row count. + * + * This is the block's ONLY panel parser. Until 2026-08 the same job ran in the workflow, as the emit-csv-meta + * Python entrypoint, and the dropdowns waited for an upload and a staging exec to fill them. Reading the file + * here fills them on the pick instead. Nothing downstream re-derives this, so the semantics below are the + * block's definition of what a panel column and a panel row ARE, not an approximation of some other parser. + * + * Parsing is delegated to `csv-parse`, as in blocks/xsv-import: RFC 4180 quoting, doubled quotes, commas and + * newlines inside quoted fields, and both LF and CRLF endings. Real panel files use CRLF, so that last one is + * load-bearing. The shape of this function follows readFileForImport in blocks/samples-and-data: bytes in, a + * value out, and a throw where the file has no header to read. + */ +export function parseTagCsvMeta(bytes: Uint8Array): CsvMeta { + // Decoding is done here rather than left to csv-parse so the block owns the one decision that a BOM forces. + // TextDecoder strips a UTF-8 BOM, which is what an Excel-exported panel needs: left in place it becomes + // part of the first header's name, and every later match against that name fails. + const text = new TextDecoder("utf-8").decode(bytes); + + // relax_column_count: a panel whose rows are shorter or longer than its header is readable, since the value + // loop below simply finds nothing at the missing indices. Refusing the file would be worse than reading the + // columns that ARE there. + const records: string[][] = parse(text, { + relax_column_count: true, + skip_empty_lines: true, + }); + + if (records.length === 0) + throw new Error("The panel CSV is empty: it has no header row and no data rows."); + + // Blank header cells are dropped, so a trailing comma on the header line does not become a nameless column + // in three dropdowns. The INDEX is kept from the original header, not from the compacted list: dropping + // cell 1 of `Barcode,,Name` must not make `Name` look like column 1 when its values are at 2. + // + // A repeated header keeps both entries in `columns` and resolves to its LAST index for values. Both halves + // are deliberate: the dropdowns show the file's headers as the file has them, and a later column silently + // shadowing an earlier one of the same name is the same rule a spreadsheet applies. + const columns: string[] = []; + const indexByColumn = new Map(); + records[0].forEach((cell, index) => { + const name = cell.trim(); + if (name === "") return; + columns.push(name); + indexByColumn.set(name, index); + }); + + if (columns.length === 0) + throw new Error( + "The panel CSV has no usable header: its first row is empty or every heading is blank.", + ); + + const distinct = new Map>(); + for (const name of indexByColumn.keys()) distinct.set(name, new Set()); + + // A row whose every cell is blank is not a row. Panels exported from a spreadsheet routinely carry a few of + // these at the end, and counting them would make rowCount disagree with the number of barcodes declared -- + // the comparison the duplicate-mapping gate is built on. + let rowCount = 0; + for (let r = 1; r < records.length; r++) { + const row = records[r]; + if (row.every((cell) => cell.trim() === "")) continue; + rowCount++; + for (const [name, index] of indexByColumn) { + if (index >= row.length) continue; + const value = row[index].trim(); + if (value === "") continue; + const seen = distinct.get(name); + if (seen === undefined) continue; + seen.add(value); + } + } + + // Sorted so the dropdowns are stable: the same panel read twice must offer its values in the same order, + // whatever order the rows happened to be in. + const valuesByColumn: Record = {}; + for (const [name, seen] of distinct) valuesByColumn[name] = [...seen].sort(); + + return { columns, valuesByColumn, rowCount }; +} diff --git a/ui/src/csvSource.ts b/ui/src/csvSource.ts new file mode 100644 index 0000000..54de23a --- /dev/null +++ b/ui/src/csvSource.ts @@ -0,0 +1,57 @@ +import type { CsvMeta } from "@platforma-open/milaboratories.feature-integration.model"; +import type { + ImportFileHandle, + LocalBlobHandleAndSize, + LocalImportFileHandle, +} from "@platforma-sdk/model"; +import { getRawPlatformaInstance, isImportFileHandleUpload } from "@platforma-sdk/model"; +import { ReactiveFileContent } from "@platforma-sdk/ui-vue"; +import type { ComputedRef } from "vue"; +import { computed } from "vue"; +import { parseTagCsvMeta } from "./csvMeta"; + +/** + * The panel CSV's metadata, read straight off the user's disk. + * + * Returns undefined for a REMOTE pick, which is not a failure: an `index://` handle names a file in remote + * storage that this machine cannot open, so those picks are served by the blob path below instead. Any real + * failure -- the file vanished between the pick and the read, the bytes are not a readable CSV -- throws, and + * the caller shows it. + * + * Same shape as blocks/immune-assay-data (setFile) and blocks/synthetic-repertoire-profiler: guard on + * isImportFileHandleUpload, then read through the ls driver. + */ +export async function readLocalCsvMeta(handle: ImportFileHandle): Promise { + if (!isImportFileHandleUpload(handle)) return undefined; + + // The cast is unavoidable and is the one assumption this module makes. isImportFileHandleUpload proves the + // handle is an `upload://` one, but LocalImportFileHandle is a SEPARATE brand meaning "openable on this + // machine, in this session", and no SDK predicate tests for it. What makes the cast sound is the caller: + // this runs synchronously from the file-picker gesture, so the handle came from the dialog this session + // just opened. Never call this with a handle read back out of `data` -- a project reopened on another + // machine carries handles whose files are not here. + const localHandle = handle as LocalImportFileHandle; + const bytes = await getRawPlatformaInstance().lsDriver.getLocalFileContent(localHandle); + return parseTagCsvMeta(bytes); +} + +/** + * The bytes of the panel CSV as the prerun imported it, for picks this machine cannot read from disk. + * + * The prerun already exports the uploaded CSV -- it has to, so that staging demands the blob and the upload + * starts before production needs it -- so this costs the workflow nothing. It serves a remote pick, and it + * also serves a project opened somewhere the original file never existed. + * + * Mirrors the assayFileBytes computed in blocks/immune-assay-data, which feeds the same parser from the same + * kind of handle. Bytes rather than text because parseTagCsvMeta owns the decoding. + */ +export function useRemoteCsvBytes( + getHandle: () => LocalBlobHandleAndSize | undefined, +): ComputedRef { + const fileContent = ReactiveFileContent.useGlobal(); + return computed(() => { + const handle = getHandle(); + if (handle === undefined) return undefined; + return fileContent.getContentBytes(handle.handle).value; + }); +} diff --git a/ui/src/pages/AntigenQcPage.vue b/ui/src/pages/AntigenQcPage.vue new file mode 100644 index 0000000..09e8716 --- /dev/null +++ b/ui/src/pages/AntigenQcPage.vue @@ -0,0 +1,526 @@ + + + + + diff --git a/ui/src/pages/MainPage.vue b/ui/src/pages/MainPage.vue index ea5dd00..26b8f3f 100644 --- a/ui/src/pages/MainPage.vue +++ b/ui/src/pages/MainPage.vue @@ -8,6 +8,7 @@ import { PlAgOverlayLoading, PlAgOverlayNoRows, PlAlert, + PlCheckbox, PlBlockPage, PlBtnGhost, PlBtnGroup, @@ -25,6 +26,18 @@ import { createAgGridColDef, makeRowNumberColDef, } from "@platforma-sdk/ui-vue"; +import type { + GroupingRule, + ReferenceSource, +} from "@platforma-open/milaboratories.feature-integration.model"; +import { + AGGREGATE_DETECTION_DEFAULTS, + groupingColumns, + QC_LINE_DEFAULTS, +} from "@platforma-open/milaboratories.feature-integration.model"; +import type { ImportFileHandle } from "@platforma-sdk/model"; +import { parseTagCsvMeta } from "../csvMeta"; +import { readLocalCsvMeta, useRemoteCsvBytes } from "../csvSource"; import type { ColDef, GridReadyEvent } from "ag-grid-enterprise"; import { ClientSideRowModelModule, ModuleRegistry } from "ag-grid-enterprise"; import { AgGridVue } from "ag-grid-vue3"; @@ -41,10 +54,41 @@ import { } from "../results"; const app = useApp(); -// Auto-open Settings for a fresh block (no FASTQ chosen yet); stay closed once configured. + +// A quality line and an aggregate-detection knob are both stored undefined until someone types one, and +// the workflow then substitutes the shipped default. The fields below bind through `setting` so an +// untouched one shows that substituted number instead of an empty box. Clearing a field writes undefined +// back and the display returns to the default. +// +// The value shown is never written to `data` on its own, so opening Settings stales nothing. +const SETTING_DEFAULTS = { ...QC_LINE_DEFAULTS, ...AGGREGATE_DETECTION_DEFAULTS }; +type SettingKey = keyof typeof SETTING_DEFAULTS; + +function setting(key: SettingKey) { + return computed({ + get: () => app.model.data[key] ?? SETTING_DEFAULTS[key], + set: (value) => { + app.model.data[key] = value; + }, + }); +} + +const cellBarcodeValidWarn = setting("cellBarcodeValidWarn"); +const cellBarcodeValidError = setting("cellBarcodeValidError"); +const readsPerCellWarn = setting("readsPerCellWarn"); +const aggregateBarcodeWarn = setting("aggregateBarcodeWarn"); +const aggregateBarcodeError = setting("aggregateBarcodeError"); +const undeclaredBarcodeWarn = setting("undeclaredBarcodeWarn"); +const undeclaredBarcodeError = setting("undeclaredBarcodeError"); +const usableReadWarn = setting("usableReadWarn"); +const usableReadError = setting("usableReadError"); +const aggregateBarcodeIqrMultiplier = setting("aggregateBarcodeIqrMultiplier"); +const aggregateBarcodeMinUmiThreshold = setting("aggregateBarcodeMinUmiThreshold"); +const aggregateBarcodeTopN = setting("aggregateBarcodeTopN"); +// Auto-open Settings for a fresh block, with no FASTQ chosen yet. Stays closed once configured. const settingsOpen = ref(app.model.data.fbFastqRef === undefined); -// Close the Settings drawer once a run starts. Watching an output → writing a local ref is not a -// hairpin (no write to server-stored data). +// Close the Settings drawer once a run starts. Watching an output and writing a local ref is not a hairpin, +// since nothing writes to server-stored data. watch( () => app.model.outputs.isRunning, (running) => { @@ -53,18 +97,39 @@ watch( ); // The block's "Analysis logs": a live completed-sample heartbeat while the run is in progress, then a -// run-level summary when it finishes (the model builds the lines from the per-sample QC). Shown in a -// wide slide-over as one text area; detailed per-sample statistics live on the QC page. +// run-level summary once it finishes. The model builds the lines from the per-sample QC. Shown in a wide +// slide-over as one text area. Detailed per-sample statistics live on the QC page. const analysisLog = computed(() => app.model.outputs.analysisLog ?? []); -// First line of the Analysis-logs drawer: point users at the richer per-sample logs, which live behind a -// double-click on each sample row (the run-level analysisLog below is only a summary heartbeat). + +// One line for the whole run, above the per-sample grid. The grid has carried live per-step progress +// all along, but a reader watching a single sample had to read it out of a cell, and a run that had +// not produced its roster yet showed nothing at all while the parser was already most of the way +// through. `analysisLog` says only "Processing..." in that window. +// +// The percent is the MEAN over samples rather than the minimum. A minimum reports the slowest sample, +// which is a different question and makes a run of twenty look stalled whenever one lags. +const runProgress = computed(() => { + const rows = sampleResults.value ?? []; + if (rows.length === 0) return undefined; + const done = rows.filter((r) => r.progress.status === "done").length; + if (done === rows.length) return undefined; + const pct = (r: SampleResult) => r.progress.percent ?? 0; + const percent = Math.round(rows.reduce((sum, r) => sum + pct(r), 0) / rows.length); + // The label of whichever sample is furthest along: it names the step the run is actually in, where + // the slowest sample's label lags a step behind and reads as if nothing had started. + const leader = rows.reduce((a, b) => (pct(b) > pct(a) ? b : a)); + const detail = [leader.progress.text, leader.progress.suffix].filter(Boolean).join(" "); + return { percent, detail, done, total: rows.length }; +}); +// First line of the Analysis-logs drawer, pointing the user at the richer per-sample logs behind a +// double-click on each sample row. The run-level analysisLog below is only a summary heartbeat. const LOGS_HINT = "Tip: double-click any sample in the progress table to open its own detailed per-step logs (parse, refine tags, count UMIs)."; const logText = computed(() => [LOGS_HINT, "", ...analysisLog.value].join("\n")); const logsOpen = ref(false); -// Per-sample report slide-over (live per-step mitool logs). Opened by double-clicking a grid row; the -// modal is shown whenever a sample is selected. +// Per-sample report slide-over of live per-step mitool logs. Opened by double-clicking a grid row, and +// shown whenever a sample is selected. const selectedSample = ref(undefined); const sampleReportOpen = computed({ get: () => selectedSample.value !== undefined, @@ -78,41 +143,245 @@ const selectedSampleLabel = computed(() => : "", ); -// No-negative-control info note in the Settings drawer: appears once the tag-feature CSV is added, -// and hides as soon as a negative control feature is selected. -const controlInfoVisible = computed( - () => !!app.model.data.tagFeatureCsvHandle && !app.model.data.controlFeature, -); - -// True while staging is still parsing the uploaded tag-feature CSV (handle set, but the column/value -// metadata hasn't resolved yet). Drives a "reading columns…" note and disables the CSV-derived -// dropdowns, so their empty state reads as "loading" rather than "no columns found". +// True while the panel CSV has been picked but not yet read: the handle is set and csvMetaSnapshot is not. +// For a local pick that window is a single tick, so the note below never appears. For a remote pick it lasts +// until the upload lands and the blob watcher parses it. Drives a "reading columns..." note and disables the +// CSV-derived dropdowns, so their empty state reads as "loading" rather than "no columns". const csvProcessing = computed(() => app.model.outputs.csvColumnsLoading === true); -// The CSV-derived tag-mapping dropdowns (barcode / feature / control / sample columns) have nothing to -// offer until a tag-feature CSV is uploaded AND its columns are parsed. Disable + dim them when no CSV -// handle exists yet, or while staging is still reading its columns — so their empty state reads as -// "waiting for a CSV" rather than "no columns found". Reuses the SDK disabled/dimmed affordance already -// used for the parse window (csvProcessing). +// The CSV-derived tag-mapping dropdowns (barcode / feature / control / sample columns) have nothing to offer +// until a tag-feature CSV is picked AND its columns are read. Disable and dim them while no CSV handle +// exists, or while the panel has not been read yet, so their empty state reads as "waiting for a CSV" rather +// than "no columns found". Reuses the SDK disabled and dimmed affordance the parse window (csvProcessing) +// already uses. const tagMappingDisabled = computed( () => !app.model.data.tagFeatureCsvHandle || csvProcessing.value, ); -// Combine-mode column options exclude the columns already bound to the barcode/feature roles: those hold -// DNA barcodes / feature names, not per-feature modes ("sum"/"all"), so offering them only invites a -// mis-pick. The model's args() also rejects such a collision (belt-and-suspenders), but filtering the -// dropdown prevents the mistake up front. -const combineColumnOptions = computed(() => +// CSV columns not already bound to the barcode-sequence or feature-name roles. A column holding DNA barcodes +// or antigen names is not a sample column, and offering it invites a mis-pick. The data layer refuses two +// roles on one column, but only at the end of the run. The model's args() rejects the collision too, and +// filtering here prevents the mistake up front. +const roleFreeColumnOptions = computed(() => (app.model.outputs.csvColumnOptions ?? []).filter( (o) => o.value !== app.model.data.barcodeSeqColumn && o.value !== app.model.data.featureNameColumn, ), ); -// Visible reason when the Combine-mode column is invalid, so a disabled Run button is explained rather -// than mysterious. The model's args() is the authoritative gate (it throws and greys out Run); this -// mirrors the same condition into an inline alert the user actually sees. Fires when the chosen column -// collides with the barcode/feature roles — e.g. a value left stale after changing the feature column. +// The panel's headers as they stand now. Snapshotted into data on the gesture that names a panel column, so +// args() can refuse a column the panel does not carry without reaching outside data. Left to a watcher this +// would be an output written back into data, which two open clients would race to write. +function snapshotPanelColumns() { + app.model.data.panelColumnSnapshot = (app.model.outputs.csvColumnOptions ?? []).map( + (o) => o.value, + ); +} + +// ---- the binding reading ---------------------------------------------------------------------- +// The settings for the binding reading. Rendered in the Main page's Settings drawer, and there ONLY. A block +// puts its settings in one place, and the Explore readout offered this same drawer until that second copy was +// removed. Do not mount it from a results page again: two drawers editing one set of controls is not the +// idiom, whatever it costs to recompute. +// +// Everything this component EDITS is below the "binding reading" line in BlockArgs, so a change here recovers +// every per-sample mitool body from cache and re-runs the verdict stage alone. It also READS three fields it +// must never edit -- tagFeatureCsvHandle, barcodeSeqColumn, sampleColumn -- to tell whether the panel has +// loaded, and to keep a role or grouping setting from naming a column the panel reader consumes as a key. +// Those three force the whole per-sample fan-out to re-run, and the Main page owns their controls. +// +// VOCABULARY, and the split is deliberate. Everything a USER reads says "baseline": the level a count must +// exceed, measured in the same cell from a tag declared to bind nothing. The DATA layer keeps `reference` -- +// `ReferenceSource`, the run-meta keys, the p-column domain values -- and those cannot follow, because domain +// is part of column identity and renaming one would change what every emitted column IS. Code comments here +// describe the data layer, so they still say reference and comparator. +// +// One user-facing word, never four. "control" is not it. The glossary keeps control and reference apart -- +// being a control is a property of the tag, and a panel may carry several, where being the reference that +// supplies the baseline is a job given to exactly one of them. This form nominates the reference, so it says +// baseline throughout and never "control". + +// The distinct values of the chosen role column -- what the comparator is designated by. +const roleValueOptions = computed(() => { + const column = app.model.data.roleColumn; + if (!column) return []; + return (app.model.outputs.csvValuesByColumn?.[column] ?? []).map((v) => ({ + value: v, + label: v, + })); +}); + +// Changing the role column drops the values chosen under the old one. They designate values of THIS column, +// and left behind they would mark no tag while still reading as a configured comparator. +// +// Declaring a baseline is also what the baseline CHOICE was made against, so changing the declaration drops +// the choice too. An override means "I want this rung given what I have declared", not a standing instruction +// that outlives the declaration it answered. Left behind, marking a baseline tag could not move the field +// onto it, which reads as the block ignoring what you just declared. +// +// The same rule the two settings either side of it follow: a setting does not outlive the thing it was chosen +// against. Only on a GESTURE, never from a watcher -- a watcher on a model output writing back into data is +// the hairpin, and two clients with the project open would race on it. +function clearBaselineChoice() { + app.model.data.referenceSource = undefined; +} + +function setRoleColumn(column: string | undefined) { + app.model.data.roleColumn = column || undefined; + app.model.data.referenceValues = undefined; + clearBaselineChoice(); + snapshotPanelColumns(); +} + +// ONE value, stored as a one-element list. `040-glossary` splits the two cardinalities: being a control is +// a property of the tag and a panel may carry several, but the reference is one job given to one of them. +// So the value that marks the baseline is singular. Several values here only ever described a panel whose +// role column spells one role more than one way, and a panel that does that is asking to be corrected +// rather than accommodated. +// +// `data.referenceValues` stays a LIST. The field name, the `--reference-values` flag and every stored +// project keep their shape, so this tightens the control without a migration. The `> 1 tag` refusal in +// `verdict.py` stays too, and is now the only thing that can fire: one value can still mark several tags, +// which is a panel fact this control cannot see. +function setReferenceValue(value: string | undefined) { + app.model.data.referenceValues = value ? [value] : undefined; + clearBaselineChoice(); +} + +// The comparator sources this panel can serve. Both the option list and the reasons come from a model output +// rather than from a watcher: the facts behind them are the panel's, and copying them into data would make +// the output depend on the data it feeds. +const referenceSources = computed(() => app.model.outputs.referenceSources); + +// The rungs this panel can serve, and the rung the run will be answered under. Both come from model outputs: +// the option list from `referenceSources`, the shown value from `effectiveReferenceSource`. +// +// NEVER derive the shown value here. Writing the rule twice -- once to decide what to display, once in +// `args()` to decide what to send -- makes the field lie the moment a stored choice stops being serviceable. +// Clearing the role values then leaves a dead "declared" behind: this component re-renders as "the panel's +// own readings" while the data still holds "declared", so the form shows a scientist the exact value they are +// being asked to supply while Run stays greyed out, and only re-picking the already-shown value fixes it. +// +// No derivation exists anywhere. The block does not choose a baseline, because a baseline nobody chose is a +// methodology nobody knows they used. `effectiveReferenceSource` is the stored choice, or the bottom rung +// where none was made. Reading an output to display it is not a hairpin: nothing here writes back. +// EVERY rung, always, and every one selectable. The scientist picks the rung first and the form then asks +// for what that rung needs. Offering only the rungs already satisfied made the declared rung unreachable: +// its requirements are the two fields that appear once it is chosen, so it could never become serviceable +// while it was hidden. +const allSources = computed(() => referenceSources.value?.options ?? []); + +// The agreement limit as a PERCENTAGE, where the data holds a share from 0 to 1. A share reads as a +// number a scientist has to translate, and this one has a floor most readers do not expect: agreement is +// the share of voting cells holding the state the verdict took, so a majority can never fall below a +// half. The useful range is 51 to 100, and "0-1" hid that. +// +// The data keeps the share. `--min-agreement` and every stored project keep their shape, so this is a +// display conversion and not a migration. Rounded on the way in, because a percentage entered as an +// integer must come back as the same integer. +const agreementPercent = computed({ + get: () => { + const share = app.model.data.minAgreement; + // NOT rounded. The field accepts a typed fraction of a percent -- `step` drives only the arrow + // buttons, and `commitValue` in PlNumberField assigns whatever was typed. Rounding here would show + // 51 for a stored 50.5, and 50 for a stored 50.001, so the number on screen would stop being the + // number in force. + return typeof share === "number" ? share * 100 : undefined; + }, + set: (percent: number | undefined) => { + app.model.data.minAgreement = typeof percent === "number" ? percent / 100 : undefined; + }, +}); + +// The chosen rung, from DATA. Each baseline brings its own rule, so a setting belonging to one rule is +// shown only where that rule runs. Read from data and never from `effectiveReferenceSource`: the form +// reveals fields against what the scientist picked, and that must not move on its own. +const chosenSource = computed(() => app.model.data.referenceSource); + +// What the chosen rung still needs, if anything. The model computes it, because whether a rung can serve +// turns on panel facts this component must not re-derive. +const chosenNeeds = computed( + () => allSources.value.find((o) => o.value === chosenSource.value)?.needs, +); +const shownSource = computed(() => app.model.outputs.effectiveReferenceSource); +// Whether the scientist has actually chosen. Read from data rather than from the output above, which answers +// "none" both for an explicit choice of no baseline and for no choice at all. The two look the same to a run +// and are opposite things to say to a reader. +const baselineUnchosen = computed(() => app.model.data.referenceSource === undefined); + +function setBaselineSource(value: string | undefined) { + app.model.data.referenceSource = value === undefined ? undefined : (value as ReferenceSource); +} + +// The identities the contending-groups editor picks from, live from the uploaded panel. +const identityOptions = computed(() => app.model.outputs.identityOptions ?? []); +// The panel-derived dropdowns have nothing to offer until the panel file is uploaded and staging has read +// its columns. Disabled and dimmed, so their empty state reads as "waiting" rather than "nothing found". +const panelUnread = computed( + () => !app.model.data.tagFeatureCsvHandle || app.model.outputs.csvColumnsLoading === true, +); + +// The panel's PROPERTY columns: every header except the ones the panel reader consumes as keys, which are +// the barcode column and the sample column where one is set. Mirrors panel.py's own rule. A column the +// reader strips is not one the grouping setting may name, and emit_verdicts.py ends the run rather than +// degrading when handed one. +const panelPropertyOptions = computed(() => + (app.model.outputs.csvColumnOptions ?? []).filter( + (o) => o.value !== app.model.data.barcodeSeqColumn && o.value !== app.model.data.sampleColumn, + ), +); + +// One control for the whole rule, taking SEVERAL columns: an identity is the distinct combination of the +// named columns' values, so naming antigen and concentration together makes the same antigen at two +// concentrations two identities. +// +// The barcode column sits in the same list as the property columns, because naming it IS a grouping -- the +// finest one available, one identity per barcode -- rather than a mode beside grouping. It cannot be offered +// as a property column, since the panel reader consumes it as the `tag` key, so it maps to the `tag` rule, +// which produces exactly that reading. A sentinel value stands for it, prefixed with a space so no real +// column name can collide. +const TAG_GROUPING_VALUE = " tag"; + +const groupingSelection = computed(() => { + const rule = app.model.data.grouping; + if (rule === undefined) return []; + if (rule.by === "tag") return [TAG_GROUPING_VALUE]; + return groupingColumns(rule); +}); + +const groupingOptions = computed(() => [ + { + value: TAG_GROUPING_VALUE, + // Sits in the same list as the property columns on purpose: naming the barcode IS a grouping, the + // finest one available, rather than a mode beside grouping. It is labelled so it cannot be mistaken + // for one of the panel's own columns, which is what naming it after the barcode column did. + label: "Each barcode on its own — one identity per barcode", + }, + ...panelPropertyOptions.value, +]); + +function setGrouping(selected: string[] | undefined) { + const picked = (selected ?? []).filter((c) => c !== ""); + // The barcode column is the finest grouping there is, so it does not combine with a coarser one: a + // combination including it is already one identity per barcode. Picking it therefore wins alone, and + // picking nothing leaves the rule absent, which reads the same way. + const rule: GroupingRule | undefined = picked.includes(TAG_GROUPING_VALUE) + ? { by: "tag" } + : picked.length > 0 + ? { by: "property", columns: picked } + : undefined; + app.model.data.grouping = rule; + // The identities ARE the values of the grouping columns, so groups declared under the previous rule name + // things that no longer exist. Cleared on the gesture that invalidates them rather than left to fail. + app.model.data.contendingGroups = undefined; + snapshotPanelColumns(); +} + +// Visible reason when the Combine-mode column is invalid, so a disabled Run button is explained rather than +// mysterious. The model's args() is the authoritative gate, throwing and greying out Run, and this mirrors +// the same condition into an inline alert. The selector is not offered today, but a project saved while it +// was, or migrated, can still carry a value that collides with the barcode or feature roles, and without +// this the Run button would simply be grey. const combineColumnError = computed(() => { const c = app.model.data.combineColumn; if (!c) return undefined; @@ -126,40 +395,15 @@ const combineColumnError = computed(() => { return undefined; }); -// Off-target designation (F2). The property dropdown offers imported per-feature property columns — -// csvColumnOptions minus the columns already bound to the barcode/feature/sample roles (those aren't -// per-feature properties). The values multi-select lists the chosen property's distinct values (from -// csvValuesByColumn). Features whose property value is selected are excluded from the dominant call -// (like the control) and enable the "cross-reactive" label. -const offtargetPropertyOptions = computed(() => - (app.model.outputs.csvColumnOptions ?? []).filter( - (o) => - o.value !== app.model.data.barcodeSeqColumn && - o.value !== app.model.data.featureNameColumn && - o.value !== app.model.data.sampleColumn, - ), -); -const offtargetValueOptions = computed(() => { - const prop = app.model.data.offtargetProperty; - if (!prop) return []; - return (app.model.outputs.csvValuesByColumn?.[prop] ?? []).map((v) => ({ value: v, label: v })); -}); -// Changing the property invalidates the selected values (they belong to the previous column), so clear -// them on that explicit gesture — same data→data pattern as clearControlOnInputChange. -function setOfftargetProperty(prop: string | undefined) { - app.model.data.offtargetProperty = prop; - app.model.data.offtargetValues = undefined; -} - -// Run mode: read-limited Preview (dry run) vs full run — same PlBtnGroup pattern as mixcr-clonotyping / -// demultiplex-fastq (Preview first). Feature-barcode is single-cell + shallow per cell, so the dry-run -// default matches mixcr's single-cell recommendation (500k reads/sample). +// Run mode: read-limited Preview (dry run) against a full run. The same PlBtnGroup pattern +// mixcr-clonotyping and demultiplex-fastq use, Preview first. Feature-barcode is single-cell and shallow per +// cell, so the dry-run default matches mixcr's single-cell recommendation of 500k reads per sample. const runModeOptions = [ { label: "Preview", value: "dry" as const }, { label: "Full run", value: "full" as const }, ]; const DRY_RUN_READS_DEFAULT = 500_000; -// Auto-fill the read limit when the user switches to Preview and hasn't set one (mirrors mixcr). +// Auto-fill the read limit where the user switches to Preview and has set none. Mirrors mixcr. watch( () => app.model.data.runMode, (mode) => { @@ -168,64 +412,213 @@ watch( }, ); -// A negative control is one of the feature-name column's values, so changing the CSV or the feature-name -// column can make the current selection reference a feature that no longer exists. Clear it on that user -// gesture. This is a data→data write on an explicit gesture — NOT a watcher on the controlOptions output -// (that would be the spec-facts-resync hairpin; see hairpin.md). Left stale, args() would still send it -// and the workflow would silently score specificity against a zero control (inflated scores, no error). -// If the control is still valid after the change the user re-picks — cheaper than snapshotting the valid -// set into data to validate in args(). -function clearControlOnInputChange() { - app.model.data.controlFeature = undefined; +// Empty stores as undefined so the no-control note below reads one condition rather than two. + +// A GESTURE IS NOT A CHANGE. Every clear below must compare the new value against the old one: a control +// re-emitting the value it already held -- a user re-picking the dataset they had picked, or a re-render +// after the block pack was updated -- otherwise discards configuration nobody touched. +// +// Concretely: re-emitting an UNCHANGED FASTQ ref wipes `sampleColumn`, the run reaches per_cell_metrics.py +// with no `--sample-col`, and its duplicate-barcode guard refuses a sample-keyed panel. The user meets that +// as a QuickJS stack trace minutes after a gesture that changed nothing. `clearOnCsvChange` is the same shape +// over nine more fields, the whole binding reading included. +// +// The previous value has to be remembered HERE: `v-model` writes the new one into data before the handler +// runs, so data holds the "after" on both sides of any comparison made inside it. Keyed by JSON, so a ref +// object and a file handle compare the same way. +const keyOf = (v: unknown) => (v === undefined || v === null ? "" : JSON.stringify(v)); +const seenFastqRef = ref(keyOf(app.model.data.fbFastqRef)); +const seenCsvHandle = ref(keyOf(app.model.data.tagFeatureCsvHandle)); +const seenFeatureColumn = ref(keyOf(app.model.data.featureNameColumn)); + +// Returns true only when the gesture carried a genuinely new value, and records it. +function changed(seen: { value: string }, next: unknown): boolean { + const key = keyOf(next); + if (key === seen.value) return false; + seen.value = key; + return true; } -// Sample-aware mapping (optional). Picking the sample column snapshots the CURRENT dataset's -// sampleId→name map into data, so the args projection stays pure (model.md) and the per-sample workflow -// body can translate its iteration key. +// Sample-aware mapping (optional). Picking the sample column snapshots the CURRENT dataset's sampleId->name +// map into data, so the args projection stays pure (model.md) and the per-sample workflow body can translate +// its iteration key. function setSampleColumn(col: string | undefined) { app.model.data.sampleColumn = col || undefined; - // Snapshot both the dataset's sampleId→name map AND the chosen column's CSV values, so args() can both - // filter per sample and gate Run (block when a dataset sample has no CSV rows) purely from data. + // Snapshot both the dataset's sampleId->name map AND the chosen column's CSV values, so args() can both + // filter per sample and gate Run -- blocking when a dataset sample has no CSV rows -- purely from data. app.model.data.sampleLabelSnapshot = col ? app.model.outputs.sampleLabels : undefined; app.model.data.sampleColumnValues = col ? (app.model.outputs.csvValuesByColumn?.[col] ?? []) : undefined; + // Clearing the sample column is what makes a duplicate barcode illegal again, so the numbers args() gates + // on are refreshed here rather than assumed present from an earlier gesture. + snapshotPanelCounts(); + clearVerdictSettingsNaming(col || undefined); } -// The snapshot goes stale if the dataset changes (different sampleId→name) or the CSV changes (different -// columns/values), so clear the sample-aware selection on those gestures — the user re-picks. -function clearSampleAwareOnInputChange() { +// The snapshot goes stale where the dataset changes, giving a different sampleId->name, or the CSV changes, +// giving different columns and values. Clear the sample-aware selection on those gestures and let the user +// re-pick. Split from its gesture handler because `clearOnCsvChange` calls it too, and THAT path must clear +// unconditionally: a new panel file invalidates the sample selection whatever the FASTQ ref does. +function clearSampleAwareState() { app.model.data.sampleColumn = undefined; app.model.data.sampleLabelSnapshot = undefined; app.model.data.sampleColumnValues = undefined; } -// CSV swap invalidates every CSV-derived selection: the barcode / feature-name columns (the new file's -// headers differ), the negative control, the off-target designation, and the sample-aware selection -// (columns/values change). Clear them all so the user re-picks against the new CSV. +function onFastqRefChanged(next: unknown) { + if (!changed(seenFastqRef, next)) return; + clearSampleAwareState(); +} + +// Picking the barcode column is what makes a duplicate mapping knowable, so it is where the two numbers +// args() needs are snapshotted. args() is data-only and the CSV meta lives on ctx.prerun, so without this the +// model can see the problem and still not refuse the run. +const seenBarcodeColumn = ref(keyOf(app.model.data.barcodeSeqColumn)); + +// Called from every gesture that can make a duplicate mapping RELEVANT, not only from the one that makes it +// knowable. Taken on the barcode-column pick alone, the gate is inert in the case that actually happens: the +// barcode column was picked long ago, and what changes now is the SAMPLE column being cleared, which turns a +// legal sample-keyed panel into an illegal duplicate one. Idempotent, so calling it from three places costs +// nothing. +function snapshotPanelCounts() { + const col = app.model.data.barcodeSeqColumn; + app.model.data.panelRowCount = col ? app.model.outputs.csvRowCount : undefined; + app.model.data.panelBarcodeDistinct = col + ? (app.model.outputs.csvValuesByColumn?.[col]?.length ?? undefined) + : undefined; +} + +// Claiming a column as a key invalidates any verdict setting that names it. The panel reader strips the +// barcode and sample columns before the properties are read, so the setting would name a column that is no +// longer a property. args() refuses the run in that state, which is a blocked Run button rather than a dead +// run, but the user still has to find the stale pick in a dropdown that stopped offering it. Clearing it on +// the gesture that invalidates it is the treatment clearOnCsvChange gives the panel swap. This is the +// reassignment case, reaching the same stale pick by a different gesture. +function clearVerdictSettingsNaming(column: string | undefined) { + if (!column) return; + if (app.model.data.roleColumn === column) { + // The values designate values of THIS column, so they go with it -- the same pairing setRoleColumn keeps. + app.model.data.roleColumn = undefined; + app.model.data.referenceValues = undefined; + } + const remaining = groupingColumns(app.model.data.grouping).filter((c) => c !== column); + if (remaining.length !== groupingColumns(app.model.data.grouping).length) { + // A grouping may name several columns, so losing one leaves the others standing. Losing the last leaves no + // rule, which reads as one identity per tag: the same state as never having set it. + app.model.data.grouping = + remaining.length > 0 ? { by: "property", columns: remaining } : undefined; + // The identities ARE the values of the grouping columns, so declared groups now name things that do not + // exist. Cleared here for the same reason setGrouping clears them. + app.model.data.contendingGroups = undefined; + } +} + +function onBarcodeColumnChanged(next: unknown) { + if (!changed(seenBarcodeColumn, next)) return; + snapshotPanelCounts(); + clearVerdictSettingsNaming(app.model.data.barcodeSeqColumn); +} + +// CSV swap invalidates every CSV-derived selection: the barcode / feature-name columns, since the new file's +// headers differ, the negative control, the sample-aware selection, and every setting of the binding reading +// that names a panel column or a panel value. The last group matters most: emit_verdicts.py ends the whole +// run when the role column or the grouping column is not one the panel carries, so a stale pick left behind +// here costs a run and reports it where the user never looks. +function onCsvChanged(next: unknown) { + if (!changed(seenCsvHandle, next)) return; + // The feature-name column is about to be cleared, so its own guard must not later read a stale key and + // decide the user's re-pick was a no-op. + seenFeatureColumn.value = ""; + clearOnCsvChange(); + // Read the panel NOW, from the file the user just chose, rather than waiting for the upload to land and a + // workflow step to describe it. `next` is the parse target, not data: v-model has already written it, but + // reading the argument makes the handler independent of listener order. + void readPanelFrom(next as ImportFileHandle | undefined); +} + +// Fills csvMetaSnapshot from the picked file. Local picks are read from disk, which is what makes the column +// dropdowns fill on the gesture. A remote pick reads nothing here and is served by the blob watcher below +// once the upload lands. +// +// The handle re-check before the write is the rapid-re-pick guard: the read is async, so a user who swaps +// files twice in quick succession can have the FIRST read resolve last. Publishing it would leave the +// dropdowns describing a file that is no longer chosen. +async function readPanelFrom(handle: ImportFileHandle | undefined) { + if (!handle) return; + try { + const meta = await readLocalCsvMeta(handle); + if (meta === undefined) return; // remote pick — the blob path serves it + if (app.model.data.tagFeatureCsvHandle !== handle) return; + app.model.data.csvMetaSnapshot = { handle, meta }; + } catch (e) { + if (app.model.data.tagFeatureCsvHandle !== handle) return; + app.model.data.csvImportError = e instanceof Error ? e.message : String(e); + } +} + +// The remote-pick path. Nothing on this machine can open an `index://` file, so the panel is read from the +// blob the prerun imported, through the SAME parser the local path uses. One parser, two byte sources. +// +// This is a watcher that writes to data, which hairpin.md tells reviewers to look at twice, and it is the +// same construction blocks/immune-assay-data uses for the same job. It cannot feed itself. The output it +// watches comes from the prerun, and the prerun re-renders only when the prerunArgs PROJECTION changes +// (canonical-JSON compared in pl-middle-layer's setStates, which gates renderStagingFor). That projection is +// tagFeatureCsvHandle alone, so writing csvMetaSnapshot cannot re-run the prerun and cannot change +// csvFileHandle. Adding the snapshot to prerunArgs WOULD close that loop, and because a staging re-render +// resets staging, each turn would throw away the uploaded blob. Leave the projection alone. +// +// Two clients open on one project both run this and both write, which is safe because they cannot disagree: +// the parse is pure and both read the same blob, so the writes are identical. The guard below stops the +// second one anyway. +const remoteCsvBytes = useRemoteCsvBytes(() => app.model.outputs.csvFileHandle); +watch( + remoteCsvBytes, + (bytes) => { + const handle = app.model.data.tagFeatureCsvHandle; + if (!bytes || !handle) return; + if (app.model.data.csvMetaSnapshot?.handle === handle) return; // already read + try { + app.model.data.csvMetaSnapshot = { handle, meta: parseTagCsvMeta(bytes) }; + app.model.data.csvImportError = undefined; + } catch (e) { + app.model.data.csvImportError = e instanceof Error ? e.message : String(e); + } + }, + { immediate: true }, +); + function clearOnCsvChange() { + // The panel metadata describes the OLD file, so it goes first: every field cleared below is derived from + // it, and readCsvMeta stops returning it the moment the handle it is tagged with stops matching. + app.model.data.csvMetaSnapshot = undefined; + app.model.data.csvImportError = undefined; app.model.data.barcodeSeqColumn = undefined; + app.model.data.panelRowCount = undefined; + app.model.data.panelBarcodeDistinct = undefined; + seenBarcodeColumn.value = ""; app.model.data.featureNameColumn = undefined; app.model.data.combineColumn = undefined; - // Off-target property/values name columns + values of the OLD CSV; a new CSV may not have them, and - // args() projects them unconditionally once set (no column-existence guard), so clear them here. - app.model.data.offtargetProperty = undefined; - app.model.data.offtargetValues = undefined; - clearControlOnInputChange(); - clearSampleAwareOnInputChange(); + app.model.data.roleColumn = undefined; + app.model.data.referenceValues = undefined; + app.model.data.grouping = undefined; + app.model.data.contendingGroups = undefined; + app.model.data.panelColumnSnapshot = undefined; + clearSampleAwareState(); } -// Sample-aware mapping sanity warning from the model (dataset samples missing from the CSV / CSV sample -// values matching no dataset sample). Only present once a sample column is chosen. +// Sample-aware mapping sanity warning from the model: dataset samples missing from the CSV, and CSV sample +// values matching no dataset sample. Only present once a sample column is chosen. const sampleMappingWarning = computed(() => app.model.outputs.sampleMappingWarning); -// Sample-aware mapping is auto-selected. When the model spots a CSV column whose distinct values match -// the dataset's sample names (suggestedSampleColumn), pre-populate the Sample column dropdown with it via -// setSampleColumn (which snapshots the sample map into data). Guarded to run only while NO column is set, -// so a manual clear or a manual pick is never overridden. Safe from the reactive-write hairpin the block -// otherwise avoids: suggestedSampleColumn is derived from the CSV meta + sample labels only — it does not -// depend on sampleColumn or the snapshot fields setSampleColumn writes, so applying it can't re-trigger -// the suggestion. Clearing (X) sticks; a CSV/dataset change re-clears (clearOnCsvChange) then re-suggests. +// Sample-aware mapping is auto-selected. Where the model spots a CSV column whose distinct values match the +// dataset's sample names (suggestedSampleColumn), pre-populate the Sample column dropdown with it through +// setSampleColumn, which snapshots the sample map into data. Guarded to run only while NO column is set, so a +// manual clear or a manual pick is never overridden. Safe from the reactive-write hairpin the block otherwise +// avoids: suggestedSampleColumn derives from the CSV meta and sample labels alone, and depends on neither +// sampleColumn nor the snapshot fields setSampleColumn writes, so applying it cannot re-trigger the +// suggestion. Clearing with X sticks. A CSV or dataset change re-clears through clearOnCsvChange, then +// re-suggests. const suggestedSampleColumn = computed(() => app.model.outputs.suggestedSampleColumn); watch( suggestedSampleColumn, @@ -248,8 +641,8 @@ const defaultColumnDef: ColDef = { sortable: false, }; -// The progress grid is now always shown (the results-table view is retired — see template). Before the -// run starts, show the "not-ready" overlay; once it begins, show "running" until the sample roster loads. +// The progress grid is always shown. Before the run starts, show the "not-ready" overlay. Once it begins, +// show "running" until the sample roster loads. const loadingOverlayParams = computed(() => app.model.outputs.started ? { variant: "running" as const, runningText: "Preparing sample list" } @@ -274,14 +667,17 @@ const columnDefs: ColDef[] = [ headerName: "Progress", headerComponentParams: { type: "Progress", - info: "Double-click a sample to open its per-step logs (parse, refine tags, count UMIs).", + info: + "Double-click a sample to open its report: read recovery, the individual quality checks, " + + "and the per-step logs (parse, refine tags, count UMIs).", } satisfies PlAgHeaderComponentParams, flex: 2, - // results.ts already produces the cell config (status / percent / text / suffix); pass it through. + // results.ts already produces the cell config (status / percent / text / suffix). Pass it through. progress: (value) => value, }), - // Quality status tag (OK / WARN / ALERT), worst-case per sample from the QC metrics (results.ts). - // Blank while the sample is still running (quality is undefined until its QC settles). + // Quality status tag (OK / WARN / ALERT): the sample's rolled-up status, taken by the software over its + // own measurements. Blank while the sample is still running, and blank on a finished sample where no + // measurement carried a line to judge it. createAgGridColDef({ colId: "quality", field: "quality", @@ -289,10 +685,11 @@ const columnDefs: ColDef[] = [ headerComponentParams: { type: "Text", info: - "Per-sample QC status.\n" + - "ALERT — no cells detected, or under 25% of reads assigned to the panel.\n" + - "WARN — under 50% of reads panel-assigned, or under 80% matched the read pattern.\n" + - "OK — otherwise.", + "The worst status among this sample's own quality measurements. A measurement carries a status " + + "only where a published or stated line stands behind it; the rest are shown with their value and " + + "no status.\n" + + "Blank means no measurement of this sample carried a line, which is not the same as OK.\n" + + "Double-click the sample to see every measurement, its value, and the reason where it has none.", } satisfies PlAgHeaderComponentParams, width: 120, cellRendererSelector: (params) => @@ -300,7 +697,7 @@ const columnDefs: ColDef[] = [ ? { component: PlAgCellStatusTag, params: { type: params.data.quality } } : undefined, }), - // Read recovery: a compact stacked bar (usable / off-panel / no pattern match). Blank until QC settles. + // Read recovery: a compact stacked bar of usable, off-panel and no-pattern-match. Blank until QC settles. createAgGridColDef({ colId: "recovery", field: "recovery", @@ -349,11 +746,21 @@ const gridOptions = { - + +
+
+ {{ runProgress.detail }} + {{ runProgress.done }} of {{ runProgress.total }} samples done +
+
+
+
+
+
+ + + - + + + + + + + Panel Settings Reading columns from the uploaded CSV… + + Could not read the tag-feature CSV: {{ app.model.data.csvImportError }} + - + + - - - Optional settings - - - - - - - - - + - + + Baseline Parameters + + + + {{ chosenNeeds }} + + + + + - + - + + + + + + + + Threshold Parameters + + + + + + + + + + + + + + + + {{ combineColumnError }} - - Specificity scores will not be computed without a negative control feature -
{{ line }}
 + + + + {{ app.model.outputs.barcodeAlphabetIssue }} + + + {{ app.model.outputs.barcodeMappingIssue }} + + + {{ app.model.outputs.unkeyedSamplePanel }} + - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + These three settings decide which barcodes count as aggregates before the + aggregate-barcode line above ever reads a number. The 0.05 warn line was calibrated + against Cell Ranger's default setting of all three; moving any of them changes what that + line is judging. + + + + + + + + + + + @@ -599,8 +1372,46 @@ const gridOptions = { - + + + diff --git a/ui/src/pages/PunchcardPage.vue b/ui/src/pages/PunchcardPage.vue new file mode 100644 index 0000000..3b6930b --- /dev/null +++ b/ui/src/pages/PunchcardPage.vue @@ -0,0 +1,415 @@ + + + diff --git a/ui/src/pages/QcSummaryPage.vue b/ui/src/pages/QcSummaryPage.vue index 5be201a..7155c86 100644 --- a/ui/src/pages/QcSummaryPage.vue +++ b/ui/src/pages/QcSummaryPage.vue @@ -24,8 +24,8 @@ const qcSummarySettings = usePlDataTableSettingsV2({ show-export-button /> - Per-sample QC metrics (reads parsed/matched, cells, features, UMIs) appear here once the block - has run. + One row per sample, carrying that sample's rolled-up status and every quality measurement, + appears here once a V(D)J dataset is chosen and the block has run. diff --git a/ui/src/pages/ResultsPage.vue b/ui/src/pages/ResultsPage.vue index 5b50782..f4591bd 100644 --- a/ui/src/pages/ResultsPage.vue +++ b/ui/src/pages/ResultsPage.vue @@ -9,9 +9,9 @@ import { useApp } from "../app"; const app = useApp(); -// The collapsed per-cell results table (one row per [sampleId, cellId]): consensus feature + per-cell -// aggregates (Max UMI / fraction / specificity) + the "Feature breakdown" string. This was the block's -// original Main page; Main is now the progress grid, so the table lives on its own tab. The perCellTable model output has stayed in place throughout. +// The collapsed per-cell results table, one row per [sampleId, cellId]: the per-cell aggregates (Max feature +// UMI count, Max feature fraction) and the "Feature breakdown" string. Main is the progress grid, so this +// table lives on its own tab, reading the perCellTable model output. const tableSettings = usePlDataTableSettingsV2({ model: () => app.model.outputs.perCellTable, }); diff --git a/ui/src/pages/SampleReportPanel.vue b/ui/src/pages/SampleReportPanel.vue index ecd9ed0..dc286ee 100644 --- a/ui/src/pages/SampleReportPanel.vue +++ b/ui/src/pages/SampleReportPanel.vue @@ -1,49 +1,42 @@ + diff --git a/ui/src/pages/SampleReportPanelLogs.vue b/ui/src/pages/SampleReportPanelLogs.vue new file mode 100644 index 0000000..9e2168b --- /dev/null +++ b/ui/src/pages/SampleReportPanelLogs.vue @@ -0,0 +1,54 @@ + + + diff --git a/ui/src/pages/SampleReportPanelQc.vue b/ui/src/pages/SampleReportPanelQc.vue new file mode 100644 index 0000000..8b83bce --- /dev/null +++ b/ui/src/pages/SampleReportPanelQc.vue @@ -0,0 +1,72 @@ + + + + + diff --git a/ui/src/pages/SampleReportPanelVisualReport.vue b/ui/src/pages/SampleReportPanelVisualReport.vue new file mode 100644 index 0000000..78a6a64 --- /dev/null +++ b/ui/src/pages/SampleReportPanelVisualReport.vue @@ -0,0 +1,219 @@ + + + + + diff --git a/ui/src/parseProgress.ts b/ui/src/parseProgress.ts index b099463..641d37d 100644 --- a/ui/src/parseProgress.ts +++ b/ui/src/parseProgress.ts @@ -1,7 +1,7 @@ import { ProgressPattern } from "@platforma-open/milaboratories.feature-integration.model"; -// Parse one mitool progress line (already stripped of the [==PROGRESS==] prefix by the accessor) into -// its stage / percent / ETA parts. Ported from blocks/peptide-extraction (same ProgressPattern). +// Parse one mitool progress line, already stripped of the [==PROGRESS==] prefix by the accessor, into its +// stage / percent / ETA parts. Ported from blocks/peptide-extraction (same ProgressPattern). export type ParsedProgress = { raw?: string; stage?: string; diff --git a/ui/src/progress.ts b/ui/src/progress.ts index 55f7a2c..b179eb8 100644 --- a/ui/src/progress.ts +++ b/ui/src/progress.ts @@ -2,9 +2,9 @@ import type { SampleStep } from "@platforma-open/milaboratories.feature-integrat import { ProgressPrefix } from "@platforma-open/milaboratories.feature-integration.model"; import { parseProgressString } from "./parseProgress"; -// Progress-cell config for the Main grid's Progress column. Maps onto the SDK's ColDefProgress: -// status → stage, percent → bar fill (undefined = indeterminate), text → label, suffix → right-hand text -// (set "" to suppress the SDK's default "0%" on an indeterminate bar). +// Progress-cell config for the Main grid's Progress column, mapping onto the SDK's ColDefProgress: status -> +// stage, percent -> bar fill where undefined is indeterminate, text -> label, and suffix -> right-hand text. +// Set suffix to "" to suppress the SDK's default "0%" on an indeterminate bar. export type ProgressCell = { status: "not_started" | "running" | "done"; percent?: number; @@ -12,12 +12,12 @@ export type ProgressCell = { suffix?: string; }; -// The per-sample pipeline runs four stages: parse → refine → tag-stat → per-cell metrics (Python). Each -// stage owns a quarter-band of the overall bar (parse 0–25, refine 25–50, tag-stat 50–75, metrics -// 75–100). A stage's own live % fills WITHIN its band; indeterminate phases hold at the band floor. The -// band floor comes from the deterministic sampleStep (report presence, which only advances), so the bar -// is MONOTONIC — it never resets to zero when a new step starts (the pre-scrap version drove the full bar -// per step, which caused that reset). The rich per-step text still comes from the live mitool stdout. +// The per-sample pipeline runs four stages: parse -> refine -> tag-stat -> per-cell metrics (Python). Each +// stage owns a quarter-band of the overall bar: parse 0-25, refine 25-50, tag-stat 50-75, metrics 75-100. A +// stage's own live % fills WITHIN its band, and an indeterminate phase holds at the band floor. That floor +// comes from the deterministic sampleStep, which is report presence and only advances, so the bar is +// MONOTONIC and never resets to zero when a new step starts. Never drive the full bar per step, which is +// what causes that reset. The rich per-step text comes from the live mitool stdout. const STEP_ORDINAL: Record = { parsing: 0, refining: 1, @@ -27,15 +27,20 @@ const STEP_ORDINAL: Record = { const TOTAL_STEPS = 4; const BAND = 100 / TOTAL_STEPS; -// The streaming mitool steps in order, with their bar ordinal. The label follows the FURTHEST of these -// that currently has a live line (see deriveProgress) — NOT the report-derived sampleStep, which advances -// a beat early (a step's report settles before the next step's live stream starts). Driving the label off -// reports made the bar flash the next step's name ("Counting UMIs") during that gap while refine was still -// streaming; keying off the live stream keeps the label honest to what is actually running. -const WF_STEPS = ["1-parse", "2-refine", "3-tagstat"] as const; -const WF_ORDINAL: Record = { "1-parse": 0, "2-refine": 1, "3-tagstat": 2 }; +// The streaming mitool steps in order, with their bar ordinal. The label follows the FURTHEST of these that +// currently has a live line (see deriveProgress), NOT the report-derived sampleStep, which advances a beat +// early: a step's report settles before the next step's live stream starts. Driving the label off reports +// made the bar flash the next step's name ("Counting UMIs") during that gap while refine was still +// streaming. Keying off the live stream keeps the label honest to what is actually running. +const WF_STEPS = ["1-parse", "2-refine", "3-tagstat", "4-metrics"] as const; +const WF_ORDINAL: Record = { + "1-parse": 0, + "2-refine": 1, + "3-tagstat": 2, + "4-metrics": 3, +}; -// Fallback step label (used when no live line is available for the current step yet). +// Fallback step label, used when no live line is available for the current step yet. const STEP_LABEL: Record = { parsing: "Parsing reads", refining: "Refining barcodes", @@ -43,15 +48,15 @@ const STEP_LABEL: Record = { metrics: "Computing metrics", }; -// mitool progress prose per step. Returns the display text + suffix, plus a localPercent (0–100 WITHIN -// the step) ONLY for monotonic phases — the caller maps that into the step's band. Indeterminate phases -// (refine correction passes, tag-stat on-disk sort) return no localPercent so the bar holds at the band -// floor rather than bouncing. +// mitool progress prose per step. Returns the display text and suffix, plus a localPercent (0-100 WITHIN the +// step) ONLY for monotonic phases -- the caller maps that into the step's band. Indeterminate phases (refine +// correction passes, tag-stat on-disk sort) return no localPercent so the bar holds at the band floor rather +// than bouncing. type StepDisplay = { text: string; suffix: string; localPercent?: number }; -// refine-tags corrects CELL → FEATURE → UMI (fb-pipeline passes -t CELL -t FEATURE -t UMI; mitool orders -// them CELL < FEATURE < UMI). Each tag's correction is recursive/non-monotonic, so we surface WHICH tag -// is in progress ("2 of 3") on an indeterminate bar rather than a jumpy %. +// refine-tags corrects CELL -> FEATURE -> UMI (fb-pipeline passes -t CELL -t FEATURE -t UMI, and mitool +// orders them CELL < FEATURE < UMI). Each tag's correction is recursive and non-monotonic, so we surface +// WHICH tag is in progress ("2 of 3") on an indeterminate bar rather than a jumpy %. const REFINE_TAGS = ["CELL", "FEATURE", "UMI"]; const REFINE_TAG_LABELS: Record = { CELL: "Cell barcodes", @@ -61,12 +66,12 @@ const REFINE_TAG_LABELS: Record = { const REFINE_INIT_LABEL = /init/i; const TAGSTAT_WRITING_LABEL = /writing/i; // tag-stat runs a hierarchical on-disk sort ("Sorting records, step N of M: X%") followed by one final -// "Writing result: X%" pass. Each sub-phase is monotonic 0→100% on its own; naively showing that % makes -// the bar bounce M+1 times. Instead we compose them into one monotonic fill: the sort passes share a -// fixed leading portion of the step (0 → SORT_PORTION%), distributed dynamically across whatever M mitool -// reports, and the write pass owns the remainder (SORT_PORTION → 100%). This is M-independent — sort step -// M always ends at SORT_PORTION and the write starts there, so the bar never jumps or steps back no -// matter how many sort passes run (the "Writing result" line carries no M, so we cannot derive it there). +// "Writing result: X%" pass. Each sub-phase is monotonic 0->100% on its own, and naively showing that % makes +// the bar bounce M+1 times. Instead we compose them into one monotonic fill: the sort passes share a fixed +// leading portion of the step (0 -> SORT_PORTION%), distributed dynamically across whatever M mitool reports, +// and the write pass owns the remainder (SORT_PORTION -> 100%). This is M-independent -- sort step M always +// ends at SORT_PORTION and the write starts there, so the bar never jumps or steps back no matter how many +// sort passes run. The "Writing result" line carries no M, so we cannot derive it there. const TAGSTAT_SORT_LABEL = /step\s+(\d+)\s+of\s+(\d+)/i; const TAGSTAT_SORT_PORTION = 80; @@ -84,8 +89,8 @@ function stepDisplay( suffix: `${REFINE_TAGS.indexOf(tag) + 1} of ${REFINE_TAGS.length}`, }; } - // Global phases keep the stable "Refining barcodes" prefix so the label doesn't jump. mitool's - // lead-in is "Initialization"; the wrap-up stages collapse to ": Finalizing". + // Global phases keep the stable "Refining barcodes" prefix so the label does not jump. mitool's lead-in + // is "Initialization". The wrap-up stages collapse to ": Finalizing". if (REFINE_INIT_LABEL.test(stage)) return { text: "Refining barcodes", suffix: "" }; return { text: "Refining barcodes: Finalizing", suffix: "" }; } @@ -97,14 +102,14 @@ function stepDisplay( if (sort) { const n = Number(sort[1]); const m = Number(sort[2]); - // Sort passes share the leading SORT_PORTION; pass M at 100% lands exactly at SORT_PORTION. + // Sort passes share the leading SORT_PORTION, so pass M at 100% lands exactly at SORT_PORTION. return { text: `Counting UMIs: sorting ${n}/${m}${percentage ? ` — ${Math.round(pct)}%` : ""}`, suffix: etaLabel ?? "", localPercent: ((n - 1 + pct / 100) / m) * TAGSTAT_SORT_PORTION, }; } - // Final "Writing result" pass owns the remainder (SORT_PORTION → 100%), so it always continues from + // Final "Writing result" pass owns the remainder (SORT_PORTION -> 100%), so it always continues from // where the sort ended, independent of how many sort passes ran. if (TAGSTAT_WRITING_LABEL.test(stage)) { return { @@ -116,7 +121,18 @@ function stepDisplay( return { text: "Counting UMIs", suffix: "" }; } - // parse (and any other monotonic step): show the live percent when present, else the bare stage name. + // The Python step names its own phase and carries no ETA: these are whole-frame operations with no + // iteration count to extrapolate from, and an invented ETA is worse than none. + if (step === "4-metrics") { + const pct = percentage ? Number(percentage) : undefined; + return { + text: stage ? `Computing metrics: ${stage}` : "Computing metrics", + suffix: "", + localPercent: pct, + }; + } + + // parse, and any other monotonic step: show the live percent when present, else the bare stage name. if (percentage) { return { text: `Parsing reads: ${percentage}%`, @@ -127,38 +143,49 @@ function stepDisplay( return { text: "Parsing reads", suffix: "" }; } -// Progress cell for one sample. Done once its QC settles (completedSamples). Otherwise the band floor -// comes from the deterministic sampleStep (monotonic), and — if the current step is streaming a live -// line — the rich mitool prose fills the text and (for monotonic phases) the within-band bar. +// Progress cell for one sample. Done once its QC settles (completedSamples). Otherwise the band floor comes +// from the deterministic sampleStep, which is monotonic, and where the current step is streaming a live line +// the rich mitool prose fills the text and, for monotonic phases, the within-band bar. +// One step's stream: its last line, and whether the stream is still open. `live` matters because a closed +// stream's last line is HISTORY, not a reading. mitool prints progress on a timer and the process usually +// finishes between ticks, so the final line is whatever tick landed last -- 97.8% with an ETA of one second +// is a normal way for a finished parse to end. Replayed as if current it reads as a stall, and that is +// exactly how it was read. +export type StepStream = { line?: string; live?: boolean }; + export function deriveProgress( sampleId: string, completed: Set, sampleStep: Record | undefined, - liveLines?: Partial>, + liveLines?: Partial>, ): ProgressCell { if (completed.has(sampleId)) return { status: "done", percent: 100, text: "Done" }; const step = sampleStep?.[sampleId] ?? "parsing"; const reportFloor = STEP_ORDINAL[step] * BAND; - // The label follows the FURTHEST streaming step that actually has a live line — not the report-derived - // sampleStep. A step's report settles a beat before the next step's live stream starts, so keying the - // label off sampleStep flashed the next step's name ("Counting UMIs") during that gap while refine was - // still emitting its last (UMI, 3/3) line. The live stream is the source of truth for WHAT is running. + // The label follows the FURTHEST streaming step that actually has a live line, not the report-derived + // sampleStep. A step's report settles a beat before the next step's live stream starts, so keying the label + // off sampleStep flashed the next step's name ("Counting UMIs") during that gap while refine was still + // emitting its last (UMI, 3/3) line. The live stream is the source of truth for WHAT is running. let liveWf: string | undefined; let liveLine: string | undefined; + let streamOpen = true; if (liveLines) { for (const wf of WF_STEPS) { - if (liveLines[wf]) { + const stream = liveLines[wf]; + if (stream?.line) { liveWf = wf; - liveLine = liveLines[wf]; + liveLine = stream.line; + streamOpen = stream.live !== false; } } } - // metrics (the Python step, no stream) or nothing streaming yet → hold at the report floor with the - // step name. metrics sits at 75% through the whole (slow) Python run. - if (step === "metrics" || liveWf === undefined || liveLine === undefined) { + // Nothing streaming yet: hold at the report floor with the step name. The metrics step used to land + // here unconditionally and sat at 75% through the whole slow Python run; it now streams like the + // others, so only the gap before a step's first line reaches this. + if (liveWf === undefined || liveLine === undefined) { return { status: "running", percent: Math.round(reportFloor), @@ -167,6 +194,19 @@ export function deriveProgress( }; } + // A closed stream means that step finished, whatever percentage its last tick happened to carry. Hold + // at the TOP of its band and drop the label's stale figures: an ETA of one second that never elapses is + // worse than no ETA, because it invites a reader to wait for something that already happened. + if (!streamOpen) { + const finishedFloor = (WF_ORDINAL[liveWf] + 1) * BAND; + return { + status: "running", + percent: Math.round(Math.max(reportFloor, finishedFloor)), + text: STEP_LABEL[step], + suffix: "", + }; + } + const parsed = parseProgressString(liveLine.replace(ProgressPrefix, "")); const d = stepDisplay(liveWf, parsed.stage ?? "", parsed.percentage, parsed.etaLabel); const liveFloor = WF_ORDINAL[liveWf] * BAND; diff --git a/ui/src/results.ts b/ui/src/results.ts index 8185754..7e05e8f 100644 --- a/ui/src/results.ts +++ b/ui/src/results.ts @@ -1,9 +1,13 @@ -import type { QcRow } from "@platforma-open/milaboratories.feature-integration.model"; +import type { + QcMeasurementStatus, + QcRow, + SampleQcReport, +} from "@platforma-open/milaboratories.feature-integration.model"; import type { Color } from "@platforma-sdk/ui-vue"; import { Gradient } from "@platforma-sdk/ui-vue"; import { computed } from "vue"; import { useApp } from "./app"; -import { deriveProgress, type ProgressCell } from "./progress"; +import { deriveProgress, type ProgressCell, type StepStream } from "./progress"; export type { ProgressCell } from "./progress"; @@ -11,36 +15,42 @@ export type SampleResult = { sampleId: string; label: string; progress: ProgressCell; - // Populated once the sample's QC has settled (i.e. it has finished) — drive the Quality + Read + // Populated once the sample's QC has settled, meaning the sample finished, to drive the Quality and Read // recovery columns. Absent while the sample is still running. quality?: QcStatus; recovery?: RecoveryBar; + // The sample's own quality report: every sample-level measurement the software declares, with its status, + // its value and, where it has none, the reason in its place. The Quality tag above is this report's rollup, + // so the tag and the report cannot disagree about one sample. + qcReport?: SampleQcReport; }; -// QC status tag shown in the Quality column (worst-case per sample). Rendered by PlAgCellStatusTag. +// QC status tag shown in the Quality column and beside each measurement. Rendered by PlAgCellStatusTag and +// PlStatusTag, whose vocabulary is upper-case; the software's is not. A measurement with no line behind it +// carries no status at all, and there is no fourth word for that. export type QcStatus = "OK" | "WARN" | "ALERT"; +const STATUS_TAG: Record = { + OK: "OK", + warn: "WARN", + alert: "ALERT", +}; + +export function qcStatusTag(status: QcMeasurementStatus | null): QcStatus | undefined { + return status === null ? undefined : STATUS_TAG[status]; +} + // Stacked-bar settings consumed by PlAgChartStackedBarCell for the Read recovery column. export type RecoveryBar = { title: string; data: { label: string; value: number; color: Color; description: string }[]; }; -// Quality status from the per-sample QC metrics (proposed cutoffs; tune here). Mirrors the analysisLog -// flags: zero cells detected or a very low panel-assigned fraction → ALERT; a low panel-assigned or -// pattern-match fraction → WARN; otherwise OK. panelAssignedFraction is "" when no refine report ran. -function qualityStatus(qc: QcRow): QcStatus { - const paf = typeof qc.panelAssignedFraction === "number" ? qc.panelAssignedFraction : undefined; - if (qc.cellsDetected === 0 || (paf !== undefined && paf < 0.25)) return "ALERT"; - if ((paf !== undefined && paf < 0.5) || qc.matchedFraction < 0.8) return "WARN"; - return "OK"; -} - -// Read-recovery funnel: split each sample's reads into usable (matched the pattern AND kept after the -// feature-barcode panel correction) / off-panel (matched but dropped) / no pattern match. Values are -// read counts summing to readsTotal; PlAgChartStackedBarCell renders them proportionally. When no -// refine report is available (panelAssignedFraction === "") the off-panel split is unknown, so only -// usable (= matched) and no-match are shown. +// Read-recovery funnel, splitting each sample's reads three ways: usable, meaning matched the pattern AND +// kept after the feature-barcode panel correction; off-panel, meaning matched but dropped; and no pattern +// match. The values are read counts summing to readsTotal, and PlAgChartStackedBarCell renders them +// proportionally. Where no refine report is available (panelAssignedFraction === "") the off-panel split is +// unknown, so only usable, which is then matched, and no-match are shown. const RECOVERY_COLORS = { usable: Gradient("viridis").getNthOf(2, 5), offPanel: Gradient("magma").getNthOf(4, 9), @@ -104,31 +114,60 @@ export const sampleResults = computed(() => { const labels = app.model.outputs.sampleLabels ?? {}; const completed = new Set(app.model.outputs.completedSamples ?? []); const qcBySample = app.model.outputs.sampleQc ?? {}; + // Written by the verdict step, so it settles later than the per-sample read QC above and only for a run + // with a V(D)J dataset. A sample without one keeps its progress and its recovery bar and shows no tag. + const reportBySample = app.model.outputs.sampleQcReport ?? {}; const sampleStep = app.model.outputs.sampleStep; - // Early roster signal: the flat parseLogStream registers per sample the moment parse starts — before - // any step report settles — so a sample appears in the grid immediately. (The bar detail comes from - // stepProgress below; this is just "does this sample exist yet".) + // Early roster signal. The flat parseLogStream registers per sample the moment parse starts, before any + // step report settles, so a sample appears in the grid immediately. This answers only "does this sample + // exist yet". The bar detail comes from stepProgress below. const parseProgress = app.model.outputs.parseProgress; const earlyRosterIds = parseProgress ? parseProgress.data.map((p) => String(p.key[0])) : []; + // It also carries parse's live LINE, and that matters for the whole of parse rather than a moment of + // it: `stepLogs` is built by fb-refine-tagstat, so nothing lands in the per-step map below until parse + // is over and the next template runs. Read only for the roster, the bar spent every parse showing the + // bare step label while a percent and an ETA sat right here. + const parseStreamBySample = new Map(); + if (parseProgress) { + for (const p of parseProgress.data) { + const v = p.value as { progressLine?: string; live: boolean } | undefined; + parseStreamBySample.set(String(p.key[0]), { line: v?.progressLine, live: v?.live }); + } + } - // Per-[sampleId, step] live progress lines (parse / refine / tag-stat). Index by sampleId → step → - // progressLine so deriveProgress can pull the line for whichever step the sample is currently on. + // Per-[sampleId, step] live progress lines (parse / refine / tag-stat). Indexed by sampleId -> step -> + // progressLine, so deriveProgress can pull the line for whichever step the sample is on. const stepProgress = app.model.outputs.stepProgress; - const lineBySampleStep = new Map(); + const streamBySampleStep = new Map(); if (stepProgress) { for (const p of stepProgress.data) { const v = p.value as { progressLine?: string; live: boolean } | undefined; - lineBySampleStep.set(`${String(p.key[0])} ${String(p.key[1])}`, v?.progressLine); + streamBySampleStep.set(`${String(p.key[0])} ${String(p.key[1])}`, { + line: v?.progressLine, + live: v?.live, + }); + } + } + // Every streaming step's live line for a sample, so deriveProgress can pick the furthest one actually + // streaming. The report-derived step advances a beat early and flashes the next step's label in the gap. + // The Python step streams on its own output rather than into the per-step map, because it runs in a + // different template. Keyed by sample alone, so it is indexed here and joined under the step name the + // bar knows it by. + const metricsProgress = app.model.outputs.metricsProgress; + const metricsStreamBySample = new Map(); + if (metricsProgress) { + for (const p of metricsProgress.data) { + const v = p.value as { progressLine?: string; live: boolean } | undefined; + metricsStreamBySample.set(String(p.key[0]), { line: v?.progressLine, live: v?.live }); } } - // All streaming steps' live lines for a sample, so deriveProgress can pick the furthest one actually - // streaming (rather than the report-derived step, which advances a beat early and flashed the next - // step's label during the gap). - const liveLinesFor = (sampleId: string): Record => ({ - "1-parse": lineBySampleStep.get(`${sampleId} 1-parse`), - "2-refine": lineBySampleStep.get(`${sampleId} 2-refine`), - "3-tagstat": lineBySampleStep.get(`${sampleId} 3-tagstat`), + const liveLinesFor = (sampleId: string): Record => ({ + // The per-step map wins once it fills, since it keeps streaming after the flat stream closes. + "1-parse": streamBySampleStep.get(`${sampleId} 1-parse`) ?? parseStreamBySample.get(sampleId), + "2-refine": streamBySampleStep.get(`${sampleId} 2-refine`), + "3-tagstat": streamBySampleStep.get(`${sampleId} 3-tagstat`), + "4-metrics": metricsStreamBySample.get(sampleId), }); // Roster: dataset labels ∪ completed ∪ QC'd ∪ any sample with a step signal ∪ early parse signal. @@ -139,7 +178,7 @@ export const sampleResults = computed(() => { ...Object.keys(sampleStep ?? {}), ...earlyRosterIds, ]); - // Roster not enumerated yet → keep the grid's loading overlay rather than flashing an empty table. + // Roster not enumerated yet, so keep the grid's loading overlay rather than flashing an empty table. if (sampleIds.size === 0) return undefined; return [...sampleIds] @@ -147,7 +186,13 @@ export const sampleResults = computed(() => { const label = labels[sampleId] ?? sampleId; // Per-sample QC settles when the sample finishes, so Quality + Read recovery fill in at completion. const qc = qcBySample[sampleId]; - const qcFields = qc ? { quality: qualityStatus(qc), recovery: recoveryBar(qc) } : {}; + const qcReport = reportBySample[sampleId]; + const qcFields = { + ...(qc ? { recovery: recoveryBar(qc) } : {}), + // The tag IS the report's rollup. Nothing here recomputes it, so the grid and the sample's own + // report state one status rather than two that can drift. + ...(qcReport ? { quality: qcStatusTag(qcReport.status), qcReport } : {}), + }; const progressCell = deriveProgress(sampleId, completed, sampleStep, liveLinesFor(sampleId)); return { sampleId, label, progress: progressCell, ...qcFields }; }) diff --git a/workflow/package.json b/workflow/package.json index fcfffcb..4b0a1da 100644 --- a/workflow/package.json +++ b/workflow/package.json @@ -7,7 +7,8 @@ "scripts": { "build": "shx rm -rf dist && pl-tengo build", "format": "/usr/bin/env emacs --script ./format.el || echo 'No emacs.'", - "check": "pl-tengo check" + "check": "pl-tengo check", + "test": "pl-tengo test" }, "dependencies": { "@platforma-open/milaboratories.feature-integration.per-cell-metrics": "workspace:*", diff --git a/workflow/src/column-guards.test.tengo b/workflow/src/column-guards.test.tengo new file mode 100644 index 0000000..bee9891 --- /dev/null +++ b/workflow/src/column-guards.test.tengo @@ -0,0 +1,32 @@ +test := import("@platforma-sdk/workflow-tengo:test") +columnSpecs := import(":column-specs") + +// The legacy per-cell contract keys features by NAME on pl7.app/feature/featureId. A new antigen table +// reusing that axis while carrying barcode SEQUENCES would leave the axis identity untouched and invert +// its value space, so nothing downstream would error and joins would return wrong rows. That is the +// worst available failure, which is why it is asserted at import. +// +// The predicate is tested rather than the assertion. Tengo has no try/catch, so a test cannot exercise a +// panicking guard at all. +Test_the_legacy_feature_axis_is_detected := func() { + legacy := [ + { column: "sampleId", spec: { name: "pl7.app/sampleId", type: "String" } }, + { column: "tag", spec: { name: "pl7.app/feature/featureId", type: "String" } } + ] + test.isTrue(columnSpecs.usesLegacyFeatureAxis(legacy), + "an antigen table keyed on the legacy feature axis must be detected") +} + +// The minted antigen axes pass, so this checks one name rather than refusing everything and making every +// caller work around it. +Test_the_minted_antigen_axes_are_not_the_legacy_axis := func() { + blockId := "blk" + minted := [ + { column: "tag", spec: columnSpecs.tagAxis(blockId) }, + { column: "panelId", spec: columnSpecs.panelAxis(blockId) } + ] + test.isFalse(columnSpecs.usesLegacyFeatureAxis(minted), + "the minted antigen axes are not the legacy axis and must pass") + test.isEqual(len(columnSpecs.guardNoLegacyFeatureAxis(minted)), 2, + "the guard returns its input unchanged when nothing is wrong") +} diff --git a/workflow/src/column-specs.lib.tengo b/workflow/src/column-specs.lib.tengo index d7e763d..4c5d7bf 100644 --- a/workflow/src/column-specs.lib.tengo +++ b/workflow/src/column-specs.lib.tengo @@ -1,30 +1,83 @@ // Export column + axis specs for the Feature Integration block. // -// Centralised here (mirrors blocks/peptide-extraction column-specs.lib.tengo and -// blocks/mixcr-clonotyping calculate-export-specs.lib.tengo) so main.tpl.tengo stays readable and the -// per-cell contract columns carry the full standard annotation set (abundance flags, order priority, -// table visibility). Annotations do NOT affect p-column identity (name + domain + axes do), so the -// downstream discovery contract is unchanged by anything in this file. +// Two layers live here. The per-cell and per-feature contract (pl7.app/feature/*) is the block's original +// surface. The verdict stage, in the second half, describes what emit_verdicts.py writes. The two never +// share an axis. +// +// Centralised so main.tpl.tengo stays readable and every column carries the full annotation set: +// abundance flags, order priority, table visibility. Mirrors blocks/peptide-extraction +// column-specs.lib.tengo and blocks/mixcr-clonotyping calculate-export-specs.lib.tengo. +// +// COLUMN IDENTITY is name + domain + axes. Annotations are excluded from it. Every note below saying +// "in the domain, not the annotations" rests on that one fact. +ll := import("@platforma-sdk/workflow-tengo:ll") maps := import("@platforma-sdk/workflow-tengo:maps") strings := import("@platforma-sdk/workflow-tengo:strings") json := import("json") +text := import("text") + +// No column this block emits may be orderable. A verdict states what the experiment could establish, and +// a count of verdicts is derived from that statement. Neither is a magnitude, so neither may be ranked. +// `pl7.app/isScore` and the `pl7.app/score/*` family are what make a column rankable downstream, so this +// refuses them outright. Ordinary filter annotations are untouched. +// +// The guard runs over every annotation map this module builds, catching a computed key as well as a +// literal one, so a reintroduction fails the workflow render instead of shipping. It cannot see a column +// built outside this module, which the block test asserts instead. +SCORE_ANNOTATION := "pl7.app/isScore" +SCORE_FAMILY_PREFIX := "pl7.app/score/" +LEGACY_FEATURE_AXIS := "pl7.app/feature/featureId" + +guardNoScore := func(annotations) { + for key, _ in annotations { + ll.assert( + key != SCORE_ANNOTATION && !text.has_prefix(key, SCORE_FAMILY_PREFIX), + "column-specs: annotation %v makes a column orderable; verdicts and their counts are filterable, never orderable", + key) + } + return annotations +} + +/* The legacy per-cell contract keys features by name on pl7.app/feature/featureId, and those columns stay + as they are. A NEW column must never key on that axis while carrying barcode sequences: the axis + identity would be unchanged and its value space inverted, so no query would fail and joins would return + wrong rows. Every antigen axis is therefore minted under its own name. The block test asserts the same + over EMITTED columns, which needs a running workflow. */ +// Split from the assertion below so it can be tested both ways. Tengo has no try/catch, so a predicate is +// the only part of a guard a test can exercise. +usesLegacyFeatureAxis := func(axesSpec) { + for axis in axesSpec { + if axis.spec.name == LEGACY_FEATURE_AXIS { + return true + } + } + return false +} + +guardNoLegacyFeatureAxis := func(axesSpec) { + ll.assert( + !usesLegacyFeatureAxis(axesSpec), + "column-specs: an antigen table keys on %v; antigen axes are minted under their own names so a barcode value space cannot inherit a feature-name axis identity", + LEGACY_FEATURE_AXIS) + return axesSpec +} -// Standard table annotations: order priority + default visibility. +// Standard table annotations: order priority and default visibility. // visibility: undefined -> hidden, true -> default, false -> optional. a := func(order, defaultVisibility, spec) { - return maps.merge(spec, { + return maps.merge(guardNoScore(spec), { "pl7.app/table/orderPriority": string(order), "pl7.app/table/visibility": is_undefined(defaultVisibility) ? "hidden" : defaultVisibility ? "default" : "optional" }) } // --- Axes (per-cell contract: [pl7.app/sampleId, pl7.app/sc/cellId, pl7.app/feature/featureId]) --- -// The sample axis is prepended by processColumn from the iteration key; these are the per-sample axes. +// The sample axis is prepended by processColumn from the iteration key. These are the per-sample axes. -// cellAxis: the single-cell barcode. `parents` records the sampleId axis it nests under (the sample -// axis name is inherited verbatim from the input FASTQ column). +// cellAxis: the single-cell barcode. `parents` records the sampleId axis it nests under, whose name is +// inherited verbatim from the input FASTQ column. cellAxis := func(sampleAxisName) { return { name: "pl7.app/sc/cellId", @@ -37,7 +90,8 @@ cellAxis := func(sampleAxisName) { } } -// featureAxis: the feature/antigen name (block-scoped domain so distinct blocks' features don't collide). +// featureAxis: the feature or antigen name. The domain is block-scoped so two blocks' features cannot +// collide. featureAxis := func(blockId) { return { name: "pl7.app/feature/featureId", @@ -48,12 +102,13 @@ featureAxis := func(blockId) { } // valueOutputs: the per-cell contract value columns, as processColumn `Xsv` output declarations. -// hasControl gates the specificity score — only meaningful with a negative control. // -// umiCount is the primary abundance (abundance/isPrimary). It is deliberately NOT isAnchor: this -// column is meant to be DISCOVERED under the downstream VDJ single-cell anchor (via the cellLinker), -// not to be an anchor itself, so marking it isAnchor would be semantically wrong here. -valueOutputs := func(blockId, sampleAxisName, hasControl) { +// No consensus or dominant-feature column, and no specificity score: a specificity score is a binding +// magnitude, the thing a four-state verdict replaces. Nothing here takes the negative control. +// +// umiCount is the primary abundance (abundance/isPrimary), and deliberately NOT isAnchor. It is meant to +// be DISCOVERED under the downstream VDJ single-cell anchor, through the cellLinker. +valueOutputs := func(blockId, sampleAxisName) { cell := cellAxis(sampleAxisName) feat := featureAxis(blockId) @@ -113,73 +168,17 @@ valueOutputs := func(blockId, sampleAxisName, hasControl) { path: ["fractions"] } - consensus := { - type: "Xsv", - xsvType: "csv", - settings: { - axes: [{ column: "cellId", spec: cell }], - columns: [{ - column: "consensusFeature", id: "consensusFeature", - spec: { - name: "pl7.app/feature/consensusFeature", - valueType: "String", - annotations: a(88000, true, { - "pl7.app/label": "Consensus feature", - "pl7.app/description": "The cell's dominant feature/antigen — assigned only when one feature's share reaches the dominance threshold (default 0.6); otherwise 'ambiguous'.", - "pl7.app/isDiscreteFilter": "true" - }) - } - }], - storageFormat: "Parquet", - partitionKeyLength: 0 - }, - name: "consensus", - path: ["consensus"] - } - - outputs := [abundance, fractions, consensus] - - if hasControl { - outputs = append(outputs, { - type: "Xsv", - xsvType: "csv", - settings: { - axes: [{ column: "cellId", spec: cell }, { column: "feature", spec: feat }], - columns: [{ - column: "specificityScore", id: "specificityScore", - spec: { - name: "pl7.app/feature/specificityScore", - valueType: "Double", - annotations: a(87000, true, { - "pl7.app/label": "Specificity score", - "pl7.app/description": "Cell Ranger's BEAM specificity score (0–100): confidence that this antigen's binding exceeds the negative control, from their UMI counts — not binding strength or affinity. Needs a designated negative control.", - "pl7.app/min": "0", - "pl7.app/max": "100", - "pl7.app/isScore": "true", - "pl7.app/format": ".1f" - }) - } - }], - storageFormat: "Parquet", - partitionKeyLength: 0 - }, - name: "specificity", - path: ["specificity"] - }) - } - - return outputs + return [abundance, fractions] } -// perCellSummaryOutput: the TABLE-ONLY per-cell collapse (one row per [sampleId, cellId]) produced by -// per_cell_metrics.py's result_per_cell_summary.csv. Its columns are the cell's max feature UMI count -// / max feature fraction (/ max specificity score, with a control) plus a "feature : umi : fraction | -// ..." summary string sorted by descending fraction. This drives the Main results table INSTEAD of the -// per-(cell x feature) matrix; the per-cell export contract (abundance/fractions/consensus/specificity) -// is unchanged. The "Max ..." labels distinguish these aggregates from the exported per-feature -// columns. maxSpecificityScore is emitted (and imported) only with a negative control — same gating as -// the per-feature specificity column. Keyed [cellId]; the sample axis is prepended by processColumn. -perCellSummaryOutput := func(blockId, sampleAxisName, hasControl) { +// perCellSummaryOutput: the TABLE-ONLY per-cell collapse, one row per [sampleId, cellId], from +// per_cell_metrics.py's result_per_cell_summary.csv. Columns: the cell's max feature UMI count, its max +// feature fraction, and a "feature (fraction%, umi), ..." string sorted by descending fraction. Drives the +// Main results table INSTEAD of the per-(cell x feature) matrix, leaving the per-cell export contract +// unchanged. The "Max ..." labels separate these aggregates from the exported per-feature columns. No +// max-specificity aggregate: it read a binding magnitude. Keyed [cellId], sample axis prepended by +// processColumn. +perCellSummaryOutput := func(blockId, sampleAxisName) { cell := cellAxis(sampleAxisName) cols := [ @@ -189,7 +188,7 @@ perCellSummaryOutput := func(blockId, sampleAxisName, hasControl) { name: "pl7.app/feature/maxUmiCount", valueType: "Int", annotations: a(87000, true, { - "pl7.app/label": "Max Feature UMI count", + "pl7.app/label": "Max feature UMI count", "pl7.app/description": "The cell's largest per-feature UMI count (the dominant feature's molecule count). The full per-feature counts are in the Feature breakdown column and the exported abundance matrix.", "pl7.app/min": "0" }) @@ -201,7 +200,7 @@ perCellSummaryOutput := func(blockId, sampleAxisName, hasControl) { name: "pl7.app/feature/maxFraction", valueType: "Double", annotations: a(86000, true, { - "pl7.app/label": "Max Feature Fraction", + "pl7.app/label": "Max feature fraction", "pl7.app/description": "The cell's largest per-feature fraction (the dominant feature's share of the cell's total feature signal).", "pl7.app/min": "0", "pl7.app/max": "1", @@ -211,26 +210,8 @@ perCellSummaryOutput := func(blockId, sampleAxisName, hasControl) { } ] - if hasControl { - cols = append(cols, { - column: "maxSpecificityScore", id: "maxSpecificityScore", - spec: { - name: "pl7.app/feature/maxSpecificityScore", - valueType: "Double", - annotations: a(85000, true, { - "pl7.app/label": "Max Specificity score", - "pl7.app/description": "The cell's highest per-feature Cell Ranger BEAM specificity score (0–100): confidence that the best-supported feature's binding exceeds the negative control. Needs a designated negative control.", - "pl7.app/min": "0", - "pl7.app/max": "100", - "pl7.app/isScore": "true", - "pl7.app/format": ".1f" - }) - } - }) - } - - // The "all features" summary string (mirrors antibody-sequence-liabilities' pl7.app/isSummary - // column). Listed last (lowest orderPriority) so the headline aggregates read first. + // The "all features" summary string, mirroring antibody-sequence-liabilities' pl7.app/isSummary + // column. Listed last, at the lowest orderPriority, so the headline aggregates read first. cols = append(cols, { column: "featureSummary", id: "featureSummary", spec: { @@ -259,14 +240,12 @@ perCellSummaryOutput := func(blockId, sampleAxisName, hasControl) { } // qcFileMapOutput: collect the per-sample QC CSV (fb-pipeline `qc` output) into a resource map keyed -// [sampleId]. A per-sample QC row is a SCALAR (0 within-sample axes); an xsv import cannot emit that -// (it requires >=1 within-file axis: xsv-import-file.lib.tengo asserts partitionKeyLength < len(axes)), -// which is why the old empty-axes Xsv output crashed the SHARED processColumn render and took the whole -// perCellTable down with it. Instead the per-sample one-row CSVs are concatenated (real sampleId -// injected per row) and imported ONCE, keyed [sampleId], by qc-summary.tpl.tengo. `type: "Resource"` -// yields a [sampleId] -> file map (same mechanism as blocks/mixcr-clonotyping's `clns`/`qc` outputs); -// the default path ["qc"] matches the fb-pipeline body's `qc:` return (the old path ["qcSummary"] was -// itself wrong — no such body output exists). Not part of the per-cell export contract; outputs-only. +// [sampleId]. A per-sample QC row is a SCALAR, and an xsv import cannot emit one: xsv-import-file +// .lib.tengo asserts partitionKeyLength < len(axes), so it needs a within-file axis. An empty-axes Xsv +// output here crashes the SHARED processColumn render and takes perCellTable down with it. So +// qc-summary.tpl.tengo concatenates the one-row CSVs, injects the real sampleId per row, and imports them +// ONCE. `type: "Resource"` yields a [sampleId] -> file map, as blocks/mixcr-clonotyping's `clns`/`qc` do. +// The default path ["qc"] matches the fb-pipeline body's `qc:` return. Outputs-only. qcFileMapOutput := func(blockId) { return { type: "Resource", @@ -281,18 +260,16 @@ qcFileMapOutput := func(blockId) { } // featurePropertyImportColumns: the xsv.importFile `columns` for the per-feature property CSV -// (emit_feature_properties.py). One column per EXTRA tag-CSV header (every column beyond the mapped -// barcode-sequence + feature-name roles — A-0026), imported generically with NO hardcoded schema. Each -// is a String p-column keyed on the shared feature axis (the caller supplies `axes` = [featureAxis]), so -// the property rides pl7.app/feature/featureId into VDJ Multiomic Integration's per-feature outputs and -// lead selection with no re-import. -// -// Identity: a single fixed name (pl7.app/feature/property) with the raw header carried in the domain -// (pl7.app/feature/propertyName) so distinct properties never collide and two different properties are -// distinct p-columns (name + domain + axes define identity; annotations do not). Each column is marked -// isDiscreteFilter and carries its distinct values (discreteValues) so downstream can group / filter -// features by it (e.g. pool all human vs all cyno antigens — A-0015). Values are imported verbatim as -// strings: Feature Integration imports "as given" and neither validates nor interprets them (A-0026). +// (emit_feature_properties.py). One String column per EXTRA tag-CSV header -- every column beyond the +// mapped barcode-sequence and feature-name roles (A-0026) -- with NO hardcoded schema, keyed on the shared +// feature axis the caller supplies as `axes` = [featureAxis]. The property therefore rides +// pl7.app/feature/featureId into VDJ Multiomic Integration and lead selection with no re-import. +// +// One fixed name (pl7.app/feature/property) with the raw header in the domain +// (pl7.app/feature/propertyName), so two properties are distinct p-columns and never collide. Each is +// marked isDiscreteFilter and carries its distinct values, so downstream can pool all human against all +// cyno antigens (A-0015). Values are imported verbatim: this block imports "as given" and neither +// validates nor interprets them (A-0026). featurePropertyImportColumns := func(propertyColumns, valuesByColumn) { cols := [] for i, name in propertyColumns { @@ -304,8 +281,8 @@ featurePropertyImportColumns := func(propertyColumns, valuesByColumn) { name: "pl7.app/feature/property", valueType: "String", domain: { "pl7.app/feature/propertyName": name }, - // order by header position (earlier headers rank higher); optional visibility so imported - // properties don't crowd downstream tables by default. + // Order by header position, so earlier headers rank higher. Optional visibility, so imported + // properties do not crowd downstream tables by default. annotations: a(50000 - i, false, { "pl7.app/label": name, "pl7.app/isDiscreteFilter": "true", @@ -317,11 +294,11 @@ featurePropertyImportColumns := func(propertyColumns, valuesByColumn) { return cols } -// qcSummaryColumnsSpec: the xsv.importFile spec for the COMBINED per-sample QC table (one row per -// sample), used by qc-summary.tpl.tengo. Keyed [sampleId] with the axis spec inherited verbatim from -// the block's sampleId axis, so the QC table's sampleId axis unifies with the per-cell contract columns. -// Each metric keeps its own valueType/label/orderPriority/format. Not part of the per-cell export -// contract; outputs-only. +// qcSummaryColumnsSpec: the xsv.importFile spec for the COMBINED per-sample QC table, one row per sample, +// used by qc-summary.tpl.tengo. Keyed [sampleId] with the axis spec inherited verbatim from the block's +// sampleId axis, so the QC table's sampleId axis unifies with the per-cell contract columns. Each metric +// keeps its own valueType, label, orderPriority and format. Not part of the per-cell export contract, +// outputs-only. qcSummaryColumnsSpec := func(sampleAxisSpec) { numCol := func(id, colName, label, order, fmt, ...desc) { spec := { @@ -347,7 +324,1286 @@ qcSummaryColumnsSpec := func(sampleAxisSpec) { numCol("featuresDetected", "featuresDetected", "Features detected", 86000, undefined), numCol("totalUniqueUmis", "totalUniqueUmis", "Total distinct UMIs", 85000, undefined), numCol("medianUmisPerCell", "medianUmisPerCell", "Median UMIs / cell", 84000, ".1f"), - numCol("panelAssignedFraction", "panelAssignedFraction", "Panel-assigned fraction", 83000, ".2p", "Fraction of feature-barcode reads kept after correcting their barcode against the panel; reads too far from any panel entry are dropped. A low value flags a panel or read-geometry mismatch.") + numCol("panelAssignedFraction", "panelAssignedFraction", "Panel-assigned fraction", 83000, ".2p", "Fraction of feature-barcode reads kept after correcting their barcode against the panel; reads too far from any panel entry are dropped. Its complement is the share landing in barcodes the panel never declared, which is where this measurement's line comes from. A low value flags a panel or read-geometry mismatch."), + numCol("cellBarcodeValidFraction", "cellBarcodeValidFraction", "Valid cell-barcode fraction", 82000, ".2p", "Fraction of reads whose cell barcode corrects onto the whitelist this chemistry produces. A low value points at the wrong whitelist or the wrong read geometry rather than at the panel.") + ], + storageFormat: "Parquet", + partitionKeyLength: 0 + } +} + +// negativeControlColumn: the xsv.importFile `columns` entry for the negative-control marker CSV +// (emit_feature_properties.py `_negative_control.csv`, holding "true" for the control). One String +// p-column (pl7.app/feature/negativeControl) on the shared feature axis. Hidden and NOT a discrete filter: +// it is not a user-facing property. It exists so the control rides pl7.app/feature/featureId into VDJ +// Multiomic Integration, which removes the control from its restriction index, breadth, fraction columns +// and dominant call. Distinct from the generic pl7.app/feature/property import (A-0026). +negativeControlColumn := func() { + return { + column: "value", + id: "negativeControl", + spec: { + name: "pl7.app/feature/negativeControl", + valueType: "String", + annotations: guardNoScore({ + "pl7.app/label": "Negative control", + "pl7.app/table/visibility": "hidden" + }) + } + } +} + +// ===================================================================================== +// The verdict stage +// +// Everything below describes what emit_verdicts.py writes: twelve CSVs plus a run-meta JSON. One function +// per CSV, each returning a complete xsv.importFile spec, so the file that names a column is also the file +// that says which CSV column it comes from. +// ===================================================================================== + +// --- Axes ----------------------------------------------------------------------------- +// +// Four axes are minted rather than reused. Putting tag keys on the existing pl7.app/feature/featureId axis +// would leave the axis identity unchanged while inverting its value space, from feature names to barcode +// sequences. No query would fail and no join would error. Joins would return rows for the wrong thing. The +// legacy pl7.app/feature/* columns are left as they are, and the verdict stage keys on its own vocabulary. + +// tagAxis: one declared barcode sequence. +// +// `tagType` sits in the domain rather than in the name, so a second kind of tag such as surface markers is +// a branch inside one vocabulary instead of a second vocabulary. blockId keeps two blocks' tags from +// colliding, exactly as it does for featureAxis. +tagAxis := func(blockId) { + return { + name: "pl7.app/antigen/tagId", + type: "String", + domain: { + "pl7.app/blockId": blockId, + "pl7.app/antigen/tagType": "antigen" + }, + annotations: { "pl7.app/label": "Barcode" } + } +} + +// identityAxis: the thing a verdict is about -- one tag, or a group of tags read as one antigen. +// +// `groupingId` is in the DOMAIN, and ships even though v1 only ever sets "per-tag". Domain is part of axis +// identity and annotations are not, so without it the axis would keep one identity while its values +// changed meaning under a regrouping, and a saved downstream filter would match nothing or match something +// else. With it, one run can carry several groupings side by side as distinct axes. `segmentedBy` tells a +// reader which domain key separates them. +identityAxis := func(blockId, groupingId) { + return { + name: "pl7.app/antigen/identityId", + type: "String", + domain: { + "pl7.app/blockId": blockId, + "pl7.app/antigen/tagType": "antigen", + "pl7.app/antigen/groupingId": groupingId + }, + annotations: { + "pl7.app/label": "Antigen identity", + "pl7.app/segmentedBy": string(json.encode(["pl7.app/antigen/groupingId"])) + } + } +} + +// panelAxis: one distinct declared tag set. No panel file names its panel, so the id is a hash of the +// sorted tag list (emit_verdicts.py `_panel_id`), stable across re-runs of the same declaration. Where one +// panel covers every sample the axis takes a single value, and the column still renders, holding that one +// value on every row: the SDK suppresses no axis for being constant. +panelAxis := func(blockId) { + return { + name: "pl7.app/antigen/panelId", + type: "String", + domain: { "pl7.app/blockId": blockId }, + annotations: { "pl7.app/label": "Panel" } + } +} + +// captureAxis: the physical capture a sample came off. Minted before any capture assignment reaches the +// block, because adding an axis to a released QC column changes that column's identity, where adding a +// value to an existing axis does not. Every capture presently rolls up to no status at all, the honest +// reading. No emitted column is keyed on it yet. The capture rollup travels as a row in the QC frame, and +// the axis is exported so the sample->capture map can be added without re-identifying anything. +captureAxis := func(blockId) { + return { + name: "pl7.app/antigen/captureId", + type: "String", + domain: { "pl7.app/blockId": blockId } + } +} + +// The QC key is three axes, not two. emit_verdicts.py writes result_qc.csv keyed +// (level, entity, measurement), and `entity` is load-bearing: under the default per-tag grouping a tag id +// and an identity id are the SAME string, so without a part of the key naming which kind of thing the row +// is about, a tag row and an identity row for the same barcode collide. A measurement is an axis VALUE, so +// the fifteenth measurement costs a row, not a column. +qcLevelAxis := func(blockId) { + return { + name: "pl7.app/antigen/qcLevel", + type: "String", + domain: { "pl7.app/blockId": blockId }, + annotations: { "pl7.app/label": "Level" } + } +} + +qcEntityAxis := func(blockId) { + return { + name: "pl7.app/antigen/qcEntity", + type: "String", + domain: { "pl7.app/blockId": blockId }, + annotations: { "pl7.app/label": "Measurement subject" } + } +} + +qcMeasurementAxis := func(blockId) { + return { + name: "pl7.app/antigen/qcMeasurement", + type: "String", + domain: { "pl7.app/blockId": blockId }, + annotations: { "pl7.app/label": "Measurement" } + } +} + +// --- Value vocabularies --------------------------------------------------------------- +// +// Each mirrors a closed enum in the software, so a filter offers every value the data can take. There is +// deliberately no default cutoff: the whole score family is refused, and a default cutoff is part of it. +// All four states stay reachable through the ordinary discrete filter. +VERDICT_STATES := string(json.encode(["bound", "not bound", "never asked", "unreliable"])) +BOOL_VALUES := string(json.encode(["true", "false"])) +UNRELIABLE_REASONS := string(json.encode([ + "never-offered", "no-comparator", "all-cells-gated", + "tie", "below-agreement-floor", "too-few-voters"])) +ADMISSIBILITY_VALUES := string(json.encode([ + "admissible", + "cell set aside by the admissibility gate", + "no comparator for this cell"])) +// Three and no fourth. A measurement with no line behind it leaves this column EMPTY rather than +// taking a fourth word, and which of the two no-status cases it is reads from the value: a number +// means computed-with-no-line, its absence means nothing computed it. +QC_STATUSES := string(json.encode(["OK", "warn", "alert"])) +// The three defence routes `Measurement.line` names, and no fourth: qc_measures.py's +// `LINE_ROUTES` plus the categorical route. Empty means the measurement carries no line. +QC_ROUTES := string(json.encode(["inherited", "categorical", "recommended-and-observed"])) +MISMATCH_DIRECTIONS := string(json.encode(["declared-never-seen", "undeclared-in-panel"])) + +// servedDomain: which comparator and which cell list actually served this run. +// +// In the DOMAIN, not the annotations. Two runs served by different comparators would otherwise emit +// columns of identical identity and be silently unioned in a pool holding both, and a verdict read against +// a declared reference tag is not the same reading as one read against the panel's own signal. `served` is +// what emit_verdicts.py RESOLVED. It always equals what was requested, since a rung that cannot serve +// refuses the run rather than falling to another, and it arrives from the run-meta JSON because that +// record is also where a run that established no baseline says so. +// +// The domain value is the MACHINE token (`declared` / `panel` / `distribution`). There is no `none`: a +// baseline is required and a run without one does not happen. Never put display prose here -- a domain +// value must outlive any wording. +servedDomain := func(served) { + return { + "pl7.app/antigen/referenceChoice": served.referenceChoice, + "pl7.app/antigen/cellListSource": served.cellListSource + } +} + +// servedNotes: the floor and the cutoff, which stay ANNOTATIONS. They are informational, moving where the +// line falls rather than what a reading means, so a change to them must not fork column identity the way a +// change of comparator does. +servedNotes := func(served) { + return { + "pl7.app/antigen/countFloor": string(served.floor), + "pl7.app/antigen/boundCutoff": string(served.cutoff) + } +} + +// --- Verdicts: result_verdicts.csv, keyed (setId, identity) ----------------------------- +// +// The set axis is taken VERBATIM from the cell linker's third axis (linkerCol.spec.axesSpec[2]), so the +// verdicts land on the very clonotype key the clonotyping run produced. Rebuilding it here would produce a +// lookalike axis that joins to nothing. +verdictsImportSpec := func(setAxisSpec, identityAxisSpec, served) { + // The importer builds each column's axesSpec from the `axes` list below, so only the domain is added + // here. Every column in this family shares it. + verdictCol := func(colName, id, spec) { + spec.domain = servedDomain(served) + return { column: colName, id: id, spec: spec } + } + return { + axes: [{ column: "setId", spec: setAxisSpec }, { column: "identity", spec: identityAxisSpec }], + columns: [ + verdictCol("state", "verdict", { + name: "pl7.app/antigen/verdict", + valueType: "String", + annotations: a(100000, true, maps.merge(servedNotes(served), { + "pl7.app/label": "Binding verdict", + "pl7.app/isDiscreteFilter": "true", + "pl7.app/discreteValues": VERDICT_STATES, + "pl7.app/description": "One of four states. 'Never asked' means the experiment did not put this antigen to these cells; 'unreliable' means it did and the data cannot settle it. Neither is a kind of 'not bound'." + })) + }), + verdictCol("unreliableReason", "unreliableReason", { + name: "pl7.app/antigen/unreliableReason", + valueType: "String", + annotations: a(96800, false, { + "pl7.app/label": "Why unsettled", + "pl7.app/isDiscreteFilter": "true", + "pl7.app/discreteValues": UNRELIABLE_REASONS, + "pl7.app/description": "A statement left unsettled by a position the experiment never asked calls for a panel change; one left unsettled by a reading that did not survive calls for a re-run. A bare 'unreliable' cannot tell them apart." + }) + }), + verdictCol("cellsCouldAnswer", "cellsCouldAnswer", { + name: "pl7.app/antigen/cellsCouldAnswer", + valueType: "Int", + annotations: a(99000, true, { + "pl7.app/label": "Cells that could answer", + "pl7.app/min": "0" + }) + }), + verdictCol("cellsAnswered", "cellsAnswered", { + name: "pl7.app/antigen/cellsAnswered", + valueType: "Int", + annotations: a(98000, true, { + "pl7.app/label": "Cells that answered", + "pl7.app/min": "0" + }) + }), + // How many of the set's cells read BOUND here. Neither of the two counts above, and not + // recoverable from `agreement`, which is the majority's share -- and the majority is not always + // bound. A position where three of ten read bound and the verdict is "not bound" still carries + // three, and `the-explore-readout` puts exactly that number in the expansion. + verdictCol("cellsBound", "cellsBound", { + name: "pl7.app/antigen/cellsBound", + valueType: "Int", + annotations: a(97500, true, { + "pl7.app/label": "Cells that read bound", + "pl7.app/min": "0" + }) + }), + // The other half of the same pair. SETTLED holds only bound and not-bound, so the two together + // always exhaust cellsAnswered. Reported alongside cellsBound rather than left for a reader to + // derive, because a not-bound reading with no count beside it looks unsupported. + verdictCol("cellsNotBound", "cellsNotBound", { + name: "pl7.app/antigen/cellsNotBound", + valueType: "Int", + annotations: a(97250, true, { + "pl7.app/label": "Cells that read not bound", + "pl7.app/min": "0" + }) + }), + verdictCol("agreement", "agreement", { + name: "pl7.app/antigen/agreement", + valueType: "Double", + annotations: a(97000, false, { + "pl7.app/label": "Cell agreement", + "pl7.app/format": ".2p", + "pl7.app/min": "0", + "pl7.app/max": "1" + }) + }), + verdictCol("wasCompeted", "wasCompeted", { + name: "pl7.app/antigen/wasCompeted", + valueType: "String", + annotations: a(96500, true, { + "pl7.app/label": "Reading was competed", + "pl7.app/isDiscreteFilter": "true", + "pl7.app/discreteValues": BOOL_VALUES, + "pl7.app/description": "True where this antigen read 'not bound' and something it was declared to compete with read 'bound' for the same clonotype. A statement can test this; the state itself is unchanged." + }) + }), + verdictCol("competedWith", "competedWith", { + name: "pl7.app/antigen/competedWith", + valueType: "String", + annotations: a(96000, false, { "pl7.app/label": "Competed with" }) + }) + ], + storageFormat: "Parquet", + partitionKeyLength: 0 + } +} + +// --- Set counts: result_set_counts.csv, keyed (setId) ----------------------------------- +// +// The family that actually reaches lead selection: a column carrying an axis the clonotype anchor does not +// have is dropped there with no error, so nothing keyed (set, identity) is visible. +// +// They count identities, never tags, and their LABELS say so -- "Identities bound", "Identities offered". +// Never label them "Antigens ...": `040-glossary` separates an identity, "a group of tags read as one +// thing", from the antigen a tag carries. +// +// Still not orderable. A count of verdicts is derived from verdicts, and ordering by it is ordering by +// binding breadth. A downstream block builds the ranked list, from these plus other assays. +// +// `settledCount` is emitted because set_counts() writes it. It always equals offeredCount - +// unsettledCount, and importing it is cheaper than every reader re-deriving it. +setCountsImportSpec := func(setAxisSpec, served) { + // `visible` is a real parameter rather than a constant because one of these ships off by default, and + // `a` renders false as "optional": present in the table's column chooser, absent until a reader turns it + // on. + countCol := func(colName, name, label, order, visible, description) { + annotations := { "pl7.app/label": label, "pl7.app/min": "0" } + if description != "" { + annotations["pl7.app/description"] = description + } + return { + column: colName, + id: colName, + spec: { + name: name, + valueType: "Int", + domain: servedDomain(served), + annotations: a(order, visible, annotations) + } + } + } + return { + axes: [{ column: "setId", spec: setAxisSpec }], + columns: [ + // Beside the clonotype's name, and default-visible. `the-explore-readout` puts the clonotype's own + // cell count there and keeps numbers out of every position: a verdict from three cells and one + // from forty print the same word, so the count is what tells them apart. It does not vary by + // identity, which is why it belongs here rather than in the card. How many of these cells could + // answer at a given identity is a different number, and travels with that identity's verdict. + countCol("cellCount", "pl7.app/antigen/cellCount", "Cells", 96000, true, + "How many cells this clonotype has. The verdicts rest on these cells; how many of them could answer at a given identity travels with that identity's verdict."), + // Off by default, which is `the-explore-readout`'s own wording: carried per clonotype, not shown + // by default, for a reader who wants it. It qualifies the cell count directly above it -- forty + // cells of which thirty-eight read nothing is a different clonotype from forty that all read + // something -- so it is ordered next to it rather than among the identity counts below. + countCol("cellsReadingNothing", "pl7.app/antigen/cellsReadingNothing", "Cells that read nothing", 95500, false, + "How many of this clonotype's cells were left with no count on any tag, the baseline included, once the minimum count had run. It changes no verdict — those cells vote not bound like any other — and it separates a negative resting on cells that read something from one resting on cells that read nothing."), + countCol("boundCount", "pl7.app/antigen/boundCount", "Identities bound", 95000, true, + "How many distinct antigen identities this clonotype bound. Breadth, never strength: nothing here says how well it bound any of them."), + countCol("offeredCount", "pl7.app/antigen/offeredCount", "Identities offered", 94000, true, + "How many identities this clonotype's cells were actually stained with — the denominator a rate must use, since a clone offered eight of ten and binding all eight failed nothing."), + countCol("settledCount", "pl7.app/antigen/settledCount", "Identities settled", 93500, true, + "Of the identities offered, how many the data could settle either way."), + countCol("unsettledCount", "pl7.app/antigen/unsettledCount", "Identities unsettled", 93000, true, + "Offered identities the data could not settle. They stay in the offered count and are reported here beside it, so one bad reading does not void every count in the run.") + ], + storageFormat: "Parquet", + partitionKeyLength: 0 + } +} + +// --- The clonotype's cell count, for the punchcard's OWN frame --------------------------- +// +// The same column setCountsImportSpec carries, imported again into the card's frame. A block's own exports +// are not in its own result pool, which is why the punchcard is an output rather than an export, so the +// count sitting in the exported setCounts family is unreadable by the very grid that has to show it. +// +// Only the cell count. The other four set counts are identity counts, and putting them here would add four +// numeric columns to a card whose whole argument is that numbers do not belong in every position. +setCellCountImportSpec := func(setAxisSpec, served) { + return { + axes: [{ column: "setId", spec: setAxisSpec }], + columns: [{ + column: "cellCount", + id: "cellCount", + spec: { + name: "pl7.app/antigen/cellCount", + valueType: "Int", + domain: servedDomain(served), + annotations: a(96000, true, { + "pl7.app/label": "Cells", + "pl7.app/min": "0", + "pl7.app/description": "How many cells this clonotype has. A verdict resting on three cells and one resting on forty print the same word, so this is what tells them apart." + }) + } + }], + storageFormat: "Parquet", + partitionKeyLength: 0 + } +} + +// --- Per-identity summary: result_identity_summary.csv, keyed (setId) -------------------- +// +// The same verdicts pivoted onto the set axis alone, one column per identity, so a per-antigen state sits +// on the clonotype trunk where lead selection can see it. The identity travels in the DOMAIN, as +// pl7.app/feature/property does, so two identities are two distinct p-columns rather than one colliding +// name. +// +// Size-gated upstream: emit_verdicts.py writes only setId once the identity count passes its limit, and +// records the limit and whether it emitted in the run meta. Callers pass the identities the CSV actually +// carries, so an empty list yields no columns rather than a failed import. +// +// The label is the identity string itself. The readable name lives in result_identity_labels.csv, which is +// imported as a column rather than read as a value, so it is unavailable while these specs are built. + +// One column per identity, shared by every pivot whatever its rows are keyed on. Factored out rather than +// copied because the label rule below is the part that goes wrong quietly: a second copy that missed it +// would render a column of raw barcodes and look like panel data. +identityPivotColumns := func(identities, groupingId, served, colName, valueType, basePriority, idPrefix, visibility, extra) { + labels := is_undefined(served.identityLabels) ? {} : served.identityLabels + // Why a label is two names joined, for the identities where it is. The header alone shows a reader two + // antigen names in one column and leaves them to guess between three stories: the barcode is shared, the + // panel is inconsistent, or the block merged two antigens. Only the middle one is true. The description + // annotation is what PlAgDataTable renders as the header tooltip, so the explanation travels with the + // column into both punch faces rather than being written twice in the UI. + conflicts := is_undefined(served.identityNameConflicts) ? {} : served.identityNameConflicts + cols := [] + for i, identity in identities { + note := {} + if !is_undefined(conflicts[identity]) { + // One barcode, several names, and the identity is unaffected. The KEY is the barcode, and the + // readings under this column are that one barcode's. Said plainly, because the reader's first + // worry on meeting a joined header is that two antigens were silently pooled. + note = { "pl7.app/description": "The panel gives this one barcode more than one name — " + + text.join(conflicts[identity], ", ") + + ". Different samples name it differently, so every name it was given is shown. This is still a " + + "single barcode and a single identity; only the naming disagrees. Check the panel file if the " + + "names should have matched." } + } + cols = append(cols, { + column: identity, + id: idPrefix + strings.substituteSpecialCharacters(identity), + spec: { + name: colName, + valueType: valueType, + domain: maps.merge(servedDomain(served), { + "pl7.app/antigen/identityId": identity, + "pl7.app/antigen/groupingId": groupingId + }), + // `extra` last: a caller that sets a description for the whole family means it, and a + // per-identity note must not outrank it. + annotations: a(basePriority - i, visibility, maps.merge(maps.merge({ + "pl7.app/label": is_undefined(labels[identity]) ? identity : labels[identity] + }, note), extra)) + } + }) + } + return cols +} + +// Both identity pivots -- the state and the support behind it -- are keyed the same way and carry the same +// column order and labels, differing only in what a column holds. One builder produces both so the pair +// cannot drift: the punchcard pairs a state with its support by identity, and a divergence here would pair +// a state with another identity's support and signal nothing. +// +// The id prefix differs because both frames name their columns after the same identities and the punchcard +// frame holds BOTH at once. One prefix would give a state column and its support column the same id inside +// one frame, where one of the pair silently wins. +// +// `visibility` differs too, and it is the difference between a punchcard and an empty grid. The punch +// family is DEFAULT-visible: every antigen is a column of the card. The exported state family stays +// optional, because lead selection ranks on a few identities out of a panel that may run to hundreds. + +identityPivotImportSpec := func(setAxisSpec, identities, groupingId, served, colName, valueType, basePriority, idPrefix, visibility, extra) { + // The column header is the identity's LABEL where the run record carries one, and the identity itself + // otherwise. They differ for exactly one kind of identity: a tag whose rows disagreed about the grouping + // column has no value to group on and stands alone under its raw barcode, the least readable thing this + // can emit. The software labels those with the names they did declare, joined. The identity -- the KEY + // -- is untouched. Only what a reader sees changes. + return { + axes: [{ column: "setId", spec: setAxisSpec }], + columns: identityPivotColumns(identities, groupingId, served, colName, valueType, basePriority, idPrefix, visibility, extra), + storageFormat: "Parquet", + partitionKeyLength: 0 + } +} + + +// --- The punchcard's pivot: result_identity_punch.csv, keyed (setId) -------------------- +// +// One column per identity whose value carries the state AND everything behind it, as +// `state|answered|couldAnswer|agreement|reason`. The UI reads all five from the one cell: the state picks +// the punch's colour, the two counts its size, and all five its hover panel. `agreement` and `reason` are +// empty where they do not apply, since a settled verdict has no reason and a set nobody could ask has no +// agreement. The field count is therefore fixed at five, and a shorter value is not a valid punch. +// +// Compound rather than three columns because `support-travels-with-the-reading` requires both counts to +// travel with the verdict wherever it appears. A punch showing state alone would make a reading resting on +// three cells look like one resting on forty, and a grid can pair a cell with another column's cell only by +// position, which no import guarantees. Carrying it as an opaque String is also why this family is kept out +// of the exported frame: lead selection filters the state pivot, and a compound value would not. +identityPunchImportSpec := func(setAxisSpec, identities, groupingId, served) { + return identityPivotImportSpec(setAxisSpec, identities, groupingId, served, + "pl7.app/antigen/identityPunch", "String", 92000, "identityPunch_", true, {}) +} + +identitySummaryImportSpec := func(setAxisSpec, identities, groupingId, served) { + return identityPivotImportSpec(setAxisSpec, identities, groupingId, served, + "pl7.app/antigen/identityVerdict", "String", 92000, "identity_", false, { + "pl7.app/isDiscreteFilter": "true", + "pl7.app/discreteValues": VERDICT_STATES + }) +} + +// --- Re-derivation material ------------------------------------------------------------- +// +// The block emits no dense per-cell-per-identity table. On a realistic run it is the largest artifact the +// block would produce, and a pMHC panel does not fit at all. The sparse per-tag counts plus the per-cell +// scalars carry every per-cell state at a small fraction of that size. +// +// Both stay INSIDE the block, as outputs rather than exports. Reading the same experiment under another +// grouping is another execution of this block, not a re-derivation a reader performs, so no consumer across +// the boundary wants them: labelling and lead selection read verdicts, never cells. What makes re-execution +// cheap is where the grouping is consumed -- it enters after the counting, so a second run over unchanged +// reads and an unchanged panel file reuses the cached counts and pays for the verdicts alone. + +// The per-cell punchcard: one row per cell, one column per identity, that cell's own reading. +// +// Same identity columns as the set-level punch, from the same builder, so the two cards name and order +// their columns identically. A reader moving between the tabs is looking at one panel, and a column that +// moved between them would read as a different antigen. +// +// The value is `state|reason`, two fields where the set-level punch carries six. A verdict rests on counts +// a reader needs beside it. A cell IS the evidence, so there is nothing to report about how much of it +// there was. +// +// `setId` is a COLUMN, not an axis. The readout shows one clonotype at a time and filters on it, and a cell +// belongs to exactly one set, so this is a property of the row rather than part of its key. As an axis it +// would also have to agree with the clonotype axis the rest of the block uses, a domain-carrying axis this +// frame has no reason to reconstruct. +cellPunchImportSpec := func(sampleAxisSpec, cellAxisSpec, identities, groupingId, served) { + cols := [ + { + column: "setId", + id: "setId", + // Not default-visible: the readout filters on it, and a column repeating one clonotype id down + // every row of a panel about that clonotype is repetition the expansion already removed once. + spec: { + name: "pl7.app/antigen/cellSetId", + valueType: "String", + domain: servedDomain(served), + annotations: a(96000, false, { "pl7.app/label": "Clonotype key" }) + } + }, + { + column: "boundIdentities", + id: "boundIdentities", + spec: { + name: "pl7.app/antigen/boundIdentities", + valueType: "Int", + domain: servedDomain(served), + // Names its grain, because the clonotype-grain count carries the plain "Identities bound". The + // two are different numbers over different populations -- this one is one cell's, that one is + // a clonotype's majority across its cells -- and a reader meeting both under one name would + // take the smaller for a subset of the larger, which it is not. + annotations: a(95000, true, { + "pl7.app/label": "Identities this cell bound", + "pl7.app/description": "How many identities this cell read as bound, over the identities its sample was stained for. A silent position counts as the not-bound it resolves to; a cell that could not be compared counts none.", + "pl7.app/min": "0" + }) + } + } + ] + for col in identityPivotColumns(identities, groupingId, served, + "pl7.app/antigen/cellPunch", "String", 94000, "cellPunch_", true, {}) { + cols = append(cols, col) + } + return { + axes: [ + { column: "sampleId", spec: sampleAxisSpec }, + { column: "cellId", spec: cellAxisSpec } + ], + columns: cols, + storageFormat: "Parquet", + partitionKeyLength: 0 + } +} + +// result_cell_counts.csv, keyed (sampleId, cellId, tag). +// +// The CSV also repeats referenceCount and inCellList on every tag row. Those are per-CELL facts and are +// imported once, at their own grain, from result_cell_scalars.csv. Importing them here as well would put +// one fact in two columns at two different keys. +cellTagCountsImportSpec := func(sampleAxisSpec, cellAxisSpec, tagAxisSpec) { + return { + axes: [ + { column: "sampleId", spec: sampleAxisSpec }, + { column: "cellId", spec: cellAxisSpec }, + { column: "tag", spec: tagAxisSpec } + ], + columns: [{ + column: "umiCount", + id: "tagUmiCount", + spec: { + name: "pl7.app/antigen/umiCount", + valueType: "Int", + annotations: a(80000, true, { + "pl7.app/label": "Tag UMI count", + "pl7.app/description": "Molecules seen for this tag in this cell, after the count floor. Sparse: a cell with no row for a tag saw nothing for it, which is a reading, not a gap.", + "pl7.app/min": "0", + "pl7.app/isAbundance": "true", + "pl7.app/abundance/unit": "molecules", + "pl7.app/abundance/normalized": "false" + }) + } + }], + storageFormat: "Parquet", + partitionKeyLength: 0 + } +} + +// result_cell_scalars.csv, keyed (sampleId, cellId). +// +// The comparator is in the domain for the same reason it is on the verdicts: referenceCount and +// admissibility ARE the comparator's own output, so two runs served differently do not carry the same +// reading under one identity. +cellScalarsImportSpec := func(sampleAxisSpec, cellAxisSpec, served) { + scalarCol := func(colName, name, valueType, annotations) { + return { + column: colName, + id: colName, + spec: { + name: name, + valueType: valueType, + domain: servedDomain(served), + annotations: annotations + } + } + } + return { + axes: [ + { column: "sampleId", spec: sampleAxisSpec }, + { column: "cellId", spec: cellAxisSpec } + ], + columns: [ + scalarCol("referenceCount", "pl7.app/antigen/referenceCount", "Int", + a(79000, true, { + "pl7.app/label": "Baseline reading", + "pl7.app/description": "What this cell's reading was compared against. Empty where the run had no baseline for the cell.", + "pl7.app/min": "0" + })), + scalarCol("admissibility", "pl7.app/antigen/admissibility", "String", + a(78000, true, { + "pl7.app/label": "Admissibility", + "pl7.app/isDiscreteFilter": "true", + "pl7.app/discreteValues": ADMISSIBILITY_VALUES, + "pl7.app/description": "Whether this cell's readings could be compared at all, and if not, why. An inadmissible cell casts no vote; it does not vote 'not bound'." + })), + scalarCol("inCellList", "pl7.app/antigen/inCellList", "String", + a(77000, false, { + "pl7.app/label": "In cell list", + "pl7.app/isDiscreteFilter": "true", + "pl7.app/discreteValues": BOOL_VALUES + })) + ], + storageFormat: "Parquet", + partitionKeyLength: 0 + } +} + +// --- Scope: which identities each sample was stained with -------------------------------- +// +// result_offered.csv, keyed (sampleId, identity). The one piece of per-sample scope the block computes and +// would otherwise throw away: without it, "never asked" is a claim a reader cannot check. Sparse, so a row +// exists only where the sample's panel offered the identity. +// +// The value is a constant "true". A file of key columns alone imports as nothing, because columns are built +// from value columns, so the fact the frame records would never leave the block. +offeredImportSpec := func(sampleAxisSpec, identityAxisSpec) { + return { + axes: [ + { column: "sampleId", spec: sampleAxisSpec }, + { column: "identity", spec: identityAxisSpec } + ], + columns: [{ + column: "offered", + id: "offered", + spec: { + name: "pl7.app/antigen/offered", + valueType: "String", + annotations: a(76000, false, { + "pl7.app/label": "Offered", + "pl7.app/isDiscreteFilter": "true", + "pl7.app/discreteValues": BOOL_VALUES, + "pl7.app/description": "Present where this sample's panel declared this antigen. Absent means the sample was never stained with it." + }) + } + }], + storageFormat: "Parquet", + partitionKeyLength: 0 + } +} + +// --- tag -> identity linker: result_tag_identity.csv -------------------------------------- +// +// One row per (tag, identity), and MANY-TO-MANY: under per-sample grouping one barcode feeds a different +// identity in each sample's panel, and every pair is real. This is what lets a reader put a tag's count +// next to its verdict without either layer knowing about the other. The linker carries both axes and its +// value is irrelevant, so it is a constant column, matching pl7.app/sc/cellLinker's shape. Order priority 0 +// and no default visibility: it is infrastructure, hidden in tables. +// +// Deliberately NOT keyed by sample, and the reason is the join. Neither side carries a sample: verdicts are +// (set, identity) over clonotypes spanning whatever samples their cells came from, and the tag-keyed +// figures are run-level. A sample axis here is an axis no participating table has. It does not sharpen the +// join, it makes the join malformed, and SDK label discovery then refuses to build a spec frame over it. +// Where a sample-level distinction is wanted, panelMismatch keys by PANEL, a declared tag set. +// +// Several rows per tag are therefore expected, and `qc-measurement-set` asks for that shape. The SAME pair +// must not appear twice: duplicate axis keys break a grid silently, so the emitter writes distinct pairs. +// +// The value column is named "1" and holds 1, the platform's cell-linker convention, and exists for the same +// reason the offered frame's does: a key-only file imports as no column at all. +tagIdentityLinkerImportSpec := func(tagAxisSpec, identityAxisSpec) { + return { + axes: [ + { column: "tag", spec: tagAxisSpec }, + { column: "identity", spec: identityAxisSpec } + ], + columns: [{ + column: "1", + id: "tagIdentityLinker", + spec: { + name: "pl7.app/antigen/tagIdentityLinker", + valueType: "Int", + annotations: a(0, undefined, { + "pl7.app/label": "Tag / antigen linker", + "pl7.app/isLinkerColumn": "true" + }) + } + }], + storageFormat: "Parquet", + partitionKeyLength: 0 + } +} + +// --- Labels ------------------------------------------------------------------------------- +// +// A label column is satisfied only by a column NAMED pl7.app/label carrying exactly one axis. The consumers +// that matter do not read the pl7.app/isLabel annotation, so it cannot stand in for the name. Two +// identities must never share a label: where two tags carry the same consistent name, emit_verdicts.py +// appends the tag. That is where the name map lives and the only place the collision is visible. +// +// A label column REPLACES its axis in a data table: PlAgDataTableV2 drops any axis that has one, whatever +// the axis's own visibility says. A hidden label column therefore leaves nothing in the axis's place, and +// the rows keyed on it stop being distinguishable. Every label column below is visible for that reason. +identityLabelsImportSpec := func(identityAxisSpec) { + return { + axes: [{ column: "identity", spec: identityAxisSpec }], + columns: [{ + column: "label", + id: "identityLabel", + spec: { + name: "pl7.app/label", + valueType: "String", + annotations: a(0, true, { "pl7.app/label": "Antigen" }) + } + }], + storageFormat: "Parquet", + partitionKeyLength: 0 + } +} + +// --- the declarations, keyed by identity: result_identity_properties.csv ------------------- +// +// `panel-file-authority` requires whatever the panel file says consistently about an identity's tags to +// travel with that identity's verdicts, so a reader sees the declaration wherever the reading appears. +// These columns carry it, and they are EXPORTED: without them a downstream reader sees that an identity was +// bound and not what the scientist declared it to be, so "binds the target and nothing in the control set" +// cannot be stated at all. +// +// Identity follows featurePropertyImportColumns: one fixed name with the raw header in the DOMAIN, so two +// properties are two distinct p-columns rather than one that collides with itself. The names are panel +// data, unknown until the software runs, so they arrive through the run record as `identities` does. Each +// is String and a discrete filter over its own value set, which lets a reader pool every human antigen or +// every one in the control group. Optional visibility: a wide panel declares many properties and none of +// them is the reading. +// +// Keys on the antigen identity axis and never on pl7.app/feature/featureId. The per-feature property +// columns ride the feature-NAME axis, and the same declaration reached from a verdict has to arrive on the +// barcode-valued axis every verdict-stage column uses. guardNoLegacyFeatureAxis enforces it at import. +identityPropertyImportSpec := func(identityAxisSpec, propertyColumns, valuesByColumn) { + cols := [] + for i, name in propertyColumns { + distinct := is_undefined(valuesByColumn) || is_undefined(valuesByColumn[name]) ? [] : valuesByColumn[name] + cols = append(cols, { + column: name, + id: "identityProp_" + strings.substituteSpecialCharacters(name), + spec: { + name: "pl7.app/antigen/property", + valueType: "String", + domain: { "pl7.app/antigen/propertyName": name }, + annotations: a(91000 - i, false, { + "pl7.app/label": name, + "pl7.app/isDiscreteFilter": "true", + "pl7.app/discreteValues": string(json.encode(distinct)) + }) + } + }) + } + return { + axes: [{ column: "identity", spec: identityAxisSpec }], + columns: cols, + storageFormat: "Parquet", + partitionKeyLength: 0 + } +} + +// A panel has no name in any real panel file, so its label is built from what a reader can recognise it by: +// how many tags it declares, and which samples came off it. +panelLabelsImportSpec := func(panelAxisSpec) { + return { + axes: [{ column: "panelId", spec: panelAxisSpec }], + columns: [{ + column: "label", + id: "panelLabel", + spec: { + name: "pl7.app/label", + valueType: "String", + annotations: a(0, true, { "pl7.app/label": "Panel" }) + } + }], + storageFormat: "Parquet", + partitionKeyLength: 0 + } +} + +// --- sample -> panel: result_sample_panel.csv ---------------------------------------------- +// +// Keyed [sampleId] with the panel as the value, so per-tag QC keyed (panel, tag) can be read back to the +// samples it covers. Where one panel covers every sample this column is constant and drops out. +samplePanelImportSpec := func(sampleAxisSpec) { + return { + axes: [{ column: "sampleId", spec: sampleAxisSpec }], + columns: [{ + column: "panelId", + id: "panelOfSample", + spec: { + name: "pl7.app/antigen/panelOfSample", + valueType: "String", + // Says WHICH panel a sample was stained with. The panel axis's own label column is the other + // "Panel", and it names the panel itself, so one word for both would put two different things + // beside each other under one heading. + annotations: a(75000, false, { + "pl7.app/label": "Panel used", + "pl7.app/isDiscreteFilter": "true" + }) + } + }], + storageFormat: "Parquet", + partitionKeyLength: 0 + } +} + +// --- Panel versus reads: result_panel_mismatch.csv ------------------------------------------ +// +// Keyed (panelId, tag), because a per-tag failure is a property of the declared tag set rather than of any +// one sample carrying it. The samples that reported it travel in the row, so nothing about where it was +// seen is lost. Emitted as a p-column rather than a raw file: a mismatch report the user cannot see defeats +// its purpose. +panelMismatchImportSpec := func(panelAxisSpec, tagAxisSpec) { + return { + axes: [ + { column: "panelId", spec: panelAxisSpec }, + { column: "tag", spec: tagAxisSpec } + ], + columns: [ + { + column: "direction", + id: "mismatchDirection", + spec: { + name: "pl7.app/antigen/panelMismatchDirection", + valueType: "String", + annotations: a(74000, true, { + "pl7.app/label": "Mismatch", + "pl7.app/isDiscreteFilter": "true", + "pl7.app/discreteValues": MISMATCH_DIRECTIONS, + "pl7.app/description": "'declared-never-seen': the panel declares this tag and no read carried it. 'undeclared-in-panel': reads carried a tag the panel never declared. Both are checked because either alone hides half the mismatch." + }) + } + }, + { + column: "samples", + id: "mismatchSamples", + spec: { + name: "pl7.app/antigen/panelMismatchSamples", + valueType: "String", + annotations: a(73000, true, { "pl7.app/label": "Samples affected" }) + } + } + ], + storageFormat: "Parquet", + partitionKeyLength: 0 + } +} + +// --- Undeclared barcodes: result_undeclared_barcodes.csv, keyed (sampleId, tag) ------------ +// +// 330: barcodes the reads carried that no panel declares get their own table, keyed by sequence, +// because they have no row above -- they are not in the panel. It is the one thing on this +// surface that carries a status: the field publishes a line for the share of a sample's reads +// landing in undeclared barcodes. 310 keeps that status the barcode's and never the sample's, +// so it sits here rather than on any row of qcImportSpec's per-sample measurement list. +// +// Usually this table has no row for a sample at all, and that is the outcome the field wants -- +// the description below says so, since an empty grid otherwise reads as a check that never ran. +undeclaredBarcodeImportSpec := func(sampleAxisSpec, tagAxisSpec) { + return { + axes: [ + { column: "sampleId", spec: sampleAxisSpec }, + { column: "tag", spec: tagAxisSpec } + ], + columns: [ + { + column: "totalWeight", + id: "undeclaredBarcodeWeight", + spec: { + name: "pl7.app/antigen/undeclaredBarcodeWeight", + valueType: "Int", + annotations: a(79000, true, { + "pl7.app/label": "Reads", + "pl7.app/min": "0", + "pl7.app/description": "Reads carrying this sequence, from the pre-refine pass -- before refine-tags would have snapped or dropped it." + }) + } + }, + { + column: "readShare", + id: "undeclaredBarcodeShare", + spec: { + name: "pl7.app/antigen/undeclaredBarcodeShare", + valueType: "Double", + annotations: a(78000, true, { + "pl7.app/label": "Share of the sample's reads", + "pl7.app/format": ".2p", + "pl7.app/min": "0", + "pl7.app/max": "1", + "pl7.app/description": "The share of this SAMPLE's reads landing in barcodes nobody declared, repeated on every one of that sample's rows here. Not this one sequence's own share." + }) + } + }, + { + column: "status", + id: "undeclaredBarcodeStatus", + spec: { + name: "pl7.app/antigen/undeclaredBarcodeStatus", + valueType: "String", + annotations: a(77000, true, { + "pl7.app/label": "Status", + "pl7.app/isDiscreteFilter": "true", + "pl7.app/discreteValues": QC_STATUSES, + "pl7.app/description": "Warns above a 0.50 share and alerts at 1.0 (inherited from the field). This status is the barcode's, never a sample's: it never rolls into any sample's own status. No row here is the outcome to want -- an empty table means every barcode the reads carried was declared." + }) + } + } + ], + storageFormat: "Parquet", + partitionKeyLength: 0 + } +} + +// --- Quality measurements: result_qc.csv, keyed (level, entity, measurement) ------------------ +// +// Every declared measurement keeps its place whether or not this run could compute it. A measurement +// nothing computed reads "not evaluated" with its reason rather than being absent, so a reader can never +// mistake "nothing computed this yet" for "this was checked and found fine". +// +// The panel is part of the KEY, never a value column beside it. A barcode recurring across panels is the +// ordinary case, since the same reagent is stained into several samples, and it writes one row per panel at +// the same (level, entity, measurement). With the panel as a value column those rows share an axis key, +// which is a duplicate-key import: it does not raise, it silently keeps one row and loses the rest, so a +// reagent misbehaving in one panel and not another reads as whichever row survived. +// +// A row that belongs to no panel -- every sample-level and capture-level row -- leaves `panelId` empty, +// which is honest: a per-sample measurement belongs to no single panel. +// --- The distributions a scientist reads in order to settle a number ------------------------- +// +// 330-the-quality-readout puts three distributions last, and two of them exist so a scientist can +// place the cutoff and the gate. That makes them plots rather than table rows: a decile encoded in a +// measurement's detail string is a number nobody can draw. +// +// Both decile sets are taken over the WHOLE run and share one frame, told apart by the +// `distribution` axis. The cutoff is one number for the run and so is the gate, so each plot has to +// show every cell the number will act on. Pooling the reference reading over samples is 330's own +// instruction, not a simplification. +qcDistributionAxis := func(blockId) { + return { + name: "pl7.app/antigen/qcDistribution", + type: "String", + domain: { "pl7.app/blockId": blockId }, + annotations: { + "pl7.app/label": "Distribution", + "pl7.app/isDiscreteFilter": "true", + "pl7.app/discreteValues": string(json.encode(["score", "referenceReading"])) + } + } +} + +qcDecileAxis := func(blockId) { + return { + name: "pl7.app/antigen/qcDecile", + type: "Int", + domain: { "pl7.app/blockId": blockId }, + annotations: { "pl7.app/label": "Percentile" } + } +} + +qcDecileImportSpec := func(distributionAxisSpec, decileAxisSpec) { + return { + axes: [ + { column: "distribution", spec: distributionAxisSpec }, + { column: "decile", spec: decileAxisSpec } + ], + columns: [{ + column: "value", id: "value", + spec: { + name: "pl7.app/antigen/qcDecileValue", + valueType: "Double", + annotations: a(90000, true, guardNoScore({ + "pl7.app/label": "Value", + "pl7.app/description": "The score at this percentile of the run's scores, or the reference reading at this percentile of the run's readings. Which one a row is about is on the Distribution axis." + })) + } + }], + storageFormat: "Parquet", + partitionKeyLength: 0 + } +} + +// The same shape as qcDecileImportSpec, kept PER SAMPLE: 330 reads the antigen-count distribution as +// one sample's own plot, never pooled across samples the way the score and the reference reading are. +// A separate column (`qcSampleDecileValue`), because adding a sample axis to `qcDecileValue` would +// change that column's identity rather than add to it -- score and referenceReading stay exactly as +// they were. +qcSampleDecileImportSpec := func(sampleAxisSpec, decileAxisSpec) { + return { + axes: [ + { column: "sampleId", spec: sampleAxisSpec }, + { column: "decile", spec: decileAxisSpec } + ], + columns: [{ + column: "value", id: "value", + spec: { + name: "pl7.app/antigen/qcSampleDecileValue", + valueType: "Double", + annotations: a(90000, true, guardNoScore({ + "pl7.app/label": "Value", + "pl7.app/description": "The total antigen count per cell barcode at this percentile, for this sample alone." + })) + } + }], + storageFormat: "Parquet", + partitionKeyLength: 0 + } +} + +// One row per (sample, tag) a population baseline fitted, at the fit's OWN grain. Aggregating to the +// tag would hide a reagent that separated in one sample and not in another, which is the comparison +// this plot exists for. The two means are read together: a background alone says nothing about +// whether the counts separated. +qcBackgroundImportSpec := func(sampleAxisSpec, tagAxisSpec) { + num := func(colName, name, label, order, description) { + return { + column: colName, id: colName, + spec: { + name: name, + valueType: "Double", + annotations: a(order, true, guardNoScore({ + "pl7.app/label": label, + "pl7.app/min": "0", + "pl7.app/description": description + })) + } + } + } + return { + axes: [ + { column: "sampleId", spec: sampleAxisSpec }, + { column: "tag", spec: tagAxisSpec } + ], + columns: [ + num("backgroundMean", "pl7.app/antigen/fittedBackgroundMean", "Fitted background", 90000, + "Mean count of the background component fitted to this tag in this sample."), + num("signalMean", "pl7.app/antigen/fittedSignalMean", "Fitted signal", 89000, + "Mean count of the signal component. Read beside the background: two means close together are a tag whose counts did not separate."), + num("backgroundWeight", "pl7.app/antigen/fittedBackgroundWeight", "Background share", 88000, + "Share of the sample's cells the background component holds.") + ], + storageFormat: "Parquet", + partitionKeyLength: 0 + } +} + +qcImportSpec := func(levelAxisSpec, entityAxisSpec, measurementAxisSpec, panelAxisSpec) { + col := func(colName, name, valueType, label, order, visible, annotations) { + return { + column: colName, + id: colName, + spec: { + name: name, + valueType: valueType, + annotations: a(order, visible, maps.merge(annotations, { "pl7.app/label": label })) + } + } + } + return { + axes: [ + { column: "level", spec: levelAxisSpec }, + { column: "panelId", spec: panelAxisSpec }, + { column: "entity", spec: entityAxisSpec }, + { column: "measurement", spec: measurementAxisSpec } + ], + columns: [ + // The measurement's readable name, ordered ahead of the value so a row reads as a sentence: what + // was measured, then what it came to. The `measurement` AXIS carries the id, which must stay + // stable -- not because it is column identity, which it is not, but because renaming it splits + // the rows: old runs carry one name and new runs another, and a table holding both reads as two + // measurements. This carries the name a reader recognises. Without it the page shows a scientist + // `antigenCountDistribution`. + col("label", "pl7.app/antigen/qcLabel", "String", "Measurement name", 73000, true, {}), + col("value", "pl7.app/antigen/qcValue", "Double", "Value", 72000, true, {}), + col("detail", "pl7.app/antigen/qcDetail", "String", "Detail", 71000, true, {}), + col("status", "pl7.app/antigen/qcStatus", "String", "Status", 70000, true, { + "pl7.app/isDiscreteFilter": "true", + "pl7.app/discreteValues": QC_STATUSES, + "pl7.app/description": "Empty means no line stands behind this measurement, which is not a pass. Its value says which: a number was computed and nothing judges it, or the reason there is no number." + }), + // The coverage triple is shown by default, beside the status rather than folded into it. A status + // says whether what was checked is wrong, and the triple says how much of the level was checked at + // all. Hiding the triple behind the column chooser would leave "nothing here is wrong" and "almost + // nothing here was checkable" looking identical on the page. + col("judged", "pl7.app/antigen/qcJudged", "Int", "Judged", 69000, true, { "pl7.app/min": "0" }), + col("unjudged", "pl7.app/antigen/qcUnjudged", "Int", "Unjudged", 68000, true, { "pl7.app/min": "0" }), + col("notEvaluated", "pl7.app/antigen/qcNotEvaluated", "Int", "Not evaluated", 67000, true, { "pl7.app/min": "0" }), + // Shown by default, both of them. Every measurement is obliged to carry what it counts and, where a + // line exists, what a bad value means, because the reader who meets a row is not the person who + // built the run. Behind the column chooser they satisfy the obligation in the data and defeat it on + // the page: a reader meets `antigenCountDistribution 637.5 0:13.0|10:75.4|...` with no way to learn + // that those are deciles of the total antigen count per cell barcode. + col("counts", "pl7.app/antigen/qcCounts", "String", "What it counts", 66000, true, {}), + col("implies", "pl7.app/antigen/qcImplies", "String", "What a bad value means", 65000, true, {}), + // The line itself, so a reader who meets `warn` can see what it warned against. Empty where + // no line stands behind the measurement; the categorical route (`cellsDetected`) has a route + // but no numeric threshold, so both stay empty there too. + col("lineWarn", "pl7.app/antigen/qcLineWarn", "Double", "Warn threshold", 63500, true, {}), + col("lineAlert", "pl7.app/antigen/qcLineAlert", "Double", "Alert threshold", 63400, true, {}), + col("route", "pl7.app/antigen/qcRoute", "String", "Line's defence route", 63300, true, { + "pl7.app/isDiscreteFilter": "true", + "pl7.app/discreteValues": QC_ROUTES, + "pl7.app/description": "Which of the three routes backs this measurement's line: a published number, a categorical fact, or a stated recommendation. Empty where no line stands behind it." + }), + // Also shown by default. A row nothing computed carries the reason in place of a value, and + // a deferral whose reason is one click away is a blank cell to everyone who does not click. That + // reads as "checked, and fine", the one thing this vocabulary exists to prevent. + col("reason", "pl7.app/antigen/qcReason", "String", "Why deferred", 64000, true, {}) + ], + storageFormat: "Parquet", + partitionKeyLength: 0 + } +} + +// One row per (panelId, tag, identity). A barcode reused for a different antigen takes a row under each +// identity, so per-tag figures repeat across a tag's rows. No status column: every figure here is read +// against the same figure on the tags beside it. +qcReagentImportSpec := func(panelAxisSpec, tagAxisSpec, identityAxisSpec) { + col := func(colName, name, valueType, label, order, visible, description) { + return { + column: colName, id: colName, + spec: { + name: name, + valueType: valueType, + annotations: a(order, visible, { + "pl7.app/label": label, + "pl7.app/description": description + }) + } + } + } + return { + axes: [ + { column: "panelId", spec: panelAxisSpec }, + { column: "tag", spec: tagAxisSpec }, + { column: "identity", spec: identityAxisSpec } + ], + columns: [ + col("samplesSeenIn", "pl7.app/antigen/reagentSamplesSeenIn", "Int", "Seen in", 79000, true, + "Samples of this panel carrying any count of the barcode. Zero means no read carried it."), + col("samplesInPanel", "pl7.app/antigen/reagentSamplesInPanel", "Int", "Samples in panel", 78500, false, + "The denominator for Seen in: the samples this panel declares."), + col("samplesSeenInNames", "pl7.app/antigen/reagentSamplesSeenInNames", "String", "Seen in (samples)", 78900, true, + "Samples of this panel carrying any count of the barcode, named. Empty means none did."), + col("samplesInPanelNames", "pl7.app/antigen/reagentSamplesInPanelNames", "String", "Declared in (samples)", 78700, true, + "The samples this panel declares the tag for, named. Reading it apart from Seen in (samples) tells a sample whose panel never declared the tag from one that declared it and saw nothing."), + col("cellsWithCount", "pl7.app/antigen/reagentCellsWithCount", "Int", "Cells with any count", 78000, true, + "Cells holding any count of the barcode, taken before the minimum."), + col("cellsAboveTheLine", "pl7.app/antigen/reagentCellsAboveTheLine", "Double", "Cells called bound", 77000, true, + "Cells the verdict read called bound. Empty for a tag supplying the baseline; the reason column names that case."), + col("medianCountPerCell", "pl7.app/antigen/reagentMedianCount", "Double", "Median count per cell", 76000, true, + "Median count over the cells holding any count, taken before the minimum."), + col("siblingDisagreement", "pl7.app/antigen/reagentSiblingDisagreement", "Double", "Disagrees with siblings", 75000, true, + "Rate at which this tag disagrees with the other tags of its identity. Empty where no sibling comparison exists."), + col("selfDisagreement", "pl7.app/antigen/reagentSelfDisagreement", "Double", "Self-disagreement", 74500, true, + "Rate at which the cells of one set contradict each other on this tag."), + col("reason", "pl7.app/antigen/reagentReason", "String", "Why a figure is absent", 74400, true, + "Names each figure with no value and why. Empty when every figure has one.") + ], + storageFormat: "Parquet", + partitionKeyLength: 0 + } +} + +// qcSampleSummaryImportSpec: the xsv.importFile spec for result_qc_summary.csv (emit_verdicts.py +// `sample_summary_rows`) -- one row per sample, one column per sample-level measurement, plus the +// sample's own rolled-up status. 330-the-quality-readout's second surface: "the same measurements +// as columns, one row per sample, carrying the sample's rolled-up status". +// +// `panelAssignedFraction` and `cellBarcodeValidFraction` keep the exact ids and column names the +// pre-330 table shipped: both are measurement-axis values elsewhere in this run's own output +// (result_qc.csv), and renaming the p-column here would split one measurement's rows across two +// names. `readsTotal` keeps its id for the same reason. The other mitool read-QC figures +// (`readsMatched`, `matchedFraction`, `cellsDetected`, `featuresDetected`, `totalUniqueUmis`, +// `medianUmisPerCell`) carry no declared measurement and are unaffected by that rule. +// +// Every description below is copied verbatim from qc_measures.py `Measurement.counts` for the +// matching id; the two are not read from one shared definition. +qcSampleSummaryImportSpec := func(sampleAxisSpec) { + num := func(id, colName, label, order, fmt, description) { + spec := { + name: "pl7.app/qc/" + id, + valueType: "Double", + annotations: a(order, true, { "pl7.app/label": label, "pl7.app/description": description }) + } + if !is_undefined(fmt) { + spec.annotations["pl7.app/format"] = fmt + } + return { column: colName, id: id, spec: spec } + } + return { + axes: [{ column: "sampleId", spec: sampleAxisSpec }], + columns: [ + { + column: "status", id: "status", + spec: { + name: "pl7.app/qc/sampleStatus", + valueType: "String", + annotations: a(96000, true, { + "pl7.app/label": "Status", + "pl7.app/isDiscreteFilter": "true", + "pl7.app/discreteValues": QC_STATUSES, + "pl7.app/description": "The worst status among this sample's own measurements that carry one, from result_qc.csv's rollup row. Empty where nothing at this sample carried a status." + }) + } + }, + num("readsTotal", "readsTotal", "Reads parsed", 95000, undefined, + "Every read the parser saw, and the share matching the tag pattern."), + num("readsMatched", "readsMatched", "Reads matched", 94000, undefined, undefined), + num("matchedFraction", "matchedFraction", "Matched fraction", 93000, ".2p", undefined), + num("cellsDetected", "cellsDetected", "Cells detected", 92000, undefined, undefined), + num("featuresDetected", "featuresDetected", "Features detected", 91000, undefined, undefined), + num("totalUniqueUmis", "totalUniqueUmis", "Total distinct UMIs", 90000, undefined, undefined), + num("medianUmisPerCell", "medianUmisPerCell", "Median UMIs / cell", 89000, ".1f", undefined), + num("panelAssignedFraction", "panelAssignedFraction", "Panel-assigned fraction", 88000, ".2p", + "Reads whose corrected barcode is on the panel, over reads matched."), + num("cellBarcodeValidFraction", "cellBarcodeValidFraction", "Valid cell-barcode fraction", 87000, ".2p", + "Reads whose cell barcode corrects onto the chemistry's whitelist, over reads entering correction."), + num("usableReadFraction", "usableReadFraction", "Usable antigen-read fraction", 86000, ".2p", + "Reads matched that also carry a cell-associated barcode and a valid UMI, over reads matched."), + num("readsPerCell", "readsPerCell", "Reads per cell", 85000, ".1f", + "Reads matched, over cells in the cell list."), + num("antigenCountDistribution", "antigenCountDistribution", "Median antigen count per cell", 84000, ".1f", + "Deciles of the total antigen count per cell barcode."), + num("aggregateBarcodeFraction", "aggregateBarcodeFraction", "Aggregate-barcode read fraction", 83000, ".2p", + "Reads in barcodes flagged as aggregates, over reads matched."), + num("floorRemoved", "floorRemoved", "Counts removed by the minimum", 82000, undefined, + "Readings the minimum zeroed, and cells whose every non-reference reading was removed."), + num("uniqueCountsPerCell", "uniqueCountsPerCell", "Unique counts per cell", 81000, ".1f", + "Reads and distinct UMIs per cell barcode."), + num("highReferenceCells", "highReferenceCells", "Sticky reference cells", 80000, undefined, + "Cells whose reference reading reached the declared gate, or the spread of those readings.") ], storageFormat: "Parquet", partitionKeyLength: 0 @@ -361,5 +1617,41 @@ export { perCellSummaryOutput: perCellSummaryOutput, qcFileMapOutput: qcFileMapOutput, qcSummaryColumnsSpec: qcSummaryColumnsSpec, - featurePropertyImportColumns: featurePropertyImportColumns + featurePropertyImportColumns: featurePropertyImportColumns, + negativeControlColumn: negativeControlColumn, + + tagAxis: tagAxis, + identityAxis: identityAxis, + panelAxis: panelAxis, + captureAxis: captureAxis, + qcLevelAxis: qcLevelAxis, + qcEntityAxis: qcEntityAxis, + qcMeasurementAxis: qcMeasurementAxis, + + verdictsImportSpec: verdictsImportSpec, + setCountsImportSpec: setCountsImportSpec, + setCellCountImportSpec: setCellCountImportSpec, + identitySummaryImportSpec: identitySummaryImportSpec, + identityPunchImportSpec: identityPunchImportSpec, + cellPunchImportSpec: cellPunchImportSpec, + cellTagCountsImportSpec: cellTagCountsImportSpec, + cellScalarsImportSpec: cellScalarsImportSpec, + offeredImportSpec: offeredImportSpec, + tagIdentityLinkerImportSpec: tagIdentityLinkerImportSpec, + identityLabelsImportSpec: identityLabelsImportSpec, + identityPropertyImportSpec: identityPropertyImportSpec, + panelLabelsImportSpec: panelLabelsImportSpec, + samplePanelImportSpec: samplePanelImportSpec, + panelMismatchImportSpec: panelMismatchImportSpec, + undeclaredBarcodeImportSpec: undeclaredBarcodeImportSpec, + qcImportSpec: qcImportSpec, + qcDistributionAxis: qcDistributionAxis, + qcDecileAxis: qcDecileAxis, + qcDecileImportSpec: qcDecileImportSpec, + qcSampleDecileImportSpec: qcSampleDecileImportSpec, + qcBackgroundImportSpec: qcBackgroundImportSpec, + qcReagentImportSpec: qcReagentImportSpec, + qcSampleSummaryImportSpec: qcSampleSummaryImportSpec, + guardNoLegacyFeatureAxis: guardNoLegacyFeatureAxis, + usesLegacyFeatureAxis: usesLegacyFeatureAxis } diff --git a/workflow/src/fanout-inputs.lib.tengo b/workflow/src/fanout-inputs.lib.tengo new file mode 100644 index 0000000..ee0a90c --- /dev/null +++ b/workflow/src/fanout-inputs.lib.tengo @@ -0,0 +1,71 @@ +// What may reach the per-sample mitool fan-out, as a closed list. +// +// The block runs in two stages, and the whole value of the split is that the expensive stage does not notice +// the cheap one. Where only the reading's parameters change -- a different grouping, a different floor -- the +// body re-executes and processColumn is re-declared with identical inputs, so every per-sample mitool body +// is recovered from cache and only the verdict stage re-runs. That holds exactly as long as no verdict-stage +// parameter and no resolved clonotype linker enters `extra` or `metaExtra`. Both are part of each per-sample +// body's identity, so one added key costs every user a full re-run of parse, refine-tags and tag-stat, and +// neither the render nor the logs says why. +// +// The extra-input block in main.tpl.tengo is where a convenience addition lands, so the list is closed here +// and checked at render time rather than reviewed later. + +maps := import("@platforma-sdk/workflow-tengo:maps") + +// Every key main.tpl.tengo may pass as a per-sample body input. All of these describe how one sample's reads +// are parsed and counted. None describes how the counts are read. +EXTRA_INPUT_KEYS := [ + "aggregateBarcodeIqrMultiplier", + "aggregateBarcodeMinUmiThreshold", + "aggregateBarcodeTopN", + "barcodeSeqColumn", + "cellWhitelist", + "combineColumn", + "featureNameColumn", + "fileExtension", + "limitInput", + "minUmi", + "pattern", + "sampleColumn", + "sampleLabels", + "tags", + "tagsCsv" +] + +// The per-sample resource meta. A distinct meta where the user overrides RAM is correct, because an +// overridden run must not dedup with the formula-sized one. Nothing else belongs here. +META_EXTRA_KEYS := [ + "mitoolBaseMemGB", + "mitoolCPUs", + "mitoolMemOverrideGB" +] + +_contains := func(list, value) { + for item in list { + if item == value { + return true + } + } + return false +} + +// unlisted: the keys of `m` that the allowlist does not cover, sorted. +// +// Iterated with maps.forEach, whose key order is sorted, so the message a render fails with is the same on +// every run rather than depending on map layout. +unlisted := func(m, allowed) { + offenders := [] + maps.forEach(m, func(key, _) { + if !_contains(allowed, key) { + offenders = append(offenders, key) + } + }) + return offenders +} + +export { + EXTRA_INPUT_KEYS: EXTRA_INPUT_KEYS, + META_EXTRA_KEYS: META_EXTRA_KEYS, + unlisted: unlisted +} diff --git a/workflow/src/fanout-inputs.test.tengo b/workflow/src/fanout-inputs.test.tengo new file mode 100644 index 0000000..2453c70 --- /dev/null +++ b/workflow/src/fanout-inputs.test.tengo @@ -0,0 +1,60 @@ +test := import("@platforma-sdk/workflow-tengo:test") +fanout := import(":fanout-inputs") +va := import(":verdict-args") + +_one := func(key) { + m := {} + m[key] = "x" + return m +} + +// The whole point of the two-stage split: changing the reading must not re-run the per-sample mitool +// chain. Every verdict-stage parameter is checked against both allowlists, so adding one to +// verdict-args.PARAMETER_NAMES and to the fan-out at once fails here. Otherwise it costs every user a +// full parse, refine-tags and tag-stat re-run with nothing in the logs to explain it. +Test_no_verdict_parameter_may_reach_the_fanout := func() { + for name in va.PARAMETER_NAMES { + test.isEqual([name], fanout.unlisted(_one(name), fanout.EXTRA_INPUT_KEYS), + "verdict-stage parameter " + name + " must not be a per-sample body input") + test.isEqual([name], fanout.unlisted(_one(name), fanout.META_EXTRA_KEYS), + "verdict-stage parameter " + name + " must not be part of the per-sample resource meta") + } +} + +// The resolved clonotype linker is a resource rather than a parameter, and it is the other thing that must +// never enter. It would re-key every per-sample body and put it behind the clonotyping chain. +Test_no_linker_resource_may_reach_the_fanout := func() { + for name in ["linker", "linkerColumn", "cellLinker", "columns", "setAxisSpec"] { + test.isEqual([name], fanout.unlisted(_one(name), fanout.EXTRA_INPUT_KEYS), + name + " must not be a per-sample body input") + test.isEqual([name], fanout.unlisted(_one(name), fanout.META_EXTRA_KEYS), + name + " must not be part of the per-sample resource meta") + } +} + +// The allowlist accepts exactly what main.tpl.tengo passes, so an over-broad list cannot satisfy the +// render-time check vacuously. +Test_the_declared_fanout_inputs_are_accepted := func() { + extra := {} + for key in fanout.EXTRA_INPUT_KEYS { + extra[key] = "x" + } + test.isEqual([], fanout.unlisted(extra, fanout.EXTRA_INPUT_KEYS), + "every declared per-sample body input is accepted") + + meta := {} + for key in fanout.META_EXTRA_KEYS { + meta[key] = 1 + } + test.isEqual([], fanout.unlisted(meta, fanout.META_EXTRA_KEYS), + "every declared resource-meta key is accepted") +} + +// Offenders come back sorted, so a failing render reports the same message every time. +Test_offenders_are_reported_sorted := func() { + m := {} + m.zzz = 1 + m.aaa = 2 + test.isEqual(["aaa", "zzz"], fanout.unlisted(m, fanout.EXTRA_INPUT_KEYS), + "unlisted keys are reported in sorted order") +} diff --git a/workflow/src/fb-downstream.tpl.tengo b/workflow/src/fb-downstream.tpl.tengo index 42d7e69..cddf583 100644 --- a/workflow/src/fb-downstream.tpl.tengo +++ b/workflow/src/fb-downstream.tpl.tengo @@ -1,16 +1,23 @@ // Downstream per-sample QC + per-cell metrics sub-template (render.create boundary). // -// The final consumers of the mitool chain: the QC-report exec (parsed/matched + cell/feature/UMI + -// panel-assigned fraction) and the per-cell-metrics Python (collapse to the per-cell (cell x feature) -// outputs). They consume tagstat.tsv (fb-tagstat) plus the parse/refine JSON reports (fb-parse / -// fb-refine). Running them inside this render.create boundary returns their per-sample outputs as render -// outputs that flatten cleanly. Mirrors blocks/peptide-extraction downstream-pipeline.tpl.tengo. +// The final consumers of the mitool chain: the QC-report exec, giving parsed and matched reads plus the +// cell, feature and UMI metrics and the panel-assigned fraction, and the per-cell-metrics Python, which +// collapses to the per-cell (cell x feature) outputs. Both consume tagstat.tsv from fb-tagstat plus the +// parse and refine JSON reports. Running them inside this render.create boundary returns their per-sample +// outputs as render outputs that flatten cleanly. Mirrors blocks/peptide-extraction +// downstream-pipeline.tpl.tengo. +// +// Nothing here knows about the binding verdicts. The verdict stage reads the run as a whole, because a panel +// is a property of a declared tag set rather than of one sample and the quality rollup spans samples, so it +// runs once after this fan-out on the gathered counts. Keeping every one of its parameters out of this +// template is what lets a change to the reading recover each per-sample body from cache instead of +// re-running parse, refine-tags and tag-stat. self := import("@platforma-sdk/workflow-tengo:tpl") exec := import("@platforma-sdk/workflow-tengo:exec") assets := import("@platforma-sdk/workflow-tengo:assets") -self.defineOutputs("abundance", "fractions", "consensus", "specificity", "perCellSummary", "qc", "qcJson", "metricsLogStream") +self.defineOutputs("abundance", "fractions", "perCellSummary", "qc", "qcJson", "metricsLogStream") metricsSw := assets.importSoftware("@platforma-open/milaboratories.feature-integration.per-cell-metrics:main") qcReportSw := assets.importSoftware("@platforma-open/milaboratories.feature-integration.per-cell-metrics:qc-report") @@ -21,28 +28,20 @@ self.body(func(inputs) { refineReport := inputs.refineReport tags := inputs.tags // { cell, umi, feature } tag names (from the model; see pattern.ts) tagsCsv := inputs.tagsCsv - control := inputs.control // undefined / "" -> no specificity score (render.create skips undefined) - dominanceThreshold := inputs.dominanceThreshold barcodeSeqColumn := inputs.barcodeSeqColumn featureNameColumn := inputs.featureNameColumn - // Optional multi-barcode antigen combine mode. "" = off (per_cell_metrics defaults every feature to - // OR/sum). When set, it names the tag-CSV column giving each feature's mode (sum | all); minUmi is the + // Optional multi-barcode antigen combine mode. "" = off, and per_cell_metrics then defaults every feature + // to OR/sum. Where set, it names the tag-CSV column giving each feature's mode, sum or all. minUmi is the // AND per-barcode "fired" floor. combineColumn := is_undefined(inputs.combineColumn) ? "" : inputs.combineColumn minUmi := is_undefined(inputs.minUmi) ? 1 : inputs.minUmi - // Optional off-target designation (F2). offtargetColumn names an imported per-feature property column - // (e.g. antigen_class); offtargetValues is the comma-separated set of its values marking a feature as - // off-target. Both empty = off (unchanged dominant call). Features so marked are excluded from the - // dominant call (like the control) and turn on the cross-reactive label. - offtargetColumn := is_undefined(inputs.offtargetColumn) ? "" : inputs.offtargetColumn - offtargetValues := is_undefined(inputs.offtargetValues) ? "" : inputs.offtargetValues - // Per-sample QC summary: reads parsed/matched (parse JSON report) + cell/feature/UMI metrics - // (tag-stat) + best-effort panel-assigned fraction (refine JSON report). + // Per-sample QC summary: reads parsed and matched from the parse JSON report, the cell, feature and UMI + // metrics from tag-stat, and a best-effort panel-assigned fraction from the refine JSON report. qc := exec.builder(). software(qcReportSw). cpu(1).mem("2GiB"). - // polars sizes its pool to all host cores by default; cap it to the granted CPU. + // polars sizes its pool to all host cores by default. Cap it to the granted CPU. envWithVar("POLARS_MAX_THREADS", "{system.cpu}"). addFile("tagstat.tsv", tagstatTsv). addFile("parse_report.json", parseReport). @@ -53,15 +52,26 @@ self.body(func(inputs) { arg("--sample-id").arg("sample"). arg("--cell-col").arg(tags.cell). arg("--feature-col").arg(tags.feature). - arg("--umi-col").arg("unique_" + tags.umi). - arg("--output").arg("result_qc.csv").saveFile("result_qc.csv"). + arg("--umi-col").arg("unique_" + tags.umi) + // The aggregate-barcode detection knobs. Added only where set, so the exec's own argparse default + // (qc_measures.py AGGREGATE_BARCODE_*) applies otherwise. + if !is_undefined(inputs.aggregateBarcodeIqrMultiplier) { + qc = qc.arg("--aggregate-iqr-multiplier").arg(string(inputs.aggregateBarcodeIqrMultiplier)) + } + if !is_undefined(inputs.aggregateBarcodeMinUmiThreshold) { + qc = qc.arg("--aggregate-min-umi-threshold").arg(string(inputs.aggregateBarcodeMinUmiThreshold)) + } + if !is_undefined(inputs.aggregateBarcodeTopN) { + qc = qc.arg("--aggregate-top-n").arg(string(inputs.aggregateBarcodeTopN)) + } + qc = qc.arg("--output").arg("result_qc.csv").saveFile("result_qc.csv"). saveFileContent("result_qc.json"). run() - // Per-cell metrics (Python). sampleId is irrelevant here — processColumn prepends the real sample - // axis from the iteration key, so we pass a constant and the CSV's sampleId column is ignored by the - // import. The distinct-UMI polars read scales with input volume, so size RAM from the tagstat.tsv blob - // (tag "tagstat") on a floor instead of a fixed request. + // Per-cell metrics (Python). sampleId is irrelevant here: processColumn prepends the real sample axis from + // the iteration key, so this passes a constant and the import ignores the CSV's sampleId column. The + // distinct-UMI polars read scales with input volume, so size RAM from the tagstat.tsv blob, tagged + // "tagstat", on a floor rather than as a fixed request. metricsBaseGB := 16 // workflow-tengo 6.7.0 fluent formula API fed via .resources({ onCPU }) (see fb-parse for the note). formula := exec.formula @@ -70,13 +80,12 @@ self.body(func(inputs) { between(formula.gib(metricsBaseGB), formula.gib(256)). staticFallback(formula.gib(metricsBaseGB)) - hasControl := !is_undefined(control) && control != "" metrics := exec.builder(). printErrStreamToStdout(). saveStdoutStream(). software(metricsSw). resources({ onCPU: { cpu: 2, ram: metricsMemFormula } }). - // polars sizes its pool to all host cores by default; cap it to the granted CPU. + // polars sizes its pool to all host cores by default. Cap it to the granted CPU. envWithVar("POLARS_MAX_THREADS", "{system.cpu}"). addFile("tagstat.tsv", tagstatTsv, { tag: "tagstat" }). addFile("tags.csv", tagsCsv). @@ -87,45 +96,34 @@ self.body(func(inputs) { arg("--umi-count-col").arg("unique_" + tags.umi). arg("--csv-barcode-col").arg(barcodeSeqColumn). arg("--csv-feature-col").arg(featureNameColumn). - arg("--dominance-threshold").arg(string(dominanceThreshold)). arg("--output-prefix").arg("result"). saveFile("result_abundance.csv"). saveFile("result_fractions.csv"). - saveFile("result_consensus.csv"). - saveFile("result_specificity.csv"). saveFile("result_per_cell_summary.csv") - if hasControl { - metrics = metrics.arg("--control").arg(control) - } - // Multi-barcode antigen combine mode (optional). Only pass --combine-col when a column is configured; - // otherwise per_cell_metrics defaults every feature to OR/sum (unchanged behaviour). --min-umi is the - // AND per-barcode "fired" floor and is harmless when no combine column is set. + // Multi-barcode antigen combine mode (optional). Pass --combine-col only where a column is configured. + // Otherwise per_cell_metrics defaults every feature to OR/sum. --min-umi is the AND per-barcode "fired" + // floor, and is harmless where no combine column is set. if combineColumn != "" { metrics = metrics.arg("--combine-col").arg(combineColumn).arg("--min-umi").arg(string(minUmi)) } - // Off-target designation (optional, F2). Only pass both flags when a column AND values are configured; - // per_cell_metrics requires them together and leaves the dominant call unchanged when absent. - if offtargetColumn != "" && offtargetValues != "" { - metrics = metrics.arg("--offtarget-col").arg(offtargetColumn).arg("--offtarget-values").arg(offtargetValues) - } metrics = metrics.run() - // specificity is meaningful only with a control feature, so the Python always writes - // result_specificity.csv (empty header-only when no --control) and we always return it. main.tpl and - // the model gate the specificity *import* on hasControl, so the empty file is simply never read. + // No consensus (dominant-feature) or Cell Ranger specificity table. per_cell_metrics.py writes neither and + // takes none of the four flags that drove them (--control, --dominance-threshold, --offtarget-col, + // --offtarget-values). The verdict stage asks the binding question of every antigen independently, so a + // cell that bound three antigens carries three verdicts rather than one winner and two absences. The + // negative-control designation survives as a per-feature marker column, emitted outside this fan-out. return { abundance: metrics.getFile("result_abundance.csv"), fractions: metrics.getFile("result_fractions.csv"), - consensus: metrics.getFile("result_consensus.csv"), - specificity: metrics.getFile("result_specificity.csv"), perCellSummary: metrics.getFile("result_per_cell_summary.csv"), qc: qc.getFile("result_qc.csv"), qcJson: qc.getFileContent("result_qc.json"), // inline content (saveFileContent) so the model can read it - // per-cell-metrics (Python) stdout — the "4-metrics" step log. Surfaced separately (not in the - // mitool stepLogs map, which fb-refine-tagstat builds before this step runs). Capturing it here is - // CID-safe via the content-hash persist of these metrics intermediates in main.tpl.tengo: the File - // outputs above ARE resolved downstream, so that persist loop (not any avoidance of resolution) is - // what keeps re-capturing this stdout stream from re-triggering CIDConflictError. + // per-cell-metrics (Python) stdout: the "4-metrics" step log. Surfaced separately rather than in the + // mitool stepLogs map, which fb-refine-tagstat builds before this step runs. Capturing it here is + // CID-safe through the content-hash persist of these metrics intermediates in main.tpl.tengo. The File + // outputs above ARE resolved downstream, so that persist loop, and not any avoidance of resolution, is + // what keeps this stdout capture from re-triggering CIDConflictError. metricsLogStream: metrics.getStdoutStream() } }) diff --git a/workflow/src/fb-feature-properties.tpl.tengo b/workflow/src/fb-feature-properties.tpl.tengo index 6b0ade1..5dc4022 100644 --- a/workflow/src/fb-feature-properties.tpl.tengo +++ b/workflow/src/fb-feature-properties.tpl.tengo @@ -1,21 +1,21 @@ // Per-feature property import (render.create boundary). // -// Feature Integration imports every EXTRA column of the tag->feature CSV — every column beyond the -// user-mapped barcode-sequence + feature-name roles — as a per-feature property, and surfaces each as -// a p-column keyed on the shared feature axis (pl7.app/feature/featureId). The properties then ride -// that axis into VDJ Multiomic Integration's per-feature outputs and lead selection, so downstream can -// group / aggregate the binding profile by property (e.g. all human vs all cyno antigens) with no -// re-import (spec A-0026 / A-0015). +// Feature Integration imports every EXTRA column of the tag->feature CSV -- every column beyond the +// user-mapped barcode-sequence and feature-name roles -- as a per-feature property, and surfaces each as a +// p-column keyed on the shared feature axis (pl7.app/feature/featureId). The properties then ride that axis +// into VDJ Multiomic Integration's per-feature outputs and lead selection, so downstream can group and +// aggregate the binding profile by property, such as all human against all cyno antigens, with no re-import +// (spec A-0026 / A-0015). // -// emit_feature_properties.py (run in main.tpl.tengo) produced two inputs: the wide per-feature CSV -// (`feature` + one column per property) and a meta JSON listing the property column names + each +// emit_feature_properties.py, run in main.tpl.tengo, produced two inputs: the wide per-feature CSV +// (`feature` plus one column per property) and a meta JSON listing the property column names and each // property's distinct values. The property NAMES are user data, unknown at graph-build time, so this -// template reads the meta content via .getData() to build one xsv import column per property (the same -// "inspect file content inside a sub-template that receives it" pattern as fb-refine-tagstat.tpl.tengo; -// the dynamic-columns-from-a-tool-output pattern mirrors VDJ Multiomic Integration's finalize.tpl). +// template reads the meta content through .getData() to build one xsv import column per property. Same +// "inspect file content inside a sub-template that receives it" pattern as fb-refine-tagstat.tpl.tengo, and +// the same dynamic-columns-from-a-tool-output pattern as VDJ Multiomic Integration's finalize.tpl. // -// Returns a BUILT pFrame (like VDJM's clonotypeProperties export) that main.tpl publishes as the -// `featureProperties` export. When the CSV carries no extra columns the frame is empty (a no-op export). +// Returns a BUILT pFrame, as VDJM's clonotypeProperties export does, which main.tpl publishes as the +// `featureProperties` export. Where the CSV carries no extra columns the frame is empty. self := import("@platforma-sdk/workflow-tengo:tpl") xsv := import("@platforma-sdk/workflow-tengo:pframes.xsv") @@ -34,14 +34,17 @@ self.body(func(inputs) { meta := json.decode(string(inputs.propertyMeta.getData())) propertyColumns := is_undefined(meta.columns) ? [] : meta.columns valuesByColumn := is_undefined(meta.valuesByColumn) ? {} : meta.valuesByColumn + // Negative-control marker for control-aware metrics: a hidden per-feature column marking the control + // feature, emitted only where a control is designated. main.tpl gates this. + hasControl := !is_undefined(inputs.hasControl) && inputs.hasControl fb := pframes.pFrameBuilder() - // No extra columns -> no properties to import; publish an empty frame. Skipping the xsv import here is - // required (an empty `columns` list fails the import spec validation) and correct (nothing to key). - if len(propertyColumns) > 0 { - // Same trace step as the per-cell contract columns (deterministic; seeded from the input FASTQ - // spec + blockId), so the properties carry consistent provenance / labels. + // Nothing to emit without extra property columns (A-0026) or a negative-control marker. Publish an empty + // frame otherwise, because an empty xsv `columns` list fails the import spec validation. + if len(propertyColumns) > 0 || hasControl { + // Same trace step as the per-cell contract columns, deterministic and seeded from the input FASTQ spec + // plus blockId, so the properties carry consistent provenance and labels. trace := pSpec.makeTrace(inputs.traceSeedSpec, { type: "milaboratories.feature-integration", id: blockId, @@ -50,16 +53,32 @@ self.body(func(inputs) { }) featureAxis := columnSpecs.featureAxis(blockId) - importColumns := columnSpecs.featurePropertyImportColumns(propertyColumns, valuesByColumn) - rawPf := xsv.importFile(inputs.propertiesFile, "csv", { - axes: [{ column: "feature", spec: featureAxis }], - columns: importColumns, - storageFormat: "Parquet", - partitionKeyLength: 0 - }, { splitDataAndSpec: true }) - for k, v in rawPf { - fb.add(k, trace.inject(v.spec), v.data) + if len(propertyColumns) > 0 { + importColumns := columnSpecs.featurePropertyImportColumns(propertyColumns, valuesByColumn) + rawPf := xsv.importFile(inputs.propertiesFile, "csv", { + axes: [{ column: "feature", spec: featureAxis }], + columns: importColumns, + storageFormat: "Parquet", + partitionKeyLength: 0 + }, { splitDataAndSpec: true }) + for k, v in rawPf { + fb.add(k, trace.inject(v.spec), v.data) + } + } + + // The negative-control marker column: hidden, and not a filter. It rides pl7.app/feature/featureId into + // VDJ Multiomic Integration, which removes the control from its antigen metrics. + if hasControl { + ctrlPf := xsv.importFile(inputs.negativeControlFile, "csv", { + axes: [{ column: "feature", spec: featureAxis }], + columns: [columnSpecs.negativeControlColumn()], + storageFormat: "Parquet", + partitionKeyLength: 0 + }, { splitDataAndSpec: true }) + for k, v in ctrlPf { + fb.add(k, trace.inject(v.spec), v.data) + } } } diff --git a/workflow/src/fb-parse.tpl.tengo b/workflow/src/fb-parse.tpl.tengo index 86948fb..aae7322 100644 --- a/workflow/src/fb-parse.tpl.tengo +++ b/workflow/src/fb-parse.tpl.tengo @@ -1,12 +1,12 @@ // mitool parse sub-template (render.create boundary). // -// Runs mitool parse and returns parsed.mic + parse_report.json + a live stdout stream (parseLogStream). -// Capturing stdout (saveStdoutStream) injects the rate-dependent stdout into the exec's content-hashed -// files map, drifting every getFile off it — which used to throw CIDConflictError on a stale re-render. -// It is safe now because main.tpl persists parsedMic/parseReport (and the refine/tag-stat intermediates) -// by content hash, pinning their CIDs so getFile resolutions stay stable across re-renders (see main.tpl -// + fb-pipeline). MI_PROGRESS_PREFIX makes mitool prefix its progress lines with [==PROGRESS==], which -// the model scrapes for a live 0–100% parse bar (getProgressLogWithInfo). Mirrors blocks/peptide-extraction. +// Runs mitool parse and returns parsed.mic, parse_report.json and a live stdout stream (parseLogStream). +// Capturing stdout through saveStdoutStream injects the rate-dependent stdout into the exec's +// content-hashed files map and drifts every getFile off it, which throws CIDConflictError on a stale +// re-render. It is safe here only because main.tpl persists parsedMic and parseReport, and the refine and +// tag-stat intermediates, by content hash, pinning their CIDs so getFile resolutions stay stable across +// re-renders. MI_PROGRESS_PREFIX makes mitool prefix its progress lines with [==PROGRESS==], which the model +// scrapes for a live 0-100% parse bar (getProgressLogWithInfo). Mirrors blocks/peptide-extraction. self := import("@platforma-sdk/workflow-tengo:tpl") exec := import("@platforma-sdk/workflow-tengo:exec") @@ -17,12 +17,12 @@ json := import("json") self.defineOutputs("parsedMic", "parseReport", "parseLogStream") -// :memory-from-limits sizes the JVM heap from the granted limits — avoids end-of-parse OOM when many -// tag writers flush (see demultiplex-fastq). +// :memory-from-limits sizes the JVM heap from the granted limits, which avoids an end-of-parse OOM as many +// tag writers flush. See demultiplex-fastq. mitoolSw := assets.importSoftware("@platforma-open/milaboratories.software-mitool:memory-from-limits") -// mitool prefixes its progress lines with this marker (MI_PROGRESS_PREFIX) so the model can scrape a -// live per-sample parse %. Same value peptide-extraction uses. +// mitool prefixes its progress lines with this marker (MI_PROGRESS_PREFIX) so the model can scrape a live +// per-sample parse %. The same value peptide-extraction uses. progressPrefix := "[==PROGRESS==]" self.body(func(inputs) { @@ -31,15 +31,15 @@ self.body(func(inputs) { inputFiles := inputs.inputFiles mitoolCPUs := inputs.mitoolCPUs memBaseGB := inputs.mitoolBaseMemGB - // 0 = no user override (Advanced Settings) -> size RAM from the input reads' blob size on a floor; - // >0 = a hard fixed request. parse OOMs at end-of-parse as tag writers flush, which scales with input - // volume (mixcr-clonotyping / demultiplex-fastq). size("reads") reads only blob metadata. + // 0 = no user override in Advanced Settings, so RAM is sized from the input reads' blob size on a floor. + // >0 = a fixed request. parse OOMs at end-of-parse as tag writers flush, which scales with input volume, + // as mixcr-clonotyping and demultiplex-fastq do. size("reads") reads only blob metadata. memOverrideGB := inputs.mitoolMemOverrideGB - // Preview read cap: 0 = full run; >0 = mitool parse reads only the first N reads (--limit). + // Preview read cap. 0 = full run. >0 = mitool parse reads only the first N reads, through --limit. limitInput := is_undefined(inputs.limitInput) ? 0 : inputs.limitInput - // RAM = base + reads*4, capped at 256 GiB (mirrors mixcr-analyze); static fallback = base for - // backends that can't evaluate resource formulas at run time. + // RAM = base + reads*4, capped at 256 GiB, mirroring mixcr-analyze. The static fallback is base, for a + // backend that cannot evaluate resource formulas at run time. formula := exec.formula memFormula := formula.gib(memBaseGB). plus(formula.size("reads").times(4)). @@ -66,8 +66,8 @@ self.body(func(inputs) { parse = parse.arg("--limit").arg(string(limitInput)) } - // inputFiles is a ResourceMap keyed by JSON-encoded [filename] arrays — decode to recover the - // mitool-expected filename (e.g. "input_R1.fastq.gz"). Tag "reads" feeds the size formula above. + // inputFiles is a ResourceMap keyed by JSON-encoded [filename] arrays. Decode to recover the + // mitool-expected filename, such as "input_R1.fastq.gz". Tag "reads" feeds the size formula above. filesMap := inputFiles.inputs() for encKey in maps.getKeys(filesMap) { key := json.decode(encKey) diff --git a/workflow/src/fb-pipeline.tpl.tengo b/workflow/src/fb-pipeline.tpl.tengo index d323223..5cb911a 100644 --- a/workflow/src/fb-pipeline.tpl.tengo +++ b/workflow/src/fb-pipeline.tpl.tengo @@ -1,17 +1,18 @@ // Per-sample feature-barcode pipeline (processColumn body). // -// One invocation per sample group (the FASTQ column is aggregated over lane + readIndex). Runs the -// mitool chain — parse -> refine-tags -> tag-stat -u — to produce per-(cell, feature-barcode) -// distinct-UMI counts, then hands that table plus the user's tag->feature CSV to the per-cell-metrics -// Python software, which collapses to the (cell x feature) outputs. +// One invocation per sample group, since the FASTQ column is aggregated over lane and readIndex. Runs the +// mitool chain -- parse -> refine-tags -> tag-stat -u -- for per-(cell, feature-barcode) distinct-UMI counts, +// then hands that table plus the user's tag->feature CSV to the per-cell-metrics Python software, which +// collapses to the (cell x feature) outputs. Also runs tag-stat -t FEATURE (no -u) on the pre-refine +// parsedMic, surfaced as rawFeatureCounts: refine-tags snaps FEATURE onto the panel whitelist, so this is +// the only point in the chain where an undeclared barcode is still visible. // -// Every exec runs inside a render.create sub-template (fb-parse / fb-refine / fb-tagstat produce the -// mitool files; fb-downstream runs the QC + per-cell-metrics consumers) so this body returns only render -// outputs. The mitool execs deliberately do NOT capture a stdout progress stream: saveStdoutStream saves -// the rate-dependent stdout into the exec's content-hashed files map, drifting every getFile off it and -// throwing CIDConflictError on a stale re-render. Per-step progress is derived deterministically instead, -// from which stage reports have settled (stepReports -> the model's sampleStep). -// Mirrors blocks/peptide-extraction (parse.tpl + umi-consensus.tpl + downstream-pipeline.tpl). +// Every exec runs inside a render.create sub-template, so this body returns only render outputs. fb-parse, +// fb-refine and fb-tagstat produce the mitool files, and fb-downstream runs the QC and per-cell-metrics +// consumers. The mitool execs deliberately do NOT capture a stdout progress stream: saveStdoutStream puts +// the rate-dependent stdout in the exec's content-hashed files map, drifts every getFile off it, and throws +// CIDConflictError on a stale re-render. Per-step progress is derived deterministically instead, from which +// stage reports have settled (stepReports -> the model's sampleStep). Mirrors blocks/peptide-extraction. self := import("@platforma-sdk/workflow-tengo:tpl") assets := import("@platforma-sdk/workflow-tengo:assets") @@ -31,7 +32,7 @@ fbParseTpl := assets.importTemplate(":fb-parse") fbRefineTagstatTpl := assets.importTemplate(":fb-refine-tagstat") fbDownstreamTpl := assets.importTemplate(":fb-downstream") -self.defineOutputs("abundance", "fractions", "consensus", "specificity", "perCellSummary", "qc", "qcJson", "stepReports", "stepLogs", "parseLogStream", "metricsLogStream", "parsedMic", "parseReport", "refineReport", "tagstatTsv") +self.defineOutputs("abundance", "fractions", "perCellSummary", "qc", "qcJson", "stepReports", "stepLogs", "parseLogStream", "metricsLogStream", "parsedMic", "parseReport", "refineReport", "tagstatTsv", "rawFeatureCounts") self.body(func(inputs) { inputData := inputs[pConstants.VALUE_FIELD_NAME] @@ -40,34 +41,29 @@ self.body(func(inputs) { pattern := inputs.pattern tags := inputs.tags // { cell, umi, feature } tag names (from the model; see pattern.ts) tagsCsv := inputs.tagsCsv - dominanceThreshold := inputs.dominanceThreshold - control := inputs.control // may be undefined / "" -> no specificity score fileExtension := inputs.fileExtension mitoolCPUs := inputs.mitoolCPUs barcodeSeqColumn := inputs.barcodeSeqColumn featureNameColumn := inputs.featureNameColumn combineColumn := is_undefined(inputs.combineColumn) ? "" : inputs.combineColumn minUmi := is_undefined(inputs.minUmi) ? 1 : inputs.minUmi - // Off-target designation (optional, F2) — property column + comma-separated off-target values. - offtargetColumn := is_undefined(inputs.offtargetColumn) ? "" : inputs.offtargetColumn - offtargetValues := is_undefined(inputs.offtargetValues) ? "" : inputs.offtargetValues - cellWhitelist := inputs.cellWhitelist // "" = de-novo CELL correction (default); else a 10x built-in + cellWhitelist := inputs.cellWhitelist // "" = de-novo CELL correction, the default. Else a 10x built-in. memBaseGB := inputs.mitoolBaseMemGB - // mitool memory override (Advanced Settings): passed to the parse/refine sub-templates. 0 = no - // override (size RAM from the input blob on a floor); >0 = a hard fixed request. A sentinel keeps the - // render.create input a plain number (the model only sets metaExtra.mitoolMemOverrideGB when the user - // configures one, so inputs.mitoolMemOverrideGB is otherwise undefined). + // mitool memory override (Advanced Settings), passed to the parse and refine sub-templates. 0 = no + // override, so RAM is sized from the input blob on a floor. >0 = a fixed request. A sentinel keeps the + // render.create input a plain number: the model sets metaExtra.mitoolMemOverrideGB only where the user + // configures one, so inputs.mitoolMemOverrideGB is otherwise undefined. memOverrideGB := is_undefined(inputs.mitoolMemOverrideGB) ? 0 : inputs.mitoolMemOverrideGB - // Preview read cap: 0 = full run (no --limit); >0 = mitool parse processes only the first N reads. + // Preview read cap. 0 = full run, with no --limit. >0 = mitool parse processes only the first N reads. limitInput := is_undefined(inputs.limitInput) ? 0 : inputs.limitInput ll.assert(!is_undefined(pattern) && pattern != "", "tag pattern is required") - // Sample-aware tag->feature mapping (optional). When a sample column is set, the tag CSV maps the same - // barcode to different features per sample, so filter it to THIS sample's rows before it feeds the - // panel (fb-refine) and the naming join (fb-downstream). - // sampleColumn == "" -> global mapping, CSV passed through untouched. + // Sample-aware tag->feature mapping (optional). Where a sample column is set, the tag CSV maps the same + // barcode to different features per sample, so filter it to THIS sample's rows before it feeds the panel + // (fb-refine) and the naming join (fb-downstream). sampleColumn == "" means one global mapping, with the + // CSV passed through untouched. sampleColumn := inputs.sampleColumn tagsCsvForSample := tagsCsv if sampleColumn != "" { @@ -86,8 +82,8 @@ self.body(func(inputs) { meta := inputData.getDataAsJson() fileMap := inputData.inputs() - // Collect R1/R2 FASTQ files and build the mitool input arg from the aggregation layout - // (readIndex -> R1/R2; optional lane). Feature barcoding is paired: cell+UMI on R1, feature on R2. + // Collect R1/R2 FASTQ files and build the mitool input arg from the aggregation layout: readIndex -> + // R1/R2, with an optional lane. Feature barcoding is paired, with cell+UMI on R1 and feature on R2. R1Files := [] R2Files := [] inputArg := "" @@ -128,8 +124,8 @@ self.body(func(inputs) { ll.assert(len(R1Files) > 0, "no R1 FASTQ files found") ll.assert(len(R2Files) > 0, "no R2 FASTQ files found (feature barcode is read from R2)") - // Step 1: mitool parse (fb-parse boundary) — extract CELL + UMI (R1) and the feature barcode (R2). - // The FASTQ files go in as a ResourceMap keyed by [filename] (decoded back inside fb-parse). + // Step 1: mitool parse (fb-parse boundary). Extract CELL + UMI from R1 and the feature barcode from R2. + // The FASTQ files go in as a ResourceMap keyed by [filename], decoded back inside fb-parse. inputFilesBuilder := pcolumn.resourceMapBuilder(1) for f in R1Files { inputFilesBuilder.add([f.name], f.file) } for f in R2Files { inputFilesBuilder.add([f.name], f.file) } @@ -145,13 +141,13 @@ self.body(func(inputs) { }) parsedMic := parseRun.output("parsedMic") parseReport := parseRun.output("parseReport") - // Live parse stdout stream — threaded into fb-refine-tagstat's stepLogs map (as the 1-parse entry) and - // also surfaced flat below to drive the model's live parse % + early sample roster (peptide pattern). + // Live parse stdout stream. Threaded into fb-refine-tagstat's stepLogs map as the 1-parse entry, and + // surfaced flat below to drive the model's live parse % and early sample roster. parseLogStream := parseRun.output("parseLogStream") - // Step 2: parse gate. mitool parse writes NO parsed.mic when zero reads match the pattern (fb-parse - // pre-wrote an empty placeholder). Read the parse report's matched count into a decision the branch - // below routes on, and synthesize the empty fallbacks. + // Step 2: parse gate. mitool parse writes NO parsed.mic where zero reads match the pattern, and fb-parse + // pre-wrote an empty placeholder. Read the parse report's matched count into a decision the branch below + // routes on, and synthesize the empty fallbacks. gate := exec.builder(). software(parseGateSw). cpu(1).mem("256MiB"). @@ -167,9 +163,9 @@ self.body(func(inputs) { saveFile("refine_report_empty.json"). run() - // Step 3: gated refine-tags -> tag-stat (fb-refine-tagstat boundary). On a match it runs the normal - // chain; on no-match it returns the gate's header-only tag-stat + empty refine report, skipping the - // mitool steps that would crash on the empty parsed.mic. Either way it yields tagstatTsv + refineReport. + // Step 3: gated refine-tags -> tag-stat (fb-refine-tagstat boundary). On a match it runs the normal chain. + // On no-match it returns the gate's header-only tag-stat and empty refine report, skipping the mitool + // steps that would crash on the empty parsed.mic. Either way it yields tagstatTsv and refineReport. refineTagstatRun := render.create(fbRefineTagstatTpl, { decision: gate.getFileContent("decision.json"), parsedMic: parsedMic, @@ -189,56 +185,58 @@ self.body(func(inputs) { refineReport := refineTagstatRun.output("refineReport") stepReports := refineTagstatRun.output("stepReports") stepLogs := refineTagstatRun.output("stepLogs") + // tag-stat -t FEATURE on the pre-refine parsedMic, before refine-tags snaps FEATURE onto the panel + // whitelist. Carries any sequence the panel does not declare, which a post-refine counts frame cannot. + rawFeatureCounts := refineTagstatRun.output("rawFeatureCounts") - // Step 4: QC report + per-cell metrics (fb-downstream boundary). These consume the drifting mitool - // files (tagstat.tsv + parse/refine reports), so they run inside a render boundary and their per-sample - // outputs are returned as render outputs — see the file header for why an inline consumer conflicts. + // Step 4: QC report and per-cell metrics (fb-downstream boundary). These consume the drifting mitool + // files -- tagstat.tsv plus the parse and refine reports -- so they run inside a render boundary and + // return their per-sample outputs as render outputs. See the file header for the inline-consumer clash. downstreamRun := render.create(fbDownstreamTpl, { tagstatTsv: tagstatTsv, parseReport: parseReport, refineReport: refineReport, tags: tags, tagsCsv: tagsCsvForSample, - control: control, // undefined -> render.create skips it -> no specificity - dominanceThreshold: dominanceThreshold, barcodeSeqColumn: barcodeSeqColumn, featureNameColumn: featureNameColumn, combineColumn: combineColumn, - minUmi: minUmi, - offtargetColumn: offtargetColumn, - offtargetValues: offtargetValues + minUmi: minUmi }) // The body must return exactly the defineOutputs set and no output may be undefined. return { abundance: downstreamRun.output("abundance"), fractions: downstreamRun.output("fractions"), - consensus: downstreamRun.output("consensus"), - specificity: downstreamRun.output("specificity"), perCellSummary: downstreamRun.output("perCellSummary"), qc: downstreamRun.output("qc"), qcJson: downstreamRun.output("qcJson"), - // per-cell-metrics (Python) stdout stream -> the 4-metrics step log (surfaced flat; the model keys - // it [sampleId] and the UI's per-step Logs panel reads it under the "4-metrics" step). + // per-cell-metrics (Python) stdout stream -> the 4-metrics step log, surfaced flat. The model keys it + // [sampleId], and the UI's per-step Logs panel reads it under the "4-metrics" step. metricsLogStream: downstreamRun.output("metricsLogStream"), - // Per-sample step-report map (parse / refine / tag-stat output files) — the model reads which are - // present to derive the sample's current step (deterministic). + // Per-sample step-report map of the parse, refine and tag-stat output files. The model derives the + // sample's current step from which are present, which is deterministic. stepReports: stepReports, - // Per-sample [step] map of live stdout streams (parse / refine / tag-stat) -> model per-step log - // handles + live parse %. Variable key set on a no-match sample (parse only). + // Per-sample [step] map of live stdout streams (parse / refine / tag-stat) -> the model's per-step log + // handles and live parse %. The key set is variable: a no-match sample carries parse alone. stepLogs: stepLogs, - // Flat parse stdout stream (also the 1-parse entry of stepLogs) -> model live parse % + early roster. + // Flat parse stdout stream, also the 1-parse entry of stepLogs -> live parse % and early roster. parseLogStream: parseLogStream, - // mitool intermediates surfaced so main.tpl can persist them by content hash (see there). These are - // the deterministic parse/refine/tag-stat outputs the downstream consumers resolve through - // (parseReport/refineReport -> qc; parsedMic/tagstatTsv -> the metrics chain). Once - // content-hash-persisted, their CIDs stop drifting across re-renders, which keeps getFile-off-a- - // streaming-exec resolutions stable and prevents the CIDConflictError (the reason stdout capture was - // removed). This also lets downstream reuse them instead of re-running the whole block on a settings - // change. No model/UI consumer — the persistence itself is the point. Mirrors blocks/peptide-extraction. + // mitool intermediates surfaced so main.tpl can persist them by content hash. These are the + // deterministic parse, refine and tag-stat outputs the downstream consumers resolve through: + // parseReport and refineReport feed qc, and parsedMic and tagstatTsv feed the metrics chain. Once + // content-hash-persisted their CIDs stop drifting across re-renders, which keeps a getFile off a + // streaming exec stable and prevents CIDConflictError. It also lets downstream reuse them instead of + // re-running the whole block on a settings change. No model or UI consumer: the persistence is the + // point. Mirrors blocks/peptide-extraction. parsedMic: parsedMic, parseReport: parseReport, refineReport: refineReport, - tagstatTsv: tagstatTsv + tagstatTsv: tagstatTsv, + // rawFeatureCounts' own exec calls neither saveStdoutStream() nor printErrStreamToStdout(), so the + // CIDConflictError mechanism above does not apply to it, and it is not in main.tpl.tengo's + // content-hash persist loop. Currently unpersisted and unconsumed; whoever wires it into a + // consumer decides then whether that consumer's resolution needs the same treatment. + rawFeatureCounts: rawFeatureCounts } }) diff --git a/workflow/src/fb-refine-tagstat.tpl.tengo b/workflow/src/fb-refine-tagstat.tpl.tengo index f2f14a9..99f4d89 100644 --- a/workflow/src/fb-refine-tagstat.tpl.tengo +++ b/workflow/src/fb-refine-tagstat.tpl.tengo @@ -1,22 +1,27 @@ // Gated refine + tag-stat (render.create boundary). // -// Branches on the parse gate's decision (parse_gate.py -> decision.json, read here via .getData() — file -// content can only be inspected inside a template that receives the file, hence this sub-template). -// - matched > 0 -> the normal chain: refine-tags (fb-refine) then tag-stat (fb-tagstat). -// - matched == 0 -> mitool wrote no parsed.mic (fb-parse pre-wrote an empty placeholder), so refine / -// tag-stat would crash on it. Skip both and return the header-only tag-stat TSV + -// empty refine report the gate already synthesized. The downstream Python is -// empty-safe (a header-only tag-stat yields empty results), so the sample completes -// with zero cells instead of failing the whole block. -// It also assembles the per-sample step-report map (the parse / refine / tag-stat output files) keyed by -// an ordinal step axis. These files are deterministic, so surfacing them is CID-safe; the model reads -// only their PRESENCE to derive which step a sample has reached (see the model's sampleStep) — a report -// appears once its step finishes. A no-match sample still gets all three (parse's real report plus the -// gate's empty refine report + header-only tag-stat), so it advances to Done like any other sample. +// Branches on the parse gate's decision (parse_gate.py -> decision.json), read here through .getData(). +// File content can be inspected only inside a template that receives the file, which is why this +// sub-template exists. +// - matched > 0 -> the normal chain: refine-tags (fb-refine), then tag-stat (fb-tagstat). +// - matched == 0 -> mitool wrote no parsed.mic, and fb-parse pre-wrote an empty placeholder, so refine and +// tag-stat would crash on it. Skip both and return the header-only tag-stat TSV and +// empty refine report the gate synthesized. The downstream Python is empty-safe, so the +// sample completes with zero cells instead of failing the whole block. +// It also assembles the per-sample step-report map of the parse, refine and tag-stat output files, keyed by +// an ordinal step axis. These files are deterministic, so surfacing them is CID-safe. The model reads only +// their PRESENCE to derive which step a sample has reached (see the model's sampleStep), and a report +// appears once its step finishes. A no-match sample still gets all three -- parse's real report plus the +// gate's empty refine report and header-only tag-stat -- so it advances to Done like any other sample. +// +// Also runs a raw FEATURE tag-stat on the pre-refine parsedMic (rawFeatureCounts), gated the same way: on +// no-match it returns the gate's header-only tagstat_empty.tsv, which already carries the FEATURE and +// totalWeight columns this output needs. self := import("@platforma-sdk/workflow-tengo:tpl") assets := import("@platforma-sdk/workflow-tengo:assets") render := import("@platforma-sdk/workflow-tengo:render") +exec := import("@platforma-sdk/workflow-tengo:exec") pcolumn := import("@platforma-sdk/workflow-tengo:pframes.pcolumn") file := import("@platforma-sdk/workflow-tengo:file") @@ -25,7 +30,11 @@ json := import("json") fbRefineTpl := assets.importTemplate(":fb-refine") fbTagstatTpl := assets.importTemplate(":fb-tagstat") -self.defineOutputs("tagstatTsv", "refineReport", "stepReports", "stepLogs") +// For the raw (pre-refine) FEATURE tag-stat below. refine-tags and tag-stat -u run inside +// fb-refine and fb-tagstat; this step is deterministic and unstreamed, so it runs directly here. +mitoolSw := assets.importSoftware("@platforma-open/milaboratories.software-mitool:memory-from-limits") + +self.defineOutputs("tagstatTsv", "refineReport", "stepReports", "stepLogs", "rawFeatureCounts") self.body(func(inputs) { shouldContinue := json.decode(string(inputs.decision.getData())).shouldContinue @@ -33,24 +42,28 @@ self.body(func(inputs) { tags := inputs.tags if !shouldContinue { - // No-match branch: use the gate's header-only tag-stat + empty refine report (refine / tag-stat - // never ran). The step-report map carries parse's real report plus those fallbacks. - // Reports are exported (file.exportFile) so each map value is a downloadable Blob the model can - // probe with getFileHandle; the model only reads their PRESENCE, never the content. + // No-match branch: use the gate's header-only tag-stat and empty refine report, since refine and + // tag-stat never ran. The step-report map carries parse's real report plus those fallbacks. Reports + // are exported through file.exportFile, so each map value is a downloadable Blob the model can probe + // with getFileHandle. The model reads only their PRESENCE, never the content. stepReports := pcolumn.resourceMapBuilder(1) stepReports.add(["1-parse"], file.exportFile(inputs.parseReport)) stepReports.add(["2-refine"], file.exportFile(inputs.emptyRefineReport)) stepReports.add(["3-tagstat"], file.exportFile(inputs.emptyTagstat)) - // Only the parse step ran, so only its log stream exists (refine / tag-stat were skipped). A - // variable key set is expected — the UI shows nothing for the absent steps (mirrors peptide's - // empty branch). parse still streamed, so the sample gets a live parse log + progress. + // Only the parse step ran, so only its log stream exists. A variable key set is expected, and the UI + // shows nothing for the absent steps, mirroring peptide's empty branch. parse still streamed, so the + // sample gets a live parse log and progress. stepLogs := pcolumn.resourceMapBuilder(1) stepLogs.add(["1-parse"], inputs.parseLogStream) return { tagstatTsv: inputs.emptyTagstat, refineReport: inputs.emptyRefineReport, stepReports: stepReports.build(), - stepLogs: stepLogs.build() + stepLogs: stepLogs.build(), + // emptyTagstat is header-only (CELL FEATURE count totalWeight unique_UMI): no data + // rows, so its extra columns are harmless, and it carries the FEATURE and + // totalWeight columns a reader of rawFeatureCounts needs. + rawFeatureCounts: inputs.emptyTagstat } } @@ -72,14 +85,41 @@ self.body(func(inputs) { mitoolCPUs: inputs.mitoolCPUs }) + // Raw FEATURE tag-stat on the PRE-refine parsedMic, so a sequence the panel never + // declared still shows up (refine-tags has not yet snapped it onto the panel or + // dropped it). No -u: the streaming TagStatsAggregator, one entry per distinct + // observed FEATURE value, rather than -u's on-disk hierarchical sort. Output columns: + // FEATURE, count, totalWeight. + // + // Measured on a generated 4.26M-read run with a 6-tag panel: 94 distinct FEATURE barcodes + // observed pre-refine, peak RSS ~489 MB for this step at -Xmx4g. The aggregator holds one + // entry per distinct observed barcode, so its memory scales with sequencing-error diversity + // rather than with panel size. Generated reads carry a simpler error model than a + // sequencer's, so this count is a lower bound on what real data produces; the formula is + // unvalidated against a real run or a large panel. + rawFeatureBaseGB := 4 + formula := exec.formula + rawFeatureMemFormula := formula.gib(rawFeatureBaseGB). + plus(formula.size("mic").times(1)). + between(formula.gib(rawFeatureBaseGB), formula.gib(64)). + staticFallback(formula.gib(rawFeatureBaseGB)) + rawFeatureCount := exec.builder(). + software(mitoolSw). + resources({ onCPU: { cpu: 1, ram: rawFeatureMemFormula } }). + arg("tag-stat"). + arg("-t").arg(tags.feature). + addFile("parsed.mic", inputs.parsedMic, { tag: "mic" }). + arg("parsed.mic").arg("raw_feature_counts.tsv").saveFile("raw_feature_counts.tsv"). + run() + stepReports := pcolumn.resourceMapBuilder(1) stepReports.add(["1-parse"], file.exportFile(inputs.parseReport)) stepReports.add(["2-refine"], file.exportFile(refineRun.output("refineReport"))) stepReports.add(["3-tagstat"], file.exportFile(tagStatRun.output("tagstatTsv"))) - // Per-step live stdout streams, same [step] axis as stepReports. The model exposes these as per-step - // log handles (PlLogView) and scrapes parse's for a live % (see fb-parse.tpl.tengo). The Python - // per-cell-metrics step (fb-downstream) is surfaced separately (metrics log), not in this mitool map. + // Per-step live stdout streams, on the same [step] axis as stepReports. The model exposes these as + // per-step log handles (PlLogView) and scrapes parse's for a live % (see fb-parse.tpl.tengo). The Python + // per-cell-metrics step (fb-downstream) is surfaced separately as the metrics log. stepLogs := pcolumn.resourceMapBuilder(1) stepLogs.add(["1-parse"], inputs.parseLogStream) stepLogs.add(["2-refine"], refineRun.output("refineLogStream")) @@ -89,6 +129,7 @@ self.body(func(inputs) { tagstatTsv: tagStatRun.output("tagstatTsv"), refineReport: refineRun.output("refineReport"), stepReports: stepReports.build(), - stepLogs: stepLogs.build() + stepLogs: stepLogs.build(), + rawFeatureCounts: rawFeatureCount.getFile("raw_feature_counts.tsv") } }) diff --git a/workflow/src/fb-refine.tpl.tengo b/workflow/src/fb-refine.tpl.tengo index d158343..c703897 100644 --- a/workflow/src/fb-refine.tpl.tengo +++ b/workflow/src/fb-refine.tpl.tengo @@ -1,14 +1,15 @@ // refine-tags sub-template (render.create boundary): emit the feature panel, then refine-tags. // -// Returns refined.mic + refine_report.json + a live stdout stream (refineLogStream) to the caller. -// Capturing stdout is safe now that main.tpl content-hash-persists the mitool intermediates (see -// fb-parse.tpl.tengo header) — it used to cause the CIDConflictError. Mirrors blocks/peptide-extraction -// umi-consensus.tpl.tengo. Only the refine-tags exec streams; the deterministic panel-emit exec does not. +// Returns refined.mic, refine_report.json and a live stdout stream (refineLogStream) to the caller. +// Capturing stdout is safe only because main.tpl content-hash-persists the mitool intermediates, which is +// what prevents CIDConflictError (see the fb-parse.tpl.tengo header). Mirrors blocks/peptide-extraction +// umi-consensus.tpl.tengo. Only the refine-tags exec streams. The deterministic panel-emit exec does not. // -// refine-tags error-corrects cell, feature, and UMI barcodes. FEATURE snaps to the panel whitelist -// (#file:panel.txt): within-Hamming-1 reads snap to a panel barcode, off-panel reads are dropped. CELL -// is de-novo corrected by default; a chemistry whitelist snaps it to a 10x built-in (#builtin:). -// The panel-emit exec is deterministic (no stream), so its panel.txt is safe to feed refine.addFile. +// refine-tags error-corrects cell, feature and UMI barcodes. FEATURE snaps to the panel whitelist +// (#file:panel.txt): a within-Hamming-1 read snaps to a panel barcode, and an off-panel read is dropped. +// CELL is de-novo corrected by default, and a chemistry whitelist snaps it to a 10x built-in +// (#builtin:). The panel-emit exec is deterministic and does not stream, so its panel.txt is safe to +// feed refine.addFile. self := import("@platforma-sdk/workflow-tengo:tpl") exec := import("@platforma-sdk/workflow-tengo:exec") @@ -19,7 +20,7 @@ self.defineOutputs("refinedMic", "refineReport", "refineLogStream") mitoolSw := assets.importSoftware("@platforma-open/milaboratories.software-mitool:memory-from-limits") emitPanelSw := assets.importSoftware("@platforma-open/milaboratories.feature-integration.per-cell-metrics:emit-panel") -// mitool progress-line marker (see fb-parse.tpl.tengo) — drives the model's per-step live log/progress. +// mitool progress-line marker (see fb-parse.tpl.tengo). Drives the model's per-step live log and progress. progressPrefix := "[==PROGRESS==]" self.body(func(inputs) { @@ -27,15 +28,15 @@ self.body(func(inputs) { tags := inputs.tags // { cell, umi, feature } tag names (from the model; see pattern.ts) tagsCsv := inputs.tagsCsv barcodeSeqColumn := inputs.barcodeSeqColumn - cellWhitelist := inputs.cellWhitelist // "" = de-novo CELL correction (default); else a 10x built-in + cellWhitelist := inputs.cellWhitelist // "" = de-novo CELL correction, the default. Else a 10x built-in. mitoolCPUs := inputs.mitoolCPUs memBaseGB := inputs.mitoolBaseMemGB - memOverrideGB := inputs.mitoolMemOverrideGB // 0 = formula-sized; >0 = hard fixed request + memOverrideGB := inputs.mitoolMemOverrideGB // 0 = formula-sized. >0 = a fixed request. - // Build the feature-barcode panel (panel.txt) from the tag->feature CSV. With no sample column the - // CSV is identical across samples, so this exec dedups across samples by CID (same tags.csv input); - // in sample-aware mode tagsCsv is already the per-sample filtered file (fb-pipeline), so the panel is - // correctly per-sample and must NOT be shared — do not "optimize" this into one panel for all samples. + // Build the feature-barcode panel (panel.txt) from the tag->feature CSV. With no sample column the CSV is + // identical across samples, so this exec dedups across them by CID on the same tags.csv input. In + // sample-aware mode tagsCsv is already the per-sample filtered file (fb-pipeline), so the panel is + // per-sample and must NOT be shared. Never collapse this into one panel for all samples. panel := exec.builder(). software(emitPanelSw). cpu(1). @@ -46,8 +47,8 @@ self.body(func(inputs) { saveFile("panel.txt"). run() - // RAM = base + reads*4, capped at 256 GiB (mirrors mixcr-analyze); static fallback = base. - // workflow-tengo 6.7.0 fluent formula API fed via .resources({ onCPU }) (see fb-parse note). + // RAM = base + reads*4, capped at 256 GiB, mirroring mixcr-analyze. The static fallback is base. + // workflow-tengo 6.7.0 fluent formula API, fed through .resources({ onCPU }). See the fb-parse note. formula := exec.formula memFormula := formula.gib(memBaseGB). plus(formula.size("reads").times(4)). diff --git a/workflow/src/fb-tagstat.tpl.tengo b/workflow/src/fb-tagstat.tpl.tengo index df39d5c..99dd739 100644 --- a/workflow/src/fb-tagstat.tpl.tengo +++ b/workflow/src/fb-tagstat.tpl.tengo @@ -1,13 +1,13 @@ // tag-stat sub-template (render.create boundary). // -// tag-stat -u computes per-(cell, feature-barcode) distinct-UMI counts and returns tagstat.tsv + a live -// stdout stream (tagstatLogStream) to the caller. Capturing stdout is safe now that main.tpl content-hash- -// persists the mitool intermediates (see fb-parse.tpl.tengo header) — it used to cause the CIDConflictError. -// Output columns: CELL FEATURE count totalWeight unique_UMI. +// tag-stat -u computes per-(cell, feature-barcode) distinct-UMI counts and returns tagstat.tsv and a live +// stdout stream (tagstatLogStream) to the caller. Capturing stdout is safe only because main.tpl +// content-hash-persists the mitool intermediates, which is what prevents CIDConflictError (see the +// fb-parse.tpl.tengo header). Output columns: CELL FEATURE count totalWeight unique_UMI. // -// --use-local-temp puts the count-unique on-disk sort in the exec work dir instead of the shared system -// /tmp — team convention for heavy on-disk-sort mitool steps (peptide-extraction, mixcr-shm-trees, -// miltenyi cite silent /tmp-exhaustion hangs). +// --use-local-temp puts the count-unique on-disk sort in the exec work dir rather than the shared system +// /tmp. Team convention for heavy on-disk-sort mitool steps: peptide-extraction, mixcr-shm-trees and +// miltenyi all cite silent /tmp-exhaustion hangs. self := import("@platforma-sdk/workflow-tengo:tpl") exec := import("@platforma-sdk/workflow-tengo:exec") @@ -17,7 +17,7 @@ self.defineOutputs("tagstatTsv", "tagstatLogStream") mitoolSw := assets.importSoftware("@platforma-open/milaboratories.software-mitool:memory-from-limits") -// mitool progress-line marker (see fb-parse.tpl.tengo) — drives the model's per-step live log/progress. +// mitool progress-line marker (see fb-parse.tpl.tengo). Drives the model's per-step live log and progress. progressPrefix := "[==PROGRESS==]" self.body(func(inputs) { @@ -25,16 +25,16 @@ self.body(func(inputs) { tags := inputs.tags // { cell, umi, feature } tag names (from the model; see pattern.ts) mitoolCPUs := inputs.mitoolCPUs - // tag-stat's distinct-UMI sort parallelizes to ~5 cores (benchmarked); beyond that no gain, so cap - // there and never exceed the granted CPUs. Ties to the same perProcessCPUs knob as parse/refine - // (default 16). + // tag-stat's distinct-UMI sort parallelizes to about 5 cores, benchmarked, with no gain beyond that. Cap + // there and never exceed the granted CPUs. Ties to the same perProcessCPUs knob as parse and refine, + // which main.tpl.tengo defaults to 8. TAGSTAT_CPU_CAP := 5 tagStatCPU := mitoolCPUs < TAGSTAT_CPU_CAP ? mitoolCPUs : TAGSTAT_CPU_CAP - // The distinct-UMI on-disk sort scales with input volume, so size RAM from the refined.mic blob - // (molecules) on a floor instead of a fixed request (same mixcr-style size formula as parse/refine). + // The distinct-UMI on-disk sort scales with input volume, so size RAM from the refined.mic blob of + // molecules on a floor rather than as a fixed request. Same mixcr-style formula as parse and refine. tagStatBaseGB := 16 - // workflow-tengo 6.7.0 fluent formula API fed via .resources({ onCPU }) (see fb-parse for the note). + // workflow-tengo 6.7.0 fluent formula API, fed through .resources({ onCPU }). See the fb-parse note. formula := exec.formula memFormula := formula.gib(tagStatBaseGB). plus(formula.size("mic").times(4)). diff --git a/workflow/src/gather-counts.tpl.tengo b/workflow/src/gather-counts.tpl.tengo new file mode 100644 index 0000000..b446f08 --- /dev/null +++ b/workflow/src/gather-counts.tpl.tengo @@ -0,0 +1,70 @@ +// One sparse count table for the whole run, gathered from the per-sample tag-stat tables. +// +// emit_verdicts.py reads every sample at once: a panel is a property of the declared tag set rather than of +// one sample, the panel-versus-reads check runs in both directions across the run, and the QC rollup spans +// samples. So the per-sample [sampleId] -> tag-stat file map the fan-out produces is concatenated here into +// one (sampleId, cellId, tag, umiCount, totalWeight) CSV. `totalWeight` is the post-refine tag-stat's +// read-weight column, carried through for `usable_read_fraction` (`qc_measures.py`): every row here already +// passed refine-tags' FEATURE correction, so restricting `cellId` to the cell list is the only condition +// `usable_read_fraction` still has to apply. +// +// The real sampleId is injected from the resource-map key, exactly as qc-summary.tpl.tengo does. The +// per-sample execs are handed a constant sample id, so the tables cannot say which sample they came from. +// +// Rendered as a separate template because the resource map must be awaited before its per-sample keys can be +// iterated, which main's body cannot do. +// +// The keys are taken through maps.getKeys, which returns them sorted. A bare `for k, v in` over a map has no +// defined order in Tengo, so the concatenated file's resource handle would vary from run to run, and every +// node downstream of it would silently lose deduplication. + +self := import("@platforma-sdk/workflow-tengo:tpl") +ll := import("@platforma-sdk/workflow-tengo:ll") +maps := import("@platforma-sdk/workflow-tengo:maps") +pt := import("@platforma-sdk/workflow-tengo:pt") + +json := import("json") + +self.defineOutputs("countsFile") + +self.body(func(inputs) { + tagstatData := inputs.tagstatData // ResourceMap: [sampleId] -> per-sample tag-stat TSV + tags := inputs.tags // { cell, umi, feature }: the mitool tag names, so the headers are known + + inputsMap := tagstatData.inputs() + ll.assert(len(inputsMap) > 0, "no per-sample tag-stat tables to gather") + + // The gather reads every sample's distinct-UMI table, which scales with the run rather than with one + // sample, so it is not a light-queue job. + // + // A FLAT grant on a step whose input grows with the sample count, the shape that OOMs late and on someone + // else's run. It is flat because the sizing formulas `exec.builder().resources()` takes are not reachable + // from this builder: `pt.workflow().mem()` hands a plain amount to the process template. Raised to the + // same 16 GiB floor that exec uses rather than left at 8, which buys headroom without pretending to be a + // formula. Not observed failing, and the risk is stated so the next run that does fail has somewhere to + // land. + wf := pt.workflow().cpu(2).mem("16GiB") + + dfs := [] + for key in maps.getKeys(inputsMap) { + sampleId := json.decode(key)[0] + df := wf.frame(inputsMap[key], { xsvType: "tsv", inferSchema: false }) + // tag-stat -u writes one row per (cell, feature barcode) with the distinct-UMI count under + // "unique_". The FEATURE tag holds the barcode SEQUENCE, corrected against the panel by + // refine-tags, and that is the key emit_verdicts.py joins the panel on. + dfs = append(dfs, df.select([ + pt.lit(sampleId).alias("sampleId"), + pt.col(tags.cell).alias("cellId"), + pt.col(tags.feature).alias("tag"), + pt.col("unique_" + tags.umi).alias("umiCount"), + pt.col("totalWeight") + ]...)) + } + + combined := len(dfs) > 1 ? pt.concat(dfs) : dfs[0] + combined.save("counts.csv", { xsvType: "csv" }) + + return { + countsFile: wf.run().getFile("counts.csv") + } +}) diff --git a/workflow/src/gather-raw-feature-counts.tpl.tengo b/workflow/src/gather-raw-feature-counts.tpl.tengo new file mode 100644 index 0000000..c8e8fab --- /dev/null +++ b/workflow/src/gather-raw-feature-counts.tpl.tengo @@ -0,0 +1,53 @@ +// Run-wide pre-refine FEATURE counts, gathered from the per-sample raw tag-stat tables. +// +// emit_verdicts.py reads the undeclared-barcode share per sample against that sample's own +// declared tag set, so every sample's pre-refine counts have to reach it in one file with +// sampleId attached. The per-sample execs are handed a constant sample id, so the tables +// cannot say which sample they came from -- injected here from the resource-map key, the same +// way gather-counts.tpl.tengo does for the distinct-UMI counts. +// +// Rendered as a separate template because the resource map must be awaited before its +// per-sample keys can be iterated, which main's body cannot do. +// +// The keys are taken through maps.getKeys, which returns them sorted, for the same reason +// gather-counts.tpl.tengo takes them that way: a bare `for k, v in` over a map has no defined +// order in Tengo, so the concatenated file's resource handle would vary run to run and every +// node downstream of it would silently lose deduplication. + +self := import("@platforma-sdk/workflow-tengo:tpl") +ll := import("@platforma-sdk/workflow-tengo:ll") +maps := import("@platforma-sdk/workflow-tengo:maps") +pt := import("@platforma-sdk/workflow-tengo:pt") + +json := import("json") + +self.defineOutputs("rawCountsFile") + +self.body(func(inputs) { + rawFeatureData := inputs.rawFeatureData // ResourceMap: [sampleId] -> per-sample raw FEATURE tag-stat TSV + + inputsMap := rawFeatureData.inputs() + ll.assert(len(inputsMap) > 0, "no per-sample raw feature-count tables to gather") + + // Small relative to the distinct-UMI gather: one entry per distinct observed barcode rather + // than per (cell, barcode), so the same 16 GiB floor is generous here rather than tight. + wf := pt.workflow().cpu(2).mem("16GiB") + + dfs := [] + for key in maps.getKeys(inputsMap) { + sampleId := json.decode(key)[0] + df := wf.frame(inputsMap[key], { xsvType: "tsv", inferSchema: false }) + dfs = append(dfs, df.select([ + pt.lit(sampleId).alias("sampleId"), + pt.col("FEATURE"), + pt.col("totalWeight") + ]...)) + } + + combined := len(dfs) > 1 ? pt.concat(dfs) : dfs[0] + combined.save("raw_feature_counts.csv", { xsvType: "csv" }) + + return { + rawCountsFile: wf.run().getFile("raw_feature_counts.csv") + } +}) diff --git a/workflow/src/main.tpl.tengo b/workflow/src/main.tpl.tengo index fc19210..10ea02e 100644 --- a/workflow/src/main.tpl.tengo +++ b/workflow/src/main.tpl.tengo @@ -1,14 +1,14 @@ -// Feature Integration — main workflow template. +// Feature Integration -- main workflow template. // // FASTQ (feature-barcode) -> per-sample mitool (parse/refine/tag-stat -u) + per-cell-metrics Python // (processColumn -> fb-pipeline.tpl.tengo) -> import the per-cell result CSVs as the per-cell contract -// p-columns keyed [pl7.app/sampleId, pl7.app/sc/cellId, pl7.app/feature/featureId], exported to the -// result pool (consumed by VDJ Multiomic Integration) and surfaced as a results table. +// p-columns keyed [pl7.app/sampleId, pl7.app/sc/cellId, pl7.app/feature/featureId], exported to the result +// pool (consumed by VDJ Multiomic Integration) and surfaced as a results table. // -// processColumn's `Xsv` outputs import each per-sample CSV and prepend the sample axis from the -// iteration key (so the per-sample CSVs need only cell/feature columns). Modeled on -// blocks/peptide-extraction main.tpl.tengo + blocks/cell-ranger process.tpl.tengo. Column/axis specs -// live in column-specs.lib.tengo (peptide-extraction pattern). +// processColumn's `Xsv` outputs import each per-sample CSV and prepend the sample axis from the iteration +// key, so the per-sample CSVs need only cell and feature columns. Modeled on blocks/peptide-extraction +// main.tpl.tengo and blocks/cell-ranger process.tpl.tengo. Column and axis specs live in +// column-specs.lib.tengo. wf := import("@platforma-sdk/workflow-tengo:workflow") assets := import("@platforma-sdk/workflow-tengo:assets") @@ -19,24 +19,39 @@ exec := import("@platforma-sdk/workflow-tengo:exec") ll := import("@platforma-sdk/workflow-tengo:ll") file := import("@platforma-sdk/workflow-tengo:file") columnSpecs := import(":column-specs") +fanout := import(":fanout-inputs") fbPipelineTpl := assets.importTemplate(":fb-pipeline") prerunTpl := assets.importTemplate(":prerun") qcSummaryTpl := assets.importTemplate(":qc-summary") featurePropsTpl := assets.importTemplate(":fb-feature-properties") +gatherCountsTpl := assets.importTemplate(":gather-counts") +gatherRawFeatureCountsTpl := assets.importTemplate(":gather-raw-feature-counts") +verdictRunTpl := assets.importTemplate(":verdict-run") +verdictImportTpl := assets.importTemplate(":verdict-import") featurePropsSw := assets.importSoftware("@platforma-open/milaboratories.feature-integration.per-cell-metrics:feature-properties") -// Resource defaults mirror the MiXCR blocks (mixcr-analyze): 16 CPUs and a 64 GiB base per mitool -// process. FI's parse/refine are as compute- and RAM-heavy as mixcr alignment, so err large by -// default; users can still override via perProcessCPUs / perProcessMemGB (Advanced Settings). -defaultMitoolMemGB := 64 -defaultMitoolCPUs := 16 +// Per-sample mitool resource defaults. The fb-parse and fb-refine formulas add size("reads")*4 on top of +// this base and cap at 256 GiB, so the base only sets the floor for small inputs. Feature-barcode +// parse/refine are far lighter than the VDJ alignment the mixcr-analyze formula was tuned for, so the floor +// is 16 GiB, against peptide-extraction's 32 GiB fixed default. Keep it low: mitool's memory-from-limits +// launcher derives the JVM -Xms from the grant, so a 64 GiB floor picks a ~32 GiB -Xms on a tiny dataset, +// swaps on desktop RAM, and stalls at "parsing reads". CPUs default to 8, matching peptide-extraction, +// because 16 exceeds the core count of a typical desktop. Advanced Settings overrides both. +defaultMitoolMemGB := 16 +defaultMitoolCPUs := 8 -// Staging imports + exports the CSV to drive its upload before production needs it, and emits the +// Staging imports and exports the CSV to drive its upload before production needs it, and emits the // feature-name list for the control dropdown (see prerun.tpl). wf.setPreRun(prerunTpl) +// Only the FASTQ dataset is resolved here, and the single-cell V(D)J dataset deliberately is NOT. +// Everything in prepare's bundle is awaited in full, spec AND data, before the body runs. Resolving the +// clonotype dataset here would leave the per-sample mitool fan-out undeclared until the whole clonotyping +// chain had finished: two chains that run concurrently would run one after the other, roughly doubling wall +// clock, and nothing would surface it until a real dataset. The body needs only the FASTQ column. The +// clonotype linker is resolved inside the verdict stage's own child templates. wf.prepare(func(args) { return { resolvedFastq: wf.resolve(args.fbFastqRef) @@ -51,74 +66,77 @@ wf.body(func(args) { sampleAxis := inputSpec.axesSpec[0] // pl7.app/sampleId — inherited verbatim fileExtension := inputSpec.domain["pl7.app/fileExtension"] - // Import the user-uploaded tag->feature CSV and share its blob across every per-sample body - // invocation (the mapping is identical for all samples). The model guarantees the handle is set. + // Import the user-uploaded tag->feature CSV and share its blob across every per-sample body invocation, + // since the mapping is identical for all samples. The model guarantees the handle is set. csvImport := file.importFile(args.tagFeatureCsvHandle) csvFile := csvImport.file - // The mitool tag pattern, assembled + validated in the model's args() and passed through verbatim - // (see model/src/pattern.ts). It carries the CELL/UMI/FEATURE tags + R2 capture the downstream - // refine-tags/tag-stat steps reference by name; any extra flanks/spacers/anchors reach mitool as-is. + // The mitool tag pattern, assembled and validated in the model's args() and passed through verbatim (see + // model/src/pattern.ts). It carries the CELL/UMI/FEATURE tags and the R2 capture that refine-tags and + // tag-stat reference by name. Any extra flanks, spacers or anchors reach mitool as-is. pattern := args.pattern - control := args.controlFeature - hasControl := !is_undefined(control) && control != "" - dominanceThreshold := is_undefined(args.dominanceThreshold) ? 0.6 : args.dominanceThreshold - // Cell-barcode whitelist for refine-tags CELL correction. "" = de-novo (default). Always a defined - // string so the extra-input field below resolves and never stalls the body. + // The negative-control features. They designate a per-feature marker column for downstream consumers + // and never reach the per-sample metrics step. A four-state verdict asks the binding question of every + // antigen independently, so there is no winner for a control to be excluded from. + // + // SEVERAL are allowed: a panel may carry more than one control. Which one supplies the baseline is a + // different setting, and it is not this one. + controls := is_undefined(args.controlFeatures) ? [] : args.controlFeatures + // Cell-barcode whitelist for refine-tags CELL correction. "" = de-novo, the default. Always a defined + // string, so the extra-input field below resolves and never stalls the body. cellWhitelist := is_undefined(args.cellWhitelist) ? "" : args.cellWhitelist - // Off-target designation (optional, F2). The model gives offtargetProperty (an imported per-feature - // property column, e.g. antigen_class) and offtargetValues (the multi-selected values of that column - // marking a feature as off-target). Join the value list into the comma-separated string the Python - // --offtarget-values expects; "" for either = feature off (unchanged dominant call). - offtargetProperty := is_undefined(args.offtargetProperty) ? "" : args.offtargetProperty - offtargetValueList := is_undefined(args.offtargetValues) ? [] : args.offtargetValues - offtargetValues := "" - for i, v in offtargetValueList { - offtargetValues = i == 0 ? v : offtargetValues + "," + v - } - - // Per-sample body inputs. processColumn binds EVERY extra key as a body input field and waits for - // each to be set before firing the body (pframes/index.lib.tengo: `renderInputs["__extra_"+k]=v`), - // so a key whose value is `undefined` creates a field that never resolves and stalls the body. - // Therefore `control` is added only when a control feature is configured. + // Per-sample body inputs. processColumn binds EVERY extra key as a body input field and waits for each to + // be set before firing the body (pframes/index.lib.tengo: `renderInputs["__extra_"+k]=v`), so a key whose + // value is `undefined` creates a field that never resolves and stalls the body. + // + // This map is a closed list, enforced below. Everything in it describes how one sample's reads are parsed + // and counted, and nothing in it describes how the counts are read. That is what lets a change to the + // reading recover every per-sample mitool body from cache instead of re-running parse, refine-tags and + // tag-stat for every sample. extraInputs := { pattern: pattern, - tags: args.tags, // mitool tag names (CELL/UMI/FEATURE) — single source is the model (pattern.ts) - // Sample-aware mapping (optional). "" = off (global mapping). When set, the per-sample body filters - // the CSV to its own sample's rows using sampleLabels[sampleId] (sampleId from the passKey key). + tags: args.tags, // mitool tag names (CELL/UMI/FEATURE). The one source is the model (pattern.ts). + // Sample-aware mapping, optional. "" = off, meaning one global mapping. When set, the per-sample body + // filters the CSV to its own sample's rows using sampleLabels[sampleId], keyed off passKey. sampleColumn: is_undefined(args.sampleColumn) ? "" : args.sampleColumn, sampleLabels: is_undefined(args.sampleLabels) ? {} : args.sampleLabels, tagsCsv: csvFile, - dominanceThreshold: dominanceThreshold, fileExtension: fileExtension, barcodeSeqColumn: args.barcodeSeqColumn, featureNameColumn: args.featureNameColumn, - // Optional per-feature combine-mode column (multi-barcode antigens). "" = off (every feature sums - // its barcodes = OR). minUmi is the AND-mode per-barcode "fired" floor; always defined (default 1). + // Optional per-feature combine-mode column for multi-barcode antigens. "" = off, so every feature sums + // its barcodes (OR). minUmi is the AND-mode per-barcode "fired" floor, always defined at 1. combineColumn: is_undefined(args.combineColumn) ? "" : args.combineColumn, minUmi: is_undefined(args.minUmi) ? 1 : args.minUmi, - // Off-target designation (F2) — property column + comma-joined off-target values ("" = off). - offtargetColumn: offtargetProperty, - offtargetValues: offtargetValues, cellWhitelist: cellWhitelist } - if hasControl { - extraInputs.control = control - } - // Preview (dry-run) read cap. Only add the field when set (an undefined extra-input field would stall - // the per-sample body, like `control`); absent -> full run (fb-parse applies no --limit). + // Preview (dry-run) read cap. Only add the field when set: an undefined extra-input field creates a body + // field that never resolves and stalls the sample. Absent means a full run, with fb-parse applying no + // --limit. if !is_undefined(args.limitInput) { extraInputs.limitInput = args.limitInput } + // The aggregate-barcode detection knobs (qc_measures.py AGGREGATE_BARCODE_*). Same rule as limitInput + // above: added only when set, so an unset knob never creates a body field that stalls waiting for a + // value nobody will supply. qc_report.py's own argparse default applies where absent. + if !is_undefined(args.aggregateBarcodeIqrMultiplier) { + extraInputs.aggregateBarcodeIqrMultiplier = args.aggregateBarcodeIqrMultiplier + } + if !is_undefined(args.aggregateBarcodeMinUmiThreshold) { + extraInputs.aggregateBarcodeMinUmiThreshold = args.aggregateBarcodeMinUmiThreshold + } + if !is_undefined(args.aggregateBarcodeTopN) { + extraInputs.aggregateBarcodeTopN = args.aggregateBarcodeTopN + } - // processColumn outputs: the per-cell contract value columns + the per-sample QC CSV collected as a - // [sampleId] file map (assembled into a table by qc-summary.tpl, NOT imported here — a per-sample - // scalar cannot be xsv-imported inside the shared processColumn) + the per-sample QC-JSON map (read - // by the model to build the Analysis logs). Column/axis specs come from column-specs.lib.tengo. - outputs := columnSpecs.valueOutputs(blockId, sampleAxis.name, hasControl) - outputs = append(outputs, columnSpecs.perCellSummaryOutput(blockId, sampleAxis.name, hasControl)) + // processColumn outputs: the per-cell contract value columns, the per-sample QC CSV collected as a + // [sampleId] file map, and the per-sample QC-JSON map the model reads to build the Analysis logs. + // qc-summary.tpl assembles the file map into a table, because a per-sample scalar cannot be xsv-imported + // inside the shared processColumn. Column and axis specs come from column-specs.lib.tengo. + outputs := columnSpecs.valueOutputs(blockId, sampleAxis.name) + outputs = append(outputs, columnSpecs.perCellSummaryOutput(blockId, sampleAxis.name)) outputs = append(outputs, columnSpecs.qcFileMapOutput(blockId)) outputs = append(outputs, { type: "Resource", @@ -130,9 +148,8 @@ wf.body(func(args) { }, name: "qcJson" }) - // Per-sample step-report map: one mitool output file per pipeline step (parse / refine / tag-stat), - // keyed [sampleId, step]. The model reads which entries are present to derive each sample's current - // step. + // Per-sample step-report map: one mitool output file per pipeline step (parse / refine / tag-stat), keyed + // [sampleId, step]. The model derives each sample's current step from which entries are present. outputs = append(outputs, { type: "ResourceMap", spec: { @@ -145,9 +162,9 @@ wf.body(func(args) { name: "stepReports" }) // Per-sample x per-step LIVE stdout-stream map (parse / refine / tag-stat), keyed [step]. The model - // exposes these as per-step log handles (PlLogView) and scrapes them for live per-step progress. - // Same generic pl7.app/log column peptide-extraction uses, distinguished from the flat parse stream - // below by having the step axis rather than a fixed step domain. + // exposes these as per-step log handles (PlLogView) and scrapes them for live per-step progress. The same + // generic pl7.app/log column peptide-extraction uses, told apart from the flat parse stream below by + // carrying the step axis rather than a fixed step domain. outputs = append(outputs, { type: "ResourceMap", spec: { @@ -159,9 +176,9 @@ wf.body(func(args) { }, name: "stepLogs" }) - // Flat parse stdout stream (the 1-parse stream, surfaced without the step axis). Registers the moment - // the body runs — before any sample finishes — so it drives the live parse % + the early sample roster - // (peptide pattern). Same pl7.app/log column, pinned to step 1-parse via domain. + // Flat parse stdout stream: the 1-parse stream, surfaced without the step axis. It registers the moment + // the body runs, before any sample finishes, so it drives the live parse % and the early sample roster. + // Same pl7.app/log column, pinned to step 1-parse through the domain. outputs = append(outputs, { type: "Resource", spec: { @@ -172,9 +189,9 @@ wf.body(func(args) { }, name: "parseLogStream" }) - // per-cell-metrics (Python) stdout — the "4-metrics" step log, surfaced flat (keyed [sampleId] by - // processColumn). Kept out of the stepLogs map because that map is built upstream (fb-refine-tagstat) - // before the metrics step runs; the model keys this at step "4-metrics" and the UI reads it there. + // per-cell-metrics (Python) stdout: the "4-metrics" step log, surfaced flat and keyed [sampleId] by + // processColumn. Kept out of the stepLogs map because fb-refine-tagstat builds that map before the + // metrics step runs. The model keys this at step "4-metrics", and the UI reads it there. outputs = append(outputs, { type: "Resource", spec: { @@ -185,15 +202,14 @@ wf.body(func(args) { }, name: "metricsLogStream" }) - // mitool intermediates persisted by content hash (dedup / CID stability). parse/refine/tag-stat are - // deterministic, but a downstream consumer resolving through their outputs (parse/refine reports -> qc; - // parsed.mic/tagstat.tsv -> the metrics chain) walks into the streaming execs' files map, whose CID - // drifts run-to-run -> CIDConflictError (the reason stdout capture was removed from the mitool steps). - // Surfacing them as content-hashed File outputs pins their CIDs so those resolutions stay stable across - // re-renders, and downstream reuses them instead of re-running the whole block on a settings change. No - // model/UI consumer — the persistence itself is the point. Mirrors blocks/peptide-extraction. Each is a - // per-sample Resource -> processColumn yields a [sampleId] map. - for pn in ["parsedMic", "parseReport", "refineReport", "tagstatTsv"] { + // mitool intermediates persisted by content hash, for dedup and CID stability. parse, refine and tag-stat + // are deterministic, but a downstream consumer resolving through their outputs walks into the streaming + // execs' files map, whose CID drifts run to run and raises CIDConflictError. Surfacing them as + // content-hashed File outputs pins their CIDs, so those resolutions stay stable across re-renders and + // downstream reuses them instead of re-running the block on a settings change. No model or UI consumer: + // the persistence is the point. Mirrors blocks/peptide-extraction. Each is a per-sample Resource, so + // processColumn yields a [sampleId] map. + for pn in ["parsedMic", "parseReport", "refineReport", "tagstatTsv", "rawFeatureCounts"] { outputs = append(outputs, { type: "Resource", spec: { @@ -206,12 +222,12 @@ wf.body(func(args) { }) } // Same content-hash persistence for the per-cell-metrics (Python) exec's file outputs. Required because - // fb-downstream now also captures that exec's stdout (the 4-metrics log): the stream makes the exec's - // files map rate-dependent, so resolving abundance/fractions/... through it drifts the CID and throws - // CIDConflictError (poisoning the whole per-sample body). Pinning these files by content hash keeps - // those resolutions stable — the metrics analogue of the mitool loop above. A `path` maps each pin to - // the existing body output without colliding with the Xsv/File output of the same name. - for pn in ["abundance", "fractions", "consensus", "specificity", "perCellSummary"] { + // fb-downstream also captures that exec's stdout, the 4-metrics log, and the stream makes the exec's files + // map rate-dependent: resolving abundance or fractions through it drifts the CID and throws + // CIDConflictError, poisoning the whole per-sample body. Pinning these files by content hash keeps those + // resolutions stable. A `path` maps each pin to the existing body output without colliding with the Xsv or + // File output of the same name. + for pn in ["abundance", "fractions", "perCellSummary"] { outputs = append(outputs, { type: "Resource", spec: { @@ -225,9 +241,9 @@ wf.body(func(args) { }) } - // mitool resource meta (mixcr semantics): pass the base memory floor always; add the override key - // only when the user set one (Advanced Settings) so fb-pipeline treats it as a hard fixed request. - // A distinct meta when overridden is correct — it must not dedup with the formula-sized run. + // mitool resource meta, with mixcr semantics. Always pass the base memory floor. Add the override key only + // where the user set one in Advanced Settings, so fb-pipeline treats it as a fixed request. The meta + // differs when overridden, which is correct: it must not dedup with the formula-sized run. metaExtra := { mitoolBaseMemGB: defaultMitoolMemGB, mitoolCPUs: is_undefined(args.perProcessCPUs) ? defaultMitoolCPUs : args.perProcessCPUs @@ -236,6 +252,18 @@ wf.body(func(args) { metaExtra.mitoolMemOverrideGB = args.perProcessMemGB } + // Both fan-out maps are closed lists, checked here rather than reviewed later. Each key is part of every + // per-sample body's identity, so one added key costs every user a full parse, refine-tags and tag-stat + // re-run, and neither the render nor the logs says why. A verdict-stage parameter, or the resolved + // clonotype linker added for convenience, is exactly the addition that would land here. + // fanout-inputs.test.tengo asserts the lists exclude both. + ll.assert(len(fanout.unlisted(extraInputs, fanout.EXTRA_INPUT_KEYS)) == 0, + "per-sample body inputs outside the allowlist: %v. Anything the verdict stage needs belongs in the verdict stage, not in the mitool fan-out.", + fanout.unlisted(extraInputs, fanout.EXTRA_INPUT_KEYS)) + ll.assert(len(fanout.unlisted(metaExtra, fanout.META_EXTRA_KEYS)) == 0, + "per-sample resource meta keys outside the allowlist: %v.", + fanout.unlisted(metaExtra, fanout.META_EXTRA_KEYS)) + perSampleResults := pframes.processColumn( { spec: inputSpec, data: inputData }, fbPipelineTpl, @@ -270,110 +298,313 @@ wf.body(func(args) { } } - // Exported per-cell contract frame (goes to the result pool -> VDJ Multiomic Integration). This is the - // full per-(cell x feature) matrix and is UNCHANGED by the table collapse below — downstream still - // consumes abundance/fractions/consensus/specificity at [sampleId, cellId, featureId]. + // Exported per-cell contract frame, which goes to the result pool and on to VDJ Multiomic Integration. The + // full per-(cell x feature) matrix, UNCHANGED by the table collapse below: downstream still consumes + // abundance and fractions at [sampleId, cellId, featureId]. fb := pframes.pFrameBuilder() addXsvTo(fb, "abundance") addXsvTo(fb, "fractions") - addXsvTo(fb, "consensus") - if hasControl { - addXsvTo(fb, "specificity") - } finalPf := fb.build() - // Main results table (table-only): one row per [sampleId, cellId] — the consensus feature plus the - // per-cell summary aggregates (max UMI count / fraction, max specificity with a control) and the - // "feature : umi : fraction | ..." string listing every feature the cell has signal for. This - // collapses the matrix above for display; the export contract is unaffected. + // Main results table, table-only: one row per [sampleId, cellId], carrying the per-cell aggregates (max UMI + // count, max fraction) and the "feature (fraction%, umi), ..." string listing every feature the cell has + // signal for. Collapses the matrix above for display, leaving the export contract unaffected. tableFb := pframes.pFrameBuilder() - addXsvTo(tableFb, "consensus") addXsvTo(tableFb, "perCellSummary") tablePf := tableFb.build() - // Per-sample QC summary (outputs-only): concatenate the per-sample one-row QC CSVs (collected above - // as a [sampleId] file map) into one table and import it keyed [sampleId]. Done in a child template - // (qc-summary.tpl.tengo) because the file map must be awaited before its per-sample keys can be - // iterated — see its header. A per-sample scalar table cannot be produced by an xsv import. + // Per-sample QC summary, outputs-only: concatenate the per-sample one-row QC CSVs collected above as a + // [sampleId] file map into one table, and import it keyed [sampleId]. Done in a child template + // (qc-summary.tpl.tengo) because the file map must be awaited before its per-sample keys can be iterated. + // An xsv import cannot produce a per-sample scalar table. qcSummaryResult := render.create(qcSummaryTpl, { qcData: perSampleResults.outputData("qc"), sampleAxisSpec: sampleAxis }) // --- per-feature properties (A-0026) --- - // Import every EXTRA tag-CSV column (beyond the mapped barcode-sequence + feature-name roles) as a - // per-feature property and surface it on the shared feature axis, so downstream can group / filter the - // binding profile by property (A-0015). Panel-global (one value per feature, same for every sample), so - // this runs ONCE on the shared CSV blob OUTSIDE the per-sample processColumn fan-out — not per sample. - // emit_feature_properties.py writes the wide per-feature CSV + a meta JSON (property names + distinct - // values); fb-feature-properties.tpl reads the meta and builds one p-column per property keyed on the - // feature axis. The sample column, when set, is a role too (not a property) and is excluded. + // Import every EXTRA tag-CSV column, beyond the mapped barcode-sequence and feature-name roles, as a + // per-feature property on the shared feature axis, so downstream can group and filter the binding profile + // by property (A-0015). Panel-global, one value per feature for every sample, so this runs ONCE on the + // shared CSV blob OUTSIDE the per-sample processColumn fan-out. emit_feature_properties.py writes the wide + // per-feature CSV plus a meta JSON of property names and distinct values, and fb-feature-properties.tpl + // reads that meta to build one p-column per property. The sample column, where set, is a role rather than + // a property, and is excluded. featurePropsSampleColumn := is_undefined(args.sampleColumn) ? "" : args.sampleColumn - featurePropsRun := exec.builder(). + featurePropsBuilder := exec.builder(). software(featurePropsSw). cpu(1).mem("1GiB"). addFile("tags.csv", csvFile). arg("tags.csv"). arg("--csv-barcode-col").arg(args.barcodeSeqColumn). arg("--csv-feature-col").arg(args.featureNameColumn). - arg("--sample-col").arg(featurePropsSampleColumn). + arg("--sample-col").arg(featurePropsSampleColumn) + // Negative-control marker for control-aware metrics: pass each chosen control so the step emits a + // dedicated per-feature marker CSV. No flag at all where none is designated, leaving it header-only. + // REPEATED rather than comma-joined, because a feature name may contain a comma. + for c in controls { + if c != "" { + featurePropsBuilder = featurePropsBuilder.arg("--control-feature").arg(c) + } + } + featurePropsRun := featurePropsBuilder. arg("--output-prefix").arg("result"). saveFile("result_feature_properties.csv"). saveFileContent("result_feature_property_meta.json"). + saveFile("result_negative_control.csv"). run() featurePropsResult := render.create(featurePropsTpl, { propertiesFile: featurePropsRun.getFile("result_feature_properties.csv"), propertyMeta: featurePropsRun.getFileContent("result_feature_property_meta.json"), + negativeControlFile: featurePropsRun.getFile("result_negative_control.csv"), + hasControl: len(controls) > 0, blockId: blockId, traceSeedSpec: inputSpec }) - return { - outputs: { - // Collapsed per-cell results table (one row per [sampleId, cellId]). - perCellTable: pframes.exportFrame(tablePf), - // Per-sample QC summary table (outputs-only): reads parsed/matched, cells/features, UMIs. - qcSummaryTable: pframes.exportFrame(qcSummaryResult.output("qcSummaryTable")), - // Per-sample QC metrics as JSON (fb-pipeline qcJson) -> model builds the live "Analysis logs" - // (completed-sample heartbeat + run-level summary) from it. - qcJson: perSampleResults.outputData("qcJson"), - // Per-sample x per-step report-file map -> model derives each sample's current step from - // which reports are present (deterministic per-step progress). - stepReports: perSampleResults.outputData("stepReports"), - // Per-sample x per-step LIVE stdout-stream map -> model per-step log handles + live progress. - stepLogs: perSampleResults.outputData("stepLogs"), - // Flat parse stdout stream -> model live parse % + early sample roster (registers first). - parseLogStream: perSampleResults.outputData("parseLogStream"), - // per-cell-metrics (Python) stdout -> model 4-metrics step log. - metricsLogStream: perSampleResults.outputData("metricsLogStream"), - // mitool intermediates kept reachable so they're persisted by content hash (dedup / CID - // stability — see the output-spec note above). No model/UI consumer; the persistence is the point. - parsedMic: perSampleResults.outputData("parsedMic"), - parseReport: perSampleResults.outputData("parseReport"), - refineReport: perSampleResults.outputData("refineReport"), - tagstatTsv: perSampleResults.outputData("tagstatTsv"), - // metrics-exec file outputs kept reachable so they're content-hash-persisted (see the persist - // loop above) — this is what makes capturing the metrics stdout CID-safe. No model consumer. - abundancePersist: perSampleResults.outputData("abundancePersist"), - fractionsPersist: perSampleResults.outputData("fractionsPersist"), - consensusPersist: perSampleResults.outputData("consensusPersist"), - specificityPersist: perSampleResults.outputData("specificityPersist"), - perCellSummaryPersist: perSampleResults.outputData("perCellSummaryPersist"), - // Expose the tag->feature CSV import HANDLE so the model can watch it via getImportProgress() - // (model/src/index.ts). That model output is the actual upload driver: it registers the import - // with the middle-layer so the CSV bytes are pushed. Without it the CSV blob never materialises - // and every per-sample body hangs on __extra_tagsCsv. Mirrors immune-assay-data - // (`dataImportHandle: importFile.handle` + model getImportProgress). - tagFeatureCsvImportHandle: csvImport.handle - }, - exports: { - perCellFeatures: finalPf, - // Per-feature properties (A-0026), keyed on the shared feature axis (pl7.app/feature/featureId). - // A separate frame keeps the per-cell contract above unchanged; the result pool is flat, so - // these are discoverable/joinable by the feature axis downstream (VDJ Multiomic Integration - // reuses this exact feature axis for its per-feature outputs). Empty when the CSV has no extra - // columns. - featureProperties: featurePropsResult.output("featureProperties") + // --- the verdict stage --- + // + // Conditional on a chosen single-cell V(D)J dataset. Without one there is no clonotype set for a verdict + // to be about, and no way to say which receptor's linker the reading would be answered against, since the + // linker query is scoped by that dataset's anchor. The block still runs and still emits its per-cell + // contract columns, the per-sample QC and the per-feature properties. A missing input narrows what can be + // answered, and nothing fails. + verdictRun := undefined + verdictImport := undefined + if !is_undefined(args.datasetRef) { + // The reading is answered across the whole run, not per sample. A panel is a property of a declared tag + // set rather than of one sample, the panel-versus-reads check runs in both directions across the run, + // and the quality rollup spans samples and panels. So the per-sample counts are gathered into one + // table first, in a child template, because the file map must be awaited before its keys can be + // iterated. + countsGather := render.create(gatherCountsTpl, { + tagstatData: perSampleResults.outputData("tagstatTsv"), + tags: args.tags + }) + + // Pre-refine FEATURE counts, gathered the same way, for the undeclared-barcode table + // (330): refine-tags snaps every barcode onto the panel before the distinct-UMI counts + // above are even computed, so a sequence the panel never declared has to be read from + // this earlier pass or it cannot be seen at all. + rawFeatureCountsGather := render.create(gatherRawFeatureCountsTpl, { + rawFeatureData: perSampleResults.outputData("rawFeatureCounts") + }) + + // Built HERE, in the body, and not in prepare -- see the note on wf.prepare above. The bundle is passed + // to both child templates unresolved. A resolved bundle is a map with methods, which cannot be + // serialized as a template input, so each template resolves it for itself. + bundleBuilder := wf.createPBundleBuilder() + bundleBuilder.ignoreMissingDomains() + bundleBuilder.addAnchor("main", args.datasetRef) + // Resolved by NAME as a MULTI query, because the linker is infrastructure: it carries + // pl7.app/isLinkerColumn and is hidden in tables, so no user can pick it. A dataset may bring one + // linker per receptor, and verdict-linker.lib.tengo selects the one whose clonotype axis IS the + // anchor's. Choosing the dataset is choosing the receptor, so a legitimate BCR + TCR run panics + // nowhere. + bundleBuilder.addMulti({ name: "pl7.app/sc/cellLinker" }, "linker") + verdictBundle := bundleBuilder.build() + + // Only the reading's own parameters, and nothing that touched the fan-out. + verdictParams := { + barcodeSeqColumn: args.barcodeSeqColumn, + featureNameColumn: args.featureNameColumn, + sampleColumn: is_undefined(args.sampleColumn) ? "" : args.sampleColumn + } + // Optional parameters are ADDED, never defaulted to undefined. verdict-args tells an absent value -- + // leave the CLI's own default, or leave the line off -- apart from a supplied one, and a key whose value + // crosses a template boundary as undefined arrives back as JSON null, a third thing neither side means. + setParam := func(key, value) { + if !is_undefined(value) { + verdictParams[key] = value + } } + // Only meaningful alongside a sample column. With no per-sample keying the panel declares one panel for + // every sample, and there is nothing to translate. + if !is_undefined(args.sampleColumn) && args.sampleColumn != "" { + setParam("sampleLabels", args.sampleLabels) + } + setParam("roleColumn", args.roleColumn) + setParam("referenceValues", args.referenceValues) + setParam("referenceSource", args.referenceSource) + setParam("panelReferenceMinMembers", args.panelReferenceMinMembers) + setParam("distributionMinCells", args.distributionMinCells) + setParam("countFloor", args.countFloor) + setParam("boundCutoff", args.boundCutoff) + setParam("minVotingCells", args.minVotingCells) + setParam("minAgreement", args.minAgreement) + setParam("gateThreshold", args.gateThreshold) + setParam("grouping", args.grouping) + setParam("contendingGroups", args.contendingGroups) + setParam("captureMap", args.captureMap) + setParam("cellBarcodeValidWarn", args.cellBarcodeValidWarn) + setParam("cellBarcodeValidError", args.cellBarcodeValidError) + setParam("readsPerCellWarn", args.readsPerCellWarn) + setParam("aggregateBarcodeWarn", args.aggregateBarcodeWarn) + setParam("aggregateBarcodeError", args.aggregateBarcodeError) + setParam("undeclaredBarcodeWarn", args.undeclaredBarcodeWarn) + setParam("undeclaredBarcodeError", args.undeclaredBarcodeError) + setParam("usableReadWarn", args.usableReadWarn) + setParam("usableReadError", args.usableReadError) + + verdictRun = render.createEphemeral(verdictRunTpl, { + columns: verdictBundle, + datasetRef: args.datasetRef, + sampleAxisSpec: sampleAxis, + countsFile: countsGather.output("countsFile"), + panelFile: csvFile, + // The combined per-sample read QC, from the same child template that imports it as a table. Without + // it readsTotal, panelAssignedFraction and readsPerCell have no source and carry no status, + // silently, since a non-evaluation is a legitimate state rather than an error. readsPerCell is the + // only sequencing-depth alert the block ships. + qcSummaryFile: qcSummaryResult.output("qcSummaryCsv"), + rawFeatureCountsFile: rawFeatureCountsGather.output("rawCountsFile"), + params: verdictParams + }) + + // The import takes the run record as an INPUT so the specs it builds carry the comparator that actually + // SERVED. The software degrades a request it cannot honour, turning a declared comparator into none + // where the panel has no reference tag, and that choice sits in the emitted columns' domain. Specs + // built inside the exec's own template would record the request instead. + verdictImport = render.createEphemeral(verdictImportTpl, { + columns: verdictBundle, + datasetRef: args.datasetRef, + blockId: blockId, + sampleAxisSpec: sampleAxis, + traceSeedSpec: inputSpec, + runMeta: verdictRun.output("runMeta"), + verdicts: verdictRun.output("verdicts"), + setCounts: verdictRun.output("setCounts"), + identitySummary: verdictRun.output("identitySummary"), + identityPunch: verdictRun.output("identityPunch"), + cellPunch: verdictRun.output("cellPunch"), + // cellCounts is deliberately NOT passed. The per-cell per-tag counts are the run's largest table and + // no reader exists for them on either side of the block boundary, so the import template does not + // build them into columns. An input it would ignore is worse than no input, because it reads as a + // consumer that is not there. See verdict-import.tpl.tengo, where the frame is built. + cellScalars: verdictRun.output("cellScalars"), + offered: verdictRun.output("offered"), + tagIdentity: verdictRun.output("tagIdentity"), + identityLabels: verdictRun.output("identityLabels"), + identityProperties: verdictRun.output("identityProperties"), + panelLabels: verdictRun.output("panelLabels"), + samplePanel: verdictRun.output("samplePanel"), + panelMismatch: verdictRun.output("panelMismatch"), + undeclaredBarcodes: verdictRun.output("undeclaredBarcodes"), + qc: verdictRun.output("qc"), + qcSummary: verdictRun.output("qcSummary"), + qcDeciles: verdictRun.output("qcDeciles"), + qcSampleDeciles: verdictRun.output("qcSampleDeciles"), + qcBackgrounds: verdictRun.output("qcBackgrounds"), + reagents: verdictRun.output("reagents") + }) + } + + blockOutputs := { + // Collapsed per-cell results table (one row per [sampleId, cellId]). + perCellTable: pframes.exportFrame(tablePf), + // Per-sample QC metrics as JSON (fb-pipeline qcJson). The model builds the live "Analysis logs" from it: + // the completed-sample heartbeat and the run-level summary. + qcJson: perSampleResults.outputData("qcJson"), + // Per-sample x per-step report-file map. The model derives each sample's current step from which reports + // are present, which is deterministic per-step progress. + stepReports: perSampleResults.outputData("stepReports"), + // Per-sample x per-step LIVE stdout-stream map -> model per-step log handles + live progress. + stepLogs: perSampleResults.outputData("stepLogs"), + // Flat parse stdout stream -> model live parse % + early sample roster (registers first). + parseLogStream: perSampleResults.outputData("parseLogStream"), + // per-cell-metrics (Python) stdout -> model 4-metrics step log. + metricsLogStream: perSampleResults.outputData("metricsLogStream"), + // mitool intermediates kept reachable so they are persisted by content hash, for dedup and CID + // stability. See the output-spec note above. No model or UI consumer: the persistence is the point. + parsedMic: perSampleResults.outputData("parsedMic"), + parseReport: perSampleResults.outputData("parseReport"), + refineReport: perSampleResults.outputData("refineReport"), + tagstatTsv: perSampleResults.outputData("tagstatTsv"), + rawFeatureCounts: perSampleResults.outputData("rawFeatureCounts"), + // metrics-exec file outputs kept reachable so they are content-hash-persisted, per the persist loop + // above. This is what makes capturing the metrics stdout CID-safe. No model consumer. + abundancePersist: perSampleResults.outputData("abundancePersist"), + fractionsPersist: perSampleResults.outputData("fractionsPersist"), + perCellSummaryPersist: perSampleResults.outputData("perCellSummaryPersist"), + // Expose the tag->feature CSV import HANDLE so the model can watch it through getImportProgress() + // (model/src/index.ts). That model output is the upload driver: it registers the import with the + // middle-layer so the CSV bytes are pushed. Without it the CSV blob never materialises and every + // per-sample body hangs on __extra_tagsCsv. Mirrors immune-assay-data. + tagFeatureCsvImportHandle: csvImport.handle + } + + blockExports := { + perCellFeatures: finalPf, + // Per-feature properties (A-0026), keyed on the shared feature axis (pl7.app/feature/featureId). A + // separate frame keeps the per-cell contract above unchanged, and the result pool is flat, so these are + // discoverable and joinable by the feature axis downstream. VDJ Multiomic Integration reuses this exact + // feature axis for its per-feature outputs. Empty where the CSV has no extra columns. + featureProperties: featurePropsResult.output("featureProperties") + } + + // Only present where a V(D)J dataset was chosen. A template output field cannot be assigned undefined, so + // the verdict results are added rather than defaulted. + if !is_undefined(verdictImport) { + // Everything a downstream block joins to: the verdicts, the set-keyed counts that are the only family + // lead selection can see, the pivoted per-identity summary, the offered scope, the tag -> identity + // linker and the label columns. + // + // No per-cell material is among them. Reading the same experiment under another grouping is another + // execution of this block rather than a re-derivation a reader performs, so per-cell state has no + // consumer across the boundary: labelling and lead selection both read verdicts, never cells. The + // per-cell reference readings are still reported, as an OUTPUT below, because they are part of what + // this run says about itself. The per-cell per-tag counts are not built at any grain. + blockExports.antigenVerdicts = verdictImport.output("antigenVerdicts") + // The same frame as an OUTPUT, because a block's own exports are not in its own result pool. Without + // this the block that produced the verdicts would be the one place that cannot read them. + // + // NOTHING READS IT TODAY. The punchcard is the block's own view of the verdicts and reads the pivoted + // frame below instead, because a table cannot turn a (set, identity) frame into one column per + // identity. So this output is a capability held open rather than a wire in use: the first page that + // wants the verdicts unpivoted reads it without a workflow change. Removing it costs nothing outside + // this block, since the EXPORT above carries the verdicts to labelling and lead selection. + blockOutputs.antigenVerdictsTable = pframes.exportFrame(verdictImport.output("antigenVerdicts")) + // The punchcard's source: the same verdicts pivoted onto the clonotype set, one column per identity for + // the state and one for the support behind it. A table cannot pivot a (set, identity) frame into + // columns, so the shape the punchcard needs is built here rather than in the model. + blockOutputs.antigenPunchcardTable = pframes.exportFrame(verdictImport.output("punchcard")) + // The run's own report. Outputs rather than exports: these are read by this block's model and UI. + blockOutputs.antigenQcTable = pframes.exportFrame(verdictImport.output("qcTable")) + // The across-samples table: one row per sample, one column per sample-level measurement, + // carrying that sample's rolled-up status, replacing the earlier mitool-only qcSummaryTable + // (which ran unconditionally). This table is gated behind a chosen V(D)J dataset -- same as + // every other antigen output here, since the rolled-up status comes from the verdict stage + // and that stage has no clonotype set to run against until a dataset is picked. + blockOutputs.qcSummaryTable = pframes.exportFrame(verdictImport.output("qcSummaryTable")) + blockOutputs.antigenPanelMismatchTable = pframes.exportFrame(verdictImport.output("panelMismatchTable")) + // 330's own table: barcodes the reads carried that no panel declares, keyed by sequence, + // carrying the one status this run's quality surface publishes outside the measurement + // list -- the share of a sample's reads landing in undeclared barcodes. Usually empty. + blockOutputs.antigenUndeclaredBarcodesTable = pframes.exportFrame(verdictImport.output("undeclaredBarcodesTable")) + // The distributions the readout draws. Outputs rather than exports, like the report above: + // they are what this block says about its own run, and nothing downstream reads them. + blockOutputs.antigenQcDistributions = pframes.exportFrame(verdictImport.output("qcDistributions")) + blockOutputs.antigenReagentTable = pframes.exportFrame(verdictImport.output("reagentTable")) + // The per-cell reference readings, kept inside the block. One row per cell: the reading itself, and + // whether a declared gate set the cell aside. They are part of what the run reports about itself. A + // reader checks here why a cell could or could not be compared, and here the block reports the cells + // carrying a high reference reading whether or not a gate was declared. + // + // Named for what it holds rather than for its grain: it is not a table of cells. The per-cell per-tag + // counts are not built at all, because no reader exists for them on either side of the boundary and + // they are the run's largest table. See verdict-import.tpl.tengo, where that frame is assembled. + blockOutputs.antigenCellReference = pframes.exportFrame(verdictImport.output("cellReference")) + // What the run was answered under, including the comparator and cell list that actually served and every + // parameter the reading used. Read as content by the model for the run summary. + blockOutputs.antigenRunMeta = verdictRun.output("runMeta") + // Every sample-level quality measurement, keyed by sample, with that sample's rolled-up status and + // coverage. Read as content by the model. It is the only source for both the Main grid's Quality + // tag and the sample detail view's Quality Checks tab. + blockOutputs.antigenSampleQc = verdictRun.output("qcBySample") + } + + return { + outputs: blockOutputs, + exports: blockExports } }) diff --git a/workflow/src/prerun.tpl.tengo b/workflow/src/prerun.tpl.tengo index 3cb84ab..bed3436 100644 --- a/workflow/src/prerun.tpl.tengo +++ b/workflow/src/prerun.tpl.tengo @@ -1,23 +1,22 @@ // Prerun (staging) template. // -// Jobs, all driven off the user-uploaded tag->feature CSV: -// 1. Import + export the CSV so the staging phase DEMANDS its blob, which drives the client upload -// before production needs it (same as blocks/immune-assay-data prerun.tpl.tengo). The production -// workflow consumes the CSV only deep inside the processColumn fan-out, and the upload driver -// never picks up an upload requested only from inside that nested body — so without driving it -// here the per-sample mitool exec stalls forever waiting on a blob whose bytes are never pushed. -// 2. Emit the CSV's metadata (emit-csv-meta entrypoint) in ONE exec, keyed off the CSV alone: the -// column headers (-> the barcode-sequence / feature-name column dropdowns) AND the distinct -// values of every column (-> the negative-control dropdown, indexed by the chosen -// feature column). One upload-triggered exec populates both dropdowns; picking the feature column -// is then a pure model recompute over the already-emitted values, with no second exec or rerun. +// One job: import and export the user-uploaded tag->feature CSV, so that the staging phase DEMANDS its blob +// and the client uploads it before production needs it (same as blocks/immune-assay-data prerun.tpl.tengo). +// The production workflow consumes the CSV only deep inside the processColumn fan-out, and the upload driver +// never picks up an upload requested only from inside that nested body -- so without driving it here the +// per-sample mitool exec stalls forever waiting on a blob whose bytes are never pushed. +// +// This template used to run a second job: an emit-csv-meta exec that described the CSV's columns and their +// distinct values for the block's dropdowns. The UI now reads the file itself -- from disk on a local pick, +// and from the csvFile blob exported below on a remote one -- so the dropdowns fill on the pick instead of +// waiting for an upload and an exec. That removed the only exec this template had, and with it the venv the +// exec's artifact needed. +// +// The csvFile export below is therefore load-bearing twice over: it drives the upload, and it is the byte +// source the UI reads for any pick it cannot read from disk. wf := import("@platforma-sdk/workflow-tengo:workflow") file := import("@platforma-sdk/workflow-tengo:file") -exec := import("@platforma-sdk/workflow-tengo:exec") -assets := import("@platforma-sdk/workflow-tengo:assets") - -csvMetaSw := assets.importSoftware("@platforma-open/milaboratories.feature-integration.per-cell-metrics:emit-csv-meta") wf.body(func(args) { if is_undefined(args.tagFeatureCsvHandle) { @@ -26,24 +25,13 @@ wf.body(func(args) { csvImport := file.importFile(args.tagFeatureCsvHandle) - // CSV metadata (headers + per-column distinct values) -> the CSV column dropdowns and the - // negative-control dropdown. Depends only on the CSV, so it runs the moment the upload lands. - meta := exec.builder(). - software(csvMetaSw). - cpu(1).mem("1GiB"). - addFile("tags.csv", csvImport.file). - arg("tags.csv").arg("csv_meta.json"). - saveFileContent("csv_meta.json"). - run() - outputs := { csvFile: file.exportFile(csvImport.file), - csvMeta: meta.getFileContent("csv_meta.json"), - // Import handle exposed so the model can drive the CSV upload during STAGING (via a ctx.prerun - // getImportProgress driver, mirroring samples-and-data). The exportFile demand above does NOT - // push the bytes on its own; without a staging-side driver the upload only fires from the main - // render, which is unreachable until args() passes — and args() needs the CSV dropdowns this - // prerun populates from the uploaded CSV. That circular dependency is the deadlock this fixes. + // Import handle exposed so the model can drive the CSV upload during STAGING, via a ctx.prerun + // getImportProgress driver, mirroring samples-and-data. The exportFile demand above does NOT push the + // bytes on its own. Without a staging-side driver the upload only fires from the main render, which + // is unreachable until args() passes -- and args() needs the panel columns the user picks from. That + // circular dependency is the deadlock this fixes. tagFeatureCsvImportHandle: csvImport.handle } diff --git a/workflow/src/qc-summary.tpl.tengo b/workflow/src/qc-summary.tpl.tengo index 4b91c8b..61c284b 100644 --- a/workflow/src/qc-summary.tpl.tengo +++ b/workflow/src/qc-summary.tpl.tengo @@ -1,26 +1,25 @@ // Per-sample QC summary assembly (feature-integration). // -// processColumn emits the per-sample QC CSV (fb-pipeline `qc` output) as a [sampleId] -> file resource -// map (column-specs.lib.tengo qcFileMapOutput). A per-sample QC row is a scalar (0 within-sample axes), -// which an xsv import cannot emit directly (it needs >=1 within-file axis) — so this template -// concatenates the one-row CSVs into a single table (injecting the REAL sampleId per row from the map -// key, overwriting the constant "sample" the Python writes) and imports it ONCE keyed [sampleId]. +// processColumn emits the per-sample QC CSV (fb-pipeline `qc` output) as a [sampleId] -> file resource map +// (column-specs.lib.tengo qcFileMapOutput). This template concatenates the one-row CSVs into one file, +// injecting the REAL sampleId per row from the map key over the constant "sample" the Python writes. +// Its single output is that file. The verdict stage reads it through --qc-summary. // -// Mirrors blocks/mixcr-clonotyping/workflow/src/export-report.tpl.tengo: iterate a per-sample file map, -// pTabler-concat with an injected sampleId literal, then a single xsv.importFile with a sampleId axis. -// Rendered as a separate template (render.create from main.tpl) because the resource map must be -// awaited before its per-sample keys can be iterated — this cannot be done inline in main's body. +// It emits no p-frame. The across-samples table is built by verdict-import from result_qc_summary.csv, +// which carries the measurement set and the rolled-up status this template cannot produce. +// +// Rendered as a separate template because the resource map must be awaited before its per-sample keys can +// be iterated, which main's body cannot do inline. self := import("@platforma-sdk/workflow-tengo:tpl") ll := import("@platforma-sdk/workflow-tengo:ll") maps := import("@platforma-sdk/workflow-tengo:maps") -xsv := import("@platforma-sdk/workflow-tengo:pframes.xsv") pt := import("@platforma-sdk/workflow-tengo:pt") columnSpecs := import(":column-specs") json := import("json") -self.defineOutputs("qcSummaryTable") +self.defineOutputs("qcSummaryCsv") self.body(func(inputs) { qcData := inputs.qcData // ResourceMap: [sampleId] -> per-sample QC CSV file @@ -29,7 +28,10 @@ self.body(func(inputs) { columnsSpec := columnSpecs.qcSummaryColumnsSpec(sampleAxisSpec) // Metric column names to carry through from each per-sample CSV. The CSV's own sampleId column is - // intentionally NOT selected — the real sampleId is injected from the resource-map key below. + // deliberately NOT selected: the real sampleId is injected from the resource-map key below. + // qcSummaryColumnsSpec is read for its column NAMES only. The across-samples p-frame is built by + // verdict-import from result_qc_summary.csv, which carries the measurement set and the rolled-up + // status this template cannot produce. metricCols := [] for col in columnsSpec.columns { metricCols = append(metricCols, col.column) @@ -40,7 +42,7 @@ self.body(func(inputs) { wf := pt.workflow().inLightQueue().cpu(1).mem("2GiB") - // Sorted-key iteration keeps the constructed pt sub-resource canonical (dedup-stable). + // Sorted-key iteration keeps the constructed pt sub-resource canonical, and so dedup-stable. dfs := [] for key in maps.getKeys(inputsMap) { sampleId := json.decode(key)[0] @@ -58,9 +60,11 @@ self.body(func(inputs) { wfResult := wf.run() csvFile := wfResult.getFile("qc-summary.csv") - qcSummaryTable := xsv.importFile(csvFile, "csv", columnsSpec, { cpu: 1, mem: "2GiB" }) - return { - qcSummaryTable: qcSummaryTable + // The combined table as a FILE. The verdict stage reads its read counts through --qc-summary. + // Returned here rather than concatenated a second time because the two must agree: the per-sample + // sampleId is injected from the resource-map key, so a second gather could key the rows differently, + // and the verdict stage would then match no sample and evaluate nothing. + qcSummaryCsv: csvFile } }) diff --git a/workflow/src/verdict-args.lib.tengo b/workflow/src/verdict-args.lib.tengo new file mode 100644 index 0000000..f57f6bd --- /dev/null +++ b/workflow/src/verdict-args.lib.tengo @@ -0,0 +1,238 @@ +// The emit-verdicts command line, built in one place and unit-tested. +// +// Every flag this CLI takes has a default, so an unthreaded parameter is silent in both directions: the +// render succeeds, the exec succeeds, and the reading is answered under a value the user never chose. +// `--qc-summary` is the sharpest case -- without it the run has no read count at all, so `readsTotal`, +// `panelAssignedFraction` and `readsPerCell` render as declared-but-unchecked rather than failing, and +// `readsPerCell` is the only sequencing-depth alert the block ships. Building the whole argument list here, +// staged input filenames included, is what makes the threading assertable without a backend. +// +// The staged filenames are exported beside the flags that name them. verdict-run.tpl.tengo stages each file +// under the same constant it is referenced by, so a rename cannot leave a flag pointing at a file the exec +// never received. + +ll := import("@platforma-sdk/workflow-tengo:ll") +canonical := import("@platforma-sdk/workflow-tengo:canonical") + +FILE_COUNTS := "counts.csv" +FILE_PANEL := "panel.csv" +FILE_LINKER := "linker.csv" +FILE_QC_SUMMARY := "qc_summary.csv" +FILE_RAW_FEATURE_COUNTS := "raw_feature_counts.csv" +OUTPUT_PREFIX := "result" + +// The block arguments this module reads. Exported so fanout-inputs.test.tengo can assert that none of them +// has leaked into the per-sample mitool fan-out, which is what keeps the fan-out cached when only the +// reading's parameters change. Extend this list whenever a parameter is added below. +PARAMETER_NAMES := [ + "aggregateBarcodeError", + "aggregateBarcodeWarn", + "boundCutoff", + "captureMap", + "cellBarcodeValidError", + "cellBarcodeValidWarn", + "contendingGroups", + "countFloor", + "datasetRef", + "distributionMinCells", + "gateThreshold", + "grouping", + "minAgreement", + "minVotingCells", + "panelReferenceMinMembers", + "readsPerCellWarn", + "referenceSource", + "referenceValues", + "roleColumn", + "undeclaredBarcodeError", + "undeclaredBarcodeWarn", + "usableReadError", + "usableReadWarn" +] + +// Defaults mirroring the CLI's own (verdict.py DEFAULT_FLOOR, BOUND_CUTOFF, combine.py DEFAULT_MIN_VOTERS, +// DEFAULT_PANEL_MIN_MEMBERS). Restated rather than left to +// argparse, so the value that produced a run is always on the command line, where the run record and a +// re-run both see it. A parameter carried only as a CLI default drifts the moment one side changes. +DEFAULT_COUNT_FLOOR := 4 +DEFAULT_BOUND_CUTOFF := 75 +DEFAULT_MIN_VOTING_CELLS := 1 +DEFAULT_PANEL_MIN_MEMBERS := 25 +DEFAULT_DISTRIBUTION_MIN_CELLS := 300 + +// The four inherited lines (qc_measures.py DEFAULT_LINES), restated here for the same reason as the +// defaults above: the value that scored a run belongs on its own command line. +DEFAULT_CELL_BARCODE_VALID_WARN := 0.75 +DEFAULT_CELL_BARCODE_VALID_ERROR := 0.50 +DEFAULT_READS_PER_CELL_WARN := 5000 +DEFAULT_AGGREGATE_BARCODE_WARN := 0.05 +DEFAULT_AGGREGATE_BARCODE_ERROR := 1.0 +DEFAULT_UNDECLARED_BARCODE_WARN := 0.5 +DEFAULT_UNDECLARED_BARCODE_ERROR := 1.0 +DEFAULT_USABLE_READ_WARN := 0.20 +DEFAULT_USABLE_READ_ERROR := 0.0 + +_num := func(value, fallback) { + return is_undefined(value) ? fallback : value +} + +// An integer flag. int() first, because a model that sends 4.0 would otherwise reach argparse as "4.0", +// which `type=int` rejects: a whole-run failure from a value that was correct. +_intArg := func(value) { + return string(int(value)) +} + +// build: the full argument vector, positionals first. +// +// `params` is the block's args map plus `hasLinker` and `hasQcSummary`, which say whether the caller staged +// those two files. A flag naming a file the exec did not receive is a hard failure, so each is emitted only +// with its file. +build := func(params) { + args := [FILE_COUNTS, FILE_PANEL] + + add := func(flag, value) { + args = append(args, flag) + args = append(args, value) + } + + if !is_undefined(params.hasLinker) && params.hasLinker { + add("--linker", FILE_LINKER) + } + if !is_undefined(params.hasQcSummary) && params.hasQcSummary { + add("--qc-summary", FILE_QC_SUMMARY) + } + if !is_undefined(params.hasRawFeatureCounts) && params.hasRawFeatureCounts { + add("--raw-feature-counts", FILE_RAW_FEATURE_COUNTS) + } + + // The panel-file column roles. Passed even when empty: an empty --sample-col declares one panel for every + // sample, and an empty --role-column declares no comparator designation. Both are readings rather than + // omissions. + add("--barcode-col", params.barcodeSeqColumn) + add("--feature-col", params.featureNameColumn) + add("--sample-col", is_undefined(params.sampleColumn) ? "" : params.sampleColumn) + add("--role-column", is_undefined(params.roleColumn) ? "" : params.roleColumn) + + // Comma-separated, because that is what the CLI splits on. The args lambda has already de-duplicated and + // sorted the list, so the string is canonical and two equivalent selections produce one cache key. + referenceValues := "" + if !is_undefined(params.referenceValues) { + for i, value in params.referenceValues { + referenceValues = i == 0 ? value : referenceValues + "," + value + } + } + add("--reference-values", referenceValues) + + // Which baseline was asked for, and the only one the run can be answered under: a rung that cannot serve + // refuses rather than falling to another, so nothing substitutes for this. What the run established is + // read back from the run record, which also says where no baseline could be. + if !is_undefined(params.referenceSource) && params.referenceSource != "" { + add("--reference-source", params.referenceSource) + } + + add("--panel-min-members", _intArg(_num(params.panelReferenceMinMembers, DEFAULT_PANEL_MIN_MEMBERS))) + // Both conditions on the tag-distribution rung. Passed on every run, not only where that rung was asked + // for, so the run record states the numbers the reading would have used and two runs answered by + // different rungs still compare their configurations. + add("--distribution-min-cells", _intArg(_num(params.distributionMinCells, DEFAULT_DISTRIBUTION_MIN_CELLS))) + add("--floor", _intArg(_num(params.countFloor, DEFAULT_COUNT_FLOOR))) + add("--cutoff", string(_num(params.boundCutoff, DEFAULT_BOUND_CUTOFF))) + add("--min-voters", _intArg(_num(params.minVotingCells, DEFAULT_MIN_VOTING_CELLS))) + + // A switch, and it carries its value rather than standing alone. The vector is flag/value pairs: `valueOf` + // reads the next element, and the parity assertion at the end catches a dropped value. A bare flag would + // make every later value read as the wrong one's. + // + // Present only when on. Off by default: the minimum removes what is not evidence of binding, and the + // comparator is not evidence of binding. Turning it on changes no verdict, because every rung reads its + // own source raw. It changes only what the run reports as removed and emptied. + + // Off by default, and off means absent rather than zero. A floor of 0 passes every majority instead of + // skipping the check, and the two are different claims about the reading. + if !is_undefined(params.minAgreement) { + add("--min-agreement", string(params.minAgreement)) + } + // The admissibility gate. Absent and zero both mean off, and zero would set aside every cell. + if !is_undefined(params.gateThreshold) && params.gateThreshold > 0 { + add("--gate-threshold", _intArg(params.gateThreshold)) + } + + // The three JSON-valued flags are encoded key-sorted, never with the plain encoder. Tengo maps have no + // iteration order, so the plain encoder can emit one rule as two different strings on two renders. That + // changes the exec's argument list, and with it the cache key, for a run identical in every way that + // matters. + // + // A rule over declared properties, not a tag map. Absent means one identity per tag, the CLI's own + // default, so the flag is omitted rather than carrying a hand-built {"by":"tag"}. + if !is_undefined(params.grouping) { + add("--grouping", canonical.encode(params.grouping)) + } + // Groups of identities declared to contend for one binding site. The args lambda drops any group of fewer + // than two members and sorts the rest, so an empty list here means nothing contends. + if !is_undefined(params.contendingGroups) && len(params.contendingGroups) > 0 { + add("--contending", canonical.encode(params.contendingGroups)) + } + // No capture assignment reaches the block yet, so this is normally absent and every capture rolls up as + // no status and no value. Threaded already, so supplying the map is a model change alone. + if !is_undefined(params.captureMap) { + add("--capture-map", canonical.encode(params.captureMap)) + } + // The panel file names samples the way the scientist does. Everything else here is keyed by sampleId. + // Without this map the two namespaces never meet, nothing is offered to any sample that exists, and every + // verdict comes back *never asked* -- a correct answer to a question nobody asked, which is why it raises + // nothing. Passed only where the block has a sample column to key the panel by. + if !is_undefined(params.sampleLabels) { + add("--sample-labels", canonical.encode(params.sampleLabels)) + } + + // The four inherited lines. Passed on every run, not only where an operator moved one, so the run + // record states the number a status was read against. + add("--cell-barcode-valid-warn", string(_num(params.cellBarcodeValidWarn, DEFAULT_CELL_BARCODE_VALID_WARN))) + add("--cell-barcode-valid-error", string(_num(params.cellBarcodeValidError, DEFAULT_CELL_BARCODE_VALID_ERROR))) + add("--reads-per-cell-warn", string(_num(params.readsPerCellWarn, DEFAULT_READS_PER_CELL_WARN))) + add("--aggregate-barcode-warn", string(_num(params.aggregateBarcodeWarn, DEFAULT_AGGREGATE_BARCODE_WARN))) + add("--aggregate-barcode-error", string(_num(params.aggregateBarcodeError, DEFAULT_AGGREGATE_BARCODE_ERROR))) + add("--undeclared-barcode-warn", string(_num(params.undeclaredBarcodeWarn, DEFAULT_UNDECLARED_BARCODE_WARN))) + add("--undeclared-barcode-error", string(_num(params.undeclaredBarcodeError, DEFAULT_UNDECLARED_BARCODE_ERROR))) + add("--usable-read-warn", string(_num(params.usableReadWarn, DEFAULT_USABLE_READ_WARN))) + add("--usable-read-error", string(_num(params.usableReadError, DEFAULT_USABLE_READ_ERROR))) + + add("--output-prefix", OUTPUT_PREFIX) + + ll.assert(len(args) % 2 == 0, "verdict-args: every flag must carry a value") + return args +} + +// has: whether the built vector contains a flag. Used by the threading test. +has := func(args, flag) { + for a in args { + if a == flag { + return true + } + } + return false +} + +// valueOf: the value following a flag, or undefined. Used by the threading test to check that a flag +// carries what the caller asked for rather than only appearing. +valueOf := func(args, flag) { + for i, a in args { + if a == flag && i + 1 < len(args) { + return args[i + 1] + } + } + return undefined +} + +export { + FILE_COUNTS: FILE_COUNTS, + FILE_PANEL: FILE_PANEL, + FILE_LINKER: FILE_LINKER, + FILE_QC_SUMMARY: FILE_QC_SUMMARY, + FILE_RAW_FEATURE_COUNTS: FILE_RAW_FEATURE_COUNTS, + OUTPUT_PREFIX: OUTPUT_PREFIX, + PARAMETER_NAMES: PARAMETER_NAMES, + build: build, + has: has, + valueOf: valueOf +} diff --git a/workflow/src/verdict-args.test.tengo b/workflow/src/verdict-args.test.tengo new file mode 100644 index 0000000..fdf8bcf --- /dev/null +++ b/workflow/src/verdict-args.test.tengo @@ -0,0 +1,203 @@ +test := import("@platforma-sdk/workflow-tengo:test") +va := import(":verdict-args") + +// A fully configured reading. Every optional parameter is set, so a flag missing from the built vector is a +// threading defect rather than a deliberate omission. +_full := { + hasLinker: true, + hasQcSummary: true, + hasRawFeatureCounts: true, + barcodeSeqColumn: "Sequence", + featureNameColumn: "Name", + sampleColumn: "Samples", + roleColumn: "Type", + referenceValues: ["Control", "Isotype"], + referenceSource: "declared", + // Every number here differs from the library's own default for it. On the + // defaults, a builder that ignored the block's value and emitted its fallback + // would produce an identical vector, and this file would pass while the + // parameter never reached the exec. + panelReferenceMinMembers: 12, + distributionMinCells: 450, + countFloor: 7, + boundCutoff: 90, + minVotingCells: 3, + minAgreement: 0.6, + gateThreshold: 40, + grouping: { by: "property", column: "family" }, + contendingGroups: [["AgA", "AgB"]], + captureMap: { S1: "lane1" }, + cellBarcodeValidWarn: 0.8, + cellBarcodeValidError: 0.55, + readsPerCellWarn: 6000, + aggregateBarcodeWarn: 0.07, + aggregateBarcodeError: 0.9, + undeclaredBarcodeWarn: 0.6, + undeclaredBarcodeError: 0.95, + usableReadWarn: 0.25, + usableReadError: 0.05 +} + +// Every flag emit_verdicts.py declares, checked against the built vector. An unthreaded parameter fails +// nothing at render or exec time, because the CLI defaults it, so this list is the only place the omission +// shows up. +Test_every_flag_is_threaded := func() { + args := va.build(_full) + flags := [ + "--linker", "--qc-summary", "--raw-feature-counts", + "--barcode-col", "--feature-col", "--sample-col", "--role-column", "--reference-values", + "--reference-source", "--panel-min-members", "--distribution-min-cells", + "--floor", "--cutoff", + "--min-voters", "--min-agreement", "--gate-threshold", + "--grouping", "--contending", "--capture-map", "--output-prefix", + "--cell-barcode-valid-warn", "--cell-barcode-valid-error", "--reads-per-cell-warn", + "--aggregate-barcode-warn", "--aggregate-barcode-error", + "--undeclared-barcode-warn", "--undeclared-barcode-error", + "--usable-read-warn", "--usable-read-error" + ] + for flag in flags { + test.isTrue(va.has(args, flag), "argument list is missing " + flag) + } + + // Presence is not threading. A flag can appear carrying the library's own + // fallback while the block's value is dropped, which is the same silence + // this file exists to break, one step further in. + expected := { + "--barcode-col": "Sequence", + "--feature-col": "Name", + "--sample-col": "Samples", + "--role-column": "Type", + "--panel-min-members": "12", + "--distribution-min-cells": "450", + "--floor": "7", + "--cutoff": "90", + "--min-voters": "3", + "--min-agreement": "0.6", + "--gate-threshold": "40", + "--cell-barcode-valid-warn": "0.8", + "--cell-barcode-valid-error": "0.55", + "--reads-per-cell-warn": "6000", + "--aggregate-barcode-warn": "0.07", + "--aggregate-barcode-error": "0.9", + "--undeclared-barcode-warn": "0.6", + "--undeclared-barcode-error": "0.95", + "--usable-read-warn": "0.25", + "--usable-read-error": "0.05" + } + for flag, want in expected { + test.isEqual(want, string(va.valueOf(args, flag)), "wrong value threaded for " + flag) + } +} + +// The per-sample read QC is the one input whose absence is invisible. Three of the fifteen measurements have +// no other source, and they render as declared-but-unchecked rather than as an error. The flag must name the +// file the run template stages, or the exec fails on a path it never received. +Test_qc_summary_points_at_the_staged_file := func() { + args := va.build(_full) + test.isEqual(va.FILE_QC_SUMMARY, va.valueOf(args, "--qc-summary"), + "--qc-summary must name the staged per-sample QC CSV") +} + +// Same requirement for the pre-refine pass: without it the undeclared-barcode table stays the +// ordinary empty case rather than raising, and a flag naming a file the exec never received is +// still a hard failure. +Test_raw_feature_counts_points_at_the_staged_file := func() { + args := va.build(_full) + test.isEqual(va.FILE_RAW_FEATURE_COUNTS, va.valueOf(args, "--raw-feature-counts"), + "--raw-feature-counts must name the staged pre-refine FEATURE table") +} + +// The two positionals lead, and each names a staged file. +Test_positionals_lead_and_name_staged_files := func() { + args := va.build(_full) + test.isEqual(va.FILE_COUNTS, args[0], "the counts table is the first positional") + test.isEqual(va.FILE_PANEL, args[1], "the panel file is the second positional") +} + +// A flag naming a file the exec never received is a hard failure, so neither file flag may appear on its +// own. Without a dataset there is no linker, and the rest of the reading still runs. +Test_file_flags_follow_their_files := func() { + args := va.build({ barcodeSeqColumn: "Sequence", featureNameColumn: "Name" }) + test.isFalse(va.has(args, "--linker"), "the linker flag must be absent when no linker was staged") + test.isFalse(va.has(args, "--qc-summary"), "the QC flag must be absent when no QC table was staged") + test.isFalse(va.has(args, "--raw-feature-counts"), + "the raw-feature-counts flag must be absent when no pre-refine table was staged") +} + +// Off means absent, not zero. A zero gate would set aside every cell, and a zero agreement floor would pass +// every majority instead of skipping the check. Both are readings, and neither is "off". +// +// The switch carries its value like every other flag. A bare flag would make the vector odd, and `valueOf` +// would then read every later value as the wrong flag's. The parity assertion in build() catches that. +Test_the_baseline_minimum_switch_is_a_flag_value_pair := func() { + args := va.build(_full) + + off := va.build({ barcodeSeqColumn: "Sequence", featureNameColumn: "Name" }) +} + +Test_optional_lines_are_absent_when_off := func() { + args := va.build({ barcodeSeqColumn: "Sequence", featureNameColumn: "Name", gateThreshold: 0 }) + test.isFalse(va.has(args, "--gate-threshold"), "the gate must be absent when off") + test.isFalse(va.has(args, "--min-agreement"), "the agreement floor must be absent when off") + test.isFalse(va.has(args, "--grouping"), "an absent grouping rule leaves the CLI's per-tag default") + test.isFalse(va.has(args, "--contending"), "no contending groups means no flag") + test.isFalse(va.has(args, "--reference-source"), "an unchosen comparator is resolved by the software") +} + +// The parameters that shape the reading are always on the command line, defaulted here rather than left to +// argparse, so the value a run was answered under is recorded with the run. +Test_shaping_parameters_are_always_present := func() { + args := va.build({ barcodeSeqColumn: "Sequence", featureNameColumn: "Name" }) + test.isEqual("4", va.valueOf(args, "--floor"), "the count floor is stated even when defaulted") + test.isEqual("75", va.valueOf(args, "--cutoff"), "the bound cutoff is stated even when defaulted") + test.isEqual("1", va.valueOf(args, "--min-voters"), "the voter minimum is stated even when defaulted") + test.isEqual("25", va.valueOf(args, "--panel-min-members"), "the panel minimum is stated") + test.isEqual("300", va.valueOf(args, "--distribution-min-cells"), "the cell condition is stated") +} + +// The four inherited lines are stated on every run too, so a status is always readable against the +// number that produced it, defaulted or not. +Test_line_thresholds_are_always_present := func() { + args := va.build({ barcodeSeqColumn: "Sequence", featureNameColumn: "Name" }) + test.isEqual("0.75", va.valueOf(args, "--cell-barcode-valid-warn"), "the barcode-validity warn line is stated") + test.isEqual("0.5", va.valueOf(args, "--cell-barcode-valid-error"), "the barcode-validity alert line is stated") + test.isEqual("5000", va.valueOf(args, "--reads-per-cell-warn"), "the depth line is stated") + test.isEqual("0.05", va.valueOf(args, "--aggregate-barcode-warn"), "the aggregate-barcode warn line is stated") + test.isEqual("1", va.valueOf(args, "--aggregate-barcode-error"), "the aggregate-barcode alert line is stated") + test.isEqual("0.5", va.valueOf(args, "--undeclared-barcode-warn"), "the undeclared-barcode warn line is stated") + test.isEqual("1", va.valueOf(args, "--undeclared-barcode-error"), "the undeclared-barcode alert line is stated") + test.isEqual("0.2", va.valueOf(args, "--usable-read-warn"), "the usable-read warn line is stated") + test.isEqual("0", va.valueOf(args, "--usable-read-error"), "the usable-read alert line is stated") +} + +// argparse declares --floor, --min-voters, --panel-min-members and --gate-threshold +// as `type=int`, which rejects "4.0". A model that rounds to a whole number still hands it over as a JSON +// number, so the conversion happens here. +Test_integer_flags_carry_no_decimal_point := func() { + args := va.build({ + barcodeSeqColumn: "Sequence", featureNameColumn: "Name", + countFloor: 4.0, minVotingCells: 2.0, gateThreshold: 40.0, + panelReferenceMinMembers: 8.0, + distributionMinCells: 300.0 + }) + test.isEqual("4", va.valueOf(args, "--floor"), "the count floor reaches argparse as an integer") + test.isEqual("2", va.valueOf(args, "--min-voters"), "the voter minimum reaches argparse as an integer") + test.isEqual("40", va.valueOf(args, "--gate-threshold"), "the gate reaches argparse as an integer") + test.isEqual("300", va.valueOf(args, "--distribution-min-cells"), + "the cell condition reaches argparse as an integer") +} + +// The role values are comma-separated because that is what the CLI splits on, and they arrive already sorted +// so two equivalent selections produce one cache key. +Test_reference_values_are_comma_separated := func() { + args := va.build(_full) + test.isEqual("Control,Isotype", va.valueOf(args, "--reference-values"), + "role values reach the CLI comma-separated") +} + +// The grouping rule travels as JSON, so a property grouping reaches the software intact. +Test_grouping_rule_travels_as_json := func() { + args := va.build(_full) + test.isEqual("{\"by\":\"property\",\"column\":\"family\"}", va.valueOf(args, "--grouping"), + "the grouping rule is passed as JSON") +} diff --git a/workflow/src/verdict-import.tpl.tengo b/workflow/src/verdict-import.tpl.tengo new file mode 100644 index 0000000..cb68d7b --- /dev/null +++ b/workflow/src/verdict-import.tpl.tengo @@ -0,0 +1,281 @@ +// The verdict import: every table emit_verdicts.py wrote, turned into p-columns. +// +// Separate from the exec (verdict-run.tpl.tengo) because the run record has to be read as a VALUE before +// any of these specs can be built. The software resolves `referenceChoice` and `cellListSource`: a declared +// comparator degrades to none where the panel carries no reference tag, and a panel comparator degrades +// below the minimum membership. Both sit in the emitted columns' DOMAIN, because a verdict read against a +// declared reference is not the same reading as one read against the panel's own signal. As an annotation +// the choice would be excluded from column identity, and two incomparable runs would emit columns of +// identical identity and be unioned in a pool holding both. Specs built inside the exec's own template +// would record the choice that was REQUESTED. +// +// Ephemeral: it awaits the bundle, for the clonotype axis taken verbatim from the linker, and the run +// record. It receives the bundle UNRESOLVED from main.tpl.tengo, because a resolved bundle is a map with +// methods, which cannot be serialized as a template input. + +self := import("@platforma-sdk/workflow-tengo:tpl") +xsv := import("@platforma-sdk/workflow-tengo:pframes.xsv") +pframes := import("@platforma-sdk/workflow-tengo:pframes") +pSpec := import("@platforma-sdk/workflow-tengo:pframes.spec") +maps := import("@platforma-sdk/workflow-tengo:maps") +ll := import("@platforma-sdk/workflow-tengo:ll") +columnSpecs := import(":column-specs") +verdictLinker := import(":verdict-linker") + +json := import("json") + +// Returns five frames: antigenVerdicts (exported), and punchcard, cellReference, qcTable and +// panelMismatchTable (this block's own). cellReference holds one row per cell, carrying the reference +// reading and the gate outcome, and deliberately not the per-cell per-tag counts. defineOutputs is +// deliberately absent: an ephemeral template takes its output names from what the body returns and never +// checks a declared list, so declaring one would be inert. +self.awaitState("columns", "PColumnBundle") +self.awaitState("runMeta", "ResourceReady") + +self.body(func(inputs) { + blockId := inputs.blockId + sampleAxis := inputs.sampleAxisSpec + + // What actually served, read from the run record rather than from the block's arguments. + served := json.decode(string(inputs.runMeta.getData())) + + // The clonotype axis is taken VERBATIM from the linker and never rebuilt. An axis assembled here would be + // a lookalike carrying a different identity, and would join to nothing. + columns := inputs.columns + anchorSpec := columns.getSpec(inputs.datasetRef) + linkerCol := verdictLinker.pick(columns.getColumns("linker"), anchorSpec.axesSpec[1]) + setAxis := linkerCol.spec.axesSpec[2] + + cellAxis := columnSpecs.cellAxis(sampleAxis.name) + identityAxis := columnSpecs.identityAxis(blockId, served.groupingId) + tagAxis := columnSpecs.tagAxis(blockId) + panelAxis := columnSpecs.panelAxis(blockId) + + // Seeded from the FASTQ dataset spec, because these columns' subject is the antigen readout that dataset + // produced. Seeding from the linker would label them after the clonotyping run instead. The clonotyping + // root is already recorded where it belongs: pl7.app/vdj/scClonotypeKey carries its clonotyping run id in + // its own domain. makeTrace is variadic, and the linker is deliberately not one of its steps. + trace := pSpec.makeTrace(inputs.traceSeedSpec, { + type: "milaboratories.feature-integration", + id: blockId, + importance: 30, + label: "Feature Barcode Profiling" + }) + + // One import per table. Keys are prefixed with the table name because two tables can carry the same column + // id -- both label tables call their column "label" in the CSV -- and one p-frame key must mean one column. + // + // The importer's result map is walked with maps.forEach, whose key order is sorted. A bare `for k, v in` + // has no defined order in Tengo, which would make the built frame's resource handle vary run to run and + // silently cost every downstream node its deduplication. + // + // Every antigen table this block imports goes through here, which makes it the one place that can check + // them all. The check is that none keys on the legacy feature axis, which is keyed by feature NAME: a new + // column reusing it while carrying barcode sequences would keep the axis identity and invert its value + // space, so downstream joins would return wrong rows rather than fail. + addTo := func(fb, name, file, spec) { + // An input the caller forgot to pass arrives here as undefined and fails eleven frames down in + // xsv.importFile as "frameInput must be provided", naming neither the frame nor this block. Adding + // an output in verdict-run without also passing it in main is the mistake that produces it, and + // the two files are far enough apart to make it easy. + ll.assert(!is_undefined(file), "verdict-import: no file was passed for %v; check that main.tpl.tengo forwards it from verdict-run", name) + columnSpecs.guardNoLegacyFeatureAxis(spec.axes) + imported := xsv.importFile(file, "csv", spec, { splitDataAndSpec: true }) + maps.forEach(imported, func(key, column) { + fb.add(name + "/" + key, trace.inject(column.spec), column.data) + }) + } + + // --- exported: everything a downstream block joins to ------------------------------------------- + exportFb := pframes.pFrameBuilder() + + addTo(exportFb, "verdicts", inputs.verdicts, + columnSpecs.verdictsImportSpec(setAxis, identityAxis, served)) + addTo(exportFb, "setCounts", inputs.setCounts, + columnSpecs.setCountsImportSpec(setAxis, served)) + + // The pivoted per-identity summary is the only per-antigen state lead selection can see: a column carrying + // an axis the clonotype anchor does not have is dropped there with no error, so nothing keyed + // (set, identity) reaches it. The pivot costs one column per identity, so the software gates it on an + // identity count and records the limit and whether it emitted. Where it did not, the CSV carries only the + // key column, and an empty identity list yields no columns rather than a failed import. + summaryIdentities := [] + if !is_undefined(served.identitySummaryEmitted) && served.identitySummaryEmitted { + ll.assert(!is_undefined(served.identities), + "the run record reports an emitted identity summary but names no identities") + summaryIdentities = served.identities + } + if len(summaryIdentities) > 0 { + addTo(exportFb, "identitySummary", inputs.identitySummary, + columnSpecs.identitySummaryImportSpec(setAxis, summaryIdentities, served.groupingId, served)) + } + + // --- the punchcard's own frame, as an OUTPUT -------------------------------------------------------- + // + // The punchcard is the view `block-set` calls this block's own: every clonotype against every identity, + // each position in one of the four states. It needs the pivoted shape, because a table cannot pivot a + // (set, identity) frame into columns, and it needs it as an output rather than an export, because a + // block's own exports are not in its own result pool. + // + // Its cell carries the state, the two support counts, the agreement among them and, where the verdict did + // not settle, which of the seven ways it failed to, as one five-field value. No glyph on that page states + // a verdict without what it rests on. Gated on the same identity count as the summary above, so either + // both pivots are present or neither is, and the page says which rather than rendering an empty grid. + punchFb := pframes.pFrameBuilder() + if len(summaryIdentities) > 0 { + addTo(punchFb, "identityPunch", inputs.identityPunch, + columnSpecs.identityPunchImportSpec(setAxis, summaryIdentities, served.groupingId, served)) + } + + // The clonotype's own cell count, beside its name. Imported into THIS frame and not only into the export, + // because a block's own exports are not in its own result pool and the copy in the exported setCounts + // family is unreadable by this grid. Ungated by the identity count: a card with no identity columns still + // says how many cells each clonotype has. + addTo(punchFb, "setCellCount", inputs.setCounts, + columnSpecs.setCellCountImportSpec(setAxis, served)) + + // The per-cell material stays INSIDE the block: an output, never an export. Reading the same experiment + // under another grouping is another execution of this block rather than a re-derivation a reader performs, + // and labelling and lead selection read verdicts, never cells. + // + // This frame carries one row per cell: the reference reading, and whether a declared gate set the cell + // aside. Those keep their place among the run's own measurements, because a reader checks here why a cell + // could or could not be compared, and because the block must report the cells carrying a high reference + // reading whether or not a gate was declared. + // + // The per-cell per-TAG counts are deliberately NOT imported, so they are never built as columns. They are + // the largest table the run produces -- a cell x tag grain, 11-20x the rows of the sparse reads they came + // from -- and nothing reads them at either side of the boundary: not this block's model, not its UI, not + // another template here, no downstream block. Importing them would build the run's biggest p-frame for no + // reader on every verdict run. + // + // The states themselves are not unwanted. emit_verdicts writes result_cell_counts.csv and + // verdict-run.tpl.tengo collects it, so they are computed and exist within the run. What stops here is + // turning them into p-columns. cellTagCountsImportSpec stays in column-specs.lib.tengo as the spec to + // reuse if a consumer at that grain appears, because rebuilding it would risk a lookalike tag axis that + // joins to nothing. + cellRefFb := pframes.pFrameBuilder() + addTo(cellRefFb, "cellScalars", inputs.cellScalars, + columnSpecs.cellScalarsImportSpec(sampleAxis, cellAxis, served)) + + // The per-cell punchcard, into the SAME frame: one row per cell, one column per identity, that cell's own + // reading. This is the by-cell face of the expansion, and the one place the block shows WHY a clonotype's + // verdict came out as it did rather than only what it came out as. It is also why the paragraph above + // argues from the absence of a consumer rather than from the grain: this frame carries a + // per-cell-per-identity grid, and only the cell x tag table has no reader. + // + // Gated the same way as the identity pivots, and additionally on a cell count in the software, which is + // the gate that matters here: the rows are cells rather than clonotypes, so the same panel width costs an + // order of magnitude more. The run record says whether it emitted, and the page reads that rather than + // inferring it from an absent frame. + cellPunchIdentities := [] + if !is_undefined(served.cellPunchEmitted) && served.cellPunchEmitted { + ll.assert(!is_undefined(served.identities), + "the run record reports an emitted per-cell punchcard but names no identities") + cellPunchIdentities = served.identities + } + if len(cellPunchIdentities) > 0 { + addTo(cellRefFb, "cellPunch", inputs.cellPunch, + columnSpecs.cellPunchImportSpec(sampleAxis, cellAxis, cellPunchIdentities, served.groupingId, served)) + } + + // Which identities each sample was actually stained with. Without it, "never asked" is a claim a reader + // cannot check. + addTo(exportFb, "offered", inputs.offered, + columnSpecs.offeredImportSpec(sampleAxis, identityAxis)) + + // The tag -> identity linker. It lets a reader put a tag's count beside its verdict, and neither layer + // needs to know about the other. It carries no sample axis, because neither side of the join has one. See + // tagIdentityLinkerImportSpec. + addTo(exportFb, "tagIdentity", inputs.tagIdentity, + columnSpecs.tagIdentityLinkerImportSpec(tagAxis, identityAxis)) + + // The label columns. A label is satisfied only by a column NAMED pl7.app/label with exactly one axis. The + // readable names are imported as columns rather than read as values, which is why they cannot be used + // while these specs are built. + addTo(exportFb, "identityLabels", inputs.identityLabels, + columnSpecs.identityLabelsImportSpec(identityAxis)) + + // The declarations that hold of each identity, keyed the way its verdicts are. `panel-file-authority` + // requires them to travel with the reading, and a downstream reader has no other route to them: the + // per-feature property columns key on the feature NAME axis, which no verdict-stage column carries. + // + // The column names are panel data, so they come from the run record. A panel declaring nothing beyond the + // barcode and the name emits none, leaving the CSV key-only, which imports as nothing. Skipped rather + // than built empty. + if !is_undefined(served.identityProperties) && len(served.identityProperties) > 0 { + addTo(exportFb, "identityProperties", inputs.identityProperties, + columnSpecs.identityPropertyImportSpec( + identityAxis, served.identityProperties, served.identityPropertyValues)) + } + addTo(exportFb, "panelLabels", inputs.panelLabels, + columnSpecs.panelLabelsImportSpec(panelAxis)) + + // sample -> panel, so per-tag QC keyed (panel, tag) can be read back to the samples it covers. Where one + // panel covers every sample this column is constant, and still renders on every row: the SDK suppresses + // no column for being constant. + addTo(exportFb, "samplePanel", inputs.samplePanel, + columnSpecs.samplePanelImportSpec(sampleAxis)) + + // --- block-local: the run's own report ---------------------------------------------------------- + qcFb := pframes.pFrameBuilder() + addTo(qcFb, "qc", inputs.qc, columnSpecs.qcImportSpec( + columnSpecs.qcLevelAxis(blockId), + columnSpecs.qcEntityAxis(blockId), + columnSpecs.qcMeasurementAxis(blockId), + columnSpecs.panelAxis(blockId))) + + // 330-the-quality-readout's across-samples table: one row per sample, one column per + // sample-level measurement, plus the sample's own rolled-up status. Its own frame, because + // its columns are per-measurement rather than per (level, entity, measurement) like qcFb above. + qcSummaryFb := pframes.pFrameBuilder() + addTo(qcSummaryFb, "qcSummary", inputs.qcSummary, columnSpecs.qcSampleSummaryImportSpec(sampleAxis)) + + // The distributions 330 puts last in the readout, as plots rather than table rows. A number + // encoded in a detail string is one nobody can draw. + distributionFb := pframes.pFrameBuilder() + addTo(distributionFb, "qcDeciles", inputs.qcDeciles, columnSpecs.qcDecileImportSpec( + columnSpecs.qcDistributionAxis(blockId), + columnSpecs.qcDecileAxis(blockId))) + // The antigen-count distribution's own deciles, kept per sample beside the two pooled decile + // sets above. Its own column: adding a sample axis to `qcDecileValue` would re-identify that + // column rather than mint a new one. + addTo(distributionFb, "qcSampleDeciles", inputs.qcSampleDeciles, columnSpecs.qcSampleDecileImportSpec( + sampleAxis, + columnSpecs.qcDecileAxis(blockId))) + addTo(distributionFb, "qcBackgrounds", inputs.qcBackgrounds, + columnSpecs.qcBackgroundImportSpec(sampleAxis, tagAxis)) + + // One row per (panel, tag, identity). Its own frame, at its own grain: the measurement table is + // keyed (level, entity, measurement) and cannot hold a row per reagent. + reagentFb := pframes.pFrameBuilder() + addTo(reagentFb, "reagents", inputs.reagents, + columnSpecs.qcReagentImportSpec(panelAxis, tagAxis, identityAxis)) + + // The panel-versus-reads check is emitted as a p-column rather than left as a file. A mismatch report the + // user cannot see defeats its purpose. + mismatchFb := pframes.pFrameBuilder() + addTo(mismatchFb, "panelMismatch", inputs.panelMismatch, + columnSpecs.panelMismatchImportSpec(panelAxis, tagAxis)) + + // 330's own table: barcodes the reads carried that no panel declares, keyed by sequence. Its own + // frame, because it keys on (sampleId, tag) rather than any axis the tables above use, and because + // panelMismatch's "undeclared-in-panel" direction is structurally unreachable there -- refine-tags + // has already snapped every barcode onto the panel before that check's counts are built, which is + // exactly why this table reads the pre-refine pass instead. + undeclaredFb := pframes.pFrameBuilder() + addTo(undeclaredFb, "undeclaredBarcodes", inputs.undeclaredBarcodes, + columnSpecs.undeclaredBarcodeImportSpec(sampleAxis, tagAxis)) + + return { + antigenVerdicts: exportFb.build(), + punchcard: punchFb.build(), + cellReference: cellRefFb.build(), + qcTable: qcFb.build(), + qcSummaryTable: qcSummaryFb.build(), + qcDistributions: distributionFb.build(), + reagentTable: reagentFb.build(), + panelMismatchTable: mismatchFb.build(), + undeclaredBarcodesTable: undeclaredFb.build() + } +}) diff --git a/workflow/src/verdict-linker.lib.tengo b/workflow/src/verdict-linker.lib.tengo new file mode 100644 index 0000000..2c36db0 --- /dev/null +++ b/workflow/src/verdict-linker.lib.tengo @@ -0,0 +1,90 @@ +// Choosing the cell linker, and refusing the two ways it can be wrong. +// +// The linker is infrastructure: it carries pl7.app/isLinkerColumn, is hidden in tables, and so no user can +// pick it. The model stores the single-cell VDJ dataset anchor instead, and the linker is resolved from the +// bundle by name. A dataset may bring several linkers, since mixcr emits one per receptor, and the anchor is +// receptor-scoped, so the linker whose clonotype axis IS the anchor's is the one the user chose. Pick by +// that identity rather than panicking on a count, and a legitimate BCR + TCR run keeps working. +// +// Shared by verdict-run.tpl.tengo and verdict-import.tpl.tengo, which each resolve the bundle +// independently. The run template holds it resolved, which cannot be serialized as a template input, so the +// import template receives the unresolved reference and resolves it again. + +ll := import("@platforma-sdk/workflow-tengo:ll") +maps := import("@platforma-sdk/workflow-tengo:maps") +canonical := import("@platforma-sdk/workflow-tengo:canonical") + +LINKER_ANNOTATION := "pl7.app/isLinkerColumn" + +// Axis identity is name plus domain, and annotations are excluded from it. Compared through the key-sorted +// encoder, because two maps holding the same pairs are not equal to each other in Tengo. +_axisKey := func(axis) { + return canonical.encode({ + name: axis.name, + domain: is_undefined(axis.domain) ? {} : axis.domain + }) +} + +sameAxis := func(a, b) { + return _axisKey(a) == _axisKey(b) +} + +// pick: the linker belonging to the chosen dataset. +// +// `clonotypeAxis` is the anchor's own clonotype axis, its second. A cell linker is keyed +// [sampleId, cellId, scClonotypeKey], so its third axis is the one to match. +pick := func(linkerCols, clonotypeAxis) { + if len(linkerCols) == 0 { + ll.panic("Antigen binding: the selected dataset has no cell linker. Choose a single-cell V(D)J dataset produced by the Import V(D)J Data block.") + } + for col in linkerCols { + axes := col.spec.axesSpec + if len(axes) == 3 && sameAxis(axes[2], clonotypeAxis) { + return col + } + } + ll.panic("Antigen binding: none of the %d cell linkers found is keyed on the selected dataset's clonotype axis (%v). Re-select the single-cell V(D)J dataset.", + len(linkerCols), clonotypeAxis.name) +} + +// sampleAxisMismatch: "" when the linker's sample axis is the block's, otherwise a readable reason. +// +// The block's sample axis comes from the FASTQ dataset. The linker's comes from the clonotyping run, and on +// a multiplexed input that can be pl7.app/sampleGroupId instead. Joining across the two produces no rows and +// no error, so the verdicts would come out empty. Named here so the caller can panic. +sampleAxisMismatch := func(linkerSpec, sampleAxisSpec) { + linkerSampleAxis := linkerSpec.axesSpec[0] + if sameAxis(linkerSampleAxis, sampleAxisSpec) { + return "" + } + return "Antigen binding: the V(D)J dataset's sample axis (" + linkerSampleAxis.name + + ") is not the feature-barcode dataset's (" + sampleAxisSpec.name + + "). The two runs were keyed differently — commonly one was multiplexed and the other was not — so " + + "no cell would match and every verdict would come out empty. Re-run clonotyping on the same samples." +} + +// plainSpec: the linker's spec with the linker marker removed. +// +// The CSV export builds a linker index for a column flagged this way, and that index requires exactly two +// connected components, which a three-axis cell linker does not have. The export wants the axis tuple as +// flat rows, so the marker is dropped from a COPY. Mutating the resolved spec would change what every other +// reader of the bundle sees. +plainSpec := func(spec) { + annotations := {} + maps.forEach(is_undefined(spec.annotations) ? {} : spec.annotations, func(key, value) { + if key != LINKER_ANNOTATION { + annotations[key] = value + } + }) + plain := maps.clone(spec) + plain.annotations = annotations + return plain +} + +export { + LINKER_ANNOTATION: LINKER_ANNOTATION, + sameAxis: sameAxis, + pick: pick, + sampleAxisMismatch: sampleAxisMismatch, + plainSpec: plainSpec +} diff --git a/workflow/src/verdict-linker.test.tengo b/workflow/src/verdict-linker.test.tengo new file mode 100644 index 0000000..f838908 --- /dev/null +++ b/workflow/src/verdict-linker.test.tengo @@ -0,0 +1,76 @@ +test := import("@platforma-sdk/workflow-tengo:test") +linker := import(":verdict-linker") + +_sampleAxis := { name: "pl7.app/sampleId", type: "String" } +_cellAxis := { name: "pl7.app/sc/cellId", type: "String" } + +_clonotypeAxis := func(runId) { + return { + name: "pl7.app/vdj/scClonotypeKey", + type: "String", + domain: { "pl7.app/vdj/clonotypingRunId": runId } + } +} + +_linkerCol := func(clonotypeAxis) { + return { + spec: { + name: "pl7.app/sc/cellLinker", + valueType: "Int", + axesSpec: [_sampleAxis, _cellAxis, clonotypeAxis], + annotations: { "pl7.app/isLinkerColumn": "true", "pl7.app/label": "Cell linker" } + }, + data: "data-placeholder" + } +} + +// A dataset carrying one linker per receptor is legitimate, and the anchor says which receptor the user +// chose. Picking by the clonotype axis rather than by count keeps a BCR + TCR run working. +Test_pick_selects_the_anchors_own_linker := func() { + tcr := _linkerCol(_clonotypeAxis("run-tcr")) + bcr := _linkerCol(_clonotypeAxis("run-bcr")) + test.isEqual(bcr.spec, linker.pick([tcr, bcr], _clonotypeAxis("run-bcr")).spec, + "the linker keyed on the chosen dataset's clonotype axis is selected") + test.isEqual(tcr.spec, linker.pick([bcr, tcr], _clonotypeAxis("run-tcr")).spec, + "selection does not depend on the order the linkers were resolved in") +} + +// Axis identity is name plus domain. Two clonotyping runs produce axes of one name whose domains differ, and +// treating them as one would join the verdicts to the wrong clonotypes. +Test_axis_identity_includes_the_domain := func() { + test.isTrue(linker.sameAxis(_clonotypeAxis("r1"), _clonotypeAxis("r1")), "same name and domain match") + test.isFalse(linker.sameAxis(_clonotypeAxis("r1"), _clonotypeAxis("r2")), + "the same axis name under a different clonotyping run is a different axis") + test.isTrue(linker.sameAxis({ name: "pl7.app/sampleId" }, { name: "pl7.app/sampleId", domain: {} }), + "an absent domain and an empty one are the same identity") +} + +// A sample-axis mismatch produces no rows and no error, so it is named rather than left to surface as an +// empty block. +Test_sample_axis_mismatch_is_named := func() { + matched := _linkerCol(_clonotypeAxis("r1")) + test.isEqual("", linker.sampleAxisMismatch(matched.spec, _sampleAxis), + "a matching sample axis reports no mismatch") + + grouped := { + spec: { + axesSpec: [{ name: "pl7.app/sampleGroupId", type: "String" }, _cellAxis, _clonotypeAxis("r1")] + } + } + test.isFalse(linker.sampleAxisMismatch(grouped.spec, _sampleAxis) == "", + "a multiplexed sample axis is reported as a mismatch rather than joining to nothing") +} + +// The marker is dropped from a copy. Mutating the resolved spec would change what every other reader of the +// bundle sees, and the CSV export is the only consumer that needs it gone. +Test_plainSpec_copies_and_keeps_everything_else := func() { + original := _linkerCol(_clonotypeAxis("r1")).spec + plain := linker.plainSpec(original) + test.isTrue(is_undefined(plain.annotations["pl7.app/isLinkerColumn"]), + "the linker marker is removed from the exported spec") + test.isEqual("Cell linker", plain.annotations["pl7.app/label"], + "every other annotation survives") + test.isEqual("true", original.annotations["pl7.app/isLinkerColumn"], + "the resolved spec itself is untouched") + test.isEqual(original.axesSpec, plain.axesSpec, "the axis tuple is unchanged") +} diff --git a/workflow/src/verdict-run.tpl.tengo b/workflow/src/verdict-run.tpl.tengo new file mode 100644 index 0000000..0067d73 --- /dev/null +++ b/workflow/src/verdict-run.tpl.tengo @@ -0,0 +1,147 @@ +// The verdict exec, and nothing else. +// +// Split from the import (verdict-import.tpl.tengo) because the software decides which comparator SERVED the +// run, not the request: reference_by_cell degrades a declared choice to none where the panel carries no +// reference tag, and a panel choice to none below the minimum membership. That choice belongs in the emitted +// columns' domain, so the specs have to be built from the run record, and reading a value an exec has not +// yet produced needs a template boundary. +// +// Ephemeral rather than pure because it resolves a PColumnBundle, whose data this template awaits. That wait +// happens HERE and not in main's prepare: prepare's bundle is awaited in full, spec and data, before the +// body runs, so a linker there would hold the entire per-sample mitool fan-out behind the whole clonotyping +// chain -- two chains that run concurrently, serialized, with nothing surfacing it until a real dataset. + +self := import("@platforma-sdk/workflow-tengo:tpl") +exec := import("@platforma-sdk/workflow-tengo:exec") +assets := import("@platforma-sdk/workflow-tengo:assets") +pframes := import("@platforma-sdk/workflow-tengo:pframes") +maps := import("@platforma-sdk/workflow-tengo:maps") +ll := import("@platforma-sdk/workflow-tengo:ll") +va := import(":verdict-args") +verdictLinker := import(":verdict-linker") + +verdictsSw := assets.importSoftware("@platforma-open/milaboratories.feature-integration.per-cell-metrics:emit-verdicts") + +// Every table emit_verdicts.py writes, paired with the output field it is returned on. One list, used as +// the saveFile set and as the returned keys, so a table cannot be saved and then quietly dropped, or named +// one thing here and another in the import. +RESULT_TABLES := [ + { out: "verdicts", file: "result_verdicts.csv" }, + { out: "setCounts", file: "result_set_counts.csv" }, + { out: "identitySummary", file: "result_identity_summary.csv" }, + { out: "identityPunch", file: "result_identity_punch.csv" }, + { out: "cellPunch", file: "result_cell_punch.csv" }, + { out: "cellCounts", file: "result_cell_counts.csv" }, + { out: "cellScalars", file: "result_cell_scalars.csv" }, + { out: "offered", file: "result_offered.csv" }, + { out: "tagIdentity", file: "result_tag_identity.csv" }, + { out: "identityLabels", file: "result_identity_labels.csv" }, + { out: "identityProperties", file: "result_identity_properties.csv" }, + { out: "panelLabels", file: "result_panel_labels.csv" }, + { out: "samplePanel", file: "result_sample_panel.csv" }, + { out: "panelMismatch", file: "result_panel_mismatch.csv" }, + { out: "undeclaredBarcodes", file: "result_undeclared_barcodes.csv" }, + { out: "qc", file: "result_qc.csv" }, + { out: "qcSummary", file: "result_qc_summary.csv" }, + { out: "qcDeciles", file: "result_qc_deciles.csv" }, + { out: "qcSampleDeciles", file: "result_qc_sample_deciles.csv" }, + { out: "qcBackgrounds", file: "result_qc_backgrounds.csv" }, + { out: "reagents", file: "result_reagents.csv" } +] + +self.awaitState("columns", "PColumnBundle") + +self.body(func(inputs) { + columns := inputs.columns + + // The anchor is the single-cell V(D)J dataset, with axes [sampleId, scClonotypeKey]. Its clonotype axis + // identifies the linker belonging to the receptor the user chose. + anchorSpec := columns.getSpec(inputs.datasetRef) + ll.assert(len(anchorSpec.axesSpec) == 2, + "Antigen binding: the selected dataset is keyed on %d axes, not the expected [sampleId, scClonotypeKey].", + len(anchorSpec.axesSpec)) + + linkerCol := verdictLinker.pick(columns.getColumns("linker"), anchorSpec.axesSpec[1]) + linkerAxes := linkerCol.spec.axesSpec + + mismatch := verdictLinker.sampleAxisMismatch(linkerCol.spec, inputs.sampleAxisSpec) + if mismatch != "" { + ll.panic("%v", mismatch) + } + + // The linker as flat rows: (sampleId, cellId, setId). Headers are bound to axes by NAME and never by + // position, so a linker whose axes were declared in another order still exports correctly. The software + // ignores the value column: the axis tuple is the whole content. + linkerCsv := pframes.csvFileBuilder() + linkerCsv.add({ spec: verdictLinker.plainSpec(linkerCol.spec), data: linkerCol.data }, { header: "linker" }) + linkerCsv.setAxisHeader(linkerAxes[0].name, "sampleId") + linkerCsv.setAxisHeader(linkerAxes[1].name, "cellId") + linkerCsv.setAxisHeader(linkerAxes[2].name, "setId") + linkerCsv.cpu(1) + // Flat, and it scales with the total cell count of the run: every cell of every sample appears once. The + // csv builder's `mem` takes a plain amount rather than one of the sizing formulas the exec below uses, so + // this cannot track its input. Raised to the exec's own 16 GiB floor rather than left at 8. Not observed + // failing, and stated so a future OOM here is recognised rather than re-diagnosed. + linkerCsv.mem("16GiB") + linkerCsvFile := linkerCsv.build() + + // The reading holds one entry per analysed cell and one per (cell, tag) count, so it scales with the + // gathered counts table rather than with any one sample. The same sizing shape the per-cell metrics step + // uses, on the same floor. + formula := exec.formula + memFormula := formula.gib(16). + plus(formula.size("counts").times(8)). + between(formula.gib(16), formula.gib(256)). + staticFallback(formula.gib(16)) + + params := maps.merge(inputs.params, { + hasLinker: true, + hasQcSummary: !is_undefined(inputs.qcSummaryFile), + hasRawFeatureCounts: !is_undefined(inputs.rawFeatureCountsFile) + }) + + b := exec.builder(). + software(verdictsSw). + resources({ onCPU: { cpu: 4, ram: memFormula } }). + // polars sizes its thread pool to every host core by default. Cap it to the granted CPU. + envWithVar("POLARS_MAX_THREADS", "{system.cpu}"). + addFile(va.FILE_COUNTS, inputs.countsFile, { tag: "counts" }). + addFile(va.FILE_PANEL, inputs.panelFile). + addFile(va.FILE_LINKER, linkerCsvFile) + + // The per-sample read QC. Without it readsTotal, panelAssignedFraction and readsPerCell have no source and + // carry no status and no value, silently, because that is a legitimate state rather than an error. + // readsPerCell is the only sequencing-depth alert the block ships. + if params.hasQcSummary { + b = b.addFile(va.FILE_QC_SUMMARY, inputs.qcSummaryFile) + } + // The pre-refine pass: without it the undeclared-barcode table stays the ordinary empty + // case rather than raising, because a run wired without this file has not checked for an + // undeclared barcode, which is a different fact from having checked and found none. + if params.hasRawFeatureCounts { + b = b.addFile(va.FILE_RAW_FEATURE_COUNTS, inputs.rawFeatureCountsFile) + } + + for a in va.build(params) { + b = b.arg(a) + } + for table in RESULT_TABLES { + b = b.saveFile(table.file) + } + run := b.saveFileContent("result_run_meta.json").saveFileContent("result_qc_by_sample.json").run() + + // The run record and the per-sample quality report travel as content because the model reads each as a + // VALUE. The tables travel as files because the import template hands them to xsv.importFile. An exec + // result object is not serializable as a template output, so each table is returned on its own field. + result := { + runMeta: run.getFileContent("result_run_meta.json"), + // The sample-level measurements keyed by sample, for the sample detail view's Quality Checks tab. + // That view holds one sample at a time and resolves it synchronously. result_qc.csv carries the + // same measurements for every other reader. + qcBySample: run.getFileContent("result_qc_by_sample.json") + } + for table in RESULT_TABLES { + result[table.out] = run.getFile(table.file) + } + return result +})