MOSAIC is a scalable pipeline that ingests per-sample gVCF files into a hybrid
PostgreSQL + Apache Iceberg store. Metadata and variant sites live in
PostgreSQL; per-sample genotype data is stored as Parquet files in
S3-compatible object storage and managed by an Iceberg catalog. The result is a
cross-sample store with population-level aggregates, ready for downstream
analysis at scale. The design, its invariants and the implementation status are described in
docs/ARCHITECTURE.md; benchmark results on up to 3,202 whole
genomes are in docs/benchmarks.md.
- Python 3.10+
- PostgreSQL 10+ (uses declarative table partitioning; 12+ recommended)
- Apache Iceberg REST catalog (or AWS Glue)
- S3-compatible object storage (MinIO for local dev, AWS S3, etc.)
- For
mosaic ingest: themosaic-ingest-workerbinary, built fromrust/with a Rust toolchain (rustup; the version is pinned inrust/rust-toolchain.toml), onPATHor named byMOSAIC_INGEST_WORKER. It is required:mosaic ingesthas no other shard worker (without it, it exits 1 before any work). The Docker image builds it in its own stage. - For
joint_call.py: themosaic-joint-callbinary, which the samerust/workspace builds intorust/target/release/beside the ingest worker;joint_call.pyfinds it throughMOSAIC_JOINT_CALL_WORKERorPATH. It is required:joint_call.pyis its driver and has no engine of its own to fall back to (see 6. Joint calling)
pip install .This installs the mosaic command and its dependencies, including
vcflib from PyPI. For a fully pinned
environment, use the lockfile:
pip install -r requirements.txt && pip install --no-deps .Then build the Rust ingest worker and joint caller and put them on PATH (or
export MOSAIC_INGEST_WORKER and MOSAIC_JOINT_CALL_WORKER with their paths):
cd rust && cargo build --release && cd ..
export PATH="$PWD/rust/target/release:$PATH" # or MOSAIC_INGEST_WORKER=$PWD/rust/target/release/mosaic-ingest-workerThe Python package does not depend on them (pip install is unchanged), but the
commands do: mosaic ingest needs mosaic-ingest-worker and joint_call.py needs
mosaic-joint-call (each exits 1 with the remedy when its binary is missing).
A local Docker stack (PostgreSQL + Iceberg REST + MinIO) for development lives
in iceberg_test/:
cd iceberg_test && docker-compose up -dA single mosaic command with subcommands:
mosaic init ──> mosaic ingest ──> [mosaic register] ──> mosaic aggregate ──> mosaic index
└──> mosaic maintain
removing samples: mosaic remove ──> mosaic aggregate ──> mosaic index + mosaic maintain
| Subcommand | Purpose |
|---|---|
mosaic init <gvcf> [--chromosomes ...] |
Create PostgreSQL + Iceberg tables and populate contigs (run once) |
mosaic ingest <gvcfs...> |
Ingest one or more gVCF files (sharded, in the mosaic-ingest-worker binary) |
mosaic register |
Register the files of batches ingested with --defer-register with Iceberg, many files per commit, and complete those batches (a no-op otherwise) |
mosaic aggregate |
Reduce the per-batch Iceberg partials into the PostgreSQL aggregates |
mosaic remove <samples...> |
Remove samples: tombstone them, write their contribution as a negative batch for the next mosaic aggregate, and delete their per-sample rows (an initial version) |
mosaic index |
Check PostgreSQL indexes: re-create the alleles GiST if missing (run after aggregation) |
mosaic maintain |
Iceberg maintenance: rewrite each table's manifests, expire old snapshots and delete the files only they needed, optionally sweep orphan files (run after aggregation) |
mosaic clean |
Reset PostgreSQL + Iceberg state |
All configuration is read from environment variables. The defaults connect to
the local Docker stack in iceberg_test/, so no variables need to be set for
local development. The default credentials (mysecretpassword / admin /
password) are the zero-config local-dev values baked into
iceberg_test/docker-compose.yml — they are not secrets.
| Variable | Default | Purpose |
|---|---|---|
MOSAIC_DB_NAME |
postgres |
Database name |
MOSAIC_DB_USER |
postgres |
Database user |
MOSAIC_DB_PASSWORD |
mysecretpassword |
Database password (local-dev default) |
MOSAIC_DB_HOST |
localhost |
Database host |
MOSAIC_DB_PORT |
5432 |
Database port |
MOSAIC_DB_SSLMODE |
(unset) | Set to require for RDS |
MOSAIC_DB_SSLCERT |
(unset) | Path to SSL CA cert (required when MOSAIC_DB_SSLMODE is set) |
MOSAIC_ICEBERG_TYPE |
rest |
Catalog type (rest or glue) |
MOSAIC_ICEBERG_URI |
http://localhost:8181 |
REST catalog URI (ignored for Glue) |
MOSAIC_S3_ENDPOINT |
(unset) | Custom S3 endpoint for MinIO; omit for AWS S3. When unset under the rest catalog the effective default is minio:9000 (pyiceberg) / localhost:9000 (boto3). |
MOSAIC_S3_ACCESS_KEY_ID |
admin |
S3 access key (local-dev default) |
MOSAIC_S3_SECRET_ACCESS_KEY |
password |
S3 secret key (local-dev default) |
MOSAIC_S3_PATH_STYLE |
true |
Path-style S3 access (for MinIO; omit for AWS S3) |
MOSAIC_S3_REGION |
us-east-1 |
AWS region |
MOSAIC_S3_BUCKET |
warehouse |
S3 bucket for Parquet data files |
MOSAIC_WAREHOUSE |
(unset) | Iceberg warehouse URI (e.g. s3://my-warehouse/) |
MOSAIC_ICEBERG_NAMESPACE |
mosaic |
Iceberg namespace |
MOSAIC_PARQUET_ROW_GROUP_ROWS |
65536 |
Rows per row group in sample_ref_blocks data files (read by mosaic ingest's writer and by mosaic init, which sets the table's write.parquet.row-group-limit for pyiceberg's rewrites: set it identically for both, or not at all) |
MOSAIC_PARQUET_SITE_ROW_GROUP_ROWS |
8192 |
The same for sample_site_info |
MOSAIC_INGEST_WORKER |
(unset) | Path of the mosaic-ingest-worker binary mosaic ingest runs; unset, mosaic-ingest-worker on PATH |
MOSAIC_JOINT_CALL_WORKER |
(unset) | Path of the mosaic-joint-call binary joint_call.py runs; unset, mosaic-joint-call on PATH |
# DB configuration
export MOSAIC_DB_HOST=my-db.xxxxxxxx.us-east-1.rds.amazonaws.com
export MOSAIC_DB_PASSWORD=<from-secrets-manager>
export MOSAIC_DB_SSLMODE=require
export MOSAIC_DB_SSLCERT=/path/to/global-bundle.pem
# Iceberg / S3 configuration
export MOSAIC_ICEBERG_TYPE=glue
export MOSAIC_S3_REGION=us-east-1
export MOSAIC_S3_BUCKET=my-warehouse-bucket
export MOSAIC_WAREHOUSE=s3://my-warehouse-bucket/
# MOSAIC_S3_ENDPOINT intentionally unset — uses AWS S3 natively.
# AWS credentials come from the instance/task role.Size the RDS volume for the aggregate's finalize, or enable storage autoscaling
(MaxAllocatedStorage) so a full disk grows instead of killing the run. Genome-wide
at 3,202 samples the database is 69.8 GB (65 GiB) after the aggregate, which ran on a 100 GiB
volume that also held four smaller test databases (16.6 GB) and ended with 15.6 GB
free.
workflow/conf/awsbatch.config and workflow/nextflow.config carry the settings
those runs used (maxForks = 30, pos_batch_size = 100000), except
aggregate_threads, which defaults to the task's cpus. At 3,202 samples
genome-wide pass --aggregate_threads 2 (or 1): at the default 4 threads the
aggregate runs out of memory in a 12 GiB task (see AGGREGATE memory and
storage below).
Creates PostgreSQL tables (without indexes) and Iceberg tables, and populates
the contigs table from a gVCF header.
mosaic init sample.gvcf.gz # all chromosomes
mosaic init sample.gvcf.gz --chromosomes chr21,chr22Ingests one or more gVCF files in the Rust worker (mosaic-ingest-worker). Each genomic
shard is processed independently; per-sample genotype data and per-batch
aggregate partials are written as Parquet, uploaded to S3, and registered with
Iceberg.
mosaic ingest sample1.gvcf.gz sample2.gvcf.gz
mosaic ingest *.gvcf.gz --threads 8 --shard-size 500000 --pos-batch-size 150000| Option | Default | Description |
|---|---|---|
--threads |
CPU count | Parallel shards, and then parallel combined files: threads of the one worker process |
--shard-size |
10,000,000 | Genomic region size (bp) per shard |
--pos-batch-size |
10,000,000 | Genomic window size (bp) per Parquet file |
--aggregate-parquet-size |
268,435,456 | Target aggregated Parquet file size (bytes). The per-sample tables pack whole samples up to it: a sample larger than the target gets a file of its own |
--region |
(unset) | Scope the run (and its checkpoint) to a region, e.g. chr21 |
--defer-register |
off | Upload the Parquet but leave its Iceberg registration to mosaic register (see below) |
--ref-gq-bands |
off | Ref-block banding: store a sample's adjacent reference blocks of one GT class and GQ band as one row (see below). A comma-separated list of ascending GQ bounds in [1, 99] (20,60) or all |
Ingestion supports checkpoint/resume: if a run is interrupted, re-running
with the same sample set resumes from the last completed shard.
--pos-batch-size controls memory use (smaller = lower memory). The CLI
defaults (10 Mbp shard and window) are deliberately coarse — the Nextflow
path and workflow/bin/fetch_and_ingest.py override them to --shard-size 500000 --pos-batch-size 150000, which we benchmarked at 200 gVCFs per batch. A bare
mosaic ingest with the defaults uses far more memory and can OOM on a large
batch; lower --pos-batch-size if that happens.
The worker. The shard work (reading the gVCFs, merging them by position and
writing the local Parquet) runs in the Rust worker: one mosaic-ingest-worker
process per batch, its shards on --threads threads. After its last shard the same
process combines the local files into the data files (the per-sample tables'
whole-sample files, the partials' concatenation) on --threads threads; mosaic ingest uploads each file as it is finished (4 upload threads) and registers them.
The test suite checks the worker's output against frozen signatures (see
Testing).
By default each mosaic ingest ends by committing its data files to the six
shared Iceberg tables itself. With many concurrent ingests those commits
contend, and each one rewrites the table's metadata.json. With
--defer-register the batch uploads its Parquet as usual, records the file
list in the PostgreSQL table ingest_files and ends in status uploaded;
mosaic register later registers every pending file with one committer per
table (--files-per-commit, default 500, files per commit) and marks the
batches complete. Until then mosaic aggregate and the readers do not see an
uploaded batch (mosaic aggregate prints a warning); re-running mosaic ingest
on it does nothing. The registrar registers to Iceberg first and records that in
PostgreSQL second, so an interrupted mosaic register is simply re-run. Only one
registrar runs at a time (a PostgreSQL advisory lock). The Nextflow workflow uses
this path. Registration does not expire Iceberg snapshots; mosaic maintain does.
mosaic ingest --defer-register batch1/*.gvcf.gz
mosaic ingest --defer-register batch2/*.gvcf.gz
mosaic register # one commit per table for both batchesMany tools band gVCFs finely: on the six chr21 test samples the median reference block is
2–3 bp long, and the per-sample reference blocks are most of the lake's bytes (80%
genome-wide at 100 samples). --ref-gq-bands merges them at ingest. It is off by
default: without it every reference block is stored as the gVCF has it.
- What merges. A sample's reference blocks that are adjacent (no gap, no variant record
between them), of the same GT class (hom-ref or no-call), whose GQ falls in the same band,
and that lie in the same
--pos-batch-sizewindow, become onesample_ref_blocksrow. The spec lists the bands' exclusive upper bounds, as GATK's--gvcf-gq-bandsdoes:20,60is the bands [0, 20), [20, 60) and [60, 99];allis one band. A block without GQ (no FORMAT/GQ and no PL) is in a band of its own. Under banding a block that crosses a window edge is cut there. - The merged values.
dpis the length-weighted mean of the blocks' depths (each the gVCF block's MIN_DP), rounded half up: the depth summed over the run's positions is kept up to that rounding, and a run whose blocks share one depth keeps it exactly. AD and PL are those of the block with the lowest GQ (ties: the first), so no position is reported better than it was;sample_countsis the same for all of them. The row's contribution tobatch_ref_blocksis emitted from the same values, so the partials stay derivable from the per-sample rows andmosaic removestays exact. - What changes in the output. PG
ref_blocks.dp, so INFO/DP and QD at variant sites (up or down), and the FORMAT DP, AD and PL thatjoint_call.pyfills in for a sample with a reference block at the site. What does not: GT, AC, AN and every other INFO field,site_info,alleles, the genotype matrix (batch_genotypes) andsample_qc, which read the raw records. - Provenance. The batch records its normalized spec in
ingestion_progress.ref_gq_bands(NULL: no banding). A resumed batch must use the same value (mosaic ingestexits 1 otherwise, naming the stored one). Batches ingested with different settings can be aggregated together.
Measured on the six chr21 samples as three 2-sample batches (--region chr21 --threads 2,
CLI defaults):
| off | 20 |
20,60 |
all |
|
|---|---|---|---|---|
sample_ref_blocks rows (all six samples) |
12,591,956 | 1,838,511 (6.8× fewer) | 5,002,492 (2.5×) | 473,893 (26.6×) |
sample_ref_blocks bytes |
112.3 MB | 17.6 MB (6.4× smaller) | 46.1 MB (2.4×) | 4.5 MB (24.7×) |
| the six ingest tables' bytes | 393.8 MB | 285.8 MB | 318.9 MB | 271.0 MB |
mosaic remove of one sample: wall (ref-block rows it reads) |
16.2–16.9 s (2,966,248) | 13.6 s (362,684) | 14.4 s (1,199,826) | 13.6 s (97,733) |
| VCF records changed (of 132,382) | — | 93,233 (70%) | 89,997 (68%) | 99,981 (76%) |
| INFO/DP changed at: up / down (median absolute change) | — | 63% of records: 40,761 / 42,995 (4 = 2.4%) | 58%: 39,027 / 37,713 (3 = 1.7%) | 70%: 43,242 / 48,851 (4 = 2.8%) |
| INFO/DP summed over all records | — | −0.14% | −0.07% | −0.72% |
| ref-filled calls with GQ < 20 (GQ from the PL) | 14.1% | 14.1% | 14.1% | 59.5% |
n=2 vs GVCFtyper, formatkeys=GT,PL,DP infokeys=DP,AC,AF,AN |
+/- 128 = 56644 |
+/- 23830 = 32942 |
+/- 21867 = 34905 |
+/- 28005 = 28767 |
n=2 vs GVCFtyper, formatkeys=GT infokeys=AC,AF,AN |
+/- 117 = 56655 |
+/- 117 = 56655 |
+/- 117 = 56655 |
+/- 117 = 56655 |
GT, AC, AN and every INFO field but DP and QD are identical at every record under every
setting, so the genotype-level comparison does not move; the DP and PL comparisons do,
since GVCFtyper reports the gVCF's own blocks and vcfdiff counts any change (at 20,60
the PL alone differs at 19,512 records, DP alone at 16,499). The mean moves INFO/DP both
ways by a few percent: the largest relative changes are at low-depth sites (at 20,60,
90% of the changed sites move by at most 5.3%). The raw blocks of these
samples have a median length of 2–3 bp; 20,60 merges them 2.4–2.7×, all 19–32×. What
bounds a merged run is the variant records between blocks, not the windows: at the
workflow's 150 kb windows the row counts are within 0.4% of these.
Measured on AWS at 20:
- Storage: on chr21 × 3,202 the ingest tables are 75% smaller (
sample_ref_blocks8.8× fewer bytes). Genome-wide at 3,202 samples the whole lake is 1.50 TB, 11.6% of the 12.98 TB of source gVCFs, and the per-sample tables are 0.39 GB per sample against 4.05 GB of gVCF. Unbanded, the ingest tables take 2.17 GB per sample genome-wide (at 100 samples). - Calls: on chr21 × 3,202, the joint calls of 17 sets of 200 samples, banded against unbanded, are identical in GT, AC, AN, AF, QUAL, FS, MQ, SOR and InbreedingCoeff at all 5.32 billion genotypes. Only INFO/DP and QD and the FORMAT DP / AD / PL of hom-ref and no-call fills differ.
After all ingest batches are registered, reduce the per-batch Iceberg partials into the cross-sample PostgreSQL aggregates — a materialized tier-tree reduce with its own checkpoint/resume.
mosaic aggregate # resumes an interrupted run if present
mosaic aggregate --fan-in 100 --threads 8
mosaic aggregate --rebuild # force a fresh run instead of resumingmosaic remove takes samples out of the cohort; the following mosaic aggregate
applies it, after which the PostgreSQL aggregates and the joint-called output are
those of a fresh ingest of the remaining samples. It is an initial version: its
exactness is tested on six chr21 samples, it has run on AWS at chr21 × 100
samples (two samples removed in 31 s, the aggregate that applied it 30 s), and it is
not part of the Nextflow workflow.
mosaic remove HG00100 HG00101 # tombstone, negative batch, delete rows
mosaic aggregate # folds the negative batch in (incremental)
mosaic index
mosaic maintain --older-than-hours 0 # the deleted rows' files leave S3 hereWhat it does, in order (each step resumable: re-run mosaic remove with the same
samples after a failure):
- Tombstones the samples (
samples.status = 'removed'): from then on the readers hide their rows andmosaic ingestrefuses them. Until the followingmosaic aggregatethe aggregates still count them, sojoint_call.pyrefuses the stale cohort. - Re-derives their exact contribution from their own per-sample rows (each row
stores it) and writes it, negated, as one ingest batch of kind
removal(batch_site_info,batch_allele_counts,batch_ref_blocks), registered with Iceberg like an ingest batch. Nothing is subtracted in place: the nextmosaic aggregatesums it like any added batch and drops the sites, alleles and reference positions only the removed samples had. - Deletes their rows from
sample_site_infoandsample_ref_blocks(a copy-on-write delete that rewrites only the files holding them). The bytes leave S3 whenmosaic maintainexpires the snapshots that still name the old files.
It refuses, changing nothing, on an unknown or already removed sample, a sample in an
ingest batch that is not complete (an uploaded one needs mosaic register
first), another unfinished removal, and a run while mosaic register or mosaic maintain holds the shared lock. It does not scrub batch_genotypes: a removed
sample keeps its slot and bytes there, and the reader (mosaic/genotypes.py) hides
its column. A removed sample cannot be re-added.
| Option | Default | Description |
|---|---|---|
--threads |
CPU count | Parallel derive workers |
--shard-size |
10,000,000 | Genomic window (bp) of one derive unit |
--aggregate-parquet-size |
268,435,456 | Target size (bytes) of the negative batch's data files |
--span-margin |
100,000 | How far before a window the derive reads deletion rows whose split spans reach into it; a longer span is an error |
Apart from mosaic clean, no other command deletes an Iceberg file: every commit
leaves a manifest list (and usually a manifest) behind, and the staging that
mosaic aggregate prunes when a run completes is only recorded as deleted, so its
Parquet stays in S3. mosaic maintain bounds that growth. For each Iceberg table
it:
- sets
write.metadata.delete-after-commit.enabledif the table lacks it (mosaic initsets it on every table), so each commit deletes themetadata.jsonversions beyond the last 100; - rewrites the manifests: the current snapshot's data manifests are merged into
files of up to
--manifest-target-size-bytes(default 8 MiB); - expires the snapshots the retention policy drops and deletes every file only
they needed, plus the files that a retained snapshot names only as deleted
(the pruned staging). A snapshot is retained when it is the current one, one
of the
--retain-lastnewest (default 1), younger than--older-than-hours(default 120), or a branch head or tag; - with
--orphans, deletes the objects under the upload prefixes (s3://<MOSAIC_S3_BUCKET>/<table>/) and the table locations that no snapshot of any table in any namespace of the catalog references, that are not pending iningest_files, and that are older than--orphan-grace-hours(default 72). The sweep assumes those prefixes belong to this catalog.
--older-than-hours 0 keeps only each table's current snapshot. That is safe
when no reader is running, and the Nextflow workflow passes it. The default keeps
five days of snapshots for a reader started by hand that planned its scans from an
older snapshot. mosaic maintain shares mosaic register's advisory lock, and
refuses to run while an aggregation run is in_progress (--force overrides, for
a run that was killed). --dry-run commits nothing and deletes nothing. Data
files are not compacted.
mosaic maintain --dry-run # report what would be done
mosaic maintain --older-than-hours 0 # no reader is running
mosaic maintain --older-than-hours 0 --orphans # also sweep orphan filesRun after aggregation. This creates the secondary indexes; the tables' primary-key and
unique constraints already exist from mosaic init. Creating the secondary indexes after
ingestion, rather than before, avoids per-row index maintenance during the bulk loads.
mosaic indexjoint_call.py writes a multi-sample VCF of every active sample (or of a subset,
--samples) over every ingested contig from the PostgreSQL aggregates and the per-sample Iceberg tables. It is also the
regression oracle of the e2e tier; it is not part of the mosaic command. It refuses (exit 2) a cohort whose latest
aggregation run is not complete or does not cover every completed ingest batch, unless
--allow-stale. Its output has been compared with Sentieon GVCFtyper on one 2-sample chr21 test
only (chr21:1-30M; the figures are in the banding table above), and it does not handle
haploid calls (see the known limitation below).
joint_call.py is the driver of mosaic-joint-call, the Rust joint caller: it runs
the gate, exports the PostgreSQL side, plans the per-sample data files and runs the
binary ($MOSAIC_JOINT_CALL_WORKER, else PATH; missing: exit 1 with the remedy,
before any work), which writes the VCF.
python joint_call.py cohort.vcf.gz --threads 4
python joint_call.py slice.vcf.gz --region chr21:10000000-12000000 # a slice
python joint_call.py two.vcf.gz --samples HG00100,HG00245 # a sample subset
python joint_call.py own.vcf.gz --samples HG00100,HG00245 \
--trim-alt-alleles --min-ac 1 # its own sites and alleles| Option | Default | Description |
|---|---|---|
output |
stdout | The VCF; BGZF-compressed when the path ends in .gz |
--region |
the whole scope | CONTIG[:START-END], 1-based, inclusive; the records equal the whole run's in the slice |
--allow-stale |
off | Joint-call a cohort the gate would refuse |
--threads |
the CPU count | Units run concurrently; the output is written in genomic order |
--window |
10,000,000 | The unit width in bp, 0 = one unit per contig; the output does not depend on it |
--unit-buffer-bytes |
67,108,864 (64 MiB) | A unit that is not the first unit not yet written keeps its compressed output in memory up to this many bytes, then spills it to a file of its own (0 or 1: every chunk spills); the output does not depend on it |
--spill-dir |
beside the output | Spill under this directory instead, in a directory of the driver's own naming (<DIR>/mosaic-joint-call-spill-<token>) |
--max-alt-alleles |
6 | The ALT cap: a record with more ALT alleles keeps this many (below); 0 = no cap |
--samples |
every active sample | NAME[,NAME...]: joint-call only these active samples (below) |
--samples-file |
— | As --samples, one name per line (blank lines ignored); not with --samples |
--trim-alt-alleles |
off | An ALT allele no written genotype names (AC 0) leaves the record (below) |
--min-ac |
0 (off) | K: write only the records whose written genotypes hold at least K ALT observations (below) |
--cohort-state |
— | Print the cohort state as one JSON line and exit; no VCF (below) |
--export-cohort |
— | DIR: write DIR/cohort.json; no VCF (below) |
--export-contig |
— | DIR with --region CONTIG: write that contig's exports into DIR; no VCF (below) |
--from-export |
— | DIR: joint-call from the export, with no PostgreSQL connection (below) |
The joint caller reads each data file once per unit, by position, as a k-way merge
across the files, and its units run on --threads threads, each compressing its own
output in 4 MiB text chunks. The head unit (the first unit not yet written) writes
its chunks to the output as it goes; every other unit keeps them in memory up to
--unit-buffer-bytes and then moves them to a spill file of its own and appends there;
when the head ends, the units after it that already finished are copied to the output in
order and the next unit still running becomes the head. A unit starts only while it is
fewer than 2 × threads units past the head, so at most 2 × threads − 1 units hold a
buffer (each at most the bound plus one chunk) or a spill file, and at one thread nothing
is buffered or spilled. Spill files go in <output>.spill-<token> beside a regular output
path (the filesystem sized for the output; at most the in-flight units' compressed bytes),
in the driver's temporary directory for a /dev/... output (stdout included), or under
--spill-dir; the directory is created at the first spill and removed when the run ends,
whether it succeeds or fails. The compressed output is the same bytes at every
--threads and every bound. It writes no tabix index beside a .gz output; index it
with tabix -p vcf or bcftools index -t. On the six chr21 test samples the whole run
takes about 5 s at one thread (2.7 s at four).
The ALT cap (--max-alt-alleles N, default 6, 0 = none). A record with more than N
ALT alleles (* counts as one) is written as a projection of the uncapped record: it
keeps the N ALTs with the largest AC (the counts of its genotypes; ties to the earlier
ALT), in their original order; a genotype that carries a dropped ALT is written ./.;
AD, F1R2 and F2R1 keep the entries of REF and the kept ALTs; PL keeps the entries of the
genotypes over those alleles and, when the uncapped PL held a 0 and all the kept entries
are above 0, has their minimum subtracted (a ref-block fill, whose PL may hold no 0, is
never shifted); DP is unchanged; AC, AN and AF are recomputed from the written genotypes, and
QUAL and the other INFO fields (DP, QD, MQ, FS, SOR, InbreedingCoeff, the site's) are
unchanged. A record with at most N ALTs is the uncapped record exactly. A genotype the cap
writes ./. keeps its AD, DP and PL over the kept alleles, and that PL is normalized over
the kept genotypes like any call's, so it no longer points at the dropped call (a het
carrier of a dropped allele usually has its 0 at the hom-ref genotype). Under a cap the
header holds ##mosaic_max_alt_alleles=N (before the ##contig lines), the catalog
cross-check leaves the capped records out (it says how many), and a closing line reports
the records capped, the genotypes written ./., the ALT observations on the dropped
alleles and the kept-allele observations lost with those genotypes. On the six chr21 test
samples the default caps 311 of 132,382 records and writes 60 genotypes ./.;
--max-alt-alleles 0 gives the uncapped output.
A sample subset (--samples A,B or --samples-file FILE). The run joint-calls only
the named active samples, and each record is the whole cohort's record projected onto
their columns: the columns are the subset's in the cohort header's order (whatever order
is given); every cohort site is still a record, and QUAL and the site's INFO fields (DP,
QD, MQ, FS, SOR, InbreedingCoeff) stay the cohort's; * leaves the ALT list when no
sample of the subset carries it (AD, F1R2 and F2R1 then lose its entry, PL its genotypes
under the rule above; a record whose only ALT was * gets ALT . and no AC or AF); a FORMAT key no
sample of the subset carries is dropped; AC, AN and AF are the subset's (bcftools view -s of the whole cohort's output gives the same genotypes, AC and AN). Under the ALT cap a
subset record is capped like any record, ranked by the subset's own AC. The header is the
cohort's but for the #CHROM line. Only the per-sample data files whose sample_id
bounds can hold a subset sample are read (with the whole-sample layout, a subset of one
batch reads that batch's files only), while the PostgreSQL export and the per-site work
still cover every cohort site. The catalog cross-check is skipped (its counts are the
whole cohort's) and a closing line says so. An unknown, removed or repeated name, or an
empty list, exits 2 naming them, before the gate. Naming every active sample gives the
whole cohort's text.
Trimming ALTs and a minimum AC (--trim-alt-alleles, --min-ac K; both off by
default; the names are bcftools view's). The options apply in a fixed order, each to
the record the previous one wrote: the subset (--samples), then the trim, then the ALT
cap, then --min-ac. --trim-alt-alleles drops every ALT allele that no written
genotype names (AC 0): the other ALTs keep their order, the GTs are re-indexed (no
genotype changes: none names a dropped allele), AD, F1R2 and F2R1 keep the entries of REF
and the kept ALTs, PL the entries of the genotypes over them under the rule above, AC and
AF lose the dropped entries, and AN, QUAL and the other INFO fields are unchanged. A record
left without ALTs is written with ALT . and no AC or AF key, as bcftools writes such a
record (unless --min-ac leaves it
out); a ref-block fill there keeps its 0/0 PL, shifted to 0 only when the fill held a 0 at
another genotype. The cap then ranks the trimmed record, so it no longer keeps an ALT of AC
0 to fill its N. --min-ac K writes a record only when the ALT observations of its
written genotypes (the sum of its INFO/AC, after the subset, the trim and the cap) are at
least K, so every record of the output satisfies it on its own INFO/AC; --min-ac 1 with
--samples keeps the sites variable in the subset. A record the cap leaves without its
only carrier (written ./.) is left out at --min-ac 1. The records left out are skipped
before their calls are built, but their sites are still read (the run still walks every
cohort site). Each option set adds a header line before the ##contig lines, in the order
the options apply: ##mosaic_trim_alt_alleles=true, ##mosaic_max_alt_alleles=N,
##mosaic_min_ac=K; and a closing line after the cross-check's, in the same order:
ALT trim (--trim-alt-alleles): R records trimmed, A ALT alleles with AC 0 dropped, the
cap's line, Minimum AC (--min-ac K): L records left out (fewer ALT observations in their written genotypes). The cap's counts are over the trimmed record when both are set, and
the trim's and the cap's counts include the records --min-ac leaves out. The catalog
cross-check compares a trimmed record on its untrimmed alleles, so the trim alone does not
change its lines; it does not count a record --min-ac leaves out. At
--max-alt-alleles 0, bcftools view [-s S] [--min-ac 1] --trim-alt-alleles of the
whole cohort's output gives the same sites, ALTs, genotypes, AC and AN as [--min-ac 1] --trim-alt-alleles [--samples S] (bcftools cuts PL without the shift), and bcftools
reads the output, its records without ALTs included, with no warning. On the six chr21 test
samples (132,382 records) the trim changes 4,469 records and drops 6,799 ALTs, and under
the default cap 47 records are still capped (311 without the trim); --min-ac 1 leaves
out 3 records of the whole cohort and 45,479 for a two-sample subset; the time is
unchanged (2.2–2.3 s at four threads). Against GVCFtyper on two samples the native trim in
place of bcftools view --trim-alt-alleles gives the same concordance as the bcftools
trim (+/- 128 = 56644, the same with both trims; with no trim at all, +/- 1614).
A cohort export (--cohort-state, --export-cohort DIR, --export-contig DIR --region CONTIG, --from-export DIR). A joint call reads PostgreSQL on every run; an export
reads it once and then serves any number of calls that open no PostgreSQL connection.
--cohort-state runs the driver's
checks (the gate, the --samples names) and prints the cohort state as one JSON line on
stdout: the aggregation run, its status and time, the completed batches, the sample counts
and a digest of the samples, and the stale reason; the line is the same until the cohort
changes. --export-cohort DIR writes DIR/cohort.json (the state, the contigs, the active
and removed samples; a stale cohort only with --allow-stale, which records the reason).
--export-contig DIR --region CONTIG writes that contig's sites and alleles exports over the
whole contig (sites.c<index>.parquet, alleles.c<index>.parquet, then
export.c<index>.json, its completion marker), after checking in the same transaction that
the database's cohort state is cohort.json's (another: exit 2, make a new export).
--from-export DIR is a joint call with every option above but --allow-stale: the contigs,
the samples and the exports come from the directory, which it does not change; it refuses
(exit 2, before any store is read) a directory of another format, a contig in scope with
sites whose files are missing or of another cohort state, and a bad sample name. The
per-sample Iceberg tables are still read from the lake. The output equals the PostgreSQL
path's byte for byte, closing lines included, for every option set.
python joint_call.py --cohort-state # one JSON line on stdout
python joint_call.py --export-cohort /scratch/exp
python joint_call.py --export-contig /scratch/exp --region chr21
python joint_call.py --from-export /scratch/exp two.chr21.vcf.gz \
--region chr21 --samples-file two.names # no PostgreSQL connectionMemory and disk at cohort scale. --unit-buffer-bytes bounds the output a run holds,
and the ALT cap bounds a site's calls and a record's text: every call is built over the
kept alleles only, so a capped record's PL has at most (N + 1)(N + 2) / 2 values per
sample. Measured on AWS (S3 and RDS) at 3,202 samples on chr21 (the 1000 Genomes DRAGEN
gVCFs): the defaults fit a 14 GiB memory limit, so no --threads or --window setting is
needed for memory at that scale. A larger chromosome needs fewer threads: the whole
cohort of chr1 × 3,202 does not fit 12 GiB at 4 threads and fits at 2 (below). Memory
follows the samples in a call, so a smaller sample set fits easily: a set of 200 samples
on chr1 × 3,202 takes about 5.5 min at 1.7 GB, a set of 800 25 min at 3.9 GB.
| Call (the default cap and window) | --threads |
Wall | Peak RSS | Output (.gz) |
|---|---|---|---|---|
| chr21 × 3,202 samples, 1,662,044 records | 4 | 30 min 19 s | 6.4 GB | 28.8 GB |
| the same | 1 | 78 min 57 s | 2.4 GB | the same bytes |
| the genome × 100 samples, 18,071,827 records | 4 | 29 min 27 s | 4.0 GB | 13.0 GB |
chr1 × 3,202 samples (banded at 20), 9,657,568 records |
4 | out of memory in 12 GiB | — | — |
| the same | 2 | 3.6 h | 8.2 GB | 173 GB |
Memory grows with the units in flight: at 3,202 samples about 1.3 GB per unit over about
1 GB held once (the reader keeps a row group and a decoded batch per data file, 386 files
there). The uncapped output (--max-alt-alleles 0) needs far more: at 3,202 samples it
completed at one thread with a peak of 13.8 GB (12.8 GiB of the 14), reached at a 605-ALT
site whose record is 2.13 GB of text (bcftools +fill-tags stops at such a record).
The spill copy (expected). With more than one thread, a unit whose compressed output
exceeds --unit-buffer-bytes has usually written most of it to its spill file before the
units ahead of it are done; when its turn comes, one thread copies the spill file into the
output. About (threads − 1) / threads of a large output is therefore written twice. At
3,202 samples and 4 threads, 21.2 GB of the 28.8 GB output was spilled, the spill directory
reached 25.8 GB, and the copies took 292 s of the 30 min. This is the cost of an output
whose bytes do not depend on --threads. Leave free space about the size of the output
beside it, or point --spill-dir at a filesystem that has it. Nothing is spilled at one
thread, or while every unit's output stays under the bound (the genome call of 100 samples
spilled nothing).
Known limitation: haploid calls (chrX outside the PARs, chrY). The joint caller writes diploid genotypes only and does not yet handle a gVCF's haploid genotypes, which callers commonly write for a male sample on chrX outside the pseudoautosomal regions and on chrY:
- a haploid ALT call is written
./.. Its AD is kept, and its PL, which has one value per allele, is spread over the diploid genotype slots with990fills; - a haploid reference block is filled
0/0.
So on those regions a record's AC and AF miss the haploid ALT calls, and its AN counts each
haploid reference call as two alleles. The catalog cross-check reports such a record as
other: in the genome-wide call of 100 samples above, 399,172 of chrX's 613,607 records
and 17,434 of chrY's 30,192. Do not rely on the joint-called genotypes, AC, AN or AF of
chrX outside the PARs or of chrY for a cohort with haploid calls. The autosomes and the
PARs are not affected, and a diploid sample's own genotypes on chrX are written as its gVCF
carries them. The ingested data is not affected either: the PostgreSQL alleles counts
include the haploid calls, a haploid call's sample_site_info row has gt_1 set and
gt_2 NULL, and the genotype matrix (batch_genotypes) and sample_qc.n_haploid keep the
ploidy. sample_ref_blocks stores no ploidy for a reference block, which a fix has to
address.
mosaic clean drops MOSAIC's PostgreSQL tables and resets the Iceberg tables.
In local mode (MOSAIC_ICEBERG_TYPE=rest) it restarts the Docker stack; in
cloud mode it drops the Iceberg tables and deletes the S3 data files: the
upload prefixes and each dropped table's own location in the warehouse. Because
the cloud reset is destructive to shared infrastructure (RDS + Glue + S3), it
requires --yes:
mosaic clean # local Docker stack
mosaic clean --yes # cloud (Glue + S3 + RDS) — confirmation requiredOnly MOSAIC's own tables are dropped, so it is safe to point at a shared database; a partial reset exits non-zero rather than reporting success.
The Nextflow project lives in workflow/ (main.nf, nextflow.config, the
conf/ profiles, and the bin/ helper). workflow/main.nf orchestrates the
full ingest → register → aggregate → {index, maintain} flow as a
pipeline: it fans a sample list out into parallel INGEST_BATCH tasks (each a
mosaic ingest --defer-register by default), then — once every ingest finishes —
runs a single mosaic register and mosaic aggregate, and after it mosaic index
and mosaic maintain --older-than-hours 0 (independent of each other).
mosaic init remains a manual one-time step (it needs a reference gVCF header and
the desired chromosome scope). Two profiles are provided: local (host machine
against the iceberg_test/ Docker stack) and awsbatch (AWS Batch jobs). A second
workflow, workflow/joint_call.nf, joint-calls the ingested cohort after that
(below).
# One-time schema setup (manual), then run the pipeline
mosaic init <reference-sample.gvcf.gz> --chromosomes chr21
nextflow run workflow/main.nf -profile local --samples samples.txt| Parameter | Default | Purpose |
|---|---|---|
--samples |
(required) | Newline-delimited sample-name file (an s3:// URL on awsbatch) |
--batch_size |
200 | Samples per INGEST_BATCH task |
--region |
(unset) | Comma-separated chromosomes to fan out over, e.g. chr1,chr2 |
--defer_register |
true |
INGEST_BATCH runs mosaic ingest --defer-register, and REGISTER registers every batch's files. false: each task registers its own files and REGISTER reports nothing to register |
--register_files_per_commit |
500 | mosaic register --files-per-commit (min 1) |
--fan_in |
100 | Batches merged per tier in mosaic aggregate (min 2); >= #batches is a single pass, smaller forces more tiers |
--shard_size |
500,000 | Passed through to mosaic ingest |
--pos_batch_size |
100,000 | Passed through to mosaic ingest (the value the 3,202-sample run completed with) |
--ref_gq_bands |
(unset: off) | mosaic ingest --ref-gq-bands, passed through when set (ref-block banding, e.g. 20,60 or all; see above). Keep it unchanged between a failed run and its -resume |
--ref_shard_size |
20,000,000 | mosaic aggregate --ref-shard-size (bp): the ref_blocks window; one unit per (chromosome, window) holds ~17 bytes per window base (~340 MB per thread at the default). 0 = one window per chromosome (~4 GB on chr1), discouraged. Output is identical regardless |
--aggregate_threads |
null (task cpus) |
mosaic aggregate --threads (reduce concurrency). On 3,202 samples genome-wide 4 threads run out of memory in a 12 GiB task and 2 complete (64 min, 12.84 GB peak): set 2, or 1 on a larger cohort. Only the main process commits to Iceberg (once per barrier and table), so threads do not contend on commits |
--aggregate_cpus / --aggregate_memory |
4 / 12.GB |
AGGREGATE task resources (awsbatch withName). Raise memory for a very large cohort only alongside a compute environment with matching instances |
--maintain |
true |
Run the MAINTAIN step (mosaic maintain) after AGGREGATE; false skips it |
--maintain_older_than_hours |
0 | mosaic maintain --older-than-hours. 0 keeps only each table's current snapshot, so the staging the aggregate just pruned is reclaimed in the same run; safe because the pipeline runs no reader meanwhile |
--maintain_orphans |
false |
Also pass --orphans (sweep the upload prefixes and table locations for unreferenced objects) |
--maintain_orphan_grace_hours |
72 | mosaic maintain --orphan-grace-hours, with --maintain_orphans |
Each mosaic ingest invocation is one batch, so the total number of batches is
#sample-batches × #regions. When that exceeds --fan_in, aggregate performs
a multi-tier merge.
AGGREGATE memory and storage. Memory is bounded in code, but not by a constant:
- finalize: streams the whole cross-sample result to PG +
agg_*in row chunks; - reduce: each worker's accumulator is bounded by output keys (variant sites or alleles on one chromosome), not by input rows.
Per-worker memory therefore still grows with the number of distinct sites per chromosome,
and peak memory is roughly --aggregate_threads × that.
Settings for the full 1000 Genomes cohort. Measured on AWS in a 4-cpu / 12 GiB task:
| Cohort | --aggregate_threads |
Wall | Peak RSS |
|---|---|---|---|
| chr21 × 3,202 | 4 | 80 s | 4.0 GB |
| genome × 100 | 4 | 6 min | 5.6 GB |
chr1 × 3,202 (banded at 20) |
4 | 6.4 min | 10.0 GB |
genome × 3,202 (banded at 20) |
4 | out of memory within about a minute | — |
genome × 3,202 (banded at 20) |
2 | 64 min | 12.84 GB, about 45 MB under the limit |
So at 3,202 samples genome-wide set --aggregate_threads 2; beyond that, 1 thread
(unmeasured) or larger instances with --aggregate_memory. An aggregate killed for memory
leaves the store consistent: rerun with -resume and fewer threads.
The finalize builds each chromosome's new partitions before dropping the old ones, so it also needs transient PostgreSQL storage: genome-wide at 3,202 samples the database is 69.8 GB (65 GiB) and the run fit a 100 GiB RDS volume. Plan the RDS size ahead, or enable storage autoscaling.
--ref_shard_size. Keep the 20 Mbp default. 0 (one window per chromosome) needs
~4 GB of arrays per thread on chr1; much smaller windows multiply the Iceberg staging
commits, which contend on one table and cause commit-conflict retries at genome scale.
A second workflow beside main.nf, sharing nextflow.config, the two profiles and
workflow/bin/, joint-calls an ingested, aggregated cohort as one VCF per sample set
and chromosome with joint_call.py (section 6), after the ingestion run has finished.
PostgreSQL is read once per run, by
its first three steps; the call tasks open no PostgreSQL connection:
COHORT_STATE:joint_call.py --cohort-stateover the whole sample list, on every run. A stale cohort or a bad sample name stops the run here (exit 2).EXPORT_COHORT:--export-cohort(cohort.json: the generation, the contigs, the samples).EXPORT_CONTIG, one per chromosome with sites:--export-contig, at most--joint_call_export_forksat a time.JOINT_CALL, one per set and chromosome:--from-export ... --region <chromosome> --samples-file <set>to the task's disk,bcftools index -tand a check of the index's record count against the driver's, then the outputs are uploaded.JOINT_CALL_SUMMARY:joint_call_summary.tsv.
Each task runs workflow/bin/joint_call_task.py, which finds the driver as
$MOSAIC_JOINT_CALL_DRIVER, else joint_call.py on PATH (the image carries it in
/opt/mosaic/bin), else the repository root (a local run from a checkout); the driver
finds mosaic-joint-call as section 6 says.
nextflow run workflow/joint_call.nf -profile local \
--samples samples.txt --batch_size 200 --region chr21,chr22
nextflow run workflow/joint_call.nf -profile awsbatch \
--samples s3://.../samples.txt --batch_size 200The sets. --samples is read as main.nf reads it (blank lines and lines starting
with # skipped) and cut into consecutive sets of --batch_size: set_0001, set_0002,
…, the last one holding the remainder. --batch_size is this run's own value: a set may
hold part of an ingest batch, several of them or the whole list (the joint caller plans a
set's data files wherever its samples were ingested). Without --samples there is one
set, cohort, of every active sample, called without a names file (so the catalog
cross-check runs). The chromosomes are those of the cohort that have sites; --region a,b,c (whole chromosomes only) narrows them, and an entry that is not a contig of the
cohort stops the run before any contig export. A chromosome without sites gets no export
and no task, and the log says so.
The outputs, all written by the helper (to a local directory or s3://):
<joint_call_out>/gen<agg_run_id>/
export/cohort.json, sites.c<index>.parquet, alleles.c<index>.parquet, export.c<index>.json
set_0001/set_0001.samples.txt
set_0001/set_0001.chr21.vcf.gz
set_0001/set_0001.chr21.vcf.gz.tbi
set_0001/set_0001.chr21.summary.json
...
joint_call_summary.tsv
A file is named by the chromosome when its name is plain (letters, digits, ., _,
-), else by c<index>. A task's summary.json is written last, after the VCF, its
index and the names file, and is its completion marker: a VCF without one is not a
finished output (a failed task leaves none). It holds the set, the chromosome, the sample
count, the options, the generation and cohort state, the record count, the VCF's bytes,
the driver's mosaic-joint-call: line and closing lines, the seconds per step (driver,
index, publish) and the published paths. joint_call_summary.tsv has one row per set and
chromosome (set, chromosome, contig index, samples, records, VCF bytes, the seconds per
step, the VCF's path), written when every task is done. export/ is a copy of the export
the tasks used. The task logs go to --log_dir as for main.nf.
| Parameter | Default | Purpose |
|---|---|---|
--samples |
(unset: the whole cohort) | The sample list; cut into sets of --batch_size |
--batch_size |
200 | Samples per set (min 1) |
--region |
(unset: every chromosome with sites) | Comma-separated chromosome names (no ranges) |
--joint_call_out |
./joint_calls (local), s3://<s3_bucket>/joint-calls/ (awsbatch) |
The output prefix; a local path is made absolute |
--joint_call_max_alt_alleles |
(unset: the driver's 6) | --max-alt-alleles; 0 = no cap |
--joint_call_trim_alt_alleles |
false |
--trim-alt-alleles |
--joint_call_min_ac |
(unset: off) | --min-ac |
--joint_call_window |
(unset: 10 Mbp) | --window |
--joint_call_threads |
(unset: the task's cpus) | --threads |
--joint_call_index |
true |
Write each VCF's .tbi and check its record count |
--joint_call_export_forks |
2 | Chromosome exports in flight: the only load on PostgreSQL |
--joint_call_max_forks |
(unset: the profile's limit, 30 on awsbatch) | JOINT_CALL tasks in flight |
--joint_call_cpus / --joint_call_memory |
4 / 12.GB |
A JOINT_CALL task's resources |
Give these on the command line (--name value). In a -c file a params block is read
after the configuration has computed what it derives from parameters, so there it does not
change the awsbatch default of joint_call_out (which keeps the profile's placeholder
bucket: set joint_call_out itself) nor a call task's cpus, memory and forks (set the
process settings instead, process { withLabel: jc_call { memory = 24.GB } }); the
parameters the workflow script reads (the sets, the chromosomes, the driver's options,
joint_call_threads) work either way. The awsbatch profile has run from chr21 × 100
to genome × 3,202: as 17 sets of 200 samples, genome × 3,202 is 425 tasks in 54 min of
wall (81 CPU-hours, at most 6.2 min and 1.9 GB per task, 1.11 TB of VCF), and the 25
contig exports kept RDS at ≤ 47% CPU. Every VCF holds its contig's record count, and the
sampled ones equal a direct joint_call.py call byte for byte. A whole-cohort set of
chr1 × 3,202 needs --joint_call_threads 2 in a 12 GiB task.
The workflow refuses, before any task, a --batch_size below 1, a negative cap or
--min-ac, an empty sample list and a --region entry that is empty or a range. An exit
status 2 of the driver (a refusal: a stale cohort, a bad name, a changed cohort under
--export-contig) ends the run after one submission of that task; any other failure
retries as the other processes do (the default errorStrategy retries up to three times
whatever maxRetries says: to bound retries in an override file, set errorStrategy as
well as maxRetries). The local profile runs one task of each process at a
time. A set's memory grows with the data files it reads (samples × chromosome size,
divided by ref-block banding): for a large set lower --joint_call_threads or raise
--joint_call_memory together with a compute environment that has such instances.
-resume. COHORT_STATE runs every time, and its one-line state is the value that
keys every task after it: on an unchanged cohort -resume runs COHORT_STATE alone and
reuses the exports and every finished call; a different --batch_size (or sample list)
reuses the exports and calls the new sets; after a new aggregate everything runs again,
into the new generation's folder. The cache does not look at the outputs: a file deleted by
hand is not rebuilt by -resume, and a cached task does not publish again. A different
option set or --batch_size on the same generation writes over the same set folders (and a
set of an earlier --batch_size that the new one does not produce stays there): give each
its own --joint_call_out.
Do not run mosaic remove, an ingest or an aggregate while a joint-call run is in
flight. The cohort is checked at the start of a run only: the call tasks read the export
and the lake, and a change under them (a removal deletes a sample's per-sample rows while
the export still lists the sample) is not detected, so the tasks before and after it would
not call one cohort and nothing would report it.
The workflow writes the joint caller's output as section 6 describes it, its known limitation on haploid calls included.
Downloads a few 1000 Genomes DRAGEN gVCFs from the public S3 bucket
(workflow/bin/fetch_and_ingest.py) but runs ingest/register/aggregate/index/maintain
against the local Docker stack. Four single-sample batches with --fan_in 2 force a
2-tier merge; the run takes a few minutes. mosaic clean in local mode restarts the
Docker stack and wipes everything on it, so run this only against a stack whose data you
do not need. mosaic init needs the header of any gVCF of the same reference (GRCh38,
with chr-prefixed contigs), for example one of the four samples' gVCFs:
cd iceberg_test && docker-compose up -d && cd ..
mosaic clean
mosaic init <one-1000G-DRAGEN-sample.gvcf.gz> --chromosomes chr21
printf 'HG00096\nHG00097\nHG00099\nHG00100\n' > samples.txt
nextflow run workflow/main.nf -profile local \
--samples samples.txt --region chr21 --batch_size 1 --fan_in 2| Table | Purpose |
|---|---|
contigs |
Chromosome names and lengths |
alleles |
The allele catalog, built by mosaic aggregate: every allele any batch listed at a site, its summed count, and del_range for deletions (derived from the allele string) |
site_info |
Aggregated per-site statistics (QUAL, DP, MQSQ, SB, GTC) |
ref_blocks |
Aggregated reference-block DP and sample counts |
samples |
Sample names, IDs and status: active, or removed (the tombstone mosaic remove sets first). A removed sample is left out of the per-sample reads of joint_call.py, scripts/gwas_benchmark.py and the genotype-matrix reader, and mosaic ingest refuses it; the aggregates count it until the mosaic aggregate that applies its removal |
ingestion_progress / shard_completion |
Ingest checkpoint/resume tracking (batch status in_progress, failed, uploaded or complete; kind is ingest, or removal for the negative batch of a mosaic remove; ref_gq_bands is the batch's --ref-gq-bands spec, NULL without banding) |
removal_progress |
One row per mosaic remove: the samples, the negative batch's batch_id, status (in_progress, registered, complete, failed) |
sample_qc / sample_qc_summary |
Per-(batch, sample, chromosome) QC counters; the summary view lists active samples only |
ingest_files |
The data files of --defer-register batches, pending until mosaic register registers them |
aggregation_progress / tier_completion |
Aggregation checkpoint/resume tracking |
sample_ref_blocks — per-sample reference blocks:
sample_id, chromosome_index, position_start, position_end,
ad_1, ad_2, dp, pl_1, pl_2, pl_3, sample_counts (1 for a
hom-ref block, 0 for a ./. block). With --ref-gq-bands a row is a merged run of
adjacent blocks (see "Ref-block banding" above)
sample_site_info — per-sample variant genotypes:
sample_id, chromosome_index, position, del_start, del_end,
alleles (the record's own allele strings: its reference base, then its ALT
alleles in the record's order), gt_1, gt_2 (local indices into alleles),
dp, ad, f1r2, f2r1, sb, pl (lists, as the record carries them), and the
record's own contribution to site_info: qual, site_dp (INFO/DP, else
FORMAT/DP; dp is FORMAT/DP), mqsq_1, mqsq_2, gtc (a list of three
doubles). On a deletion carrier's row, del_ref_spans lists the (start, end)
spans of the sample's own reference blocks that the deletion overlaps (see
below; joint_call.py does not read it).
Layout of the two per-sample tables. Both are partitioned by chromosome_index.
Each data file holds whole samples of one ingest batch and one chromosome, packed in
sample_id order up to --aggregate-parquet-size; a file never splits a sample. Inside a
file, rows are sorted by position, then sample_id (the sort order declared on both
tables), in row groups of MOSAIC_PARQUET_ROW_GROUP_ROWS / MOSAIC_PARQUET_SITE_ROW_GROUP_ROWS
rows. So:
- deleting one sample's rows (a
sample_idfilter) rewrites only the files that hold it, one or a few per chromosome per batch, instead of every file of its batch; a file that holds only that sample is dropped without a rewrite. pyiceberg 0.11 cannot write merge-on-read deletes (it falls back to copy-on-write with a warning), so this packing is what makes such a delete cheap; - a scan for a subset of samples skips the files of the other samples;
- a scan of a position window skips the row groups outside the window, but plans every file of the chromosome, since each file spans it.
mosaic remove deletes a removed sample's rows this way. The files such a delete
rewrites are written by pyiceberg, in row groups of the table's
write.parquet.row-group-limit, which mosaic init sets to the same caps, so a
position-window scan reads a rewritten file as cheaply as an ingested one.
Ingestion also writes append-only per-batch partials (batch_ref_blocks,
batch_site_info, batch_allele_counts, with gtc summed in fixed point
(2^-32 units, int64) and the record and listing counts n_records /
n_listed); mosaic remove writes the
same tables with negative values. mosaic aggregate reduces them
through staging tables (tier_*) into the materialized aggregates (agg_*) and
finalizes to PostgreSQL. Every table is created with
write.metadata.delete-after-commit.enabled; mosaic maintain expires old
snapshots and deletes their files.
The partials are a sum of the per-sample rows. Each batch's partial rows are exactly the sum, over the batch's samples, of what their per-sample rows store, and the test suite checks this by recomputing every partial row from the two per-sample tables:
batch_site_infois the sum of the contribution columns (each row'sgtcquantized to 2^-32 units first);n_recordsis the number of rows at the site.batch_allele_countsis the count of thegt_1/gt_2hits;n_listedis the number of rows whose own allele list holds the allele.batch_ref_blocks, per position, is the sum of thesample_ref_blocksrows'dpandsample_counts, plus eachdel_ref_spansspan with its row'ssite_dpand a sample count of 0. A reference block that a sample's own deletion overlaps counts the deletion's depth there.
So a sample's contribution to every aggregate can be re-derived from its
Iceberg rows without its gVCF, which is what mosaic remove does. The finalize
of mosaic aggregate drops a site whose net n_records is 0 and an allele whose
net n_listed is 0, and writes the integer part of the fixed-point gtc sum to
PostgreSQL. All of these are integer sums, so the result is exact whatever the
order of summation (a removal, an incremental run or --rebuild).
Alleles are stored using a prefix notation:
| Type | Format | Example |
|---|---|---|
| Reference | first base | A |
| SNP/MNP | x + alt |
xT |
| Insertion | + + inserted bases |
+AT |
| Deletion | - + deleted bases |
-ATG |
| Non-ref symbolic | * |
* |
Two test layers run from a clone with no extra data: the Python unit tests and the Rust workspace's tests.
./run_tests.sh --fast # Python unit tests (no Docker, no test data)
cd rust && cargo test # Rust unit tests (plus: cargo fmt --check, cargo clippy --all-targets -- -D warnings)run_tests.sh runs pytest under python3; set MOSAIC_TEST_PYTHON to another
interpreter if the dependencies are installed elsewhere. Unit tests that run the Rust
binaries use the release builds under rust/target/release/ when they exist and skip
otherwise.
The other tiers are for maintainers:
./run_tests.sh --integration # integration tests (needs the Docker stack)
./run_tests.sh --e2e # full pipeline vs. a recorded vcfdiff baseline (Docker, slow)
./run_tests.sh --all # unit -> restart -> other integration -> e2eThey need the Docker stack in iceberg_test/ (cd iceberg_test && docker-compose up -d)
and chr21 gVCF test data under test_data/ (1000 Genomes DRAGEN gVCFs and slices derived
from them), and the e2e tier also a vcfdiff.py comparison script at the repository root.
Neither the test data nor that script is distributed with the repository, so tests that
need them do not run from a clone. The tests use their own PostgreSQL database
(jc_test), Iceberg namespace (jc_test) and S3 bucket (jc-test-warehouse), so they do
not touch pipeline data on the same stack.
The ingest worker's and the joint caller's outputs on the test data are checked against
frozen digests: tests/golden/ingest_signatures.json (per case a SHA-256 over the
worker's rows, floats as bits) and tests/golden/joint_call_digests.json (per scenario
the SHA-256, record and byte counts of the decompressed VCF and its closing lines). A
digest is re-recorded only when a change is meant to alter the output:
MOSAIC_INGEST_GOLDEN_RECORD=1 with MOSAIC_INGEST_GOLDEN_REASON="<why>" (on
tests/integration/test_rust_engine.py), or MOSAIC_JOINT_CALL_GOLDEN_RECORD=1 with
MOSAIC_JOINT_CALL_GOLDEN_REASON="<why>" (on tests/integration/test_rust_joint_call.py).
Licensed under the BSD 2-Clause License — see LICENSE.