WIP Option to extend existing pangenome - #2005
Open
glennhickey wants to merge 1 commit into
Open
Conversation
A Minigraph-Cactus pangenome has been immutable: adding one genome to an HPRC-scale graph meant rebuilding all of it. minigraph construction is iterative in the input genomes and dominates the wall time of a large run -- weeks, for a release -- so that is the cost this removes. Adding 50 genomes to a graph of 450 now costs 50 genomes' worth of construction. cactus-pangenome --extendGFA seeds construction with an existing graph and --extendGAF reuses that run's mappings. cactus-minigraph --extendGFA and cactus-graphmap --extendGAF/--remap are the same thing at the step-by-step level, and the two interoperate in both directions: a graph built either way extends either way, and an extended pangenome is an ordinary one that can be extended again. Neither half needed a new algorithm. minigraph_construct_in_batches already chains "minigraph -cxggs prev.gfa a.fa b.fa" in batches of 50, so extending is seeding batch 0 with a graph somebody else wrote. The mappings did not need translating at all. cactus-graphmap runs minigraph without --vc, so the GAF it publishes as <outName>.gaf.gz is raw minigraph output in *stable* coordinates -- rGFA SN/SO names and offsets -- and it is published before any filtering. Adding genomes never moves those: nodes are appended and existing ones are only ever split, so the stable sequence a node covers stays where it was. The node-space PAF cactus consumes is derived from that GAF afterwards, by gaf2unstable | gaffilter | gaf2paf, so re-expressing 450 genomes against an augmented graph is running those three commands again with the new GFA, and gaf2unstable resolves the stable coordinates into the new, finer node ids on the way past. That chain is now stable_gaf_to_paf(), shared verbatim by mapping and by reuse. One thing does not survive the change of granularity. gaf2paf reads a record's path start as an offset into its *first* step, which holds for a GAF gaf2unstable resolved against the graph it was mapped to, where each of minigraph's stable steps is one node. Against a graph that has since been extended, the same stable step resolves into the finer nodes it was split into and the offset can reach past the first of them, which gaf2paf asserts on. On yeast that is 11 of SK1's 52 records; primates never hits it. trim_unstable_gaf() takes off the end steps that hold none of the alignment and moves the offsets with them -- a no-op, byte for byte, when the graph has not changed. It belongs in gaf2unstable, which is the thing changing the granularity; it is here so that this does not wait on a cactus-gfa-tools release. Two things make an extended pangenome differ from one built all at once, and it is worth being precise about which is which. Construction order. Building A B C D in one go sorts all four by mash distance; building A C and extending with B D gives A C B D, because the genomes already in the graph cannot be reordered around the ones being added. There is no way around that. It is also the only construction-side difference: with minigraphSortInput="none" so both sides use the same order, extending and building in one go produce a byte-identical GFA. Reused mappings. The genomes being added are mapped against the whole extended graph, so they come out exactly as a from-scratch run maps them. The ones already in the graph do not, so they never see nodes contributed by the new arrivals. Largely self-limiting -- a node is only there because some genome carries that allele, and an existing genome carrying it would have contributed it when the graph was built -- but not nothing. Measured against a from-scratch mapping on the same graph: primates (2 extended by 2) puts all 1183 alignments at identical coordinates with 5 cigars differing by a 2bp indel shift; yeast (3 extended by 3) agrees on 97.8% of alignments, with the reused genomes' aligned bases differing by at most 0.09%. --remap removes this difference entirely and reproduces a from-scratch PAF and GAF byte for byte, at the cost of the mapping stage -- which, unlike construction, is embarrassingly parallel. --mgSplit and --collapse are rejected with explicit errors. --mgSplit has per-chromosome graphs and mappings that would need extending as well; --collapse self-alignments come from minimap2 rather than from the GAF, so there is nothing in the GAF to reuse. A graph whose reference is not already in it is refused too: it would otherwise be constructed in last, at the highest rGFA rank rather than rank 0. Verification, all run locally: 24 offline unit tests covering the PanSN round trip on a published GAF, splitting a merged GAF back into the per-genome pieces it was concatenated from, and trim_unstable_gaf's invariants extending by nothing reproduces the GFA, PAF, GAF and minigraph fasta byte for byte -- what pins the whole reuse path -- plus an uncompressed output path, since an unchanged graph is the one graph construction hands back without writing it and so the one that can arrive at the wrong compression for where it is going extending and building in one go produce the same graph, the reused mappings land at from-scratch coordinates, and --remap reproduces the from-scratch PAF exactly the primates pipeline end to end, against the same MAF accuracy baseline as the from-scratch runs the yeast pipeline end to end, which is the one that puts a translated PAF through cactus-graphmap-split, checked against the same pinned graph statistics as the from-scratch six-strain run Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0189H4mFGx16gLTMmqhLc2ku
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
If you have a GFA/GAF from a previous
cactus-pangenomerun, you can now pass these in via the new--extendGFA/extendGAFoptions tocactus-pangenome.This lets you save on rerunning minigraph construction and mapping on the original data. You construct/map only the new samples, the redo the remaining steps of the pipline: split/align/join with the data.
Pending some scaled up experiments before merging. But this should help especially for wall time which is dominated by minigraph.