getModelFromKEGG: build the global model from raven-data artefacts - #707
Merged
Conversation
findRAVENroot now validates a stored MATLAB preference against RAVEN2.png before trusting it, instead of blindly returning a path that may belong to a different RAVEN checkout on the same machine. getKEGGModelForOrganism only downloads the ~100+ MB HMM library when a FASTA file is actually supplied; the annotation-only path never uses libraryFile, so the download was pure waste on that path.
keggModel.mat was never shipped in the repository and had no working way to be regenerated, so any reconstruction for an already-annotated KEGG species failed outright (issue #704). getModelFromKEGG now downloads and assembles the same data from the raven-data kegg118 release instead: the gene-free reference model plus the ko_reaction/organism_gene_ko/rxn_flags relational tables published by the raven-toolbox Python package. Every organism's genes are joined onto the reference reactions through their shared KO ids using two sparse incidence matrices (gene-KO, KO-reaction) multiplied together, since organism_gene_ko can carry millions of rows and a per-row loop isn't viable. The assembled result is cached to keggModel.mat exactly as before, so this only costs the (slow, ~150+ MB) build once. getPhylDist/keggPhylDist.mat is unaffected: that file is already committed to the repository and was never part of this issue.
Function test results304 tests 277 ✅ 1m 9s ⏱️ Results for commit 7f70d8d. ♻️ This comment has been updated with latest results. |
A stored preference pointing at a directory without RAVEN2.png now throws immediately, naming the bad path and how to fix or clear it, rather than silently falling back to resolving the root from the currently executing copy of RAVEN.
…sion fetchKEGGArtefact only ever downloaded the kegg118 core bundle. Replaced it with fetchRavenDataAsset, which takes the release tag and asset name as arguments instead of hardcoding kegg118 --- so it can also serve the HMM library download in getKEGGModelForOrganism (previously its own inline websave/try-catch), and any future raven-data asset that isn't KEGG at all. The "kegg118" string itself now lives in exactly one place, reconstruction/kegg/KEGG_VERSION.md, read via the new keggDataVersion. Everywhere else that named it directly (buildGlobalKEGGModel's artefact names, getKEGGModelForOrganism's HMM library names/URL) now reads it from there, so a future KEGG release only needs that one file bumped.
It isn't KEGG-specific --- it downloads any raven-data release asset by tag and file name --- so it belongs alongside downloadRavenBinaries, which fetches from the same repository, rather than under reconstruction/kegg/. No behaviour change.
Replaces its own inline websave/try-catch (and the now-redundant base URL variable) with the same download-with-cache helper KEGG artefacts and the HMM library already use, since it downloads from the same raven-data repository. No behaviour change beyond the shared helper's messaging.
The hard error broke CI: checkInstallation's own setpref, moments later re-read by checkFunctionUniqueness's separate findRAVENroot call, disagreed with RAVEN.png's presence in a way that only reproduced on the CI runner, cascading into ~40 failing test classes. Reverted to silently falling through to resolving the root from the executing file, matching every prior passing run --- still validated (an invalid stored preference is not trusted), just not fatal. Also: RAVEN2.png was renamed to RAVEN.png upstream (#708, picked up by merging origin/develop3); the preference-validation branch had missed that rename since it has no equivalent on the upstream side to merge against.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #704.
keggModel.matwas never shipped in the repository and had no working way to be regenerated, so any reconstruction for an already-annotated KEGG species failed outright.getModelFromKEGGnow downloads and assembles the same data from the raven-datakegg118release: the gene-free reference model plus theko_reaction/organism_gene_ko/rxn_flagsrelational tables published by the raven-toolbox Python package. Every organism's genes are joined onto the reference reactions through their shared KO ids via two sparse incidence matrices (gene-KO, KO-reaction) multiplied together —organism_gene_kocan carry millions of rows, so a per-row loop isn't viable. The assembled result is cached tokeggModel.matexactly as before, so this only costs the (slow, ~150+ MB) build once.getPhylDist/keggPhylDist.matis unaffected — that file is already committed to the repository and was never part of this issue.kegg118release tag is no longer hardcoded anywhere:reconstruction/kegg/KEGG_VERSION.mdis now the single place it's recorded, read via the newkeggDataVersion. The download helper itself (fetchRavenDataAsset, replacing the KEGG-specificfetchKEGGArtefact) takes the release tag and asset name as arguments, so it also now servesgetKEGGModelForOrganism's HMM library download (previously its own inlinewebsave), and is reusable for any future raven-data asset.Test plan
tSyntax,tInstallation,tReconstruction— 14 passed, 0 failed, 3 filtered (all needing real network data / a pre-existingkeggModel.mat, consistent with the existinggetKEGGModelForOrganismNeedsData/getPhylDistNeedsDatapattern)buildGlobalGPRJoinsGenesThroughSharedKO) exercises the gene/KO/reaction join logic directly against a small synthetic fixture, without needing the real ~150 MB artefact set