Skip to content

Removing generation of matrices which can't be modeled - #30

Merged
amcim merged 8 commits into
mainfrom
skip-unusable-test-sets
Aug 6, 2026
Merged

Removing generation of matrices which can't be modeled#30
amcim merged 8 commits into
mainfrom
skip-unusable-test-sets

Conversation

@amcim

@amcim amcim commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Two changes were made to generate_matrices_ml.R code. The motivation is the number of matrices that are created but did not have enough of the minority class, enough genomes, etc. This PR stops the attempt to model these.

  • Skip unusable derived matrices: New .skipUnusableMatrix(), called from .parquet2LOOMatrix, .parquet2LOODrugMatrix and .parquet2CrossDrugTestMatrix. These matrices are built by removing genomes, so they can end up tiny or single-class. There is no point modeling those, so they're now skipped with a logged reason instead of written.
  • Run the balance check after the join, not before: matrix is built by joining metadata (drug phenotypes) to the feature tables (gene counts) on genome_id. Some genomes are in metadata but have no feature rows, so the join drops them, but skipImbalancedMatrix was running beforehand and counting genomes that would never make it in. The check now sees the genomes that survive the join.

@AbhirupaGhosh AbhirupaGhosh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The scripts ran without any error. I made some minor changes I observed in downstream functions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants