Metabolomics-pcpfm

Overview

A Biodepot workflow builder (Bwb) implementation of a metabolomics preprocessing and annotation pipeline for the MorPhiC consortium, built on PCPFM (Python Centric Pipeline For Metabolomics) and asari. The pipeline takes raw LC-MS instrument output through conversion, sample sheet consolidation, feature detection, and annotation against curated compound libraries. Original project GitHub can be found at https://github.com/shuzhao-li-lab/MorPhiC_Data_Releases.

Pipeline steps

The app/ directory contains the scripts that make up the pipeline, run in order:

Step 0: .raw to .mzML conversion (step0_mzML_convert.sh) Converts a directory of Thermo .raw files to centroided .mzML using ThermoRawFileParser.

• Requires: RAW_DIR (input directory of .raw files), OUTPUT_MZML_DIR (output directory for converted .mzML files)

Step 1: Combine sequence sheets (step1_combine_sequences.py) Merges multiple per-run sequence CSV files into a single combined_sequences.csv, tagging each row with its source file (OriginSequence), instrument method (Method), and a derived Harvest field (Supernatant or Pellet) based on filename content.

• Requires: SEQUENCE_DIR (directory of sequence CSVs), COMBINED_CSV_DIR (output directory)

Step 2: Add sample metadata (step2_add_pcpfm_metadata.py) Applies a configurable set of substring matching rules from a JSON file to populate or correct the Sample Type column of the combined sequence sheet, producing combined_sequences_modified.csv.

• Requires: COMBINED_CSV_DIR (directory holding the Step 1 output), SEARCH_DICT (path to a JSON file defining the field/substring/label rules)

Step 3: Map raw file names to .mzML paths (step3_map_mzML.py) Searches a directory (recursively) for the .mzML file corresponding to each File Name in the sequence sheet and writes the matched paths to pcpfm_sequence.csv. Rows without a matching .mzML file are written separately to missing_mzml.csv for review, rather than silently dropped.

• Requires: COMBINED_CSV_DIR (directory holding the Step 2 output; also where outputs are written), MZML_DIR (directory containing .mzML files, searched recursively)

Step 4: PCPFM processing (step4_pcpfm_process.sh) Runs the core pcpfm workflow for every filter JSON found in FILTERS_DIR, one filter per acquisition mode/sample type combination (e.g. HILIC-negative supernatant, RP-positive cell pellet). For each mode, this:

1. Assembles an experiment from pcpfm_sequence.csv using the mode's filter (pcpfm assemble)
2. Runs asari feature detection on the assembled .mzML set (pcpfm asari), with adjusted peak-height/intensity thresholds for DnCl/DnHz-labeled modes
3. Builds empirical compounds from the feature table (pcpfm build_empCpds)
4. If all eight MS2 directories (HILIC/RP x pos/neg x supernatant/cellpellet) are supplied, maps MS2 spectra to features (pcpfm map_ms2) before annotation
5. Annotates features against the HMDB/LMSD library (pcpfm l4_annotate) and the MoNA MS/MS library (pcpfm l2_annotate)
6. Generates the final annotated feature table (pcpfm generate_output)

• Requires: COMBINED_CSV_DIR (directory containing pcpfm_sequence.csv), OUTPUT_PREFIX, PCPFM_OUTPUT_DIR, FILTERS_DIR (directory of per-mode filter JSON files)
• Optional: MS2_HILICNEGSUPERNATANT_DIR, MS2_HILICPOSSUPERNATANT_DIR, MS2_RPNEGSUPERNATANT_DIR, MS2_RPPOSSUPERNATANT_DIR, MS2_HILICNEGCELLPELLET_DIR, MS2_HILICPOSCELLPELLET_DIR, MS2_RPNegCELLPELLET_DIR, MS2_RPPosCELLPELLET_DIR (all eight must be set to enable MS2 mapping and annotation; otherwise this step is skipped and annotation runs on the MS1-only feature table)

Step 5: Steroid derivative annotation (step5_steroid_annotate.py) For DnCl and DnHz derivatized feature tables (both cell pellet and supernatant), calculates expected m/z values for steroid derivatives (base compound mass plus 0 to 3 added derivatization groups, using mass2chem) and annotates matching features within a configurable ppm tolerance, requiring nonzero signal across all standard columns. Writes an annotation_table.tsv alongside each matched feature_table.tsv.

• Requires: OUTPUT_DIR (directory tree containing feature_table.tsv files from Step 4), STEROID_CSV (CSV of steroid names and chemical formulas)

Batch annotation notebook

A separate batch annotation notebook extends a single-table Li lab annotation workflow to run across all reprocessed feature tables with a matching method/ion-mode library. It matches features to:

• An authentic compound library (Shuzhao Li lab, 2024), covering HILIC (8 minute) and RP (12 minute) methods in positive or negative ion mode
• A MorPhiC-specific MS/MS library, filtered by cosine similarity

Currently only hilicneg and rppos feature tables are processed, since the DnCl/DnHz tables (metabolomics2, and the DnCl/DnHz portion of metabolomics3) do not yet have a matching library from the Li lab. Results are exported as one JSON file per table under annotated_features/.

Data

Expected data inputs are metabolomics .raw or .mzML files, along with per-run sequence CSV files describing sample metadata. Any .raw files must first be converted to .mzML (Step 0). Sample types currently distinguished are cell pellet and supernatant, across four LC-MS acquisition modes (HILIC/RP, positive/negative ion mode), plus DnCl/DnHz-derivatized steroid runs.

Requirements

• Bwb
• Python >= 3.11
  • asari-metabolomics
  • pcpfm >= 1.1.3
  • pandas
  • mass2chem (for Step 5 steroid mass calculations)
• Mono and ThermoRawFileParser (v1.4.0), for .raw to .mzML conversion
• Docker

A Dockerfile is included in app/ that builds an Ubuntu 22.04 image with Python 3.11, the packages above, ThermoRawFileParser, and the MoNA/HMDB/LMSD annotation libraries pre-downloaded via pcpfm download_extras.

References

Mitchell, J.M., Chi, Y., Thapa, M., Pang, Z., Xia, J. and Li, S., 2024. Common data models to streamline metabolomics processing and annotation, and implementation in a Python pipeline. PLOS Computational Biology, 20(6), p.e1011912. (https://doi.org/10.1371/journal.pcbi.1011912)

Li, S., Siddiqa, A., Thapa, M., Chi, Y. and Zheng, S., 2023. Trackable and scalable LC-MS metabolomics data processing using asari. Nature Communications, 14(1), p.4113. (https://www.nature.com/articles/s41467-023-39889-1)

https://github.com/shuzhao-li-lab/PythonCentricPipelineForMetabolomics

https://github.com/shuzhao-li-lab/asari_pcpfm_tutorials

https://github.com/shuzhao-li-lab/MorPhiC_Data_Releases