Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -117,3 +117,31 @@ logs/
pytorch_*.err
pytorch_*.out
scratch/

# Python cache
__pycache__/
*.pyc

# Local accidental CLI files
--config
--log-dir

# Local backups
*.bak
*.bak_*

# Large MFX/generated outputs
runs/
outputs/
mfx_shoebox*/
mfx_shoebox_chunks*/
diagnostics_qi_25files/
diagnostics_qi_meeting_25files/
diagnostics_qp_mean_profiles/
diagnostics_qp_mean_three_models_25files/
*.npy
*.parquet
*.ckpt
*.mtz
*.png
*.log
30 changes: 29 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,5 +37,33 @@ micromamba activate integrator-dev
micromamba env create -f environment-cuda-dev.yml
micromamba activate integrator-cuda-dev
```
## MFX Workflow

Each installs DIALS, laue-DIALS, PyTorch, the integrator package (editable), and the upstream reciprocalspaceship build required to read DIALS `.refl` files.
This repository includes an end-to-end workflow for applying the variational-inference Integrator to experimental MFX serial femtosecond crystallography data from LCLS.

The MFX workflow extends the base Integrator framework with:

- scalable Jungfrau shoebox extraction from cctbx.xfel `.refl/.expt` files
- chunked preprocessing and GPU training for datasets containing more than 100 million reflections
- alternative intensity posteriors including FoldedNormal and LogNormal
- Normal and Student-t observation likelihoods for floating-point and negative-valued detector data
- Monte Carlo KL fallback for posterior/prior combinations without a built-in analytic KL divergence
- MFX prediction write-back from Integrator intensities and uncertainties to the original reflection tables
- downstream cctbx.xfel scaling/merging and PHENIX refinement for direct crystallographic comparison
- diagnostic tools for shoeboxes, learned spot profiles, training behavior, and resolution-dependent merging statistics

### Current MFX study

The current production study uses experiment `mfx101555026`, covering runs 269–289 excluding 275:

- **124,975** indexed MFX `.refl/.expt` pairs
- **~115 million** extracted 25 × 25 Jungfrau shoeboxes
- two production Integrator models using **ASINH** and **sqrt-squareplus** preprocessing with a **FoldedNormal intensity posterior**
- matched cctbx.xfel baselines for apples-to-apples comparison
- evaluation through **CC½, Rint, Rsplit, R-work, and R-free**

The current results establish an end-to-end proof of concept: Integrator can scale to a full experimental MFX dataset, produce probabilistic intensity estimates, write them back into the crystallographic workflow, and generate refinable structures. The cctbx.xfel baseline currently performs better overall, while the elevated high-resolution Integrator CC½ behavior remains an active validation question.

See the full workflow, commands, configs, Slurm jobs, PHIL files, diagnostics, and results here:

- [`workflows/mfx101555026`](workflows/mfx101555026/README.md) — complete MFX cctbx.xfel → Integrator → scaling/merging → PHENIX workflow
Empty file added inspect_d_script.py
Empty file.
1 change: 1 addition & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ laue = ['laue-dials']
"integrator.predict" = "integrator.cli.predict:main"
"integrator.make_shoeboxes" = "integrator.cli.make_shoeboxes:main"
"integrator.resolve" = "integrator.cli.resolve:main"
"integrator.preprocess" = "integrator.cli.preprocess:main"

[project.urls]
Homepage = "https://github.com/Hekstra-Lab/integrator/tree/integrator_mvn"
Expand Down
Loading
Loading