From Alignment to Synthesis Contrastive Volumetric Grounding for Text-to-CT Generation

Daniele Molino¹ · Camillo Maria Caruso¹ · Filippo Ruffini^1,2 · Valerio Guarrasi³ · Paolo Soda^1,2

_{¹ Università Campus Bio-Medico di Roma  ·  ² Umeå University  ·  ³ UniCamillus}

Paper  ·  Method  ·  Dataset  ·  Getting started  ·  Citation

---

Important

🎉 Accepted at BMVC 2026

This work has been accepted at BMVC 2026 in a substantially extended form. The paper now asks a different question: is the bottleneck for semantic controllability in Text-to-CT the richness of the text encoder, or the quality of 3D vision-language grounding? We show it is the latter, and introduce a generation-oriented 3D-CLIP encoder trained with text-level structured hard negatives.

📄 Updated paper: arXiv:2506.00633 (v3) 🚧 Code: what you find below is the release accompanying the original preprint. The BMVC version — hard-negative contrastive training and updated checkpoints — will land here . ⭐ the repo to get notified. 📦 Previous version: preserved under the v1-preprint tag.

---

🧠 Model Overview

Generating semantically controllable 3D CT volumes from radiology reports requires more than a rich text encoder — it requires vision-language alignment grounded in volumetric space. Existing Text-to-CT approaches condition generation on encoders pretrained with language-only or 2D vision-language objectives: linguistically expressive, but volumetrically blind.

Our framework combines:

• A generation-oriented 3D-CLIP encoder, trained contrastively on paired CT volumes and radiology reports with structured hard negatives operating exclusively at the text level. Because confusable reports cost nothing in 3D memory, this raises contrastive difficulty without the small-batch ceiling that constrains volumetric encoders. Beyond conditioning, the encoder reaches state-of-the-art zero-shot pathology classification and volumetric retrieval.
• A volumetric VAE compressing CT scans into a low-dimensional latent space.
• A fully end-to-end latent diffusion model operating directly in 3D latent space, with cross-attention conditioning — no cascaded super-resolution, and therefore none of the spatial artifacts and cross-slice inconsistencies it introduces.

Evaluated on CT-RATE across 18 pathological conditions, the method reaches state-of-the-art performance on both image fidelity and factual correctness, at lower inference time and GPU memory than competing approaches. A recurring finding of the paper: fidelity metrics such as FID can look healthy while the model quietly ignores the conditioning report — only task-oriented metrics expose it.

---

📦 Synthetic Dataset

We release 1,000 synthetic chest CT scans generated with our model for the VLM3D Challenge.

➡️ Synthetic Text-to-CT Dataset on Hugging Face

---

🚀 Getting Started

Note

The instructions below reproduce the preprint version of the model. They will be updated alongside the BMVC code release.

Environment

Python 3.10.8

pip install -r requirements.txt

Weights

Download from 🤗 dmolino/text2ct-weights and place in models/:

File	Component
autoencoder_epoch273.pt	Volumetric VAE
unet_rflow_200ep.pt	Diffusion UNet
CLIP3D_Finding_Impression_30ep.pt	3D-CLIP text/vision encoder

from huggingface_hub import snapshot_download

snapshot_download(
    repo_id="dmolino/text2ct-weights",
    repo_type="model",
    local_dir="your_local_path",
)

Then set the paths in the configs:

Config field	Points to
trained_autoencoder_path	autoencoder
existing_ckpt_filepath / model_filename	unet
clip_weights	clip

Data

We use the CT-RATE dataset.

python scripts/download_ctrate.py     # pull volumes from HF
python scripts/preprocess_ctrate.py   # reorient to RAS, clip HU, resample to fixed spacing/shape

After download and preprocessing, make sure:

• dataset/ contains the CT volumes
• data/train_data_volumes.json and data/validation_data_volumes.json list volumes with relative paths (e.g. dataset/train/...)
• data/train_reports.csv and data/validation_reports.csv contain the reports (VolumeName, Findings_EN, Impressions_EN)

Precomputing embeddings

Recommended — it speeds up training considerably.

1. VAE latent embeddings (CT)

python scripts/diff_model_create_training_data.py \
  --model_def ./configs/config_rflow.json \
  --model_config ./configs/config_diff_model.json \
  --env_config ./configs/environment_diff_model_train.json \
  --num_gpus 1 \
  --index 0

Key fields in environment_diff_model_train.json:

• data_base_dir → dataset
• embedding_base_dir → output folder for latents (e.g. ./embeddings)
• trained_autoencoder_path → ./models/autoencoder_epoch273.pt

2. Report embeddings (3D-CLIP)

python scripts/save_embeddings_ctrate.py \
  --train_json data/train_data_volumes.json \
  --val_json data/validation_data_volumes.json \
  --train_reports data/train_reports.csv \
  --val_reports data/validation_reports.csv \
  --data_base_dir dataset \
  --embedding_base_dir ./embeddings \
  --clip_weights ./models/CLIP3D_Finding_Impression_30ep.pt \
  --report_encoder_model xgem_3D

Training

python scripts/diff_model_train.py \
  --model_def ./configs/config_rflow.json \
  --model_config ./configs/config_diff_model.json \
  --env_config ./configs/environment_diff_model_train.json \
  --num_gpus 1

Use existing_ckpt_filepath to resume from your own checkpoint.

Inference

python scripts/diff_model_infer.py \
  --model_def ./configs/config_rflow.json \
  --model_config ./configs/config_diff_model.json \
  --env_config ./configs/environment_diff_model_eval.json \
  --num_gpus 1 \
  --index 0 \
  --resize 512

Outputs are written to the output_dir set in environment_diff_model_eval.json.

Demo — generate from your own report

python scripts/diff_model_demo.py \
  --model_def ./configs/config_rflow.json \
  --model_config ./configs/config_diff_model.json \
  --env_config ./configs/environment_diff_model_eval.json \
  --num_gpus 1

Edit example_report inside the script. Output: predictions/demo.nii.gz.

Script reference

Script	What it does
scripts/download_ctrate.py	Download CT-RATE volumes from HF
scripts/preprocess_ctrate.py	Reorient / clip / resample to fixed spacing and shape
scripts/save_embeddings_ctrate.py	Encode reports with 3D-CLIP, save impressions as .npy
scripts/diff_model_create_training_data.py	Extract VAE latent embeddings for CT volumes
scripts/diff_model_train.py	Train the diffusion UNet
scripts/diff_model_infer.py	Batch inference over data lists
scripts/diff_model_demo.py	One-off generation from a provided report

---

📝 Citation

If you find this work useful, please cite:

@inproceedings{molino2026alignment,
  title     = {From Alignment to Synthesis: Contrastive Volumetric Grounding for Text-to-CT Generation},
  author    = {Molino, Daniele and Caruso, Camillo Maria and Ruffini, Filippo and Guarrasi, Valerio and Soda, Paolo},
  booktitle = {British Machine Vision Conference (BMVC)},
  year      = {2026}
}

---

📬 Contact

Questions or collaborations — Daniele Molino, daniele.molino@unicampus.it

🙏 Acknowledgements

This repository builds on:

• MAISI tutorials (MONAI)
• XGeM