DR CVRIE

Contagions Vulnerability Robotic Infirmary Engineer — Epitech AIA coursework covering unsupervised clustering of patient symptom testimonies and supervised multi-class brain tumor classification from MRI scans, implemented with scikit-learn in Jupyter notebooks.

---

Overview

Two independent ML pipelines are developed end-to-end in src/:

• Unsupervised — 1011 unlabelled patient testimonies are cleaned with NLP (lowercasing, stop-word removal, lemmatization), vectorized with TF-IDF (1000 features), and clustered with KMeans, DBSCAN, and Agglomerative Clustering to group symptom descriptions by pathology family without any prior labels.
• Supervised — Brain MRI scans (glioma, meningioma, notumor, pituitary) are preprocessed through a grayscale → histogram equalization → square padding → 128×128 resize → normalize → PCA pipeline, then classified by 8 scikit-learn models. SVM with RBF kernel achieved the best test accuracy (86.3%) while being 150× faster to train than working on raw pixels thanks to PCA.

Each pipeline follows the same four-notebook structure: eda → preprocessing → training → evaluation.

---

Notebooks

Unsupervised — src/unsupervised/

Notebook	What it does	Key result
eda.ipynb	Loads the 1011-entry CSV, analyses text length (avg ~30 words/testimony), word frequency, and the mysterious 0x000000 flag present on 614 entries (meaning withheld until defense).	Stop words dominate raw frequencies; medical vocabulary (pain, ache, skin, cough…) visible after manual filtering.
preprocessing.ipynb	4-step cleaning: lowercase → regex punctuation removal → NLTK English stop-word removal → WordNet lemmatization. Vectorizes with TF-IDF (1011×1000 sparse, 98.6% sparsity). PCA 2D projection shows exploitable structure.	Top TF-IDF terms: feel, pain, ache, sore, skin, chest, ear, cough, throat.
training.ipynb	Trains KMeans (elbow + silhouette to select K), DBSCAN (eps grid search), and Agglomerative Clustering. Compares all three with Silhouette Score, Davies-Bouldin Index, and Calinski-Harabasz Index.	KMeans selected as best model (K=14 by silhouette). DBSCAN struggles due to the curse of dimensionality in TF-IDF space.
evaluation.ipynb	PCA and t-SNE 2D projections coloured by cluster and by the 0x000000 flag. Top-10 keyword bars per cluster. Auto-named clusters (e.g. "pressure/blood/chest", "chest/cough/breath", "knee/front/ache", "rash/itchy/new").	KMeans produced 14 clusters with medically coherent keyword groupings. Low silhouette scores are expected for overlapping symptom text (normal for NLP clustering).

The script src/unsupervised/clustering.py replicates the KMeans pipeline from the command line and prints cluster assignments + metrics to stdout (run via make run-unsupervised).

Supervised — src/supervised/

Notebook	What it does	Key result
eda.ipynb	Class distribution (1400 train / 400 test per class, perfectly balanced), sample MRI grids, image size scatter (200–750 px, variable aspect ratios), per-class pixel intensity histograms.	Classes are balanced; sizes vary enough to require resizing; pixel intensity profiles differ slightly between tumor types.
preprocessing.ipynb	Pipeline: grayscale → ImageOps.equalize → pad-to-square (aspect-preserving) → resize 128×128 (Lanczos) → normalize [0,1] → flatten → StandardScaler → PCA. Plots each step quantitatively (before/after equalization stats, padding ratios, normalization). PCA retains 95% variance.	16384 raw features → 1393 PCA components (95% variance). Data saved to data/processed/*.npy.
training.ipynb	Trains 8 classifiers: Logistic Regression, SVM (RBF), SVM (Linear), Random Forest (300 trees), Extra Trees (300 trees), KNN (k=5), Gaussian Naive Bayes, MLP (256→128). Explains log loss, hinge loss, and Gini impurity. Includes a PCA-vs-raw comparison on SVM (RBF).	SVM (RBF) 86.3%, MLP 85.9%, KNN 84.5%, Random Forest 83.2%, Extra Trees 83.7%, Logistic Regression 83.8%, SVM (Linear) 81.6%, Naive Bayes 49.7%. PCA gives 149× speedup (4.6 s vs 693.5 s) with identical accuracy.
evaluation.ipynb	Final confusion matrix (raw + normalized), misclassified image grid, per-class accuracy bar chart, full classification report, model ranking table, and detailed justification for choosing SVM (RBF).	SVM (RBF) test accuracy 86.3%, weighted F1 0.859. Pituitary: 95% recall. No-tumor: 100% recall. Glioma: 67% recall (most confused with meningioma).

---

Setup

Prerequisites: Python 3.10 and uv.

# Create virtualenv and install dependencies
make install

# Download the Brain Tumor MRI dataset from Kaggle (~170 MB)
# (requires a Kaggle account; the Makefile uses the public Kaggle API endpoint)
make download

If you prefer plain pip without uv:

python3.10 -m venv build
source build/bin/activate
pip install -r requirements.txt

Dependencies (requirements.txt): scikit-learn, pandas, numpy, matplotlib, notebook, ipykernel, Pillow, nltk.

---

Running the notebooks

Via make (executes notebooks in-place with nbconvert)

make run-supervised     # runs all 4 supervised notebooks sequentially
make run-unsupervised   # runs all 4 unsupervised notebooks + clustering.py

Notebooks are run from their own directory so relative paths (data/…) resolve correctly.

Interactively

source build/bin/activate
jupyter notebook

Open notebooks in order: eda → preprocessing → training → evaluation. Each notebook saves intermediate data (.pkl or .npy) that the next one loads; run them in sequence within each module.

---

Data

Unsupervised — patient testimonies

File	Size	Description
src/unsupervised/data/dataset.csv	~158 KB, 1011 rows	No header. Columns: integer ID, optional 0x000000 flag (present on 614/1011 entries, meaning withheld), free-text symptom testimony.

The 0x000000 column is treated as a hidden label during development; its meaning is revealed at defense. No external license known — assumed to be Epitech-provided synthetic/curated data.

Supervised — brain tumor MRI

Downloaded by make download from the public Kaggle dataset Brain Tumor MRI Dataset (Masoud Nickparvar). Not committed to this repository.

Split	Per-class count	Classes
Training	1400	glioma, meningioma, notumor, pituitary
Testing	400	glioma, meningioma, notumor, pituitary

Images are variable-size JPEGs (roughly 200–750 px on each side, greyscale MRI scans). After make download they land in src/supervised/data/training/<class>/ and src/supervised/data/testing/<class>/.

---

Project layout

Cvrie-publish/
├── Makefile                    # install / download / run targets (uses uv)
├── requirements.txt            # pip dependencies
├── docs/
│   └── README.md               # course subject summary (PDF not redistributed)
└── src/
    ├── unsupervised/
    │   ├── data/
    │   │   └── dataset.csv     # 1011 patient testimonies
    │   ├── eda.ipynb
    │   ├── preprocessing.ipynb
    │   ├── training.ipynb
    │   ├── evaluation.ipynb
    │   └── clustering.py       # standalone CLI replication of KMeans pipeline
    └── supervised/
        ├── data/               # created by `make download`
        │   ├── training/       # 5600 MRI images (4 class subdirs)
        │   └── testing/        # 1600 MRI images (4 class subdirs)
        ├── eda.ipynb
        ├── preprocessing.ipynb
        ├── training.ipynb
        └── evaluation.ipynb

---

Authors

• @sobsh — Gabriel Brument
• @nawfal-hassani — Nawfal Hassani
• @matyyyy — Maty Millien

---

Origin

Developed as part of the Epitech curriculum — AI & Algorithmics module, 4th semester (S4). The original project subject PDF is not redistributed here per Epitech policy.