English | 中文

RiskScore

Credit Risk Scoring

WOE/IV · XGBoost/LightGBM · Leakage-free · SHAP Interpretable

🏠 Primary: Gitee · GitHub: MeaFew/riskscore

English | 中文

---

Headline

On real Kaggle Home Credit Default Risk data (307,511 samples / 208 features / 8.07% default rate), XGBoost achieves the following under 5-fold stratified cross-validation with leakage-free out-of-fold (OOF) evaluation:

OOF AUC 0.784 · KS 0.429 · Gini 0.569 — a +0.019 AUC lift over the single-table baseline (OOF AUC 0.766, 141 features) after aggregating 6 auxiliary tables, clearly above the single-table logistic-regression baseline (AUC 0.653) and essentially at the competition Top 10% band (~0.795).

Model	AUC	KS	Gini
Logistic Regression	0.653	0.229	0.307
Random Forest	0.760	0.390	0.519
XGBoost	0.785	0.430	0.569
LightGBM	0.784	0.431	0.569

Source: reports/model_results.json — mean of 5-fold stratified CV on multi-table features (application_train + 6 auxiliary tables: bureau, bureau_balance, previous_application, POS_CASH_balance, credit_card_balance, installments_payments; 141 → 208 features, 2026-07-27). Target encoding is fitted per CV fold inside a sklearn Pipeline on the fold's training split only, so validation rows never encode their own target. See reports/multitable_upgrade_report.md for the single-table → multi-table comparison.

Leakage-free Design (core selling point)

A credit-risk model is only as trustworthy as its evaluation is honest. This pipeline corrects three common leakage patterns:

• Target-encoding leakage: an earlier version computed target encoding on the full training set, then fed it into 5-fold CV — so a validation row's own target leaked into its encoded feature and inflated AUC. Target encoding now lives in a sklearn Pipeline and is fit on each fold's training split only. IV-based feature selection was removed from the modeling path (kept only as the analytic data/processed/iv_report.csv) to avoid the same class of leak via "select features with the full-set target, then use them inside CV folds".
• Fabricated test AUC: Home Credit's application_test.csv has no TARGET, so there is no labeled holdout. An earlier version split the training set 80/20, retrained on the full data, and evaluated — producing a 0.80 "test AUC" that was a resubstitution number and, worse, higher than the honest CV AUC. This metric was removed; we now report OOF AUC.
• Preprocessing leakage: outlier caps and median imputation are now fit on train only, then transformed onto test.

Scorecard (WoE/IV + PSI)

Gradient boosting wins on raw performance, but banks still deploy classical scorecards because every point of the final score is auditable. src/riskscore/scorecard.py implements the full industry-standard stack alongside XGBoost:

1. Optimal binning — the top-15 features by IV (data/processed/iv_report.csv) are binned monotonically with optbinning (monotonic_trend="auto", min_bin_size=5%; a ChiMerge + pool-adjacent-violators fallback ships in-module for environments without the library). Per bin: bounds, counts, bad rate, WoE, IV → reports/scorecard_bins.csv.
2. WoE + logistic regression — binned features are WoE-encoded and modeled with an L1-regularized LogisticRegression (the classic scorecard estimator). L1 sparsity doubles as feature selection: the raw EXT_SOURCE_* terms drop out because their squared/interaction versions carry the signal.
3. Score scaling — log-odds are mapped to points with base_score=600 @ odds=1/50, PDO=20 (doubling the default odds costs exactly 20 points). Points decompose additively per feature per bin → the readable scorecard table reports/scorecard_points.csv.
4. PSI monitoring — Population Stability Index of the total score and of every model feature (train vs. application_test) → reports/scorecard_psi.csv.

Run it with make scorecard (or python -m riskscore.scorecard).

Scorecard excerpt (EXT_SOURCE_2 × EXT_SOURCE_3, IV = 0.54 — higher score = lower risk):

Bin	% of pop.	Bad rate	WoE	Points
(−inf, 0.050]	6.2%	25.2%	−1.344	25.9
(0.050, 0.110]	9.8%	15.6%	−0.743	30.9
(0.110, 0.234]	25.3%	9.5%	−0.181	35.6
(0.234, 0.361]	31.0%	5.7%	0.368	40.2
(0.361, inf]	27.6%	2.8%	1.104	46.4

Scorecard vs. XGBoost — identical leakage-free 5-fold stratified OOF protocol (binning and LR are refit inside each fold, so bin boundaries never see validation targets):

Model	AUC	KS	Gini
XGBoost (208 features)	0.784	0.429	0.569
WoE scorecard (15 features)	0.732	0.344	0.465

PSI verdicts (train → application-test):

Object	PSI	Verdict
Total score	0.002	✅ stable
14 / 15 features	< 0.08	✅ stable
CREDIT_TO_ANNUITY_RATIO	1.030	🔴 drift (test mass concentrates in the low-ratio deciles)

Honest boundaries. Textbook experience says a scorecard gives up 1–3 AUC points to a GBDT — that is the price of full interpretability. Our gap is larger (~5 AUC points) because the scorecard is deliberately restricted to the top-15 IV features while XGBoost exploits all 208 plus free-form interactions; widening the feature set narrows the gap but erodes the auditability that is the scorecard's whole point. The scorecard is also uncalibrated for absolute PD by design here — it ranks risk and allocates points; a production deployment would add a calibration step to the master scale.

Overview

End-to-end credit risk scoring pipeline built on the Kaggle Home Credit Default Risk dataset: professional-grade feature engineering (WOE/IV) → model comparison (LR → RF → XGBoost → LightGBM) → SHAP-based interpretability → a Streamlit risk-calculator dashboard.

Quick Start

# Clone from GitHub
git clone https://github.com/MeaFew/riskscore.git
# or from Gitee (faster in China): git clone https://gitee.com/zeroonei1/riskscore.git
cd riskscore

# Create and activate a Python 3.11 virtual environment
python -m venv .venv
# Linux / macOS: source .venv/bin/activate
# Windows PowerShell: .venv\Scripts\Activate.ps1

# Install locked dependencies, the package, and development tools
make setup
# Windows without GNU Make: python -m pip install -r requirements.lock
#                           python -m pip install -e ".[dev]"

# Download real dataset (GitHub Releases, ~40MB)
bash download_data.sh

# Run full pipeline
make all
# Windows without GNU Make: python run_all.py

# Or step by step
make preprocess
make features
make train
make evaluate
make shap

# Launch dashboard
make dashboard

# Quality gates
make verify

Key Highlights

• Feature Engineering: WOE binning (analytic reference), per-fold target encoding (leakage-free), cross-features
• Model Stack: Logistic Regression (baseline) → Random Forest → XGBoost → LightGBM
• Evaluation: AUC, KS, Gini, calibration curves, confusion matrix at optimal threshold
• Interpretability: SHAP summary, dependence plots, force plot for individual cases
• Delivery: Streamlit dashboard with risk calculator

Tech Stack

Layer	Tools	Notes
ETL	pandas, scikit-learn	Missing value imputation, outlier capping
Feature Eng	Custom WOE/IV	Quantile-based binning with smoothing
Modeling	XGBoost, LightGBM, sklearn	5-fold stratified CV
Interpretability	SHAP	TreeExplainer for gradient boosting models
Evaluation	scipy, sklearn	AUC, KS, Gini, PR curve, calibration
Delivery	Streamlit	Interactive risk calculator + model comparison
Quality	pytest, ruff, GitHub Actions	CI runs lint + tests on every push

Project Structure

.
├── src/riskscore/
│   ├── generate_mock_data.py      # Synthetic data generator (for CI)
│   ├── preprocess.py              # Data cleaning & missing value handling
│   ├── feature_engineering.py     # Cross-features + WOE/IV analytic report (target encoding moved into CV Pipeline)
│   ├── train_models.py            # LR / RF / XGB / LGBM with CV
│   ├── evaluate.py                # ROC, PR, calibration, confusion matrix
│   ├── shap_analysis.py           # SHAP summary, dependence, force plots
│   ├── aggregate_auxiliary_features.py
│   └── merge_auxiliary_features.py
├── dashboard/
│   └── app.py                     # Streamlit interactive dashboard
├── tests/
│   └── test_pipeline.py           # Unit + integration tests
├── Makefile                       # Workflow orchestration
└── requirements.lock              # Reproducible dependency pins

Benchmark

Based on Kaggle Home Credit Default Risk (7,190+ teams, metric: AUC-ROC).

Reference	AUC	Notes
Kaggle Starter Baseline	0.688	Official starter notebook, no feature engineering
Single-table Logistic Regression	0.748	application_train only + GridSearchCV
Single-table LightGBM	0.749	Same as above, gradient boosting
Competition Median	~0.72-0.75	Leaderboard median
Competition Top 10%	~0.795	Multi-table features + ensemble
This Project (single-table)	0.766	Per-fold target encoding (leakage-free), 141 features, application_train only
This Project (multi-table)	0.785	+ 6 auxiliary tables (bureau, bureau_balance, previous_application, POS_CASH_balance, credit_card_balance, installments_payments), 208 features, XGBoost (5-fold CV / OOF), 2026-07-27

Note: the competition Private Leaderboard is closed. Scores above are from local 5-fold stratified cross-validation plus leakage-free out-of-fold (OOF) evaluation on real Kaggle data (307,511 train samples). The multi-table score (0.785) is validated under the exact same protocol as the single-table score (0.766); see reports/multitable_upgrade_report.md.

Additional evaluation plots

Related Projects

Project	Repo	Description
E-commerce User Analytics	MeaFew/shoplytics	29M real user behavior records, 10 analytical modules
Marketing Attribution & MMM	MeaFew/attributor	MMM + multi-touch attribution + budget optimization
Multivariate Time Series	MeaFew/foresight	LSTM / Transformer / XGBoost time series forecasting
Graph Fraud Detection	MeaFew/graphguard	GNN illicit transaction detection (Elliptic)

License

MIT