CreditVintage

A point-in-time credit cohort evaluation, from application records to a reviewable risk report. It is designed to demonstrate the data controls a banking or risk analytics team should ask about before trusting a score.

Live synthetic report · Observed public credit benchmark · Five-split public comparison · Early score monitor · SQL business analysis · 中文业务案例 · Review workflow / 复核闭环 · Published report files · Test predictions · Run summary · Artifact hashes

Data status: The point-in-time event experiment is synthetic. A separate UCI German credit experiment uses 1,000 historical observed records. Its native good/bad target and random split cannot validate the event pipeline's time controls or a deployable credit policy.

Try observed public data

creditvintage public-benchmark --out outputs/public-credit
creditvintage verify-public outputs/public-credit

After installing the package as below, these commands work offline. The frozen 600/200/200 stratified split compares a constant baseline, logistic regression and independent isotonic calibration. Preprocessing fits only on train. On the 200 held-out records, logistic ROC-AUC is 0.774, Brier 0.162; calibration worsens Brier to 0.169, and that result stays visible. Test predictions, split membership, original attributed bytes and a label-free 20-record queue are exported. Verification refits from those bytes instead of trusting hashes alone. Protocol, provenance and limits.

Compare fixed models across five public-data splits

The separate repeated comparison reuses the packaged UCI source, native labels and the same 17-feature contract. A constant baseline, logistic regression and histogram gradient boosting share five fixed stratified 800/200 train/test splits. All 15 results, 3,000 test scores and every split assignment are saved. Preprocessing is fitted on training rows only; model settings are fixed. Protocol and interpretation.

Method	Mean ROC-AUC	Observed AUC range	Mean Brier ↓	Mean top-10% precision
Constant	0.500	0.500–0.500	0.2100	0.230
Logistic	0.767	0.732–0.822	0.1714	0.740
Histogram gradient boosting	0.771	0.728–0.844	0.1692	0.730

Each test set has 60 bad labels and each method receives 20 inspection slots. The constant method's top-20 subset is an arbitrary source-row tie break. The tree does not win every metric. These five overlapping splits provide descriptive means and observed ranges, not independent trials or confidence intervals. This comparison uses different splits from the calibration experiment above; their metrics must not be directly combined.

From the source checkout after installing the project:

python examples/german_credit.py --out outputs/public-credit-comparison
python examples/german_credit.py --verify outputs/public-credit-comparison

Data loads from the installed package, so there is no second dataset copy or network fetch. Saved-score verification checks source labels, held-out membership, metrics and inspection flags without refitting. The package's separate verify-public command refits its original calibration experiment.

What is actually evaluated

New: same-budget policy comparison compares a fixed business rule, raw/calibrated logistic regression, a training-only WOE scorecard and histogram gradient boosting at 5%, 10% and 20% review capacity. It exports monthly/segment outcomes, paired month-bootstrap intervals and label-free raw-model explanations. Protocol and limitations. Run creditvintage benchmark --out outputs/benchmark, then creditvintage verify-benchmark outputs/benchmark. This dated-event policy experiment remains synthetic.

Each loan has a decision date, a principal amount, an application-time feature history, and dated performance reports. The model predicts an operational 180-day 90+ days-past-due event. This is an explicit research label, not a claim of regulatory default, IFRS 9 expected credit loss, or credit approval fitness. Regulatory default definitions can include other criteria; see the Basel Framework.

Control	Enforced behavior
Feature availability	Only the latest value published by the decision date is used. Later revisions cannot change historical applications.
Label maturity	Every loan waits 180 days plus two reporting days. A nondefault needs all six scheduled reports at days 30, 60, 90, 120, 150 and 180. Missing coverage is flagged, never treated as good.
Outcome revisions	A performance report is visible only after its available_at date; later corrections do not alter an earlier snapshot.
Prospective stages	Training outcomes mature before the first calibration application. Calibration outcomes mature before the first test application.
Probability quality	Fixed logistic baseline and a separate isotonic calibration sample. ROC-AUC, average precision and Brier are all shown on untouched test vintages.
Decision capacity	The top 10% review capacity is fixed in advance; precision and default capture are measured on test.
Monetary scenario	predicted probability × origination principal × assumed 45% LGD. This is a scenario proxy, not observed recovery loss or an accounting provision.
Early monitoring	A frozen model scores visible test applications before their labels mature. Raw-score distribution is compared with the calibration cohort without using current outcomes.

Dates are calendar dates with end-of-day availability semantics. Real intraday decisions need timestamps and a source-system availability audit. Loan identity, date types, principal cents, segment, feature ranges, duplicate updates, and report coverage are validated. Incomplete evaluation cohorts fail explicitly rather than shrinking the denominator silently.

Published run

Seed 20260927, 80 generated loans per origination month. Training: Jan–Dec 2022 (960 loans); calibration: Jul–Dec 2023 (480); untouched test: Jul–Dec 2024 (480). The two six-month gaps let outcomes mature before the next stage; the test observation cutoff is 2025-07-01.

Test method	ROC-AUC	Average precision	Brier ↓
Constant training event rate	0.500	0.223	0.174
Fixed logistic model	0.747	0.434	0.155
Logistic + independent isotonic calibration	0.735	0.407	0.158

Calibration worsened Brier here. The result is retained. The test event rate was 22.3% versus a mean calibrated estimate of 18.8%, illustrating the value of inspecting drift instead of reporting AUC alone. A 300-resample vintage-block bootstrap gives a descriptive ROC-AUC interval of 0.653–0.801. Only six test months exist; that interval is not a guarantee under future distribution shift. The fixed 10% review group contains 48 loans and captures 23.4% of test defaults. All rounded headline numbers come from committed summary.json; predictions.csv contains every test label, raw score, calibrated score, and review flag.

From model output to business analysis

The analyze command verifies an existing mature evaluation report before running SQLite cohort queries. It exports portfolio, customer-segment, origination-vintage and vintage × segment views, plus the original fixed-capacity review queue without future outcome labels. Every group is checked against independent Python counts and sums. No database server or new dependency is needed.

creditvintage analyze docs/demo --out outputs/business-case
creditvintage verify-analysis outputs/business-case

Choose an output directory that does not yet exist. Open its index.html; the folder contains 21 group rows, 48 queue rows, the actual SQL, a complete frozen source-report copy and a digest manifest. Verification reconstructs the CSV, summary, queue, SQL and page from the embedded source, so moving the folder does not break replay. Invalid source evidence fails before the output is created. Pages generates and verifies the public analysis from the original committed report during publication.

The synthetic source has 107 research events in 480 mature applications. The 48-application review group contains 25 events: precision is 25/48 = 52.08%, while event capture is 25/107 = 23.36%. Overall calibrated mean PD is 18.78%; principal-weighted PD is 18.49%. The preset 10% capacity is global, not a separate 10% quota for every month or segment. Zero denominators remain null; comparisons use the previous observed vintage in the same segment. These are descriptive historical measures, not evidence that an intervention reduced losses. See the Chinese business walkthrough for denominator checks, customer-segment interpretation, a minimal SQL query and interview follow-ups.

A durable human review workflow

Move the verified, label-free queue into a SQLite workspace with claim, release, resolve and reopen actions. Each conclusion requires a reason and evidence reference. Concurrent claims are serialized in one transaction and checked against a case version; identical request retries return the original event. Review history is append-only, and current state is reconstructed from the frozen queue and events.

creditvintage review-init outputs/business-case --db outputs/reviews.sqlite
creditvintage review-status outputs/reviews.sqlite

The five-minute walkthrough includes complete PowerShell commands, competing-reviewer and retry examples, design choices and interview prompts. After review, creditvintage review-export outputs/reviews.sqlite --out outputs/review-evidence delivers a portable CSV/JSON evidence bundle. creditvintage verify-reviews outputs/review-evidence replays all cases without the database. The delivery walkthrough shows how a separately saved --expected-sha256 checkpoint also detects a consistent rewrite; internal hashes alone do not authenticate the author or evidence. The workspace stores no future performance labels. Its completion rate counts resolved work, not captured credit events or prevented loss. This is a trusted local CLI workspace: actor names are declarations, evidence references are not authenticated documents, and a hosted multi-user service would need identity, authorization and attachment controls.

Before outcomes mature: score monitoring

The separate monitor command freezes the same train and calibration stages, then scores only applications visible by a chosen cutoff. It needs no performance reports from the current cohort. Its raw-score population stability index (PSI) compares current applications with the calibration cohort; it is a distribution diagnostic, not a default-rate, quality, or significance test. It never triggers automatic approval, retraining, or alerts.

The monitoring path uses performance reports only for the train and calibration cohorts. It builds the current application snapshot without performance reports, so current outcomes cannot change the score distribution or whether the monitor passes its cohort checks. The input digest still covers all supplied event rows; it identifies the source bundle, including rows the monitor did not use.

creditvintage monitor --as-of 2024-10-01 --out outputs/early
creditvintage verify-monitor outputs/early

The published synthetic monitoring example has 480 reference and 320 visible current applications. Its PSI is 0.041 overall, with a 0.220 September vintage diagnostic on 80 loans. These are descriptive values on generated data, not validated decision thresholds. Ten quantile bins are learned only from reference raw scores, repeated edges are collapsed, and each bin receives a stated 0.5-count smoothing term. A monthly PSI is omitted below 20 loans. The score rows, reference scores, bin contributions, and manifest let verify-monitor independently recompute the result. Current labels are absent from the monitor output.

With supplied CSVs, pass the same three input paths shown below and an appropriate --as-of date. The train and calibration outcome gaps still need complete reports; the current window may have none. Keep borrower-level monitor files private.

Run and verify

Python 3.12+, from the repository root. On Linux or macOS:

python -m venv .venv
source .venv/bin/activate
python -m pip install -e ".[dev]"
creditvintage run --out outputs/latest
creditvintage verify outputs/latest
creditvintage monitor --as-of 2024-10-01 --out outputs/early
creditvintage verify-monitor outputs/early
pytest -q
ruff check src tests
mypy src

On Windows PowerShell, run the environment's executables directly; activation and execution-policy changes are unnecessary:

python -m venv .venv
.\.venv\Scripts\python.exe -m pip install -e ".[dev]"
.\.venv\Scripts\creditvintage.exe run --out outputs/latest
.\.venv\Scripts\creditvintage.exe verify outputs/latest
.\.venv\Scripts\creditvintage.exe monitor --as-of 2024-10-01 --out outputs/early
.\.venv\Scripts\creditvintage.exe verify-monitor outputs/early
.\.venv\Scripts\python.exe -m pytest -q
.\.venv\Scripts\ruff.exe check src tests
.\.venv\Scripts\mypy.exe src

For later examples on Windows, replace creditvintage with .\.venv\Scripts\creditvintage.exe and python with .\.venv\Scripts\python.exe. CreditVintage writes its HTML and JSON with UTF-8 and LF endings, and .gitattributes preserves committed evidence bytes even with Git's core.autocrlf=true. Regenerate older Windows exports with the updated package before verifying them. CI checks generated and committed reports, the lineage download and its replay on Linux Python 3.12/3.13 and Windows Python 3.12.

Check the package a new user actually installs

The separate Installed distributions CI jobs build a source archive, build its wheel, and install that wheel in a clean virtual environment. All commands run outside both repositories, so an editable checkout cannot hide a missing SQL file, HTML template or command entry point. They exercise evaluation, monitoring, SQL analysis and a 48-case review workspace, including an identical retry and a competing stale-version claim. The core paths run before PITBridge is installed; missing integration fails cleanly without creating an output. The pinned PITBridge wheel is then installed and the full lineage bundle replayed.

To run the same check locally, first clone the pinned PITBridge reference as shown in Source-to-prediction lineage, then:

python -m pip install build
python -m build --outdir dist
python -m build --wheel --outdir dist ../PITBridge
python scripts/check_distribution.py dist

Use the virtual environment's python.exe on Windows, as above. The replay script creates and removes its own isolated environment and temporary synthetic reports. Runtime dependencies may be downloaded during wheel installation; it does not publish reports or change an existing review workspace. Its explicit summary distinguishes package installation from numerical or business validity.

The command creates a standalone index.html, CSV tables, summary.json, and a manifest of SHA-256 output digests plus an input-event digest. verify checks file digests and independently recomputes test AUC, average precision, Brier, the test default rate, and the constant baseline from the prediction rows. It also replays all 300 whole-month bootstrap draws from saved predictions and the recorded seed, checking interval endpoints and valid replicate counts. It enforces the full training, calibration and test outcome-maturation gaps, then checks prediction dates against the stated test window and their vintage labels. It also checks that the published review flags select exactly the highest-ranked loans at the stated capacity, with application ID breaking score ties and the recorded configuration defining the same capacity. Both verifiers reject non-finite numeric claims (NaN, infinity and overflow), duplicate JSON keys, duplicate or malformed CSV columns, and fractional cohort counts. verify-monitor also validates current application identities, segments and the reference-label maturation gap. Both rebuild the HTML page from the verified evidence, rejecting a rehashed webpage with invented claims. These checks run even when the output hashes have been updated to match edited files; the regression suite covers those inconsistent reports alongside the original published examples. The summary records Python, NumPy and scikit-learn versions; exact floating-point output across environments is not promised. The hashes detect accidental edits, not an adversary who can rewrite files and manifest together.

To use privately held event data, supply all three UTF-8 CSV paths:

creditvintage run --applications applications.csv \
  --features features.csv --performance performance.csv --out outputs/private

CSV	Required columns
applications.csv	application_id,decision_at,principal_cents,segment
features.csv	application_id,name,value,observed_at,available_at
performance.csv	application_id,as_of,available_at,days_past_due

Dates are YYYY-MM-DD. Segments are retail or small_business; feature names are utilization, debt_to_income, prior_delinquencies. principal_cents is a positive integer USD-cent amount. The default vintage windows above apply when --config is omitted. Supply a strict JSON configuration to run, benchmark, monitor or preflight to use your own dates, LGD scenario and review fraction. The file must give all seven calendar dates; unknown keys, duplicate keys, non-finite numbers and shortened maturity gaps fail explicitly. An explicit --seed overrides the JSON seed.

creditvintage preflight --applications applications.csv --features features.csv \
  --performance performance.csv --config examples/cohort-config.json
creditvintage run --applications applications.csv --features features.csv \
  --performance performance.csv --config examples/cohort-config.json --out outputs/private

preflight checks the mature train/calibration/test cohorts without fitting or writing a report. It counts excluded applications so an incorrect date range cannot look like complete coverage. It checks input consistency, not source authenticity or model quality. For immature current cohorts use monitor; preflight intentionally requires mature evaluation outcomes. The synthetic generator still uses its documented fixed dates. Custom-input walkthrough. Keep borrower-level files private.

Design limits

• The event-demo labels are generated from the same synthetic risk drivers used as model inputs. Stronger-than-chance metrics validate the pipeline, not real-world default prediction.
• Six exact 30-day reporting anchors simplify coverage. Actual servicing feeds need calendar-aware schedules, partial repayment, prepayment, restructuring, recoveries, exposure at default, and audited source timestamps.
• Isotonic calibration is measured against a fixed logistic baseline; no hyperparameter search or test-driven calibration choice is performed. A larger independent calibration set may be required in practice. See the scikit-learn calibration guidance.
• No protected attributes are used, but their absence does not prove fairness. Selection bias, approvals-only labels, population drift, and policy impacts need separate study before any real use.
• Score PSI alone cannot tell whether calibration, realized loss, or fairness has changed. The published early example includes four current vintages; binning, smoothing, sample size, and shifting acceptance policy all affect the diagnostic.

Related open-source work

scorecardpy provides traditional scorecard development and PSI evaluation; Evidently provides general data and model monitoring. Credit risk case studies such as Lending Club Credit Risk and RiskLens show vintage maturity, calibration, and drift in broader pipelines. CreditVintage focuses its small, testable contract on application-time availability, explicit outcome maturity, and the transition from label-free monitoring to out-of-time evaluation.

The package has a standard-library event contract, NumPy/scikit-learn model evaluation, a dependency-free static HTML report, strict tests for timing and corruption, and CI on Python 3.12/3.13. This is research software, not a credit decision service.

Source-to-prediction lineage

Open the PITBridge × CreditVintage explorer · Download the complete synthetic evidence bundle. The lineage-demo command writes a standalone index.html: select a held-out application to trace its three model features to the source record, revision, publication, ingestion and decision cutoff. Its default generated synthetic cohort has 480 applications, 1,440 selected features and 120 held-out predictions. The original evaluation above remains a separate 480-test-application example. The public example is generated and verified during the Pages build. Its source events, input CSVs, predictions and manifests are available together in the downloadable bundle; generated outputs are kept out of Git history.

The adapter is a real code path, not a diagram connecting two independent demos. It verifies PITBridge's saved bundle through SQL and its separate Python oracle, enforces application/entity mapping and feature units, exports model inputs, and runs CreditVintage on those exported CSVs. The source fixture includes 192 observations unavailable at their decision, including corrections published on the decision date but ingested the next day.

Install the pinned reference implementation before using this optional integration:

git clone https://github.com/dev-belly/PITBridge.git ../PITBridge
git -C ../PITBridge checkout ed19dc698534f45a2b646fb4976ff6b01966b0cc
python -m pip install -e ../PITBridge
creditvintage lineage-demo --out outputs/lineage
creditvintage verify-lineage outputs/lineage

Open outputs/lineage/index.html in your browser after local generation. Its download section includes evidence.zip, generated alongside the page, containing the 18 evidence artifacts and the root manifest. Alternatively, download and extract the online evidence ZIP, then run creditvintage verify-lineage lineage against the extracted folder. verify-lineage checks file hashes, replays the original PIT sources, reconstructs every feature and lineage row, independently verifies the evaluation artifacts, and refits the model from saved applications/features/performance. It compares saved predictions using a numerical tolerance. The integration contract explains the units, UTC convention, dependency pin and boundaries.

The original run, monitor, verify and verify-monitor commands do not need PITBridge installed. CI exercises the integration with a pinned PITBridge checkout on Python 3.12 and 3.13.