Data integrity: wgi_processed.csv k_eff is inverted on every row (4933/4933); three inconsistent institutional pipelines

[emooreatx]

Summary

data/institutional/wgi_processed.csv ships precomputed k, rho, k_eff columns in which every analyzable row is defective. This is a data-integrity issue, not a paper-corrections item — it was surfaced during the CCA v3 audit (#11, defect C-18) but its blast radius is anything downstream of that file, not the paper.

Verified

Of 5083 rows, 4933 have non-null k_eff. All 4933 have k < 1 and k_eff > k. 4884 report k_eff > 1 derived from a fractional "count". Zero exceptions.

       country  year         k       rho     k_eff
0  Afghanistan  1996  0.064967  0.820024  0.911534
3  Afghanistan  2002  0.182463  0.768456  1.116875

Effective diversity exceeding nominal constituent count is impossible for a valid Kish design effect.

Root cause. k is not a count. ratchet/data/institutional_loader.py:326 computes k = (xconst - 1) / 6.0 — the Polity executive-constraints ordinal (1–7) rescaled to the unit interval. Kish's k is a cluster count with k >= 1. Below that boundary the formula inverts: d(k_eff)/d(rho) = -k(k-1)/(1+rho(k-1))^2 > 0 for k < 1, so k_eff increases with correlation. Every interpretation built on these columns reads backwards.

rho is not a correlation either. institutional_loader.py:339 computes rho = vdem_corr * (1 - k) — a corruption index times a constraint deficit. No covariance enters. At xconst = 7, k = 1 and rho === 0 for any corruption level.

Three mutually inconsistent pipelines are live

1. ratchet/data/institutional_loader.py:339 — rho = corruption x deficit
2. experiments/exp0_cca_validation/wgi_polity_validation.py:29-42 — rho = 1 - 2*cv across six WGI indicators (cross-indicator dispersion for one country-year; again no covariance)
3. ratchet/engines/institutional.py:94 — silently applies k_scaled = self.k * 10 before the Kish formula, an undocumented rescale

These give three different k_eff for the same country. Venezuela: 0.55 (published table), 0.741 (formula on the printed inputs), 2.47 (engine).

Also

InstitutionalDataLoader.load_all() raises RuntimeError: No datasets loaded against data/institutional/ as committed — the required qog_std_ts_jan25.csv (institutional_loader.py:152) and p5v2018.csv (:153) are absent. The pipeline that defines k is not runnable from the repo, so these are received numbers with no live path back to data.

Proposed

• Quarantine wgi_processed.csv — it should not be read by anything until regenerated
• Audit consumers of the k/rho/k_eff columns; anything downstream inherits the inversion
• Decide the institutional k semantics, or record that none is available. Note: the only count-valued institutional variable on disk is V-Dem v2lgbicam (legislative chambers), which is 1 for every country-year checked — the degenerate k=1 case. Reconstruction may not be possible from present data.
• Add a domain guard: reject k < 1 at the Kish call site rather than returning an inverted result
• Remove or document the * 10 rescale at institutional.py:94
• Reconcile the three pipelines to one

Refs

#11 (CCA v3 corrections, defect C-18), CCA_PAPER/CORRECTIONS_v3.md.