Reproduction repository for the paper "Capturing Non-linear Neighbourhood Structure: An Autoencoder Approach to Census-based Dimensionality Reduction for Residential Differentiation"
Dimensionality reduction is fundamental to applied spatial data analysis, condensing high-dimensional indicators into parsimonious representations for mapping and modelling. Principal Component Analysis (PCA) remains a dominant approach, yet its linearity assumptions constrain its capacity to capture the complex, non-linear dependencies characteristic of spatial socio-economic data. We evaluate the effectiveness of autoencoders (AEs) as a flexible, parametric alternative for constructing composite measures of neighbourhood structure. We demonstrate the approach using small-area data from the 2021 Census of England and Wales, evaluating how effectively AEs summarise spatial variability and recover composite dimensions of residential differentiation compared with PCA.
├── .gitignore
├── README.md
├── pyproject.toml / uv.lock / requirements.txt # environment specification
├── data/ # input data (NOT in git — see Data sources)
│ ├── census_data/ # raw census tables
│ ├── geofiles/ # boundaries, lookups, IMD
│ ├── OAC/ # 2021 Output Area Classification
│ └── cache/ # auto-regenerated caches (spatial weights etc.)
├── AE_outputs/ # model outputs
├── torchgeodemo/ # autoencoder package
└── notebooks/
├── 1_prepare_data.ipynb
├── 2a_train_ae_stability.ipynb
├── 2c_lad_blocked_cv.ipynb
├── run_lad_spcv.py / run_lad_parallel.sh
├── make_pca_reconstructions.py
├── 3_rmse_vs_dimension.ipynb
├── 4_cv_generalisation_table.ipynb
├── 5_spatial_autocorrelation.ipynb
├── 6_error_by_geography.ipynb
├── 7a…7d clustering notebooks
├── prototype_notebooks/
│ └── 2b_identity_ae_control.ipynb
├── clustering_results/
├── plots/
└── logs_lad/
Python 3.12. uv sync reproduces the exact paper environment, including CUDA-enabled
PyTorch from the CUDA 12.8 wheel index.
git clone https://github.com/GeographicDataService/census_dimreduction_autoencoder.git
cd census_dimreduction_autoencoder
uv sync # creates .venv from uv.lock (exact paper environment)Without uv: create a venv and run both commands below from the repository root — the last
entry in requirements.txt is the relative path ./torchgeodemo, which only resolves from
there. requirements.txt takes torch from PyPI, which is CUDA-enabled but currently ships
CUDA 13.0 wheels; install from the cu128 index first to match the CUDA 12.8 build used in
the paper.
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128
pip install -r requirements.txtThe autoencoder implementation lives in the torchgeodemo package. A copy of the package source is in this repository at torchgeodemo/ — this is the version (0.0.4) used to produce the results in the paper, and it is what uv sync / requirements.txt install.
The actively maintained release of this package is deepgeodemo: https://deepgeodemo.readthedocs.io/en/latest/.
data/ is not tracked in git. The required input files and their sources are:
eng_raw_csvs/ts0XX.csv— 52 bulk tables (ts001–ts075, OA level, England & Wales), from Nomis Census 2021 bulk downloads
Output_Areas_(December_2021)_Boundaries_EW_BFE_(V9)_and_RUC.geojson— ONS Open Geography Portal, GeoJSONMiddle_layer_Super_Output_Areas_(December_2021)_Boundaries_EW_BFE_(V8)_and_RUC/— ONS Open Geography Portal, ShapefileLower_layer_Super_Output_Areas_Dec_2011_Boundaries_Full_Clipped_BFC_EW_V3_2022_*.gpkg— ONS Open Geography PortalOutput_Area_to_Lower_layer_Super_Output_Area_to_Middle_layer_Super_Output_Area_to_Local_Authority_District_(December_2021)_Lookup_in_England_and_Wales_v3.csv— ONS Open Geography Portaluk_imd2019.csv— Index of Multiple Deprivation 2019oabounds_2021approx.parquet— ships with this repo; simplified version of the full OA boundaries above, for plotting
OAC_assignment.csv,OAC_cats.csv— Output Area Classification 2021
Run the notebooks in numeric order from the notebooks/ directory. The stage-2 notebooks (2a–2c) train models and are GPU-intensive.
| Step | What it does |
|---|---|
1_prepare_data.ipynb |
Cleans and scales the raw census tables to the 408-variable model input |
2a_train_ae_stability.ipynb |
Trains the AE at 8 latent dimensionalities × 10 random seeds (500 epochs) |
run_lad_parallel.sh → run_lad_spcv.py |
5-fold LAD-blocked spatial CV of the AE (whole Local Authority Districts held out); parallel launcher over (dim, repeat) pairs |
2c_lad_blocked_cv.ipynb |
Builds folds, PCA CV comparator, and aggregates the spatial-CV results |
make_pca_reconstructions.py |
Fits PCA once and writes the per-dimensionality baseline reconstructions used by notebooks 3, 5 and 6 |
3_rmse_vs_dimension.ipynb |
RMSE vs latent dimensionality, AE vs PCA |
4_cv_generalisation_table.ipynb |
Train vs held-out RMSE table from the LAD-blocked CV |
5_spatial_autocorrelation.ipynb |
Global Moran's I of reconstruction errors (queen contiguity) |
6_error_by_geography.ipynb |
Error by IMD/density decile and OAC group; exports per-OA/MSOA error data for mapping |
7a_geodemographic_clustering.ipynb |
k-means (k=8, 10,000 inits) on the 100-d AE latent space and PCA comparator |
7b_clustering_quality_maps.ipynb |
Cluster quality metrics and regional maps |
7c_clustering_pub_plots.ipynb |
Publication cluster maps and OAC cross-tabulation |
7d_clustering_decoding.ipynb |
Decodes cluster centroids back to census variables |
Models were trained on a pair of NVIDIA A6000 GPUs (each individual model trains in under 10 minutes with a <2 MB footprint). Approximately 130 GB of disk is needed for a full end-to-end reproduction.