Skip to content

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

19 Commits

Folders and files

Repository files navigation

AE_CDR — Autoencoder Census Dimensionality Reduction

Reproduction repository for the paper "Capturing Non-linear Neighbourhood Structure: An Autoencoder Approach to Census-based Dimensionality Reduction for Residential Differentiation"

Left: 8-cluster geodemographic classification of England and Wales from the 100-dimensional autoencoder embedding. Right: reconstruction RMSE against latent dimension, autoencoder below PCA at every dimension.

Abstract

Dimensionality reduction is fundamental to applied spatial data analysis, condensing high-dimensional indicators into parsimonious representations for mapping and modelling. Principal Component Analysis (PCA) remains a dominant approach, yet its linearity assumptions constrain its capacity to capture the complex, non-linear dependencies characteristic of spatial socio-economic data. We evaluate the effectiveness of autoencoders (AEs) as a flexible, parametric alternative for constructing composite measures of neighbourhood structure. We demonstrate the approach using small-area data from the 2021 Census of England and Wales, evaluating how effectively AEs summarise spatial variability and recover composite dimensions of residential differentiation compared with PCA.

Project structure

├── .gitignore
├── README.md
├── pyproject.toml / uv.lock / requirements.txt   # environment specification
├── data/                     # input data (NOT in git — see Data sources)
│   ├── census_data/          #   raw census tables
│   ├── geofiles/             #   boundaries, lookups, IMD
│   ├── OAC/                  #   2021 Output Area Classification
│   └── cache/                #   auto-regenerated caches (spatial weights etc.)
├── AE_outputs/               # model outputs
├── torchgeodemo/             # autoencoder package 
└── notebooks/
    ├── 1_prepare_data.ipynb
    ├── 2a_train_ae_stability.ipynb
    ├── 2c_lad_blocked_cv.ipynb
    ├── run_lad_spcv.py / run_lad_parallel.sh
    ├── make_pca_reconstructions.py
    ├── 3_rmse_vs_dimension.ipynb
    ├── 4_cv_generalisation_table.ipynb
    ├── 5_spatial_autocorrelation.ipynb
    ├── 6_error_by_geography.ipynb
    ├── 7a…7d clustering notebooks
    ├── prototype_notebooks/  
    │   └── 2b_identity_ae_control.ipynb
    ├── clustering_results/    
    ├── plots/                 
    └── logs_lad/              

Installation

Python 3.12. uv sync reproduces the exact paper environment, including CUDA-enabled PyTorch from the CUDA 12.8 wheel index.

git clone https://github.com/GeographicDataService/census_dimreduction_autoencoder.git
cd census_dimreduction_autoencoder
uv sync          # creates .venv from uv.lock (exact paper environment)

Without uv: create a venv and run both commands below from the repository root — the last entry in requirements.txt is the relative path ./torchgeodemo, which only resolves from there. requirements.txt takes torch from PyPI, which is CUDA-enabled but currently ships CUDA 13.0 wheels; install from the cu128 index first to match the CUDA 12.8 build used in the paper.

pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu128
pip install -r requirements.txt

The torchgeodemo autoencoder package

The autoencoder implementation lives in the torchgeodemo package. A copy of the package source is in this repository at torchgeodemo/ — this is the version (0.0.4) used to produce the results in the paper, and it is what uv sync / requirements.txt install.

The actively maintained release of this package is deepgeodemo: https://deepgeodemo.readthedocs.io/en/latest/.

Data sources

data/ is not tracked in git. The required input files and their sources are:

data/census_data/

data/geofiles/

  • Output_Areas_(December_2021)_Boundaries_EW_BFE_(V9)_and_RUC.geojson — ONS Open Geography Portal, GeoJSON
  • Middle_layer_Super_Output_Areas_(December_2021)_Boundaries_EW_BFE_(V8)_and_RUC/ — ONS Open Geography Portal, Shapefile
  • Lower_layer_Super_Output_Areas_Dec_2011_Boundaries_Full_Clipped_BFC_EW_V3_2022_*.gpkg — ONS Open Geography Portal
  • Output_Area_to_Lower_layer_Super_Output_Area_to_Middle_layer_Super_Output_Area_to_Local_Authority_District_(December_2021)_Lookup_in_England_and_Wales_v3.csv — ONS Open Geography Portal
  • uk_imd2019.csv — Index of Multiple Deprivation 2019
  • oabounds_2021approx.parquet — ships with this repo; simplified version of the full OA boundaries above, for plotting

data/OAC/

Reproducing the paper

Run the notebooks in numeric order from the notebooks/ directory. The stage-2 notebooks (2a–2c) train models and are GPU-intensive.

Step What it does
1_prepare_data.ipynb Cleans and scales the raw census tables to the 408-variable model input
2a_train_ae_stability.ipynb Trains the AE at 8 latent dimensionalities × 10 random seeds (500 epochs)
run_lad_parallel.sh → run_lad_spcv.py 5-fold LAD-blocked spatial CV of the AE (whole Local Authority Districts held out); parallel launcher over (dim, repeat) pairs
2c_lad_blocked_cv.ipynb Builds folds, PCA CV comparator, and aggregates the spatial-CV results
make_pca_reconstructions.py Fits PCA once and writes the per-dimensionality baseline reconstructions used by notebooks 3, 5 and 6
3_rmse_vs_dimension.ipynb RMSE vs latent dimensionality, AE vs PCA
4_cv_generalisation_table.ipynb Train vs held-out RMSE table from the LAD-blocked CV
5_spatial_autocorrelation.ipynb Global Moran's I of reconstruction errors (queen contiguity)
6_error_by_geography.ipynb Error by IMD/density decile and OAC group; exports per-OA/MSOA error data for mapping
7a_geodemographic_clustering.ipynb k-means (k=8, 10,000 inits) on the 100-d AE latent space and PCA comparator
7b_clustering_quality_maps.ipynb Cluster quality metrics and regional maps
7c_clustering_pub_plots.ipynb Publication cluster maps and OAC cross-tabulation
7d_clustering_decoding.ipynb Decodes cluster centroids back to census variables

Hardware

Models were trained on a pair of NVIDIA A6000 GPUs (each individual model trains in under 10 minutes with a <2 MB footprint). Approximately 130 GB of disk is needed for a full end-to-end reproduction.

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors

Languages