Large language models (LLMs) have become central to modern scientific computing, yet for most practitioners they remain opaque systems — input goes in, text comes out, and the internal mechanism is a mystery. Mechanistic interpretability (MI) is the emerging discipline of reverse-engineering what specific components of a neural network actually do. So far, MI tooling has been built almost exclusively around PyTorch and specialised libraries such as TransformerLens, placing a barrier between the scientific Python community and this important area of research.
This tutorial takes a different approach. Using Andrej Karpathy's microgpt — a fully self-contained, 200-line, dependency-free GPT implementation in pure Python — as our subject, we systematically dissect what a trained language model has learned. No PyTorch, no specialised ML frameworks: just plain python tools for a genuinely novel problem.
The model is tiny by design: 4,192 parameters, a 27-token vocabulary (a–z + a special token), trained on 32,000 names in roughly one minute on a laptop. This makes it the ideal subject for interpretability work — every attention weight is inspectable, every embedding printable, every head ablatable. The scientific question driving the tutorial is: "What has this model actually learned about the structure of names?"
This tutorial uses Andrej Karpathy's microgpt as a dissection subject. We reverse-engineer what a tiny trained language model has learned.
No PyTorch. No specialised ML frameworks. Just plain Python.
Prerequisites: Familiarity with neural network knowledge preferred. No GPU required.
mechanistic interpretability, transformers, GPT, neural networks, explainable AI, visualisation
Intermediate
uv initPython 3.9+ required. No GPU needed. Training takes ~1 minute on a laptop.
| # | File | Technique | Question it answers | Evidence type |
|---|---|---|---|---|
| 1 | 01_logit_lens.py |
Logit lens (nostalgebraist, 2020) | Where in the network does the prediction form? | observational |
| 2 | 02_attention.py |
Attention visualization (cf. Olsson et al., 2022, induction heads) | What does each head look at? | observational |
| 3 | 03_patching.py |
Activation patching / causal tracing (Meng et al., 2022) | Which activations causally carry a piece of information? | causal |
| 4 | 04_ablation.py |
Zero & mean ablation (cf. Wang et al., 2022, IOI) | How much does the model need each head/MLP? | causal |
| 5 | 05_probing.py |
Linear probes (Alain & Bengio, 2016) | Is a feature (next-char-is-vowel) linearly encoded? | correlational |
Each script prints its results with an interpretation guide, runs in
seconds-to-a-couple-minutes, and takes an optional name argument where
relevant (e.g. uv run 01_logit_lens.py jasmine).
1–2 generate hypotheses by looking; 3–4 test them causally by intervening; 5 asks what information is represented rather than what is computed. That observe -> intervene -> probe loop is the standard mech-interp workflow on real models too.
Swap the dataset by replacing input.txt before running part 1 like city names, etc.
The rest of the code adapts automatically. Try to observe whether the same circuits re-emerge on a different domain.
- Karpathy's microgpt blog post: https://karpathy.github.io/2026/02/12/microgpt/
- Anthropic circuits work: https://distill.pub/2020/circuits/
- TransformerLens (MI at scale): https://github.com/neelnanda-io/TransformerLens
- Neel Nanda's MI tutorials: https://neelnanda.io/mechanistic-interpretability