nrjNET is my very first neural network, built completely from scratch in plain Python. No PyTorch, no TensorFlow, no NumPy β just ~250 lines of code where I wrote every single piece myself:
- π’ a tiny autograd engine that computes gradients automatically
- π backpropagation using the chain rule
- π§± neurons, layers and a multi-layer perceptron
- π a gradient descent training loop
I built it while following Andrej Karpathy's brilliant lecture The spelled-out intro to neural networks and backpropagation.
The network starts out guessing randomly (π©· pink). After 100 training steps the loss drops from 5.23 β 0.012 and every prediction (π green) lands right next to its target.
git clone https://github.com/nrjai/nrjNET.git
cd nrjNET
pip install -e . # installs nrjnet (it has zero dependencies!)
python examples/train.py # train the networkMLP[Layer(3 -> 4), Layer(4 -> 4), Layer(4 -> 1)] | 41 parameters
step 0 loss 5.230518
step 10 loss 0.213094
...
step 99 loss 0.011572
Or skip the setup and open the notebook in Colab π
Every number remembers how it was made. Doing maths with Values quietly builds a graph:
from nrjnet import Value
a = Value(2.0)
b = Value(-3.0)
c = Value(10.0)
d = a * b + c # d = 4.0, and d remembers: "I came from (a*b) + c"
d.backward() # walk the graph backwards...
print(a.grad) # -3.0 -> nudging a up makes d go DOWN 3x as fast
print(b.grad) # 2.0
print(c.grad) # 1.0Each operation knows its own tiny local derivative. backward() visits nodes from the output back
to the inputs and multiplies along the way:
gradient of an input = local derivative Γ gradient flowing in from the output side
def __mul__(self, other):
out = Value(self.data * other.data, (self, other), '*')
def _backward():
self.grad += other.data * out.grad # d(a*b)/da = b
other.grad += self.data * out.grad # d(a*b)/db = a
out._backward = _backward
return outinputs (3) hidden (4) hidden (4) output (1)
x1 ββββββββΊ β ββββββββββΊ β
x2 ββββββββΊ β ββββββββββΊ β ββββββββββΊ β β prediction
x3 ββββββββΊ β ββββββββββΊ β
β β
each neuron: out = tanh(w1Β·x1 + w2Β·x2 + ... + b)
from nrjnet import MLP
model = MLP(3, [4, 4, 1]) # 41 trainable parameters
prediction = model([2.0, 3.0, -1.0])for step in range(100):
loss = sum((model(x) - y) ** 2 for x, y in zip(xs, ys)) # 1. how wrong are we?
model.zero_grad() # 2. clear old gradients
loss.backward() # 3. compute new gradients
for p in model.parameters():
p.data -= 0.05 * p.grad # 4. step downhillYes, and it's checked two independent ways:
| Check | How | Result |
|---|---|---|
| π Numerical | nudge each input by a tiny h, measure how much the output moves |
matches backprop |
| π₯ PyTorch | run the same neuron through torch.autograd |
matches to every decimal |
PyTorch nrjNET
o 0.7071 0.7071
x1 -1.5000 -1.5000
w1 1.0000 1.0000
x2 0.5000 0.5000
w2 0.0000 0.0000
Run the tests yourself:
python -m unittest discover tests -vnrjNET/
βββ nrjnet/
β βββ engine.py # π’ Value: the autograd engine
β βββ nn.py # π§± Neuron, Layer, MLP
β βββ viz.py # π¨ draw the computation graph (optional, needs graphviz)
βββ examples/
β βββ train.py # ποΈ train nrjNET
β βββ plot_training.py # π makes the chart above
βββ notebooks/
β βββ nrjNET_from_scratch.ipynb # π the full story, step by step
β βββ archive/ # ποΈ my original messy notebook
βββ tests/
βββ test_engine.py # β
gradient checks + PyTorch comparison
My first notebook had a few bugs that taught me more than the working code did:
| Mistake | What happened | Lesson |
|---|---|---|
p.data += lr * p.grad |
The loss went up to ~8. Weights exploded to -34 and tanh got stuck at 1.0 |
The gradient points uphill, so always step the opposite way (-=) |
| Forgot to zero the gradients | Old gradients piled up and training went haywire | += in backprop means you must reset grad = 0 every step |
| Ran notebook cells out of order | Printed predictions that hadn't been computed yet | Hidden state is sneaky, so always Run all from the top |
The original notebook is kept in notebooks/archive/ as a before-and-after π
- Add
reluand compare it withtanh - Train on a real 2D dataset (the "moons") and plot the decision boundary
- Add a learning-rate schedule
- Rebuild nrjNET with tensors (NumPy) to make it fast
- Andrej Karpathy for the lecture and the original micrograd, which this project follows closely
- Built and documented by Niranjan P as a learning project
made with π§ + β β one gradient at a time
