Skip to content
nrjaiPublic

About

🧠 nrjNET β€” my first neural network, built from scratch in pure Python. Autograd engine + backprop + MLP, gradients verified against PyTorch.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

Β 

History

3 Commits

Folders and files

NameName
Last commit message
Last commit date
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

nrjNET - my first neural network, built from scratch

tests python 3.8+ zero dependencies verified vs PyTorch MIT license

Open the notebook in Colab


πŸ‘‹ Hi! This is nrjNET

nrjNET is my very first neural network, built completely from scratch in plain Python. No PyTorch, no TensorFlow, no NumPy β€” just ~250 lines of code where I wrote every single piece myself:

  • πŸ”’ a tiny autograd engine that computes gradients automatically
  • πŸ”— backpropagation using the chain rule
  • 🧱 neurons, layers and a multi-layer perceptron
  • πŸ“‰ a gradient descent training loop

I built it while following Andrej Karpathy's brilliant lecture The spelled-out intro to neural networks and backpropagation.

✨ Watch it learn

Loss falling from 5.23 to 0.012, and predictions moving to match their targets

The network starts out guessing randomly (🩷 pink). After 100 training steps the loss drops from 5.23 β†’ 0.012 and every prediction (πŸ’š green) lands right next to its target.

πŸš€ Quick start

git clone https://github.com/nrjai/nrjNET.git
cd nrjNET
pip install -e .            # installs nrjnet (it has zero dependencies!)

python examples/train.py    # train the network
MLP[Layer(3 -> 4), Layer(4 -> 4), Layer(4 -> 1)] | 41 parameters

step   0   loss 5.230518
step  10   loss 0.213094
...
step  99   loss 0.011572

Or skip the setup and open the notebook in Colab πŸ““

🧠 How it works

1. Value β€” a number with a memory

Every number remembers how it was made. Doing maths with Values quietly builds a graph:

from nrjnet import Value

a = Value(2.0)
b = Value(-3.0)
c = Value(10.0)

d = a * b + c      # d = 4.0, and d remembers: "I came from (a*b) + c"
d.backward()       # walk the graph backwards...

print(a.grad)      # -3.0  -> nudging a up makes d go DOWN 3x as fast
print(b.grad)      #  2.0
print(c.grad)      #  1.0

2. Backpropagation β€” the chain rule, automated

Each operation knows its own tiny local derivative. backward() visits nodes from the output back to the inputs and multiplies along the way:

gradient of an input  =  local derivative  Γ—  gradient flowing in from the output side
def __mul__(self, other):
    out = Value(self.data * other.data, (self, other), '*')
    def _backward():
        self.grad  += other.data * out.grad   # d(a*b)/da = b
        other.grad += self.data  * out.grad   # d(a*b)/db = a
    out._backward = _backward
    return out

3. Neuron β†’ Layer β†’ MLP

inputs (3)     hidden (4)      hidden (4)     output (1)
   x1 ───────►   β—‹  ─────────►   β—‹
   x2 ───────►   β—‹  ─────────►   β—‹  ─────────►   β—‹   β†’ prediction
   x3 ───────►   β—‹  ─────────►   β—‹
                 β—‹               β—‹

each neuron:   out = tanh(w1Β·x1 + w2Β·x2 + ... + b)
from nrjnet import MLP

model = MLP(3, [4, 4, 1])         # 41 trainable parameters
prediction = model([2.0, 3.0, -1.0])

4. Training β€” repeat four steps

for step in range(100):
    loss = sum((model(x) - y) ** 2 for x, y in zip(xs, ys))   # 1. how wrong are we?
    model.zero_grad()                                          # 2. clear old gradients
    loss.backward()                                            # 3. compute new gradients
    for p in model.parameters():
        p.data -= 0.05 * p.grad                                # 4. step downhill

βœ… Is it actually correct?

Yes, and it's checked two independent ways:

Check How Result
πŸ“ Numerical nudge each input by a tiny h, measure how much the output moves matches backprop
πŸ”₯ PyTorch run the same neuron through torch.autograd matches to every decimal
        PyTorch    nrjNET
o        0.7071    0.7071
x1      -1.5000   -1.5000
w1       1.0000    1.0000
x2       0.5000    0.5000
w2       0.0000    0.0000

Run the tests yourself:

python -m unittest discover tests -v

πŸ“ Project structure

nrjNET/
β”œβ”€β”€ nrjnet/
β”‚   β”œβ”€β”€ engine.py          # πŸ”’ Value: the autograd engine
β”‚   β”œβ”€β”€ nn.py              # 🧱 Neuron, Layer, MLP
β”‚   └── viz.py             # 🎨 draw the computation graph (optional, needs graphviz)
β”œβ”€β”€ examples/
β”‚   β”œβ”€β”€ train.py           # πŸ‹οΈ train nrjNET
β”‚   └── plot_training.py   # πŸ“‰ makes the chart above
β”œβ”€β”€ notebooks/
β”‚   β”œβ”€β”€ nrjNET_from_scratch.ipynb     # πŸ““ the full story, step by step
β”‚   └── archive/                      # πŸ—‚οΈ my original messy notebook
└── tests/
    └── test_engine.py     # βœ… gradient checks + PyTorch comparison

πŸ› Things I learned the hard way

My first notebook had a few bugs that taught me more than the working code did:

Mistake What happened Lesson
p.data += lr * p.grad The loss went up to ~8. Weights exploded to -34 and tanh got stuck at 1.0 The gradient points uphill, so always step the opposite way (-=)
Forgot to zero the gradients Old gradients piled up and training went haywire += in backprop means you must reset grad = 0 every step
Ran notebook cells out of order Printed predictions that hadn't been computed yet Hidden state is sneaky, so always Run all from the top

The original notebook is kept in notebooks/archive/ as a before-and-after πŸ™‚

🌱 What's next

  • Add relu and compare it with tanh
  • Train on a real 2D dataset (the "moons") and plot the decision boundary
  • Add a learning-rate schedule
  • Rebuild nrjNET with tensors (NumPy) to make it fast

πŸ’œ Credits

made with 🧠 + β˜• β€” one gradient at a time

About

🧠 nrjNET β€” my first neural network, built from scratch in pure Python. Autograd engine + backprop + MLP, gradients verified against PyTorch.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages