Day 1:How Neural Networks Learn — Neurons, Activations & Coding `Autograd` Backpropagation From Scratch
Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz
Week 6 — Day 1: How Neural Networks Learn — Neurons, Activations & Coding Autograd Backpropagation From Scratch
Mentor's Note (10+ YOE Staff AI Engineer POV): Why did Andrej Karpathy (founding member of OpenAI, former Director of AI at Tesla) start his famous Deep Learning course by coding a tiny
Valueautograd engine (micrograd)? Because if you understand howloss.backward()walks backward through a Directed Acyclic Graph (DAG) applying the Chain Rule of Calculus, then PyTorch, Transformers, and LoRA fine-tuning become 100% transparent!
Part 1: Why Neural Networks Need Non-Linear Activation Functions
Suppose you stack two linear layers ( and ) without a non-linear activation function in between: Proof: A 100-layer neural network without non-linear activation functions collapses mathematically into a 1-layer linear regression!
The 4 Essential Activation Functions in Modern AI
| Activation | Formula | Derivative | Where Used in Modern AI |
|---|---|---|---|
1. ReLU | $1 \text{ if } x > 0 \text{ else } 0$ | Classic CNNs & MLPs (Watch out for "Dead ReLU" when permanently!). | |
2. GELU | Smooth non-zero gradient near $0$ | GPT-2, GPT-3, BERT feed-forward layers. | |
3. SiLU / Swish (SwiGLU) | Smooth, prevents dead neurons | Llama-3, Qwen-2.5, Mistral, DeepSeek Transformer blocks! | |
4. Tanh | $1 - \tanh^2(x)$ | LSTM gates, bounded hidden states. |
Part 2: Backpropagation & The Chain Rule of Calculus
In a computation graph , how does a small nudge in affect the final Loss ?
- Addition Gate (): Local derivative is
1.0Addition is a "Gradient Router" that passes the upstream gradient equally to both and ! - Multiplication Gate (): and Multiplication is a "Gradient Switcher"!
Part 3: Hands-On Lab — Building micrograd_engine.py From Scratch (micrograd_engine.py)
Check out src/micrograd_engine.py (in Tab 2: Codes — click Run in Browser to execute) and run the script and tests (where we verify that our custom autograd engine computes the exact same gradients to 7 decimal places as PyTorch!):
python week06_deep_learning_and_pytorch_from_scratch/day01_neurons_activations_and_backprop_autograd/src/micrograd_engine.py
python -m unittest discover -s week06_deep_learning_and_pytorch_from_scratch/day01_neurons_activations_and_backprop_autograd/tests -v
In Simple English: What Is Backpropagation (Chain Rule)?
Imagine a 3-person relay team passing a baton: Alice -> Bob -> Carol (Final Score). If Carol misses the target by 6 points, who should adjust and by how much?
- First calculate how much Carol's step caused the miss.
- Multiply that sensitivity backward through Bob (), then back to Alice!
That backward blame assignment is Backpropagation, and an Autograd engine records every
+,*, and activation step automatically so it can nudge every weight in the right direction! Runmicrograd_engine.pyin Tab 2 (Codes) right inside your browser to watch scalar autograd live!