Week 6 • Day 1Week 6

Day 1:How Neural Networks Learn — Neurons, Activations & Coding `Autograd` Backpropagation From Scratch

Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz

Week 6 — Day 1: How Neural Networks Learn — Neurons, Activations & Coding Autograd Backpropagation From Scratch

Mentor's Note (10+ YOE Staff AI Engineer POV): Why did Andrej Karpathy (founding member of OpenAI, former Director of AI at Tesla) start his famous Deep Learning course by coding a tiny Value autograd engine (micrograd)? Because if you understand how loss.backward() walks backward through a Directed Acyclic Graph (DAG) applying the Chain Rule of Calculus, then PyTorch, Transformers, and LoRA fine-tuning become 100% transparent!


Part 1: Why Neural Networks Need Non-Linear Activation Functions

Suppose you stack two linear layers (y1=W1x+b1y_1 = W_1 x + b_1 and y2=W2y1+b2y_2 = W_2 y_1 + b_2) without a non-linear activation function in between: y2=W2(W1x+b1)+b2=(W2W1)x+(W2b1+b2)=Wcombinedx+bcombinedy_2 = W_2 (W_1 x + b_1) + b_2 = (W_2 W_1) x + (W_2 b_1 + b_2) = W_{\text{combined}} x + b_{\text{combined}} Proof: A 100-layer neural network without non-linear activation functions collapses mathematically into a 1-layer linear regression!

The 4 Essential Activation Functions in Modern AI

ActivationFormulaDerivativeWhere Used in Modern AI
1. ReLUmax⁡(0,x)\max(0, x)$1 \text{ if } x > 0 \text{ else } 0$Classic CNNs & MLPs (Watch out for "Dead ReLU" when x<0x < 0 permanently!).
2. GELUx⋅Φ(x)x \cdot \Phi(x)Smooth non-zero gradient near $0$GPT-2, GPT-3, BERT feed-forward layers.
3. SiLU / Swish (SwiGLU)x⋅σ(x)=x1+e−xx \cdot \sigma(x) = \frac{x}{1 + e^{-x}}Smooth, prevents dead neuronsLlama-3, Qwen-2.5, Mistral, DeepSeek Transformer blocks!
4. Tanhex−e−xex+e−x∈(−1,1)\frac{e^x - e^{-x}}{e^x + e^{-x}} \in (-1, 1)$1 - \tanh^2(x)$LSTM gates, bounded hidden states.

Part 2: Backpropagation & The Chain Rule of Calculus

In a computation graph x→u=g(x)→L=f(u)x \to u = g(x) \to L = f(u), how does a small nudge in xx affect the final Loss LL? ∂L∂x=∂L∂u⋅∂u∂x(Upstream Gradient×Local Derivative)\frac{\partial L}{\partial x} = \frac{\partial L}{\partial u} \cdot \frac{\partial u}{\partial x} \quad (\text{Upstream Gradient} \times \text{Local Derivative})

  • Addition Gate (c=a+bc = a + b): Local derivative is 1.0   ⟹  \implies Addition is a "Gradient Router" that passes the upstream gradient equally to both aa and bb!
  • Multiplication Gate (c=a⋅bc = a \cdot b): ∂c∂a=b\frac{\partial c}{\partial a} = b and ∂c∂b=a\frac{\partial c}{\partial b} = a   ⟹  \implies Multiplication is a "Gradient Switcher"!

Part 3: Hands-On Lab — Building micrograd_engine.py From Scratch (micrograd_engine.py)

Check out src/micrograd_engine.py (in Tab 2: Codes — click Run in Browser to execute) and run the script and tests (where we verify that our custom autograd engine computes the exact same gradients to 7 decimal places as PyTorch!):

powershell
python week06_deep_learning_and_pytorch_from_scratch/day01_neurons_activations_and_backprop_autograd/src/micrograd_engine.py python -m unittest discover -s week06_deep_learning_and_pytorch_from_scratch/day01_neurons_activations_and_backprop_autograd/tests -v

In Simple English: What Is Backpropagation (Chain Rule)?

Imagine a 3-person relay team passing a baton: Alice -> Bob -> Carol (Final Score). If Carol misses the target by 6 points, who should adjust and by how much?

  • First calculate how much Carol's step caused the miss.
  • Multiply that sensitivity backward through Bob (∂L∂u⋅∂u∂x\frac{\partial L}{\partial u} \cdot \frac{\partial u}{\partial x}), then back to Alice! That backward blame assignment is Backpropagation, and an Autograd engine records every +, *, and activation step automatically so it can nudge every weight in the right direction! Run micrograd_engine.py in Tab 2 (Codes) right inside your browser to watch scalar autograd live!