Week 7: The Transformer Architecture from First Principles (Attention to Llama-3 / nanoGPT)

Month 2 — Classical ML, Deep Learning, Transformers & LLM Evaluation Goal: Open the black box of modern Large Language Models (GPT-4, Llama-3, Claude, Gemini) by deriving and coding every mathematical block of a modern Transformer in PyTorch from scratch.

---

Weekly Architecture & Daily Breakdown

• Day 1: Scaled Dot-Product Attention — $Q, K, V$ Projections & the $\frac{1}{\sqrt{d_k}}$ Variance Proof (day01_scaled_dot_product_attention_math/)
• Day 2: Multi-Head Attention (MHA) & Autoregressive Causal Masking (day02_multi_head_attention_and_causal_masking/)
• Day 3: Positional Information — Sinusoidal Encodings vs Rotary Position Embeddings (RoPE) (day03_positional_encodings_sinusoidal_vs_rope/)
• Day 4: Modern Transformer Block Internals — Residual Streams, Pre-Norm RMSNorm & SwiGLU FFN (day04_rmsnorm_swiglu_and_residual_streams/)
• Day 5: Inference Optimization — KV-Cache Mechanics, VRAM Math & Grouped-Query Attention (GQA) (day05_kv_cache_and_grouped_query_attention_gqa/)
• Day 6: Transformer Families — Bidirectional Encoder (BERT/Embeddings) vs Causal Decoder (GPT/Llama) (day06_encoder_decoder_bert_vs_gpt_architectures/)
• Day 7: Week 7 Mini-Project — Coding & Training a Modern MiniGPT (RoPE + RMSNorm + SwiGLU) from Scratch (day07_week7_mini_project_nanogpt_from_scratch/)