Week 7, Day 7 (Mini-Project): Coding & Training a Modern Llama-Style MiniGPT from Scratch

1. Architecture of Our Modern MiniGPT (Llama-3 Blueprint)

In this Week 7 capstone mini-project, we assemble all the building blocks we derived across Days 1–5 into a complete, working Llama-style Autoregressive Language Model (MiniGPT) in PyTorch:

• RMSNorm Pre-Normalization
• RoPE Rotary Position Embeddings
• Grouped-Query Attention (GQA) with Causal Masking
• SwiGLU Feed-Forward Network
• Weight-Tied Token Embedding & LM Head (self.lm_head.weight = self.tok_emb.weight)

Week 7 Capstone Lab: From Zero Math to Working MiniGPT

Look at what your custom MiniGPT combines inside Tab 2 (Codes) — every modern component used by Llama 3 and Claude:

1. Token Embedding Table: Turns word IDs into vectors.
2. Causal Multi-Head Attention + RoPE: Lets tokens talk to earlier tokens without peeking ahead.
3. Pre-Norm RMSNorm + SwiGLU MLP + Residual Stream: Stacks clean Transformer blocks stably.
4. Autoregressive Generation Loop: Generates new tokens step-by-step using Temperature & Softmax!

Step-by-Step Study Checklist

• Read minigpt_llama.py in Tab 2 (Codes) top-to-bottom — notice how short (~150 lines!) an entire GPT architecture really is once you understand every piece!
• Study the Cross-Entropy Training Loss step: predicting token t+1 from tokens 0..t.
• Test yourself in Tab 3 (Interview Questions) and Tab 5 (Quiz) before starting Week 8!