Week 6 • Day 2Week 6

Day 2:`PyTorch` Fundamentals — Tensors, Broadcasting, `autograd` & GPU Memory

Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz

Week 6 — Day 2: PyTorch Fundamentals — Tensors, Broadcasting, autograd & GPU Memory

Mentor's Note (10+ YOE Staff AI Engineer POV): Why is PyTorch (torch.Tensor) different from NumPy (np.ndarray)?

  1. Automatic Differentiation (requires_grad=True): Every operation on a PyTorch tensor automatically builds a dynamic computation graph (grad_fn) ready for .backward().
  2. Hardware Acceleration (device="cuda" / "mps" / "cpu"): Moving a 10,000 × 10,000 matrix from CPU RAM to GPU VRAM takes a single .to(device) call!
  3. Why with torch.no_grad(): Is Mandatory During LLM Inference: During inference/evaluation, you never call .backward(). Wrapping inference inside with torch.no_grad(): stops PyTorch from storing intermediate activation tensors for backpropagation—cutting RAM/VRAM usage by 50%+ and speeding up inference by 2x!

Part 1: Tensor Shapes, Reshaping (view vs reshape) & Broadcasting

In LLM & Transformer code, every tensor has a 3D or 4D shape:

  • (Batch_Size, Seq_Len, Hidden_Dim) = (B, T, D) (e.g., (32, 512, 768) = 32 sentences, 512 tokens per sentence, 768 numbers per token).
  • (Batch_Size, Num_Heads, Seq_Len, Head_Dim) = (B, H, T, d_k) inside Multi-Head Attention!

Part 2: Hands-On Lab — PyTorch Tensor Mechanics & torch.no_grad() (pytorch_tensor_mechanics.py)

Check out src/pytorch_tensor_mechanics.py (in Tab 2: Codes — click Run in Browser to execute) and run the script and tests:

powershell
python week06_deep_learning_and_pytorch_from_scratch/day02_pytorch_tensors_autograd_and_gpu_memory/src/pytorch_tensor_mechanics.py python -m unittest discover -s week06_deep_learning_and_pytorch_from_scratch/day02_pytorch_tensors_autograd_and_gpu_memory/tests -v

In Simple English: Tensors, Broadcasting & Why GPUs Are 50x Faster Than Loops

Think of a PyTorch Tensor as an upgraded spreadsheet built for graphics cards:

  • 0D Tensor (Scalar): One single temperature number (36.6).
  • 1D Tensor (Vector): A list of 3 word scores ([0.2, 0.9, -0.4]).
  • 2D Tensor (Matrix): A classroom grade table (30 students x 5 exams).
  • 3D Tensor: A batch of sentences (batch_size x sequence_length x embedding_dim).

What Is Broadcasting?

Suppose you have 1,000 student scores in a matrix and want to add +5 bonus points to every single entry. Instead of duplicating 5 a thousand times in RAM, Broadcasting virtually stretches that single scalar across the whole matrix with zero extra memory copy!

Crucial GPU Rule (Memory Lifecycle)

Every tensor lives either in CPU RAM (cpu) or GPU VRAM (cuda). Trying to multiply a CPU tensor by a GPU tensor crashes immediately with RuntimeError: Expected all tensors to be on the same device! Always keep model weights and input batches on the same device (.to(device)).