Week 7 • Day 6Week 7

Day 6:Week 7, Day 6: Transformer Families — Bidirectional Encoder

Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz

Week 7, Day 6: Transformer Families — Bidirectional Encoder (BERT) vs Causal Decoder (GPT/Llama)


1. The Three Branches of the Transformer Family Tree

  1. Encoder-Only (Bidirectional Attention — Every token sees EVERY token):
    • Models: BERT, RoBERTa, DeBERTa-v3, BGE-M3, text-embedding-004
    • Best For: Dense Vector Embeddings, Semantic Search, Cross-Encoder Reranking, Classification, NER.
  2. Decoder-Only (Causal Lower-Triangular Mask — Token tt sees $1 \dots t$ only):
    • Models: GPT-4o, Llama-3, Claude, Gemini, Qwen, DeepSeek
    • Best For: Text Generation, Reasoning, Code & Agentic Workflows.
  3. Encoder-Decoder (Bidirectional Encoder + Cross-Attention + Causal Decoder):
    • Models: T5, BART, Whisper (Audio Encoder →\rightarrow Text Decoder)
    • Best For: Speech-to-Text (Whisper), Multimodal Encoders, Seq2Seq Translation.

2. Bi-Encoder (Embedding Model) vs Cross-Encoder (Reranker) in RAG

  1. Bi-Encoder (O(1) dot product per candidate): Encodes Query QQ and Document DD separately into vectors u,v∈Rdu, v \in \mathbb{R}^d, computing cos⁡(u,v)\cos(u, v). Searches 10M documents in 5 ms via HNSW vector index.
  2. Cross-Encoder Reranker (O(T^2) full self-attention): Concatenates [CLS] Query [SEP] Document [SEP] and runs full bidirectional self-attention across every query token and document token simultaneously on the Top-50 candidates returned by the Bi-Encoder.

Comparing the 3 Transformer Architectures in Simple Terms

ArchitectureReads Text How?Famous ModelsBest Production Job
1. Encoder-OnlyBoth directions at once (sees left & right context simultaneously)BERT, RoBERTa, BGE-M3Fast Embeddings, Semantic Search, Cross-Encoder Rerankers, Guardrail Classifiers (<15ms!).
2. Decoder-OnlyLeft-to-right only (Causal Mask blocks future tokens)GPT-4o, Claude, Llama 3, QwenChatbots, Code Generation, Agent Tool Calling, Reasoning!
3. Encoder-DecoderBidirectional reader + autoregressive writerT5, Whisper (audio), BARTMachine Translation, Speech-to-Text transcription.

Beginner Rule of Thumb: In modern AI engineering, Encoders retrieve and filter (fast, cheap), while Decoders reason and write (smart, flexible)!