Week 7 • Day 6Week 7
Day 6:Week 7, Day 6: Transformer Families — Bidirectional Encoder
Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz
Week 7, Day 6: Transformer Families — Bidirectional Encoder (BERT) vs Causal Decoder (GPT/Llama)
1. The Three Branches of the Transformer Family Tree
- Encoder-Only (Bidirectional Attention — Every token sees EVERY token):
- Models:
BERT,RoBERTa,DeBERTa-v3,BGE-M3,text-embedding-004 - Best For: Dense Vector Embeddings, Semantic Search, Cross-Encoder Reranking, Classification, NER.
- Models:
- Decoder-Only (Causal Lower-Triangular Mask — Token sees $1 \dots t$ only):
- Models:
GPT-4o,Llama-3,Claude,Gemini,Qwen,DeepSeek - Best For: Text Generation, Reasoning, Code & Agentic Workflows.
- Models:
- Encoder-Decoder (Bidirectional Encoder + Cross-Attention + Causal Decoder):
- Models:
T5,BART,Whisper(Audio Encoder Text Decoder) - Best For: Speech-to-Text (
Whisper), Multimodal Encoders, Seq2Seq Translation.
- Models:
2. Bi-Encoder (Embedding Model) vs Cross-Encoder (Reranker) in RAG
- Bi-Encoder (
O(1)dot product per candidate): Encodes Query and Document separately into vectors , computing . Searches10Mdocuments in5 msvia HNSW vector index. - Cross-Encoder Reranker (
O(T^2)full self-attention): Concatenates[CLS] Query [SEP] Document [SEP]and runs full bidirectional self-attention across every query token and document token simultaneously on the Top-50 candidates returned by the Bi-Encoder.
Comparing the 3 Transformer Architectures in Simple Terms
| Architecture | Reads Text How? | Famous Models | Best Production Job |
|---|---|---|---|
| 1. Encoder-Only | Both directions at once (sees left & right context simultaneously) | BERT, RoBERTa, BGE-M3 | Fast Embeddings, Semantic Search, Cross-Encoder Rerankers, Guardrail Classifiers (<15ms!). |
| 2. Decoder-Only | Left-to-right only (Causal Mask blocks future tokens) | GPT-4o, Claude, Llama 3, Qwen | Chatbots, Code Generation, Agent Tool Calling, Reasoning! |
| 3. Encoder-Decoder | Bidirectional reader + autoregressive writer | T5, Whisper (audio), BART | Machine Translation, Speech-to-Text transcription. |
Beginner Rule of Thumb: In modern AI engineering, Encoders retrieve and filter (fast, cheap), while Decoders reason and write (smart, flexible)!