Day 4:Week 9, Day 4: Dense Embedding Models — `InfoNCE` Contrastive Loss & Matryoshka Embeddings
Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz
Week 9, Day 4: Dense Embedding Models — InfoNCE Contrastive Loss & Matryoshka Embeddings (MRL)
1. How Dense Embedding Models Are Trained: InfoNCE Contrastive Loss
How does a Transformer Encoder (text-embedding-004, BGE-M3, E5, Nomic-Embed) learn to map semantically similar texts to nearby points on the unit hypersphere ?
Given a batch of (query, positive_passage) pairs , the model uses In-Batch Negatives ( for plus mined Hard Negatives ) and minimizes the temperature-scaled InfoNCE Contrastive Loss:
where is a temperature hyperparameter that sharpens the similarity distribution.
2. Matryoshka Representation Learning (MRL — Kusupati et al., 2022)
Standard 1536-dimensional or 3072-dimensional embeddings spread information evenly across all coordinates: if you slice the first 256 coordinates of an old embedding model, retrieval accuracy collapses!
Matryoshka Embeddings (used in text-embedding-004, OpenAI text-embedding-3, Nomic-Embed-v1.5) apply InfoNCE loss simultaneously at nested prefixes during training:
Full Matryoshka Embedding Vector (e.g., 768 dims):
[ ─── 64d (Coarse Core) ─── | ── 128d ── | ─── 256d (Sweet Spot!) ─── | ───── 768d (Fine Detail) ───── ]
The Mandatory Matryoshka Truncation Rule: Slice Re-Normalize!
When you truncate an -normalized vector to its first dimensions , its norm is now ! You must re-normalize after slicing:
In Simple English: How Embedding Models Learn & Why Matryoshka (MRL) Saves 80% RAM
- Contrastive Loss ( — Magnet Physics): Training an embedding model works like magnets: attract matching pairs (
"How do I reset my password?"+ Password Reset Doc) closer together, and repel unrelated negative docs away! - Matryoshka Embeddings (
MRL— Russian Nesting Dolls): Normally you need all 3,072 numbers per sentence. Matryoshka training packs the most important meaning into the first 256–512 numbers! You can slice off the tail and keep ~98% of search accuracy while shrinking your Vector Database RAM bill by 6x–12x!