Week 9 • Day 4Week 9

Day 4:Week 9, Day 4: Dense Embedding Models — `InfoNCE` Contrastive Loss & Matryoshka Embeddings

Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz

Week 9, Day 4: Dense Embedding Models — InfoNCE Contrastive Loss & Matryoshka Embeddings (MRL)


1. How Dense Embedding Models Are Trained: InfoNCE Contrastive Loss

How does a Transformer Encoder (text-embedding-004, BGE-M3, E5, Nomic-Embed) learn to map semantically similar texts to nearby points on the unit hypersphere Sd−1\mathbb{S}^{d-1}?

Given a batch of BB (query, positive_passage) pairs {(qi,di+)}i=1B\{(q_i, d_i^+)\}_{i=1}^B, the model uses In-Batch Negatives (dj+d_j^+ for j≠ij \ne i plus mined Hard Negatives di−d_i^-) and minimizes the temperature-scaled InfoNCE Contrastive Loss:

LInfoNCE=−1B∑i=1Blog⁡exp⁡(cos⁡(qi,di+)/τ)exp⁡(cos⁡(qi,di+)/τ)+∑j≠iexp⁡(cos⁡(qi,dj)/τ)\mathcal{L}_{\text{InfoNCE}} = -\frac{1}{B} \sum_{i=1}^{B} \log \frac{\exp\big(\cos(\mathbf{q}_i, \mathbf{d}_i^+) / \tau\big)}{\exp\big(\cos(\mathbf{q}_i, \mathbf{d}_i^+) / \tau\big) + \sum_{j \ne i} \exp\big(\cos(\mathbf{q}_i, \mathbf{d}_j) / \tau\big)}

where τ∈[0.01,0.07]\tau \in [0.01, 0.07] is a temperature hyperparameter that sharpens the similarity distribution.


2. Matryoshka Representation Learning (MRL — Kusupati et al., 2022)

Standard 1536-dimensional or 3072-dimensional embeddings spread information evenly across all coordinates: if you slice the first 256 coordinates of an old embedding model, retrieval accuracy collapses!

Matryoshka Embeddings (used in text-embedding-004, OpenAI text-embedding-3, Nomic-Embed-v1.5) apply InfoNCE loss simultaneously at nested prefixes m∈{64,128,256,512,768,1536}m \in \{64, 128, 256, 512, 768, 1536\} during training:

LMRL=∑m∈Mcm⋅LInfoNCE(q1:m,d1:m)\mathcal{L}_{\text{MRL}} = \sum_{m \in \mathcal{M}} c_m \cdot \mathcal{L}_{\text{InfoNCE}}\big(\mathbf{q}_{1:m}, \mathbf{d}_{1:m}\big)

code
Full Matryoshka Embedding Vector (e.g., 768 dims): [ ─── 64d (Coarse Core) ─── | ── 128d ── | ─── 256d (Sweet Spot!) ─── | ───── 768d (Fine Detail) ───── ]

The Mandatory Matryoshka Truncation Rule: Slice →\rightarrow L2L_2 Re-Normalize!

When you truncate an L2L_2-normalized vector v∈RD\mathbf{v} \in \mathbb{R}^{D} to its first k<Dk < D dimensions v1:k\mathbf{v}_{1:k}, its norm ∥v1:k∥2\|\mathbf{v}_{1:k}\|_2 is now <1.0< 1.0! You must re-normalize after slicing:

v^1:k=v1:k∥v1:k∥2\hat{\mathbf{v}}_{1:k} = \frac{\mathbf{v}_{1:k}}{\|\mathbf{v}_{1:k}\|_2}


In Simple English: How Embedding Models Learn & Why Matryoshka (MRL) Saves 80% RAM

  • Contrastive Loss (LInfoNCE\mathcal{L}_{\text{InfoNCE}} — Magnet Physics): Training an embedding model works like magnets: attract matching pairs ("How do I reset my password?" + Password Reset Doc) closer together, and repel unrelated negative docs away!
  • Matryoshka Embeddings (MRL — Russian Nesting Dolls): Normally you need all 3,072 numbers per sentence. Matryoshka training packs the most important meaning into the first 256–512 numbers! You can slice off the tail and keep ~98% of search accuracy while shrinking your Vector Database RAM bill by 6x–12x!