Week 5 • Day 4Week 5

Day 4:Text Vectorization (`TF-IDF`) & Sub-Millisecond Classical Text Classifiers

Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz

Week 5 — Day 4: Text Vectorization (TF-IDF) & Sub-Millisecond Classical Text Classifiers

Mentor's Note (10+ YOE Staff AI Engineer POV): Why do we study TF-IDF (Term Frequency – Inverse Document Frequency) on Day 4?

  1. It is the direct ancestor of BM25 Sparse Search in Month 3 Hybrid RAG!
  2. When NOT to use an LLM: If an interviewer asks: "We have 50,000 server log lines per second and want to classify them into 5 known alert categories with < 1ms latency and $0 API cost—how do you build it?" If you say "Call GPT-4 on all 50,000 logs per second", you fail the interview immediately (that would cost $50,000/day and take 800ms!). The Senior AI Engineer answer is: Train a TF-IDF + Logistic Regression / LinearSVC / XGBoost classifier that runs in 0.05ms on CPU!

Part 1: The Mathematics of TF-IDF

Why can't we just count raw word frequencies (Bag-of-Words)? Because words like "the", "server", or "log" appear in 100% of documents and carry almost zero distinguishing signal, whereas rare technical words like "oomkilled" or "flushdb" appear in only 2% of documents and carry huge signal!

TF-IDF multiplies two numbers together for every word tt in document dd across a corpus of NN documents:

  1. Term Frequency (TF(t,d)\text{TF}(t, d)): How often word tt appears in document dd: TF(t,d)=count(t in d)total words in d\text{TF}(t, d) = \frac{\text{count}(t \text{ in } d)}{\text{total words in } d}
  2. Smooth Inverse Document Frequency (IDF(t)\text{IDF}(t)): Penalizes common words and boosts rare words (df(t)\text{df}(t) = number of documents containing word tt): IDF(t)=ln⁡(1+N1+df(t))+1\text{IDF}(t) = \ln\left(\frac{1 + N}{1 + \text{df}(t)}\right) + 1
  3. Final Score: TF-IDF(t,d)=TF(t,d)×IDF(t)\text{TF-IDF}(t, d) = \text{TF}(t, d) \times \text{IDF}(t)

(Note: Why do we add +1+1 inside 1+N1+df(t)\frac{1+N}{1+\text{df}(t)}? That is Laplace Smoothing so we never divide by zero if a word appears in 0 training documents!)


Part 2: Hands-On Lab — TF-IDF Vectorizer & Sub-Millisecond Incident Classifier (tfidf_classifier.py)

Check out src/tfidf_classifier.py (in Tab 2: Codes — click Run in Browser to execute) and run the script and tests:

powershell
python week05_math_data_science_and_classical_ml/day04_tfidf_vectorization_and_fast_text_classifiers/src/tfidf_classifier.py python -m unittest discover -s week05_math_data_science_and_classical_ml/day04_tfidf_vectorization_and_fast_text_classifiers/tests -v

In Simple English: How TF-IDF Finds Important Words

Imagine searching 1,000 hospital reports for the query "urgent insulin allergy":

  • TF (Term Frequency): How often does the word appear in this one report?
  • IDF (Inverse Document Frequency): How rare is the word across all 1,000 reports?
    • Common words like patient or the appear in 990 reports →\rightarrow tiny IDF score (almost ignored!).
    • Rare words like insulin appear in only 12 reports →\rightarrow huge IDF score!
  • Result: TF×IDF\text{TF} \times \text{IDF} automatically boosts specific keywords and ignores boring filler words in under 1 millisecond!