Week 5 • Day 6Week 5

Day 6:ML Evaluation Metrics Deep Dive — Confusion Matrix, Precision, Recall, F1 & ROC-AUC

Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz

Week 5 — Day 6: ML Evaluation Metrics Deep Dive — Confusion Matrix, Precision, Recall, F1 & ROC-AUC

Mentor's Note (10+ YOE Staff AI Engineer POV): Imagine a candidate tells an interviewer: "My prompt-injection detector achieved 99.0% Accuracy!" The interviewer asks: "What percentage of your test dataset was actually prompt injections?" Candidate: "1.0%." Interviewer: "So a dummy function return False that NEVER catches a single attack also gets 99.0% Accuracy!"

That is called The Accuracy Paradox on Imbalanced Data—and today you master the exact mathematical metrics (Precision, Recall, F1-Score, ROC-AUC, and PR-AUC) that every Senior AI Engineer uses!


Part 1: The Confusion Matrix & The 4 Core Metrics

Predicted Positive (1 = Alert / Attack)Predicted Negative (0 = Normal)
Actual Positive (1)True Positive (TPTP) — Correctly caught!False Negative (FNFN) — Missed a real attack! (Type II Error)
Actual Negative (0)False Positive (FPFP) — False alarm! (Type I Error)True Negative (TNTN) — Correctly ignored!

1.1 Precision ("When the model cries Wolf, how often is there actually a Wolf?")

Precision=TPTP+FP\text{Precision} = \frac{TP}{TP + FP}

  • Optimize for High Precision when False Positives (FPFP) are expensive: e.g., An autonomous AI Agent that automatically blocks a paying customer's credit card or deletes a user account!

1.2 Recall / Sensitivity ("Out of all real Wolves in the forest, how many did we catch?")

Recall=TPTP+FN\text{Recall} = \frac{TP}{TP + FN}

  • Optimize for High Recall when False Negatives (FNFN) are catastrophic: e.g., Detecting CRITICAL production database outages, cancer screening, or catching PII/secret leaks before sending text to an external LLM!

1.3 F1-Score (Harmonic Mean of Precision and Recall)

Why do we use the Harmonic Mean instead of the simple average P+R2\frac{P + R}{2}? Because if a dummy model predicts 1 for everything, it gets Recall=1.00\text{Recall} = 1.00 and Precision=0.01\text{Precision} = 0.01. The simple average would be a misleading 0.505 (50.5%), whereas the Harmonic Mean (F1) punishes extreme imbalance and drops to 0.0198 (1.9%)! F1=2⋅Precision⋅RecallPrecision+Recall=2TP2TP+FP+FNF_1 = \frac{2 \cdot \text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}} = \frac{2TP}{2TP + FP + FN}


Part 2: Hands-On Lab — Classification Metrics & Threshold Optimizer (metrics_auditor.py)

Check out src/metrics_auditor.py (in Tab 2: Codes — click Run in Browser to execute) and run the script and tests:

powershell
python week05_math_data_science_and_classical_ml/day06_ml_evaluation_metrics_precision_recall_f1_auc/src/metrics_auditor.py python -m unittest discover -s week05_math_data_science_and_classical_ml/day06_ml_evaluation_metrics_precision_recall_f1_auc/tests -v

In Simple English: Spam Filter vs Cancer Screening (Precision vs Recall)

Never get Precision and Recall confused again using two real-world jobs:

  • Precision ("How trustworthy are my alarms?"): Out of 10 alerts the AI flagged as urgent, 9 were truly urgent →\rightarrow Precision = $9 / 10 = 90%$. Vital for spam filters (you don't want your bank OTP sent to Spam!).
  • Recall ("Did I catch all real emergencies?"): Out of 10 real heart attacks in the ER, the AI caught 8 and missed 2 →\rightarrow Recall = $8 / 10 = 80%$. Vital for medical screening & fraud!
  • F1F_1 Score: A single balanced score (2⋅P⋅RP+R\frac{2 \cdot P \cdot R}{P + R}) that punishes cheating (e.g., 100% Recall by guessing "YES" on everything gives poor Precision, dragging F1F_1 down!).