Day 6:ML Evaluation Metrics Deep Dive — Confusion Matrix, Precision, Recall, F1 & ROC-AUC
Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz
Week 5 — Day 6: ML Evaluation Metrics Deep Dive — Confusion Matrix, Precision, Recall, F1 & ROC-AUC
Mentor's Note (10+ YOE Staff AI Engineer POV): Imagine a candidate tells an interviewer: "My prompt-injection detector achieved 99.0% Accuracy!" The interviewer asks: "What percentage of your test dataset was actually prompt injections?" Candidate: "1.0%." Interviewer: "So a dummy function
return Falsethat NEVER catches a single attack also gets 99.0% Accuracy!"That is called The Accuracy Paradox on Imbalanced Data—and today you master the exact mathematical metrics (Precision, Recall, F1-Score, ROC-AUC, and PR-AUC) that every Senior AI Engineer uses!
Part 1: The Confusion Matrix & The 4 Core Metrics
Predicted Positive (1 = Alert / Attack) | Predicted Negative (0 = Normal) | |
|---|---|---|
Actual Positive (1) | True Positive () — Correctly caught! | False Negative () — Missed a real attack! (Type II Error) |
Actual Negative (0) | False Positive () — False alarm! (Type I Error) | True Negative () — Correctly ignored! |
1.1 Precision ("When the model cries Wolf, how often is there actually a Wolf?")
- Optimize for High Precision when False Positives () are expensive: e.g., An autonomous AI Agent that automatically blocks a paying customer's credit card or deletes a user account!
1.2 Recall / Sensitivity ("Out of all real Wolves in the forest, how many did we catch?")
- Optimize for High Recall when False Negatives () are catastrophic: e.g., Detecting
CRITICALproduction database outages, cancer screening, or catching PII/secret leaks before sending text to an external LLM!
1.3 F1-Score (Harmonic Mean of Precision and Recall)
Why do we use the Harmonic Mean instead of the simple average ?
Because if a dummy model predicts 1 for everything, it gets and . The simple average would be a misleading 0.505 (50.5%), whereas the Harmonic Mean (F1) punishes extreme imbalance and drops to 0.0198 (1.9%)!
Part 2: Hands-On Lab — Classification Metrics & Threshold Optimizer (metrics_auditor.py)
Check out src/metrics_auditor.py (in Tab 2: Codes — click Run in Browser to execute) and run the script and tests:
python week05_math_data_science_and_classical_ml/day06_ml_evaluation_metrics_precision_recall_f1_auc/src/metrics_auditor.py
python -m unittest discover -s week05_math_data_science_and_classical_ml/day06_ml_evaluation_metrics_precision_recall_f1_auc/tests -v
In Simple English: Spam Filter vs Cancer Screening (Precision vs Recall)
Never get Precision and Recall confused again using two real-world jobs:
- Precision ("How trustworthy are my alarms?"): Out of 10 alerts the AI flagged as urgent, 9 were truly urgent Precision = $9 / 10 = 90%$. Vital for spam filters (you don't want your bank OTP sent to Spam!).
- Recall ("Did I catch all real emergencies?"): Out of 10 real heart attacks in the ER, the AI caught 8 and missed 2 Recall = $8 / 10 = 80%$. Vital for medical screening & fraud!
- Score: A single balanced score () that punishes cheating (e.g., 100% Recall by guessing "YES" on everything gives poor Precision, dragging down!).