Classic ML Debugging, Data Leakage, Metrics & Production Evals (30 Q&As + Code)

Target leakage—including a post-event attribute (like chargeback_settled_30d, manual review outcome, or update timestamp) unavailable when checkout authorization happens. Fastest diagnostic ("Drop and Watch"): ablate suspicious columns one by one; if dropping 1 feature plunges AUC from 0.99 toward 0.52, that feature leaked the label.

# BAD: post-event column unknown at real-time checkout
FEATURES = ["amount", "velocity_1h", "chargeback_settled_30d"]
# GOOD: strict point-in-time feature audit (available <= decision_ts)
SAFE_FEATURES = ["amount", "velocity_1h", "merchant_risk_30d_lag"]

Training-serving skew occurs whenever live inference computes feature values differently from offline training—such as duplicating z-score constants (STALE_MEAN = 950.0), using mismatched timezone cutoffs, or handling NaNs differently. Fix cleanly by fitting transforms strictly inside one sklearn.pipeline.Pipeline / ONNX / Feature Store artifact shared verbatim across offline and online endpoints.

from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
# Freeze training statistics inside ONE versioned artifact
pipe = Pipeline([("impute", SimpleImputer(strategy="median")), ("scale", StandardScaler())])
pipe.fit(X_train) # Fit ONLY on X_train!