Loading Academy Hub...
Eliminate 99% of Python loop latency: replace row-by-row df.apply() and iterrows() with SIMD vectorized NumPy expressions and np.select() conditional arrays.
[!IMPORTANT] Staff Engineer Rule: Calling
df.apply(lambda row: ..., axis=1)is simply a slow Pythonforloop wrapped in syntactic sugar. It converts every row into a separatepd.Seriesobject, discarding all C-level vectorized SIMD acceleration.
| Method | Execution Time | Relative Speed |
|---|---|---|
for index, row in df.iterrows(): | 420.5 seconds | 1x (Abysmal) |
df.apply(lambda row: ..., axis=1) | 84.2 seconds | 5x faster |
np.select(conditions, choices) | 0.34 seconds | 1,200x faster! |
1x baseline): NumPy ufuncs & Pandas vectorized arithmetic (df["a"] * df["b"], np.where, np.select, .str Arrow accessors).2x–4x): Categorical / mapping lookups (.map(lookup_dict)).50x–200x): df.apply(func, axis=1) (still executes Python callback row-by-row!).500x+): df.iterrows() (constructs a brand-new pd.Series object for EVERY SINGLE ROW—never use in production ETL!).Review the production code, company interview Q&As, incident traces, and comprehension quiz inside this lesson.
import pandas as pd
import numpy as np
df = pd.DataFrame({
'amount': np.random.uniform(10, 5000, size=1_000_000),
'is_fraud_flag': np.random.choice([True, False], size=1_000_000, p=[0.02, 0.98])
})
# SLOW: df.apply() takes ~15 seconds
# df['risk_tier'] = df.apply(lambda r: 'HIGH' if r['is_fraud_flag'] or r['amount'] > 2000 else 'NORMAL', axis=1)
# FAST: np.select() takes 0.04 seconds (375x faster!)
conditions = [
df['is_fraud_flag'] == True,
df['amount'] > 2000.0
]
choices = ['CRITICAL_RISK', 'ELEVATED_RISK']
df['risk_tier'] = np.select(conditions, choices, default='NORMAL')Job killed: Execution time exceeded 30-minute threshold on 5M rows