Loading Academy Hub...
Prevent silent data corruption in Pandas pipelines: chaining assignments, schema enforcement, and validation with Great Expectations.
This warning occurs when performing chained indexing:
df[df['amount'] > 100]['discount'] = 0.1
__getitem__ then __setitem__.__getitem__ returns a temporary copy. The assignment modifies the throwaway copy, leaving the original DataFrame completely unchanged!.loc[]:
df.loc[df['amount'] > 100, 'discount'] = 0.1In automated production pipelines, never assume incoming CSV/Parquet data adheres to your business rules. Always enforce expectations:
SettingWithCopyWarning Matters: Writing sub = df[df.a > 1]; sub["b"] = 0 leaves behavior undefined—sometimes mutating df, sometimes discarding the edit silently! Fix cleanly with explicit df.loc[df["a"] > 1, "b"] = 0 OR sub = df[df["a"] > 1].copy().DtypeWarning: Columns (12, 18) have mixed types: Happens when read_csv guesses types using chunk #1 (all ints) and hits "N/A" in row 500,000, creating mixed-type object columns that break downstream Parquet writes! Fix: pass explicit dtype={...} and na_values=["N/A", "NULL", ""].errors="coerce": Always log invalid_cnt = series.isna().sum() - raw_na after pd.to_numeric(..., errors="coerce") so corrupt strings don't vanish silently into NaN!Review the production code, company interview Q&As, incident traces, and comprehension quiz inside this lesson.
import pandas as pd
def validate_and_clean_orders(df: pd.DataFrame) -> pd.DataFrame:
# 1. Enforce critical non-null constraints
if df['order_id'].isnull().any():
null_count = df['order_id'].isnull().sum()
raise ValueError(f"CRITICAL: Found {null_count} records with NULL order_id!")
# 2. Enforce positive amounts
invalid_amounts = (df['amount'] <= 0).sum()
if invalid_amounts > 0:
print(f"WARNING: Dropping {invalid_amounts} records with negative or zero amount.")
df = df[df['amount'] > 0].copy()
return df
SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame.