Loading Academy Hub...
Break massive multi-gigabyte CSV and SQL query results into manageable chunks using pd.read_csv(chunksize=...) and stream aggregations directly to Parquet.
[!NOTE] Standard
pd.read_csv('large_file.csv')attempts to load all data into a contiguous memory block in RAM, requiring 5x-10x the raw file size in virtual memory. Passingchunksizeturns Pandas into a streaming batch engine.
[ 50 GB CSV / SQL Table ]
│
▼ (chunksize = 100,000 rows)
[ Iterative DataFrame Buffer in RAM: ~25 MB ]
│
▼ (Transform & Aggregate)
[ Append to Partitioned Parquet on S3 / Disk ]
chunksize: Reading with pd.read_csv(path, chunksize=100_000, usecols=[...]) yields bounded DataFrames so peak memory remains constant (~80MB) even across a 50GB input log!usecols): Never load 60 unused columns only to drop 55 of them after read_csv or read_parquet—specify usecols / Parquet columns=[...] at read time!Review the production code, company interview Q&As, incident traces, and comprehension quiz inside this lesson.
import pandas as pd
chunk_size = 100_000
total_revenue = 0.0
total_records = 0
# Stream processing: Memory never exceeds ~50 MB!
for chunk in pd.read_csv('massive_transactions.csv', chunksize=chunk_size, usecols=['amount', 'status']):
valid_txns = chunk[chunk['status'] == 'COMPLETED']
total_revenue += valid_txns['amount'].sum()
total_records += len(valid_txns)
print(f"Processed {total_records:,} transactions. Total: ${total_revenue:,.2f}")MemoryError: Unable to allocate 14.2 GiB for an array with shape (180000000, 10)