Fast Columnar I/O: Parquet & Apache Arrow Engine

[!IMPORTANT] Staff Engineer Rule: Never store intermediate production pipeline datasets as .csv or .json. CSV files lack schema enforcement, have no column statistics, require full scans for single-column queries, and consume 5x-10x more disk space.

1. CSV vs Parquet Performance Benchmark (10 Million Rows)

Columnar Parquet & PyArrow Engine Best Practices

• Partitioned Writes + Compression: Write analytical outputs via df.to_parquet(path, engine="pyarrow", compression="snappy", partition_cols=["event_date"])—ZSTD saves even more S3 storage (~25–35% smaller than Snappy) for archival Silver/Gold layers.
• Zero-Copy Arrow Interop: Using dtype_backend="pyarrow" lets Pandas share memory buffers seamlessly with PyArrow, DuckDB, Polars, and PySpark via Arrow IPC/Zero-Copy without expensive Python-to-JVM row deserialization!
• Avoid Small Files in Partitioned Writes: Do NOT set partition_cols=["user_id"]—creating 100,000 tiny directories will cripple S3 and Athena metadata listing.