Python Performance Optimization Playbook

1. Slashing Object Overhead with __slots__

By default, every Python class instance stores its attributes in an internal dictionary (__dict__), consuming ~152 bytes per object. Adding __slots__ tells CPython to store attributes in a fixed C-array without __dict__, shrinking object memory from 152 bytes to 48 bytes (68% savings)! For pipelines handling 10M records in memory, this saves over 1 GB of RAM!

2. Vectorization vs For Loops

Pure Python for loops iterate through boxed PyObject pointers. Using NumPy or Polars invokes compiled C SIMD instructions that operate on 16 numbers in a single clock cycle.

High-Performance Python Tuning: Vectorization Over Loops

• Avoid Python Loops on Millions of Rows: Replacing a pure-Python for row in rows dict calculation with NumPy/PyArrow/Polars vectorized arrays speeds up execution 30x–100x by executing tight C/SIMD machine loops without GIL or PyObject boxing overhead.
• Collection Lookup Complexity: Membership checks x in bad_ids_list scan O(N) per lookup; converting bad_ids to a set makes every lookup O(1) (turning 100M checks from hours into milliseconds!).
• String Concatenation: Never append strings with out += chunk inside large loops—accumulate in a list and run "".join(chunks) once.