Optimization: Repartition by Partition Key Before Write

Medium
Optimization
Repartition
Parquet
Write
Amazon
Databricks
Netflix

Optimization: Repartition by Partition Key Before Write

Writing unaligned partitions creates dozens of tiny files across every output directory.

  1. Call .repartition(4, 'region') on dfs['sales'] sorted by sale_id.
  2. Write the repartitioned dataset as Parquet in overwrite mode partitioned by 'region' to '/output/partitioned_sales'.
  3. Return the DataFrame (sale_id, region, amount) sorted by sale_id.
Browser Engine: Evaluates PySpark DataFrame & SQL logic locally in Pyodide WebAssembly.

Input data

0 DataFrames

These DataFrames are passed to solve(spark, dfs). Access one with dfs['name'].

Browser engine
Loading Python editor…
Click Run to try the sample cases, or Submit to check the hidden cases. First run caches the browser Python engine (~8–15s once); all code stays on your device.