Schema Drift: Reordered Columns

Medium
Schema
CSV
Union
Data Quality
Uber
Airbnb

Schema Drift: Reordered Columns

Two monthly extracts of the same table arrive with the columns in different orders. A positional union would silently mix amount into customer_id.

Read paths['january'] and paths['february'] as plain header CSV (all strings), project each one onto the canonical order order_id, customer_id, amount, cast the values, and return the combined table sorted by order_id.

A positional union is the trap: Spark matches columns by position for union, by name for unionByName.

Browser Engine: Evaluates PySpark DataFrame & SQL logic locally in Pyodide WebAssembly.

Input files

2 Files
Loaded automatically

These files are loaded into this browser when you run. Their paths are handed to your code as the third argument: solve(spark, dfs, paths). Read one with spark.read.csv(paths['users'], header=True), or skip the read and use dfs['users'].

paths[january]
csv
datasets/schema/orders_jan.csv
order_id(string)
customer_id(string)
amount(string)
paths[february]
csv
datasets/schema/orders_feb.csv
amount(string)
order_id(string)
customer_id(string)
Browser engine
Local files
Loading Python editor…
Click Run to try the sample cases, or Submit to check the hidden cases. First run caches the browser Python engine (~8–15s once); all code stays on your device.