Schema Drift: Missing Required Column

Medium
Schema
Data Quality
CSV
Validation
Stripe
Databricks

Schema Drift: Missing Required Column

An upstream job sometimes drops customer_id from orders.csv. Loading that file silently produces nulls, so the bad rows only surface days later as unattributed revenue.

Rule: inspect the header before you apply the schema. If customer_id is missing, raise a ValueError whose message contains customer_id. When the column is present, return order_id, customer_id, amount sorted by order_id.

This is a fail-fast contract: a broken file must stop the pipeline, not produce a table with nulls.

header = spark.read.option('header', True).csv(paths['orders'])
Browser Engine: Evaluates PySpark DataFrame & SQL logic locally in Pyodide WebAssembly.

Input files

1 File
Loaded automatically

These files are loaded into this browser when you run. Their paths are handed to your code as the third argument: solve(spark, dfs, paths). Read one with spark.read.csv(paths['users'], header=True), or skip the read and use dfs['users'].

paths[orders]
csv
datasets/schema/orders_missing_customer.csv
order_id(string)
amount(string)
currency(string)
Browser engine
Local files
Loading Python editor…
Click Run to try the sample cases, or Submit to check the hidden cases. First run caches the browser Python engine (~8–15s once); all code stays on your device.