Loading Academy Hub...
Build production medallion lakehouse pipelines with Unity Catalog governance, Auto Loader cloudFiles schema evolution, Bronze-to-Silver DLT streaming with expectation quarantine, Silver-to-Gold Liquid Clustering MERGE, and vector index synchronization.
Inspect console configs, simulate Happy Path vs Fault-Injection & DLQ recovery, or copy production IaC/code.
Centralized ACLs, row filters, column masks, and end-to-end data lineage across workspaces.
[!IMPORTANT] Delta Live Tables (DLT) is Databricks' declarative ETL framework that abstracts away infrastructure, streaming checkpoints, and state management. DLT pipelines run on Unity Catalog, enforcing three-tier namespace governance (
catalog.schema.table) and ACID data quality expectations.
format('cloudFiles')) to ingest arriving JSON/Parquet files with automatic schema inference and rescue data columns (_rescued_data).@dlt.expect_or_drop or @dlt.expect_or_fail to eliminate null primary keys or invalid business logic.CLUSTER BY (customer_id, event_date)) for sub-second business reporting.Legacy Hive partitioning (PARTITIONED BY date) frequently causes data skew and the small file problem. Delta Liquid Clustering dynamically reorganizes files based on write frequency and query patterns, supporting multiple cluster keys without over-partitioning penalties.
Review the production code, company interview Q&As, incident traces, and comprehension quiz inside this lesson.
import dlt
from pyspark.sql.functions import col, current_timestamp
# Bronze Table: Streaming Ingestion via Auto Loader
@dlt.table(
name="bronze_orders",
comment="Raw streaming orders ingested from S3 drop folder"
)
def bronze_orders():
return (
spark.readStream.format("cloudFiles")
.option("cloudFiles.format", "json")
.option("cloudFiles.schemaLocation", "/checkpoints/bronze_orders/schema")
.option("cloudFiles.inferColumnTypes", "true")
.load("s3://enterprise-lakehouse/landing/orders/")
)
# Silver Table: Cleansed Orders with Data Quality Expectations
@dlt.table(
name="silver_orders",
comment="Cleaned orders with verified foreign keys and positive amounts"
)
@dlt.expect_or_drop("valid_order_id", "order_id IS NOT NULL")
@dlt.expect_or_drop("positive_amount", "amount > 0")
def silver_orders():
return (
dlt.read_stream("bronze_orders")
.select(
col("order_id").cast("string"),
col("customer_id").cast("string"),
col("amount").cast("double"),
col("timestamp").cast("timestamp"),
current_timestamp().alias("ingested_at")
)
)com.databricks.pipelines.common.ExpectationFailedException: Expectation 'valid_order_id' failed on 42 records: [order_id IS NOT NULL]