Loading Academy Hub...
Implement sub-minute declarative streaming pipelines in Snowflake: Snowpipe Streaming low-latency ingestion, CDC change tracking with Streams and Tasks, declarative Dynamic Tables with targeted lag policies, and Cortex LLM feature extraction.
Inspect console configs, simulate Happy Path vs Fault-Injection & DLQ recovery, or copy production IaC/code.
Direct streaming into Snowflake micro-partitions with sub-5 second latency over gRPC.
[!IMPORTANT] Snowflake's modern streaming architecture has evolved beyond legacy batch tasks. With Snowpipe Streaming and Dynamic Tables, data engineers build declarative, continuous data transformation pipelines with sub-minute lag targets and automated DAG dependency resolution.
TARGET_LAG = '1 minute'). Snowflake computes the optimal incremental refresh schedule automatically.Review the production code, company interview Q&As, incident traces, and comprehension quiz inside this lesson.
-- 1. Raw Landing Table with Change Tracking Enabled
CREATE OR REPLACE TABLE raw_customer_events (
event_id VARCHAR(64),
customer_id VARCHAR(64),
event_type VARCHAR(32),
payload VARIANT,
ingested_at TIMESTAMP_NTZ DEFAULT CURRENT_TIMESTAMP()
);
-- 2. Declarative Dynamic Table with 1-Minute Target Freshness
CREATE OR REPLACE DYNAMIC TABLE silver_customer_activity
TARGET_LAG = '1 minute'
WAREHOUSE = compute_wh
AS
SELECT
event_id,
customer_id,
event_type,
payload:device_os::VARCHAR AS device_os,
payload:amount::NUMBER(12, 2) AS transaction_amount,
ingested_at
FROM raw_customer_events
WHERE event_type IS NOT NULL;
-- 3. Gold Aggregated Dynamic Table depending on Silver
CREATE OR REPLACE DYNAMIC TABLE gold_customer_spend_hourly
TARGET_LAG = '5 minutes'
WAREHOUSE = compute_wh
AS
SELECT
customer_id,
DATE_TRUNC('hour', ingested_at) AS event_hour,
COUNT(*) AS total_transactions,
SUM(transaction_amount) AS total_spend
FROM silver_customer_activity
GROUP BY customer_id, DATE_TRUNC('hour', ingested_at);SQL compilation error: Dynamic table 'SILVER_CUSTOMER_ACTIVITY' missed target lag of 60 seconds due to warehouse queueing on COMPUTE_WH.