Azure Data Factory (ADF) & Azure Databricks Medallion: Interactive Pipeline Studio

[!IMPORTANT] Azure Data Factory (ADF) is Microsoft Azure's cloud-native ETL and data orchestration service. In modern enterprise lakehouses, ADF serves as the control-plane orchestrator, coordinating data movement from on-premises sources into ADLS Gen2, and handing off heavy transformations to Azure Databricks Photon clusters.

1. Zero-Trust Pipeline Architecture

A production ADF pipeline follows strict security and reliability rules:

1. Azure Key Vault (AKV) Linked Service: All database connection strings, passwords, and service principal secrets are stored in AKV and retrieved dynamically via System-Assigned Managed Identity.
2. Secure Input & Secure Output: Always enabled on activities handling secrets to prevent sensitive credentials from leaking into ADF execution logs.
3. Control-Table Watermarking: Incremental ingestion uses an Azure SQL control table to query WHERE ModifiedDate > LastWatermark, ensuring idempotent, incremental processing without full table scans.

2. High-Throughput Ingestion & Integration Runtimes

• Self-Hosted Integration Runtime (SHIR): Connects ADF securely to on-premises Oracle and SQL Server databases without opening inbound firewall ports. Limit batchCount to 10-15 per node to avoid ephemeral port exhaustion.
• Copy Activity DIU Scaling: Assign 32 to 256 Data Integration Units (DIUs) and enable physical source partitioning to divide massive tables into 8-16 parallel streaming read slices.
• Databricks Job Clusters: Always launch new automated job clusters with Photon runtime rather than shared interactive clusters, reducing DBU costs by up to 50% while ensuring memory isolation.

3. Visual Pipeline Canvas Simulation

Interact with the drag-and-drop pipeline canvas above to inspect real ARM parameters, activity dependencies, and Senior Architect failure recovery strategies.