Loading Academy Hub...
Master enterprise failure forensics and production triage: Azure Monitor, Log Analytics, Kusto Query Language (KQL), ADF CopyActivity failure triage, Synapse tempdb exhaustion, and ADLS Gen2 HTTP 429 throttling mitigation.
[!NOTE] Senior data engineers are defined by their ability to troubleshoot complex, distributed failure cascades under strict production SLAs. This module provides a diagnostic runbook for the top Azure data platform production incidents.
All ADF pipeline runs, Synapse queries, and ADLS Gen2 storage metrics should stream into an Azure Log Analytics Workspace. Using Kusto Query Language (KQL), engineers query execution logs across all resources from a single console:
ErrorCode=UserErrorFailedFileOperation): Occurs when source files are corrupted, truncated, or locked by another writer during ingestion.Error 110802): Caused by large hash join shuffles or unpartitioned CTAS queries running under small resource classes (smallrc).ServerBusyException): Triggered when hundreds of concurrent Spark executors exceed the storage account's 20,000 IOPS limit.403 AuthorizationPermissionMismatch, ADF IR Timeouts & Synapse Memory Grants403 Variants: AuthorizationPermissionMismatch means your Entra ID principal authenticated cleanly, yet lacks Storage Blob Data Contributor / Reader RBAC on the storage account/container (Owner or Contributor control-plane roles do NOT grant data-plane read/write!). In contrast, AuthorizationFailure usually indicates VNet/Firewall IP denial.Query aborted: resource class concurrency Queueing: Small Resource Classes (smallrc) limit a single query to ~3% of node memory, spilling complex hash joins to tempdb and running 10x slower. Assign ETL transforms to largerc or Workload Groups with explicit MIN_PERCENTAGE_RESOURCE and CAP_PERCENTAGE_RESOURCE.copyActivity concurrency (MaxConcurrentConnections), tuning cloudDataMovementUnits, and ensuring outbound port 443 / TLS 1.2 compliance.Review the production code, company interview Q&As, incident traces, and comprehension quiz inside this lesson.
// Log Analytics: Kusto Query to identify failed ADF activities in last 6 hours
ADFPipelineRun
| where TimeGenerated >= ago(6h)
| where Status == "Failed"
| project TimeGenerated, PipelineName, RunId, Parameters
| join kind=inner (
ADFActivityRun
| where Status == "Failed"
| project ActivityName, ActivityType, ErrorMessage=parse_json(Output).errors[0].message, RunId
) on RunId
| order by TimeGenerated desc;ErrorCode=UserErrorFailedFileOperation, Message=The file 'raw/transactions_2026.csv.gz' is corrupted or invalid. Unexpected end of stream encountered.