1. When should I rely on Spark AQE instead of manual broadcast or repartition hints?
Enable Adaptive Query Execution (spark.sql.adaptive.enabled=true) as your baseline so Spark dynamically switches SortMergeJoin to BroadcastHashJoin and splits skewed partitions at runtime using actual shuffle bytes. Keep explicit broadcast(small_df) hints or two-phase salting only when upstream statistics are missing or a single hot key (such as NULL or a default merchant ID) dominates more than 70% of rows.