Loading Academy Hub...
Master AWS metadata governance: AWS Glue Data Catalog, Crawlers vs Partition Projection in Athena, Glue DynamicFrames vs standard PySpark DataFrames, and schema evolution handling.
[!NOTE] The AWS Glue Data Catalog provides a unified Apache Hive Metastore-compatible metadata repository across all AWS analytical services (Athena, EMR, Redshift Spectrum, SageMaker).
MSCK REPAIR TABLE or crawler execution.resolveChoice(specs=[('id', 'cast:long')]) cleanly resolves mixed data types without dropping rows.GetPartitions RPC overhead during Athena and Spark planning. For predictable schemas (e.g., year/month/day/hour or fixed region enums), use Athena Partition Projection (projection.enabled=true) or migrate tables to Apache Iceberg / Delta Lake so metadata pruning lives inside manifest files rather than thousands of Hive metastore calls.table_12345) on minor schema drift or CSV header differences. Instead, register explicit schemas via Terraform/IaC and enforce contract checks in ETL.DataCellsFilter) instead of brittle IAM bucket prefix policies when sharing curated datasets across dozens of downstream consumer accounts.Review the production code, company interview Q&As, incident traces, and comprehension quiz inside this lesson.
import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
# Read messy JSON from S3 into a Glue DynamicFrame
dynamic_frame = glueContext.create_dynamic_frame.from_options(
connection_type="s3",
connection_options={"paths": ["s3://lakehouse/landing/raw_orders/"]},
format="json"
)
# Resolve schema drift where order_id arrived as both string and integer
resolved_df = dynamic_frame.resolveChoice(specs=[('order_id', 'cast:long')])
# Convert to Spark DataFrame for complex SQL transformation
spark_df = resolved_df.toDF()
cleaned_df = spark_df.filter("amount > 0").dropDuplicates(["order_id"])
# Write back to S3 partitioned by year and month in Snappy Parquet
cleaned_df.write \
.partitionBy("year", "month") \
.format("parquet") \
.option("compression", "snappy") \
.save("s3://lakehouse/silver/orders/")
job.commit()Query failed: HIVE_EXCEEDED_SPLIT_LIMIT. Query exceeded maximum split limit of 1,200,000 splits. Please refine your query predicate.