Interactive Prompt Engineering Lab

Test and score your prompts against real-world enterprise scenarios. Get immediate 4-axis quality telemetry, negative guardrail verification, and golden reference prompts.

Read Syllabus

1. Authoritative Persona

Expert role + zero fluff

25 pts

2. Deterministic Schema

XML tags + JSON / keys

25 pts

3. Negative Guardrails

NEVER / fallback rules

25 pts

4. Token Efficiency

150–1,800 chars concise

25 pts

Data Engineering & RAGStaff

Apache Spark DataFrame PySpark Code Generator with Schema Guard

Generate an optimized, production-grade PySpark ETL pipeline that ingests raw telemetry logs, parses JSON strings, dedupes events by device_id and timestamp, and writes partitioned Parquet files.

Target Input Context / Untrusted Data:

Source: s3://raw-telemetry/events/*.json (500GB daily, billions of records)
Fields: device_id (string), event_time (iso timestamp), payload (raw json string with battery_level, cpu_temp)
Target: s3://curated-telemetry/partitioned_by_date/
Quality: Filter out corrupt records, dedupe by device_id keeping latest event_time.

Quality Requirements & Guardrails:

Define a Principal Data Engineer (Apache Spark & PySpark) persona.

Define an explicit StructType schema instead of using runtime schema inference.

Use window functions (row_number().over(Window.partitionBy(...))) for distributed deduplication.

Write data in Parquet format partitioned by year/month/day with snappy compression.

Your System Prompt Draft

81 chars~20 tokens

Golden Reference Architecture (100% Quality Benchmark)

<system_instructions>
You are a Principal Big Data Architect specializing in Apache Spark 3.5 & PySpark optimization.
Write an enterprise-grade PySpark ETL pipeline for high-throughput telemetry deduplication.

PERFORMANCE & ARCHITECTURAL RULES:
1. NO SCHEMA INFERENCE: Explicitly declare the DataFrame schema using `StructType` and `StructField`. Runtime schema inference on 500GB crashes driver nodes.
2. WINDOW DEDUPLICATION: Use `Window.partitionBy("device_id").orderBy(col("event_time").desc())` with `row_number()` to filter duplicates at scale without expensive groupBy shuffles.
3. ADAPTIVE QUERY EXECUTION: Enable `spark.sql.adaptive.enabled = true` and tune shuffle partitions to prevent data skew.
4. EFFICIENT WRITES: Write to S3 using `.format("parquet").partitionBy("date").option("compression", "snappy")`.
5. Output clean, documented Python code inside <pyspark_pipeline> tags.
</system_instructions>