Delta Lake Internals: _delta_log JSON Transactions, ACID & Time Travel

[!NOTE] Delta Lake is an open-source storage layer that brings ACID transactions to Apache Spark and big data workloads. Delta tables are physically stored as standard Apache Parquet files paired with a forward-only transactional commit log in the _delta_log/ directory.

1. Inside the _delta_log Commit Protocol

Every write, update, delete, or merge to a Delta table creates a new commit file:

• Checkpoints: Every 10 commits, Delta Lake consolidates the historical commit JSONs into an optimized checkpoint Parquet file. When a reader opens a table at version 50, it reads checkpoint 40 plus JSONs 41-50, avoiding reading 50 individual JSON files.

2. Optimistic Concurrency Control (OCC)

Delta Lake uses OCC to guarantee serializable isolation:

1. Record the Starting Version: Job reads table state at version $V$.
2. Execute Write in Staging: Job writes new Parquet data files to storage.
3. Attempt Commit: Job tries to write $V+1$.json. If another job committed $V+1$.json first, Delta checks if the two jobs mutated the same files or partitions.
   • If partitions are disjoint: Delta automatically fast-forwards and commits $V+2$.json.
   • If partitions overlap: Delta throws ConcurrentModificationException and aborts.

Staff Architecture: Delta Lake _delta_log, Deletion Vectors & Time Travel Boundaries

• Optimistic Concurrency & _delta_log Checkpoints: Every Delta commit writes an immutable atomic 00000N.json log file and builds a Parquet checkpoint every 10–100 commits so readers never replay 100,000 tiny JSON files. Concurrent UPDATE/MERGE jobs use Row-Level Concurrency (with Deletion Vectors enabled) to avoid false ConcurrentAppendException aborts when touching disjoint rows in the same partition.
• Deletion Vectors (Merge-on-Read Speedup): Enabling delta.enableDeletionVectors = true marks updated/deleted row positions inside lightweight Roaring Bitmap .bin files instead of rewriting entire 1 GB Parquet files during MERGE/DELETE, speeding up GDPR & CDC upserts by 5x–10x.
• VACUUM Safety Window: Never run VACUUM table RETAIN 0 HOURS while active streaming readers or long queries run—keep the default 168 hours (7 days) safety floor so DESCRIBE HISTORY / Time Travel remains deterministic and uncorrupted.