Week 13 • Day 5Week 13
Day 5:Week 13, Day 5: Durable Checkpointing, Time-Travel Replay & Human-in-the-Loop (`HITL`) Interrupts
Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz
Week 13, Day 5: Durable Checkpointing, Time-Travel Replay & Human-in-the-Loop (HITL) Interrupts
1. Why Agents Cannot Block a Thread Waiting for a Human
Suppose your SRE Agent diagnoses a production outage and proposes:
"Promote standby replica
pg-node-02to primary usingpg_ctl promote."
Because promoting a database replica is a high-blast-radius destructive action, we require Human-in-the-Loop (HITL) Approval from the on-call engineer on Slack/PagerDuty.
What if the engineer takes 20 minutes to click "Approve" (or what if your Python pod restarts in the meantime)?
You cannot hold an HTTP request or time.sleep() open in RAM for 20 minutes!
2. Durable SQLite Checkpointing (thread_id, checkpoint_id) & Interrupt/Resume
- Every Graph Super-Step Writes a Durable Checkpoint to SQLite:
Keyed by
(thread_id, step_index), storingcurrent_node,next_node, andserialized_state_json. - Pre-Execution
interrupt_beforeGate: When the graph reaches a node listed ininterrupt_before = {"execute_remediation"}:- The engine saves the checkpoint with
status = "INTERRUPTED_FOR_HITL"and exits cleanly (0CPU/RAM used while waiting!). - When the human clicks Approve, Edit (modifying the command parameters), or Reject 20 minutes later, the API loads the latest checkpoint for
thread_id, applies the human's decision, and resumes execution fromexecute_remediation!
- The engine saves the checkpoint with
- Time-Travel Debugging (
fork_from_step): Because every historical step0, 1, 2, ...is persisted in SQLite, an engineer can loadstep_index = 2from a failed production run, tweak a state variable, and replay forward from Step 2!
In Simple English: Game Save Points (Checkpoints) & Calling a Human Before Destructive Actions (HITL)
- Durable Checkpointing: Imagine playing a 5-hour RPG game where any server crash restarts you from Level 1! Checkpointing saves your graph state to SQLite/Postgres after every single step. If the server restarts at Step 4, it resumes from Step 4 in milliseconds!
- Time-Travel Replay: Engineers can rewind to Step 2, tweak a bad tool result, and replay from there to debug agent bugs.
- Human-in-the-Loop (
HITLInterrupt): Before running anything dangerous (deleting data, sending money, restarting production servers), the graph hits Pause, asks a human "Approvekubectl delete pod? [Yes/No]", and sleeps using $0 compute until clicked!