Week 17 • Day 5Week 17

Day 5:Asynchronous Background Tasks & Webhook Event Queues

Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz

Day 5: Asynchronous Background Tasks & Webhook Event Queues

1. First-Principles Problem: Protecting the HTTP Critical Response Path

In high-performance AI services, only operations required to generate the user's tokens belong in the synchronous HTTP request path.

Consider an incident copilot endpoint that executes:

  1. Retrieval + LLM Generation: $1,200\text{ms}$
  2. Audit DB Logging: $45\text{ms}$
  3. Slack Webhook Notification: $350\text{ms}$
  4. Asynchronous LLM-as-a-Judge Eval: $2,100\text{ms}$
  5. Vector Cache Refresh: $400\text{ms}$

Latency Comparison:

Synchronous Response Time=1200+45+350+2100+400=4,095ms\text{Synchronous Response Time} = 1200 + 45 + 350 + 2100 + 400 = 4,095\text{ms} Decoupled Background Time=1,200ms(70.7% lower user-perceived latency!)\text{Decoupled Background Time} = 1,200\text{ms} \quad \mathbf{(70.7\% \text{ lower user-perceived latency!})}

Furthermore, if the external Slack API experiences a network timeout, a synchronous endpoint crashes with HTTP 500, failing the entire user query even though the AI response was computed successfully!


2. Decoupled Event Architecture & Dead Letter Queues (DLQ)

Architecture Flow Diagram
Client HTTP Request->API[FastAPI Endpoint
|1. Immediate Stream / Return (1200ms)| Client
|2. Enqueue Background Task| Q[In-Memory / Redis Queue
W[Asynchronous Worker
DB[Audit DB Write
WH[Webhook Dispatcher (HMAC Signed)
|Permanent Failure| DLQ[Dead Letter Queue (DLQ)
  1. Fire-and-Forget / Non-Blocking Queue: The main event loop delegates post-processing tasks immediately after response transmission.
  2. Cryptographic Webhook Signatures: Outgoing webhook payloads are signed via HMAC-SHA256 in the X-Copilot-Signature header, allowing subscribers to verify payload integrity and authenticity.
  3. Dead Letter Queue (DLQ): Failed webhook deliveries are retried up to NN times with exponential backoff. If all retries fail, payloads are archived in a DLQ for manual inspection without crashing the application.

In Simple English: Don't Make the User Wait for Slow Email / Analytics Work

Suppose after answering a chat question (1,200 ms), your server also writes analytics to Snowflake (350 ms), sends a Slack webhook (2,100 ms), and emails an audit receipt (400 ms):

  • Doing Everything Inline: User waits 4,095 ms (over 4 seconds!) just so Slack can finish!
  • Background Queue (Decoupled): Return the AI answer immediately in 1,200 ms and drop the Slack/email jobs onto a fast background queue (with a Dead Letter Queue DLQ catching any failed webhooks for safe retry later)!