Chain-of-Thought (CoT) & Scratchpad Reasoning
When and how to instruct models to think step by step, using <thinking> tags and reasoning model architectures.
Core Conceptual Brief
Chain-of-Thought prompting drastically boosts accuracy in math, algorithmic design, and complex data verification. Master the use of scratchpads and understand modern reasoning models (o1, o3-mini, DeepSeek-R1).
Theoretical & Empirical Architecture
Standard autoregressive decoding generates one token at a time without backtracking. If an LLM begins generating a complex answer without intermediate scratchpad steps, it often commits to an early mistake because every future token is conditioned on earlier generated tokens.
Chain-of-Thought (CoT) forces the model to emit its intermediate reasoning path before producing the final answer:
Modern frontier reasoning models (OpenAI o1/o3-mini, DeepSeek-R1) integrate CoT directly into their inference pipeline via Reinforcement Learning on reasoning trajectories.
Under the Hood: Key Engineering Mechanics
1The Dual-Tag Pattern (<thinking> and <answer>)
Instruct the model to conduct its reasoning inside <thinking> tags, followed by its final response in <answer>. This allows client applications to stream or hide the thinking process from end users while retaining 100% of the cognitive benefits.
2Self-Correction and Edge Case Verification
Prompt the scratchpad to specifically verify assumptions: "In your thinking process, verify: (1) Does the code handle empty arrays? (2) Are there off-by-one errors in loop boundaries? (3) What is the worst-case time complexity?"
3When NOT to use Chain-of-Thought
CoT multiplies the output token count by 3x to 10x, directly increasing API latency and billing. Never use CoT for simple tasks (sentiment classification, short translations, or structured field extraction) where zero-shot or few-shot is already 99%+ accurate.
Production Chain-of-Thought Prompt for Algorithmic Verification
Analyze the following PySpark job for skew and memory pressure.
You MUST structure your response into two distinct sections:
<thinking>
In this section, systematically work through:
1. Identify all groupBy, join, and distinct operations.
2. Check the join key cardinality: is the key highly skewed (e.g. nulls, default values)?
3. Calculate shuffle partition estimates assuming 200MB default partition target.
4. Formulate the mitigation: Salting vs Broadcast Join vs Adaptive Query Execution (AQE).
5. Verify if broadcast join is safe (< 10MB default threshold).
</thinking>
<recommendation>
Output the final architecture review with:
- Identified Bottlenecks (with line references)
- Drop-in PySpark optimization code
- Expected memory and execution time impact
</recommendation>Production Guardrails & Takeaways
- CoT trades latency and token cost for algorithmic accuracy. Use it where logical mistakes are expensive.
- Enclosing scratchpad reasoning in <thinking> tags allows frontend applications to toggle reasoning visibility.
- Reasoning models (o1, o3, R1) perform CoT natively during inference; traditional chat models require explicit scratchpad prompting.
Real Staff AI Systems Interview Question
Q: Why does Chain-of-Thought prompting improve an LLM's performance on multi-step reasoning problems?
In a Transformer, each generated token can attend to all previously generated tokens. When a model outputs intermediate reasoning steps, it breaks a complex non-linear calculation into smaller linear steps. The attention mechanism can refer back to the exact intermediate values calculated in earlier tokens rather than trying to map the entire problem to the final answer in one step.