Week 8 • Day 4Week 8

Day 4:Week 8, Day 4: Statistical Significance, Confidence Intervals & CI/CD Regression Gates

Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz

Week 8, Day 4: Statistical Significance, Confidence Intervals & CI/CD Regression Gates


1. Is a +2% Eval Gain Real or Just Random Noise?

Suppose Prompt A passes 40 / 50 test cases ($80%) and Prompt B passes `41 / 50` test cases (\82%$). Is Prompt B actually better, or is 1 test case within random noise?

Standard Error & 95% Confidence Interval of a Pass Rate

Treat each test case outcome as a Bernoulli random variable with sample pass rate p^=kn\hat{p} = \frac{k}{n}. The Standard Error (SE) of the mean over nn evaluation cases is:

SE(p^)=p^(1−p^)n\text{SE}(\hat{p}) = \sqrt{\frac{\hat{p}(1 - \hat{p})}{n}}

And the 95% Confidence Interval is:

CI95%=p^±1.96⋅p^(1−p^)n\text{CI}_{95\%} = \hat{p} \pm 1.96 \cdot \sqrt{\frac{\hat{p}(1 - \hat{p})}{n}}

Example Calculation:

For n=50n = 50 and p^=0.80\hat{p} = 0.80: SE=0.80×0.2050=0.0032≈0.0566  ⟹  ±1.96×0.0566=±11.1%!\text{SE} = \sqrt{\frac{0.80 \times 0.20}{50}} = \sqrt{0.0032} \approx 0.0566 \implies \pm 1.96 \times 0.0566 = \mathbf{\pm 11.1\%!} With only 50 independent binary questions, a 95% confidence interval is [68.9%, 91.1%]! However, when we use paired comparison on the exact same Golden Dataset at temperature=0.0 and track continuous metrics (Faithfulness, Fact Recall, Safety Gate), we can enforce strict Zero-Regression CI/CD Gates!


2. Automated CI/CD Regression Gates (.github/workflows/evals.yml)

Every Pull Request that touches prompts/, src/, or models/ triggers an automated CI pipeline:

  1. Run the candidate pipeline on the Golden Dataset.
  2. Compare metrics against baseline_metrics.json.
  3. Hard Gate (sys.exit(1) if violated):
    • safety_pass_rate < 1.00 (Zero tolerance for unsafe commands!)
    • faithfulness_mean < baseline.faithfulness_mean - max_allowed_drop
    • overall_pass_rate < threshold

In Simple English: Why Testing Only 5 Questions Lies to You

Suppose you flip a coin 4 times and get 3 heads (75%). Is the coin rigged? No — with tiny sample sizes (nn), luck causes wild swings! Now look at the 95% Confidence Interval numbers from our formula:

  • If you test 50 questions (n=50n = 50) and pass 40 (p^=80%\hat{p} = 80\%): your true accuracy could reasonably be anywhere from 69% to 91% (±11%\pm 11\% uncertainty!).
  • If you expand your Golden Dataset to 400 questions (n=400n = 400) and pass 80%: uncertainty shrinks by 8≈2.8×\sqrt{8} \approx 2.8\times down to ±3.9%\pm 3.9\% (true score is safely between 76% and 84%) — solid enough to block bad deployments automatically in GitHub Actions!