Day 4:Week 8, Day 4: Statistical Significance, Confidence Intervals & CI/CD Regression Gates
Battle-tested daily lesson note • 100% runnable code, interview Q&A, test suite & interactive quiz
Week 8, Day 4: Statistical Significance, Confidence Intervals & CI/CD Regression Gates
1. Is a +2% Eval Gain Real or Just Random Noise?
Suppose Prompt A passes 40 / 50 test cases ($80%) and Prompt B passes `41 / 50` test cases (\82%$). Is Prompt B actually better, or is 1 test case within random noise?
Standard Error & 95% Confidence Interval of a Pass Rate
Treat each test case outcome as a Bernoulli random variable with sample pass rate .
The Standard Error (SE) of the mean over evaluation cases is:
And the 95% Confidence Interval is:
Example Calculation:
For and :
With only 50 independent binary questions, a 95% confidence interval is [68.9%, 91.1%]!
However, when we use paired comparison on the exact same Golden Dataset at temperature=0.0 and track continuous metrics (Faithfulness, Fact Recall, Safety Gate), we can enforce strict Zero-Regression CI/CD Gates!
2. Automated CI/CD Regression Gates (.github/workflows/evals.yml)
Every Pull Request that touches prompts/, src/, or models/ triggers an automated CI pipeline:
- Run the candidate pipeline on the Golden Dataset.
- Compare metrics against
baseline_metrics.json. - Hard Gate (
sys.exit(1)if violated):safety_pass_rate < 1.00(Zero tolerance for unsafe commands!)faithfulness_mean < baseline.faithfulness_mean - max_allowed_dropoverall_pass_rate < threshold
In Simple English: Why Testing Only 5 Questions Lies to You
Suppose you flip a coin 4 times and get 3 heads (75%). Is the coin rigged? No — with tiny sample sizes (), luck causes wild swings! Now look at the 95% Confidence Interval numbers from our formula:
- If you test 50 questions () and pass 40 (): your true accuracy could reasonably be anywhere from 69% to 91% ( uncertainty!).
- If you expand your Golden Dataset to 400 questions () and pass 80%: uncertainty shrinks by down to (true score is safely between 76% and 84%) — solid enough to block bad deployments automatically in GitHub Actions!