Statistical significance
The threshold at which an A/B test result is unlikely to be due to chance — typically 95% confidence.
Statistical significance is the threshold at which an A/B test result is unlikely to be due to random chance, traditionally set at p<0.05 (95% confidence). Below the threshold, you can't reliably attribute the observed difference between variants to the variable you changed — it could just be sampling noise.
For email A/B tests, sample sizes matter more than most marketers realize. To detect a 1 percentage-point lift in open rate (say, 25% vs. 26%) with statistical significance, you typically need 5,000+ recipients per variant. Smaller tests can still be useful for directional decisions, but declaring a "winner" on 200 recipients per variant is statistical theater.
Tools like Optimizely, ABTestguide, or built-in ESP A/B calculators handle the math. The practical lesson: design tests so the changes are large enough to detect at your actual list size. Testing tiny variations on small lists wastes sends; reserve A/B tests for changes meaningful enough to matter.