A/B Testing Math: When to Call a Winner and When to Keep Running

Kavya Mehra 3 min readJuly 3, 2024

Pre-commit to a sample size

Calculate required sample size before the test starts. Don't peek and stop early. Stopping when you like the result inflates false positives.

Regardless of sample size, run at least 2 full weeks. Captures weekday/weekend seasonality. 1 week is too noisy.

Standard industry threshold. Higher confidence (99%) needed for high-traffic changes where variance in wins/losses at 95% compounds.

Before any A/B test, we run qualitative research. Here's the four-tool stack that generates experimentable hypotheses.

Most CRO programs start with tests. Ours starts with research. Here's the 2-week research sprint we run first.

Three mobile CRO tests that reliably produce lift across industries.

Our team ships cro programs every week. Book a free consult — we'll tell you what would move the needle for your brand.