A/B Testing
A controlled experiment that compares two versions of a product element (A = control, B = variant) to determine which performs better on a defined metric.
What is A/B Testing?
A/B testing (also called split testing) is a randomised controlled experiment that shows two versions of a product element to different user segments simultaneously to determine which version drives better outcomes.
A/B test anatomy
| Element | Description |
|---|---|
| Control (A) | The existing version |
| Variant (B) | The new version being tested |
| Hypothesis | "Changing X will increase Y because Z" |
| Primary metric | The one metric that determines the winner |
| Guard metrics | Metrics you must not harm (e.g. revenue) |
| MDE | Minimum detectable effect — smallest improvement worth detecting |
| Sample size | Calculated from MDE, baseline conversion, and significance level |
Statistical significance
A result is significant at 95% confidence when there's only a 5% chance the observed difference occurred by random chance. Don't stop tests early — running until significance is reached after peeking inflates false positive rates.
Common mistakes
- Too many variants — multivariate tests require much larger samples
- Stopping early — peeking at results and stopping when significant inflates false positives
- Wrong primary metric — optimising for clicks while the real goal is revenue
- No holdout — running a test without a control group
Free templates for A/B
Frequently asked questions
How long should an A/B test run?
Until you've reached the required sample size calculated upfront — not until you see a significant result. For most SaaS products, this is 1–4 weeks. Never stop a test early because it looks like B is winning.
What's the difference between A/B testing and multivariate testing?
A/B testing compares two versions of one element. Multivariate testing compares multiple variations of multiple elements simultaneously. MVT requires much larger sample sizes and is harder to interpret — use it only when you have very high traffic and are optimising multiple elements at once.
Apply A/B Testing to your real product data
PMRead ingests customer feedback, interviews, and Slack threads — and generates PRDs grounded in real evidence.
Related terms
North Star Metric
A single metric that best captures the core value your product delivers to customers — and that the whole company optimises toward.
Conversion Rate
The percentage of users who complete a desired action — signup, upgrade, purchase, or feature activation — out of all users who had the opportunity to do so.