*originally published February 1, 2020*

Here’s a simple tool that you can use to test whether the results of your A/B Tests are statistically significant. Happy growth hacking!

### Instructions

Plug in your two variations sample sizes (`n1`

and `n2`

) and estimated success rates (`p1`

and `p2`

), and scroll down to Interpreting The Results, to understand the results of your test.

The Hypothesis Specification explains how to formulate your A/B Test experiment.

https://codepen.io/stedmanblake/pen/ZEjapYB

Sample distributions. Red: Variation A. Green: Variation B. The Null Hypothesis is that the true, population average of the distribution that generated the red sample is *higher than* the true, population average of the distribution that generated the green sample. The Alternative Hypothesis is that green’s underlying distribution has a higher average.

### The Magic of Statistics

In statistics vernacular, we’re doing a test of “difference in proportions”, or a “two-proportion z-test”.

The data that we’re considering is analogous to a repeated coin toss. You flip the coin, and it either comes up heads or tails. Then you do it again, and again, …

The distribution that this sort of data follows is called a “Binomial Distribution”. It’s characterized by two parameters: sample size (denoted by the variable *n*, for *number* of coin flips), and probability of success on any given “coin flip” (denoted by the variable *p*, for *probability* of success).

Many business applications with a discrete outcome follow a Binomial Distribution:

**ad click-through** (n = number of ad impressions, p = probability of click-through),
**email open** (n = number of emails sent, p = probability of an email being opened),
**website sign-up** (n = number of website visitors, p = probability of a visitor signing up),
**checkout conversion** (n = number of users who go to the checkout page, p = probability of successful checkout)

As such, we often gather this data in the course of growing our businesses: optimizing our ads, websites, and funnels for conversion.