# Types of statistical Errors

Type I and type II errors

• Error that is rejecting null hypothesis
• Rolling out A/B tests that is performing worse than the baseline
• Error that is confirming wrong null hypothesis
• Rolling back A/B tests that were significantly better than the baseline

# A/B Testing

• Some see this version (A)
• Others see this version (B)

## Measurements of...

• Samples (Users, Sessions, Impressions)
• Conversions (Number of clicks or goal competition

## Why do we need testing instead of direct comparison

• If number of sample is not enough, the result isn't statistically significant enough
• Flipping a coin 10 times may lead to 6 heads and 4 tails, or maybe 7 tails and 3 heads.
• We need better decision making framework

## Bayesian A/B Testing

Bayesian A/B Testing Calculator

• Use bayesian theory
• Why?
• To test problems when there's relatively low amount of data
• What is different?
• We set the prior probability distribution (i.e, uniform distribution)

# Null hypothesis significance testing

• Validate the following hypothesis
• two or more groups have basically same distribution (ex: uniform distribution for example)