# Types of statistical Errors

Type I and type II errors

- Error that is rejecting null hypothesis
- Rolling out A/B tests that is performing worse than the baseline

- Error that is confirming wrong null hypothesis
- Rolling back A/B tests that were significantly better than the baseline

# A/B Testing

- Some see this version (A)
- Others see this version (B)

## Measurements of...

- Samples (Users, Sessions, Impressions)
- Conversions (Number of clicks or goal competition

## Why do we need testing instead of direct comparison

- If number of sample is not enough, the result isn't statistically significant enough
- Flipping a coin 10 times may lead to 6 heads and 4 tails, or maybe 7 tails and 3 heads.
- We need better decision making framework

## Bayesian A/B Testing

Bayesian A/B Testing Calculator

- Use bayesian theory
- Why?
- To test problems when there's relatively low amount of data

- What is different?
- We set the prior probability distribution (i.e, uniform distribution)

# Null hypothesis significance testing

- Validate the following hypothesis
- two or more groups have basically same distribution (ex: uniform distribution for example)