Building a program from the ground up

• Experimentation at Yelp: Harvard Business School case on the work my team and I did to build the Yelp experimentation program
• Building a Culture of Experimentation by Stefan Thomke
• The Yelp Story: Building an Experimentation Program From the Ground Up by Rohan Katyal

Terminology

• Power analysis: [Primer](https://machinelearningmastery.com/statistical-power-and-power-analysis-in-python/#:~:text=Statistical power is the probability,effect size%2C and statistical power.), Calculator
• Multiple comparison: Primer, Bonferroni correction
• Sample ratio mismatch: Primer & Diagnosis
• Confidence intervals
• Simpson's Paradox
• Hypothesis

Avoiding the common gotchas

• Ron Kohavi's talk on pitfalls in online controlled experiments
• Good Experiment, Bad Experiment by Fareed Mosavat
• Avoid the Pitfalls of A/B testing
• Diagnosing Sample Ratio Mismatch

Writing a great hypothesis

• A strong hypothesis

Prioritization

• Darius Contractor's experimentation prioritization framework called EVELYN