Chris Van Patten
#wcpub 2019
- Experimentation is scary
- Often vague (product managers hate it)
- Chance of failure (executives hate it)
- Experimentation is valuable
- Keeps "creative" talent engaged
- Can lead to valuable breakthroughs
- Experimentation is just small scale innovation, a word that executives love
- Reason it's so scary: often done ad-hoc, without structure
- Solution: create framework for experimentation
- Framework:
- Scope your experiment
- Should not be too specific/tactical
- Focus on outcomes/"desired future state"
- Time constraints are essential
- Prepare for failure (try to avoid the word though)
- What is your rollback strategy?
- Essential piece of the puzzle; have a plan for reverting your experiment before you start the experiment
- Outline the risk
- What would happen if the experiment does not meet its goals?
- Tag the stakeholders
- Notify anyone who might be impacted by the experiment
- This should be more of a 'heads up', not a formal Request For Comment
- SPOT
- Answer those questions in a document
- Keep it simple — 2–3 sentences for each answer
- Should be somewhat informal; this is not a technical design document or an executive briefing
- I'm not going to talk too much about the process of writing/building/executing the experiment
- Schedule a 20 minute meeting after the end of the experiment, before the experiment starts
- Keeps you honest to your time constraints
- Ensures you review results
- How else do you determine if an experiment met its goals?
- Record the results with the original experiment proposal
- Opportunity to discuss next steps
- "Can we formalize the experiment?"
- "Do we need to revert the experiment?"
- Finally, make sure your process is documented and encouraged
- Encourage your team to come up with experiments
- Try to run experiments regularly. Best way to normalize the process of experimentation
- Keep experiments in a central place — a corner of your internal wiki, a Google Docs folder, GitHub repo, etc. — and encourage your team to review past experiments.