Chris Van Patten

#wcpub 2019

  1. Experimentation is scary
    1. Often vague (product managers hate it)
    2. Chance of failure (executives hate it)
  2. Experimentation is valuable
    1. Keeps "creative" talent engaged
    2. Can lead to valuable breakthroughs
  3. Experimentation is just small scale innovation, a word that executives love
  4. Reason it's so scary: often done ad-hoc, without structure
  5. Solution: create framework for experimentation
  6. Framework:
    1. Scope your experiment
      1. Should not be too specific/tactical
      2. Focus on outcomes/"desired future state"
      3. Time constraints are essential
    2. Prepare for failure (try to avoid the word though)
      1. What is your rollback strategy?
      2. Essential piece of the puzzle; have a plan for reverting your experiment before you start the experiment
    3. Outline the risk
      1. What would happen if the experiment does not meet its goals?
    4. Tag the stakeholders
      1. Notify anyone who might be impacted by the experiment
      2. This should be more of a 'heads up', not a formal Request For Comment
    5. SPOT
  7. Answer those questions in a document
    1. Keep it simple — 2–3 sentences for each answer
    2. Should be somewhat informal; this is not a technical design document or an executive briefing
  8. I'm not going to talk too much about the process of writing/building/executing the experiment
  9. Schedule a 20 minute meeting after the end of the experiment, before the experiment starts
    1. Keeps you honest to your time constraints
    2. Ensures you review results
      1. How else do you determine if an experiment met its goals?
      2. Record the results with the original experiment proposal
    3. Opportunity to discuss next steps
      1. "Can we formalize the experiment?"
      2. "Do we need to revert the experiment?"
  10. Finally, make sure your process is documented and encouraged
    1. Encourage your team to come up with experiments
    2. Try to run experiments regularly. Best way to normalize the process of experimentation
    3. Keep experiments in a central place — a corner of your internal wiki, a Google Docs folder, GitHub repo, etc. — and encourage your team to review past experiments.