Your goal: design a clean test, size it, read the results honestly, and make a launch call
you can defend.
How this works (about 30 minutes)
- One question, gone deep: you're a PM at Plaid, helping a streaming service decide on bank
transfers at checkout. Ben has a fact sheet and, later, a results sheet, both made up for
the exercise. Ask for what you need; he won't volunteer.
- A shared notepad is there if you want to sketch your design or your math. Ask Ben to open it,
and tell him when you've written something.
- Lots of follow-ups and one curveball, then a trade-off turn on a second result: one number
rises while another falls. Ship, hold or change?
- No hints and no scores during the call. Your report comes after.
What scores well
- A hypothesis and a unit of randomization with a reason
- One primary metric, guardrails, and a window long enough for the outcome to show
- Size and duration worked out with numbers, in whole weeks
- Reading an interval, not just "significant", and what the headline hides
- A launch call with a ramp, a holdback and what would change your mind
- Your plan said out loud, and answers that land on a decision
Tips
- Take PM Coach · A/B Tests first.
- Say what decision the test will make before you say how to run it.