A/B Test Design Framework

Help me design a rigorous A/B test.

**What I want to test:**
[The change you want to make — be specific about what's different between control and variant]

**Hypothesis:**
[Your prediction: "If we [change], then [metric] will [increase/decrease] because [reason]"]

**Context:**
- Product or page: [Where does this test run?]
- Current baseline metric: [The metric you're trying to move and its current value — e.g. conversion rate is 3.2%]
- Minimum improvement that matters: [What's the smallest change worth shipping? e.g. +0.5% conversion]
- Weekly traffic or users: [How many people flow through this area?]

**Constraints:**
- Maximum test duration: [How long can you run the test? e.g. 4 weeks]
- Anything that could confound results: [Seasonality, concurrent tests, known data issues]

Design the test:
1. **Refined hypothesis** — restate it in the format: "We believe [change] will cause [metric] to [direction] for [audience] because [mechanism]"
2. **Sample size calculation** — the minimum users needed per variant for 80% power at 95% confidence, given your baseline and MDE
3. **Test duration** — based on traffic, how long to run it
4. **Success and guardrail metrics** — primary metric to move, secondary metrics to monitor, guardrails not to break
5. **Segmentation plan** — any user segments to analyze separately
6. **Analysis plan** — how to read results when the test ends, including what to do if results are inconclusive
7. **Common pitfalls** — novelty effect, peeking, multiple comparisons — which apply here and how to handle them