MBA DM Phase 2: The Agency Simulation
Session 15
A/B Test Design for Ads & Landing Pages
Testing, not guessing.
Today's Agenda
- Why single-variable testing matters
- Test design: hypothesis, variable, success metric
- Statistical significance basics (sample size intuition)
- Deliverable: A/B test plan (A2 input)
The One-Variable Rule
Test ONE variable at a time (headline OR image, not both) — otherwise you can't attribute the result to the right change.
Test Design Template
| Element | Example |
|---|
| Hypothesis | "Version B headline increases CTR" |
| Variable | Headline only |
| Success metric | CTR ≥ 15% lift |
| Duration/sample | Run until ≥100 clicks per variant |
Lab: Design Your A/B Test Plan
- Pick one ad or landing page element to test
- Write your hypothesis and success metric
- Define minimum sample size / run duration
Deliverable for Session 15
A/B Test Plan
Third component of your A2 package.