Session 15: A/B Test Design for Ads & Landing Pages

Block 2: Content Scaling, AI Workflows & A2 Part 1

View lecture slides: Session 15 Slides

1-Hour Lecture (Theory & Strategy)

  • A real A/B test plan needs: one variable changed at a time, a hypothesis, a success metric, a minimum sample/duration, and a decision rule (“if variant B beats A by X%, we roll it out”).
  • Common student mistake: testing 3 things at once (headline + image + CTA) and being unable to attribute the win to anything.
  • Read the full theory here: A/B Testing & Landing Page Optimization.
  • Statistical common sense (no heavy stats required): small sample sizes produce noisy results — state the caveat rather than over-claiming a “winner” from 20 clicks.

1-Hour Lab (Agency Execution)

  1. Pick ONE variable from your Session 14 ad set to test (headline, image, or CTA — not all three).
  2. Write a one-sentence hypothesis: “Changing X to Y will increase [metric] because [reason].”
  3. Define your success metric (CTR, form-fill rate, cost-per-lead) and a realistic minimum run duration/spend given a student budget.
  4. Write your decision rule: what result would make you roll out the winner vs. call it inconclusive?
  5. Do the same for one landing-page element (headline, hero image, or form length) on your Phase 1 sandbox site.

Deliverable: Feeds Assignment A2 (Part 3 of 4)

A documented, feasible A/B test plan — the “A/B Test Plan” rubric row of A2.