Session 15: A/B Test Design for Ads & Landing Pages

--:-- --
↓ Scroll for more
MBA DM Phase 2: The Agency Simulation

Session 15

A/B Test Design for Ads & Landing Pages

Testing, not guessing.

Today's Agenda

  • Why single-variable testing matters
  • Test design: hypothesis, variable, success metric
  • Statistical significance basics (sample size intuition)
  • Deliverable: A/B test plan (A2 input)

The One-Variable Rule

Test ONE variable at a time (headline OR image, not both) — otherwise you can't attribute the result to the right change.

Test Design Template

ElementExample
Hypothesis"Version B headline increases CTR"
VariableHeadline only
Success metricCTR ≥ 15% lift
Duration/sampleRun until ≥100 clicks per variant

Worked Example: Pathshala Tutors

Landing-page test (lead rate): baseline 8%; goal is to detect a lift to 11% (d = 0.03). n ≈ 16 × 0.08 × 0.92 ÷ 0.03² = 1.178 ÷ 0.0009 ≈ 1,310 visitors per variant, about 2,620 in total. Paid search sends about 312 clicks a month (NPR 12,500 ÷ NPR 40 CPC), so this test would take 8+ months. Not feasible.

Worked Example: Pathshala Tutors (cont.)

Redesigned plan: test ad CTR instead.

  • Variable: headline (Variant A question hook vs. Variant B number hook).
  • Hypothesis: "A number-led headline will raise CTR because it signals a concrete benefit."
  • Baseline CTR 4%; detect 5% (d = 0.01): n ≈ 16 × 0.04 × 0.96 ÷ 0.01² ≈ 6,144 impressions per variant.
  • At about 7,800 impressions a month (3,900 per variant), that takes about 7 weeks.
  • Decision rule: "If B's CTR is at least 20% higher after 6,144 impressions per variant and the difference is significant, roll out B. Otherwise, keep A and test a new offer."

For the landing page, test a bold change instead: a 3-field form with a WhatsApp button versus the current 7-field form, and report the result as directional.

Full walkthrough in the session note.

Check: Test Validity

Prompt: cold-call a student.

Lab: Design Your A/B Test Plan

  1. Pick one ad or landing page element to test
  2. Write your hypothesis and success metric
  3. Define minimum sample size / run duration

Deliverable for Session 15

A/B Test Plan

Third component of your A2 package.

Review Questions 📝

  1. List the five components of an A/B test plan.
  2. Using the rule of thumb, how many visitors per variant are needed to detect a lift from 5% to 7%?
  3. What is "peeking" and why is it a problem?

Apply it: A boutique hotel website gets 900 visits a month and converts 2% to booking enquiries. Is an A/B test on button colour sensible? Propose a better test design.

Thank You!

Open Session 15 Note