Session 15: A/B Test Design for Ads & Landing Pages

Block 2: Content Scaling, AI Workflows & A2 Part 1

View lecture slides: Session 15 Slides

Template: document each test in the Agency Content & A/B Test Matrix — one isolated variable and one falsifiable hypothesis per row.

1-Hour Lecture (Theory & Strategy)

A real A/B test plan needs: one variable changed at a time, a hypothesis, a success metric, a minimum sample/duration, and a decision rule (“if variant B beats A by X%, we roll it out”).
Common student mistake: testing 3 things at once (headline + image + CTA) and being unable to attribute the win to anything.
Read the full theory here: A/B Testing & Landing Page Optimization.
Statistical common sense (no heavy stats required): small sample sizes produce noisy results — state the caveat rather than over-claiming a “winner” from 20 clicks.

flowchart TD
    VAR[One variable only] --> HYP[Hypothesis]
    HYP --> MET[Success metric]
    MET --> DUR[Min sample / duration]
    DUR --> RULE[Decision rule]
    RULE --> OUT{Roll out or inconclusive?}

Figure 1: A/B test plan structure

Key Concepts in Depth

A test plan has five parts: variable, hypothesis, metric, sample/duration, decision rule.

How big a sample? A practical rule of thumb (about 80% power, 5% significance) is:

n ≈ 16 × p × (1 − p) ÷ d²

where n is visitors per variant, p is the baseline conversion rate, and d is the absolute lift you want to detect. Small lifts on small rates need very large samples.

When traffic is low, managers have three honest options:

Test a higher-funnel metric with more volume (ad CTR instead of lead rate).
Test a bolder change (a different offer, not a button colour), so the expected lift is larger.
Run the test longer, or treat the result as directional and say so.

Run tests for full weeks (to cover weekday/weekend patterns), avoid peeking and stopping early, and do not change anything else during the test.

Worked Example: Pathshala Tutors (Running Case)

Landing-page test (lead rate): baseline 8%; goal is to detect a lift to 11% (d = 0.03).

n ≈ 16 × 0.08 × 0.92 ÷ 0.03² = 1.178 ÷ 0.0009 ≈ 1,310 visitors per variant, about 2,620 in total.

Paid search sends about 312 clicks a month (NPR 12,500 ÷ NPR 40 CPC), so this test would take 8+ months. Not feasible.

Redesigned plan: test ad CTR instead.

Variable: headline (Variant A question hook vs. Variant B number hook).
Hypothesis: “A number-led headline will raise CTR because it signals a concrete benefit.”
Baseline CTR 4%; detect 5% (d = 0.01): n ≈ 16 × 0.04 × 0.96 ÷ 0.01² ≈ 6,144 impressions per variant.
At about 7,800 impressions a month (3,900 per variant), that takes about 7 weeks.
Decision rule: “If B’s CTR is at least 20% higher after 6,144 impressions per variant and the difference is significant, roll out B. Otherwise, keep A and test a new offer.”

For the landing page, test a bold change instead: a 3-field form with a WhatsApp button versus the current 7-field form, and report the result as directional.

1-Hour Lab (Agency Execution)

Pick ONE variable from your Session 14 ad set to test (headline, image, or CTA — not all three).
Write a one-sentence hypothesis: “Changing X to Y will increase [metric] because [reason].”
Define your success metric (CTR, form-fill rate, cost-per-lead) and a realistic minimum run duration/spend given a student budget.
Write your decision rule: what result would make you roll out the winner vs. call it inconclusive?
Do the same for one landing-page element (headline, hero image, or form length) on your Phase 1 sandbox site.

Common Mistakes

Changing headline, image and CTA together, so you cannot tell which change worked.
Declaring a winner from 20 clicks.
Stopping the test the day the variant looks ahead (“peeking”).
No decision rule written in advance, so the result is interpreted to suit opinions.

Deliverable: Feeds Assignment A2 (Part 3 of 4)

A documented, feasible A/B test plan — the “A/B Test Plan” rubric row of A2.

Summary

A feasible A/B test changes one variable, states a hypothesis, picks one metric, sizes the sample, and fixes a decision rule in advance. The rule of thumb n ≈ 16 × p × (1 − p) ÷ d² shows when a test is unrealistic. Low-traffic businesses should test higher-funnel metrics or bolder changes, and label results honestly.

Key Takeaways

One variable per test.
Size the sample before you start.
Write the decision rule before seeing results.
Low traffic: test CTR or bold changes, and call results directional.

Review Questions

Short answer

List the five components of an A/B test plan.
Using the rule of thumb, how many visitors per variant are needed to detect a lift from 5% to 7%?
What is “peeking” and why is it a problem?

Applied (case) questions

A boutique hotel website gets 900 visits a month and converts 2% to booking enquiries. Is an A/B test on button colour sensible? Propose a better test design.
Write a complete test plan (variable, hypothesis, metric, sample, decision rule) for a Meta ad for a Kathmandu bakery’s pre-order cakes.

Session 15: A/B Test Design for Ads & Landing Pages