A/B testing (split testing) compares two versions of a page or element to see which better achieves a defined outcome. Landing page optimization applies CRO (conversion rate optimization) principles so traffic from SEO, ads, and email converts.
Learning Objectives
By the end of this chapter, you will be able to:
- Run an A/B test from hypothesis to decision.
- Explain statistical significance in simple terms and avoid common testing errors.
- Apply a landing page CRO checklist.
- Calculate conversion lift and judge whether a result is reliable.
A/B Testing Process
- Hypothesis: “Changing the CTA from ‘Submit’ to ‘Get a free quote’ will increase form key events because intent is clearer.”
- One primary change: Test headline or CTA or hero image — not everything at once if you need clear learning.
- Split traffic: Send comparable users to A (control) and B (variant).
- Run long enough: Avoid calling winners after one quiet afternoon; need adequate sample size.
- Decide with a metric: Key event rate, not “which looked nicer.”
What to test
| Element | Example A | Example B |
|---|---|---|
| Headline | “Quality Furniture in Kathmandu” | “Custom Sofas Delivered in 10 Days” |
| CTA | “Submit” | “Get my free quote” |
| Hero image | Product on white background | Product in a real Nepali living room |
| Form length | 8 fields | 3 fields |
| Social proof | No reviews | Three Google reviews near the button |
Statistical Caution
- Premature stopping creates false winners.
- Seasonality matters in Nepal (Dashain/Tihar traffic ≠ normal weeks).
- Mobile vs desktop mixes can hide effects — check device splits in GA4.
- Low-traffic SME sites may need sequential tests or qualitative tools (Clarity) before formal A/B platforms.
Worked example: is the winner real?
| Version | Visitors | Conversions | Conversion rate |
|---|---|---|---|
| A (control) | 2,000 | 60 | 3.00% |
| B (variant) | 2,000 | 85 | 4.25% |
- Relative lift = (4.25 − 3.00) ÷ 3.00 = 41.7%
- A two-proportion z-test gives z ≈ 2.1. Because this is above 1.96, the difference is statistically significant at the 95% confidence level: it is unlikely to be due to chance alone.
Now imagine the same rates with only 200 visitors per version (6 vs about 9 conversions). The lift looks identical, but z is well below 1.96, so the result is not reliable. Small samples produce false winners, which is why tests must run long enough. Online significance calculators do this arithmetic for you.
Landing Page CRO Checklist
- Message match: Headline mirrors the ad/keyword promise.
- One primary CTA: Call, WhatsApp, or form — not five competing buttons.
- Proof: Reviews, licenses, photos of real location/team.
- Speed & mobile: Fast on 4G; thumb-friendly buttons.
- Friction cuts: Fewer form fields; show trust near the submit button.
- Track the CTA: GA4 events for form submit,
tel:clicks, WhatsApp clicks.
Nepal practice: Many ad clicks fail because the landing page asks for long forms when users expect a call or WhatsApp chat. Test a short form vs click-to-WhatsApp CTA and measure inquiry volume and sales-qualified rate.
Case study: Form vs WhatsApp CTA
A mortgage broker tested long application form vs “Chat on WhatsApp for eligibility.” Chat variant produced more conversations; form variant produced fewer but higher-documentation leads. They kept both paths: WhatsApp for speed, form for serious applicants — scored separately in the CRM.
Learn more: conversion optimization services in Nepal.
Summary
A/B testing compares a control and a variant to see which better achieves a defined outcome. A good test starts with a hypothesis, changes one main element, splits comparable traffic, runs long enough, and decides on a key metric. Statistical significance protects against false winners from small samples, seasonality, and device mix. Landing page optimization applies message match, one clear CTA, proof, speed, reduced friction, and tracking so that traffic from every channel converts.
Key Takeaways
- Hypothesis → one change → split traffic → enough sample → decide on the key metric.
- Relative lift = (B − A) ÷ A; check significance before declaring a winner.
- Avoid testing during unusual weeks such as Dashain unless that is the goal.
- Message match and one primary CTA are the foundations of landing page CRO.
Discussion Questions
- Suggest three A/B tests for a Nepali e-commerce product page.
- Why can stopping a test early lead to the wrong decision?
- Why did the mortgage broker keep both the form and the WhatsApp path?
Exam-Style Questions
Short answer
- What is A/B testing?
- What is a hypothesis in A/B testing? Give an example.
- What is meant by message match?
- Why is sample size important in A/B testing?
Long answer
- Explain the A/B testing process and the statistical cautions marketers should follow.
- Version A of a landing page had 3,000 visitors and 90 conversions; version B had 3,000 visitors and 120 conversions. Calculate the conversion rates and relative lift, and explain what else you would check before declaring B the winner.
