A/B Testing For CRO: Best Practices, Examples & Steps

A/B is one of the most effective ways to increase conversions on a website, be it newsletter subscriptions, sales, average order value, or software signups.

That's why A/B testing is a crucial element of CRO strategies.

In this guide, you'll learn how to get started with A/B testing for increase conversions and conduct your CRO experiments the right way and with the right mindset from the very first run.

Go to sections:

• What is A/B testing in the context of CRO? 

• Benefits of doing A/B tests for CRO

• Key website areas to test for improving conversions

• Examples of A/B testing for CRO

• Types of testing for optimizing conversions

• How to set up an A/B test for CRO

What is A/B testing in CRO?

A/B testing, or split testing, is a CRO tactic in which two or more versions of a website page or onsite campaign are compared to test a pre-developed hypothesis about their performance. This way, A/B testing measures the real value of changes or tweaks you make on our website.

Regular and ongoing A/B testing is essential to increasing conversions because it's the most effective way to consistently improve visitor engagement and other marketing results based on real data.

Benefits of A/B testing for CRO

• Better ROI. Converting more visitors will help you achieve better returns on your paid ad and email campaigns

• Higher conversions. Testing allows to find and implement website changes that lead to higher conversion rates

• Helpful customer insights. Experimenting helps with learning more about your visitors' behavior, audience, and pain points

• Improved user experience. A/B testing allows for continuous improvement in website design and functionality, leading to a better experience for visitors

• More sustainable growth. Split testing helps to optimize campaigns where you have the most control: your website, which is more sustainable compared to other growth strategies

Key areas to test for boosting conversions

Although you can test nearly everything on your website, online businesses focus on these areas to improve their conversions:

• CTA buttons (text, placement)

• headlines (length, phrasing)

• signup forms (headings, incentives, fields, content, placement)

• website popups (headings, incentives, content, placement)

• onsite campaign format (eg. popups vs website bars)

• unique selling point (value propositions, tone, length)

• landing page designs (layout, copy, CTA button placement)

• email campaigns (subject lines, content, send times, incentives)

• website menus (placement, links, color, text)

• checkout (number of pages, upsell offers, payment methods)

📕 Note:

These key areas cover the entire onsite customer journey, allowing you to test unique customer insights and less obvious elements, such as campaign timing, emotional tone of marketing messages, gamification tactics, multi-offer strategies, and entry vs exit offers.

Examples of A/B testing for CRO

One more example:

See how to create an A/B test in which one popup campaign with a discount code contains a countdown timer and the other doesn't:

Types of A/B testing for CRO

• A/B (split) testing. Compares two versions of a page or campaign (control version vs. variant) to see which one performs better

• A/B/n testing. Involves comparing the original with two or more versions, mainly to test multiple ideas in one run

• Multivariate testing. Tests multiple elements on a single page at once, allowing for more insights on how different combinations affect conversions

• Multi-armed bandit testing. A type of testing that uses traffic allocation algorithms to track engagement with the tested versions and adjusts the percentage of visitors based on ongoing results

How to set up an A/B test for CRO

Doing a simple A/B test means following these steps:

1. Choose an area for improvement

2. Come up with a hypothesis

3. Create versions A and B

4. Calculate statistical significance

5. Run the test

6. Review the results

7. Publish the winning version

Step 1: Choose an area for improvement

Decide on a target area you'd like to improve.

These can be:

• web pages with high bounce rates

• signup forms with very low conversions

• checkout pages with high abandonment rates

💡 Expert tip:

Consider testing the elements/campaigns you find the most impactful for your business. Although running many tests is easy with A/B testing tools, start with simple experiments that test elements or campaigns that should affect conversions. For example, instead of testing colors and copies of every button, begin by experimenting with different discount sizes and go from there.

Step 2: Come up with a hypothesis

A hypothesis is simply a statement where you describe the desired outcome of the test and provide the reasoning for your prediction.

Here's a simple formula to create a hypothesis for split testing:

Changing (element tested) from _________ to ________ will increase (a conversion metric) because (the reasoning).

Examples:

• "If we reduce the number of signup fields in our newsletter form from two to one, we will improve our lead generation by 10%. because it will be easier to convert."

• "If we add a countdown timer to our banner with the offer for returning visitors, we will increase our conversion rate by 15% because of the sense of urgency it creates."

• If we change the copy of the homepage CTA from "Book a demo" to "Request a free personalized demo," we can increase demo signups by 20% due to the more personalized and less commitment-heavy language."

Start by testing one element (button copy, discount size, etc.) to build your confidence and collect some useful insights to inform further experiments.

As you gain more experience, you can test multiple elements (like below, an online store is testing different discount sizes, campaign format, and copy) and even create multivariate tests to see how different combinations affect conversions.

Step 3: Create versions A and B

If you're testing the change of only one element, make two campaigns.

• Version A (the control): the original version

• Version B (with the change): a modified version

In A/B testing software, creating modified versions can be very easy: many times, it just boils down to duplicating version A and making the change.

After both versions are done—

You may need to choose the visitor category to display your campaigns to (new/returning, desktop/mobile, etc.).:

Also, you'll need to define the percentage of traffic that each version will get. In most cases, A/B testing tools automatically divide it evenly (50%/50%).

Step 4: Calculate statistical significance

Your A/B test must be run for a sufficient duration to give you reliable results (e.g. be statistically significant).

Although the time depends on the business, in many cases, you'll need at least 1,000 conversions per version. A "conversion" doesn't have to be a sale—it can also refer to actions like signing up for an email list or downloading a resource.

To calculate how long the test needs to run for before you can call it a success use the A/B test statistical significance calculator.

Step 5: Set a goal for the experiment

Many A/B testing software also allows you to track goals for your experiments.

For example, if you're testing the impact of different homepage headlines on your signups, your goal might be "to start the signup process," e.g. visit the first page of the signup flow.

In this case, all you have to do is set your goal as "page view" and set the URL of the signup page as the "goal value."

The settings for this might look something like this:

Step 6: Run the test

The results you got from the statistical significance calculator should be your guideline. But you'll still need to monitor the test manually and decide when it's time to act on your results.

Again, the timing may vary, but consider running the test for at least a month before doing the initial manual review. The longer it runs, the more reliable results will be. Ending the test too early may result in picking the wrong version, even if it seems like the clear winner at first.

Step 7: Review the results

Once the statistical significance is achieved, it's time to evaluate the performance. Analyzing the data is crucial to understand what worked and what didn't.

Things to do here:

• Check conversion rates. Did version B outperform A? If yes, by how much?

• Consider the cost-benefit. Decide if the increase in conversions justifies the cost of implementing version B

• Evaluate other metrics. Depending on the type of test, look at other relevant metrics, such as bounce rate, time on page, email collected etc., to gain a complete understanding of the impact of your changes

Ask yourself why that positive change happened. A/B testing is all about learning, so analyze the winning campaign to know why visitors or customers preferred it.

Step 8: Publish the winning version on your website

After analyzing the results of your A/B test, enable the winning version and delete the other. Now, your task is to monitor its performance to understand the impact over time.

If you feel like other parts of your website or campaigns may benefit from the change that produced better results, consider applying this knowledge and starting some more A/B tests.

Summary

A/B testing is an effective way to improve conversion rates and better understand your audience's preferences. By following a structured process, you can make data-driven decisions to optimize your website or campaigns.

Keep learning about CRO:

• Guide to A/B testing in ecommerce

• 20 ways to improve the conversion rate on Shopify stores

• What is a website conversion funnel? A beginner's guide

• Strategies how to reduce customer acquisition costs