The Difference Between A/B Testing and Multivariate Testing

Both A/B testing and multivariate testing share the same concept: splitting visitors between several designs of your website or webpage to determine which of these designs generates more conversions.

So, which of these two methods is best suited to help you increase your website conversion rate?

AB Testing

AB Testing (or Split testing) allows you to test one or more variations for each page on your website against each other.

Let’s take few examples:

• On an e-commerce website, you can create a new design of your product page to test whether the product image should be placed on the left or the right side of the page.
• On a lead generation website, you can use A/B testing to create multiple designs of a landing page where each design uses different headlines or different images.
• On a blog subscription page, you can test different designs displaying assorted colors and text on the subscribe button.

Editor note: Please note that we use button colors, fonts, headlines as a way to clarify the concept of testing. Succesful AB and multivariate tests should include more sophisticated changes to your page.

There are two general approaches to creating AB tests:

1. Testing new, radical designs of an entire page or process. As an example, testing three entirely different and distinct designs of a homepage
2. Testing small changes on a page: each design makes a slight shift in the same element. For example, each variation will test different hero images and so on

In our experience, we recommend using AB testing to measure the performance of two to five new designs compared to the original design. We typically do not run more than seven designs against an original page. It is also important to remember that the more designs you introduce in a split test:

• The test will require more time to reach statistical significance (conclude)
• Your results are more likely to suffer from  statistical errors

In an ideal world, each challenger to the control will include one change. However, this approach does not work in reality. Doing so will require both a long time and a significant budget. Testing more radical changes in each variation is a better approach. At the same time, it is important to remember that:

• A single and coherent hypothesis drives every change you make on a page. The hypothesis can drive several changes to several elements on a page
• Radical change to a page does NOT mean that you are randomly changing elements on the page. You should always tie any change to your test hypothesis

Multivariate Testing

Multivariate testing allows you to test several variations of multiple elements on a webpage all at once.

For example, you can test multiple options of the product name on an e-commerce product page, along with various alternatives of the product image and different types and colors of add-to-cart buttons.

You can also use multivariate testing to evaluate various headlines and several designs of the CTA on a lead generation landing page.

MVT testing allows you to zoom in and focus on the changes you are making at an element level.

While multivariate testing is great in theory to analyze element level impact on conversion rates, it has three drawbacks:

1. It requires significant levels of traffic or conversions to conclude a test
2. If you are not careful when designing your multivariate test, you can end up testing thousands of designs against the original
3. A single multivariate test includes several simultaneous changes to multiple elements; thus it is hard to isolate the exact reason a particular design performs a certain way. It is also difficult to assess the impact of elements interaction on your original design

MVT testing is very powerful. However, you must be careful when using it to optimize your conversion rates. Most companies forget about the large volume of visitors and conversions required to conclude a test. As a result, they find themselves running tests for several weeks without bringing them to a conclusion.

Additionally, since testing software allows them to do so, many companies tend to alternate elements randomly without thinking about the rationale behind the change. This mistake alone is enough to reduce the impact (or even kill) the benefits of any testing program.

Multivariate test vs. an A/B test: what type of test should you use?

Should you start with an A/B or multivariate test for a particular page?

There is no correct answer to this question. A/B tests are useful for testing alternate designs of an entire page or a process. We usually recommend using them while deciding high-level or radical changes to the optimized page (or area). MVT testing, on the other hand, allows for fine-grained testing on a particular page. They are helpful in determining the most impactful elements on visitors.

We highly recommend using AB testing in the following scenarios:

• If you are starting out with the testing process
• If you have limited number of website visitors
• If you have limited number of conversions
• If you are looking for a radical departure from your existing designs

We highly recommend using multivariate testing in the following scenarios:

• If you have been conducting AB tests for at least 8-12 months
• If you have a large number of monthly visitors
• If you have a large number of monthly conversions (more than 10,000 conversions per month)
• If you are looking to fine-tune existing designs

For most companies, we recommend starting with an A/B experiment to assess significant design changes to the website.

However, if you already have an existing page, you can start with a small multivariate test (less than 12 or so different scenarios). The goal of this initial test is to determine which of the elements (headline, image, benefit list, etc.) resonates most with visitors. Analysis of the first test results will help guide the need for further MVT or A/B tests.

Five steps when testing your website

1. Determine which of your website pages are good candidates for optimization by finding which of your pages are leaking the most visitors. Calculate the potential revenue impact you can gain from fixing the leak.
2. Check the number of visitors to your webpage. The number of visitors who will go through the actual web page or process impacts your testing velocity. Although your site may have 500,000 visitors a month, a particular page that you want to test may receive fewer visitors in a single month
3. Do not run your tests for less than two weeks nor longer than four weeks
4. Pick the right conversion goal. Not every test should have the goal of increasing the macro conversions. Many successful tests help in increasing micro-conversion rates. This is particularly important if your website or landing page does not receive enough conversions. Starting out with micro conversion tests allows you to conduct experiments on a smaller scale.
5. Examine what elements you should test. Not all elements on a page will have the same impact on your conversion rate. Determine which elements will have the most impact on your bottom line based on the voice of customer data, persona development, and analytics analysis.