Comparing Multivariate & AB Testing Tools

It is time now to select an AB or multivariate testing platform to start running your tests. There are over thirty different tools available in the marketplace for you to choose from. So, you might ask:

• What is the right split testing tool for my website?
• Will I affect the success of my conversion optimization program by choosing one particular tool?

The market is full of testing tools that range from simple free tools to sophisticated enterprise split testing software. This article will walk you through the pros and cons of different split testing tools available.

We will also suggest criteria to help you select the software that best fits your needs, considering ease of use, the agility of results, cost of running, testing support, operational issues, and testing analytics.

The different types of AB testing tools available

There’s no shortage of conversion rate optimization tools in the market. Cancelled 

CRO tools can help you understand the performance of your website by showing you how visitors use it. Simply put, these tools collect data about your visitors and website, therefore helping you identify problematic areas on your website that are making visitors struggle.

Before I give you a list of the best CRO tools out there, it’s important to mention that these tools fall into three categories:

1. Web analytics CRO tools
2.  Behavior analytics CRO tools
3.  CRO testing tools
4. Web analytics CRO tools: Show you what’s happening on your site

You can’t improve or optimize your site’s conversion rate if you don’t know what’s happening on it, can you?

This is where Web analytics CRO tools come in.

Web analytics CRO tools help you understand your site in terms of numbers. They offer you quantitative data to answer questions like:

1. How long are visitors staying on my site?
2.  Which landing pages have the highest bounce rate?
3.  Which stage of the conversion funnel am I losing visitors?

Knowing the answers to the above questions is important as it gives your optimization efforts a high chance of being successful.

1. Behavior analytics CRO tools: Shows the behavior of your website visitors

To improve your website’s conversion rate, you need to provide an excellent experience that is in line with what your visitors want. So, how do you know what visitors want?

Well, one way of doing that is by looking at how they behave on your website.

Unlike web analytics tools, user behavior analytics tools can show you exactly what visitors are doing on your website by collecting qualitative data. You can rely on behavior analytics tools to answer the following questions:

1. What’s causing visitors to leave the site without purchasing?
2.  Are visitors finding what items they are looking for?
3.  Which website areas are frustrating visitors?

Without knowing the answers to the above questions, your optimization efforts are going to be fruitless.

2. CRO testing tools: Allow you to compare and measure website changes

Testing is an important piece of the conversion rate optimization puzzle. The process usually involves testing two or more design versions of a web page to determine which version has more conversions.

Before you permanently implement changes on your website, you need to measure and see if they make your website better or not. This is why you need to have a reliable A/B testing tool.

RELATED ARTICLES: Top 10 A/B Testing Tools Used By CRO Experts in 2021

CRO testing tools can be used to test different web elements such as the position of a call to action button, web form size/placement, landing pages, website copy, website navigation, checkout process of an eCommerce site, pricing models, etc.

Now that you know how these tools work, here’s a list of the best conversion rate optimization tools:

• Google Analytics 360
•  FigPii
•  Heap
•  Adobe Analytics
•  Crazy Egg
•  FullStory
•  Istanpage
•  VWO

Some of the above tools have similar features and only differ in price plans. However, our CRO team relies on FigPii when it comes to understanding user behavior, and it is the only tool on that list that has a recommendation feature that pinpoints problem areas you should fix right away on your site. We also use Google Analytics as our main tool for tracking traffic data.

Classifying A/B testing tools based on price

Split testing tools vary in price from free of charge to tools that can cost tens of thousands of dollars.

1. Tools priced based on site metrics

These tools are priced based on specific website metrics. Different tools use different metrics to set the price, including:

1. The number of tested visitors in a time period (typically a month)
2. Number of the website’s monthly visitors
3. Number of monthly page views
4. The number of calls to the testing tool.

These tools will vary in price and range anywhere from $50/month to tens of thousands of dollars.

2. Free AB testing tools

These tools provide the customer the ability to conduct split testing free of charge, but they offer limited sets of features, including a smaller set of segmentation, targeting and reporting functions. The leading player in this space is Google Optimize which continues to add more features on a regular basis.

Classifying AB testing tools based on testing technology (client side vs. server side)

There are two different types of ways testing tools are used on a website:

1. Client-side testing tools

In these tools, the testing scripts are executed by the browser of the website visitor. When a visitor comes to a webpage, the testing software javascript is executed, and the particulars of the test are loaded on the page.

Client-side testing tools typically come with a point-and-click visual editor that allows users to manipulate a web page.

In theory, these tools require limited involvement from the user’s technical team. Creating new designs with these tools is much faster compared to their counterparts.

Client-side testing tools are great for creating new designs for one page, but they are less suited to conducting more complex testing, such as visitor flow tests. The cornerstone “visual editor” of these platforms breaks when modifying websites that rely heavily on JavaScript or dynamic content.

These tools come with a low-cost monthly subscription starting at $50 per month. The subscription cost will vary based on the number of visitors a website receives in a month.

Amongst client-side testing tools, you can find:

• Optimizely
• Visual Website Optimizer
• Convert
• Omni convert

2. Server-side testing tool

In these tools, the testing script runs on the testing platform as opposed to the visitors’ browser. The visitor’s browser gets the final version to be displayed.

There are several ways testing is implemented using server-side tools. Adobe tests and tags, for example, rely on the markings made by users on specific areas of the tested webpage. Variations of the marked areas are then created in the testing software by the optimization or development team. Each new test will require involvement from the development team.

Server-side testing tools frequently involve custom configuration to retrieve meaningful visitor tracking data. They also require organizations to have a full-time dedicated resource(s) to manage the software and to analyze the results.

While many of these tools do not come with an easy-to-use “visual editor” to make quick changes on a webpage, they do provide detailed reporting and targeting features.

Monthly plans for these tools start around $1,000 and vary based on the number of page views a customer uses in a month. In addition to the software cost, we estimate a $100,000 annual in-house cost of running tests on enterprise testing packages.

Server-side testing platforms are useful in conducting sophisticated testing such as multi-page testing, mobile application testing or visitor flow testing. However, they are overkill when doing quick A/B testing for a webpage design due to the substantial cost associated with the deployment of each test. The leading player in this field is Adobe Test and Target. 

Selecting the right testing tool for your website

With many tools available, the following criteria can help you choose the best split-testing software to meet your requirements.

1. Ease of use: How fast can you deploy a test?

We evaluate the ability of a testing tool to deploy tests quickly as a way of measuring its ease of use.

When you are conducting a conversion optimization program, your goal is to implement tests efficiently and not to have technology as a barrier to your efforts of increasing conversion rates.

Testing tools vary tremendously when it comes to ease of use. Some of the more sophisticated tools require complicated test setups, which can take days. Client-side testing tools outperform any of their counterparts in this area.

2. What is the cost of running the AB testing software?

The cost of running testing software varies from zero to thousands of dollars per month.

In addition to the cost of running the software itself, there might be additional required investments. Some split testing software is too complicated and will require having full-time staff to manage in addition to involvement from your development team. That, of course, will increase the cost of running your testing program.

3. Testing support

Sooner or later, you will need some help from the testing software company. Many smaller companies ignore this one area when they are doing the initial assessment of the testing packages. We highly recommend evaluating the different alternatives available.

4. Operational issues (scale, performance, and high availability)

How long does it take the testing software to load up a particular design? Does the testing software use CDN (content delivery network) to deliver designs to visitors who live in different parts of the world?

We highly recommend evaluating the response time for each of the different software packages. Avoid any testing package that takes longer than 500 milliseconds to deliver a design.

5. Testing analytics

In addition to reporting the conversion rate for each variation, some split testing platforms will also report the following metrics for each of the variations:

• Revenue per visit
• Bounce rate
• Exit rate
• Visitor type (new vs. repeat)
• Traffic source
• Traffic medium

This data is essential to understand the impact of the new designs on all aspects of the visitor experience on your website. We recommend evaluating what metrics the testing software reports in a test as that can impact your final analysis.

6.  What multivariate approach does the software use?

This point is particular in evaluating multivariate testing software. When launching a multivariate test, the software will display the original and new variations to different visitors to determine the winning design. To select the winning design, the software will use either full factorial or fractional factorial testing:

Full Factorial Testing

The testing software will test all of the different combinations of elements and their alternatives. So, if a test has four elements and each element has three different combinations in it, the testing software will test all possible 3^4= 81 designs.

Fractional Factorial testing (Taguchi method)

The testing software will select a subset of all possible combinations of the different elements and their alternatives. So, in our example above, the testing software will fully test a smaller set (less than 30) to determine the winner. Fractional factorial testing allows tests to run faster since they do not fully test all possible combinations. Critics of this method point out that it is less accurate compared to full factorial testing.

The debate between full factorial vs. fractional factorial has been going on for years.

Most MVT testing software uses full factorial in determining the winner of an experiment or a mix between the two different methods. While this might be a sticking point for some conversion experts, for 90% of online businesses, it is not a critical point in selecting which software to use.