User Usability Testing Vs A/B Testing : Friends or Foes?

Reading Time: 18 minutes

Difference Between A/B Testing and Usability Testing

• Usability testing and A/B testing aren’t poles apart as far as their purpose is concerned.
• The ultimate purpose of User testing and A/B testing is to improve the overall on-site experience for users and improve conversion rates

Purpose Of A/B Testing

• The objective or core purpose of A/B testing is to compare two or more website pages against each other to find out which one performs better.
• The focus of A/B testing should be on discovering which website version is improving your content engagement, reducing bounce rate, increasing conversions, increasing sales, and reducing cart abandonment.

Purpose Of User Usability Testing

• Usability testing is done to discover how people interact with the website, what problems they face while accomplishing a desired goal (for example, if they find it difficult to ‘add products to cart on mobile), and make improvements on the basis of the findings.

Why Usability Matters

• Usability comes under the umbrella of user experience or UX.
• One of the advantages of Usability testing is that it enables users to learn how to use the website/product without needing guidance and assistance.
• Another benefit of running usability tests is that it enables users to be able to complete tasks with the website and product with maximum accuracy.
• Error tolerance is another big win for usability testing because it helps to minimize the number of errors and ensure that the user can easily avoid the outcomes of those errors.
• Usability testing exposes efficiency issues that when addressed enable users to complete the goal or desired action in minimum possible steps, in the shortest possible time.

Pros Of A/B Testing

• You might have a million ideas that you think will help you improve conversion rates. However, only testing can prove which of those ideas are actually feasible and would give desired results.
• One main benefit of A/B testing is that it helps you make the most of your existing traffic. You’re already spending top dollar in getting traffic to your website, with A/B testing, you’re to find out which design will convert more of that traffic for you.
• Another benefit of A/B testing is that it encourages a culture of experimentation.
• A/B testing allows you to test almost all elements of your website, simple as well as complex.

Disadvantages of A/B testing

• A/B testing is time-consuming. Starting out, you might lot of ideas you want to test, also there might be clashes about which elements to test, and these are separate from actual testing which could take some time because you’re trying to achieve statistical significance.
• Getting any real lift is challenging and takes patience.
• A/B tests are slow to give results. In fact, the time required complete your A/B test will be inversely proportional to the traffic of the page.

Advantages Of Usability Testing

• Usability or user testing is free of statistical errors, is easier than AB testing to execute, and takes no more than a day or two to execute and give results.
• What you test is based on users’ exhibited behaviors. Therefore, experimentation bias gets eliminated.
• What you create is based on the direct feedback from your target audience. Therefore, you make ‘real’ improvements to your product, website, or app.
• If your optimization team members are not on the same page regarding what will perform better, they can resort to user reactions and feedback on different opinions
• The actual friction and bottlenecks that users face get fixed. Therefore, the chances of your product failing after launch are reduced to a minimum when the prototype is tested for usability and changes are made as per users’ exhibited behavior, feedback, and reaction.

Disadvantages Of Usability Testing

• User behavior during a usability test and in their natural environment is often different. When the user is in the natural environment his behavior and the way he interacts with the website is natural. This natural interaction and engagement behavior is what gives better insights into user behavior in a natural, day-to-day environment.
• Usability testing isn’t as deep as market research. With market research, you need a lot of respondents, with usability testing, 5–10 participants is good enough.
• Finding an audience who will take the usability test for genuinely finding flaws and giving feedback is challenging. Most people take a usability test just for the sake of money. They are not interested in helping improve the usability of the product or website. This brings in the issue of the accuracy and reliability of test results.

When to Pick User Testing and When to Choose AB Testing in the Optimization Process

• If you are looking for answers to questions like “Which landing page will give more sign-ups?” or “Which navigation menu will give you higher visits on a certain category?”, etc., then A/B testing is your friend. In such cases, you are testing how making a certain change can impact your returns or conversion rates.
• On the other hand, if you need answers to questions like “Are users successfully able to achieve a certain objective?” “Are users able to navigate smoothly to checkout?” or “Do users face distractions and bottlenecks on their on-site journey to end goal?”, then user testing is what you need.

The Combined Power of User Testing and A/B Testing — Reasons to use them together and the outcomes

• You can start with usability testing and move on to A/B testing if you have a new website or app.
• Insights from qualitative research done in usability testing can help obtain insights that optimizers can use in creating user-behavior-based hypotheses.
• Combining qualitative insights as a foundation for validating quantitative results increases reliability and accuracy.
• Follow this process: conduct usability tests for collecting user behavior-based inputs and then run AB tests for getting insights to find out which design alternative performs best. Moreover, your design options tend to be bulletproof when they are based on actual insights rather than plain guesswork.

P.s: Check the main article for the section on combining usability testing with A/B testing with a focus on the checkout process.

Benefits Of Combining A/B Testing With Usability Testing

• Complete use of quantitative and qualitative data-informed decision making
• Comprehensive view into how users interact with your app/website and how is their experience of using your app or website
• Insights into additional ideas to run tests that can increase engagement and conversions.

 

Here’s A Longer And More Detailed Version Of The Article.

---

Both user usability testing and a/b testing are important techniques used during conversion optimization. They aren’t poles apart as far as their purpose is concerned. You can say that they are indeed two sides of the same coin.

The ultimate purpose of user testing and a/b testing is to improve the overall on-site experience for users and to improve the conversion rates of websites. However, one must understand the 8 differences between the two and decide when to pick the right test based on its objective. So, let’s find out what is the purpose of each of these tests.

Purpose of A/B Testing

The objective or core purpose of A/B testing is to compare two or more website pages against each other to find out which one performs better. Or, in other words determine a winner amongst two or more website version on the basis of which one gets better conversions.

For example, if your website goal is to get more free trials, you could try using a floating free trial call-to-action on one of the website version ‘B’. On the control or Version ‘A’, let’s assume that you have the ‘free-trial’ call to action only at the end of the scroll. You can now run an A/B test and see which version is getting you more free trials. There are a number of other simple, as well, as complicated A/B tests that you can run on your website.

The focus of A/B testing should be on discovering which website version is improving your content engagement, reducing bounce rate, increasing conversions, increases sales, and reducing cart abandonment. At one time, however, A/B testing can be done only for one defined goal.

Purpose of User or Usability Testing

Usability testing is done to discover how people interact with the website, what problems they face while accomplishing a desired goal (for example, if they find it difficult to ‘add products to cart on mobile), and make improvement on the basis of the findings.

Based on these observations of user on-site behavior, usability testing aims at improving design and website functionality. As a result, the improved interface that enables users to navigate to their goals easily and quickly, and intuitively know how to work with the application, system, or website. According to Clifford Chi:

“Usability testing lets you address your product’s or website’s issues before you spend a ton of money creating something that ends up having a poor design.”

Why Usability Matters

Usability comes under the umbrella of user experience or UX. And, it is not just about ‘ease of use’. It is important to test for usability and improve it so as to excel in the following objectives:

Learnability: Enable users to operate the software/product/website by themselves, without needing any support or guidance. In a short period of time users become skilled enough to use the product/application on their own.

Effectiveness: Ensure that users complete the desired action or goal with maximum accuracy. An example of this would be fixing a credit card field on the form such that the form accepts only valid credit card details.  This enables users to perform the tasks or end goal ‘payment’ correctly.

Error Tolerance: Minimize the number of errors and ensure that the user can easily avoid the outcomes of those errors. For example, giving people to re-type their password if they have not entered it correctly. Also, in order to reduce errors, users should be presented with minimum options to pick the correct choice of action. For example, if you don’t want the user to enter the date in DD/MM/YYYY format, do not provide that option.

Efficiency: Enable users to complete the goal or desired action in minimum possible steps, in shortest possible time. Are users taking too long to fill the fields in your form? Provide autofill option. Can you reduce the number of steps needed to sign-up or subscribe to your service? If yes, then you must cut down on those steps that are unimportant. Unnecessary steps decrease user-friendliness and can put off users.

All the above objectives of usability are validated by user testing. Therefore, with user or usability testing you can tweak your website/product/app such that it complies with human behavior and interaction.

Now that we’ve discussed the purpose or importance of both A/B testing and user testing, let’s also dig into their benefits and disadvantages.

Pros and Cons of A/B Testing

Benefits of A/B Testing

You might have a million ideas that you think will help you improve conversion rates. However, only testing can prove which of those ideas are actually feasible and would give desired results. AB testing validates whether your ideas will give improved results over your current website or campaign on implementation.

One main benefit of A/B testing is that it helps you make the most of your existing traffic. What this means is that though you might be spending top dollars to get more traffic to your website, all that money might not trickle down to profits. However, if you focus on A/B testing for conversion optimization you can convert a greater percentage of users that form your current visitor base into customers. The cost of a starter A/B testing plan is approximately the same as that of 8 to 10 Google Adword clicks.

Another benefit of A/B testing, or scientific A/B testing, is that it encourages a culture of experimentation. Why we have specifically mentioned ‘scientific A/B testing’ is that earlier most companies randomly tested ideas. However, A/B testing is a waste of time, effort, and money if not done in a logical and scientific manner. Hypotheses building should be based on user observations, and hypotheses must be prioritized based on ease, importance, and cost of the test. All this makes A/B testing more efficient a testing process, setting the right expectations from the outcome, and reducing opportunity cost. The end result – an organization reaps the benefit of an experimentation culture, which is being able to actively generate actionable insights from data.

The third advantage of A/B testing is that it allows you to test almost all elements of your website, simple as well as complex. Some simple A/B tests you can run include testing:

• Headlines and subheadings
• Text
• Call to action button and text
• Social proof
• Images
• Testimonials
• Links
• Header and footer

However, even though AB testing allows you to tackle smaller less complex elements, it still requires you to use due diligence in knowing if a test is viable. Therefore, doing some type of research and constructing a hypothesis on that basis is important.

Some more challenging or advanced elements that you can A/B tests include pricing structures, free trial lengths, navigation, free shipping, free or paid delivery etc.

Lastly, A/B testing allows you to optimize one step at a time. So, if you have a big website or run a number of websites, you need to test everything at once. With A/B testing you can start small. Once you get comfortable with the process you can start A/B testing more complex and the main pages on your website. According to Michael Reddy:

“Powerful A/B testing provides more information about your customers than you may realize. It enables your ability to segment your customers on multiple levels in order to gain a deeper understanding of their behaviors.”

Disadvantages of A/B testing

There are some inherent problems with A/B testing that you must take into consideration before using it as an approach to optimize conversions. For example,

1. A/B testing is time consuming: When you are starting out as a business or website you will have hundreds of ideas that might work or won’t. Testing each idea isn’t feasible. To add on to the complexity, Individual opinions might clash while taking decisions such as which variables should be included in the test. Also, optimizers often lack sound statistical knowledge that is necessary to run the test as well as interpret test results accurately. As a result, AB tests are prone to statistical errors, which makes a lot of the efforts go to waste.

However, certain tools have made it really simple to run A/B tests. Further, to ensure that only the best of ideas are tested, hypotheses formulation has now become a more logic and scientific process. This has reduced significant efforts, time and money from going to the drain.

2) Getting ‘real’ life is challenging and takes patience: AB tests might fail to get you any real lift in conversion because of two main reasons: a) an AB testing measures the outcomes of something you ‘assume’. B) your A/B test is affected by external variables that can give you false positives. Let’s dive a bit into both of the above points:

• If based on ‘assumptions’, hampers test result accuracy: Often, the person who runs a test has a bias towards a preferred result. He or she proposes the hypotheses based on what he or she thinks will improve results. Even if proposed version B outperforms the control or version A, there is not enough reason or logic to prove that version B is the best. What if users were more likely to convert on a version C or D or E? When user behavior is not taken into account your with A/B testing you are not actually giving the users the best version that converts the highest.
• External variables increase chances of false variables: While interpreting test results, optimizers often fail to identify, measure, and exclude certain confounding variables. Free from the impact of such confounding external variables, AB testing would have given highly accurate results. However, AB tests are impacted by any changes take place while the test is running. For example, you might be running an AB test to test conversions on floating free trial CTA on version B, against control or version A that has a fixed CTA. Version B wins. However, during the AB test you were running Google Ad campaigns on your free trial landing page. And, it might actually be this campaign that got you more traffic and higher conversions. So, the real lift in conversions might actually be taking place because of the Adword campaign. Did you take that into account?

3) AB tests are slow to give results: Not only is it time consuming to decide what ab tests to run, it also takes a lot of time before one can get the results from their ab tests. Infact, the time required complete your A/B test will be inversely proportional to your the traffic of the page. It can take a month or even more to reach statistically significant results. And, even after running the test for THAT duration to reach the right sample size and statistical significance, there is no complete guarantee that the results are actually providing any real lift in conversions.

The point to note here is that getting statistically significant results in less time is difficult for websites that are new and have low traffic. On the other hand, if your website has been around for long and has a good traffic, AB testing would be a good optimization choice.

This brings us to user testing’s advantages and disadvantages. Let’s dive in:

Pros and Cons of Usability Testing

While we are certainly not saying that user testing is a better alternative to a/b testing, understanding the advantages and disadvantages of this testing technique will help you figure out if you must opt for it while optimizing your website/app/product.

Advantages of User Usability testing

• Usability or user testing is free of statistical errors, is easier than AB testing to execute, and takes no more than a day or two to execute and give results.
• What you test is based on users’ exhibited behaviors. Therefore, experimentation bias gets eliminated.
• What you create is based on the direct feedback from your target audience. Therefore, you make ‘real’ improvements to your product, website, or app.
• If your optimization team members are not on the same page regarding what will perform better, they can resort to user reactions and feedback on different opinions
• The actual friction and bottlenecks that users face get fixed. Therefore, the chances of your product failing after launch are reduced to minimum when the prototype is tested for usability and changes are made as per users’ exhibited behavior, feedback, and reaction.

In a nutshell, user testing is absolutely customer-centric. When conducting usability testing you aren’t serving limited assumption based options to your users to decide which one in better. Instead, you are taking the overall experience of your users while they use the product. You optimize the overall journey and make it friction free for your target audience.

Disadvantages of user testing

Actual behavior vs. exhibited behavior during test: User behavior during a usability test and in their natural environment is often different. A home shopper might, for example, be listening to music, talking to someone, or doing several other tasks other than browsing and buying on-site. However, while a usability test, the user’s full attention is directed towards how he navigates the site, what actions he takes etc. Therefore, while he might be able to get better feedback and catch some obvious flaws on the website during the test, when the user is in the natural environment his behavior and the way he interacts with the website is natural.

This natural interaction and engagement behavior is what gives better insights into user behavior in natural, day to day environment. For example, a participant is likely to be much more careful about making a purchase in regular, natural or day-to-day environment as compared to a purchase that they might pretending to complete for the sake of usability testing.

Not as deep as market research: Unlike market research, usability testing can be completed even with as few as 5 to 10 users. Even just a handful of people can detect usability flaws easily and give reliable results. However, this does not solve market research problems such as design preference or whether people are actually willing and ready to use the product. For understanding these deeper issues, a much larger sample size is required to drive reliable and accurate conclusions.

Testing junkies often are in the test just for the money: Finding an audience who will take the usability test for genuinely finding flaws and giving feedback is challenging. Most people take a usability test just for the sake of money. They are not interested in helping improve the usability of the product or website. This brings in the issue of accuracy and reliability of test results.

One point not included – usability tests can be conducted in a very incorrect manner if the person doesn’t have know how. At times, the moderator “coaches” the visitor. These interference skew results.

We aren’t putting one against the other. That’s not the purpose of this article. However, both these techniques answer different questions and come into play at different stages of optimization. Let’s take a look at what questions they answer and where to deploy each during optimization.

When to Pick User Testing and When to Choose AB Testing in the Optimization Process

User testing vs. AB testing can also be defined as testing for user behavior vs. testing for user preferences.

Let’s explain it this way – If you are looking for answers to questions like “Which landing page will give more sign-ups?” or “Which navigation menu will give you higher visits on a certain category?”, etc., then A/B testing is your friend.  In such cases, you are testing how making a certain change can impact your returns or conversion rates. To get answers to such questions you define a conversion goal and create a version of your hypothesis based on insights obtained from qualitative and quantitative data. You then run your AB test and find out which version gets you better conversions.

On the other hand, if you need answers to questions like “Are users successfully able achieve a certain objective?” or “Are users able to navigate smoothly to checkout?” or “Do users face distractions and bottlenecks on their on-site journey to end goal?”, then user testing is what you need. This means that essentially you are gauging and tracking how users interact with a product/webpage/app, for completing a given task. This is unlike tracking ‘the number of people who successfully completed that given task/desired goal.

You can also see it this way – If you want to find out ‘What’ performs better then use A/B testing. And, if you want to find out ‘Why’ something works better or fails to work perform usability test.

Another point that needs to mentioned here is that usability testing is something that you can start out with when your website is new and has never been tested for conversions. Initially, you can keep improving on the usability to make the best website/product available to your users.

Once you’ve created your website/product/app based on genuine user feedback and user behavior, and are getting no more value from user testing, you can start the process of hypotheses building, idea generation, and carry out AB tests to further keep improving conversions on your website/product/app. According to Priya Viswanathan:

“In the earlier stages of the product, when the adoption is low, the best way to obtain quick feedback about usability is by doing User testing.”

The Combined Power of User Testing and A/B Testing – Reasons to use them together and the outcomes

As discussed above, you can start with user testing and then keep the conversion optimization process going with A/B testing, if you have a new website/product/app. However, you can use the combined power of user testing and AB testing regularly and make it a part of your testing culture.

An experienced and wise optimization expert would agree that one of the key prerequisites of getting valid AB testing results is building a strong hypotheses that is free of bias. Usability testing can be used while formulating hypotheses. Insights from qualitative research done in usability testing can help obtain insights that optimizers can use in creating user-behavior based hypotheses.

Combining qualitative insights as a foundation for validating quantitative results increases reliability and accuracy. So, what we are saying is that you must conduct usability tests for collecting user behavior based inputs and then run AB tests for getting insights to find out which design alternative performs best. Moreover, your design options tend to be bulletproof when they are based on actual insights rather than plain guesswork.

Follow the given steps when using user testing and AB testing together:

Step 1:  Collect user input and qualitative feedback using usability testing. If an AB test is driving your current project, usability testing should be highly focused on that particular test. For example, if you are planning to AB test a landing page talk to a small number of target users and gather general feedback on the page. This will give you tremendous insights into crafting better content or design.

Step 2: Evaluate the inputs from your usability study and create design alternatives. A key step here would be to prioritize your qualitative research findings. While there might be multiple inputs to improve the design, you need to come to a consensus and test just one design at a time. You should not change too many things on the landing page being AB tested.

Step 3: Create a wireframe and design the alternate version, once you and your team have reached a consensus. Push the control and the new design version live for A/B testing.

Step 4: Run the test till it gives conclusive results and compare the outcomes.

Step 5: Pre-Launch Prototyping: Additionally, you could opt for prototyping. Before rolling out certain additional functionalities in your app, for example, you can create a mock app or website that your users can assess for feasibility or viability before you actually launch. A tip here is that you should share the wireframe or prototype with users while it is still in the developmental phase. You can start as soon as you have a rough mockup idea ready. The earlier you get feedback from users, the sooner you’ll be able to address drawbacks and not have to make major changes at the end of phase. According to Anthony Brebion:

“Starting with a usability test allows you to locate the points of friction to address. You can then conduct an A/B test to validate the relevance of the changes you are considering.”

Example of Combining User Testing with AB Testing

Studying and Optimizing Checkout Process

Have an eCommerce website? One of the most important thing that you need to optimize for is the convenience of your checkout process. You also must optimize checkout such that it is completed in minimum steps and takes minimum time.

Begin with usability testing. Observe participants in your checkout usability test. See how they navigate from one page on your website to another. Notice how they make ‘add to carts’. Understanding obstructions and bottlenecks while checking out is the key to improving usability. For example, is user hesitant to ‘register’ while ‘adding to cart’? Notice facial expressions and body language to know where users are getting stuck or confused. Take note of every small and big feedback. They point out that there isn’t any ‘support’ number that they might need while placing the order, note that down. They point out that you don’t have ‘order confirmation’ message showing post checkout, note that down. They point out that your ‘checkout’ options don’t have ‘payment wallet option’, note that down.

Create new variation based on this feedback. This version that you have developed qualifies as for basic feasibility that has addressed all those bottlenecks that were pointed out during the usability test.

AB Testing for Improved Conversions. Once you have optimized your checkout for basic usability and ease, you can spin more ideas that could further help optimize the checkout process. For example, you could hypothesize that providing option to avail free shipping on orders exceeding a certain value can help increase AOV on checkout. You could design and run an AB test and optimize based on test results.

How User Testing Fueled A/B Tests Improve Optimization Results

A/B testing and usability testing are conversion optimization techniques that work best when used together. The clear cut benefits of combining A/B testing with usability testing, in a nutshell, include:

• Complete use of quantitative and qualitative data informed decision making
• Comprehensive view into how users interact with your app/website and how is their experience of using your app or website
• Insights into additional ideas to run tests that can increase engagement and conversions

Now that we’ve come to the conclusion that both user testing and AB testing must be used together for optimizing apps for experience, usability, and ultimately conversions, let’s find out if any of you have used both these techniques together. Have you? And, what were your findings? Drop us a comment.