Remote Usability Testing – How To Do Usability Testing on a Budget

Disclaimer: This section is a TL;DR of the main article and it’s for you if you’re not interested in reading the whole article. On the other hand, if you want to read the full blog, just scroll down and you’ll see the introduction.

• Remote usability testing is when the test participant and the conversion optimization specialist are in separate environments and interact by using screen-sharing software or online remote usability vendors.
• The worst thing to do when trying to refine a product, despite its nature, is to ignore the need for testing.
• The data you can obtain through usability testing is unmatched. Usability testing uncovers issues, alludes to possibilities, and provides a roadmap for immediate necessary change.
• Remote usability testing has the main feature; participants and experimenters are in different locations.
• Participants don’t take their tasks in recreated environments like labs but in their natural surroundings where those tasks are presented through an online platform.

Advantages of Remote Usability Testing

• Location is no longer an issue: When conducting remote usability testing, the location is no longer an issue, especially if the participants are geographically dispersed.
• Diversity of participants: You can involve as many participants as possible from different geographical areas worldwide.
• Fewer expenses: Remote tests are generally less expensive than in-person tests in recruiting participants and evaluators.
• Non-sterile experience: It is also an excellent opportunity for you to see how participants use their machines to interact with your system in their natural environment.

Drawbacks Of Remote Usability Testing

• Limitation of documentation: Documentation is an issue in remote testing. The fact that you are remotely conducting a test makes it harder to follow up with participants’ activities in case you lose connection or something out of control occurs.
• Inability to fix the tasks and assist the participants: another frequent problem is adjusting your test or assisting your participants. Especially with unmoderated testing, there is nothing you can do before the test is finished.
• Inability to avoid disruptions: If any disruptions occur, there is no way to avoid them if you don’t control the environment.

Advantages Of In-Person Usability Testing

• Non-verbal communication: One significant upside of in-person testing is enabling the experimenter to witness the participants’ body language.
• Tracking equipment: Another advantage is the use of high-tech equipment attached to the user to track eye movements, brain activity, and pulse activity. Each of those areas can provide other insights and information depending on the complexity of the test.
• Empirical research/data: In-person testing provides you the flexibility to control the environment without any disruptions or issues to ensure the validity of the results.

Drawbacks Of In-Person Usability Testing

• Costs: In-person testing can be tremendously expensive, mainly if you have limited resources.
• Sterility: sterile environment may have its advantages; the fact that it’s far removed from what a user is used to can impact how they behave during the usability test as well as the results of the test.

Types Of Remote Usability Testing

• There are two main categories, moderated and unmoderated (however, within moderated, the level of moderation varies from one type to the next).
• Moderated remote testing is the most similar to in-person testing in that both participants and experimenters are online simultaneously, and the experimenter witnesses what the participants are doing.
• The participants complete unmoderated testing sessions alone, usually via specific services or apps. The experimenter inputs tasks and questions in advance and either ask the participant to speak out loud or not, depending on the specific insights they are seeking.

General Guidelines For Conducting Remote Testing

• The goal
• Design your test protocol ahead
• Test your technology
• Be available
• Recruit additional participants
• Keep your tasks simple

Here’s A Longer And More Detailed Version Of The Article.

The worse thing to do when trying to refine a product, despite its nature, is to ignore the need for testing and rely on guess work under the pretense of scarcity of resources, time limitation, lack of knowledge or whatever your excuse might be.

Regardless that many think of usability testing as hill climbing, instead of seeing it as a solution, because there is much unknown, and they may lack experience.

The bottom line is that the data you can attain through usability studies is unmatched. Usability testing uncovers issues, alludes to possibilities, and provides a roadmap for immediate necessary changes.

We can all agree that usability testing can be extremely overwhelming and expensive sometimes, especially in-person testing. Not everyone can afford a fancy full-blown lab, expensive equipment, or even a dedicated room for testing.

On the flip side, remote testing has proven over the past years to be easy and cost effective with tremendously satisfying results and it changed the way marketers perceive and think about testing.

In this article, you will come across some of the affordable remote usability testing methods, mainly moderated and unmoderated remote testing, the PURE method, and heuristic evaluation.

But before diving in, let’s look at the pros and cons of in-person and remote usability testing.

Remote vs In-Person Usability Testing

Remote usability testing is a usability assessment with a main feature: participants and experimenters are in two different locations.

Instead of completing tasks in recreated environment such as a dedicated lab, participants ultimately take the tasks in their natural surroundings where those tasks are presented through an online platform.

Advantages of Remote Usability Testing

• Location is no longer an issue: When conducting a remote usability testing, the location is no longer an issue, especially if the participants are geographically dispersed.
• Diversity of participants: You can involve as many participants as you wish and from different geographical areas all over the world. You can, this way, have a larger pool of participants and end up with more heterogeneous groups for more insights.
• Less expenses: Remote tests are generally less expensive than in-person tests, in terms of recruiting participants and evaluators. You can easily find your participants online.
• Non-sterile experience: It is also a great opportunity for you to see how participants use their own machines to interact with your system in their natural environment.

Drawbacks of Remote Usability Testing

And because there is no such thing as a perfect test, remote testing poses some challenges as well which can affect the process.

• Limitation of documentation: Documentation is an issue in remote testing. The fact that you are remotely conducting a test makes it harder to follow up with participants’ activities in case you lose connection or something out of control occurs.
• The inability to fix the tasks and assist the participants: another frequent problem is adjusting your test or assisting your participants. Especially with unmoderated testing, there is nothing you can do before the test is finished.
• Inability to avoid disruptions: If there are any disruptions that occur, there is no way to avoid it if you don’t control the environment.

In-person testing remains a corner stone in usability testing.

But for many experimenters, conducting it is just logistically impractical and a burden for their limited budget and time.

Advantages of In-Person Usability Testing

• Non-verbal communication: One of the major upsides of in-person testing is enabling the experimenter witnessing the participants’ body language. You can derive great amounts of input from participants’ facial expressions aggregated with actions and words when having them think out loud.
• Tracking equipment: Another advantage is the use of high tech equipment that is attached to the user to track eye movements, brain activity, pulse activity. Each of those areas can provide with other insights and information depending on the complexity of the test.
• Empirical research/data: In-person testing provides you the flexibility to control the environment without any disruptions or issues to ensure the validity of the results.

Drawbacks of In-Person Usability Testing

• Costs: In-person testing can be tremendously expensive, especially if you have limited resources. Some experimenters drop the idea of testing as they face the need for recruiting participants, setting up a full-blown testing lab, trying to imitate the natural environment and to help participants act naturally and feel comfortable.
• Sterility: While sterile environment may have its advantages, the fact that it’s far removed than what a user is used to can impact how they behave during the usability test as well as the results of the test.

What Is Remote Usability Testing

Just like in-person testing, there are several ways to conduct remote testing. The two main categories include moderated and unmoderated (however within moderated, the level of moderation varies from one type to the next).

Moderated remote testing is the most similar to in-person testing, in that both participants and experimenters are online simultaneously and the experimenter witnesses what the participants are doing. As mentioned above, depending on the type of testing and task organization, the level of prompts and moderations varies.

You can always resort to this kind of testing when there is a need to develop an understanding for a certain issue.

On top of low cost and faster processes, moderated remote testing also provides on-going communication between experimenters and participants through an online platform, allowing the experimenters to ask for clarification even when the task is completed.

On the other hand, unmoderated testing sessions are completed by the participants alone, usually via specific services or apps. The experimenter inputs tasks and questions in advance, and either asks the participant to speak out loud or not, depending on the specific insights they are seeking.

There is no opportunity for the experimenters to ask questions or have any sort of involvement during the task even if needed. Keep in mind that this kind of testing is best in getting large inputs and volumes of data.

The more specific your questions and tasks are, the more effective those tests become.

It is also recommended to include qualitative research questions when possible and analyze self-reported questions in a way to compensate somehow the gap left from conducting a remote test.

General Guidelines for Conducting Remote Testing

When developing a remote usability test, be it moderated or unmoderated, keep in mind a few guidelines:

• The goal: Like any type of testing or research, you must first identify the goal behind it. Ask yourself, what are you trying to accomplish by conducting this usability study? Having this clearly defined will help you create more to the point tasks.
• Design your test protocol ahead: Determine the tool or tool(s) that you are going to use during the test, make sure that your participants have free access to it.
• Test your technology: Ensure that you have chosen the adequate and compatible tool and platform for conducting your tests. Experimenters frequently select a tool that participants are unwilling or unable to download. Also make sure that whatever you are using to conduct these tests is accessible for your participants and they actually know how to use it.

It’s always advisable to run a pilot test to examine your technologies and the tools through mocking the real test and its circumstances, to prepare yourself for any occurrences during the actual test.

• Be available: Always be available to provide any assistance if needed, either through email, by phone, or any other means.
• Recruit additional participants: Lack of communication with participants may rise their frustration, which may lead to their drop out at some point in the task, that’s why it is to recruit additional ones.
• Keep your tasks simple: Keep the tasks focused and simple to keep your participants engaged as much as possible. Take into account the task duration. Remember also that in total, the tasks not exceed 15 to 30 minutes which approximately translates into 3 to 5 tasks.

Remote Usability Testing of a Website: A Case Study

This section brings to you a remote usability test ran by our company.

The purpose behind this test was to evaluate visitors’ engagement on site to eventually convert them into actual buyers of an online store that features a range of memorable gifts and home décor items.

And because the test was supposed to be a live-intercept study, the recruitment of the participants was based on two different metrics:

• They should be interested in the content of the website
• They should be real users of the site

After qualifying the users, our team recruited five participants. The study was conducted remotely on desktop version of the website. Three participants used “Chrome” and two of them used “Safari.”

As for the tools, our team of experimenters chose the “Zoom” app, and provided participants with necessary instructions on how to use the tool. Experimenters also asked participants to use GoToMeeting so we could capture video of their facial movements.

The goal of the experiment: To test navigation and ease of locating items as well as up-sells on the site.

Tasks:

1. Please make sure you speak as you go through the site experience.
2. Browse a specific artist category to locate a popular item on the site.
3. Add an up-sell that it is attached to the product.
4. Finally, add another item within that same artist category.

The Steps for Conducting The Test :

1. Starting a “Zoom” session on the experimenter’s computer.
2. Give instructions to the participants over the phone to help them join “Zoom” session.
3. Making the participant the presenter, to see their screens.
4. Then start the recording in Zoom.

The Test Results:

The usability testing revealed the following problems:

1. Top navigation taxonomy was not clear to users
2. Users showed blindness for left navigation on category pages
3. Users could not locate up-sells easily
4. Users could not locate product details easily to verify if products were compatible
5. There was a small “Add to cart” popup misalignment

Concluding remarks:

• Users spent a lot more time browsing the website.
• The first task can account for the length of time and browsing time spent as demonstrated in the graph below.

Other forms of remote testing

The PURE Method

If remote testing is not an option, there are other methods that you can employ to gather uncover usability options.

The Pragmatic Usability Rating by Experts, or the PURE method, is another testing method that you can conduct from behind your desk office.

According to Christian Rohrer, the PURE method is used to estimate the amount of “friction” a typical user is likely to experience when using an interface. This testing is usually conducted by experts.

The user’s friction with an interface is measured through a preset scoring rate that each expert uses. The results are combined together into a final score.

In order to make the scoring easy and consistent, Rohrer came up with colored bars heights. Each task has a series of bars. Each color has a significance.

One of the great features of this methodology is that you can compare the scores of the same task within different product versions and keep recordings of the scoring.

Using the PURE Method to Evaluate a Website: A Case Study

For our preliminary evaluation, we tend to conduct a quick PURE evaluation to see if the website needs any quick fixes before digging any deeper for more.

Our last evaluation involved a governmental website that has a large online visitors base. Our main goal was to see if the average visitor could walk through the happy paths without difficulties.

We were 4 experts rating the tasks and two observed in the room to take notes and witness our body language and frustration mostly.

The tasks were the following:

Scenario:

You will be traveling to another country on August 1^st, you would like to obtain a visa. You login to the website to check the following:

5 Fundamental Tasks:

1. Determine if you need a visa to visit this country
2. Determine the visa requirements to this country
3. Determine the length of time it takes to process the visa
4. Determine how much does it cost to get the visa
5. Place an order for a visa

Results of the study:

At the end of the study the team walked through the tasks again and discussed why each expert rated a task a certain way. If there are varying ratings, the team discusses until they reach a consensus on a rating.

In this example, the expert panel concluded that most of the tasks were difficult to accomplish by the average user as demonstrated by the final consensus scoring below:

Heuristic Evaluation

One of the greatest features of heuristic evaluation is that you can do it yourself from behind your desk office without the need to worry about recruiting participants.

Heuristic evaluation may be labeled as a remote usability test because it’s based on inferences made by a usability expert, with the objective to inspect and investigate usability using well-known heuristics.

Bear in mind that heuristic evaluation mostly identifies issues different from the ones identified by other usability testing methodologies and you can always use it to back up your research when needed.

The fact that heuristic evaluation can provide some quick and relatively inexpensive feedback, makes it one of most used methods by most of experimenters.

The knowledge and experience of reviewers must be taken into consideration because they can affect the findings drastically.

Severity rating in heuristic evaluation:

0: No problem at all

1: Cosmetic issues only

2: Minor problems are present from some users

3: Major problems are present

4: Catastrophic and nearly unusable for all users, imperative to fix

There are three fundamental steps to conduct a heuristic evaluation:

The Preparation phase:

__  Agree on who will do the review

__  Determine the key principles or guidelines of the evaluation

__  Select the key content areas for evaluation that will cover a wide range of evaluation type.

The Execution phase:

For each heuristic provide as severity rating from 0 to 4, through determining how severe each problem is, either individually where each evaluator gives their own rating or as a group

The Consolidation phase:

__  Discuss the aggregated findings and point out the problem areas; these problems can be either details or it can be something that had to deal with a more general pain point.

__  Look for patterns

__  Suggest appropriate recommendations, testing ideas and allocate the appropriate resources to fix those problems through realistic solutions.

A Case study of Heuristic Evaluation by Invesp Team

Our usability team conducted a heuristic evaluation for one of the largest clothing companies in Asia.

As a preliminary stage, the team began the assessment assigning quick fixes, potential research questions, or research opportunities. A quick fix means that there is a broken heuristic or framework element that needs to be fixed immediately. A research question is something that must be investigated further through analysis. A research opportunity is something that can be tested through usability, A/B or Multivariate testing.

In order to conduct this walkthrough, the team first assess the different pathways and scenarios that would bring a visitor to the site. Then based on each scenario the team lists “the fundamental tasks” that the visitor would conduct in order to accomplish the goal that brought them to the site to begin with.

The rationale behind making a list of the best tasks is to check how well, or bad, the website does against that list while we are performing them on the screen, and find more holes and to define additional testing ideas and fixes.

The image below, shows some of the fixes recommended by our team for the desktop version of the site.

And here is an example of some of the fixes carried out at the level of the product page.

Original Product Page Design

• The price of the product did not update as visitors added or changed the customization options.
• The website unique value proposition was not emphasized on the product pages.
• The product customizations were not explained well through copy.
• The social media icons placement was a distraction from the page primary conversion goal.

The New Product Page Design

Results after some fixes:

• When visitors clicked on the different product options, the product price updated accordingly.
• A new emblem appeared on the product image, focusing on the custom tailoring offered by the website (a unique value proposition).
• Social media share icons were removed.
• New copy was added around the product customization

The amount of issues uncovered during the heuristic alone was tremendous. Testing just the above ones listed within the PDP resulted in a 34% uplift in conversions.

Over to you

The idea of user testing has changed and evolved over time. The availability/scarcity of resources are major factors contributing to these changes.

Remote testing addresses the scarcity problem and offers other advantages, as a larger and diversified audience, the ability to gather great amount of input for a deep, and a better understanding of your users and the market in general.

There is no perfect method in the UX world. The method you choose always depends on the goal you want to achieve on your test. Tweaking certain aspects of these testing and evaluation methods while maintaining the integrity and methodology was critical to the success of our UX team and processes.

Ayat Shukairy

My name is Ayat Shukairy, and I’m a co-founder and CCO at Invesp. Here’s a little more about me: At the very beginning of my career, I worked on countless high-profile e-commerce projects, helping diverse organizations optimize website copy. I realized, that although the copy was great and was generating more foot traffic, many of the sites performed poorly because of usability and design issues.

See Full Bio