As Optimizers, we are obsessed with the idea of giving visitors a great user experience. Even when we open a random website, we have this tendency of looking for anything that can impede users from having a great experience on that site.
We conduct research, take months gathering data using different CRO techniques and we spend weeks categorizing, classifying and organizing the data into valuable insights.
The data we gather can either be quantitative or qualitative, but regardless of the type, it’s important that we analyze it so that we know how to communicate the findings in a way that is understandable to clients.
Data analysis determines the success or failure of research projects. You can use the right research techniques and monitor the right KPIs, but if you fail to answer the why question in the analysis process, your research might just be a wasted effort.
In all honesty, data analysis is tricky, even for the best and brightest of us. There are infinite ways to interpret data, too many metrics to wade through and it’s easy to get caught up in some metrics and to ignore some important ones.
Yes, correct data analysis can be tricky to master but fortunately, it’s not impossible.
Most people think that analysis in research is done when the data has been collected. But that’s wrong. Research analysis should start right before the ultimate research commences —the best starting point would be when you are designing your research objectives.
Analysis at the beginning of the research will help you define your goals, select a sample size and collect data in predefined themes that are aligned to your research goals. It also leaves room for periodic analysis, and not waiting until the end of the research to analyze the collected data.
In this article, we will assume that the data has already been gathered, so we won’t focus on the planning and facilitation part of the research, but on the analysis and interpretation of the data. We c
Let’s get straight into it.
UX Data Analysis, what is it?
UX data analysis is a fancy name for a process of transforming raw data into valuable information. With data, there is always more than meets the eye, hence the process of data analysis. Considering that it is a process that helps us to make informed decisions that can improve the user experience, data analysis is a vital part of any type — qualitative or quantitative — of research.
In every data analysis, the ultimate “actionable” takeaways are completely subjective. If you take 5 Digital Marketers, give them the same raw data, and ask them for their thoughts, the analysis process can be the same but you will probably get 5 different actionable takeaways.
It takes effort to get to the right conclusion when analyzing data but as said by Ronald Coase
If you torture data long enough, it will confess to anything.
Data analysis is only as good as the questions you ask, but the primary question you should be looking to answer during the analysis process is: why. Ignoring this question can result in misleading findings that can lead your whole team down the path of solving a wrong problem.
Confirmation bias tends to get in the way of most Digital Marketers to the point that they forget to ask the primary question when analyzing data. We all have preconceived ideas and beliefs whenever we see something, but they should be mitigated as much as possible during the data analysis process.
Let’s say you are analyzing session replay videos, you can avoid falling to the confirmation bias trap by having some team members watch the same set of videos separately and then you compare your notes afterward.
Well, at Invesp, instead of typing, we prefer jotting down notes during the data analysis process because we are sold to the belief that note-taking using pen and paper helps you to focus on the most pertinent information during the analysis.
UX Data Analysis Steps
Now that you have gathered your raw data, it’s time to go from a mass of data into meaningful, actionable insights. However, the way you analyze your data depends on the research technique — quantitative and qualitative — used, but the steps you will have to follow are probably the same.
When gathering data, let’s say during customer interviews, you will get the general idea or feeling of what your users and what your results may look like, but once you start drilling down the data during the analysis, you are bound to be surprised by how much more information is contained in the raw data.
In order to get the meaning of data more easily, there are different methods of analysis that have been used by researchers over the years. These methods include:
- Coding and marking the underlying ideas in the data
- Grouping similar kinds of information together in categories
- Relating different ideas and themes to one another
So regardless of the kind of method you will use, there are certain steps you should follow when analyzing your user research data and for this process to be successful, you have to make sure that all the fundamentals are being followed. Here are the 5 key steps of data analysis:
- Define your analysis objectives
- Organize the data
- Mine the data
- Categorize your findings
- Identify insights
Having said that, here is a discussion around each step.
Step 1: Define your analysis objectives
Analysis objectives should be set at the very beginning of the analysis as they serve as a go-forward guardrail that will help you ensure that you gain useful and relevant insights. Your primary analysis objective summarizes the main reason you are conducting the analysis —why are users behaving in a certain way.
The analysis objectives should be set with your business goal in mind; the reason why you are conducting analysis in the first place.
You may be seeking to understand the needs of your users because you want to give them a better experience and keep them engaged on your site.
So whenever you are about to set your analysis objectives, ask yourself this question: What is it exactly that I want to learn from the analysis, and how will this help my business goal?
Step 2: Organize the Data
When doing your research, you are more likely to capture unwanted data. Not everything said by your users during your research process will be useful to you. So, the first step of analysis is to organize your data and remain with the exact slices that are in accordance with what you are trying to find out. Revisit your research objectives, and see which data relates to your goals and separate it from the chaff.
Let’s say you are analyzing session replay videos, the software you are using can record thousands of videos. But some of the videos captured can be irrelevant to the research objectives you have. Fortunately, with session replays you can easily organize your data (thus videos) by filtering according to the type of device, browser, page visited or duration —again, this depends on the research questions you are trying to answer.
Your data has to be organized in a way that is easy to look at, and that allows you to go through the information to pick out concepts and recurring themes.
Once your data has been organized, you can move onto the next step: picking out insights and organizing them into categories.
Step 3: Mine the data
Identifying salient themes, patterns and recurring ideas is probably the most intellectually challenging part of the analysis —and one that can make or break the entire endeavor. In order to be able to interpret and come up with feasible insights, you have to know what to observe during analysis.
Let’s say you have conducted customer interviews and you were taking notes or recording. So, when analyzing one particular question you may have asked your customers, look for specific words or ideas that are actually being repeated by your customers —and make notes of different ideas (keep a list) as you are going through different responses.
Analyzing user feedback doesn’t end with looking out for similar words. You also have to find meaning in the language that is being used by your users. I mean, users have their own way of expressing themselves when giving feedback, but you should pick out frequently used expressions and look at the meaning as well as the underlying implications of those expressions.
So, if you are analyzing your users’ statements — collected via online polls or customer interviews — it’s important not to alter or clean up the grammar they used. The translation has to be as true as the original speech and this means that if you were taking notes during your research, your notes should give the best reflection possible of how the conversation happened.
Step 4: Categorize your findings
Once you have you have identified frequently used words and phrases, it’s now time to organize these findings into categories. Categorizing can take little or much time, but important because it will lead you on the path to focus, prioritize and take action.
Your findings categories should be distinct or at least overlap as little as possible. You may also catch interesting statements you may have overlooked in the previous phase. In itself, this process of categorizing the findings is iterative, so this is why you should stop when you notice that you are not adding anything relevant anymore.
At Invesp, we usually classify our findings according to these categories:
Research Opportunity (RO)
Research Opportunity can be defined as a chance to gain more in-depth knowledge in a certain subject or area. But in this context, you can think of RO as items, identified during the data analysis process, that present an opportunity to increase conversions. Chances are extremely high that you can generate a Research Opportunity from the data analysis process.
For instance, let’s assume you conducted the Jobs-to-be-done interviews so as to find out the stages that customers go through when making a purchase decision for your product.
As part of your findings, you may discover that your customers’ main concern is the trust aspect of your product. So, in this case, trust symbols and testimonials can be regarded as items that can help you to increase conversions.
Investigate Further (IF)
The process of data analysis requires one to be patient because there are instances when one analysis is not enough. Before implementing the findings, sometimes you may need to have two or three further investigations to understand the behavior of visitors.
For instance, suppose you are running an A/B test on the product page and you trigger an exit-intent poll that asks users to specify what they found beneficial about the page. 20% of users may tell you that they like the design of the page and 10% may tell you that they like the images.
But in such a scenario, it should be further investigated which page design they are referring to between the two variations, and what is it about the design that makes them like the page.
Fix Right Away (FRA)
This is probably self-explanatory, right? Some of the findings you may discover during your analysis may require you to act immediately without further delay. I mean, if your usability test findings show that your site is broken or has other forms of usability issues, you should fix it right away.
Instrument (IT)
As part of our data analysis process, we also like to give a general comment on the website data or user behavior. You might not think of it as a necessity, but we believe that they are valuable contributions that provide guidance in regard to the insights obtained from the analysis process.
Step 5: Identify Insights
More often than not, findings and insights are two words that are used interchangeably. But they do not have the same meaning. You cannot base your analysis conclusions on findings, you need insights. Teri Slavik-Tsuyuki gives a sweet explanation of the difference between findings and insights:
Findings are based on hard and fast observations, and things we observe, that help us put all the data in buckets, and people in categories. But findings alone don’t lead to the why. They just call out and help us categorize the what. Insights are penetrating, discerning understandings that unlock an opportunity. They cut to the chase and get to the why. And finding them takes listening, quiet observation, and active participation.
With that said, it’s now time to take it to the next level and identify insights and core truths about your users from the patterns you have found.
Here are the steps you need to follow to identify insights:
- Discuss all the categorized findings as a team. Talk about why you think each category is important and how best can it help to optimize the user experience. Recall the exact words used by your users, facial expressions, hesitation to click, click rages and also the participants’ emotions and feelings.
- Articulate, in one simple statement, the insights that emerge out of each category. You can use post-it notes to write down the insights. But remember, these can be changed when you are revisiting them.
- After you are done articulating your insights in simple statements. Leave them and focus on another work just for a period of time or even for a day.
- Check them out again and let everyone in your team have a chance to re-articulate insights again. You should all take turns to think of different ways of articulating or expressing them. If by any chance you all concur with what you have written the first time, then it’s best not to change anything.
- Socialize your insights. Talk about them to some of your team members who were not part of the analysis. Give them some context, show them the insights and get their reactions. This will reveal to you whether the insights resonate. Do people get them? Do your insights speak to people as the truth? Can your design team create a meaningful product from the insights?
Conclusion
When analyzing UX data, you have to be open-minded so as to avoid missing some of the important findings that may help you come up with actionable insights. Is that the same approach you take when analyzing your UX data? If not, tell us about the approach you use in the comments section.
