Customers don’t think like we do. And similarly, what we may think looks or sounds right for our company and website, or what may be a primary motivator of visitors, could (and is most likely) completely off.
Elizabeth Wellington explains;
“Customers give information that reveals their attitudes (how they think they should behave) rather than behaviors (how they actually make decisions)”
But conversions and the path that leads to those conversions is highly driven by the customer. The image of a funnel is what comes to mind when many think of conversions, however, because the customer is irrational and highly distracted, it looks more like this:
There are ways of conducting qualitative research to gather customer data and understand more about what they are thinking, what is motivating them.
How To Determine When To Do A Qualitative Research
- Little is known or present understanding is inadequate
- Making sense of complex situations or social processes
- Learn from participants about their experiences (beliefs, motivations, opinions)
- Construct a hypothesis/theory from data
- Understand phenomena deeply and in detail
You’re essentially putting an organized framework on the messiness of life:
“Chaos often breeds life, when order breeds habit.” Henry Adams
But one must be wearied with some of the results because surveys and polls aren’t always accurate. Results of those qualitative studies aren’t always accurate. There is a science in the way that you ask and how you ask it (the method to use).
“Write neutral questions that don’t imply particular answers or give away your expectations.” Keep it short. Every extra question reduces your response rate, decreases validity, and makes all your results suspect.”
Qualitative research participants often answer something, but there subconscious and actions do something entirely different. Many studies have been conducted on how faulty relying on qualitative data can be. Participants tend to say the first thing that comes to mind or what they think they want. But you dig deeper to realize that there’s a lot more than you must be uncovered, and what they think they want or why they think they choose is a product or service isn’t the true motivation.
So although it’s important to assume the majority of qualitative research participants aren’t maliciously telling untruths, it happens often enough to make you really think about why it’s happening and what you can do about it.
It’s also very possible that they simply don’t remember an answer. If you are asking a question about whether or not someone will buy from you again in the future (or subscribe to your service), you are likely to get a positive response that doesn’t mirror the true future.
These all point to one overwhelmingly clear pattern: a clearly poorly designed qualitative research method. Some questions shouldn’t be asked in that specific format, or the way the question is asked almost forces a specific kind of response, leading to a greater change of untruths. Growing up in the 1990’s and early 2000s, I had to fill plenty of forms and always got asked the question of my ethnicity. I could never figure out where I belonged exactly – in what group. It felt limiting and likely lead me to select what felt right but wasn’t actually right.
So let’s break it down – why do participants lie and what can you do about it?
- Participants care about appearances: When you ask for demographical details, income details, employment details, people hate appearing like they’re worse off than others.
- Participants want to stay socially viable: If there is something socially unacceptable, people will tend to tell untruths about the reality of their situation. Think about a topic like voting – it’s socially undesirable to be a citizen that simply does not vote or participate.
- Participants are sensitive too: Social and personal questions, including illegal behaviors, may lead participants to tell untruths about realities they deal with.
- Participants aren’t malicious and want to “help:” If they think that’s the answer the research wants to hear, they will give the answer they believe the research “needs” in order to please or help them.
- Participants are human which means greedy: If there is something in it for them, or they think there is, they will shape the answer to meet their needs.
Ok, this is a little disheartening, but qualitative research is still quite invaluable. The key is how to use it, using what method, how to conduct it and analyze it. But first…
The basics: what is qualitative research?
Image Source: qualitativeresearch
Elizabeth Wellington describes it well:
“In his book “How Customers Think: Essential Insights into the Mind of the Market,” Zaltman characterizes this part of the mind as a “wonderful, if messy, stew of memories, emotions, thoughts, and other cognitive processes we’re not aware of or that we can’t articulate.” As customers, we make decisions from that convoluted (and unexamined) place. Without the amygdalae — the instinctual part of our brain that generates emotions — we wouldn’t be able to make any decisions at all.”
It’s a not a single approach, but rather a very broad systematic collection and organization and interpretation of textual information.
There is no one way to conduct qualitative research. It is definitely complicated. The wrong question be it open ended questions or close-ended questions can lead to useless data. It definitely requires much organization and categorization. And finally, qualitative data requires deep interpretation and many methods of doing that, one of which is textual analysis. A well-formed open ended question is the key to good research and often leads to more research and more questions. When you consider what a good research question is, consider that it should include the following qualities:
- It is inductive, exploratory.
- It’s impartial in the statement, and there are no underlying hints given
- Careful focusing on a single idea, don’t overdo it by asking too much
How To Conduct Qualitative Research
Image Source: qualitativeresearchprocess
1. Set a goal
Never begin a qualitative research activity without having oriented yourself with what it is you are trying to learn and achieve. Otherwise, you will not be able to formulate the right question, nor come up with the correct mechanism and/or setting to conduct your research.
If you know your goal, and you are familiar with the different methods of conducting qualitative research and the application of those methods; which will be covered later in this blog, you will have a better outcome with the research. Ultimately whatever method of research you select based on your goal, will help identify what processes must be followed for the best results.
Finally, part of your goals should include who the participants that you are targeting are. Throwing a wide net to all visitors of a website is less meaningful then selecting a segment depending on the question(s) being asked.
2. Consider the Outcomes
There are few possible products or outcomes that can come out of qualitative research. The outcomes are highly dependent on the type of qualitative research you conduct as well. Some of the outcomes you can expect include:
- Recurrent themes and hypotheses
- Survey instrument measures
- Taxonomies
- Conceptual models (theories)
“Not everything that can be counted counts; not everything that counts can be counted.” Albert Einstein
One thing to understand about qualitative research is it’s almost never about the numbers, but rather the overall meaning. You’re essentially capturing a specific aspect of social or psychological life.
According to Priscilla Esser and Ditte Mortensen:
“When you’re sharing results from qualitative user research efforts, you’re most likely focusing on creating an understanding for the lives people lead, the tasks that they need to fulfill, and the interactions they must effect so as to achieve what they need or want to do.”
So, what does that mean for you? Well for one, there are some important considerations to keep in mind, which will better help you interpret the data:
Don’t expect a single answer. You will always get a multitude of answers that you must decipher. Looking at the data, don’t look at it from a quantitative analysis perspective because you’re looking for quantitative data, because it’s not like that. You’re merely telling a story. Your story may not be entirely true; however, it MUST be: plausible, and coherent, and if you want some truth behind it and a stronger case: quantitative data to back it up.
3. Know the context to better understand the responses
A key area to think about when it comes to qualitative data is that it’s relative.
“Truth can be compelling without being absolute.” Remember ultimately that many truths are relative, especially considering qualitative data.
So, you need to understand and think about context when you conduct and analyze the data coming from qualitative research.
Context includes:
Source of traffic – understand the journey that brought them to your site will help you understand why they answer what they answer. A great place to start is by analyzing the sources of traffic that brought these visitors to your site in the first place. That way you can understand their journey that much better.
Bias – There’s no way around it, qualitative data is tainted with bias. While quantitative data enjoys an absolute truth because numbers are numbers, qualitative data is quite the opposite. Trying to gauge human bias in your sample group is important for context.
Subjectivity – we always say as a golden rule to optimization is that you want to get from subjectivity to objectivity and data-driven changes. However, it’s important to recognize that qualitative data is quite subjective during the analysis process. That’s why you want to try and pain some of that qual data with analytics or other qual data to provide objectivity to a particular issue you uncovered.
4. Eliminate Researcher Bias
You’re biased. Everyone is. As a researcher, however, this bias can stand in the way of asking the right questions and seeing other perspectives. So, our recommendation is to have some sort of a questionnaire or checklist before going into the research to ensure that you practice good reflexivity:
- Consider 5 factors how participants will view you (presumptions)? Because you are the company asking, how will that affect the answers they give?
- Then consider the 5 factors that impact the way you analyze data (as a researcher). Consider what assumptions you, as the researcher, have about life, people, website or business, and how those factors may shape your reading of the data.
That brings us to two important terms that everyone who conducts qualitative research should know: Ontology and Epistemology.
Ontology refers to what is nature of the research vs. the reality. Ultimately, the researcher’s philosophical assumptions about the nature of reality.
Epistemology refers to what is possible to know and how we can generate meaningful knowledge. So, the researcher’s assumptions on the best ways to inquire to find out the truth. Being aware of these two terms helps you understand that you too have a bias. Knowing it and understanding what will help combat it can give allow for a better chance to ensure your research will not be polluted.
Knowing these two terms will help you construct “3 fundamental questions to your research:
The ontological question: What is the form and nature of reality?
The epistemological question: What is the basic belief about knowledge (what can be known)?
The methodological question: How can the research go about finding what she believes can be known?” (Guba, Lincoln 1994)
5. Beware of subjectivity
Ultimately, researchers often fall into the mistake of subjecting histories, assumptions, and values into research the research they conduct. This is why as researchers we must be consciously practicing reflexivity.
Reflexivity refers to the reflective process in which a researcher considers how the findings were produced, taking into consideration the prior knowledge, and our role in it.
6. Understand the Who?
The second most important consideration is the Who of your research. While quantitative research often samples large groups of people randomly, there is a lot more purpose and specificity to the selection of smaller samples for qualitative research. Because the samples are smaller, they must be less random.
Image source: Slideshare
Data Sampling: There are many ways of sampling data, however, for the purpose of this article, we will talk about two ways:
- Probability sampling: This is selecting a truly random, yet statistically representative sample that can be generalized to a better understanding of the greater population (market) This is used in quantitative data collection.
- Purposeful sampling: selecting “information-rich cases” (Patton 1990) for deeper research. From these smaller, yet sought-out samples, you can learn a great deal about issues that are central to the research. The purpose is to “select information-rich cases whose study will illuminate the question under study.” (Patton) This is used in qualitative data collection.
Determine key informants: Again, the aim is at being purposeful when it comes to sampling. Seek out, specific individuals.
Breadth not representatives: You aren’t looking for general public, yet people with the extreme circumstance that make them eligible for the study. For instance, I’m looking to understand what makes individuals loyal to my brand. The average loyal customer shops 2 – 3 times a year. I would seek out those customers who shop over 8 or 9 times a year.
Sample size
- depends on the complexity of inquiry and type of qual research (cannot be clearly determined in advance)
- determined by “theoretical saturation (a point at which no new concepts emerge from data)”
7. Select the right Qualitative Research method
Image Source: methodsuxdesign
Qualitative research involves different formats and methods, each with a specific use and aim.
The methods include face-to-face or phone interviews, focus groups, observation (natural settings), textual (polls, surveys), listening to audio recordings of conversations between customer support and customers.
“Qualitative methods such as interviews, ethnographic field studies, and (to some degree) usability tests are often more exploratory and seek to get a more in-depth understanding of the individual users’ or user group’s experiences, motivations, and everyday lives.”
Interviews
Interviews should be considered as “conversations with a purpose.” There are two ways to conducting interviews. Face-to-face or via phone. While in today’s digital world, people are less likely to conduct face-face, the chances of accuracy are always less likely when you can’t view the person and their expressions when they answer.
Topics and issues to be covered are specified in advance, in outline form; the interviewer decides sequence and the outline increases the comprehensiveness of the data and makes data collection somewhat systematic for each respondent. Logical gaps in Important and salient topics may be inadvertently omitted. Interviewer flexibility in sequencing and wording questions can result in the wording of questions in the course of the interview. data can be anticipated and closed. Interviews remain fairly conversational and situational. substantially different responses from different perspectives, thus reducing the comparability of responses.
Useful for:
- Individual perspectives and experiences
- Sensitive topics
- Situations where there is perceived danger of reprisal
- Topics that cannot be investigated through surveys
- Gathering in-depth information about a topic
Focus Groups
When we think focus groups, a lot of times, the picture of a group of people evaluating some sort of product that is going to be rolled out pops into mind.
According to the Marketing Research Association,
“A focus group is the meeting of a small group of individuals who are guided through a discussion by a trained moderator (or consultant). The goal of the focus group is to get beyond superficial answers and uncover insights on consumer attitudes and behavior.”
The key behind focus groups is trying to generate narrative data in a focused discussion. The nature of this format is about the group dynamics offering a wide range of perspectives and views on a common experience. Because many people are participating, it helps to activate forgotten details. It’s also revealing in the way people speak about issues.
Useful for:
- Characterizing social and cultural norms
- Sharing and comparing
- Revealing how people talk about an issue
- Exploring sensitive topics
Observation
Observation is defined as the recording of the behavior of the sample. Many usability experts use observation and eye-tracking tools when they conduct a study online to specifically gauge more of the participant’s facial and body language. Although participants may say they weren’t frustrated by a feature or experience post the observation, their body language, and facial expressions tell a different story. It is best used when applied to visitors in their natural environments. For instance, if a company that offers teachers classroom management tool wants to understand more about how and when teachers use their product answering questions about the agility of the software, they would send a team to classrooms using the software and observe over a number of days how the teacher interacts with it.
Useful for:
- Individual perspectives and experiences
- Difficult to gauge through interview experiences
- Complimenting usability studies
- Topics that cannot be investigated through surveys
- Gathering in-depth information about a topic
Textual (polls and surveys)
The go-to for most marketers because it’s easiest and least invasive. However, with all of the positives, the responses you can be getting about certain questions may need to be taken with a big spoonful of salt. Of course, if you know how to use polls and surveys, and what type of questions work with this format, it can prove useful. What’s also key to polls and surveys is first identifying, of course, the goal, but then the segment that would best provide you with the answers that have the greatest specificity and accuracy.
Useful for:
- Specific segment perspectives and experiences
- Broad topics
- Situations where participants can feel free to criticize
- Demographical data
Keep your online surveys as short and concise
The vast majority of the customers have a tendency to avoid long surveys. People do not have the patience nor the time to click through and answer any questions. There’s a higher likelihood that people will fill out short and clear surveys. From my own experience, if I see a question to rate an experience, for example, where I need to click through in order to get to another screen where the question is listed, I will likely opt out. Let it be easy and visible from the outset to increase response rate.
Image source: Econsultancy
Response rate
For qualitative data, the response is more heavily focused on the quality, and not the quantity. For surveys and polls, there is a possibility to tap into a greater number of participants. However, the aim should not be solely focused on the number of respondents, but rather if you were able to get quality responses from the specific segment you were after.
That being said, with some kinds of polls and surveys (especially quality or experience assurance or NPS) you need to factor quantity as well. There are several factors that determine the validity of a survey. Population size, margin of error, confidence level, sample size, and response rate.
The Response rate: Number of valid responses / total number of people invited to respond. When you make an investment about anything, you expect high ROI (return on investment) of a project right? The response rate is the ROI of a survey. You need high response rate to be sure that you are collecting valid data.
Checkout 34 ways to improve your survey response rate:
By implementing some of the above strategies, you can improve your response rates from 30% to 55%.
PeoplePulse, IBM’s innovation center, ran B2B online surveys from 2000 to 2001. After some changes in their survey structure, they managed to double their response rate. The two variables they tested on the surveys were offering incentives and shortening the survey. The incentive increased response rates 10-15%.
8. Asking the right questions
Your questions will vary depending on the goal and format you will be conducting the qualitative research. Whether it’s interviews, ethnographic research, focus groups, or surveys, questions must be identified clearly before beginning the process.
You’ve considered your goal, thought about the biases that may pollute your reading of the responses and consider who it is you are asking: your job now is to think about the questions.
When it comes to asking research questions, a lot of bias towards the “answer you expect” can sway your approach to constructing sound, unbiased questions.
Here are some tips to consider:
Use a verb: Remember, your role is to ask them to describe, characterize, understand, consider. That line of questioning will put the ball in their court.
Identify the topic of interest: Be clear about what the actual topic is first, then ask the question, in order to give them context.
Non-directional neutral language: Remember, many times we inject our personal feelings into a question which will, in turn, create a bias in the responses.
Define the sample and setting: How will you conduct your research – what method? The question may differ based on those decisions. A statement of why you are conducting the research is helpful at this point: like the goal is to XXX.
As precise as possible: Most of the work happens in the question construction phase, if you rush and create a haphazard question, you are likely to get haphazard answers that add no value to your research.
Samples:
To explore the views of men who read XXX skin research blog
To understand XXX visitor views on XXX
To describe and classify the factors that determine motivations of visitors for xxx
To understand the knowledge, attitudes, beliefs, preferences, and barriers regarding XXX amongst XX visitors
To characterize barriers to XXX among male/female visitors, who are currently enrolled, registered, new, returning, oligarchs,
Mass communication theory has some good ideas on how to structure the questions you will be asking.
They recommend formatting it into what I would describe as almost a data tree structure.
- The Research Problem
- Goal: or qualitative purpose statement
- Determine method of research that is best suited to achieve the goals
- The research questions (that narrow the purpose):
- Central
- Sub-questions
To hone in on the right questions, Mass Communication Theory suggests using the following scripts to identify the relation between the question and the qualitative purpose statement:
“Central question script (use only one):
“What does it mean to _________________ (central phenomenon)?”
“How would ______________ (participants) describe (central phenomenon)?”
Sub-question script: “What _________ (aspect) does __________ (participant) engage in as a _____________ (central phenomenon)?”
Other tips for asking questions
Take a look at a chart NN Group mentions how to ask open-ended questions:
Here are the possible questions to be asked upon buying process:
- What matters to you when buying product/service?
- How long did the buying process take on average? Do you think it is short/long?
- What are we doing worse/better compared to our competitors?
- Which hesitations did you have before buying? Are there still any, please clarify?
- What made you buy from us?
- What would you recommend us to better improve our product/service for next time?
“When surveying your customers, you don’t want to just survey anyone and everyone. The best way to do it is to survey only recent customers because they won’t hold a biased opinion, and their experiences with you are still fresh. If possible, survey them immediately after they purchase. The faster you survey post-purchase customers, the more accurate your results will be.”
9. Conduct Qualitative Data analysis
The part of digital marketing that always baffles me is the amount of data available for analysis but left unused. There is so much effort that goes into the creation of research, but by the time it’s said and done, marketers feel exhausted and don’t pay close attention to what the results are really saying.
The key to conducting qualitative data analysis is to ensure there is sufficient, quality data. Looking at that data, there are several ways to analyze it. You will have likely considered the analysis needed for your specific qualitative research method previous to receiving the results.
Thematic analysis: identifying themes and patterns of meaning across a dataset in relation to a research question
Grounded theory: Questions about social and/or psychological processes; focus on building theory from data.
Interpretative phenomenological analysis: Analysis aimed at seeking insights into the meanings that events and experiences hold for people.
Discourse analysis: Analysis of understanding more about how certain terms are being used. What is said and why it might be said?
Narrative analysis: Analysis based on seeking greater understanding of unique perspectives brought on by individuals. It is based on seeing how individuals make meaning using stories. How these individuals make sense of their external and internal world.
Types of qualitative research methods in UX design
1. Grounded theory.
This systematic methodology was developed by sociologists Glaser and Strauss in the 1960s.
Grounded theory is a qualitative research method in which researchers come up with theories and hypotheses that are grounded in empirical data/observations.
Instead of researchers developing a theory before examining data, grounded theory works the other way around. Researchers come up with theories after examining the data and finding patterns along with categories.
2. Card sorting.
Card sorting typically presents users with a series of categories or topics written on cards and invites them to organize them in such a way that makes sense to them. Card sorting is a great way to understand what might be intuitive to users and what web or platform design structure users expect. It is also a relatively quick, easy, and cheap method to execute.
3. Diary studies.
By getting users to log their habits over time, diary studies offer a research approach that yields insightful data that can then help businesses understand users’ daily habits, needs, and behaviors. It also meets users’ where they are—having them document their day as it’s happening instead of asking for them to recount an interaction after the fact.
4. Qualitative data analysis.
Also known as thematic analysis, this method involves analyzing qualitative data for themes that can help answer a research question or find meaning within a data set. The results of a thematic analysis can influence design decisions by helping developers zero in on user needs.
Wrapping Up
Qualitative research is a type of research in the social sciences that focuses on non-numerical data as opposed to quantitative data that’s full of numbers.
To get the best from conducting any qualitative research, you need to focus on the quality of answers, not the number of respondents.
Quantitative data is focused on how large the sample size is; qualitative data is focused on the quality of the information from a smaller sample size.
Also, don’t limit yourself to one type of qualitative research method; employ many, like ethnographic research methods, interviews, focus groups, textual analysis, etc.
Whoever said qualitative data was easy or required little precision or attention to detail are the same people that sold AB testing as an easy-to-use tool that any marketer can simply apply to their activities. It is precise, scientific, and driven by knowing who you are targeting, being conscious of your biases, and constructing sound problem statements, goals, and questions.
Additional Resources
1. Open-ended questions and close-ended questions: What they’re and how they affect user research.
2. The Paradox Of User Behavior In Web Design: Novel vs. Routine Tasks
3. A Guide To UX Data Analysis For Actionable Insights
