So you have decided to conduct a usability test for your product —so as to understand your product from the users’ perspective. You then realize that you have to come up with relevant questions to ask participants.
“That sounds like a piece of cake,” some may presume.
Writing effective usability questions seems deceptively easy, but the harsh reality is it’s not as simple as you may think it is. There are quite a number of mistakes to avoid.
How you phrase your questions will directly impact the quality and value of your user testing results. Ask your participants the wrong questions, or use the wrong words to structure the questions, and you will gather incorrect feedback. Wrong questions can contaminate the whole research —leading to misleading quantitative data and qualitative data.
Before you come up with any usability questions, there is a great need for creative thinking, which involves answering the question – what you intend to learn from the test. Begin by asking yourself this: what information do I need from this research?
Taking time to answer that question will help you narrow down all the possible wrong directions you might head before getting to the qualitative data or quantitative data you need. The ultimate goal could be as simple as finding out if users will click on your search result listing.
When you have a clearly defined goal, writing usability questions isn’t a task that is hard to do.
By default, your usability questions could either be open-ended questions or close-ended questions.
Whenever we conduct any user research at Invesp, our conversations with our participants have a natural rhythm.
We ensure this by using open-ended questions and closed-ended questions in unison.
This article will explore open-ended questions and close-ended questions in great detail, focusing on how to write them and when to use them.
What are open-ended questions?
An open-ended question is what it is: a question that is open to any answer. In the context of user research, open-ended questions are questions that do not limit users to one- or two-word answers. Instead, they have multiple potential responses, and they often give room for further probing by the moderator.
Open-ended questions are versatile in nature, and they prompt users to describe their feelings and thoughts in their own voice. In this regard, the Digital Marketing Evangelist for Google, Avinash Kaushik, says:
The greatest nuggets of insights are in open ended questions because it is the Voice of the Customer speaking directly to you (not cookies and shopper_ids but customers).
There is something about asking open-ended questions that makes participants feel more comfortable during a usability test. People, in general, tend to open up and express themselves better when they are given room to answer in their own words.
For example, rather than asking, “Was the new feature easy to use?” You can try something like, “How would you describe your experience of using the new feature?”
The most common response to the first question would have been, “Yes, it was ” or “No, it wasn’t” and there was no way you could have understood the context behind the user’s response. But the second question allows users to respond freely, and there is a high chance of getting unique answers that you might not have anticipated.
Open-ended questions are ideal for starting and holding a conversation in any circle. They empower users, giving them 100% control of what they intend to say —and this is something that cannot be done using closed-ended questions.
Tips for writing effective open-ended questions.
Coming up with the right kind of questions requires practice. There’s an art to asking questions that prompts people to think before giving a response. Unless it’s an open-ended question, not every question you ask your users will motivate them to give a detailed answer. Here are a few tips that will help you write effective open-ended questions.
1. Begin your question with how, why, and what
What makes a question open-ended is the wording of the sentence. For an effective open-ended question, start the question with words such as how, what, why, and can. This way, you give your users freedom to say more, and in the process, there is a possibility of uncovering rich insights.
Avoid using more specific words such as did, would, which, when and was —these usually prompt one-worded answers.
For example, a question like “Which part of your experience was unsatisfactory?” does not evoke users to give in-depth details about what they found unsatisfactory with the application. Instead, you can elicit users to reflect on their experience and give a more insightful response if you ask it this way “What challenges did you face during your experience with XYZ?”.
Here are a couple more examples of how to attain a more precise answer by changing the wording of your question:
| Closed-ended Questions | Open-ended Questions |
| Was our product easy to use? | What do you think about our product? |
| Did you enjoy using our product? | How would you describe the experience you had using our product? |
| Did anything persuade you to use our product? | What persuaded you to use our product among other products? |
2. Clarity and Analysis
Ensure that your question requires users to be analytical and to clarify their points. One defining trait of an open-ended question is its ability to propel users to put more thought to their responses.
Analytical questions do not require users to generalize their answers. For instance, in a Conversion Optimization research, you might ask your participants these questions:
- What is the significance of a certain element in a website?
- How important is the new feature on the site?
- Why did you choose to use this service/product?
All these three questions are different, but they all have one thing in common: they would require participants to be clear and describe their answers in more detail.
By motivating users to give clarity in their responses, open-ended questions can also be used as a way of motivating their participants to verify their answers, especially when your previous question was closed-ended.
Suppose you ask this close-ended question: “Did you find the product you were looking for?” you can then verify the given answer by asking this follow-up open-ended question: “Why were you looking for that product?”
3. Avoiding leading participants into a certain answer
In any research approach, open-ended questions are asked so as to elicit valuable insights from users, not to confirm the moderator’s existing beliefs. So, if your questioning subtly prompts users to answer in a certain way or gives hints at the expected answer, then you need to revise the phrasing of your questions.
The wording of the question shouldn’t be suggestive of any answers to the participants as this biases the users into giving a predetermined answer.
Let’s say you ask this question: “Which feature made you visit our site?”
The problem with this question is that already suggests an answer for the users. It implies that it’s a feature that made users visit the site. Come to think of it, what if it wasn’t a feature but a service that lured the users to the site?
Examples Of Open-Ended Questions.
You can use these sample questions as conversation starters and to also make your participant explain more.
1. How satisfied or dissatisfied are you with this process?
2. What would (did) you expect to happen when you … ?
3. Did you find it?
4. How would this fit into your work?
5. How might this change the way you do that today?
6. What do you think about that?
7. What kinds of questions or difficulties have you had when doing this in the past?
8. What happened when you did this before?
9. Please describe your level of experience with …
10. What’s most confusing or annoying about … ?
11. What worked well for you?
12. How do you know … ?
13. How do you normally … ?
14. What just happened?
15. What was that?
16. What would you most want to change about … ?
17. Which things did you like the best about … ?
18. What were you expecting?
How Do You Ask Open-Ended Questions In UX Research?
Open-ended questions in UX research are used to gather qualitative data and gain insights into users’ thoughts, feelings, and behaviors.
These types of questions allow participants to provide detailed and unstructured responses rather than simply choosing from a set of pre-determined options.
To ask open-ended questions in UX research, you can use prompts such as:
- “Can you tell me about a time when you used a similar product or service?”
- “What do you like/dislike about the current design?”
- “How would you describe your experience using this feature?”
- “What would you change about this feature if you could?”
- “Can you walk me through your thought process as you complete this task?”
When asking open-ended questions, it is important to create a comfortable and non-threatening environment for participants and to listen and probe actively for more information as needed. Additionally, it is important to avoid leading questions and to keep the question open-ended to gather unbiased answers.
When To Use Open-Ended Questions.
In some situations, the only way to get valuable insights is to give respondents some sense of control over the conversation by allowing them to answer in their own words. There is a high chance of bumping into something completely unique and valuable if you allow users to have the freedom to express themselves.
1. In sales
SalesHacker made an interesting observation about interactive discussions prompted by open ended questions: when you have a conversation with your potential customers, and they talk for at least 30% of the time, your conversion sales will likely increase. But if they talk for less than 30% of the time, your sales conversion rates will drop.
With open-ended questions, you are not only guaranteed an increase in sales, but they can also help you:
- Explore the needs of your customers.
- Provide you with a better idea of what your customers think about your product.
- Foresee and minimize risks.
- Trigger a meaningful and insightful conversation with your customers.
- Discover new opportunities.
- They can play a significant role in building a good rapport with your customers.
However, not all open-ended questions are good. In an article written by Business 2 Community, they gave examples of some of the “bad open-ended questions” that won’t work well in sales:
- How much are you willing to spend?
- What is your worst pain?
- What kind of goods or services are you ready to pay for
- What don’t you like about our service?
- What don’t you like about our service?
2. Open-ended questions in conversion optimization research
The essence of an effective CRO program is not only based on getting tactics right and testing this and that, but it’s also about knowing the mindset of your customers. You first have to see your product or service from the customer’s lens to deliver a product or the services they desire.
One way of getting into the customer’s head is by asking open-ended questions.
In this regard, JeremySaid put out a handy list of questions you can ask your current customers when you intend to increase your conversion rate. Here are some of the open-ended questions they recommended:
- How was your overall experience?
- Why are you here today?
- What about this product/service struck out to you?
- What do you know about our company?
- What would you like to know about our company?
- What problems have you experienced in the past with similar products?
- What would you like to see us do more online?
The more you let your customers feel comfortable, the more they will reveal what drove them to consider purchasing your product. So, whether you are conducting a usability test, focus group, customer interviews, or surveys, keep your customers TALKİNG and use the information to your advantage. And the best way of doing this is by opening up the conversation.
What Are Closed-Ended Questions (With Examples)
If you can imagine a question restricting participants to a set of predefined answers, then that’s a close-ended question. It aims to get precise and clear-cut answers — without leaving any room for users to express themselves.
According to Wikipedia:
A close ended question refers to any question for which a researcher provides research participants with options from which to choose a response. Close ended questions are sometimes phrased as a statement which requires a response. A close ended question contrasts with an open ended question, which cannot easily be answered with specific information.
Asking closed-ended questions will give you specific answers, aka quantitative data. Do they want to purchase your product? Are they shopping around with your competitors for the same service? The answers are simple and direct.
Although close-ended questions provide limited insights, that doesn’t make them any less important. In most cases, close-ended questions are used in a quantitative research approach where insights gathered are numerical.
Close-ended questions have their place in user research, and they are wonderfully effective in guiding participants into giving certain answers.
For example, you conduct a usability test to determine if your app store listing will convert well. So you ask your participants this question:
Which of the following pieces of information made you download this app on the Play Store?
- App Icon
- Screenshots
- Customer reviews
- Price
From this example above, the question eliminates any element of surprise by setting boundaries for the participants’ responses. Participants are not expected to give an answer outside the set of predefined responses.
So, if close-ended questions do not require participants to express themselves, then isn’t that a disadvantage? Well, it is. Respondents are biased into responding in a certain way. But sometimes, it’s necessary to use these types of questions in user research as they make it easier and quicker for respondents to answer.
Although close-ended questions have different forms, they all have this in common: they are similar in the kind of answer they draw out from the respondents —clear-cut answers.
Specific questions
Specific questions are precise, clearly defined and they leave no room as to the intended meaning. At times, they come as multiple-choice questions that consist of two sections: (1) the stem which is the question itself and (2) a list of response alternatives, choices or answers that respondents will have to select an answer from.
Example: Suppose you want to evaluate your marketing channels and find out which platform is your brand more visible in. So you ask your participants:
How did you first learn about our product/website?
As alternative responses to the question, you can give these to your participants:
- AdWords
Among all kinds of questions, specific or multiple-choice questions are considered to be the most versatile type of questions. In user research, you can use specific questions to discover facts or to gain an understanding of user behavior.
Implicit questions
If anything is said to be implicit, then it means that it is not directly pointed out, but it is somehow suggested in the statement. So, an implicit statement is as an expression that prompts a certain reaction.
With that said, what then is an implicit question?
An implicit question can be defined as a leading question that gives hints about the type of answer needed. Think of them as leading questions, that pushes participants to respond in a specific manner.
Example: If you ask users this question: How many times do you visit our website?
Using “how many times” in the above question implies that the participants have visited the website before. There is an element of conjecture and assumptions and in this case, participants are persuaded to give a numerical answer.
If the question was to be phrased as a direct question, it would have been:
Have you ever visited our website?
This second question doesn’t influence the participants’ responses —and this means that the question doesn’t cultivate any biases in respondents.
Tips For Writing Effective Close-Ended Questions
Close-ended questions should not always be thought of as simple questions that anyone can easily answer merely because they do not require a detailed answer.
1. Begin sentences with Where, Which, When, Did
To make a question a close-ended, there is a certain way you should phrase it. In his book Conversationally Speaking, Alan Garner suggests that you use these few words to begin close-ended questions:
- Are
- Would
- Who
- When
- Where
- Which
- What
Using these words, here’s a list of some examples of close-ended questions you can use:
- Are you happy with your experience when using our site?
- Would you recommend our product/service?
- What challenges did you face when you were using this website?
- Which elements on our mobile applications were easy to use?
None of these closed-ended questions prompt participants to give detailed answers. They all can be answered with a one-word answer, as they aim to find out the ‘what’ and not the ‘why’.
2. Be clear and simple
Needless to say, when you ask a clear and simple question, you allow the possibility of a clear-cut answer. So the starting point is to remove extra verbiage that may end up distracting or confusing the respondents.
Good example: Would you recommend our website?
Bad example: You have used the website for more than 10 minutes and you have visited all the pages and clicked on all, so does it mean that you will recommend our website to other people?
Once you frustrate users with wordy questions, you risk compromising the value of your feedback. Here’s a list of clear and simple close-ended questions:
- Did you experience good customer service?
- Would you consider using our product or service again?
- Did you like our product or service?
- What product or service were you looking for today?
- Are you happy with your experience with us?
- Did you find what you were looking for today?
Having clear and simple close ended questions won’t just make it easy for users to infer the intended meaning, but you, as a moderator, will understand the given answers without any hassles.
3. Relevant answer choices
If you intend to use multiple-choice questions, then make sure that your suggested answers are plausible. Participants usually have different experiences even after using the same product, so you should have several alternative answers that best describe answers.
Use at least four alternatives in each multiple-choice question so as to give users a variety of alternatives.
When to use close-ended questions.
Generally, an online poll can have close-ended questions and open-ended questions as long as they require short feedback. But to give users a simple experience, you can use close-ended questions as they are easy to answer and do not require a detailed answer. Since they require limited answers, Susan Farrell from Nielsen Norman Group says this:
Closed ended questions are often good for surveys, because you get higher response rates when users don’t have to type so much. Also, answers to closed ended questions can easily be analyzed statistically, which is what you usually want to do with survey data.
Here’s an example of one of the close-ended questions we use on the FigPii polls. The question asked users the reason for their visit on the site and it allowed them to select one answer from four alternatives.
Such close-ended questions which have multiple-choice forms have higher completion rates as users do not have to come up with their own responses. Although the answers given provide a general sentiment of insights, you can always follow up with an open-ended question so as to see things from the users’ perspectives.
For instance, you can ask users to further elaborate on their answers by asking this as a follow-up:
What is the most important feature of our product/service for you? This way you can understand the context behind the users’ decisions.
Considering that they don’t demand much explanation from the respondents, close ended are perfect in quantitative usability research where you’d need to measure usability metrics such as task completion rates, error rates, and post-task satisfaction.
Insights gained using close ended questions allow researchers to categorize respondents based on the answers they have selected. How so?
Let’s say you have an online store, and you need to know the demographics of people who visit(ed) your site and left without completing a purchase. To decipher this demographic information, you can conduct an online survey that asks these close ended questions:
Question 1: Can you please specify your gender?
- Female
- Male
Question 2: You are in which age group?
- 18-24 years
- 25-34 years
- 35-44 years
- 45-54 years
- 55-64 years
- 65+ years
Question 3: What is your annual income range?
- $18-24k
- $24-45k
- $45-80k
- $80-120k
This knowledge would help you target the right kind of marketing campaign to the exact customers you’d want to attract.
Similarly, the Nielsen Norman Group gave this list that indicates when to use close ended questions. Here are the situations where you should use this type of questions:
- In quantitative usability studies, where you are measuring time on task and error rates, and you need to compare results among users
- In surveys where you expect many (1000+), respondents
- When collecting data that must be measured carefully over time, for example with repeated (identical) research efforts
- When the set of possible answers is strictly limited for some reason
- After you have done enough qualitative research that you have excellent multiple-choice questions that cover most of the cases.
Final Thoughts
Whenever you hear any CRO consultant or agency saying that they will fish out all the ‘barriers that inhibit conversion’ on your site, all they mean is that they will ask relevant questions until they achieve better results.
In this CRO business, assumptions can ruin what might have been a good relation with your customers. Foster the culture of asking questions, after all, the source of valuable knowledge is attained through asking questions.
So, whether it’s open-ended or closed-ended you decide to use, make sure you can answer Yes to the following questions before coming up with any questions:
- Do your participants have relevant prior knowledge needed to respond to your questions?
- Does your question address one of the important aspects that your users may have experienced during the course of the research?
- Is your question clearly outlined, using the appropriate language that can be easily interpreted by your customers/users?
- Are your questions grammatically correct?
Additional Resources
1.Open ended questions and close ended questions: What they are, and how they affect user research.
2. 6 DTC ecommerce websites with killer value proposition.
3. 11 customer service psychology secrets that go down the funnel, not the drain.
