A GUIDE TO CONDUCTING QUALITATIVE USABILITY STUDIES

Reading Time: 26 minutes

If you want to build a multimillion-dollar online business, then you must understand how visitors view your website and how they interact with it.

Prior to making changes on your website and before starting any A/B or multivariate test, you need to conduct qualitative and quantitative researches. The data you collect from both types of research help you assess your site for potential problems, areas of weakness, as well as areas of strength.

Over the years, I heard companies claim that they conduct qualitative analysis to understand online visitor behavior. However, when you start digging deep into what these companies are really doing, you discover that they are merely paying lip service to qualitative research.

Why is that?

The fields of qualitative and quantitative research have been used extensively in social and psychological research for over a hundred years. While the literature is rich with books that explain how to use the research for conducting social studies and how to analyze the results, little has been produced in terms of using the research to increase website conversions.

When using qualitative and quantitative researches with the goal of increasing website conversion rates, you need a conversion-oriented approach to plan, conduct, and analyze your research. This approach assures you find valuable insights about your website visitors.

In this post, we will share with you the 3 steps we developed at Invesp, throughout the years, to get the best results from planning, conducting, and analyzing conversion-oriented qualitative research.

What is Qualitative Research?

Qualitative research is the process of observing, analyzing and explaining visitors’ interaction with a website.

With a focus on general user trends, qualitative research attempts to explain how visitors view a website, its offering, and its competitors.

To conduct qualitative research, you interact with website visitors, either virtually or physically, by asking them predesigned questions with the goal of uncovering potential problems on your website.

Among the different methods of conducting qualitative research, you can find focus groups and user testing. Focus groups involves small groups of participants, could be as small as four individuals, who interact with your website while you observe them and ask questions that relate to your offer. User testing involves users completing specific tasks on your website, while you observe them.

Qualitative research goes through three general steps:

• Planning the research: you determine the research goal, what questions to ask, what methods to use, how many people will participate.
• Conducting the research: you carry out your research and gather data.
• Analyzing the results: you analyze the collected data looking for patterns and actionable insights.

“Conducting the research,” step 2, is the easiest. However, the success of your research depends greatly on step 1, where you invest the time making sure that you are asking the right questions to the right people, and step 3, where you analyze the results.

Most companies think that qualitative research is limited to step 2. So, they do not invest the time in thinking through their questions, and they definitely do not invest the time in looking for meaningful insights from their research.

One of the earlier qualitative research projects we conducted was attempting to understand high checkout abandonment rate for a website which sold nursing uniforms. Abandonment rates for un-optimized checkout are usually anywhere from 45% to 80%. This particular client reported checkout abandonment rates close to 82%. Nothing in the checkout explained this above-average abandonment rate.

The team then conducted a usability test. Nurses were invited to place an order with the site while the optimization team observed and conducted exit interviews to gather information from participants.

Most participants completed the checkout process successfully and placed an order with the website without any problem. The test revealed nothing glaring in the website that would explain the high abandonment rates.

Exit interviews, however, revealed the real reason behind high abandonment rates. Participants acknowledged that the biggest problem causing them to abandon the site was their fear of paying too much for an item. Through our research, we discovered that nurses are price-conscious, so if they can buy the same item from other competing websites or bricks-and-mortar stores at a cheaper price, they will do so. So, price played a big role in deciding where to buy a uniform.

Our client was aware of nurses’ price sensitivity. The client offered a money back guarantee and a 100% price match guarantee. The problem was that most of the site visitors landed on category and product pages first, and the company’s price assurances were only displayed on the home page. Therefore, most visitors knew nothing about these assurances.

With both insights, about the visitors’ behavior on price and the problem of price assurances only displayed on the home page, the team added an “assurance center” on the left-hand navigation of the cart page reminding visitors of the 100% price match and the money back guarantee. The new version of the page resulted in a 30% reduction in shopping cart abandonment.

What is Quantitative Research?

Quantitative research is the process of analyzing analytics software data and numbers that relate to visitors’ behavior around the website.

With a focus on numerical specifics, in conducting quantitative research, you set up goals and funnels around the website to understand the different visitor navigation paths.

So, if you are in an e-commerce website, you setup several funnels in your preferred analytics program. These funnels should include (but are not limited to):

• Homepage to top level category pages;
• Top level category pages to mid-level category pages;
• Bottom level category pages to product pages;
• Product pages to cart page;
• Checkout process.

Let’s take the checkout process funnel. The goal of this funnel is to track how visitors navigate through the different steps of your checkout process. This funnel will also provide you with the checkout abandonment rate which shows the percentage of visitors who start the checkout process but never finish it.

If you are a lead generation website, you will also setup several funnels to track visitor navigation around your website. These funnels should include:

• Homepage to offer pages;
• Offer pages to contact form page;
• Offer page to supporting pages (for example from your services pages to your blog post pages).

The Difference Between Quantitative and Qualitative Research

Quantitative and qualitative research complement each other because they provide different results:

• While quantitative research uses general data on all the visitors of a website, qualitative research focuses on a very small set of website visitors.
• Quantitative research does not concentrate on the complexity of the human interaction with the website. Qualitative research, on the other hand, produces small, narrow-focused, but rich data.
• By its nature, quantitative research produces general patterns for visitor interaction with the website. Qualitative research tries to deduce patterns of visitors’ interaction with the website; however, it strives to explain the variance in collected data.
• Quantitative research is focused on the what. It analyzes data to determine visitors’ behavior on the website. Qualitative research focuses on the why. It tries to explain why people behave in a specific way.

There is a tendency to assume that qualitative research supports the findings of quantitative research. That is not the case. Each research stands on its own to explain the visitors’ behavior and the reasons for their behavior.

Both qualitative and quantitative research should produce valid findings that drive actionable marketing activities. We do not believe in conducting research for the sake of research. You want meaningful and actionable insights that you can implement on your website and in your marketing campaigns.

Should you start with quantitative or qualitative research?

Quantitative research tends to point out possible potential problems, and aids in the designing of our qualitative research questions and scenarios. Your quantitative data might tell you that you have a problem in product pages and that not enough people are clicking on the add to cart button, while your qualitative research will help you dig deeper and go beyond product pages trying to understand where the problem is initiated.

What Do You Need to Conduct Qualitative Research?

Conducting qualitative research requires a skillset that is quite different than the one required for quantitative research.

We noticed over the years that many analytics and online marketing professionals are good with numbers, but they lack the skills and the critical eye required to understand how visitors view a website and interact with it.

Quantitative research is your attempt at understanding “what” questions. Qualitative research is your attempt at understanding “why” questions.

To be successful in conducting qualitative research and analysis, you will have to train yourself on the following skills:

• Real interest in understanding why visitors behave in a certain way on a website;
• Ability to ask questions and critically analyze human behavior, so you never take things for granted;
• Analytical eye and ear, as well as ability to think quickly on your feet, listen to what people are saying, and as a result, question further.

As you go through the process of conducting your research, you will find yourself always asking questions and seeking answers to them.

You can find three types of questions in qualitative research:

1. Your research question(s): what you are trying to find out;
2. The question you ask participants to generate data (NB: only in qualitative research that collects data from participants);
3. The questions you ask of your data, in order to answer your research question(s).

Step 1: Planning Your Qualitative Research

Most companies jump into conducting qualitative research without any proper preparation. Thus, they create surveys, and interview customers or launch exit surveys, but they ask all the wrong questions.

And when you ask the wrong questions, you will collect the wrong data from participants. You cannot use the wrong data to draw reasonable and actionable insights.

As you plan your research around your website, Virginia Braun and Victoria Clarke, in the book Successful Qualitative Research: A Practical Guide for Beginners, suggest asking the following questions:

• What do I want to know?
• Why do I want to know it?
• What assumptions am I making about research, and knowledge (what are my theoretical and methodological positions)?
• What type of data would best answer these questions?
• What type of data will I use to tell me what I want to know? (The answer to this and to the previous question is not always the same)
• How much data will I need?
• How will I collect my data?
• If my research involves participants: Who will I need to collect data from?
• How will I access and recruit those participants?
• How will I analyze my data in order to answer my questions?
• What particular ethical issues do I need to consider?
• Are there any pragmatic or practical factors that I need to take into account?

a.    Identify the Goal of the Research

Before you start thinking about the questions you want to ask participants, you should clearly identify the goal of the research.

Your research goal is vital and will impact all aspects of your research. You can identify your goal with one single question that the research tries to answer. When you determine the research question, you are able to classify the questions which you will ask in your research.

So, what are you trying to accomplish by conducting the research? Are you casting a wide net such as “why is my website not converting?” Or are you taking a very narrow approach, as “how effective is the search functionality of my website?”

It is our experience that focused research topics will drive more focused analysis and better results.

A good approach is to sit down with your team and think of four to eig
ht possible wide research topics or questions that you want to answer. You then take each of these topics and break them into smaller topics. That will drive more focus to your work.

For an e-commerce website, you can think of the following sample research goals:

1. How do you potential buyers go about locating and ordering an item that you carry?
2. What barriers do potential buyers see to placing an order with your website?
3. What factors influence potential buyers when buying an item which you carry?
4. What is the visitor experience using the search function of the website?
5. What is the visitor experience locating a product on our website using website search?
6. What is the visitor experience locating a product on our website using website navigation?
7. What is the impact of using price incentives on reducing visitors’ anxieties around the website?
8. What is the impact of using urgency incentives on reducing visitors’ anxieties around the website?
9. What is the impact of using scarcity incentives on reducing visitors’ anxieties around the website?
10. What information do the visitors need to see on the product page to be persuaded to buy?
11. What information do the visitors ignore on a product page and consider non-relevant?
12. What are the factors impacting visitors completing the checkout process?
13. What factors can enhance visitor engagement on product pages?

Again, all of these could make good goals for your research.

For a lead generation website, you can think of the following research questions:

1. How do potential buyers go about locating and choosing a vendor in your space?
2. What factors influence potential leads when locating a service that you offer?
3. What barriers do potential buyers see contacting your website?
4. What is the visitor experience locating a particular service you offer on your website?
5. What is the visitor experience as they land on your homepage vs. competitors’ homepages?
6. What is the impact of trust factors on reducing visitors’ anxieties around your website?
7. What is the impact of using research on enhancing visitor trust reducing anxieties?
8. What is the visitor experience in learning more about your company history?

These research goals are not comprehensive by no means.

However, these should give you a way to start thinking of potential goals for different research you should carry on your website. Notice how each research goal can drive actionable insights that can be implemented to enhance the visitor experience.

 b.    Select the Research Method

On top of guiding the actual questions you will ask participants, your research goal is fundamental in defining the method of research you will conduct.

You can use many different research methods, but there are four main ones we use in conducting qualitative research for online marketing:

• One-on-one (or in-depth) interviews: As the name suggests, you will conduct one-on-one interviews with existing or potential customers asking a pre-set list of questions and getting an in-depth knowledge about their personal experience. You, for instance, can ask people that call in to the call center of whether or not they would like to partake in that sort of activity.
• Focus groups: these are moderated group discussions where the moderator directs and leads the discussion.
• Usability studies: you ask participants to complete a list of tasks on your website (or a competitor website).
• Surveys: you ask participants to answer a set of questions to reveal their feelings around the research topic.

We will get into more details about each of these types and how to use them correctly.

So, which research method is right for your situation? This will depend on your research goal. One-on-one interviews are great in understanding how the participant feels about a particular product or service. They provide a deep insight into the participant views.

Focus groups allow for an open discussion among participants which provide a wide range of views on your market or website. Focus groups are an excellent tool to understand general issues which potential customer might have buying your offer.

Usability studies are best suited to observe the participants using a particular website functionality. They are not a good method if you are trying to understand general market trends and issues.

Finally, surveys allow participants to provide insights into both general market issues as well as specific website functions. However, due to the fact that participants answer questions either on a paper or virtually, getting an in-depth knowledge using surveys is difficult.

c.    Design Your Research Questions

The number of questions you should ask varies based on the setting and method. So, website exit surveys should not have more than 6 questions. Web usability studies can have multiple tasks and questions (up to 15). Focus groups and one-on-one interviews allow for more questions.

Anyway, each question you ask should relate to your research goal.

Let’s assume that you run an e-commerce website that sells cell phone accessories and that your research goal is to understand the visitor experience locating a product on our website using website search.

This type of goal lends itself to a usability study where you ask participants to locate an item on your website using the search function.

When conducting the usability study, you can either:

• ask participants to locate an IPhone 7 hard cover case using the website search function;
• ask participants to locate a specific model (model name) using the website search function;
• ask participants to locate cell phone hard cover case for their own phone using the website search function.

Each of these represent a valid research task and they will most likely produce different insights.

As the participant uses the search function, to complete your usability study, you will ask him:

• How easy was it to locate the search bar?
• How do you like the layout of the search results?
• Is there any information that is missing from our search results page?
• Can we enhance our search results page?
• Is there unnecessary information on the search results page?
• Do the search results displayed make sense?

You should carefully observe the participant as he completes the given task:

• Is the participant able to complete the task successfully?
• How fast is the participant able to complete the task? Was he able to complete on the first run or did it take him multiple tries?
• Did the user click on items which you did not expect him to click on?
• Did he use change the default sorting functionality?
• Did he have to click to 2^nd or more pages of the search results?
• Did any of the steps in locating and placing an order for the item seem confusing?

d.   Recruit Participants

Depending on the research method you choose to conduct, you have different options of finding and recruiting participants, but you should always pay close attention to finding the participants who fit your research criteria.

If your target market is black females aged 30 to 45 and you end up recruiting white males aged 20 to 30, your research will fail miserably.

Finding the right participants should not be too complicated. You need to do the following:

1. Start by identifying your ideal research participant;
2. Determine where ideal candidates usually hang out (physically or virtually);
3. Choose the proper way to reach out to them and ask them to join;
4. State very clearly to each participant the goal of the research;
5. We recommend having each participant sign a waiver form that allows you to completely own the result of their data.

Different research methods provide a different way to recruit participants:

 Usability studies: You can use one of the multiple websites that offer usability study services. Another option is to target website visitors and ask them to join your usability study. Finally, if you know a location (physical or virtual) where potential participants hang out, you might post an ad there asking for people to join.

Surveys: You can target your own email list asking those who joined it to answer the survey. Some surveys are better answered by people who converted on your website within the last 30 days. Other surveys are best answered by those who never converted.

One-on-one interviews & Focus groups: You can recruit participant for one-on-one interviews by either targeting your own website visitors or by identifying where ideal candidates hang out and reaching out to them there. A recent client of ours conducted 30 one-on-one physical interviews by targeting their website visitors and inviting them to join the research.

As you recruit participants for one-on-one interviews as well as focus groups, beware of data saturation. This happens when adding more participants does not provide any additional insights rather just emphasizes the same results.

Should I Offer Research Participants a Compensation?

There is a lot of debate around compensation for research participants. The fear is that some participants might not provide accurate insights if they receive compensation. While this might apply to social research, we have not seen this impacting our research findings in the last 10 years. Some of the forms of compensation you can think about:

• Offering participants a gift certificate they can use on your website (anywhere from $10 to $20);
• Offering participants a chance to win a gift certificate from a large retailer ($100 Amazon gift certificate). In this case, you draw a single winner from the pool of participants.

Sample Size and Diversity for Qualitative Research

In the book Qualitative Research & Evaluation Methods, Michael Quinn Patton indicates that “there are no rules for sample size in qualitative inquiry”. However, Virginia Braun and
Victoria Clarke suggest in Successful Qualitative Research: A Practical Guide for Beginners:

“Qualitative research tends to use smaller samples than quantitative research. Sometimes only a single participant or text is analyzed in depth”

For one-on-one interviews, we typically recommend a sample size of 30 participants. For focus groups, 4 to 6 groups could satisfy what you are looking for.

Keep in mind, that using 4 different focus groups (with 6 people in each) provide 4 data points. It does not provide 24 data points!

Another important criterion when selecting your participants is that you select a diverse group.

As cited by Virginia Braun and Victoria Clarke, Margarete Sandelowski explains:

When qualitative researchers decide to seek out people because of their age or sex or race, it is because they consider them to be good sources of information that will advance them toward an analytic goal and not because they wish to generalize to other persons of similar age, sex, or race. That is, a demographic variable, such as sex, becomes an analytic variable; persons of one or the other sex are selected for a study because, by virtue of their sex, they can provide certain kinds of information.

Step 2: Conducting Your Research

Now that you have identified your research goals, the appropriate method, with the questions you want to ask, and have recruited participants, you are ready to carry out the actual research.

One-on-One Interviews

Conducting one-on-one interviews requires you to prepare a list of questions that you want to ask the participant. However, you should not follow the questions to the letter. This method of research allows you more freedom based on the responses of the interviewee. As you listen to the interviewee, you try to understand his feeling and rationale when he is interacting with your website or buying an item that you carry (either from you or a competitor).

You can conduct interviews either physically or virtually. However, to get the most out of these interviews, we recommend conducting them physically. If your circumstances force you to conduct interviews virtually, then we suggest you turn the camera on, so you can see the participants facial expressions.

Just like with any other type of qualitative research, you must invest time in preparing for the interview. Create a list of topics and questions that you plan on asking during the interview. Since you want to hear personal feeling from the interviewee, it is good to establish trust and rapport initially, so they can feel comfortable answering your questions.

We like to start our interviews by asking the participant to tell us a bit about him or herself. We can then move to asking the questions which are planned for the interview.

Make sure that you listen carefully to what the interviewee is telling you and pick up on any queues he or she gives you. Remember that the questions, which you prepared, are not set in stone.

At the end of the interview, we recommend asking a question such as: “Anything else you want to add?” or “anything you think we should think about which we have not discussed?”

Sample script for an e-commerce website that sells cell phone accessories can look like this:

You meet the interviewee and introduce yourself. You first explain what you are trying to accomplish with the interview. For example, you might say, “I am trying to understand how people shop for cell phone accessories and what factors they consider when buying one.”

• To start with, can you tell me more about yourself?
• Can you tell me how often do you buy a new cell phone?
• How do you decide on which cell phone do you buy?
• Typically, cell phone carriers offer accessories at the shop when you get a new phone, do you usually take them up on their offer? Why (if yes or no)?
• What places do you think about when you need to buy a phone accessory?
• Do you think that getting original cell phone accessory will impact its quality?
• Can you tell me about the last time you bought a cell phone accessory?
• Did you encounter any problems selecting your cell phone accessory?
• Do you buy any cell phone accessory for your family members?
• What advice do you give to anyone needing to buy a cell phone accessory?
• Is there anything else you would like to add?

These are sample questions. You can adapt the questions to meet your specific situation and website.

Final recommendations for one-on-one interviews:

• We recommend taking notes during the interviews as well as recording them (both video and audio).
• Since the interviewee is donating his or her time, try to accommodate a time and a place that works well for him.
• Avoid loud places where it is difficult to focus.

Focus Groups

Using group discussion is another way to learn more from your potential website visitors. Focus groups are excellent in getting a wide range of opinions and behaviors on a certain topic.

You may have seen or heard of this type of research conducted during political campaigns. Using it for marketing can produce significant results. In a survey on 261 companies, 47% of them cited that they had used focus groups; 81% of consumer goods companies in the respondents indicated that they used focus groups, followed by 61% of firms in the marketing research field;  79% of consumer goods companies indicated the use of one-on-one interviews while 91% of marketing research firms indicated that they used one-on-one interviews.

Similar to one-on-one interviews, you prepare a list of topics and questions that you will present to the focus group. Focus groups use a guided discussion with you playing the role of a moderator, asking questions and directing the conversation.

While you can conduct one-on-one interviews physical or virtually, focus groups must be conducted physically to get the best results out of them.

Focus groups should not include more than 6 to 8 participants. Anything more than that could easily allow the discussion to get out of hand, and the moderator to lose control.

One consideration when conducting a focus group is the mix of people who will participate. Let’s say you are trying to target three different market segments. Should you include people from each segment in one focus group or should you conduct one focus group per market segment?

There is no correct answer here and it will depend on your particular situation. Do you think that having three different groups will provide you with different information? If the answer is yes, then it is obvious that you should conduct three different ones. However, if the different focus groups will turn up the same information, then there is no need to conduct three of them.

Some of our clients discuss the statistical significance of focus groups. We do not think of them that way. We think they provide qualitative insights away from statistics. You can later validate your findings using quantitative research or split testing.

Do not to confuse focus groups with one-on-one interviews. Conducting three focus groups with 6 participants in each is not the same as conducting 18 one-on-one interviews. By their nature, focus groups do not provide in-depth personal knowledge about participants.

Usability Studies

When you observe users completing different tasks on your website, you can better understand how visitors interact with different website elements.

As the name suggests, usability studies provide great insights into users interacting with your website. They are not meant for understanding general market trends.

Several services available online allow you to quickly recruit participants and conduct user testing on your website. However, conducting a physical study produces better results.

We recommend starting your usability study by selecting the correct type of user you want to participate in the study.

You can choose to run a usability study on a wide topic, such as placing an order from your website, or on a narrower topic, such as using the website search function to locate an item.

Our recommendation is to conduct several narrow usability studies on your website to gauge how visitors are interacting with the site at different points throughout the process.

In addition, we recommend running the same usability study on your top three to five competitor’ websites.

If you run paid search campaigns, ask the users to simulate taking the action that will trigger both your ads as well as competitor ads to display and then to click on them and follow the conversion path of your website.

With the prominence of mobile devices, we highly recommend running the same test for both desktop and mobile websites.

Finally, make sure to ask participants questions about their general perception of your website after completing the usability study.

Case Study:

After beginning a project and having initial CRO discussions with a client in the homeware business, we conducted some qualitative research in order to show them some of the concerning trends we were seeing from the quantitative side.

We conducted a usability testing through a user testing website. We wanted users to take advantage of the savings on the website, and purchase an item from the clearance section: 100% of the visitors did not see the huge banner that took up about a quarter of the top of the homepage to go directly to the clearance section. All of them struggled to complete the task. This was alarming because the company had spent so much time and effort on creating the marketing materials and banners on the website.

Surveys

When you run a survey, you ask participants to answer a list of questions that relate to your research goal.

Surveys are excellent in providing large dataset which you can analyze to understand user perspective on your market, website, products or services.

When selecting participants for your survey, consider the following segments:

• Everyone who joined your email list: these are people who are interested in what you have to offer. The email list provides a mix of participants between those who might have converted and those who did not convert.
• Everyone who converted within a certain time period: converted users provide great insights into what works and what does not work on your website. For an e-commerce website, those who converted within the last couple of weeks provide the best data set (if there are enough of them).
• Everyone who never converted: non-converted users can help you understand what stopped them from converting and what additional elements you need to add or modify on your website.

For an e-commerce website, you should consider the following possible groups for your survey:

• Customers who placed a single order with your website
• Customers who placed 2 – 4 orders with your website
• Customers who placed more than 5 orders with your website
• Customers who placed orders with low average order value
• Customers who placed orders with high average order value
• Customers who specifically buy a certain category of products that you carry

How Many Questions Should I Use in My Survey?

When creating a survey, gather as much relevant data as possible. Lengthy surveys, while they seem attractive since you are “getting” more data, have a less chance of getting completed.

So, what is a good response rate for your surveys?

MarketingSherpa, posted the following response rates:

• For prospects, a respectable click-to-completion rate is between 1% and 5%
• For customers, the click-to-completion rate should be anywhere from 20% to 55%
• For a consumer marketer, open rates are typically above 15% and click-to-completions greater than 25%

Best Practices for Getting the Most of Your Online Surveys

1. Group your questions

As you prepare the list of questions you want to ask participants, group them in different themes. Each theme will present a list of questions that dig deep into the topic from a different perspective.

One theme might ask participants about the general market information. Another might ask participants about their shopping behavior. A third, might ask about their interaction with your website or service.

2. Keep It short

Don’t go overboard with the number of questions you ask participants to answer. Our recommendation is to not exceed 30 questions. We typically like to group these into 4-5 themes. As you design your survey, make sure to layout the survey on multiple pages. Each page covers one of the survey themes.

Also, make sure that the survey software you are using is collecting data as participants complete each step. So, even if a user fills out two pages of the survey, then that data is still stored and collected.

3. Only ask important questions

Many marketers go overboard with asking too many questions when creating surveys. As you add each question to your survey, make sure that it relates to the goal of the research you are conducting.

Evaluate each question in your survey with the, “so, what?”, question.

“So, what” if I know the participants does not think of pricing when buying this item?

“So, what” if I know that participants only shops for this item offline?

“So, what” if I know that participants do not know what makes an offer a good deal?

Notice how each of the questions above will give you actionable insights you can implement on your website.

4. Pay Close Attention to Email invites

With online surveys, you will use email to invite participants to answer your questions. Pay close attention to every little detail in your email invite.

1. Use a strong email headline

Your email headline is the first chance to convince visitors to open the email. Strong headlines focus on the benefits or results which the user will get from participating in the survey.

2. Personalize your email by using the participant’s first name.

Personalization of the survey increases the chances of getting more responses. This will of course depend on how much data you have on your participants. You can use the participant’s name in the subject of the email as well as the body of the survey.

3. Let the visitor know how long it will take to complete the survey.

Letting visitor know that it will take less than 3 or 5 minutes to complete the survey will also increase your chances of getting more clicks to the survey.

4. Use a strong and nicely designed call to action button.

Needless to say that your email should have a nicely designed CTA inviting email readers to click to the survey. We recommend having at least two CTAs in your email:

• One is a text call to action;
• The other, an image (button) call to action.

The reason you use a text call to action is that many online email services will block your images from downloading. So, if you use only an image CTA, readers of your email might not see the CTA.

Let’s take couple of examples of surveys:

Tempo timesheets sent me this survey after we decided to switch to another service.

The survey does few things right:

• It uses my first name.
• It uses a nicely designed CTA.
• It lets me know that it will take less than 3 minutes to complete.

Notice this survey from a sales training course:

While the email subject is good, notice the few problems in it:

• The images are not displaying in this email (due to me using Outlook).
• The body of the email uses paragraphs that are not well formatted.
• The CTA does not appear above the fold.

5. Incentivize participants

What will participants get from answering your survey? Using incentives with email surveys will increase its response rate. There are several ways to encourage participation in the survey:

• You can offer a small gift certificate or a discount code to everyone who completes the survey.
• You can offer a one lucky participant a large gift certificate.
• You can offer participants first access to the results of the survey.

6. Test Everything about Your Survey

Everything in your survey should be tested.

To start with, you should test your email invite. At a minimum, you should test:

• Different email subjects;
• The body of the email;
• The calls to action in the email;
• The delivery time of the email.

When it comes to the actual survey, at a minimum you should test:

• Survey designs;
• The number of questions you ask on the survey;
• The wording on different questions.

Step 3: Analyzing Research Results

Now that you have collected the data from your research, the next task is to start the analysis phase. The analysis phase has two main components to it:

• Transcribing data;
• Data-mining results.

Transcribing Your Data

If you conducted one-on-one interviews or focus groups, it is important that you transcribe all the data you collected.

Having transcripts of different interviews, you analyze them and look for different themes in them. While transcribing sounds looks like a straight forward task, there is a bit more involved.

Why?

Because not only do you have to capture the words which participants use but also attempt to capture their state of mind.

A participant might pause or might not be too certain when saying something. That will definitely impact how you understand that data. Capturing these emotions is the biggest challenge.

We recommend transcribing in two different stages:

1. Transcribing the audio recording: hiring a professional transcription service or a freelancer will do the job. We have been using the same freelance transcriber for years. Her job is to capture the data from our audio recording as close as possible. There are several companies that offer this service or you can use a website such as Upwork to find a reliable transcriber. At the end of this task, you should have an initial transcript.
2. Video is even better: If you can video record focus groups and one-on-one interviews, it will give you the 360 view you are looking for.
3. Transcribing the emotions and verbal queues: in this phase, you take the transcript produced in step 1 and listen to the recording again. Your goal is to add the emotions to the conversation. If you don’t have video, this will all be from memory. Look for instances such as:

• Laughing or coughing;
• Certainty;
• Uncertainty;
• Identify pauses;
• Non-verbal utterances;
• Emphasis of certain words.

You should use some sort of a coding system to identify any of the above emotions or verbal queues.

Data-Mining: Looking for Insights

Well-designed and professionally conducted research provides you with a wealth of data on your market trends, market segments, visitors’ impressions of your business, and users interaction with your website.

When you are analyzing the results of qualitative research, you are data-mining the results with two goals:

1. Identify data patterns;
2. Determining actionable insights.

Getting Intimate with Collected Data

The process of data mining starts by familiarizing yourself and your team with the collected data. This typically means listening to interviews and reading scripts with the team multiple times.

We typically ask our team members to do two read-outs of each transcript. The goal is to start internalizing the different opinions of participants.

After completing the two initial readings, you will do two more readings but this time you start underlining and putting your thoughts around what participants are saying.

There is no right or wrong answer here.

What one team member may underline or notice, no else on the team may pay heed too. The goal is to start putting initial thoughts. Each team member is capturing their personal views of the participants.

Coding the Transcripts

The next stage is to start coding different parts of the interviews. This is typically a group exercise where we read the transcripts aloud and start coding different sections of what the participants are saying.

Since we use the Conversion Framework for data analysis and determining conversion problems, we code what participants are saying to different elements of the conversion framework.

Let’s take the following example from a focus group where participants were discussing buying toner ink for their printers:

What participant is saying	Coding
I am worried about paying too much for the toner ink	FUDs (fears, uncertainty, doubts)
It is faster to just drive down to best-buy and pick up the ink from there	FUDs (fears, uncertainty, doubts)
Yeah, I can wait on ordering the item, but I like taking the break from the office and just going out to buy the toner	Engagement
The problem is that so many online stores offer the toner and their prices all look comparable	Trust (value proposition)
The problem is, ehh, I heard that that non-original toner ink will break down the printer	FUDs

You can code either the whole transcript or you can code a portion of the transcript.

Defining Themes & Looking for Patterns

After finishing the coding process of the transcripts, the next stage is to look for different themes that appear in our data.

While there are different approaches to data analysis, we recommend using a thematic analysis for online marketing.

In this thematic analysis, we are looking for themes that appear in different datasets. Virginia Braun an Victoria Clarke, in “Using Thematic Analysis in Psychology,” tells us that a theme:

“captures something important about the data in relation to the research question, and represents some level of patterned response or meaning within the data set”.

So, a theme combines different codes looking for consistent patterns in them.

One question we get with clients as we work on creating different themes from data is how many themes we should create. There is no correct answer to this question.

We have worked on projects in which we are able to identify 4 to 5 themes. We also worked on projects where we identified up to 8 different themes.

The only rule we have is let the data guide you.

Some themes tend to intersect with each other very closely. Think of general theme around trust factors on a website. At the same time, another theme might be anxiety producing factors on a website. It is natural that the lack of trust factor will increase visitor anxiety. So, how would you classify lack of trust factor? You can classify it under the trust theme and at the same time you can classify it as an anxiety theme.

Deep Analysis and Marketing Insights

This will be the final stage in your research. At this point, you have a lot of data that you have collected. You have results from your research methods, coded transcripts, themes and patterns.

The final step is to put all this data together and start looking for actionable marketing insights.

Some companies and consultants like to conduct research for the sake of research. That might be fine, if you have the luxury to do so. Most of the time that is not the case.

Our research should produce specific items that will impact on how we design a website or structure a campaign.

To deduce actionable marketing insights from the research, approach your findings with a deep analytical eye.

Again, let the data guide your findings.

While you might have preconceived notions on the research subject, the goal from the research is not to prove that you are correct. The goal from the research is to discover deep insights from your participants about the research subject.