Expert Answers to 9 Tough CRO Questions

1. Are there any risks involved in an optimization project?

Every project comes with risks. That’s probably a known fact. But does this ring true in the world of conversion optimization? If it’s true, what kind of risks are there in a CRO project? And how can you avoid falling victim to them? 

Well, according to Georgi: 

“Yes, similar to any other project. The difference is that most optimization projects rely on experiments to manage those risks, so usually, such projects involve much less risk than other types of marketing or product development work. This means that if your CROs do not know what they are doing, this will become evident during testing (assuming the tests are done properly), which is not necessarily the case with PPC or SEO. While experiments are possible in PPC, I don’t think they are as widespread as in CRO. Experiments in SEO are very difficult if not outright impossible and of much more limited value.”

Lorenzo says that he always see a couple of risks in optimization projects: 

“The first one involves the people running an optimization program without proper experience. The second thing that could also be risky is optimizing for the wrong KPI. Most people always obsess about optimizing the conversion rate. But that’s just one side of the coin in terms of trying to achieve the company’s overall goal. Sometimes, there could be some opportunity to increase the AOV or retention rate, but if you are only obsessed with conversion rate, you might miss other important opportunities.” 

It’s funny that CRO should help you avoid risks, and yet it comes with its own set of possible risks, says André. He also goes on to mention three potential risks you should avoid when doing conversion optimization: 

“CRO should be about risk mitigation, but it also has its own risks. If you do not know what to optimize and why you don’t need to A/B test anything, you could save a lot of money by not doing A/B testing and optimization. 

The first risk associated with CRO is testing weak hypotheses. 

The second risk is weak operationalization. This usually happens when you test things and deduce the wrong assumptions from the results. Sometimes you may have already tested something, not get positive results, and then decide to stop at that point. But, that doesn’t mean they might not be another way of testing it. 

The third risk is the wrong implementation. Sometimes your variations may work in another browser, and you think that’s alright, and you forget to test on other browsers and devices. That’s a huge risk that can cost you a lot of money.”

Chad says there are dozens of risks in a CRO. He also warns against implementing a test without proper Quality Assurance: 

“There are many risks, and one I can think about is the implementation of the test. Rolling out an experiment without doing Quality Assurance. We once ran an experiment where we changed the priority order of a list of offers for our customers. When the experiment was rolled out, the swipe to go down the page wasn’t working. There was a bug. And because of that bug, we saw a pretty big negative impact.” 

“Nothing of value is without risks,” says Tim. He also goes on to say, “the biggest risk in experimentation is people not doing it in the way it is intended and trying to treat it like some kind of shortcut to money.” Tim also mentions different kinds of CRO risks you should look out for: 

“Another risk is to try CRO, but not give it the correct thinking, and you continue to do the wrong thing. Then there’s the risk of making decisions based on too little data on the false assumptions that some data is better than no data and making the wrong decisions believing that they are the right ones.

2. What comes first, CRO, SEO, or PPC?

If you are reading this, you probably know the difference between CRO, SEO, and PPC. But do you know which one should come first? If CRO has to come first, why is that so? Well, Georgi says they should work hand-in-glove and not as foes: 

“I’ve always seen them as complementary and not as mutually exclusive options.”

André agrees with Georgi, as he says, “traffic multiplied with conversion rate equals revenue.” But he also adds that CRO problems are usually the plague of most businesses:

“So, of course, if you have constraints in your traffic acquisition, it’s as bad as having problems with your conversion rate. The thing is, I strongly believe that SEO has a 10 years maturity advantage. The SEO industry is quite mature, the knowledge is quite mature, and it’s not that complex. I see much more companies having much more conversion rate challenges compared to traffic acquisition. But in general, it’s two independent variables.”

On the other hand, Chad says CRO should always be given the first priority because it enables you to unlock your customer pain points: 

“CRO should always be first. But the problem is that most people think of good CRO as a particular set of techniques. And that’s not the whole idea of CRO. Experimentation and CRO is simply the thought process of thinking about the problems of our customers. We need to build engineering solutions to solve those problems. And then, we need to measure the results. So we have some idea whether it worked or not…that should be the core of every business. You can always keep your business strong – even without SEO or PPC – if you build more things and test to see if they work.  

A great example of this is Airbnb. They cut their marketing spend by 25% during the pandemic, and they didn’t see a significant drop in sales. The reason that people use Airbnb is not because of the marketing. It’s because they’ve developed an excellent, fantastic product that people love to use, and it solves a problem. I’m not saying that marketing is terrible. SEO, PPC, or Facebook ads can be a good solution. Especially if customers have never heard of you.” 

Lorenzo agrees with both Georgi and André in putting CRO, SEO, and PPC in the same box, “they should all work together.” But he also says that if you are to choose which one you should start with first, go for CRO. 

“But it’s better to start by doing CRO and figuring out – from that perspective – the customers’ decision-making process. And based on those findings, then you can create an SEO or PPC strategy.” 

Tim says that what comes first is determined by the kind of problem you have on your site. He also goes on to say that not all CRO issues should be given first priority over SEO or PPC: 

“Sometimes, when clients approach me, I look at their data, and they don’t have a CRO problem. Or, if they do have a CRO problem, it might not be the problem they should be solving first, so I tell them to fix their SEO or PPC. Part of it is down to maturity or situation. 

Sometimes, I look at what type of CRO program fits their business. I mean, if their Search Engine Marketing is poor, then the conversion optimization effort can be spent on trying to understand why it is performing poorly. This can help us answer the following questions: 

• Are they communicating the wrong message to the wrong people? 

• Do they have the right message but to the wrong people? 

• Or maybe they have the wrong product-market-fit?. 

Ultimately, CRO, SEO, and PPC are all tied together. The idea is to work towards the ultimate end-goal of the business and not try to isolate.”

3. Should CRO be framed as a marketing initiative or a business strategy?

Before you read the experts’ answers, what’s your take on this? Think about it for a second. Most companies tend to CRO as a marketing initiative. But is it the right way of going about it, or should they turn it into a business strategy? 

In this regard, Georgi says: 

“In my mind, it [CRO] sits between marketing and product development, but I’ve not delved on the issue of taxonomy too much.“

Lorenzo says CRO shouldn’t be considered as a marketing initiative or business strategy, “It should be pitched more as a process or mindset, and not as a business strategy or marketing initiative.” He also adds, “this is because CRO allows you to learn a lot of valuable things that other departments – like product development and customer support – can take advantage of.” 

André agrees with Lorenzo, but he also adds that calling it CRO is wrong: 

“100% I would always try to frame CRO as a mindset, give it a different name towards management level and frame it as a strategic cultural mindset in the company. I don’t even think CRO is the proper term to address its importance towards the management level. As soon as you’re talking CRO, you’re talking tactics, and management is not interested in tactics. 

CRO is basically a mindset of analyzing a challenge carefully, being very customer-centric, understanding what customers want, then deducing hypotheses and prioritizing them to find out what has the business impact.” 

Chad is wary about framing CRO as a marketing initiative. He believes that framing CRO as a marketing initiative would breed another problem of comparisons: 

“If you frame it as a marketing initiative, then it is always going to be compared to other marketing efforts like email campaigns, advertising campaigns that have a clear line to revenue. When using Google ads, you will have a dashboard, and it will show you how many conversions have resulted from those Google ads. But with optimization, this isn’t always the case, and it’s not even the way you should use experimentation. Sometimes you just want to run experiments to learn things, and you’re not expecting any type of ROI.“

Instead of framing CRO as a marketing initiative or business strategy, he suggests treating it as a department that is as important as other departments in a company: 

“CRO should be framed as a business strategy. In fact, CRO should be treated the same way someone would treat a security team. You don’t invest in a security team because you expect it to make you money. You invest in a security team because it makes sense to be safe. And with CRO, it makes sense to always be learning and be making reliable measurements.”

According to Tim, the main reason why most people treat CRO as a marketing initiative is that “it has the quickest and most evident effect on KPIs that are traditionally used by marketers.” But also goes on to support the idea of treating conversion optimization as a business strategy: 

“CRO works best when it is planned as a business-wide strategy that helps to improve the whole business. The whole business should have an experimentation mindset and determination to get better at what they do.” 

4. What’s the key to an effective Conversion Strategy?

The importance of a conversion strategy cannot be overstated. Whether you have a small or enterprise business, a well-crafted conversion strategy will always benefit your company one way or the other. But is there anyone thing key element that you should always include in your conversion strategy: 

Georgi says everything involved in CRO is equally important; you can’t single out one thing:

“Quality of the data, analysis, hypothesis generation, prioritization, and statistical testing are all crucial. Get one wrong, and the rest will function much worse than optimal.”

André also agrees with Georgi on this one, “the big challenge with conversion optimization is that it is not one thing. It’s a connection of disciplines, and it’s always a collective effort.” He also says that most business struggle because they make the mistake of focusing on one element:  

“Many companies fail because they see it as a data thing, and they completely forget about the customer. But if I am to condense it, I would single out two things, and that’s being customer-centric and data-driven.”

Chad believes that the essential thing to do is to focus on customers: 

“I think the key is to focus on problems, not solutions. Often, people come up with ideas like changing button colors and thinking that will increase their conversions. But that’s starting with a solution and assuming that there’s going to be an impact. Customers have so many problems. Reach out to them and ask about their pain points. Sometimes you might find that their pain point is not even related to your website. Maybe it’s the customer service that’s causing people not to buy anything. So, focusing on customer problems is the key to a conversion strategy.” 

Tim says that the crucial thing to do in your conversion strategy is to align CRO and business goals. He also advises against copying your competitors or other more prominent brands in your industry: 

“There’s no one thing because there are a dozen things that make this up. But you have to align your CRO program with your business. People often make the mistake of copying bigger brands just because they do a lot of testing. They copy other brands and forget that they don’t have that ability; they don’t have the budget; they don’t have the traffic; they don’t have an audience that fits what they see big brands doing. And so they try and do a conversion strategy that doesn’t fit them. The secret of success in a conversion strategy is matching your ability against your needs. There has to be a balance between your goals and ability.”

5. How do you calculate the revenue impact of a CRO program?

If you ask any seasoned optimizers, they will tell you that calculating the revenue impact when running a CRO program is challenging. Most optimizers don’t really know if they should be doing it or not? But should you really calculate the revenue or impact? If yes, how do you do it? 

According to Georgi:

“It’s one part business accounting, one part meta-analysis of projections from the estimated effects from initial and confirmatory tests, and one part art. The last one refers to anticipated changes in the entire environment and how that would affect the future performance of the implemented changes, and it becomes more of a factor for longer-term estimates of the revenue impact.” 

Lorenzo says that “instead of trying to calculate the impact, you should be focusing more on understanding the learnings that the program is giving you. And learn how to use the process to make decisions based on a scientific methodology.” 

André says that there’s no way you can calculate the revenue and the impact of a CRO program, “that’s a big myth with no one answer. If you reduce experimentation to direct business impact, it becomes hard to calculate. And that’s not even the real value of experimentation.” He also adds that: 

“There are many other values – faster learnings, better insights, more agility, and customer-centricity – you can get from experimentation. But you have to always ask your clients for what they consider to be non-financial values of expectation. And you try to pile them up and make sure the value is perceived from expectation.”

Avoiding the revenue impact is the best thing to do, says Chad. He goes on to say that big brands don’t even bother to do the calculations: 

“The best way of thinking about it is to avoid calculating the revenue impact. Look at the top companies in the world that are doing experimentation, Google, Twitter, Amazon. You will see that they don’t calculate the revenue impact of an optimization program. They understand that doing conversion optimization is as essential as having a security team. 

So, the value of experimentation and conversion optimization lies in having a reliable measurement. Metrics will always be variable, customer behaviors will change due to certain seasons, and if you don’t have experimentation, you won’t know what might be causing changes on your website.” 

Although Tim also believes that there’s no way you should calculate the revenue impact, he also thinks that you should define what a CRO success is to you and your clients before you even start the project: 

“The truth of the matter is that there’s no correct answer or the right number. You can’t say, “I did this, and it resulted in this many dollars and cents.” That’s not possible. Can we estimate? Yes, but we will be basing on some assumptions, parameters, and data. If any of those parameters and data change or are incorrect, whatever you forecast as a potential impact is wrong or, arguable, right.

So, you need to have an agreement with the client you are working with. Agree on what you will define as the success of a CRO program. If you are just measuring it on revenue, then you need to make sure that everything that relates to making money for that business is also controlled by the optimizers.” 

6. Do performance-based CRO projects work?

A performance-based CRO project is a plan whereby payment is granted to an optimizer after achieving a specific outcome or results. Some CRO consultants and agencies prefer having such contracts. But do they really work?

This is what Georgi had to say: 

“Not any more than performance-based contracts in any industry other than sales and simple manufacturing tasks where the performance relies entirely upon the person executing the tasks. A performance bonus may have benefits, but otherwise, I think it is a bad deal for both parties. Such a structure makes it easy to claim credit and receive rewards without merit and suffer without reason. The incentives are never properly aligned in such a case.”

Chad also doesn’t like the idea of performance-based CRO contracts as he believes that this concept might inhibit optimizers from testing to learn:  

“No. This kind of model limits people from going for the learning. It might sound like a good idea to the client, but that will prompt optimizers to always go for the lowest denominator over and over again and not try new things. It also doesn’t encourage risk-taking. If you incentivize risk-taking and failure, your optimization team is going to try innovative ideas.” 

Such projects are risky, according to André. He also believes that performance-based projects can disadvantage everyone involved in a project: 

“That is dangerous because you will be risking a healthy relationship with your clients. You can have higher maturity and understanding of the underlying statistics, but your clients may not have that. Clients always have constraints, and you may not understand their limitations. Basically, it becomes a disadvantage on both sides. These kinds of contracts require a lot of trust and confidence on both sides to make it work. If you consider performance-based contracts, you should think about how you can always achieve great results.”

7. Can you do testing on a site with low traffic/conversions?

Conversion optimization can be done on any website. It doesn’t matter if the traffic or conversions of that particular site are high or low. But when it comes to A/B testing, it can be tricky. Can you do it on a low traffic/conversion site, or you can’t? 

Georgi believes that you can also test on any site. But he also says that “if low traffic and conversions are also accompanied by low revenue, then the relative returns on such efforts are significantly less attractive compared to when testing is applied to a larger project.” He also adds: 

“Obviously, the statistical design of the tests would be different in a project with low traffic to make testing make business sense at all. Even with an optimal design, though, a smaller project will benefit less than a larger one, all else being equal. I’ve given an example of how the math works out in part on Economics of testing: small sample size vs. large sample size in which the bigger business gets a 40% better bang for their buck simply due to their larger size.”

According to Chad, “The bigger the sample size, the easier it is to run an experiment and get a result you can act on.” However, Chad also says that testing can be done on any website: 

“If you don’t have a big sample size, it doesn’t mean you can’t do experimentation. Many people forget that experiments happen all the time using tiny samples in the academic world. Psychology and health studies usually have a low number of samples. But the problem is that they usually last a long time.”

On the other hand, Lorenzo says that instead of testing, there many different alternatives that you can do on a low traffic or conversions site: 

“If a site has about 20 or 50 conversions, you can’t launch an A/B test and use conversions as the primary KPI. But with such websites, the best thing you can do is choose a top-of-the-funnel KPI (like Add to Cart or visits to the pricing page) and focus on increasing those KPIs. 

The other thing you can do is gather insights from research, make the changes on your site and compare the before-and-after metrics. This approach might not be as accurate as testing, but it is worth trying. But when you make these changes, make significant changes, not just button colors, you can even go beyond layout design changes.” 

André believes that “It depends on the amount of validity you want.” He also goes on to say that “when there are no conversions, then that means there’s no business, and that makes it difficult for optimizers. I mean, what’s there to optimize?”

Just like Lorenzo, André also shares some ideas of what to do instead of testing:  

“So, if I’m an early-stage startup with no traffic or micro-conversions, I won’t do A/B testing. I would seek for other ways like qualitative research, quantitative research with online panels, and remote testing to validate my ideas.”

8. How long does it take to see results from CRO?

Many components are needed to launch a successful CRO program. Suppose you now have what it takes to successfully execute a CRO program. But you are still wondering how long it will take to see the results. Will it take a month? Two months? Or maybe more than 3 months? 

Well, Georgi says that “if the prioritization is right and the current CRO efforts are the first for the project, then results should arrive almost immediately.” He also adds: 

“Capturing the low-hanging fruit needs little more than the time to run the first batch of online experiments, which is usually from a week to a month for these types of tests. If a user experience has already been through dozens of rounds of improvements, then it would be much harder to get substantial results on top of that as the marginal value of each additional attempt becomes lower.”

Chad says that it may take longer than three months before you begin to see the results. But he also adds that the benefits of CRO go beyond the numbers:  

“With CRO, the results are not going to be immediately apparent in a quarter, and sometimes even in six months. One of the biggest value-adds of an optimization program isn’t even the results. It’s the culture that it creates. Conversion optimization establishes a culture of accountability, where people are responsible for the metrics they believe they can change. And it also creates a culture of science where people are thinking in terms of hypotheses and outcomes. 

When the entire business adopts this culture of science, it’s going to enable people to be moving in a much more measured way; a much more robust way; and holding each other’s standards of accountability.” 

According to Lorenzo, the number and quality of experiments you launch will determine when you will see the results:   

“It depends on how you run your experimentation process. But it all comes down to the number of experiments you’re running and the quality of those experiments. By experiment quality, I mean wherein the funnel are you running those experiments. Are you running experiments on pages that are more likely to have a bigger impact in terms of revenue or elsewhere? Or on pages with low revenue impact?“

André agrees with Chad that CRO is not about the results only. But he also says that results may start to show up after six months: 

“If you take into account that a great CRO project is not only about delivering results but changing an organization, enabling people, making them understand the importance of being customer-centric and data-driven, I would say at least six months. If you ask me how long does it take to set up an experiment and get results, I would say a couple of weeks, maybe four weeks.” 

9. What type of benefit should I expect from a CRO project? Should I expect an increase in revenue or conversions? What %?

People that are getting into conversion optimization for the first time like asking this question. And because of the name CRO, most people tend to assume that it’s all about conversions. But is CRO only limited to conversions? If that’s the case, what percentage of conversions increase can one expect from a CRO project? 

Lorenzo believes that you can’t put an exact percentage to expect from a CRO project. He also adds that the rate is determined by the maturity of your CRO program: 

“Well, percentage-wise, it depends on how mature your optimization program is. So, suppose you are optimizing a website like Booking.com (one of the top experimentation organizations in the world). In that case, it’s tough to make huge impacts because their site has been optimized for years by smart optimizers. But if you optimize a site that nobody has ever done experimentation before, you will more likely see bigger benefits.” 

According to Georgi, a CRO project with proper target metrics should result “in an improvement in a metric as close to the bottom line as is possible to measure.” He also goes on to say: “Ideally that would be profit, but as this is rarely possible, average revenue per user would be the next best thing. Conversions per user come next in the list.”

Considering how different businesses are, it’s impossible to predict the type of benefit you should expect from a CRO project, says Chad. But he also goes on to echo the same sentiments as Lorenzo on how the optimization program maturity can also be a determinant factor: 

“It’s impossible to predict because it differs from one business to another. If you have a very mature business and it’s been around for a long time, there might be certain metrics that will be challenging to change. For instance, Google has been around for about 20 years, and they have optimized their search page repeatedly to the point where they change it anymore. So, if I’m going to try and increase the number of searches on Google’s homepage, I’m probably not going to make an impact at all.“

André says that the benefits can also be influenced by the timeframe of your CRO program. But he also has an illustrative story about this question: 

“Depends on the timeframe. But it’s perilous to imply a significant business impact. We once helped a client get a conversion uplift of nearly 40%. When they came to us again, I told them they can’t expect 40% more conversions because many factors impact the experiment. 

So this is a question of expectation management. And this is why it’s important to have workshops with clients and ask them what they expect. Sometimes the financial return might not be visible, or it’s not direct that you can attribute the accumulated uplift of experiments to your bank account. 

But on the other hand, you’ll get many different values that are not measurable, and you have to be aware of that. So, yeah, expectation management is king.” 

Tim believes that the way you structure your business and CRO goals will decide the kind of benefit you can expect from your project: 

“It depends on what your business needs. So, should you expect an improvement in conversion, revenue, or both? If that’s what you’ve designed your business strategy to do, if that’s what you’ve commissioned your team or the agency you’ve hired, then yes, you should expect your revenue and conversions to improve.” 

Conclusion

Well, there you have it. The toughest CRO questions answered by the top pros themselves. Special thanks to every expert that contributed! If you want to keep on receiving such expert tips in your inbox, you can join our community of over 30k marketers.