The average success rate of A/B testing programs sits between 12% and 15%.
Yeah, I know…that’s very low.
Many brands always start AB testing hoping that it will significantly boost their conversions. A few months down the line, they are bitterly disappointed.
They probably don’t understand that it is not the A/B testing that boosts the conversions. But the quality of research that comes before the experiment implementation often determines the success of your test.
Sometimes, you can even conduct adequate research, and your A/B test will still fail to produce the results you expect – this also happens often. Ask any CRO expert; they will tell you that they’ve been humbled by A/B tests results many times.
If you want to improve the success rate of your A/B testing programs, there are some do’s and don’ts you should follow religiously. In this article, we will talk about eight lessons we learned from running 500+ A/B testing in the space of 12 months.
These lessons have helped us achieve a 65% success rate on A/B testing programs.
1. Strategic CRO + Tactical CRO = Effective CRO Program
A good CRO program consists of both strategic and tactical optimization.
Strategic optimization focuses more on the business as it seeks to answer the following questions:
- What can be done on the website to grow the business’s revenue?
- Which other revenue opportunities can we explore?
- How to position the company in the mind of the target audience?
- What’s the best way to communicate the brand message on our website?
On the other hand, tactical optimization focuses on improving the customer experience on the website. It’s all about uncovering conversion roadblocks, removing friction, and adding the right triggers that encourage visitors to turn into customers.
Depending on the client, the CRO program can be 25% strategic and 75% tactical in some instances. Sometimes it can be the other way round.
If your CRO program is only strategic – then you won’t learn anything. And if it is only tactical, you will also have short-term success.
2. High Testing Velocity is Better
In the space of 12 months, we managed to launch 500+ A/B tests across our clients’ websites – this means that our client-delivery CRO team averaged 40 tests per month.
That’s a high-testing velocity if you ask me.
I have to mention here that there was no single experiment about button colors in all those tests. We have a high talent density in our team, and we never compromise the quality of experiments just to launch dozens of tests. Our experiments are hypothesis-driven, and we prioritize gaining valuable insights over any other thing.
You may be wondering why do we have to launch more experiments?
Our team is fast, and we understand there is a positive correlation between increasing testing velocity and improving return on investment (ROI).
However, we don’t just go on a high-testing velocity on every client we work with. Sometimes it makes sense to have a low-testing velocity program, especially when the client doesn’t have adequate resources (such as traffic, sizable team).
Also, when a client is just launching their first CRO program, it makes sense, to begin with, a low-effort testing program to help them understand how the process works. And as the momentum builds, we gradually shift towards high-level testing programs.
3. Best Practices fail
Best practices do not universally work on every website, and they are not a success factor.
Best practices will tell you to use videos to increase user engagement on your landing pages. According to a study done by Google, almost 50% of internet user watch product videos before they make a purchase – and the other 50% doesn’t need to watch videos to make a purchase.
That clearly shows when a conversion tactic works wonders on some landing pages, it can still affect conversions on other websites.
When it comes to CRO, best practices have their place. But you shouldn’t rely on them to achieve repeatable, sustainable growth.
Some of the clients we have worked with have been adhering to conversion optimization best practices, but they still couldn’t see any improvement in conversions, and the ROI has been low.
When you stick to CRO best practices and still do not see your conversions improving, they probably don’t align with your audience preferences.
Every site has its own challenges. It’s better to take a deep dive into research and look into the site’s challenges than relying on best practices.
4. Effective Project Management is a Requirement
Launching one A/B test has its own complications. There are so many elements – designing, developing, QA’ing – involved in running a successful A/B test.
If launching one experiment is that complicated, now, imagine having to handle 40 tests in one month – that’s arcane!
You’ll need at least two or three project managers to oversee and guide your team members, resources, workflows, and even time. The agency side needs to have its own project managers, and the client also needs to have their own.
All project managers from both sides have to be involved in the CRO process from the first optimization step to closing the whole project. This is an important thing to do.
Did you just ask why?
Well, because when optimizing a website, you’re more likely to bump into other conversion opportunities, you’d need to have project managers who make sure that your team does not deviate from the CRO goal you are trying to achieve. And they also have to make sure that the business objectives and CRO goals remain aligned throughout.
After all, has been said and done, with the help of project managers, you will increase productivity and efficiency in your CRO program. You will achieve more in less time when your CRO project is well-managed.
5. Follow a process – but never set it in stone
Besides the high talent density within our CRO team, having a solid CRO process has helped us manage to achieve a 65% success rate in our A/B testing programs.
A CRO process will help you eliminate guesswork and move faster since you already know the series of steps you must take to get to a particular end.
There are several CRO processes used in the industry. A typical conversion optimization process includes the following:
- Qualitative research
- Quantitative research
- Prioritization
- Hypothesizing
- Testing
- Post-test analysis
At Invesp, we follow a well-documented SHIP process: scrutinize, hypothesize, implement, and propagate. Here’s what it looks like:
Even though we have a CRO process, we don’t always stick to it. Sometimes we modify it based on the length of a CRO project we are working on. And sometimes, the modifications are done based on the type of website we are working on.
If you have room in your head for only one CRO wisdom from this section, make it this one:
A CRO process is good, and it will help you move fast. But a CRO process set in stone is not. Always be ready to refine and reinvent your processes.
6. Align Business Objectives with experiments
Never launch a CRO program without aligning it with business objectives first.
That has to be a top priority.
One of the first things we do before starting any conversion optimization process is to sit down with a client (via Zoom, of course) and talk about the company objectives. We do this because we know that business objectives are a solid foundation to build a CRO roadmap.
Without proper alignment between your CRO program and business goals, it’s easy to get pulled in different directions, and you are more likely to end up producing results that do not help you achieve business goals.
Here’s an example:
We once worked with a client who was unclear on their business goals. All they wanted us to do was increase their website conversion rate. Their e-commerce website was small compared to their brick-and-mortar stores in terms of conversions. When we started working on their site, the conversion rate improved significantly – but then they started getting complaints from their retail stores because foot traffic was down tremendously due to the changes that we were doing on their site.
On the flip side, another client had their business objectives clear: they wanted us to increase conversions on their site and increase foot traffic to their physical stores. Knowing this business objective, we helped them introduce Buy Online Pick in Store (BOPIS) experience on their site, launched an experiment, and saw a 20% uplift. Their foot traffic on their store increased as well.
So what’s the point here? You want to make sure there’s an alignment and vision of what you are trying to achieve as an organization before you engage any conversion optimization agency.
So how do you make sure that your business objectives are aligned to your CRO goals? Well, this is what you do:
Set realistic business goals – Your business goals impact your CRO program. If you set realistic business goals, your CRO program is more likely to be successful. The reverse is also true. Conversion optimization is already a tricky process, don’t make it complicated by setting unrealistic business goals.
Create a CRO roadmap – The alignment between your business and CRO goals must be clearly defined. This means creating a CRO roadmap with tests that contribute to achieving business goals at large.
Communicate the goals relentlessly – Everyone involved in the CRO project knows the business goals and the CRO roadmap. You can’t align business objectives and CRO goals if you still have siloed approach.
Avoid distractions – When optimizing a website, there’s a possibility that you come across different conversion opportunities along the way. But always prioritize opportunities that help you move towards achieving your business goals.
If you want to learn more about aligning your business objectives with your CRO goals, we have written an article in this blog that explains all you need to know.
7. Stop Copying Competitors
An ancient adage says, “Imitation is the sincerest form of flattery,” but it’s not a good idea when it comes to conversion optimization.
When we do a competitive analysis, the goal is not to replicate what your competitors are doing but to differentiate your brand from them.
It’s okay to conduct a competitor analysis to find inspiration and spark ideas for your own. But it’s never okay to imitate what they have implemented on their sites.
If customers can’t differentiate between your brand and your competitors’, they will look at the prices to find the difference. You do not want your differentiation factor to be based on price. You want to create a copy that connects your users emotionally and shows how you stand out from the rest.
8. Don’t Take Shortcuts
A strategic CRO program that pays dividends takes a lot of hard work behind the scenes.
Before you even launch an A/B test, there’s a ton of work that has to be done. In fact, A/B testing is 10% of the process. Depending on the website, 70% to 80% of the CRO process is spent on research.
You don’t just want to launch a test to win (although that is satisfying), but you also run tests so that you can learn something from your users. Experiments that give you valuable insights are driven by a research-backed hypothesis.
It takes a lot of time, effort, and resources to launch an A/B test, so if you have the opportunity to run tests, make sure that you do it right. An A/B test that fails but was driven by a research-backed hypothesis is worth more than a test that wins without the hypothesis behind it.
Conclusion
Launching an A/B testing program can be exciting. But that excitement can quickly die down when what you were expecting to achieve doesn’t show up. Before you even start your A/B test, make sure that you have done adequate research and you QA your tests before they are rolled out to the production stage. Make it a point that each and every test you launch has a hypothesis behind it – this will help you learn something whether your test wins or loses.
