Disclaimer: This section is a TL;DR of the main article, and it’s for you if you’re not interested in reading the whole article. On the other hand, if you want to read the full blog, just scroll down, and you’ll see the introduction.
- Some online retailers take weeks, while some take months to plan out their holiday strategy.
- It seems impossible to develop a conversion strategy in a few days, but we did so in 2019 when a client approached us.
- Seasonal events like Black Friday and Cyber Monday need to be approached thoughtfully, and you need all the customer data you can get to execute successful campaigns.
Why Do Companies Hate A/B Testing During Holidays?
- Most online retailers aren’t willing to run A/B tests during the holidays, and this is something we see with some of our clients.
- The fear of introducing new bugs into their site is one of the reasons why clients don’t like running A/B tests during the holiday season.
- Whether it’s a holiday season or not, there’s a possibility of new bugs creeping into your variations when A/B testing – especially when your A/B testing has to do with the website functionality.
- The second reason many businesses are skeptical of conducting tests during the holiday season is the fear of revenue reduction due to introducing new designs.
Is Holiday A/B Testing Worth It?
- Holiday A/B Tests come with their own set of risks and rewards
- Most online retailers experience an influx of visitors during holiday seasons, but it can be tricky to run tests if you’re not experienced.
Why You Should A/B Test During Holidays
- Because you get an influx of new visitors.
- You get a lot of comparison shoppers who are driven by product price and availability.
- If you’re not running tests, you’ll never understand what kind of offers drive your customers during the holiday season.
Why You Shouldn’t A/B Test During The Holidays
- A/B testing is not just about maximizing revenue – at its core, it’s about understanding how your visitors think and behave by analyzing the test results.
- The problem with running tests during the holidays is that the results might easily be skewed. You can’t rely on holiday test results to permanently change your website.
- Remember, during holidays; you’re dealing with consumers driven by the cost and inventory – this means that their behavior is different from your usual customers during the year.
Here’s a longer and more detailed version of the article
How far out do you need to plan your holiday A/B testing strategy?
Some online retailers take two weeks to plan. Some take a couple of months. And, others need a few weeks to prepare.
There seems to be no right or wrong answer to that question.
At some point, I used to think that it was impossible to come up with and execute a successful holiday testing program in few days.
That was before I witnessed our CRO team create and execute one three days before Black Friday.
Yes, it’s a true story!
Three days before the Black Friday of 2019, an eCommerce company signed up with us and tasked us to handle their holiday testing program.
Seasonally events – like Black Fridays and Cyber Mondays – need to be approached thoughtfully, and you need all the customer data you can get to execute successful campaigns.
“If they hadn’t provided us with insights on their customers and inundated us with a ton of information – we would not have pulled off a successful campaign. We also needed to compare the traffic and their activity from the prior year to gauge customer behaviors during this period.”
– says Ayat, who was also part of our CRO team that created and executed a testing program in 3 days.
In other words, If you’re entirely unfamiliar with the website, the market, and the customers, then your A/B testing program is more likely to be a disaster.
When it comes to holiday campaigns, it’s not about the promotions – but rather the marketing strategies will engage and persuade the visitor during and keep the visitor loyal post.
In this article, we will look at why companies hate holiday testing and the pros and cons of running tests during holidays. We are also going to touch on the dos and don’ts of holiday testing.
Why do companies hate A/B testing during holidays?
Most online retailers are not willing to run A/B tests during holidays – and this is something we have seen with some of our clients.
In our experience, there are two reasons why companies want to go through a code freeze during holidays:
Fear of introducing new bugs
You launch a test with two or three variations against the control. And you think that your testing tool is collecting correct data. You then discover that one of the variations is bugged – and it’s not working for Safari users.
Whether it’s a holiday season or not, there’s a possibility of new bugs creeping in your variations when A/B testing – especially when your A/B testing has to do with the website functionality.
In his article – about A/B testing sampling issues – Khalid writes:
“Bugs are typically introduced while coding the test variations. Since most tests are coded using JavaScript and not served from the server-side, there is a good chance that one of your variations might contain bugs.”
Fortunately, there’s a way to discover the bugs before the test is launched.
We usually use extensive Quality Assurance (QA) on a staging environment and addressing any bugs (if any) before moving the test to production.
In other words, no test has to be pushed to the production servers before the data is validated in a QA environment.
Risk of reduction in revenue as a result of new designs
Whenever you launch a test, you are not 100% sure that it will increase conversions. That’s why you have to test it first.
In other words, with every A/B test you launch, there’s a risk of decreasing conversions.
So some clients would instead go with the original design and not risk experimenting – especially during the holiday season where they’re expecting to cash in lots of revenue.
So, how can this be mitigated, you ask?
Well, the solution that we have for a lot of our clients is to do gradual testing.
For example, instead of directing 50% of the traffic to the new design, we start by 10% and 90% to the original design. And then we gradually increase or decrease the traffic according to how o how the new design is performing.
This way, our clients feel more comfortable running A/B tests during holidays.
Holiday testing – Is it worth it?
Most online retailers experience an influx of visitors during holiday seasons.
Does that mean you should take advantage of this seasonal spike in traffic and launch some tests?
Well, that’s a good but tricky question.
I have to say, testing during holidays comes with its own set of risks and rewards.
You have to be experienced in the field of CRO if you want your tests to be successful.
ALSO READ: What they don’t tell you about A/B testing
Why you should AB test during Holidays
It’s not every day that you get an increase in:
- New visitors to your site,
- Comparison shoppers, who are driven by product price and availability,
- Visitors from new or less popular sources (and sometimes locations),
- And average order and cart size.
– so, not running tests during holidays can feel like a missed opportunity.
You might also be leaving more money on the table by not testing during holidays that you could have got through new variations.
For instance, how do you know that your holiday offer is performing as well as it could be? If you are not running tests, you will never understand what kind of offer drives your customers during the holiday season.
In such periods of high urgency, you can quickly learn how your customers think, behave, and what kind of offers drive them to take the desired action on your site.
There are tons of A/B testing tools that have editor WYSIWYG that allows you to create variations without touching your site’s code – especially when the variations don’t change the page functionality.
Using the above tools, you don’t have to change the code to remove, add, resize, or even rearrange your web page copy.
Why you shouldn’t A/B test during Holidays
A/B testing is not just about trying to maximize revenue – at its core, it’s about understanding how your visitors think and behave by analyzing the test results.
ALSO READ: How to analyze A/B testing results and statistical significance
But the problem with running tests during the holidays is that the results might easily be skewed. This means that you can’t rely on holiday test results to make permanent changes to your website.
Remember, during holidays; you’re dealing with consumers who are driven by the cost and inventory – this means that their behavior is different from your usual customers during the year.
For instance, holiday shoppers tend to have a higher sense of urgency, lower bounce rates, and more engagement with particular items. They also tend to have more intent than regular browsing – meaning that there are more likely to purchase their first visit.
Most online retailers tend to increase their Ad spend during holidays to attract more customers. And this sometimes means bringing in an audience from new or less popular sources – this kind of audience is more likely to have a different mindset because they are not within your usual target audience.
During the holiday season, what might have worked might not be as efficient during the rest of the year. In other words, your winning test during the holiday might not be the winner when you test it on regular days.
ALSO READ: Holiday Marketing Strategies: 11 Tactics Every Small Business Should Leverage
So, relying on the insights obtained during holiday testing is something you should carefully think about. Before you think of using the insights you gained from holiday tests, we recommend rerunning your holiday tests on regular days to validate your results.
Conclusion: Yes, holiday testing is worth it!
Testing during the holiday is a good idea. You can still gain some valuable insights that you can use during high urgency periods.
All you have to do is put in place an A/B testing strategy and be mindful of the unique visitor behavior you will be dealing with.
Your testing strategy should also include conducting extensive quality assurance. You should also test high-impact elements – product page descriptions, value propositions, return policy, promoted products, etc. – and not any other elements.
Avoid testing variations that have to do with the functionality of your website, as they may cause some technical issues on your website.
ALSO READ: 10 simple ways to boost your holiday conversion rate
