Should you run the same test across all devices?
Or let me rephrase, should you be A/B testing on mobile or desktop separately, or not?
Well, the correct answer is of course that you should run separate tests on mobile and desktop.
That is a no-brainer…
But when you answer that question, the way I did, you generate another smaller, tangential question such as:
Why should you A/B test on mobile and desktop separately?
Now, there are dozens of reasons why you should test separately. In this article, I will give you five reasons:
User Behavior Varies by Device
It’s no secret that mobile and desktop visitors exhibit different behaviors. Most people tend to browse using their mobiles and complete the purchase using their personal computers.
Understanding users’ motivations for visiting your site can help you with insights that may be used as the foundation for how to optimize different experiences on your website.
Even if both mobile and desktop users use the search on your website, how they go about it is not similar.
In general, mobile device users don’t want to spend time idling on your site. Their aim is to get things done as efficiently as possible. But that’s not the case when it comes to desktop users – they are more open to having more elements on a website.
These nuances show that user behavior is something that is highly dependent upon the type of device being used. It’s essential that you understand your audience’s preferences then come up with an A/B testing strategy for each device type.
What works for Mobile doesn’t necessarily work for desktop
This point sounds like a bleed-off from the previous section. But, here me out, it deserves a whole section of its own.
Some of the web elements that help increase conversions on desktop may actually affect conversions on mobile devices. In other words, if it works for desktop, it doesn’t necessarily mean that it will work for mobile.
This is the reason why you see some sites – after being optimized for conversions – having different experiences on mobile and desktop. For instance, when you are on Facebook using your desktop you will find that you have too many choices than when you are on a mobile device.
Over the years, we have seen winning designs for a mobile test being very different from the winning design of the desktop test of the same page. This, therefore, means that you will have to serve two different experiences – on the same website – based on the device type that the visitor is using.
Having two different web designs – mobile and desktop – for the same webpage can lead to development problems. This is why some companies tend to avoid being in this situation by using a similar design. But that doesn’t necessarily solve the problem, if anything, it ignores it.
Traffic Varies By Devices
Recent research indicates that mobile usage continues to increase (with no signs of slowing down) at a faster rate when compared to desktop. More users are using their mobile devices to surf the web, shop, and make purchasing decisions.
Mobile and desktop traffic are never the same. Some websites have more traffic coming from their mobile devices and less traffic from desktop devices. But, for some websites, it’s vice-versa.
So, when you run the same test, at the same time, on both mobile and desktop, how are you going to split the traffic evenly? Take a few seconds to think about it…
You may have enough sample size combined, but when you look at the data of your test, you will notice that one device type contributed more traffic than the other.
And this means that at the end of the day, whatever result you have, it won’t be specific to a particular device. It will be on average – and as you may (or may not) know, you can’t make decisions based on ‘on average’ results.
In trying to come up with a solution for this, people end up extending the test duration so as to get enough sample size on each device. When this happens, you will be eating into time that you could be using to prepare for other tests.
Less Development and QA time
When you run an A/B test on desktop and mobile devices at the same time, this means more development and QA work for your team. And suppose the test is complicated, you will need more development and QA time. When this happens, you end up eating into time that could have been working on other tests.
But if you test separately, on desktop and mobile, the development won’t be as complicated and the QA time spent on each test won’t be that much.
Mobile traffic purchase behavior varies
Mobile users are now willing to spend more and more money. In Q1 of 2019, Paypal’s mobile payment volume increased by 52%.
Recent research also shows that in the last six months, 79% of smartphone users have purchased something online using their mobile devices.
And according to Google, 20% of smartphone users make a purchase online on a daily basis.
There’s no denying that the number of mobile users who are using their smartphones to purchase is still increasing even to this day.
But when you take a deep dive into the purchase behavior of mobile users, you will see that it varies. Even though Android users are the majority on most websites, research shows that iPhone users tend to spend more than any other smartphone user.
What we have seen over the years is that what works for iOS doesn’t necessarily mean that it will also increase conversions on Android or Windows mobile devices.
Perhaps, the best way to solve this is to segment your users by their mobile operating system and target them separately or even better, do a post-test analysis and check how each operating system performed.
Another segmentation to think about is to test mobile devices using wifi and mobiles on 5G/4G separately as that could show different buying behaviors.
Conclusion
For reasons mentioned in this article, it’s always recommended to test on mobile and desktop separately. Doing so will make your life easier and you will be able to run more and more tests without any development complications.
Additional Resources
