Disclaimer: This section is a TL;DR of the main article, and it’s for you if you’re not interested in reading the whole article. On the other hand, if you want to read the full blog, just scroll down, and you’ll see the introduction.
- The only way to continually learn is through rigorous experimentation and documentation.
- But all too often, small startups are disadvantaged by frowning upon the documentation part or viewing it as an optional formality.
- Not documenting experimentation insights makes building a meaningful experimentation program challenging and impedes learning. Not to mention, it’s also good practice.
What Is The Goal Of Documenting Experiments?
- You can tell the maturity and culture of an organization by listening to why they document experiments.
- it has everything to do with promoting continuous and efficient learning culture.
- Insights and learnings should be the primary KPIs you use to track the value of documenting experiments.
Why You Should Care About Documenting Experiments
- If you have a high testing velocity, then it’s critical to document your experiments.
- You want to keep track of every test you run – whether it wins or not – because those records will make your life down the line, and they can even help you achieve better results in your future tests.
Benefits Of Documentation
- Documentation facilitates education and cultural buy-in.
- Documentation helps prevent the duplication of tests.
- Documentation upholds your standards.
- Documentation saves money.
How To Document Experiments
- Each experimentation document should have these sections; introduction, problem, hypothesis, experimentation design, results, and conclusion.
- Write in first person active or 3rd person passive. Always document using past tense.
Here’s A Longer And More Detailed Version Of The Article.
Most of us in the conversion optimization world know that we should always be learning.
And as any experienced optimizer knows, the only way to continually learn is through rigorous experimentation and documentation.
But all too often, small startups put themselves at a disadvantage by frowning upon the documentation part or viewing it as an optional formality.
Unknown to them, not documenting experimentation insights makes building a meaningful experimentation program challenging and impedes learning.
By documenting your A/B testing insights, you can ensure that you and your team constantly learn, improve, and can quickly reproduce results.
Not to mention, it’s just good practice!
So without further ado, let’s look at the benefits of documenting your A/B testing insights and how you can do it.
What is the Goal of Documenting Experiments?
You can tell the maturity and culture of an organization by listening to why they document experiments.
For some organizations, documenting experiments is an afterthought because it’s a practice that can’t be tied to revenue.
But that’s being myopic.
Mature organizations that are pushing the needle understand that documenting experiments have nothing to do with revenue but it has everything to do with promoting continuous and efficient learning culture.
Insights and learnings should be the primary KPI you use to track the value of documenting experiments.
Why You Should Care About Documenting Experiments?
Tell you what, Although it might not look like it at the time, documenting your A/B testing insights is one of the few things that you will be glad that you did.
This process of documenting experiments is even more critical when you have a high-testing velocity program.
It can be challenging to keep track of what changed and why when running multiple tests simultaneously.
Without good documentation, it will be very hard – if not impossible – to determine which changes led to success (or failure).
You want to keep track of every test you run – whether it wins or not – because those records will make your life down the line, and they can even help you achieve better results in your future tests.
Still on the fence on whether or not to document your experiments? Here are a few key benefits that will convince you:
Documentation Facilitates Education and Cultural Buy-in
Proper documentation of experiments facilitates education, cultural buy-in, and transparency.
In other words, documenting experiments truly unlocks the power of experimentation by breaking down silos and allowing different departments to have access to insights.
When barriers between departments are broken down, the quality of decision-making of everyone is increased, as well as the collective knowledge of everyone.
Therefore, this empowers multiple departments to be more collaborative, strategic, and take actions that support the company’s overall strategy. This is how it can play out:
- For example, developers receive feedback from a CRO specialist that customers abandon the cart because the website is loading slowly.
- The developers address that by eliminating the bug causing this issue – improving website speed.
- This improvement is then communicated to the following departments: UX designers who make sure that the changes don’t affect the customer experience;
- Marketing team who then retarget customers who abandoned their carts;
- Customer support team who delight customers with such improvement.
- Community managers who spread this development in brand communities.
Documentation helps Avoid Duplication of Tests
I bet, at some point, you were planning to work on a new project only to realize that one of your team members has done that before.
One of the main pitfalls of not documenting your experiments is that it can lead to repeating experiments that have already been done. And that means a waste of precious time and resources.
According to Effective Experiments founder, Manuel Da Costa:
“History is important because those who know history are not deemed to repeat it. The whole purpose of documenting experiments is the same, you want to use that information to learn from it, to really understand whether what you have done has moved the business forward or not. Documenting tests helps build the history of innovation of an organization. Where it starts; where it progresses; and where it continues.”
But by documenting your experimentation insights, you don’t always have to start from scratch and make all the mistakes. You can just build on the experiments that have already been launched.
Expert Tip: Experimentation insights should be preserved, so newcomers can build on the company’s past rather than reinventing the wheel.
Documentation Upholds Your Standard
When everyone follows the same process, consistency is almost guaranteed – meaning the quality of experiments improves.
You want to ensure consistent results, whether you have a high- or low-testing velocity.
As mentioned earlier, documentation promotes knowledge sharing, empowering your team to understand the process and finished experiments.
With adequately documented experiments at their fingertips, your team members don’t need psychic powers to maintain the consistency of repeated projects.
They still have room for creativity while confirming that they are ticking all of these must-have boxes.
Documentation Saves Money
Earlier on, I mentioned that the ultimate goal of experiments or documentation is not about increasing revenue.
But we can’t deny that documenting experiments can help you save money.
How, you ask?
Answer: By helping you avoid the risk of running meaningless tests that do not contribute to the overall company’s objectives. Remember, experiments are not free. They cost a lot of resources.
How to Document Experiments
Manuel Da Costa believes that as you document your experiments, you should seek to answer two questions:
“First, why are you doing this exercise? Because if you’re just documenting as a tick-box exercise, the quality of the input will be compromised.
Second, who are you documenting it for? Most practitioners think they are documenting it for themselves. And that might be true in organizations where senior management is not involved with experimentation.
But in organizations where experimentation is strategic, the team is documenting information for the consumption and use by others in the organization and not themselves.”
The format used to document experiments is largely a subjective preference. However, Manuel says that “there are two ways that companies go about documenting experiments. The first one is firing up a spreadsheet and then putting the information about that experiment. This process is fine, but it’s not useful in the long run.” He also goes on to say:
“Good documentation of experiments organic and it’s not a once-off event. It’s something you should also include the input of designers, developers, and everyone else involved in the experiment.”
Each experimentation document should have the following sections:
- Introduction or Background
- Problem
- Hypothesis
- Experimentation Design
- Results and Observations
- Conclusion
Pro Tip: Write in 1st person active or 3rd person passive. Results are written in the past tense.
With that said, now let’s take a look at what each section includes:
Introduction or Background
This is the section where you explain the experiment’s premise so that whoever goes through the document doesn’t feel disoriented or confused.
However, as you inform the reader what the experiment is about, don’t bury the reader in too much detail – just prove that you clearly understood why the test was done.
You don’t have to wait for the experiment to conclude before preparing this section. In fact, this section can be drafted in advance of even executing the test.
Problem
Although you might have touched on this in the introduction, the purpose of this section is to put the problem you intend to solve in context. A concise and concrete problem section should:
- Put the problem you intend to solve in context.
- Describe the exact problem that the experiment plans to solve.
- Indicate the relevance of the issue you intend to address.
- Set the objectives of the experiment.
Here’s an example of a problem section:
“Based on the analytics analysis, (X%) of the visitors who reach cart page XYZ have been exiting the page without completing the checkout process. We will examine the presentation of the trust elements within that cart page to determine whether [Element X] is prominent enough to alleviate visitor FUDs (Fears, Uncertainties, Doubts), and if not, which other trust elements we can use to reduce FUDs.”
Of course, your problem section doesn’t have to be lengthy as the one above. What’s important is to answer the following questions:
- What is the audience for this test?
- What elements/pages do you need to optimize?
- What is the supporting data for your assumption?
- What is the data source?
- In which aspects do the experiment aims to provide insight?
Like the first section, this portion can also be documented before you even execute the experiment.
Experimentation Hypothesis
An experimentation hypothesis is a research-backed statement that explains an observed trend and justifies the solution used in the experiment.
ALSO READ: Expert Advice on Developing a Hypothesis for Marketing Experimentation
The hypothesis section makes an educated prediction of what will happen in the experiment in terms of the outcome.
It should also indicate why experimenters think the changes in the variation will breed better results than the control. That is important as it will make it easy for others to understand the big picture and see why specific changes were made.
Experiment Design
This section about the experiment structure should discuss how the whole test was designed down to the last detail. The idea here is to have enough information that someone can use to replicate the exact experiment.
So, this means displaying the correct data set, audience segments involved in the experiment, the sample size used, test duration, when the test was launched (date and time), statistical significance, the number of variations involved in the experiment should be called out.
ALSO READ: How to calculate A/B testing sample size
If possible, the screenshots of the variations should also be displayed to highlight how they are different from the original.
Everything that happened during the tests should be mentioned in this section. Were there any changes – unforeseen changes in traffic volume from a given traffic source – that you may have noticed while the test was running? Were there any false positives recorded while the experiment was running? Were there any signs of validity threats seen during the experiment?
Results and Observations
An excellent place to start for this section is to restate the experiment’s aim and objective and prove that the test was carried out with no validity threats.
ALSO READ: Validity threats to your A/B test and how to minimize them
In case there were validity threats noticed during the experiment, they should be mentioned at the beginning of this section, along with how they were fixed in a way that doesn’t pollute the test outcome.
Having done that, you will then go ahead and explain the data you gathered and the observations made.
The results segment gives you the opportunity to:
- Display the raw, unprocessed data as it is and,
- Report on the observations from a relevant, unbiased, and appropriate interpretation.
So as you display your findings, feel free to inject screenshots and reports from the testing tool used to conduct the experiment. As you present the results, you can compare and state your observations.
Conclusion Section
Having zoomed in on the problem, you now have to summarize your overall observations and exciting takeaways from the experiment. It is in this section that you make it clear whether or not the problem you were seeking to address through your experimentation was solved or not.
Parting words…
For conversion optimizers, reliable documentation is always a must. You don’t have to use fancy tools to document your experiment. Don’t let perfection be the enemy of progress.
What’s important is to make sure that your team is well trained to keep track of the documentation appropriately and understand the importance of doing so.
Let me sum up by using this quote by Manuel:
“Documentation must be a priority that a Chief Marketing Officer must enforce on practitioners if they want to run a meaningful focussed experimentation program. The second priority is how to ensure that not only does everyone have access to it, but can utilize it (and that is also tracked)
Why?
It’s company IP.
Practitioners are given the mandate to run experiments but really the mandate must be about providing insights that create business impact. And it all starts with governance around the data.”
