The Anatomy of a Great Test Plan
Updated: Nov 8, 2018
There are two types of A/B tests for website and product updates:
Those that change something minor and may or may not have a small impact
Those that change, remove, or introduce something critical that may have a big impact on user experience or key metrics
In a culture that encourages experimentation, it’s important not to create excessive process that hinders velocity. Therefore, when it comes to minor tests that require minimal effort, a simple one-page brief about the experiment is enough to build and launch it.
On the other hand, when it comes to launching something that can have a major impact or takes significant effort, having a good test plan is critical to having a successful launch. Failing to create a thorough plan may lead to miscalculating the impact, spending more time than necessary on development and revisions, or measuring the wrong things.
With a good plan in hand, building enhancements and getting to market is much easier. If the test is a big success, then your methodical test plan will give others confidence and you’ll be able to implement the new features permanently. And if your test is not a success, you’ll be justified in running it, gain new data to help you learn and improve the results of your research moving forward.
So, you may ask: “what does a great test plan include?”
I’m glad you asked! But briefly, it should include the following sections:
Opportunity & Solution
Audience & Ramp Plan
Team and Responsibilities
1. The Brief
The test plan brief should contain a summary of the what, why, where, who, and when. The intention is for someone to be able to read this single page and understand the test at a high level.
Each of the following items should be included in a sentence or two max within the brief:
What is being tested? – The feature that is being introduced, removed or changed.
Why are you doing this? – The challenge you are trying to overcome or opportunity you want to take advantage and why you think this update will have a positive impact.
Where will this be launched? – The page or pages and locations on these pages where the launch will occur.
Who will be affected? – The audience segment(s) that will experience your changes.
When will the launch occur? – The launch date, and if possible, the end date… or at least the estimated end date for your experiment.
2. Opportunity and Solution
Building and launching website and product features takes significant effort and resources. Even when you are launching a test of temporary prototype, you still need planning, design, development, QA testing and analysis. Therefore, whatever you decide to test should be tied to a real opportunity that was supported with sound research and data.
This is the section to clearly explain the following:
The issue or opportunity – What issue are you trying to improve? Or what opportunity are you trying to gain from? Explain the process you took to discover the issue/opportunity, and data (quantitative or qualitative) that supports this being a worthwhile undertaking.
Your solution – Explain the change you are making that will maximize this opportunity or fix the issue you have discovered.
Your hypothesis – Provide an evidence-backed explanation of what you think is going to occur when you launch and why you think it will occur. Evidence can come in the form of quantitative analysis, qualitative research, or examples of something similar working for other products or websites.
3. Audience & Ramp Plan
This section will detail who will be in the test at any given time. I like to break this down into the audience and the ramp plan.
Given the various devices people use and data companies have about visitors, there is no reason why you should ever launch the same experience for everyone. Therefore, it’s important to define who your audience will be. Here are some examples of ways you can define your audience:
Device type – Desktop, mobile phone, tablet, TV, etc.
Geography – Continent, country, state, county, city, zip code, etc.
Behavior – After visiting certain pages or triggering a string of key events.
Time – Morning, evening, weekdays, weekends, etc.
Browser – Chrome, Firefox, Internet Explorer, Safari, etc.
Other – There are countless ways to segment people. Make sure that you understand which segments you will be targeting and put this in your test plan.
Additionally, it’s important to have a plan for ramping your changes because it often doesn’t make sense to ramp your test experience to 50% right away. Especially when you have a high-volume website, the stakes become really high. It will probably make sense to begin with a smaller ramp and work your way up after collecting enough data to know that your changes won’t hurt your business in a meaningful way.
An example of a ramp plan could be the following:
Days 1-2, ramp to 5% – If results are not detrimental to key metrics, continue with next ramp. Otherwise stop test and identify ways to improve performance before restarting.
Days 3-4, ramp to 10% – If results are not hurting key metrics in a statistically significant way, continue with next ramp. Otherwise stop test and identify ways to improve performance before restarting.
Days 5-7, ramp to 25% – If results are not hurting key metrics in a statistically significant way, continue with next ramp. Otherwise stop test and identify ways to improve performance before restarting.
Days 8+, ramp to 50% – If results are not hurting key metrics in a statistically significant way, continue until significance is reached or your timeframe is ends. Otherwise stop test and identify ways to improve performance before restarting.
The days and percentages will vary based on your needs.
4. Measurement Plan
Your measurement plan details the key metrics you will be measuring, where the data will come from, and any new tracking that needs to be implemented as part of the launch. It’s important to think of this ahead of time to ensure that all the tracking is in place to collect the right data, so you can create actionable reporting.
The following are types of key metrics that should be measured:
Primary metric – This is the metric you expect to be most heavily affected by your launch. An example of this can be click through to the next page or next step.
Secondary metrics – These are other metrics that could be affected by your launch.
Counter metrics – Counter metrics are metrics that move in the opposite direction of your primary or secondary metrics. For example, if you get more people clicking from your homepage to your promotions page, you may have fewer people clicking to your bundles page.
Business metric – I call this the “veto metric” because it’s the one that you can’t risk hurting. An example of a business metric could be marketing qualified leads (MQL). Taking this example, if you increase your homepage click through rate by 100%, but decrease MQLs by 10%, the negative impact on MQLs will veto your positive impact on Homepage click through rate.
Don’t forget to define where all of this data will come from. It will be important to know this so that you can get the right data and your results can be replicated easily by someone else.
Sometimes the metrics that you need to measure don’t exist yet and will need to be implemented. This is the section where you want to call that out so that your team can implement the right tracking.
5. Functional Requirements
Functional requirements detail how any new or updated elements should function under various circumstances. This is a pretty broad category because depending on the elements and features you’re introducing or changing, it could be very different.
Examples could include what happens with you hover over an element, click an element, scroll on the page, fail to take a desired action, etc. The important thing to consider here is to provide your engineering team with a clear vision of what should be built so that they have enough detailed information to build it as expected.
It is often helpful to combine functional requirements with wireframes so that your engineering team can visualize what they are going to build. A slick way to do this is to annotate the wireframe itself with numbers or letters in the appropriate places, then listing the requirements for each number or letter off to the side. It’s like creating an instruction manual!
Wireframes are rough visual representation of the new experience. There are varying opinions about how a wireframe should look. Some people create wireframes that are void of all design elements and only illustrate layout and structure. Other people create wireframes that look a lot like what the finished product will be.
Example of Low Fidelity Wireframe
Example of High Fidelity Wireframes
Regardless of the approach, you should create some wireframe that helps visually communicates the idea you will test. This wireframe will then go to a designer which will turn it into an on-brand experience that you would be proud to have on your website.
Even if you include wireframes in your functional requirements, make sure you have a dedicated section for wireframes in your test plan where they can be reviewed thoroughly. They will be the basis for the new experience, so making sure your engineering and design team understand them well is important.
7. Team and Responsibilities
Launching new website and product features, even in the form of a test, requires many different skill sets. This means that you will be working with a cross-functional team to get things done.
It’s helpful to include each team member and their responsibilities in your test plan. That way everyone is clear on who is involved and what they are doing, and it becomes easier to reach out to the right people when necessary.
A lot of times when we uncover a great a opportunity, we’re tempted to jump in and start building immediately. I’ve been there and understand the temptation. However, if you’re able to avoid this urge and take the methodical approach of planning before building, you will be better off for it.
Feel free to reach out to me for any questions… always happy to chat!