How to Find Great Optimization Ideas
Starting an A/B testing program is an exciting time. When you first start A/B testing, you’ll find a ton of opportunities for improvement and can quickly develop creative solutions. Easily identifiable issues are the easiest to solve, and when you fix glaring weaknesses, you may feel like an optimization all-star.
But what happens after you consume all the low-hanging fruit?
After you conquer obvious problems and get some great wins, companies tend to develop a huge hunger for testing. However, as you A/B test more and your website improves, it becomes more and more challenging to improve upon your improvements. Your win rate decreases and you end up with too many zero-sum A/B tests.
This optimization stuff stops being so easy and testing ideas that yield no results become incredibly frustrating.
Quantitative Data is NOT the Answer
Well, at least qualitative data by itself is not the answer. Initially, it was easy to dig into web analytics to find pages with high traffic and low clickthrough rates and then come up with ways to make it better. You could look at your hotjar or SessionCam reports and easily spot where people were clicking and not clicking and think of ways to improve your pages.
But guess what? That data did not give you answers or ideas. It only told you where to look. You and your team came up with the ideas and answers!
Your Audience Has All The Answers
We are all too close to our products and services. We know too much and have too many biases. This creates blinders that make it difficult for us to understand why issues exist and would could be done to improve them.
You will probably continue to come up with great ideas from time to time. But we're all limited by our own way of thinking, familiarity with our products and services and because we are only one person.
Thankfully, technology and the growing gig economy has given rise to a Internet-based User Testing. User Testing is a low cost way to find people in your target audience, present them with a scenario, and have them go through a series of steps while you record what they are doing and saying.
User testing shows how your audience interacts with your website, why they do/don’t click, and why some areas of the site fail to engage. You get startlingly relevant insights from studies conducted with an audience that mirrors your ideal customer. These studies move you beyond assumptions that undermine your A/B testing program. Simply stated, user testing is premium fuel for your idea tank.
Building Significance... User Testing Studies
One user test can help you identify many things to fix and opportunities to purse. However one person is rarely a good sample size. Therefore, it’s a good idea to run 5 – 10 user test about one specific journey. This is called a User Testing Study.
By having 5-10 different people in your target audience go through a focused journey, you will get varied points of view that will allow you to develop trends. The strength of these trends will make it possible for you to prioritize what to fix now and test later.
The results of a well-conducted user testing study are almost always startling. You will find issues you never would have thought were issues. Your audience will outright give you some great A/B testing ideas, and through observation, you will develop new A/B test ideas that would have never occurred to you otherwise.
You Will Need Some Help
The average single user test will run between 20-30 minutes. That means that a single study, with 10 participants, will produce between 200 – 300 minutes of video. This is a lot of video review, analyze, and synthesize.
It’s also important to document your study and build a plan around your learnings. These studies should contain a your findings, trends, critical improvements and A/B test ideas that will help you and your organization plan and take action. It’s the equivalent of putting together a focused business performance report from extensive data and analysis.
Without this study, you will just have a database full of videos that you’ll dread sifting through.