What Types of A/B Tests are There?
How many times have you heard someone use that dreaded blue button vs. yellow button example when explaining the value of A/B testing? Yes, it’s an easy-to-understand example, but it really undersells the power of A/B testing and isn't very creative.
In an attempt to add more examples to our idea bank, I’ve created a list of some popular types of digital A/B test experiments. I also welcome my fellow experimenters to comment and contribute with their own examples.
#1 Complete redesign tests
I’m going to say it, and I’m not ashamed to say it: I really don't like site-wide redesigns. I think they often turn into creative-expression exercises that are justified with light research, and they tend to fail. And even when extensive research is conducted by some really smart people and really talented user experience professionals create them, they still tend to fail. If you’re curious why they tend to fail, check out my other post: “Avoid the Redesign Trap.”
Now that I’ve gotten that out of my system, I’ll admit that sometimes, but not very often, they are necessary. In a redesign test, everything is fair game and can be changed. You can change, move, remove, and add whatever your heart desires in the name improving your digital experience.
Pros: Redesign tests are good for when you have something that you know is a bad experience and needs to be completely replaced. At that point having incremental optimization is not going to cut it. Sometimes what you need is to start over and not to optimize something that is bad. In other words, as my friend Mike King once told me, “you can shine a piece of shit every single day, but at the end of a month you’ll simply have a really shiny piece of shit.”
Cons: When you change too many things at once, it’s practically impossible to learn what helped or hurt your digital experience. You may have used a new hero image that increased conversion by 10%, but changed an important description that decreased conversion by 15%. Since you changed both at once, all you would know is that you had a decrease in conversion of 5%, which could cause you to disregard an awesome hero image. It’s a shotgun approach that I often compare to high-stakes gambling. Redesigns can be great when executed well, but they come with a lot of risk, are difficult to plan around, and you always give up a lot of learning. Use them sparingly.
Tip: If you’re going redesign your entire experience, make sure you maximize this opportunity by doing a lot of due diligence in understanding what your visitors want. Don’t just assume that you can think like a visitor. Have an experienced analyst dive into your web analytics to understand how visitors use your current site and develop some actionable insights. Conduct some usability testing. Go through common use cases. And most importantly, optimize while you slowly ramp your new experience – don’t ramp it to too many people until you know it converts at least as good as your old website.
#2 Description tests
Words matter, A LOT. Changing what you say in a page title, photo caption, call to action, or product description can have dramatic effects with practically no creative assets needed.
Pros: Description tests are a great way to incrementally improve a page that you don’t think is that bad. Additionally, since most of these experiments involve testing some kind of HTML text, you’re unlikely to need a designer or heavy web development. They are also pretty easy to implement because changing text usually isn’t too technically complicated.
Cons: Usually, people don’t take the time to read on the Internet, and a common occurrence with description tests is that you may see no conversion difference. In order to make an impact, you need to make sure that you are testing descriptions that people are actually reading.
Tip: Look through your web analytics to identify any keywords that may drive significant traffic to your page or look through your own internal search from that page to identify any popular keywords. You are more likely to get visitors’ attention if you use keywords or phrases that you know people are interested in.
#3 Promotion tests
In promotion-centric tests, you can experiment with different prices, promotions, and the way you position promotions to determine what visitors respond to and how much they respond.
Pros: Promotional tests can be very useful in determining an optimal price for your products and how to best position promotions. They give you the freedom to try promotions on a sample population and ensure that you offer the most effective one to the entire population.
Cons: Promotional tests where you offer discounts can be trickier to interpret. In most cases, larger discounts create higher conversion rates, but if your discounts are larger than your conversion increases, you may start eating into your profits. It takes a little more detailed monitoring and analysis to ensure that you are making a good business decision. In addition, you can really upset your potential customers if you’re not careful. Be wary of offering different pricing to different people. Some big companies have received a lot of bad PR for these types of tactics.
Tip: Try the same discounts to everyone, but positioning them differently in your test variations. You may find that your customers respond more to a percentage discount (e.g. 25% off) compared to a flat dollar discount (e.g. get $50 off $200). It’s also possible that visitors are insensitive to $9.99 off but are more likely to convert if you offer $10 off.
#4 Image tests
Sometimes just using a different image on your page can make a big difference. If you’re a vacation-planning site, lifestyle images of people having fun may work better than a nice shot of the beach. On the other hand, if you’re selling furniture, visitors may want to see the piece of furniture by itself.
Pros: Imagery is something that people actually pay attention to, and sometimes the right image can really move the needle in right direction.
Cons: It’s easy to get lazy with these experiments and let your subjective judgment dictate the images you test. As with any test hypothesis, maximize the likelihood of a successful test by using research-based insight to determine your imagery.
Tip: Many times, less is more. If you have a page with 3 smaller images, try testing one bigger image that makes an impact. When there are too many things competing for your attention, nothing stands out. Also, there are plenty of simple tools out there such as EyeQuant or Attention Wizard that can help evaluate how your images may impact users’ ability to focus on your key value propositions and CTAs.
#5 Design tests
This is where you try to optimize by testing color, fonts types and sizes, shapes, or layout. These aren’t complete redesigns; rather, it’s changing certain aspects of your design.
Pros: Design-oriented tests not only help improve your conversion, they also allow you to test different design prototypes to ensure they don’t hurt conversion before you release them to the world. Remember that avoiding the implementation of something that hurts conversion is even more important than finding something that improves conversion.
Cons: Design-oriented tests usually require help from a designer, and depending on the complexity, a web developer. This often makes them more resource heavy, political, and slower to produce. Additionally, finding evidence to support that changing things such as button color or font styles will increase conversion may be difficult, so you’re sometimes left with creating less data-driven test variations.
#6 Targeting tests
Many will say that targeting is not the same as testing. I agree, but you can test your targeting approach to ensure that you are being as effective as you can be. For example, your targeting may be based on a referring keyword, but conversion rate may be higher if your targeting is based on visitors being Mac users.
Pros: Targeting tests can really help you capture hard-to-reach customers by fine-tuning your personalization tactics. These types of tests are the next level, once you have harvested all the low hanging fruit with your other testing.
Cons: Testing different targeting tactics is not as straightforward as testing visual items and requires a pretty deep understanding of your visitors. You will also need to invest in a tool that allows you to target because some of the more basic A/B testing offerings may not have this capability.
Tip: For the most part don’t listen to anyone that says they tried targeting and it didn’t work. Except for rare situations, targeting will help increase conversion if you find a recipe that resonates with users. Like I mentioned in the cons section, good targeting requires a deep understanding of your visitors, but when you figure it out, you can really provide a great experience and improve conversion.