A/B Testing: Lipstick On a Pig?

Are you putting lipstick on a pig?

So, your company wants to increase revenue. They want more conversions, more clicks, more shares, and more users. What do they tell you to do first?  Well, A/B test! Compare two versions of a page, define a key goal (ex. clicks), and see if you get more clicks. But, does this actually work?  Is it really the approach you should take? Let’s look at the data.

For apps with millions of users, small cosmetic changes to your app, like color, layout, and language could net you some noticeable increases in your key metrics, like more clicks and user engagement. The real question is this: for companies with smaller user bases, should you try to focus on different “Sign Up!” button colors or focus on actually making your product better?

A/B testing for many companies becomes a band-aid for poor value propositions. If your content is not being shared, maybe your content is actually not share-worthy, regardless of how amazing you make your “Share Now!” button.

Survey Data

This data is from a qualitative and quantitative A/B testing survey of 26 A/B testing practitioners conducted from May 1 to May 30 of this year. While this is not the end-all be-all of surveys, it could still give us some meaningful insights.

Key Points

• 45 percent of respondents said that their companies start a new AB test once every 3 months, with 40 percent starting one per week or month;
• 60 percent of respondents said that their AB tests last between 1 week and 1 month;
• 38 percent of respondents said that less than 10 percent of their AB testing experiments actually translate into action; and
  
• 45 percent of respondents said that application changes from AB testing persist for 1 to 6 months.

Interview Data

To supplement the quantitative research, A/B testers (from 2 to 6 years of A/B testing experience) were asked open-ended questions regarding the efficacy of A/B testing. Here are some of the key take-aways:

• 10 of 12 interviewees noted that a major drawback of A/B testing was that 90 percent or more of experiments ‘failed’;
• 6 of 12 interviewees noted that it was very difficult to make product decisions based on AB testing results since most results were inconclusive or clearly failed.  This means that the status quo was maintained roughly 90 percent of the time; and
• 10 of 12 interviewees noted that the primary benefit of A/B testing was to ‘increase revenue’.  

Making A/B Testing Useful

Overall, what does this data tell us?  Companies may be running A/B tests too frequently for too little time, contributing to a high failure rate that makes A/B test results less valuable and meaningful.  Here are some tips to help make A/B testing useful for your application.

1. Don’t get distracted
   • Changing colors, call-to-action text, and layout will have marginal impact on your key performance metrics.  However, these results seem to be very short-lived. Sustainable growth does not result from changing a button from red to blue, it comes from building a product that people want to use.
2. Don’t put lipstick on a pig
   • The better version of two pigs is still a pig.  If you are trying to sell a pig, then you are doing great.  If not, then focus on creating better user experiences and a better value proposition.
3. Use actual statistics
   • Do not rely on simple 1 on 1 comparison metrics to dictate what works and does not work.  “Version A yields a 20 percentconversion rate and Version B yields a 22 percent conversion rate, therefore we should switch to Version B!”  Please do not do this.  Use actual confidence intervals, z-scores, and statistically significant data.
4. The longer, the better
   • The longer you run your test, the better your data will account for fluctuations and extraneous variables.  Do not run a test over memorial day weekend using a red/white/blue theme and then switch to that theme for the rest of the year.
5. Failure is okay, but failure is expensive
   • If you keep releasing versions of your application that people hate, then what impact is that having on your metrics?  If most experiments fail, then are you doing more harm then good?  How much time are you spending designing and implementing A/B testing experiments?  Failing and experimenting are natural byproducts of building a company.  If something is not working, maybe it is not because your button needs to be a brighter color, maybe it is because you need to make your feature better.