The LARGE Problem With A/B Testing
Full video: • Optimization is NOT Just About A/B Testing...
This is a clip from the first video of a series where we’re answering the question of how to optimize your website or digital experience if you don’t have a large amount of traffic.
A/B testing has rightfully earned its place as the gold standard for validating ideas and driving optimization.
Nothing is more trustworthy (well, meta-analyses of trustworthy experiments are right up there1) than adequately powered A/B tests, run from a balanced backlog, and rigorously checked for perverse incentives.
But what happens when you don’t have enough traffic to run a valid A/B test? This is a common challenge many teams face, especially those working on low-volume flows or niche products. Reaching statistical significance in those instances can take months.
Getting around the low traffic problem in statistically sound ways isn’t a walk in the park — you’d need a data scientist onboard. But that doesn’t mean we should abandon optimization altogether when A/B testing is off the table.
A/B testing is still the best way to mitigate risk. No other method can match the quantitative scale and rigor of A/B testing.
We must keep these limitations in mind when working with “lower-quality evidence.”
And above all, triangulation is key. Triangulation, in this context, means looking at your digital experience and data from multiple different angles to glean deeper, more informed insights.
If we want customers to improve their user experience successfully and thus their conversion rates — we have to lay out a viable plan B. This guide to optimization beyond A/B testing is just that — a responsible plan B for when traffic is low but the desire for improvement is high.
In this series, we'll explore the hierarchy of evidence, discuss how to triangulate multiple research methods and provide best practices for mitigating risk without controlled experiments.
In this video, we discuss the roles that quantitative (i.e. site visits, average session duration) and qualitative data (surveys, reviews, etc) have when layering evidence and using triangulation.
Full article: https://www.convert.com/blog/a-b-test...
Triangulation article: https://www.convert.com/blog/a-b-test...
_____________________________
CONVERT WEBSITE
⭐ Learn more about A/B testing and Growth Engines: https://www.convert.com/shopify-ab-te...
⭐ Learn more about Convert Experiences: https://www.convert.com/features/
SOCIAL MEDIA
⭐ Follow on LinkedIn: / convert-com
⭐ Follow on Twitter: / convert
⭐ Follow on Instagram: / convertinsightsinc
⭐ Follow on TIkTok: / convert.insights
_____________________________
Music provided by @PacificProduced, • Rockabilly 60s Inspired Type Beat x Hit Th...