Home » The Ingot Blog » Posts » How Much Traffic You Need To Test (Hint: It’s Less Than You Think)

How Much Traffic You Need To Test (Hint: It’s Less Than You Think)

do you need high traffic for ab testing

If you’re just starting a website for your business or cause, you’re likely short on time, and are having to prioritize between all the things you “should do” versus reality. A/B testing is one of the many tools you should be using. When deciding when to use A/B testing, the first thing you might consider is what type of testing you are doing and what kind of traffic you have.

A/B Testing Basics

A/B testing is a bit of a generic term, and encompasses a lot of different types of technologies and methodologies. It can be difficult to take your knowledge that you “need A/B testing” into actionable tasks. Additionally, the different recommended ways of doing A/B testing require a different amount of traffic. Most A/B testing uses an algorithm that would be very familiar to you if you remember statistics 101.

Old Ideas on Traffic for A/B Testing

I’m going to skip the long statistics lecture, but the end story is this: for traditional A/B testing, the answer is you need a lot of traffic to get statistically meaningful results. Until you reach a statistically meaningful sample, which could be tens of thousands if not hundreds of thousands of pageviews, the different variants in your tests are being chosen at random.


That’s a problem for most new businesses that don’t have hundreds or thousands of pageviews a day. Even if you did have that, if you put a bad test variant into your test, you’re going to have to run it tens of thousands of times — that’s a lot of lost sales.

New Ideas on Traffic for A/B Testing

One alternative method of performing A/B testing, really multi-variant testing, is the “multi-armed bandit” strategy. In this method, you only choose a random variant a certain percentage, normally 33% of the time, and the rest of the time, you choose the best performing variant.


This creates a balancing act, where all of your variants are in a race, and slowly the bad performers fall behind. Over time it goes from being an educated guess, to being a highly accurate model.


Ingot uses a variation on this strategy. We add a short period of randomness at the beginning of the test, and Ingot also will eventually disable under-performing variants.


This “greedy” strategy pulls the winning alternatives to the front of the pack quickly so you don’t lose conversions while testing.


We chose this strategy both for the quality of results — again, this is not a “nerd out on statistics article” but trust me this is a sexy algorithm — but also because it works with any amount of traffic. We released Ingot because of our experiences running a small online business. Improving the conversion rate is everything, but our sites never got huge traffic. We know that entrepreneurs like us need something that works right away.

Mini Case Study: Our Team’s Former Experiences

Sounds great, right? However, don’t forget that the system does get better the more data you give it, especially a system like Ingot that is also gathering a ton of analytics data on conversions and failed conversions. So, not only can you start right away, but also, the sooner you get started, the better.


When I started my first online business, I didn’t put Facebook and Twitter tracking pixels on my site from the day I launched. I couldn’t afford to do pay per click advertising and I wasn’t going to anytime soon. When I was ready, I realized that to target my ads properly, I needed to use the data gathered from my tracking pixels, which I didn’t have yet.


Time travel continues to not be possible, unfortunately.


That’s OK, but I’m glad that I didn’t make the mistake that so many businesses do — only thinking about advertising when they could afford it. Online advertising is a data-driven enterprise. Pretty much all online advertising platforms are driven by the accumulated data from tracking pixels.


By the way, you do have those pixels installed, right?


Anyway, we learn from our mistakes. Ingot is all about using every interaction on your site to gather more data to do better in the future. That’s the best part about data-driven web development and sales strategy — you can’t turn every prospect into a sale, but with the right analytics collection, you get value from every single interaction you have. 


There is one comment

Join the conversation

Your email address will not be published. Required fields are marked *