Display ad testing protocol - Minimum impressions for A/B testing

Published by: Sam Karow ,
Published on: February 26, 2014

What are the minimum impressions to run in order to have a stable sample for a display ad creative A/B test?

Or, should a minimum number of orders or conversions be the metric, if the goal of the campaign is orders?

We can't seem to find any research on this topic. We may go with wanting to see 50 orders on each ad prior to making a call on which ad performed better.

Thanks for your help!



I found a couple of calculators on this subject of statistical significance in display ad testing.

a transparent breakdown of A/B Testing minimum sample size. Great if you like to understand all the formulas:
Scott Yacko's site: http://vuurr.com/split-testing-determine-sample-size/

the second one i found is based on responses, orders and conversion rate. there's an excel calculator you can download:
Avinash Kaushik's site: http://www.kaushik.net/avinash/excellent-analytics-tip1-statistical-sign...

ad headline testing in Paid Search, based on impressions, clicks and CTR. Excel calculator can be downloaded:
Serge's site: http://www.pointit.com/ad-copy-testing-for-statistical-significanc/

Hope these help. Happy testing!

Subscribe to: Receive notification emails for replies to this post.It makes me so satisfied your vast understanding and wisdom have a new channel for trying into the world.
Breaking Bad Season 5 DVD Box Set
The L Word Seasons 1-6 DVD Box Set
Game of Thrones Season 3 DVD Box Set
Chuck Seasons 1-5 DVD Box Set
Cane Seasons 1-5 dvd box set
Walt Disney’s 100 Years Of Magic 172 Discs DVD Box Set

Rocket Fuel