# bias of peeking at AB test data and adjusting minimum detectable effect

Let's say we're running an A/B test on my website comparing blue button clicks (baseline) to green button clicks.

• I use http://www.evanmiller.org/ab-testing/sample-size.html to calculate my required number of subjects per branch with the following parameters:

• significance level of 5%
• statistical power of 80%
• an observed historical baseline conversion rate of 5%
• a desired minimum detectable effect of 1% (ie. conversions between 4% and 6% will be indistinguishable from the baseline)

Using the calculator, I determine that we need 7,663 pageviews to declare a result.

Now let's say everyone gets impatient and decides to check in on the experiment after only 900 pageviews.

The Game Plan:

1) If it turns out that the green button is at least 3% better than baseline, we will decide to conclude the experiment and declare the green button as the winner (a 3% MDE given the same other initial parameters requires only 894 pageviews according to the calculator).

2) If it turns out that the green button is less than 3% better than baseline after 900 pageviews, we will decide to keep the experiment running to it's full course of 7,663 pageviews and then make a conclusion at that time.

Are we introducing bias with this Game Plan?

Setting your stopping condition based on the significance of your interim analyses is, in general, not a great idea. The worst possible thing you could do would be to re-run your analysis after every page view and stop as soon as you got a significant result. You've decided, by setting your $\alpha=0.05$, that you're willing to tolerate a 5 percent chance of making a Type I error (i.e., claiming there's an effect even though there actually is not). This repeated "peeking" inflates that more than 5-fold, so there is actually a 1:4 chance your effect is due to random noise, rather than a 1:20 on. That is clearly bad. By peeking only once instead, you're not doing nearly as badly, of course.

This problem has been studied extensively, largely under the name of "Interim Monitoring" or "Sequential Experimental Design" and comes up a lot in the design of clinical trials. This review, by Jennison and Turnbull, covers a bunch of approaches. It looks like your ad-hoc idea is pretty close to the "stochastic curtailment" approach, so that and the references therein might point you in the right direction if you want to do this completely correctly.

I'm not sure I would use the term bias here.

Following the procedure you suggest, you are accepting a risk of inappropriately selecting the green button on inadequate evidence.

To quantify that risk, you would need to calculate the probability of an unfavorable outcome (presumably Blue > Green) of an n=7663 test GIVEN (Bayes Rule) the observed outcome at n=900.

I created a simple simulation in Excel to help me evaluate the problem. Specifically I randomly created 10 series with a true underlying 5% "pass/conversion rate". Within 300 iterations, all 10 had converged to within 1.5% (e.g they ranged from 3.5 - 6.6% cumulative rates). (note, this implies about 450 iterations in your model, as you have 900 with 2 levels)

Having said that, there was still a lot of noise, and overall convergence was much stronger (within 1%) by about 850 iterations.

In other words, if you are seeing a 3% difference in performance (e.g. 8% vs. 5%) after 450 iterations of each, the chances of that being purely a statistical artifact are quite low.

• Doesn't this run into the same problem as described here though (if you check at n=900, 1000, 1100, etc..)? evanmiller.org/how-not-to-run-an-ab-test.html Basically that repeated checking for significance discounts the possibility of not having significant results.
– mark
May 25, 2013 at 1:27
• @Mark I don't think it does. He references what I am basically saying under the title "Bayesian experiment design" (see my 3rd sentence). Beyond this, the point made on the Evan Miller page is valid, as far as it goes, but I dislike his recommendation because it's a form of willful ignorance. At any value of N you are accepting some risk that given more n the answer would change. Since the "bias" he describes can be quantified, I find it difficult to endorse an artificial straightjacket as a solution.
– JBK
Jun 6, 2013 at 0:17