16
$\begingroup$

I am in charge of presenting the results of A/B tests (run on website variations) at my company. We run the test for a month and then check the p-values at regular intervals until we reach significance (or abandon if significance is not reached after running the test for a long time), something I am now finding out is a mistaken practice.

I want to stop this practice now, but to do that, I want to understand WHY this is wrong. I understand that the effect size, the sample size (N), the alpha significance criterion (α) and statistical power, or the chosen or implied beta (β) are mathematically related. But what exactly changes when we stop our test before we reach the required sample size?

I have read a few posts here (namely this, this and this), and they tell me that my estimates would be biased and the rate of my Type 1 error increases dramatically. But how does that happen? I am looking for a mathematical explanation, something that would clearly show the effects of sample size on the outcomes. I guess it has something to do with the relationships between the factors I mentioned above, but I have not been able to find out the exact formulas and work them out on my own.

For eg., stopping the test prematurely increases Type 1 error rate. Alright. But why? What happens to increase type 1 error rate? I'm missing the intuition here.

Help please.

$\endgroup$
6
  • 1
    $\begingroup$ might be useful evanmiller.org/how-not-to-run-an-ab-test.html $\endgroup$
    – seanv507
    Nov 7, 2016 at 14:51
  • 1
    $\begingroup$ Yes I went through this link, but I just did not understand the example given. $\endgroup$
    – sgk
    Nov 8, 2016 at 8:51
  • $\begingroup$ sorry Gopalakrishnan - hadn't seen that your first link pointed to that already. $\endgroup$
    – seanv507
    Nov 8, 2016 at 9:30
  • 1
    $\begingroup$ Can you explain what you don't understand. The maths/intuition seems pretty clear: its not so much stopping before the required sample size, but repeatedly checking. $P(\cup _{i \in 1\dots N} x_i>\theta) \ge P( x_N>\theta)$, so you can't use a test designed for single checks multiple times. $\endgroup$
    – seanv507
    Nov 8, 2016 at 9:41
  • $\begingroup$ @GopalakrishnanShanker mathematical explanation given in my answer $\endgroup$
    – tomka
    Nov 8, 2016 at 12:02

2 Answers 2

6
$\begingroup$

A/B tests that simply test repeatedly on the same data with a fixed type-1 error ($\alpha$) level are fundamentally flawed. There are at least two reasons why this is so. First, the repeated tests are correlated but the tests are conducted independently. Second, the fixed $\alpha$ does not account for the multiply conducted tests leading to type-1 error inflation.

To see the first, assume that upon each new observation you conduct a new test. Clearly any two subsequent p-values will be correlated because $n-1$ cases have not changed between the two tests. Consequently we see a trend in @Bernhard's plot demonstrating this correlatedness of p-values.

To see the second, we note that even when tests are independent the probability of having a p-value below $\alpha$ increases with the number of tests $t$ $$P(A) = 1-(1-\alpha)^t,$$ where $A$ is the event of a falsely rejected null hypothesis. So the probability to have at least one positive test result goes against $1$ as you repeatedly a/b test. If you then simply stop after the first positive result, you will have only shown the correctness of this formula. Put differently, even if the null hypothesis is true you will ultimately reject it. The a/b test is thus the ultimate way of finding effects where there are none.

Since in this situation both correlatedness and multiple testing hold at the same time, the p-value of the test $t+1$ depend on the p-value of $t$. So if you finally reach a $p< \alpha$, you are likely to stay in this region for a while. You can also see this in @Bernhard's plot in the region of 2500 to 3500 and 4000 to 5000.

Multiple testing per-se is legitimate, but testing against a fixed $\alpha$ is not. There are many procedures that deal with both the multiple testing procedure and correlated tests. One family of test corrections is called the family wise error rate control. What they do is to assure $$P(A) \le \alpha.$$

The arguably most famous adjustment (due to its simplicity) is Bonferroni. Here we set $$\alpha_{adj} = \alpha/t,$$ for which it can easily be shown that $P(A) \approx \alpha$ if the number of independent tests is large. If tests are correlated it is likely to be conservative, $P(A) < \alpha$. So the easiest adjustment you could make is dividing your alpha level of $0.05$ by the number of tests you have already made.

If we apply Bonferroni to @Bernhard's simulation, and zoom in to the $(0,0.1)$ interval on the y-axis, we find the plot below. For clarity I assumed we do not test after each coin flip (trial) but only every hundredth. The black dashed line is the standard $\alpha = 0.05$ cut off and the red dashed line is the Bonferroni adjustment.

enter image description here

As we can see the adjustment is very effective and demonstrates how radical we have to change the p-value to control the family wise error rate. Specifically we now do not find any significant test anymore, as it should be because @Berhard's null hypothesis is true.

Having done this we note that Bonferroni is very conservative in this situation due to the correlated tests. There are superior tests that will be more useful in this situation in the sense of having $P(A) \approx \alpha$, such as the permutation test. Also there is much more to say about testing than simply referring to Bonferroni (e.g. look up false discovery rate and related Bayesian techniques). Nevertheless this answers your questions with a minimum amount of math.


Here is the code:

set.seed(1)
n=10000
toss <- sample(1:2, n, TRUE)

p.values <- numeric(n)
for (i in 5:n){
  p.values[i] <- binom.test(table(toss[1:i]))$p.value
}
p.values = p.values[-(1:6)]
plot(p.values[seq(1, length(p.values), 100)], type="l", ylim=c(0,0.1),ylab='p-values')
abline(h=0.05, lty="dashed")
abline(v=0)
abline(h=0)
curve(0.05/x,add=TRUE, col="red", lty="dashed")
$\endgroup$
1
  • 2
    $\begingroup$ This works for me. I will have to translate this to business-speak to get my point across to my seniors now, but that's my own problem. Thank you very much $\endgroup$
    – sgk
    Nov 8, 2016 at 13:18
8
$\begingroup$

If the null hypothesis is true, then people often expect the p value to be very high. This is not true. If the null hypothesis is true, then p is a uniformely distributed random variable. Meaning, that from time to time is will be below 0.05 just randomly. If you look at a lot of different subsamples, sometimes the p value will be below 0.05.

To make that easier to grasp, here is a small simulation in R:

This will throw a coin 10,000 times and we know, it is a fair coin:

set.seed(1)
n=10000
toss <- sample(1:2, n, TRUE)

Starting from the 5th toss, this will perform a binomial test for fairness after every toss and save the p values:

p.values <- numeric(n)
for (i in 5:n){
     p.values[i] <- binom.test(table(toss[1:i]))$p.value
}

And this will plot the p-values one after the other:

plot(p.values, type="l")
abline(h=0.05)

enter image description here

As you can see, the p value dips below 0.05 a couple of times just to recover and finally end up being far above p=0.05. If we stopped the trial any time p was "significant", we would come to the wrong conclusion. One might argue "We have a sample of roughly over 4000 trials i.i.d. and p was below .05. Shurely we can stop sampling any further". The more frequently you check the p-value, the more likely you are to check at a random dip. In this case we generated the data under the $H_0$ and know, that $H_0$ is true.

(Just to be perfectly open, I've tried more than one seed for the number generator before it was as clear as this example, but that is fair for educational purposes. If you have Rinstalled and running, you can easily play with the numbers.)

$\endgroup$
9
  • $\begingroup$ Thanks for the simple experiment. But say I stopped the test at one such stage (when p-value < 0.05), what will my results mean? (other than the fact that it is wrong). Is it possible for me to compensate by reducing the p-value threshold? $\endgroup$
    – sgk
    Nov 8, 2016 at 8:59
  • $\begingroup$ +1 Note the correlated tests and related multiple testing problem. See my extended answer with adjustment options below, based on your (very good) example. $\endgroup$
    – tomka
    Nov 8, 2016 at 10:42
  • $\begingroup$ You can not quite compensate it by reducing the p-value threshold $\alpha$. Choosing an $\alpha$ is always a compromise in the area of conflict between type I error and type II error, also known as alpha error and beta error. Of course you can compensate for one of them totally or for each of them partially, but finally this conflict grows increasingly difficult the more often you test. I have read that Baysians claim they have less of a problem with that but with to little knowledge I think this holds only for the modelling of believe in the value of a statistic and not in yes/no decisions. $\endgroup$
    – Bernhard
    Nov 8, 2016 at 12:05
  • $\begingroup$ My main point is that of controlling family wise error (FWER) rate or false discovery rate (FDR) both targeting type-1 error. Controlling type-2 error is less of an issue in a/b tests due to usually very large samples. $\endgroup$
    – tomka
    Nov 8, 2016 at 12:11
  • 1
    $\begingroup$ "finally end up being far above p=0.05" sounds misleading to me: shouldn't we expect the curve to keep reaching down to 0.05 level after any arbitrarily large $n$? Is there any guarantee that the curve will "end up" somewhere? I don't think so. $\endgroup$
    – amoeba
    Nov 9, 2016 at 23:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.