# Why does significance not equal validity? Why can't I stop a test as soon as it has significant results?

According to this article, there's an issue with concluding a test as soon as it reaches significance (the "peeking effect"). Instead you need to determine the sample size ahead of time, and only look at the results once the test has gathered the predetermined number of observations.

This article also says that "statistical significance is not a stopping rule", and that just because a variation has 99% probability of beating the original it doesn't mean you can stop early. But what I don't understand fully is "why". If something has 99% probability of beating the original, that means given the observed effect you should make the correct decision 99% of the time?

So I have two questions:

1. Doesn't a confidence interval/significance account for the sample size? That is, even if the test swings wildly early on, an early 99% confidence still means that given the few observations, the results are still extreme enough to fall outside of the 99% of expected results? Isn't that the whole point of significance?

2. Let's say I do determine that I need to run each variation say 10K times each. At the end of this I find there is no conclusive evidence. So I decide to let the test continue to run for another 10K observations per variation. How does this affect power and significance?

• The idea in the article is ok but its recommendation is not a great one. There are good ways to perform sequential tests where the sample size is not determined ahead of time. That seems to accord with the intuition behind the question (1). The main problem doesn't really stem from performing a sequence of tests, but rather that when you compute "confidence" after each observation, you have to do it correctly. The wrong way is to use a calculation designed for a single decision made from a sample of a fixed size.
– whuber
Nov 17 '15 at 18:25
• @whuber Can you give a citation for a decent intro text or two about good ways to perform sequential tests? Nov 17 '15 at 19:11
• @Alexis There is a discussion in Kendall & Stuart's Advanced Theory of Statistics (fifth ed.). See Volume 2, chapter 24. Some of the papers are useful, too. I recently used the results from Bill Meeker's work of 30 years ago (related to his PhD thesis) with some success on an A/B testing problem.
– whuber
Nov 17 '15 at 21:38
• I understand that if you check daily if it has reached significance your giving the test multiple "chances" which will increase the type I error rate and mess up significance. But still seems to me, if your test reaches 99% significance in 24 hours and you act on it, you should be making the right decision 99% of the time? In his example, isn't getting only 1 conversion in 110 observation such an extreme result that we should only expect it 1 in a 100 times if h0(= no diff) was true? So was his example just a freak accident? Nov 18 '15 at 0:25
• (1) if your significance level is higher than you think, the p-values are similarly wrong. $\$ (2) with enough peeking your true p-value might exceed 0.5, even when your calculated p-value is 0.01. $\$ (3) Even if it were correctly calculated, a p-value of 0.01 doesn't mean "you have a 99% chance of making the right decision". Nov 18 '15 at 1:52