5
$\begingroup$

According to this article, there's an issue with concluding a test as soon as it reaches significance (the "peeking effect"). Instead you need to determine the sample size ahead of time, and only look at the results once the test has gathered the predetermined number of observations.

This article also says that "statistical significance is not a stopping rule", and that just because a variation has 99% probability of beating the original it doesn't mean you can stop early. But what I don't understand fully is "why". If something has 99% probability of beating the original, that means given the observed effect you should make the correct decision 99% of the time?

So I have two questions:

  1. Doesn't a confidence interval/significance account for the sample size? That is, even if the test swings wildly early on, an early 99% confidence still means that given the few observations, the results are still extreme enough to fall outside of the 99% of expected results? Isn't that the whole point of significance?

  2. Let's say I do determine that I need to run each variation say 10K times each. At the end of this I find there is no conclusive evidence. So I decide to let the test continue to run for another 10K observations per variation. How does this affect power and significance?

Improve this question
$\endgroup$
11
  • 2
    $\begingroup$ The idea in the article is ok but its recommendation is not a great one. There are good ways to perform sequential tests where the sample size is not determined ahead of time. That seems to accord with the intuition behind the question (1). The main problem doesn't really stem from performing a sequence of tests, but rather that when you compute "confidence" after each observation, you have to do it correctly. The wrong way is to use a calculation designed for a single decision made from a sample of a fixed size. $\endgroup$
    – whuber ♦
    Commented Nov 17, 2015 at 18:25
  • 1
    $\begingroup$ @whuber Can you give a citation for a decent intro text or two about good ways to perform sequential tests? $\endgroup$
    – Alexis
    Commented Nov 17, 2015 at 19:11
  • 3
    $\begingroup$ @Alexis There is a discussion in Kendall & Stuart's Advanced Theory of Statistics (fifth ed.). See Volume 2, chapter 24. Some of the papers are useful, too. I recently used the results from Bill Meeker's work of 30 years ago (related to his PhD thesis) with some success on an A/B testing problem. $\endgroup$
    – whuber ♦
    Commented Nov 17, 2015 at 21:38
  • $\begingroup$ I understand that if you check daily if it has reached significance your giving the test multiple "chances" which will increase the type I error rate and mess up significance. But still seems to me, if your test reaches 99% significance in 24 hours and you act on it, you should be making the right decision 99% of the time? In his example, isn't getting only 1 conversion in 110 observation such an extreme result that we should only expect it 1 in a 100 times if h0(= no diff) was true? So was his example just a freak accident? $\endgroup$
    – L Xandor
    Commented Nov 18, 2015 at 0:25
  • $\begingroup$ (1) if your significance level is higher than you think, the p-values are similarly wrong. $\ $ (2) with enough peeking your true p-value might exceed 0.5, even when your calculated p-value is 0.01. $\ $ (3) Even if it were correctly calculated, a p-value of 0.01 doesn't mean "you have a 99% chance of making the right decision". $\endgroup$
    – Glen_b
    Commented Nov 18, 2015 at 1:52

1 Answer 1

Reset to default
4
$\begingroup$

This prompts the question: why not stop early when you saw that your results were insignificant?

The answer in both cases, you are falling to confirmation bias, i.e., you ignore any new evidence once what you believe has been confirmed. This would render your analysis very biased towards your initial viewpoint.

We have to be careful not to rely too much on statistical significance. After all, statistical significance just means is "this result would be very strange if the null hypothesis is true".

The American statistical association published a statement critical of p-values recently, addressing this issue.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I appreciate the edit @gung, I need to remember to clean up my answers a bit more. $\endgroup$
    – Chris P
    Commented Apr 28, 2016 at 21:09
  • $\begingroup$ Welcome to the site, @ChrisP. It's not a problem. It takes a while to learn the various editing & formatting options the site affords. $\endgroup$
    – gung - Reinstate Monica
    Commented Apr 28, 2016 at 21:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.