2
$\begingroup$

Will the significance and power levels hold constant if I don't see statistically significant results and decide to wait to allow for more samples to be collected?

Improve this question
$\endgroup$
4
  • $\begingroup$ @Dave's answer has covered why what you have proposed is dangerous under a classical null hypothesis significance test setting. You might want to consider employing sequential testing or Bayesian testing (e.g. via Bayes factor) techniques to get round this problem before you start your experiment. $\endgroup$
    – B.Liu
    Commented Jul 23, 2021 at 19:11
  • $\begingroup$ look at this tag sequential-analysis $\endgroup$
    – kjetil b halvorsen ♦
    Commented Jul 23, 2021 at 19:11
  • $\begingroup$ Also see stats.stackexchange.com/questions/20676/…. $\endgroup$
    – whuber ♦
    Commented Jul 23, 2021 at 19:42
  • $\begingroup$ The answer here and at the duplicate address $\alpha$ but not power. Generally speaking, increasing $\alpha$ will move the whole power curve with it (that is, increasing $\alpha$ increases power). $\endgroup$
    – Glen_b
    Commented Jul 24, 2021 at 10:34

1 Answer 1

Reset to default
3
$\begingroup$

No! This is a dangerous practice. In effect, you’re playing something like the following game.

Flip a coin, betting \$1000 on heads. It lands on tails (no rejection), a result you do not like. Now bet \$2000 on the next flip. It comes up heads, you collect your money, you walk away from the game, and you declare yourself a master coin-flip player since you won a bunch of money.

You can simulate this with a true null hypothesis to see how your type I error rate inflates. Just use some conditional (if/else) logic in a loop to keep adding observations to your sample until you either achieve a rejection ($p\le\alpha$) or reach some absolute maximum sample size. You will see a high error rate, much higher than the $\alpha$-level you deem acceptable.

Improve this answer
$\endgroup$
3
  • $\begingroup$ True, but you leave out some other very important considerations. First, more observations allow a better estimation of the parameters of population or model. Second, even within the hypothesis testing framework an increase in sample will bring an increase in power along with the increased risk of a false positive. Third, the idea of false positive errors is only relevant in a subset of inferential approaches and settings. To simply say "don't do it" hides relevant factors. $\endgroup$
    – Michael Lew
    Commented Jul 23, 2021 at 21:51
  • 1
    $\begingroup$ It’s always easy to increase power at the expense of false positives…just set $\alpha=1!$ $\endgroup$
    – Dave
    Commented Jul 23, 2021 at 22:04
  • $\begingroup$ Another trivialising response... $\endgroup$
    – Michael Lew
    Commented Jul 23, 2021 at 22:28