1
$\begingroup$

In null-hypothesis significance testing, if we were to optionally stop data collection every so often and check the p value on the data, the null hypothesis may be eventually be rejected even when it is true.

I would like to simulate some data to help myself understand this problem more deeply, but I'm having trouble getting some R code together to demonstrate it. I was hoping someone could provide some guidance on how to generate a dataset in R to demonstrate the optional stopping problem. That is, to show that even when sampling from a population where the null is true we may eventually attain p < .05.

$\endgroup$
3
$\begingroup$

First of all, I'd like to clear a possible misconception in the first statement of the question. In any hypothesis test, the null hypothesis may be rejected even when it is true. The probability of rejecting the null hypothesis if it is true equals the signification, usually 0.05. In wrong settings like stopping data collection when the null hypothesis is rejected, the probability of rejecting becomes larger than the signification, but there is no guarantee that that data collection will stop at any point - at least, at any reasonable point.

To simulate that in R (or any other program), I would do the following:

  • Generate a large ordered sample with the conditions of the null. For example, if I were to use t.test, I would generate a sample with rnorm(n).
  • Do a loop over i (for example with for or a more idiomatic r structure) taking the first i elements from the sample and perform a t.test on them.
  • For each i record p.value and check if you would have stopped data collecting at any point.

A more complex simulation would repeat that a lot of times to estimate what is the probability of rejecting the null hypothesis with that flawed way of stopping data collection.

$\endgroup$
3
  • $\begingroup$ While both answers were excellent, I feel like this one is best because it discusses long-run error probability as part of the simulation. $\endgroup$
    – pomodoro
    Feb 15 '19 at 0:16
  • $\begingroup$ I've written my own function that (I think) achieves this. I'd like to gather some feedback, but I'm not sure whether it is appropriate to edit my question here with my code or to ask a completely new question? $\endgroup$
    – pomodoro
    Feb 15 '19 at 0:28
  • $\begingroup$ I'm afraid gathering feedback on a program is not in scope here, although any statistical question that might arise from the program and its outcome would be in scope - for a new question. Concrete programming questions may be in scope in Stack Overflow. Feedback about the code can be in scope in codereview.stackexchange.com, although I have no experience with that site. $\endgroup$
    – Pere
    Feb 15 '19 at 17:44
2
$\begingroup$

Something like this?

We check every 10 values if a t-test is significant. Here the loop in unlimited (in reality it usually isn't). In a next step we show what happens if you continue sampling (to an arbitrary total number).

set.seed(42) #for reproducibility
mean <- 0
sd <- 1

x <- numeric()
test <- FALSE

while (!test) { #this loop is slow
  x <- c(x, rnorm(10, mean = mean, sd = sd))
  test <- t.test(x)$p.value < 0.05
}

length(x)
#[1] 153770

y <- rnorm(2e5 - length(x), mean = mean, sd = sd)

t.test(x)
#One Sample t-test
#
#data:  x
#t = -1.9635, df = 153770, p-value = 0.04959
#alternative hypothesis: true mean is not equal to 0
#95 percent confidence interval:
#  -1.001688e-02 -8.940868e-06
#sample estimates:
#  mean of x 
#-0.005012912

t.test(c(x, y))
#One Sample t-test
#
#data:  c(x, y)
#t = -1.2524, df = 2e+05, p-value = 0.2104
#alternative hypothesis: true mean is not equal to 0
#95 percent confidence interval:
#  -0.007202942  0.001586512
#sample estimates:
#  mean of x 
#-0.002808215 

Note that a false positive result is also possible without optionally stopping.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.