Simulating optional stopping in data collection with respect to p values

Question

In null-hypothesis significance testing, if we were to optionally stop data collection every so often and check the p value on the data, the null hypothesis may be eventually be rejected even when it is true.

I would like to simulate some data to help myself understand this problem more deeply, but I'm having trouble getting some R code together to demonstrate it. I was hoping someone could provide some guidance on how to generate a dataset in R to demonstrate the optional stopping problem. That is, to show that even when sampling from a population where the null is true we may eventually attain p < .05.

Answer 1

First of all, I'd like to clear a possible misconception in the first statement of the question. In any hypothesis test, the null hypothesis may be rejected even when it is true. The probability of rejecting the null hypothesis if it is true equals the signification, usually 0.05. In wrong settings like stopping data collection when the null hypothesis is rejected, the probability of rejecting becomes larger than the signification, but there is no guarantee that that data collection will stop at any point - at least, at any reasonable point.

To simulate that in R (or any other program), I would do the following:

• Generate a large ordered sample with the conditions of the null. For example, if I were to use t.test, I would generate a sample with rnorm(n).
• Do a loop over i (for example with for or a more idiomatic r structure) taking the first i elements from the sample and perform a t.test on them.
• For each i record p.value and check if you would have stopped data collecting at any point.

A more complex simulation would repeat that a lot of times to estimate what is the probability of rejecting the null hypothesis with that flawed way of stopping data collection.

Answer 2

Something like this?

We check every 10 values if a t-test is significant. Here the loop in unlimited (in reality it usually isn't). In a next step we show what happens if you continue sampling (to an arbitrary total number).

set.seed(42) #for reproducibility
mean <- 0
sd <- 1

x <- numeric()
test <- FALSE

while (!test) { #this loop is slow
  x <- c(x, rnorm(10, mean = mean, sd = sd))
  test <- t.test(x)$p.value < 0.05
}

length(x)
#[1] 153770

y <- rnorm(2e5 - length(x), mean = mean, sd = sd)

t.test(x)
#One Sample t-test
#
#data:  x
#t = -1.9635, df = 153770, p-value = 0.04959
#alternative hypothesis: true mean is not equal to 0
#95 percent confidence interval:
#  -1.001688e-02 -8.940868e-06
#sample estimates:
#  mean of x 
#-0.005012912

t.test(c(x, y))
#One Sample t-test
#
#data:  c(x, y)
#t = -1.2524, df = 2e+05, p-value = 0.2104
#alternative hypothesis: true mean is not equal to 0
#95 percent confidence interval:
#  -0.007202942  0.001586512
#sample estimates:
#  mean of x 
#-0.002808215 

Note that a false positive result is also possible without optionally stopping.