Increasing sample size to obtain width of CI / SE: p-hacking?

Question

I'm involved in running experiments, where we want to obtain sufficient sample size to obtain a certain width of CI (or equivalently a certain power).

We currently run a pilot, of a few hundred units, calculate the variance (we ignore the size of the effect) and then estimate the sample that would be required to obtain a CI width that we desire.

The sample is an estimate, so sometimes (~half) the CI ends up being smaller than we expect, and sometimes larger. When it's larger, the customer is unhappy.

One approach that's been suggested is to keep sampling until the CI width is sufficiently small. This feels uncomfortably close to p-hacking to me, but we're not calculating p-values, and we're (still) not looking at the size of the effect.

Is this legitimate?

Answer 1

Sampling until a nominated confidence interval width is obtained is technically similar to sequential testing and might be thought by some to be similar to p-hacking, but that does not mean that you not do it!

If your concern is accurate estimation of the population variance then a 'stop when CI is less than' strategy is going to give you low estimates more often than not, because the sampling is more likely to stop after an observation that lowers the sample standard deviation than after an observation that increases it. However, that bias may be quite small and thus might well be of no practical concern. It will depend on the sample size and thus the nominated CI width. It will be less with a large sample because the large sample will have a relatively stable CI estimate prior to stopping whereas a small sample CI will fluctuate much more with each new observation.

If your concern is to accurately estimate the population mean then I don't think there is any issue because your stopping rule is not dependent on the sample mean.

P-hacking is not an all or none phenomenon and procedures that might sometimes be illegitimate may in other circumstances be good practice! It depends on inferential objectives as well as experimental design considerations. See section 3 here: https://link.springer.com/chapter/10.1007/164_2019_286

Answer 2

As a partial answer to my question, I ran a simulation to see if it led to inappropriate rejection of $H_0$.

library(dplyr)
set.seed(1234)
start_n <- 20
increment_n <- 20

target_se <- 0.05

vec_p <- numeric()
vec_se <- numeric()
vec_n <- numeric()
vec_mean <- numeric()

# H0 true
for (i in 1:1000) {
  y <- rnorm(start_n)
  keep_running <- TRUE
  while(keep_running == TRUE) {
    se <- sd(y) / sqrt(length(y))
    p <- t.test(y)$p.value
    keep_running <- se > target_se 
    y <- c(y, rnorm(increment_n))
  }
  vec_se <- c(vec_se, se)
  vec_p <- c(vec_p, p)
  vec_n <- c(vec_n, length(y))
  vec_mean <- c(vec_mean, mean(y))
}

mean(vec_p < 0.05)
table(vec_n)

Which gives:

Type I error rate:  0.045
vec_n
320 340 360 380 400 420 440 460 480 500 520 
  1   2  17  56 166 242 289 161  55   9   2 

(vec_n is the sample size that was reached before the experiment stopped.)

The type I error rate tends to be a touch lower than 0.05, which is explained by @Michael Lew's answer.