Continuous sample size determination based on the control group

Question

I am curious if and where the following reasoning breaks down.

Traditionally, sample size determination is done as part of the design phase. To this end, one has to have an understanding of the baseline performance, such as the mean and standard deviation of the metric in question. One might use recent historical data to obtain reasonable estimates.

Suppose we are not necessarily interested in knowing the minimal sample size ahead of time and simply launch the experiment. The control group is the baseline. Each day, we take the statistics of the control group and perform sample size determination. With each new day passed, we get a better and better understanding of when to stop.

Apart from the fact that one might launch an experiment that is doomed to fail, are there any other problems with the above logic?

Clarification 1: It should be noted that this question is not about whether it is sound to check some p-value on a daily basis and stop whenever it goes below a predefined level. The inadequacy of this procedure is well understood.

Clarification 2: Here we are not assuming any clinical or otherwise setting with serious implications and ethical concerns. Rather, one can imagine something trivial, such as an experiment with a webpage where the color of a button is subject to change.

Clarification 3: The question describes a specific stopping rule: every day, run a power analysis using the statistics of the data for the control group collected up to today and terminate the experiment as soon as the actual sample size exceeds the one given by the power analysis, potentially skipping a few first days to avoid pathological cases.

Answer 1

A priori sample size calculations are only sensible if there is a fixed cost to sampling

This is an excellent question, and it raises some common-sense issues that are commonly misunderstood in sampling theory. When we are considering sample size questions, the first thing we need to ask ourselves is what (if any) impediment, inconvenience, cost, etc., there is to just collecting all the data there is on the matter of interest. If there is no cost to this (construed broadly as including time, money, etc.) then we might as well go out and collect all possible data on the matter and then we don't need statistical inference procedures at all. Consequently, the idea of using a limited sample size that is less than what is possible only arises as a sensible procedure when there is some cost to sampling.

Once we have identified the costs of sampling (again, construed broadly to include more than just money), we need to decide if there are any fixed costs that accrue when we conduct a sampling "excursion", or not. If there are no fixed costs then there is no need to limit our data collection by any a priori calculation --- we can just collect some data, decide if it is enough, collect some more, decide if it is enough, and so on. So in this case, we can indeed do something like what you describe; sampling without any a prioiri sample size calculation and then stopping when we have enough data for the inferences we want to make. (We need to be a bit careful with this method, because certain kinds of "stopping rules" combined with particular statistical procedures can bias analysis, but assuming we are cognisant of this, we should be able to formulate an appropriate "stopping rule" and stop when we have the data we need.) Various statistical control processes operate in exactly this way --- they collect data as it is generated and update inferences with each new data point.

If there are fixed costs to sampling, but the fixed costs are quite low relative to the unit costs, it might still be useful to sample over more than one "excursion". In such a case we might make an initial a priori sample size calculation for an initial "pilot" set of data, then use this data to make an updated sample size calculation for the next set of data, and so on. In this case there might be a number of data collection excursions, but each will involve a sample size calculation. This is also somewhat similar to what you describe. Finally, if the fixed costs of sampling are quite substantial relative to the unit costs, then the "traditional" method of using only one sampling "excursion" is probably going to be optimal. In this case the sample size is fully determined by a single a priori calculation.

So, in terms of where your reasoning breaks down, it is only in the possibly exaggerated claim that "[t]raditionally, sample size determination is done as part of the design phase." This is only true if there are substantial fixed costs to sampling that make it desirable to sample "in one go" and thereby necessitate an a priori computation of the entire sample size. Your observation that it can be sensible to collect data "as you go" is perfectly legitimate (under appropriate circumstances), and it is actually an underappreciated principle of sampling theory.

Answer 2

Example: Suppose you have $n = 67$ observations from a normal population with unknown population mean $\mu$ and standard deviation $\sigma.$ You wish to test $H_0: \mu = 106$ against $H_a: \mu \ne 106.$ and you want the power of a one sample t test at the 5% level to be about 90% when the true mean is $\mu = 100.$

Then a simple simulation in R shows that $n = 67$ observations is sufficient to give the desired power:

set.seed(2021)
pv = replicate(10^6, t.test(rnorm(67, 100, 15), mu = 106)$p.val)
mean(pv <= 0.05)
[1] 0.896917

However, the P-values corresponding to t statistics computed after $n = 2, 3, \dots, 67$ may be smaller than 5% for some values of $n$ smaller than $67.$

Below we plots of such 'intermediate' P-values for nine experiments with $n = 67.$ About 80% samples will show P-values below $0.05$ exactly at $n = 67.$ That may be the first step at which the value is small enough to reject, but that is typically not the case. Clearly, it would not be appropriate to make a decision based on the first step at which a P-value is smaller than 5%.

Of course, some of these P-value traces may never dip as low as $0.05$ within $n = 67$ observations.

R code for one trace is shown below:

pv = numeric(67)
x = rnorm(67, 100, 15)
pv = numeric(67);  pv[1]= 1  # no t stat for n=1
for(i in 2:47) 
 { pv[i] = t.test(x[1:i], mu=106)$p.val }
plot(pv, type="l", ylim=0:1)
 abline(h=.05, col="blue")
 m = min(which(pv < .05));m
 abline(v=m, col="red") 

Answer 3

I feel that your statement about study design misrepresents what it is about.

Traditionally, sample size determination is done as part of the design phase. To this end, one has to have an understanding of the baseline performance, such as the mean and standard deviation of the metric in question. One might use recent historical data to obtain reasonable estimates.

Study Design Basics

Study design is all about risk management and should not be based on empirical data such as mean and standard deviation. Rather study design should be based on what is a minimally useful normalised effect size (inter-group difference over intra-group variation) balanced against maximum acceptable risk. If the effect size is useful you want to ensure your experiment detects it at a level of risk that is acceptable to stakeholders. To see why look at a basic sample size calculator:

$$n=\frac{2(Z_{\alpha} +Z_{ \beta})^2\sigma^2}{\Delta^2}$$

Basically we fix a constant by deciding alpha and beta based on stakeholder sensitivity to rejecting null when it is true vs accepting it when it is false. If we use empirical data to fill in $\sigma$ and $\Delta$, then our calculation will use the same information that we would use to calculate p values, e.g. the t-test calculates

$$ t = \frac{\Delta}{\sigma}$$

Which is then transformed into Z scores. You'll notice that t has the same variables, but inverted. I.e. using empirical data in a sample size calculator will simply tell us what we would already know by looking at the p value, albeit in a transformed way.

Proposed approach

The process described in the OP is performing exploratory data analysis and appears to be attempting to use it as confirmatory analysis. To manage risks it seeks to use statistical methods to provide respectability to the output.

One of the critical needs for confirmatory analysis is the concept of independence. Hypotheses need to be tested on data that is independent of the data used to originally generate them, otherwise you end up with familywise errors as the errors at later stages will be implicitly tied to the errors at the early stage since the variation in the early samples persists and influences all subsequent analyses. The process in the OP cannot be used to provide falsification of the hypothesis that the experiment is itself generating. If you look at the basic formula for sample size calculation above this becomes clear.

Apart from the fact that one might launch an experiment that is doomed to fail, are there any other problems with the above logic? Of course, we are not assuming a clinical setting but rather a service of some kind.

Launching an experiment doomed to fail is probably the most insignificant possible problem as all that is wasted is the resource put into the experiment. A much bigger risk is that the experiment leads to an inappropriately weighted decision that misunderstood the magnitude of risk and led to an overconfident response. Here the effect goes way beyond the resource associated with performing the experiment. A couple of examples of what it may lead to committal of:

• new investment in something flawed that will fail
• cancelling a product that could have made a positive impact
• releasing a product or service that has a negative impact on users
• fraud, if the risks were willfully not clearly detailed

Many proof of concepts start out life using informal or semi-formal processes similar to what is described in the OP. However, it must be clear to all stakeholders that it is exploratory in nature and unconfirmed by independent testing. It would be advisable to undertake a statistical design of the process to be sure you understand the risks, but to be honest this sounds like a much more difficult job than a standard study design approach based on first understanding stakeholder needs, risks and benefits. The amount of effort put into this should be proportional to the expected risk/benefits of the final use case of the research outputs. Changing the colour of a web site button probably warrants 2 minutes within a design meeting to establish these, whereas a medical device will require domain experts and a full benefit-risk analysis.

When presenting output of the analysis it must be qualified and the appropriate caveats must be attached to all downstream decisions and recommendations. All stakeholders (anyone affected by the decisions in any way) should be made fully aware of the risk associated with the data.

Conclusion

It would be difficult to see how to justify the approach without a much more robust defence and a mathematical/statistical proof of how it performs. This would be much more work than it is proposing to save.

Answer 4

Maybe it's me, but I feel all the answers here completely missed the point.

Correct me if I'm wrong - you want to update the sample size estimate by using statistics from the control group instead of calculating it once using pre-experiment statistics? So the question here is not about stopping the experiment, but about estimating control group statistics, which is what pre-experiment mean/variance is doing anyway.

So, for sample size calculation you need an MDE, alpha, beta, variance (or mean, if you have a proportional - i.e. Bernoulli - variable). Keeping alpha, beta and MDE fixed as part of the experiment design, you update the sample size based on the control mean/variance estimates.

I think this should be equivalent to the pre-experiment estimate if SUTVA is satisfied, and there are no other effects causing the change in the control group. But then, if there are, they would be there anyway, and your pre-experiment estimate wouldn't correspond to the control group statistics.

The only caveat I can see is that the pre-experiment estimate will be more reliable as you can have a higher sample size, thus lowering the confidence interval and getting an estimate close to the "true" mean/variance. This is important, because at worst case, you can get an underpowered experiment, which has quite a lot of caveats, for example, overestimated effect in case of a significant result.

It's an interesting question, and would be nice to see some proofs and simulations (I might perform some soon). But in general, I think it's all about how fast the estimate of mean/variance converges to the "true" mean using control group data, and if you have enough data to estimate it reliably.

Answer 5

The fundamental problem is the phrase "we get a better and better understanding of when to stop." On what basis do you decide "when to stop?"

What you propose is that each day (after you have collected some new data) "we take the statistics of the control group and perform sample size determination." But this doesn't seem to make sense, since "determining sample size" (power analysis) takes a certain error tolerance, standard deviation and alpha as inputs and gives you the N that you need to estimate a quantity within that tolerance. As you note this is obviously a useful analysis to run BEFORE You design the study. But in your case, each day you already KNOW whatever new, larger sample size you have as of that day. So if the output of the formula is already known, what exactly are you doing "each day?" What exactly do you put into this formula each day and what value are you expecting to get out?

My guess is that what you really meant to say is "each day I calculate confidence intervals around the control group using the data that I've collected up to that point, and when the confidence intervals get small enough I stop collecting new data."

But this is where you get into trouble, because what criteria could you use to decide if the confidence intervals are "small enough" to stop collecting new data?

If you are "running an experiment" then the goal is presumably to test a hypotheses: is the treatment group significantly different from the control group, with respect to some outcome (at a certain level of confidence). Obviously you could say "I'm going to keep collecting data until the confidence intervals around the control group no longer overlap with the estimate for the treatment group." which would obviously undermine the logic of significance testing for the reasons that others have described. But if you don't want to use this as your criteria then what else could you do?

If you just want to say "I'll stop collecting data when my Confidence intervals are less than X points wide" that's fine....but in that case you are back to where you started, since you could have done this before you started collecting data: that's what power analysis is for after all!

Here's the question you have to answer: what "new information" would you get each day that would help you decide "when to stop" that you didn't already know before hand, but which is also NOT related to the hypothesis you are trying to test?