# How do you experimentally determine a minimum sample size to characterize a response?

I want to repeatedly perform a measurement until the distribution of response values stabilizes. Is there a standard metric to stistically conclude that new data does not contain new information? I want to identify when I have a sufficient number of samples to characterize the response distribution.

My ad hoc approach was to calculate the mean and standard deviation after each new sample. With each new measurement, these values eventually stabilize to a consistent value. I can easily check the size of the change for each step, then put a cuttoff when a change is sufficiently small. Perhaps I could stop measuring after the change in standard deviation is less than 5% of the mean 3 measurements in a row.

Is there a more formal approach to experimentally determining a sufficient sample size?

Here is an plot of a mock experiment where samples are randomly chosen between 0 and 1. This illustrates how the mean and standard deviation level off as more samples are obtained.

• A productive approach might be defining what an acceptable error threshold is and then using power calculations to determine the number of data points needed to reach this threshold within a XX% confidence interval. Oct 1, 2018 at 19:21

One approach is to calculate the confidence interval of the mean value after each sample. The relative size of this interval compared to the mean value could be used to identify when to stop measuring.

The calculation begins by calculating the standard deviation of the sample, $$s$$, and then the standard error of the mean, $$s_{\bar x}=s/\sqrt{n}$$, where $$n$$ is the number of samples. The confidence interval is defined as $$\bar x \pm t s_{\bar x}$$ where $$t$$ is obtained from a t-distribution table, based on the confidence level and degrees of freedom. Using the t-value in this situation is more appropriate because of the small sample size. Note that if you specify a 95% confidence level, you would use a 1-sided t-table with $$\alpha=0.5$$.

It seems reasonable to set a minimum number of samples to obtain, then stop measuring after the confidence interval is repeatedly less than a prescribed percentage of the mean. For example, stopping after the confidence interval is less than 5% of the mean, $$2ts_{\bar x} < 0.05 \bar{x}$$, 3 samples in a row.

Several online references (see below) discuss identifying the required sample size based on an estimated standard deviation of the sample. The approach here instead calculates the sample standard deviation as you go. The cuttoff criteria is somewhat arbitrary but the method for calculating the confidence interval is statistically sound.

• Since it is indeed ad hoc, could you clarify the sense in which it has any rigor? For instance, for somebody using this procedure, what would be the actual risk that they make an egregiously wrong decision? This is a fair question to ask, and in light of your claim it deserves to be answered, because rigorous statistical procedures (at a minimum) enable one to control decision error rates.
– whuber
Oct 3, 2018 at 15:53
• re the edit: Although some method for computing CIs may be published, that does not imply it is appropriate for this particular application. Indeed, it definitely is not, because you are computing a sequence of potentially a great number of strongly interdependent CIs and thereby making a sequence of closely related decisions. Intuitively something like this ought to work in some circumstances, so what is of interest--and of importance--is to understand (a) to what extent this abuse of CIs might still be a good approximation and (b) to characterize when that will be true.
– whuber
Oct 3, 2018 at 16:42