3
$\begingroup$

I want to repeatedly perform a measurement until the distribution of response values stabilizes. Is there a standard metric to stistically conclude that new data does not contain new information? I want to identify when I have a sufficient number of samples to characterize the response distribution.

My ad hoc approach was to calculate the mean and standard deviation after each new sample. With each new measurement, these values eventually stabilize to a consistent value. I can easily check the size of the change for each step, then put a cuttoff when a change is sufficiently small. Perhaps I could stop measuring after the change in standard deviation is less than 5% of the mean 3 measurements in a row.

Is there a more formal approach to experimentally determining a sufficient sample size?

Here is an plot of a mock experiment where samples are randomly chosen between 0 and 1. This illustrates how the mean and standard deviation level off as more samples are obtained.

enter image description here

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ A productive approach might be defining what an acceptable error threshold is and then using power calculations to determine the number of data points needed to reach this threshold within a XX% confidence interval. $\endgroup$
    – Aaron Springer
    Commented Oct 1, 2018 at 19:21

1 Answer 1

Reset to default
0
$\begingroup$

One approach is to calculate the confidence interval of the mean value after each sample. The relative size of this interval compared to the mean value could be used to identify when to stop measuring.

The calculation begins by calculating the standard deviation of the sample, $s$, and then the standard error of the mean, $s_{\bar x}=s/\sqrt{n}$, where $n$ is the number of samples. The confidence interval is defined as $$\bar x \pm t s_{\bar x}$$ where $t$ is obtained from a t-distribution table, based on the confidence level and degrees of freedom. Using the t-value in this situation is more appropriate because of the small sample size. Note that if you specify a 95% confidence level, you would use a 1-sided t-table with $\alpha=0.5$.

It seems reasonable to set a minimum number of samples to obtain, then stop measuring after the confidence interval is repeatedly less than a prescribed percentage of the mean. For example, stopping after the confidence interval is less than 5% of the mean, $2ts_{\bar x} < 0.05 \bar{x}$, 3 samples in a row.

Several online references (see below) discuss identifying the required sample size based on an estimated standard deviation of the sample. The approach here instead calculates the sample standard deviation as you go. The cuttoff criteria is somewhat arbitrary but the method for calculating the confidence interval is statistically sound.

  1. Western Michigan STAT 216: Determining Sample Size for Estimating the Mean
  2. PennState STAT 506: Selection Sample Size for Estimating Population Mean and Total
  3. Select Statistical Services: Population Mean - Sample Size
  4. R Tutorial: Sampling Size of Population Mean
Improve this answer
$\endgroup$
2
  • $\begingroup$ Since it is indeed ad hoc, could you clarify the sense in which it has any rigor? For instance, for somebody using this procedure, what would be the actual risk that they make an egregiously wrong decision? This is a fair question to ask, and in light of your claim it deserves to be answered, because rigorous statistical procedures (at a minimum) enable one to control decision error rates. $\endgroup$
    – whuber ♦
    Commented Oct 3, 2018 at 15:53
  • $\begingroup$ re the edit: Although some method for computing CIs may be published, that does not imply it is appropriate for this particular application. Indeed, it definitely is not, because you are computing a sequence of potentially a great number of strongly interdependent CIs and thereby making a sequence of closely related decisions. Intuitively something like this ought to work in some circumstances, so what is of interest--and of importance--is to understand (a) to what extent this abuse of CIs might still be a good approximation and (b) to characterize when that will be true. $\endgroup$
    – whuber ♦
    Commented Oct 3, 2018 at 16:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.