0
$\begingroup$

At my work, we have used a 'modified Coefficient of Variation', defined as the Standard Error over the mean, and I wanted to ask if there is any statistical justification for this. We are interested in determining the necessary sample size needed to be confident in our estimates of a mean value (like a simple, rule-of-thumb power analysis). I've been told that for our work, a value of 20% is an acceptable cutoff for SE/Mean. I recently encountered a different group of plant ecologists using SE/Mean in the same manner: to assess whether they had done enough surveys; however, I can not find broad online support for this statistical technique.

Based on searches here at CrossValidated and the internet more broadly, true CV is defined as standard deviation over the mean and lets you assess relative variability between separate sample groups that otherwise might have different units. I also know that SE is already a measure of how confident we are in the estimate of the mean. Does it make sense to standardize SE by the mean as a rule-of-thumb estimate for the variability in your sample?

I think the appeal of using SE/mean is that there is a clear decline in this metric with increasing sample size - but it feels arbitrary. Any insight would be appreciated.

EDIT: Real life examples

In my own work: We 'over sampled' an area of interest by doing a number of marine fish surveys within an area (say 20), then used SE/mean and bootstrapping to estimate how this metric changed with sample size. It was determined that the SE/mean had reduced to .20 or 20% by ~6 surveys so that in future years, we will now only use 6 surveys.

From the plant ecologist: SE/mean can be used post-hoc to determine if you collected enough transect data using the rule of thumb that SE/mean should be 10% or less. If SE/mean is greater than 10% - then you didn't survey enough and cannot be as confident in your results

Though one of these examples is from a terrestrial ecologist, both of these uses seem a little fishy..

$\endgroup$
  • 1
    $\begingroup$ Speaking about SE/ mean: all I can think of is the coefficient of variation which you already meantioned (sd/ mean) or the confidence interval of the mean (mean +/- 1.96* SE). How does the SE/ mean value provide information for power analysis? For example if SE/ mean = 0.13, what does this rule of thump mean for the power? Maybe you could mention an example? $\endgroup$ – stats.and.r Apr 26 at 17:18
  • $\begingroup$ Neither SE, nor SE/mean, will provide an estimate of the variability in your sample - it provides an estimate of the variability in your estimated parameter. If you have two samplings of the same distribution, both will have the same SD, but the larger sample will have a lower SE. SE won't tell you anything about how variable your data is, only about how variable your estimate of the mean is. From the first paragraph, it seems like you're using it appropriately, but it was less clear from your question in the second paragraph. $\endgroup$ – Nuclear Wang Apr 26 at 17:52
  • 1
    $\begingroup$ @stats.and.r, I've tried to add two examples - the two real-world scenarios I've heard it suggested to use SE/mean. It is always in a post-hoc fashion where the user suggests that at some cutoff (say SE/mean = 10%), then we have a sufficient sample size to reduce variability around the mean enough to be confident in the results $\endgroup$ – Kodiakflds Apr 26 at 21:42
1
$\begingroup$

This seemingly arbitrary rule of thumb has a reasonable basis when it comes to trying decide, during study design, how many samples to collect.

If you intend to compare mean values, your study needs to have enough samples to provide adequate power (probability of finding a true difference) to distinguish them with a test having a specified false-positive probability criterion. Often, this is specified as having 80% power to find a true difference when the statistical significance criterion is p < 0.05.

There is a useful rule for how to apply those criteria (80% power at p < 0.05) in the case of a normal distribution: the values have to be at least 2.8 standard errors apart to provide that power. This is nicely illustrated in Figure 20.2 of this chapter from Gelman. So to obtain that power, if you have a reasonable estimate of the standard deviation and a difference that you would like to detect, you choose a sample size that will give you at least that small a standard error. Depending on the design, that could be the number of cases to distinguish the mean of a single group from a pre-specified value (based on its own standard error of the mean), or the numbers of cases in, say, two groups that are expected to have a particular difference between their means (based on the standard error of the difference between the means).

Your rule of thumb specifies that you should aim for a sample size such that the mean value is 5 times the standard error, well over the above criterion. With 2 sets of samples from normal distributions each collected according to this rule of thumb, you would for example have 80% power to distinguish the mean values of those sets if their means differed by about 0.8 standard deviations. Prior work in your field presumably has determined that such magnitudes of differences are of substantive interest.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.