2
$\begingroup$

Background: It has been shown and widely referenced (applets even exist, etc.) that for even a highly-skewed numeric variable, a sample size of $n\ge{}$30 is often "large enough" for the Central Limit Theorem (CLT) to take effect, and thus for the distribution of the sample mean to be considered normal for the purposes of inference. Some sources do suggest $n$ = 40 or even 50 for very highly-skewed data, however.

Query: It seems then, that for the one-way ANOVA, this suggestion should apply to each factor level (group) as well, since the data in each factor level need to be individually normally distributed (or the sample size "large enough") to meet the normality assumption of the one-way ANOVA. I'm wondering if anyone has any references that address this specifically.

I'm mostly curious because this post from Minitab (linked) says simulation studies have shown for:

  • 2-9 groups, $n \ge$ 15 for each group is sufficient
  • 10-12 groups, $n \ge$ 20 for each group is sufficient

The wording is somewhat vague, but the author implies these values are when the data are highly skewed (and this is when we would care anyway). No citation is given and I'm having quite the time finding other explicit discussions of the CLT in the framework of a one-way ANOVA.

I've found one source that suggests the 30 per group cut-off, but it's a text (preview given at link below) and I have no way to track down the reference they used for this statement: Biostatistics: The Bare Essentials (Norman & Streiner): "From the Central Limit Theorem (Chapter 4), the means will be normally distributed, regardless of the original distribution, especially when there are at least 30 or so observations per group."

Sources I've tried to no avail:

  • Design and Analysis of Experiments (8th Ed.), Montgomery: Doesn't discuss CLT for a one-way ANOVA
  • Design of Experiments: Statistical Principles of Research Design and Analysis (2nd Ed.), Kuehl: Doesn't discuss
  • Statistical Inference (2nd Ed.), Casella & Berger (pg. 524): "Of course, with reasonable sample sizes and populations that are not too asymmetric, we have the Central Limit Theorem (CLT) to rely on."
  • Extensive internet search that gives various suggestions, including sometimes for the ANOVA overall (overall sample size across all factor levels), which doesn't make sense to me (doesn't seem this would alleviate the issue in any one group): No references provided by any sources

Summary: I feel that to be "safe," if any given factor level in a one-way ANOVA is notably skewed, we should have at least a sample size of 30 in that factor level to ensure the CLT has taken effect. However, I would appreciate references that confirm this.

$\endgroup$
  • $\begingroup$ What makes things pretty complicated is that not only non-normality might invalidate severly the F-Test, but also non-equality of true group variances. Both assumptions are never 100% met, but it is difficult to specify settings in which they are sufficiently met. $\endgroup$ – Michael M Feb 14 '16 at 11:38
  • $\begingroup$ Yes, this assumption would need to be considered as well. Here, I'm assuming that we're correctly accounting for unequal variances if they exist, but still have skewed data, e.g. $\endgroup$ – Meg Feb 14 '16 at 15:29
2
$\begingroup$

That is not a correct interpretation of the CLT. The CLT is a limiting argument and only helps you with respect to type I error, not type II error. Confidence intervals using the CLT can be horrendously inaccurate for sample sizes in the thousands when the data distribution is very skewed (e.g., lognormal distribution). If the 2-sample $t$-test doesn't work in the face of much asymmetry, ANOVA will not fare any better. Note also that the CLT in how it's usually invoked only works when the population variance is known.

Rand Wilcox has written nice papers showing that with data distributions that are only slightly non-normal the distribution of the $t$ statistic can be very far from the $t$ distribution. Note that it is not relevant that critical values of the $t$ distribution are very close to $z$ critical values for $n\geq 20$ or $30$; these both pertain to Gaussian data.

Perhaps the easiest way to understand why the CLT is irrelevant to the real world is to remember that the standard deviation is only a very good measure of dispersion if the data distribution is symmetric and not too heavy tailed, and to note that a single outlier can destroy the standard deviation estimate.

$\endgroup$
  • $\begingroup$ Thank you, Frank. I actually was not aware that the CLT was not applicable for the two-sample t-test - I thought that if each group, individually, had a "large" sample size, the CLT applied. So is it correct that the CLT only applies to the one-sample t-test? Do you have a reference for the lack of applicability of the CLT in the case of more than one group? Also note I edited my post, because I did find one text that talked about 30 observations per group. Would you then say that this text, along with the Minitab blog post, are incorrect? $\endgroup$ – Meg Feb 13 '16 at 23:21
  • $\begingroup$ And, actually, add Casella & Berger to that list (their quote regarding the CLT and ANOVA is in my post, although the do say, "...populations that are not too asymmetric..." - is this the key?). I also can find countless examples from .edu websites that apply the CLT to the two-sample t-test, so it seems this is a common misconception. $\endgroup$ – Meg Feb 13 '16 at 23:25
  • $\begingroup$ It is ultimately applicable to the $t$-test but "ultimate" may correspond to an enormous sample size. It applies equally (bad or good) to 1-sample tests, 2-sample $t$ test, and ANOVA. The text quoting 30 per group is incorrect. See how the CLT fails for n=20,000 in an example in stats.stackexchange.com/questions/186957/… $\endgroup$ – Frank Harrell Feb 13 '16 at 23:27
  • 1
    $\begingroup$ I answered these questions by expanding my original answer. The CLT has misled practitioners for a very long time. We statisticians are to blame because of how we teach the CLT. Most courses covering it do not even mention type II error in the CLT context! $\endgroup$ – Frank Harrell Feb 14 '16 at 13:47
  • 1
    $\begingroup$ @Meg In another question on site here I gave a different example where n=20000 was not quite sufficient to apply a two sample t-test (a two-group anova); in the example I had the underlying distribution was gamma. $\endgroup$ – Glen_b Feb 14 '16 at 15:31
1
$\begingroup$

The t-statistic has a numerator and a denominator.

For this discussion, assume both numerator and denominator are now divided by $\sigma/\sqrt{n}$. The numerator is then of the standardized form required for the CLT.

The CLT only gives you that (under certain conditions) as $n\to\infty$ a standardized numerator will go to standard normal. the CLT does not addresss the denominator and it does not address the rate at which normality is approached.

You can invoke Slutsky's theorem for a ratio to get an asymptotic normal distribution for the t-statistic, but that's still not saying anything about the rate.

For that you need something like Berry-Esseen (which does at least tell us something about how quickly the numerator might go toward the normal); a quick glance at that theorem will show you why a rule like "n=30" doesn't make sense.

Note that Berry-Esseen also tells us that the quote in your question "not too asymmetric" still isn't quite getting at the issue even so, since one may have a symmetric distribution for which the absolute third moment is not small.

However even with all that extra machinery we still haven't dealt with the rate at which the entire statistic goes to normal. What is clear at least is that "n=30" isn't any use unless we restrict consideration to cases which would make the rule reasonable, but I've seen no suitable rule of thumb which is useful for deciding when the original rule of thumb (if it deserves the name at all) applies.

(My suggestion is to use simulation to investigate the behavior of the statistic under a variety of conditions relevant to the kinds of problems once faces; these will tend to be different in different areas of application.)

$\endgroup$
  • $\begingroup$ Thank you for your comments, Glen. They're clarifying. As such, at the bottom of pg. 153 of the paper at rctdesign.org/techreports/arphnonnormality.pdf, would you say the statement, "That is, the Central Limit Theorem guarantees that the t-statistic in Equation 2 will be normally distributed, but it may not have variance equal to 1.This is not a problem in practice because we can always use Equation 1..." is simply incorrect? Or not, because it could take effect eventually, we just don't know when (so "in practice" this statement about the CLT really isn't helpful)? $\endgroup$ – Meg Feb 15 '16 at 12:57
  • $\begingroup$ You have a Welch-Satterthwaite type of t-statistic there in eqn 1 (compared with the usual two-sample statistic in eqn 2). In both cases the CLT alone does not guarantee normality; it will give asymptotic normality of an appropriately scaled numerator. Invoking Slutzky's theorem along with CLT would give normality for the statistic. But asymptotic normality doesn't tell you about the behavior at specific finite sample sizes. What's there isn't wrong so much as the correct justification is missing, and what is justifiable is not automatically useful. $\endgroup$ – Glen_b Feb 16 '16 at 1:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.