When a population is not normal, can F ratio be used for ANOVA analysis?

Question

As it is known a F ratio assumes that the random variables in the numerator and denominator (variances in the case of ANOVA) follow a chi squared distribution, which is true for the sampling distributions of variances of normal distributed populations. However, if a large sample is taken which is not normally distributed, we cannot make inferences about the distribution of the population and so we cannot state that the variances follow a chi squared distribution. Is in this case valid to make an ANOVA analysis (assuming the rest assumptions are true)? (for large sample sizes the normality assumption is not required due to CLT, meaning that the samples means are good estimators of the population mean, but I cant find any information about the variances)

Answer 1

There are a few senses in which we might say "yes, at least sort of", depending on what you are prepared to modify. For example, the distribution you use for critical values/p-values, what you calculate the statistic on, or modifications to the form of the statistic itself. I don't present an exhaustive list (e.g. I haven't discussed data transformation beyond briefly mentioning the rank transform).

For what follows I'm going to be assuming you're in a one-way ANOVA-like situation (comparing means - or perhaps some other way of assessing location - of $k$ groups, with an interest in testing whether the population means are not all the same). Some of the discussion would carry over more broadly (and I do mention more general situations once or twice in passing).

1. You could use the usual F-statistic, but it won't generally have an F-distribution, and particularly in smaller samples (a few tens of observations per sample, or fewer say) or with substantial skewness/heavy tails it might not be close enough for your purposes, particularly if you're going well out into the tails (adjusting typical $\alpha$s for many multiple comparisons say). That said, it's reasonably level robust to 'moderate' deviations from normality if $\alpha$ is not too small (in a similar sense to a t-test in the two sample case).

   The issue with level can be dealt with in smallish samples by using a permutation test (noting that under the null, the values in different groups should be exchangeable). This should work well as long as sample sizes are not so small that the available significance levels becomes too limited, though usually by that time you have larger concerns. In somewhat largish samples, especially for situations that are more complex, particularly where you don't have exchangeability under the null, you might consider a bootstrap test instead.

   However, getting the level close to right with a permutation approach doesn't solve the issue with power; it can easily happen that the F-statistic is not making anywhere near efficient use of the information in the sample about differences in mean.

   Where you think you have some other nearly suitable parametric model (as in 2.) but are not quite so confident that it's accurate enough to get the level close enough, some other parametric test statistic can be used in a permutation test. If the model happens to be right (or close to it in the right sense), the test should still have good power when you use such a test, but at least the level will be correct (up to the effect of the discreteness of the permutation distribution of the test statistic) even if the model is not particularly accurate.

2. If you expect that some particular kind of distribution will be a reasonable model for the conditional distribution of the response variable, you may be able to obtain an efficient test for the differences in mean via a generalized likelihood ratio test type approach (or some asymptotically equivalent test).

   This is particularly convenient when the model comes from the exponential dispersion family (including Gamma, Poisson, binomial, inverse Gaussian, etc), where you get generalized linear models.

   This will not be an F-ratio as such, but there are corresponding quantities involving the deviance.

   However, other models could be used; e.g. some other distributions are conveniently implemented in parametric survival regression packages which would allow testing the hypothesis of interest with little additional effort. Or you could go directly to working with the likelihood, with a little more effort.

3. You could use an F statistic with a rank based test instead - in particular a Kruskal-Wallis test statistic is equivalent to (/a monotonic function of) an F-statistic computed on the ranks. The difficulty here is that any of the rank based tests will not inherently correspond to a test of means, though under assumptions that give a pure location shift alternative it would be suitable.