# How robust is ANOVA when group sizes are unequal and residuals are not normally distributed?

I understand that there are degrees of trust one can have in the output of ANOVA. However, naturally I want to maximize the amount I can trust my results.

Question: If I have data that violate the assumption of normality (of residuals), how robust is a factorial ANOVA when cell sizes are unequal? My cell sizes range from 42 to 55, for a 2x3 between-subjects ANOVA.

• Just to clarify, are you talking about one-way ANOVA or factorial ANOVA? In particular, an assortment of issues arise when the data is not balanced in factorial ANOVA (i.e., the predictors are correlated). Commented May 4, 2012 at 1:31
• Good comment; I've edited my question for clarification. Since the groups were all obtained by randomization there should be no correlation between them.
– Lee
Commented May 4, 2012 at 5:38

I suppose it depends on what you mean by "robust", there are different ways the process can go awry. If your residuals are distributed symmetrically, but simply not normally, and you don't have other issues that you don't mention (e.g., missing data, this isn't a factorial ANOVA as @JeromyAnglim points out), then your parameter estimates should be unbiased. On the other hand, depending on how far your data differ from normality, 42 - 55 may not be enough for the Central Limit Theorem to kick in and cover for you. That is, your p-values may be off. How far off (as you may have guessed from the previous sentence), will depend on how much your residuals differ from normality, with just small differences, you're probably fine. On a slightly different note, remember that if your variances are not equal the test will not be maximally efficient (read: reduced power). One other tip: with respect to non-normality, skew, especially with different cells skewed in different directions, is worse than having excess kurtosis not equal to 0.

Update: Unless you have a clear reason for believing that your data come from some other specific distribution (e.g., they're counts), the question is simply about the skewness and the excess kurtosis. The best discussion I've seen of these issues is here. Note that under skewness -> interpreting, there are some arbitrary guidelines that may be helpful, and that under kurtosis -> visualizing, you can see what the possible range of values is [-2, $$\infty$$). Again, the issue is: will the Central Limit Theorem cover for you, and that depends on the manner and extent of non-normality and how much data you have. Answering that analytically is going to be extremely difficult, although it's not too bad via simulation. I've run some simulations using distributions like the uniform and the chi-squared with df=1, to explain the idea of the CLT; even from these very non-normal distributions, the sampling distribution of the mean converges to normal impressively fast, IMO. Thus, my guess is that if you only have a little bit of skewness, you are probably fine, given your sample sizes, but of course I can't give you a final, analytical answer.

The issue with unequal cell sizes in factorial ANOVA is that the factors are correlated with each other. That means that using standard tests (which amounts to using type III sums of squares) don't properly use all of the information available. I discuss these issues here, with some complementary information here.

• The data are skewed, but only slightly, such that transformations of the usual sort (log, square root, inverse) all fail to work. I guess the real question becomes how can I tell "how far" the data differ from normality, and what is an acceptable range to trust the ANOVA results? Also P.S. I mentioned above that it is a 2x3 between-subjects ANOVA.
– Lee
Commented May 4, 2012 at 5:42
• I don't think I saw your updated response before. I wanted to thank you for it, +1. I've had a look at my data by group, and they are mostly in the moderately skewed range for the variables I care about. From your answer and from some other discussions I've had about the CLT, it seems as though my analysis can proceed with ANOVA, although I still wish I could be more certain that this is the correct approach.
– Lee
Commented May 17, 2012 at 3:05
• @Lee, glad to help. We'd all like to be more certain, but I really do think that you're fine here. Incidentally, if this is what you needed, you may want to accept the answer by clicking the check mark next to it. Commented May 17, 2012 at 18:14
• Well, you have given the best guidelines I've seen, certainly. I was holding out for something more concrete for an answer (e.g., specific skew/kurtosis numbers for a given N), but perhaps these don't exist yet. If you ever see them let me know.
– Lee
Commented May 17, 2012 at 21:29

If you are doing a one-way ANOVA, you do not need to worry about the range of Ns you mention. If you are asking before collecting data, or before completion of data collection, you might also undertake a power analysis to get a comfort level that you have reasonable statistical power to detect a real difference if on exists.

• It is a 2x3 factorial ANOVA, and the Ns given are for cells. What do you mean about not worrying, though -- the ANOVA will be robust? Also, I'd love to do a power analysis, but I'm still learning how. Good resources on this would be appreciated.
– Lee
Commented May 4, 2012 at 5:36
• Since you are dealing with a 2-way ANOVA, you need to pay particular attention to Gung's last paragraph. Your question of whether you can trust the ANOVA results may be obliquely asking about the power of your test, that is, the probability that you will reject a false null hypothesis. In some research areas, research with N's such as your will have good power. In other areas, you need much larger N's. Power depends on the effect size as well as N. Commented May 6, 2012 at 2:58