Can I trust ANOVA results for a non-normally distributed DV? I have analyzed an experiment with a repeated measures ANOVA. The ANOVA is a 3x2x2x2x3 with 2 between-subject factors and 3 within (N = 189). Error rate is the dependent variable. The distribution of error rates has a skew of 3.64 and a kurtosis of 15.75. The skew and kurtosis are the result of 90% of the error rate means being 0. Reading some of the previous threads on normality tests here has me a little confused. I thought that if you had data that was not normally distributed it was in your best interest to transform it if possible, but it seems that a lot of people think analyzing non-normal data with an ANOVA or a T-test is acceptable. Can I trust the results of the ANOVA?  
(FYI, In the future I intend to analyze this type of data in R with mixed-models with a binomial distribution)
 A: Specifically regarding error rates as a DV, Dixon (2008) very cogently demonstrates that null hypothesis testing via ANOVA can cause both increased false alarm rates (calling effects "significant" when they're not) and increased miss rates (missing real effects). He also shows that mixed effects modelling, specifying binomially distributed error, is the more appropriate approach to analyzing rate data. 
A: You cannot trust your ANOVA with that much skew and a large number of 0s.  A more appropriate method would be to use number of errors as your DV (thus turning your DV into count data) and doing a Poisson analysis.  This approach would require using a mixed effects analysis and specifying the error distribution family as Poisson.  The Dixon (2008)* article mentioned by Mike Lawrence uses mixed effects analysis in R but with binomial outcomes.  I have completely moved to doing R for most of my repeated measures analyses because so many of my outcome variables are binomial.  The appropriate R package is lme4.
$*$ Dixon, P. (2008). Models of accuracy in repeated-measures designs. Journal of Memory and Language, 59(4), 447-456.
A: Like other parametric tests, the analysis of variance assumes that the data fit the normal distribution. If your measurement variable is not normally distributed, you may be increasing your chance of a false positive result if you analyze the data with an anova or other test that assumes normality. Fortunately, an anova is not very sensitive to moderate deviations from normality; simulation studies, using a variety of non-normal distributions, have shown that the false positive rate is not affected very much by this violation of the assumption (Glass et al. 1972, Harwell et al. 1992, Lix et al. 1996). This is because when you take a large number of random samples from a population, the means of those samples are approximately normally distributed even when the population is not normal.
It is possible to test the goodness-of-fit of a data set to the normal distribution. I do not suggest that you do this, because many data sets that are significantly non-normal would be perfectly appropriate for an anova.
Instead, if you have a large enough data set, I suggest you just look at the frequency histogram. If it looks more-or-less normal, go ahead and perform an anova. If it looks like a normal distribution that has been pushed to one side, like the sulphate data above, you should try different data transformations and see if any of them make the histogram look more normal. If that doesn't work, and the data still look severely non-normal, it's probably still okay to analyze the data using an anova. However, you may want to analyze it using a non-parametric test. Just about every parametric statistical test has a non-parametric substitute, such as the Kruskal–Wallis test instead of a one-way anova, Wilcoxon signed-rank test instead of a paired t-test, and Spearman rank correlation instead of linear regression. These non-parametric tests do not assume that the data fit the normal distribution. They do assume that the data in different groups have the same distribution as each other, however; if different groups have different shaped distributions (for example, one is skewed to the left, another is skewed to the right), a non-parametric test may not be any better than a parametric one.
References


*

*Glass, G.V., P.D. Peckham, and J.R. Sanders. 1972. Consequences of failure to meet assumptions underlying fixed effects analyses of variance and covariance. Rev. Educ. Res. 42: 237-288.

*Harwell, M.R., E.N. Rubinstein, W.S. Hayes, and C.C. Olds. 1992. Summarizing Monte Carlo results in methodological research: the one- and two-factor fixed effects ANOVA cases. J. Educ. Stat. 17: 315-339.

*Lix, L.M., J.C. Keselman, and H.J. Keselman. 1996. Consequences of assumption violations revisited: A quantitative review of alternatives to the one-way analysis of variance F test. Rev. Educ. Res. 66: 579-619.

A: Juan has offered a lot, although I'll echo others and repeat that for best accuracy the variables themselves can be nonnormal as long as their residuals aren't.  Also, a simplified and slightly more structured answer (via an annotated flow chart) is available at yellowbrickstats.com.
A: Ceiling effects are the problem here. A non parametric test is your safest bet, although ANOVAs are robust to this violation of normality if n is large. Typically people just use a histogram to test this, but if the issue is with residuals it might be more advanced than that. Also bear in mind HOW this effects your results (not just that it does). Pallant (2007) would probably say this increases your chance of type one error, so if you reduce your critical alpha you mitigate that. 
