18
$\begingroup$

A few months ago I posted a question about homoscedasticity tests in R on SO, and Ian Fellows answered that (I'll paraphrase his answer very loosely):

Homoscedasticity tests are not a good tool when testing the goodness of fit of your model. With small samples, you don't have enough power to detect departures from homoscedasticity, while with big samples you have "plenty of power", so you are more likely to screen even trivial departures from equality.

His great answer came as a slap in my face. I used to check normality and homoscedasticity assumptions each time I ran ANOVA.

What is, in your opinion, best practice when checking ANOVA assumptions?

Improve this question
$\endgroup$

5 Answers 5

Reset to default
18
$\begingroup$

In applied settings it is typically more important to know whether any violation of assumptions is problematic for inference.

Assumption tests based on significance tests are rarely of interest in large samples, because most inferential tests are robust to mild violations of assumptions.

One of the nice features of graphical assessments of assumptions is that they focus attention on the degree of violation and not the statistical significance of any violation.

However, it's also possible to focus on numeric summaries of your data which quantify the degree of violation of assumptions and not the statistical significance (e.g., skewness values, kurtosis values, ratio of largest to smallest group variances, etc.). You can also get standard errors or confidence intervals on these values, which will get smaller with larger samples. This perspective is consistent with the general idea that statistical significance is not equivalent to practical importance.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ +1 for the great answer that wraps everything up. How to apply the mentioned numerical procedures is nicely and applicably described in Tabachnik and Fidell's Using Multivariate Statistics (for SPSS and SAS): amazon.com/Using-Multivariate-Statistics-Barbara-Tabachnick/dp/… (But see the Erratas on the accompanied web page) $\endgroup$
    – Henrik
    Commented Sep 20, 2010 at 7:53
  • $\begingroup$ Well, I think most of the time summaries like skewness and kurtosis have little value, their sampling variation is just to large. One could consider replacing them with L_skewness and L-kurtosis, though. $\endgroup$
    – kjetil b halvorsen ♦
    Commented Mar 11, 2016 at 13:03
  • $\begingroup$ @kjetilbhalvorsen I guess it depends on what kind of sample sizes you typically work with. In my experience, plots and and skewness stats are very helpful in understanding the distribution of the data. $\endgroup$
    – Jeromy Anglim
    Commented Mar 11, 2016 at 22:57
  • $\begingroup$ @Jeromy Anglim: OK. Then I guess you usually have very big sample sizes! Did you try to bootstrap your skewness/kurtosis coefficients? $\endgroup$
    – kjetil b halvorsen ♦
    Commented Mar 11, 2016 at 23:05
16
$\begingroup$

A couple of graphs will usually be much more enlightening than the p value from a test of normality or homoskedasticity. Plot observed dependent variables against independent variables. Plot observations against fits. Plot residuals against independent variables. Investigate anything that looks strange on these plots. If something does not look strange, I would not worry about a significant test of an assumption.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Good advice most of the time, but what about the case of large datasets, where you can't feasibly look through all the data manually? $\endgroup$
    – dsimcha
    Commented Sep 19, 2010 at 0:51
  • 1
    $\begingroup$ @dsimcha It also depends of sample size per group. It is known, for example, that when the samples are of equal size the t-test is robust against departure from the homoscedasticity assumption; if $n_1\neq n_2$, then the probability of a type I error will be $<\alpha$ if the larger $\sigma^2$ is associated with the larger sample, and vice versa. See Zar, JH Biostatistical Analysis (4th Ed., Prentice Hall, 1998) for further references. $\endgroup$
    – chl
    Commented Sep 19, 2010 at 13:17
  • 2
    $\begingroup$ @dsimcha re large datasets: depends on what you mean by "large". Many observations? Use good graphics (boxplot, jittered dotplots, sunflowerplots). Many independent variables? Yes, you have a point there... But if you have so many IVs that you can't plot the DV against each IV, I'd question using an ANOVA at all - it looks like it may be hard to interpret in any case. Some smart machine learning approaches may be better (Brian D. Ripley: "To paraphrase provocatively, 'machine learning is statistics minus any checking of models and assumptions'.") $\endgroup$
    – Stephan Kolassa
    Commented Sep 19, 2010 at 19:13
  • $\begingroup$ Good comment, +1. Even though this specific question is about ANOVA, I was thinking on a more general level about the question of plots vs. tests when I wrote my response. $\endgroup$
    – dsimcha
    Commented Sep 21, 2010 at 1:28
5
$\begingroup$

The are some very good web guides to checking the assumptions of ANOVA & what to do if the fail. Here is one. This is another.

Essentially your eye is the best judge, so do some exploratory data analysis. That means plot the data - histograms and box plots are a good way to assess normality and homoscedascity. And remember ANOVA is robust to minor violations of these.

Improve this answer
$\endgroup$
5
$\begingroup$

I agree with others that significance testing for assumptions is problematic.

I like to deal with this problem by making a single plot that exposes all the model assumptions needed to have accurate type I error and low type II error (high power). For the case of ANOVA with 2 groups (two sample t-test) this plot is the normal inverse of the empirical cumulative distribution function (ECDF) stratified by group (see QQ plot comment in an earlier post). For the t-test to perform well, the two curves need to be parallel straight lines. For the $k$-sample problem of ANOVA in general you would have $k$ parallel straight lines.

Semi-parametric (rank) methods such as the Wilcoxon and Kruskal-Wallis tests make far fewer assumptions. The logit of the ECDF should be parallel for Wilcoxon-Kruskal-Wallis tests to have maximum power (type I error is never a problem for them). Linearity is not required. Rank tests make assumptions about how distributions of different groups are related to other, but do not make assumptions about the shape of any one distribution.

Improve this answer
$\endgroup$
1
4
$\begingroup$

QQ Plots are pretty good ways to detect non-normality.

For homoscedasticity, try Levene's test or a Brown-Forsythe test. Both are similar, though BF is a little more robust. They are less sensitive to non-normality than Bartlett's test, but even still, I've found them not to be the most reliable with small sample sizes.

Q-Q plot

Brown-Forsythe test

Levene's test

Improve this answer
$\endgroup$
1
  • $\begingroup$ Relative-distribution plots (or instance, comparing to the normal distribution) could be a good replacement, since their interpretation could be clearer for beginners. $\endgroup$
    – kjetil b halvorsen ♦
    Commented Mar 11, 2016 at 13:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.