Bartlett's test vs Levene's test

Question

I am currently trying to address violations to ANOVA assumptions. I have used Shapiro-Wilk to test normality, and have dabbled with both Levene's test and Bartlett's test of variance equality. I have since log transformed my data to try and remedy the unequal variances. I reran the Bartlett's test on the log transformed data, and still received a significant p-value, and out of curiosity also ran the Levene's test and got a non-significant p-value. Which test should I rely on?

Answer 1

Probably neither. It would be better to look at your data and see how bad the violations are. Linear models (e.g., ANOVA) are fairly robust to minor violations when the group $n$s are equal. A rule of thumb for heteroscedasticity is that the maximum group variance can be as much as 4 times the minimum group variance without too much damage to your analysis. If you are worried that there may be violations, an even better approach is to simply use analyses that are robust to the possible violations from the start, rather than trying to detect violations and then make decisions based on that¹.

For what it's worth, Wikipedia says that Bartlett's test is more sensitive to violations of normality than Levene's test. So you may have non-normal data instead of heteroscedastic data. Again, a more robust analysis may be preferable².

<sub>1. See: A principled method for choosing between t test or non-parametric e.g. Wilcoxon in small samples.
2. For various ways of dealing with problematic heteroscedasticity, see: Alternatives to one-way ANOVA for heteroskedastic data.</sub>

Answer 2

For a less sensitive test for non-normal conditions than Levene's test at least sometimes use Conover's test, AKA squared ranks non-parametric test. I have found this to be at least sometimes preferred to Bartlett's test in the Mathematica implementation of the VarianceEquivalenceTest.

Here is a list of variance tests methods and assumptions copied from the Variance Equivalence link above

 Bartlett       normality       modified likelihood ratio test
 BrownForsythe  robust          robust Levene test
 Conover        symmetry        Conover's squared ranks test
 FisherRatio    normality       based on variance ratio
 Levene         robust,symmetry compares individual and group variances 

What should be obvious from that list is that the violations of assumptions are testable, although the Mathematica documentation is not specific as to how, for example, the Conover symmetry test is being performed, or even why one tests for symmetry. And, so far no one has answered that question.

So, the answer to the OP question is that only testing of conditions can suggest which method is preferable in any particular case. Moreover, if all 5 tests are attempted, and are not excluded because of violation of assumptions, then one can generally distinguish between better and worse answers with whichever answers are generated.

As a worst case, one can perform Monte Carlo simulation using known truth values to explore which conditions lead to what probabilities. But, without more information as to the problem itself, the question cannot be answered in terms of the OP's data set. If the OP wants a data oriented specific answer, please provide the data.

Answer 3

If any test of assumptions gives you a very low p-value, evaluate the data graphically to determine the interpretation and most appropriate remedies. Some of us think the graphs more more important than the tests. The most common transformations, including logarithmic, will correct positive skewness (tail drawn out more towards large values) but I doubt would help you with an outlier on the low end. Also, in case other investigators in your field have dealt with the same response variable, what do they consider most useful?

As for Bartlett, the p-value is computed under assumptions of independence, normality, and equal variance. If the p-value is very small, there is evidence that one or more of those assumptions is false. So, for example, if you have a big outlier the test is likely to be "significant." (It does not loose power for heavy tailed distributions.) In that situation, you might think of an outlier as a deviation from normality, or as a deviation from equal variances (between the group that has it, and the others). Does that matter? Either way the ANOVA F test would be unreliable and the rank based analogue (Kruskal-Wallis) may be a better bet. The low p-value from Bartlett may help to support your decision.

Thus in the scenario I am describing it would seem silly for you to be taken to task with some knee-jerk thing about sensitivity of Bartlett to normality. The entry in Wikipedia might be edited to take the relevant literature into account, perhaps starting with the Enc. Stat. Sciences entry.