Bartlett's test vs Levene's test

Question

I am currently trying to address violations to ANOVA assumptions. I have used Shapiro-Wilk to test normality, and have dabbled with both Levene's test and Bartlett's test of variance equality. I have since log transformed my data to try and remedy the unequal variances. I reran the Bartlett's test on the log transformed data, and still received a significant p-value, and out of curiosity also ran the Levene's test and got a non-significant p-value. Which test should I rely on?

Answer 1

Probably neither. It would be better to look at your data and see how bad the violations are. Linear models (e.g., ANOVA) are fairly robust to minor violations when the group $n$s are equal. A rule of thumb for heteroscedasticity is that the maximum group variance can be as much as 4 times the minimum group variance without too much damage to your analysis. If you are worried that there may be violations, an even better approach is to simply use analyses that are robust to the possible violations from the start, rather than trying to detect violations and then make decisions based on that¹.

For what it's worth, Wikipedia says that Bartlett's test is more sensitive to violations of normality than Levene's test. So you may have non-normal data instead of heteroscedastic data. Again, a more robust analysis may be preferable².

<sub>1. See: A principled method for choosing between t test or non-parametric e.g. Wilcoxon in small samples.
2. For various ways of dealing with problematic heteroscedasticity, see: Alternatives to one-way ANOVA for heteroskedastic data.</sub>

Answer 2

For a less sensitive test for non-normal conditions than Levene's test at least sometimes use Conover's test, AKA squared ranks non-parametric test. I have found this to be at least sometimes preferred to Bartlett's test in the Mathematica implementation of the VarianceEquivalenceTest.

Here is a list of variance tests methods and assumptions copied from the Variance Equivalence link above

 Bartlett       normality       modified likelihood ratio test
 BrownForsythe  robust          robust Levene test
 Conover        symmetry        Conover's squared ranks test
 FisherRatio    normality       based on variance ratio
 Levene         robust,symmetry compares individual and group variances 

What should be obvious from that list is that the violations of assumptions are testable, although the Mathematica documentation is not specific as to how, for example, the Conover symmetry test is being performed, or even why one tests for symmetry. And, so far no one has answered that question.

So, the answer to the OP question is that only testing of conditions can suggest which method is preferable in any particular case. Moreover, if all 5 tests are attempted, and are not excluded because of violation of assumptions, then one can generally distinguish between better and worse answers with whichever answers are generated.

As a worst case, one can perform Monte Carlo simulation using known truth values to explore which conditions lead to what probabilities. But, without more information as to the problem itself, the question cannot be answered in terms of the OP's data set. If the OP wants a data oriented specific answer, please provide the data.