Variance ratio below 3

Question

Can it be assumed that the repeated measures (RM) ANOVA is robust, even after having a significant Levene test for a variable? What about if the ratio between the maximum and minimum variance is below 3?

Answer 1

I'm not sure that there is some magic number when it comes to the Levene test. This Monte Carlo simulation paper I read recently seems to suggest a ratio of 1.5 and below is ideal, but I still have my qualms about adding yet another arbitrary criterion to the list (previously others have suggested maximum ratios of 3, 4, 5, and even 10 in recent years). In any case, I don't tend to trust $p$-value based assumption checking, at least not by itself, for the following reasons:

• The outcome is highly dependent on how many observations are used. If we have too many observations, then it will be overpowered and flag any minor differences in variance between groups. If we have too few observations, then the results may be unreliable due to more erratic sampling error from small samples.
• The hypothesis that drives it, the null, is silly. The basic null hypothesis in Levene's test is the following: Is the difference in variances among groups essentially zero? In practice, this will never be the case, and so we are forced to ask ourselves whether this even matters. Only visualization and descriptive statistics can actually answer the question of how much is too much.
• The premise of equality of variance is dubious anyway. On a related note, how often is it the case that groups vary exactly or almost exactly? Almost universally this is an issue in observational data, and this can only be approximately controlled in experimental designs like RCTs.

Regarding the ratio technique, the linked paper above still makes the following caveats:

First, F-test is robust with monotonic patterns of variance when the group sample sizes are equal, regardless of the number of groups, of the ratio between the largest and smallest variance, and of the total sample size. With a variance ratio as large as 9, F-test can, at least for the number of groups and sample sizes considered here, still be used without the Type I error rate being affected by heterogeneity when the design is balanced.

Second, F-test is not robust with unequal sample sizes under certain conditions. The results showed that, in general, robustness depends on the variance ratio, the pairing of variance with group size, and the coefficient of sample size variation, with the procedure being more robust when variance ratios are small, the pairing of variance is either zero or positive, and the coefficient of sample size variation is smaller.

So it seems like everything in statistics, it depends. I'd at least plot the differences in variance and see if it really is the big deal your Levene test suggests before making any moves to fix it. One can always utilize a non-parametric version of ANOVA in these cases (here the Friedman test), but non-parametric tests come at the cost of statistical power.