What affects Levene's-Test Sig. value?

Question

I did a Standard Error Plot of my two treatments, and homogeneity of variance appeared to be justified (there was only a slight difference in variance).

However, then I ran Levene's during my independent t-test and got a Sig. value of 0.000.

My sample size was in the thousands for each treatment, so I was wondering how does sample size affect the outcome of Levene's? My assumption is that: as sample size increases, the accuracy of the sample in relation to the population increases and therefore only a slight difference in variance would be really magnified.

Some clarification on this would be awesome, thanks!

Answer 1

Your intuition is correct. As with all tests, a higher $n$ will lead to higher power all other things being equal. A larger actual difference in variances obviously also makes it more likely that the Levene test can reject the null hypothesis. But with large $n$, even effects (here difference in variance) that are practically too small to worry about become statistically significant.

You will also note that using the Levene test this way (and I don't really know another way to use it) brings a fundamental conflict of interest: You want to prove the null hypothesis! You do this Levene test to conclude equality of variances, not to reject it. But you shouldn't try to prove the null hypothesis. Trying to prove the null hypothesis rewards lazy data collection or in your case penalizes abundant data.

If your primary goal is to learn about those variances, your best shot are equivalence testing or confidence interval based approaches. With a high $n$, you will have rejected $H_0 \quad \sigma_1=\sigma_2$, but you can also get an upper- and lower bound of that difference from the confidence interval. The larger your $n$, the more precisely you will know how much difference there is in variances.

It is more likely that you did that test as a precursor to another test that requires equality of variances. Know that unequal variances are only really a problem when the sample sizes are very different as well. Also, for t-tests for example you could just use the Welch version and not worry about variances at all.

Answer 2

What affects Levene's-Test Sig. value?

1. differences in spread in the sample value

2. sample size, at least when the null is not strictly true (as with any consistent test)

My sample size was in the thousands for each treatment,

This is why tiny differences are significant ... and it's also why hypothesis tests of assumptions of procedures make little sense. They answer the wrong question. The relevant question isn't "is this assumption true" (it almost certainly isn't, not exactly), it's "how much difference does it make?" and that's not what the hypothesis test tells you about. It takes a fairly substantial difference in variance to badly impact the analysis in a really appreciable way; with large samples you'll detect trivial differences.

Correspondingly failure to reject in a test of assumptions is also of little comfort if you lack power to detect differences that matter to the original analysis.

There's also the problem that choosing between procedures on the basis of the test of hypothesis impacts the properties of the procedures you're choosing among.

A diagnostic plot that lets you see how different the spreads are is a more useful indicator than a test (it shows you how different they are, which is somewhat closer to what you need to know)

An equivalence test -- if you could identify suitable bounds on what would be reasonable to call equivalent would make more sense, but it would still have an issue with the "choosing between procedures impacts their properties" part.