Using a Fixed-Effects ANOVA to help decide whether to conduct a multilevel analysis

Question

Heck et al (2013) write that

Generally, the first step in a multilevel analysis is partitioning the variance (referred to as $\sigma^2$) in an outcome variable into its within- and between-group components. If it turns out that there is little or no variation (perhaps less than 5%) in outcomes between groups, there would be no compelling need for conducting a multilevel analysis.

To that end, they suggest running a Random-Effects ANOVA, from which output like the following gets generated:

This example relates to math test results being predicted from schools. I understand that 66.551 represents variation in individual scores within schools, and 10.642 represents between-school variation. To calculate the intraclass correlation Heck et al take 10.642/(10.642+66.551) = 0.138, and conclude that 13.8% of the variance in achievement lies between schools. They write that "the intraclass correlation provides a sense of the degree to which differences in the outcome variable Y exist between Level 2 units".

Their approach makes some intuitive sense to me. However, what I don't understand is why we can't just run a regular fixed effects ANOVA and compare the between-groups sum of squares with the within-groups sum of squares.

By analogy, I'd take 100708.437/530078.816 = .190, which is not the same value that I got from the Random Effects ANOVA. However, isn't a Fixed-Effects ANOVA also also measuring the "degree to which differences in the outcome Y exist between Level 2 units"?

I understand that output from a Random-Effects ANOVA and a Fixed-Effects ANOVA are not the same thing, but is it wrong (or sub-optimal) to use the results from a Fixed-Effects ANOVA to decide whether to proceed with a multilevel analysis? Why/why not?

Answer 1

There is nothing at all wrong with doing a fixed effects analysis and leaving it there. In fact, there are situations where that would be the best approach. For example, if the effect in question is just a nuisance factor, fixed effects are a good way to reduce the variation in the factor of interest. An example would be a classic agricultural field trial where you are comparing treatments on different fields at the Ag Station. The field effect is fixed and the analysis eliminates that source of variation from comparison of the treatments.

The downside is that you can't use an experiment like that to tell farmers what sort of results they are likely to get in their own fields - only which treatment is better.

In a random effects design, the blocking effects (the fields, or the schools in your case) are selected at random and confidence intervals can be produced to get a sense of what sort of outcomes people might get in practice, allowing for their different circumstances (different schools or different fields).

I sort of don't like the idea of doing a fixed analysis first and then doing a random effect analysis, because strictly speaking the p-value will not longer be what you think it is. Your random effects analysis exists conditionally on having received a significant result on the fixed effects analysis, so the distribution of the random effects model under the null hypothesis is ... some horrible thing that might yield to a simulation but that I would not want to attempt to calculate.

However, real world applied statisticians don't worry about that sort of subtlety. I have sometimes run a fixed analysis as a reality check when the sample size is small, since mixed effects models may run into convergence issues. There are times when it makes sense to do that, I believe.

Bottom line, the difference between fixed and random effects turns on the research question you have and how the data were collected.