School-level covariates without a multilevel model-how does it affect results?

Question

I'm reviewing a paper and have a question about a stats choice the authors made about how to deal with the nested structure of their data.

The analysis is a piecewise growth model exploring trajectories before and after the transition to high school. The dataset is based on about 2000 students drawn from 11 middle schools who are followed from 7th to 10th grade. Based on descriptive information, there are clearly school-level differences. The authors chose not to add a level 3 for clustering within schools (because of substantial missing data, about 20%, in middle-school to high school "school feeder patterns"). Instead, the authors account for these differences with school-level covariates.

My questions are these.

1. In a longitudinal study, won't the standard errors be significantly biased if you don't account for the complex structure of the data (clustering by school)?

2. If the authors included school-level covariates in the model at the same level as individual-level variables, how will that affect their results? Is this an appropriate alternative or will it cause significant problems? Are those problems just the biased SEs or something else?

Note: They also mention that "when running models with the CLUSTER function on the reduced sample results are consistent with those reported here." They don't report the difference in SEs, though. So, I'm not sure what to make of that, because the coefficients might be consistent, but if the SEs are different, you might draw different conclusions.

Answer 1

This is a very complex question without a simple answer. There are two reasons to use multilevel modeling: one is to account for cluster-dependent errors second is to obtain correct degrees of freedom for inference. It sounds like the primary exposure is individual level, such that kids with and without this value are coresiding within clusters. If it is a between-cluster covariate, then the inference must be performed differently.

Theoretically, the only reason cluster-dependent errors arise is because of (possibly 1,000s of) unmeasured fixed effects which are common to all individuals within a cluster. Adjusting for school level covariates partly explains between-cluster heterogeneity, and thus reduces the intraclass correlation coefficient. Extant research about school "choice" suggests that fixed effects can actually do quite a lot to explain between-school heterogeneity: median income, percent below FPL, racial distribution, and density/deprivation indexes are important predictors of most outcomes.

Mixing cluster level and individual level fixed effects is totally kosher. Inference on cluster-level effects and individual-level effects are performed differently however (they have different degrees of freedom). It's important to recall that the conditional interpretation of fixed effects changes by the addition of other covariates. Ecological fallacy is often observable in such analyses. For instance, household income can often times take on unwieldy values because rich people live in rich neighborhoods and poor people live in poor neighborhoods, and the neighborhood is the ultimate factor that confers protective/deleterious effects. When adjusting for both median neighborhood income and household income is the effect of the latter summarizes living as a poor person in a rich neighborhood or a rich person in a poor neighborhood (or some mixture of the two) which defies our expectation--but it was our expectation that was wrong.

Exploratory analyses should actually address the sufficiency of cluster level fixed effects to reduce intraclass correlations. The most important one is to actually calculate the intraclass correlation coefficient before and after addition of those covariates. The change should be substantial and/or the resulting ICC should be sufficiently small.

Many people think that multilevel models means running random effects. Random effects models can have convergence issues. Generalized estimating equations actually provide point estimates which are consistent with unconditional models, but correct standard errors that account for heterogeneity, and their ease of use is a bonus. I don't know what "Running effects with CLUSTER" means, although for scientific publication, the authors should not appeal to the software jargon but rather use correct statistical terminology. It sounds like Stata and I presume it fits either a mixed effects model or GEE. They should say which.

The note about running effects with CLUSTER is a pet-peeve of mine: technically the cluster model is the "right" model. If the effects didn't change, then fit the right model rather than defend the "possibly right" model.