Implicit Selection Effects in Mixed Models

I'm struggling with a question that I haven't seen talked about anywhere. Potentially it just illustrates some ignorance I have about how effects are estimated in mixed models.

Let's set up the problem such that we have an educational intervention treatment that is being administered to students within school settings. The hope is that this improves the student passing rate of their grade, a binary variable pass. However, the students being embedded in schools and classrooms (school, class) needs to be controlled for with random effects. Assuming the treatment effect is constant for each student we could model this at the student level with with:

glmer(pass ~ treatment + (1 | school/class), family='binomial' ...)

However, let's assume that the treatment effect is not distributed as randomly as we would like. Some classrooms administered it correctly and half their class were treated while the other half were not treated (split_treatment). Some classrooms treated all their students by accident (all_treatment), others forgot completely that they were supposed to administer this treatment (missed_treatment). The number of classes fitting each criteria are illustrated below:

split_treatment: 50
all_treatment: 500
missed_treatment: 500


In this situation, is the fixed effect for treatment essentially only estimated from the split_treatment classes that contain students who were both treated and untreated?

How would we account for or investigate potential disagreement between models? For example, imagine the specific mixed-effects model above indicates that the treatment improves student's pass rate. However, a solely fixed effects model for glm(pass ~ treatment) shows a negative impact of the treatment, ie. we essentially see Simpson's paradox with this data. Should we just chalk it up to the selection effects of treatment administration? I'm struggling with the fact that it feels like we ignore 1000 all_treatment or missed_treatment classes while potentially only estimating based on the 50 split_treatment classes.

My answer depends on you accurately measuring whether or not a student got the treatment. If you assigned treatment randomly and then didn't keep track of what teachers did in terms of administering treatment only to assigned students, the whole class, or whatever, you're sunk.

It's always important to remember that in mixed effects models, predictor variables measured at the lowest level of the data hierarchy (students) have variation at the higher levels of the data hierarchy (classes and schools). You should investigate the degree to which this variation is meaningful in understanding the effect of treatment. This can be done in a couple of ways.

First, you can subject the treatment variable to it's own mixed effect model:

glmer(treatment ~ 1 + (1|class) + (1|school), family='binomial' ...)


You can examine the degree to which the variance in treatment sits at the higher levels of the data hierarchy.

Second, you can include potentially two more predictors in the model to help you recover the within-classroom effect separate from the between-classroom and between-school effects. Create one variable that is the proportion of students who received treatment in the classroom (cmn_trt) and another that is the proportion of students who received treatment in the school (smn_trt). Then model them:

glmer(pass ~ treatment + cmn_trt + smn_trt + (1 | school/class), family='binomial' ...)
`

Assuming you knew about the different ways treatment was administered in the classroom or could reconstruct it from knowing whether students received treatment, you could also include a series of 0/1 predictors for the classroom treatment configuration as a classroom-level predictor. But I think the class and school means of proportion of treated students is more useful.

• Thanks, Erik. This is very interesting re: investigating the selection effect itself with another model as well as the cmn_trt and smn_trt investigation. Can you provide any intuition on the bolded question re: how the fixed effects are estimated in an unbalanced design like this? If so, I'm happy to accept this answer as it will then answer the full question – Aaron Springer May 19 at 21:38