Hierarchical modeling when data not missing at random

Question

Gelman & Hill (2006) explain how a multilevel regression model can compensate for data sparsity by pooling parameter estimates. But what happens if the data are not sparse at random?

For example, suppose we are evaluating a new blood pressure medication. On every visit we measure a patient's blood pressure, administer the treatment, and measure blood pressure again. We may have fewer opportunities to observe the effect of a treatment in some patients than in others, so we may want to use pooling. But the number of observations per patient could be related to the fact that the treatment is ineffective for some patients (so they do not want to come in), or causes side effects two hours later (so they do not want to come in), or the medication puts patients in a coma and they drop out of the study.

Is pooling no longer appropriate in such a study?

Answer 1

As far as I can see pooling is no more (or less) of a problem when you have a non-ignorable missing data mechanism. NMAR data potentially spoils any kind of model, whether pooled or not. In the following I'll assume below that 'pooling' means a hierarchical/mixed/multi-level model.

As @Yevgeny suggests, adding covariates is sometimes helpful, not so much to avoid the NMAR effects but rather to make the missingness MAR instead. That is, finding something to measure something that predicts missingness, presumably something related to the effectiveness of the treatment or the probability of not turning up due to side-effects. There are two cases to consider:

In the first case, only the response / dependent variable is sometimes missing. Then if variables that predict its missingness are in the pooling model then there is nothing more to do to avoid bias. Just run the model.

In the second case there are also missing covariates. The only unbiased solution here is to impute missing data first, run a pooling analysis on each imputation, then combine the results.

There are, unsurprisingly, some delicate aspects to this 'impute then pool' suggestion. A introductory general discussion (ch 25 of Gelman and Hill for other readers) is here with the tricky bits alluded to on p.541. Basically, bias from having an imputation model that doesn't know about the data structure.

Answer 2

Pooling can be still useful since it reduces variance but in this case it can also introduce bias (due to the endogeneity you've mentioned). One way to get around that is to include the number of visits/observations as one of the covariates in your model (so pooling will "shrink" the model parameters toward the sample mean while controlling for the effect you've mentioned)