Missing data and communicating bias

Question

I'm a graduate student, so I'm looking for a little more expertise around including family income in a regression model because of a high percentage of missing data (50%). I'm hoping you all can correct my thinking about multiple imputation, practical limitations in the use of variables with a high percentage of missing data, and provide some thoughtful ways to proceed.

Context:

I was brought into a project in its later phases. The project is exploring the relation between teacher training and student achievement. The sample size of models without family income is 20 thousand, while the sample size with including income is around 10K. So, this is not an issue of loosing power because of including a variable with a lot of missingness; rather, of understanding how estimates are biased, and effectively communicating that result.

Thoughts on missing data for income variable:

Income is often one of those variables that contains a high percentage of missing data. I've taken courses on multiple imputation, but I feel like this particular situation falls outside of what I am used to. In the past I've worked on projects where we use multiple imputation in datasets that contain income because we had variables in the survey that helped us understand that the data missing for income was likely MAR. Maybe it is my still nascent understanding of MI, but I've always wondered how can MI deal with the lower bounds of income that were not reported by individuals? Obviously, we don't know what the lower bound of income would be because we'd have to follow-up with individuals that did not report income. On other projects I've worked with methodologists have successfully argued income was MAR; however, I've still always wondered: Aren't the results still biased upwards to those who report income, and are likely not to be reflective of lower income families?

With this particular dataset the demographic data is sparse, income being one of the only student-level covariates, and in probing the data I can't find evidence of MAR. Theoretically, I feel like income should be in the model. There is a vast literature showing the relation between family income and student achievement. However, literature shows that people with low incomes are less likely to report it on surveys. So, while the only way to prove NMAR would be to go back to collect data from people, I feel there's a compelling enough rational to assume it is NMAR. MI would not apply, the same would go for FIML approaches to estimation of parameters. This would also make the listwise deletion approach invalid. I feel conflicted with how to proceed. Theoretically, it makes sense to keep income in the models and just accept the bias. Is it okay to do that, as long as that bias is communicated in the results? It would seem not including income, when we know from the literature that it matters, would also introduce some bias in the estimates. Is this a "pick your poison" situation?

Your thoughts and guidance is appreciated.

Answer 1

The trick to the problem is to think big with causal and conceptual models. A causal model is, of course, interested in the measurable (and unmeasurable) things in an analysis and their relationship with an outcome of interest. A conceptual model builds on this slightly to incorporate things that might predict other factors in the analysis like your exposure of interest.

With missing data analysis and causality, these models are expanded to include the missingness variable, which in a sense "predicts" the missing measure in question (using principles from weighting and standardization).

Multiple imputation does not correct bias. It improves efficiency. When people say something like, "50% of X is missing" it is both the best situation to apply MI when the assumptions are met, and the worst time to apply MI when they are not. You are saying that 50% missingness is a problem because you're losing those data, not because they're "unlike" anyone else. Complete case analysis and MI provide unbiased estimates when you have NMAR and MAR. Basically adjustment for the right thing creates strata that interpolate what would have been answered by those who didn't answer, provided the modeling assumptions are correct.