1
$\begingroup$

There is lots of literature regarding to the treatment of Missing Data like Multiple Imputation and so on. And I know there is help from graphics but graphics are not as explicit as a test.
But I haven’t yet found any concrete strategy on how to diagnose the Missing Data Mechanism apart from Little’s Test that is more for global assessment of the Missing Data Mechanism of a whole data set and apart from t-tests. Here my example:

X … pretestscore

R … missingness indicator variable for X, where R = 0 is observed and R = 1 is missing

Y … actual testscore (numeric variable)

I have a predictor variable X and its missingness indicator variable R. (And I have other categorial covariates Z that can also have missing values).
First the taxonomy:

1) MCAR
P(R = 1|Xobs, Xmiss, Z) = P(R = 1)
Means: the probability of missing values in X (R=1) does neither depend on the observed values of X or another covariates nor on the missing values itself.

2) MAR
P(R = 1|Xobs, Xmiss, Z) = P(R = 1|Xobs, Z)

3) NMAR
P(R = 1|Xobs, Xmiss, Z)! = P(R = 1|Xobs, Z)

Here is my try to find the Mechanism:

  1. I did a t-test on R and Yobs -> there is a significant Bonferroni-corrected result and a Cohen’s d from about 0.27.

  2. To check I performed a simple linear regression E(Yobs|R): the coefficients are significant, but the R² is near 0 (0.02). I conclude that a linear regression is not a fitting model for my data points.

  3. I performed a logistic regression (R|Yobs) and AIC is again miserable from about 14000 or so.

  4. The numerical prestest variable X will actually be categorized in the end to predict testscore Y. So why not the other way round. I orderd all values of Y and categorized them into quartiles and performed a Chi²-test. The significant result supports the stochastic-dependance-hypothesis between R and Y.

My first issue: the t-test shows a significant difference in the means but the linear regression model's determination coefficient R² says the model doesn't fit. Can I then because of a significant t-test mathematical-logically conclude a stochastic dependence between R and Yobs? (Well, I think no!)
If so, the Mechanism would be at least MAR, if not NMAR, because pretest and actual test scores are correlated?
My second issue: I know one cannot infer stochastic independence from regessional independence. But in this case I’m wondering because it's a binary independant variable. So E(R|Y) = P(R = 1|Y) (the regression is the probability). As the AIC says there is no regressional depencance I conclude that there is no stochastical dependance as well. Is my thinking correct?
Now on the contrary my result from the chi²-test shows stochastic dependance between R and Y.
I'm confused. Is there nowhere a step-by-step-recipe on how to diagnose the Missing Data Mechanism. How do you do it?
What if there is a dependance between R and Y if I control for another variable, say type of school. I can hardly control for all covariates (gender, subject, retention, mother tongue...)
Thanks for any hints.

$\endgroup$
1
$\begingroup$

I think you may be barking up the wrong tree here.

First of all, you are right in thinking that you can test if the missing data is MCAR vs MAR; basically, are there any significant variables associated with detecting the missingness?

However, in certain ways, this is of little concern! If your data is MCAR, you can drop your subjects with missing data and still have unbiased estimation, whereas if the missingness mechanism is MAR, dropping your subject with missing data can lead to biased estimation. But even in the case of MCAR, dropping the subjects with missing data will still provided for less efficient estimation (as you are not using the partial information from the subjects with missing data). So there's really no reason not to build a model to account for the missingness.

And what you should be really concerned about it NMAR, of course. By definition, there is no way to test this from the observed data. It must considered through logical reasoning rather than statistical inference.

Also, just a small note: saying something like "The AIC was 14,000" gives no information to a reader, outside that you probably have a reasonably large dataset (and even that's not certain; maybe you just have a huge variance). AIC can be used to compare two models (i.e. "model 1 had AIC of 14,000, while model 2 had AIC 13,500" is a useful statement).

$\endgroup$
  • $\begingroup$ (I'm not familiar with logistic regression models. So thanks for small note.) $\endgroup$ – Frikko Sep 7 '15 at 6:33
  • $\begingroup$ "So there's really no reason not to build a model to account for the missingness." !!! Thanks a lot. Still, I know one can just discuss plausibility of NMAR. But I could show at least a MAR mechanism for pretest variable X if I was able to draw the right conclusions out of my tests. So maybe I just report the significant t-tests as hints for a stochastic dependance. $\endgroup$ – Frikko Sep 7 '15 at 6:46
  • $\begingroup$ What's your sample size? Also, are the Y values heavily skewed? $\endgroup$ – Cliff AB Sep 7 '15 at 15:08
  • $\begingroup$ My sample size for Y in group R = 0 is 10 814 and for R = 1 it's 3 604. The distribution is heavily left skewed: for group R = 0 -0.385 and for R = 1 -0.265. Anderson-Darling-Test is highly significant, but I assumed normal distribution because of the amount of data. $\endgroup$ – Frikko Sep 8 '15 at 10:43
  • $\begingroup$ I would be inclined to used the t-test. Using simple linear regression on a binary outcome is almost always a poor fit. Using logistic regression is a better choice. However, having a couple of extreme outliers that buck the trend can be very problematic. Given your sample size, I would say just using a t-test is the most robust way to go. $\endgroup$ – Cliff AB Sep 8 '15 at 15:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.