There is lots of literature regarding to the treatment of Missing Data like Multiple Imputation and so on. And I know there is help from graphics but graphics are not as explicit as a test.
But I haven’t yet found any concrete strategy on how to diagnose the Missing Data Mechanism apart from Little’s Test that is more for global assessment of the Missing Data Mechanism of a whole data set and apart from t-tests. Here my example:
X … pretestscore
R … missingness indicator variable for X, where R = 0 is observed and R = 1 is missing
Y … actual testscore (numeric variable)
I have a predictor variable X and its missingness indicator variable R. (And I have other categorial covariates Z that can also have missing values).
First the taxonomy:
P(R = 1|Xobs, Xmiss, Z) = P(R = 1)
Means: the probability of missing values in X (R=1) does neither depend on the observed values of X or another covariates nor on the missing values itself.
P(R = 1|Xobs, Xmiss, Z) = P(R = 1|Xobs, Z)
P(R = 1|Xobs, Xmiss, Z)! = P(R = 1|Xobs, Z)
Here is my try to find the Mechanism:
I did a t-test on R and Yobs -> there is a significant Bonferroni-corrected result and a Cohen’s d from about 0.27.
To check I performed a simple linear regression E(Yobs|R): the coefficients are significant, but the R² is near 0 (0.02). I conclude that a linear regression is not a fitting model for my data points.
I performed a logistic regression (R|Yobs) and AIC is again miserable from about 14000 or so.
The numerical prestest variable X will actually be categorized in the end to predict testscore Y. So why not the other way round. I orderd all values of Y and categorized them into quartiles and performed a Chi²-test. The significant result supports the stochastic-dependance-hypothesis between R and Y.
My first issue: the t-test shows a significant difference in the means but the linear regression model's determination coefficient R² says the model doesn't fit. Can I then because of a significant t-test mathematical-logically conclude a stochastic dependence between R and Yobs? (Well, I think no!)
If so, the Mechanism would be at least MAR, if not NMAR, because pretest and actual test scores are correlated?
My second issue: I know one cannot infer stochastic independence from regessional independence. But in this case I’m wondering because it's a binary independant variable. So E(R|Y) = P(R = 1|Y) (the regression is the probability). As the AIC says there is no regressional depencance I conclude that there is no stochastical dependance as well. Is my thinking correct?
Now on the contrary my result from the chi²-test shows stochastic dependance between R and Y.
I'm confused. Is there nowhere a step-by-step-recipe on how to diagnose the Missing Data Mechanism. How do you do it?
What if there is a dependance between R and Y if I control for another variable, say type of school. I can hardly control for all covariates (gender, subject, retention, mother tongue...)
Thanks for any hints.