32
$\begingroup$

I have a large set of feature vectors which I will use to attack a binary classification problem (using scikit learn in Python). Before I start to think about imputation, I am interested in trying to determine from the remaining parts of the data if the missing data are 'missing at random' or missing not at random.

What is a sensible way to approach this question?

It turns out a better question is to ask if the data is 'missing completely at random' or not. What is a sensible way to do that?

Improve this question
$\endgroup$
7
  • $\begingroup$ If there is no association between the missing indicator and any observed variable, then the missing data mechanism is MCAR. $\endgroup$
    – Randel
    Commented Sep 19, 2015 at 16:30
  • $\begingroup$ @Randel What is a good test to apply in practice to test this ? $\endgroup$
    – Simd
    Commented Sep 19, 2015 at 16:43
  • $\begingroup$ Nothing special than a correlation test or regression. $\endgroup$
    – Randel
    Commented Sep 19, 2015 at 17:53
  • 3
    $\begingroup$ It is not something you test, it is something you assume. $\endgroup$
    – Tim
    Commented Sep 25, 2015 at 15:20
  • 6
    $\begingroup$ Just to be clear: missing completely at random means that the missingness probability is a constant, it depends on nothing. Missing at random means that missingness depends on some measured factors, like age or sex, so that you can use some models to fill in the missing patterns. Missing not at random means missingness depends on things you did not measure. In the question OP says NMAR vs. MAR but OP means MAR vs MCAR. $\endgroup$
    – AdamO
    Commented Feb 9, 2018 at 22:18

6 Answers 6

Reset to default
14
+25
$\begingroup$

I found the information I was talking about in my comment.

From van Buurens book, page 31, he writes

"Several tests have been proposed to test MCAR versus MAR. These tests are not widely used, and their practical value is unclear. See Enders (2010, pp. 17–21) for an evaluation of two procedures. It is not possible to test MAR versus MNAR since the information that is needed for such a test is missing."

Improve this answer
$\endgroup$
3
  • $\begingroup$ The question asks about MAR vs MNAR, but your answer is about MCAR vs MAR. MCAR is completely different to MNAR. $\endgroup$
    – Tim
    Commented Feb 23, 2017 at 4:12
  • $\begingroup$ If you can determine the data is MAR then that should suffice. As Bjorn says, it is not possible to tell if it is MAR/MNAR, but this answer is a good proxy for his question I believe. If you were to do Enders test and find that it is MCAR, then you wouldn't need imputation. If you find that it is MAR, then you can impute, or take a hard look at your data to see if there is reason to believe it may be MNAR. $\endgroup$
    – RayVelcoro
    Commented Feb 23, 2017 at 23:08
  • 1
    $\begingroup$ @RayVelcoro It is an identifiability issue: it's possible for NMAR data to appear MCAR. Tim is right that NMAR (or the converse) is not something for which we test, it's something we assume. To your point about MCAR vs MAR, the more (most?) important thing is: if the data are MCAR and you use MAR methods, is there really any net effect on the data? I don't think so. Given the penetration, availability, and ease of use for MAR methods, maybe it's better to just use the non-parametric weighting or imputation procedure than engage in a rhetorical goose chase of tests and tests. $\endgroup$
    – AdamO
    Commented Feb 9, 2018 at 22:21
21
$\begingroup$

This is not possible, unless you managed to retrieve missing data. You cannot determine from the observed data whether the missing data is missing at random (MAR) or not at random (MNAR). You can only tell whether the data is clearly not missing completely at random (MCAR). Beyond that only appeal to plausibility of MCAR or MAR as opposed to MNAR based on what you know (e.g. reported reasons for why data is missing). Alternatively, you might be able to argue that it does not matter too much, because the proportion of missing data is small and under MNAR very extreme scenarios would have to happen for your results to be overturned (see "tipping point analysis").

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ Thank you very much. What's a good way to tell if the data is MCAR ? $\endgroup$
    – Simd
    Commented Sep 13, 2015 at 16:53
  • $\begingroup$ @Björn, above may need to be reworded to reflect essential asymmetries in what we can learn from data. Although it is possible to falsify a hypothesis that data are MCAR (viz., by building a model that exploits the observed covariates to account for some part of the missingness), it is not possible to confirm MCAR or any other such hypothesis. $\endgroup$
    – David C. Norris
    Commented Sep 24, 2015 at 19:53
  • $\begingroup$ Good point. I made that clearer. $\endgroup$
    – Björn
    Commented Sep 24, 2015 at 23:57
4
$\begingroup$

A method I use is a shadow matrix, in which the dataset consists of indicator variables where a 1 is given if a value is present, and 0 if it isn't. Correlating these with each other and the original data can help determine if variables tend to be missing together (MAR) or not (MCAR). Using R for an example (borrowing from the book "R in action" by Robert Kabacoff):

#Load dataset
data(sleep, package = "VIM")

x <- as.data.frame(abs(is.na(sleep)))

#Elements of x are 1 if a value in the sleep data is missing and 0 if non-missing.
head(sleep)
head(x)

#Extracting variables that have some missing values.
y <- x[which(sapply(x, sd) > 0)]
cor(y)

#We see that variables Dream and NonD tend to be missing together. To a lesser extent, this is also true with Sleep and NonD, as well as Sleep and Dream.

#Now, looking at the relationship between the presence of missing values in each variable and the observed values in other variables:
cor(sleep, y, use="pairwise.complete.obs")

#NonD is more likely to be missing as Exp, BodyWgt, and Gest increases, suggesting that the missingness for NonD is likely MAR rather than MCAR.
Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ In VIM, you can also check out spinoplots. They give a histogram of two variables with the missingness in each. We can plot two variables, and see how the missingness in one varies with the other. For example, if we we plot survival time and treatment assignment, if we see a right skewed distribution of missingness, we can posit that lower survival times are associated with more missingness...i.e. that missingness in treatment is MAR because it depends on the observed variable survival time. $\endgroup$
    – RayVelcoro
    Commented Nov 23, 2015 at 19:10
  • 1
    $\begingroup$ The question asks about MAR vs MNAR, but your answer is about MCAR vs MAR. MCAR is completely different to MNAR. $\endgroup$
    – Tim
    Commented Feb 23, 2017 at 4:13
  • $\begingroup$ @Tim As AdamO stated in a comment below the question, OP meant MAR vs MCAR. $\endgroup$
    – Phil
    Commented Aug 12, 2018 at 0:22
4
$\begingroup$

This sounds quite doable from a classification standpoint.

You want to classify missing versus non-missing data using all other features. If you get significantly better than random results, then your data aren't missing at random.

Improve this answer
$\endgroup$
0
2
$\begingroup$

You want to know whether there is some correlation of a value being missed in feature and the value of any other of the features.

For each of the features, create a new feature indicating whether the value is missing or not (let's call them "is_missing" feature). Compute your favourite correlation measure (I suggest using here mutual information) of the is_missing features and the rest of the features.

Note the if you don't find any correlation between two features, it is still possible to have a correlation due to group of features (a value is missing as a function of XOR of ten other features).

It you have a large set of features and a large number of values, you will get false correlations due to randomness. Other than the regular ways of coping with that (validation set, high enough threshold) You can check if the correlations are symmetric and transitive. If they are, it is likely that they are true and you should further check them.

Improve this answer
$\endgroup$
1
$\begingroup$

There is a useful package called finalfit check here it has a missing_pairs(outcome VAR, explanatory Vars) were you can explore patterns of missingness and decide whether data is MCAR or MAR. It produces pairs plots to show relationships between missing values and observed values in all variables.

Improve this answer
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.