1
$\begingroup$

When dealing with real world data, we often see some of the columns have missing values. I frequently see that people using the following way to deal with missing values.

Assume column Z has 50% of NaNs, then they just leave those NaNs there and add a new column denoted by Z_indicator which contains only 0 and 1 such that for the entry in column Z with NaN, then the corresponding entry in Z_indicator is 1 and if the entry in column Z is not NaN, then the corresponding entry in Z_indicator is 0.

I don't understand why this way of dealing with missing value is "better" than other ways? Is there any theoretically justification for that?

Improve this question
$\endgroup$
3
  • 3
    $\begingroup$ Schafer, J. L. (1999). Multiple imputation: a primer. Statistical Methods in Medical Research, 8:3–15. $\endgroup$
    – Alexis
    Commented Apr 15, 2017 at 4:46
  • $\begingroup$ @KevinKim You have to clarify how this data matrix is used. I presume the columns are your predictors that feed into a model to explain some response, correct? $\endgroup$
    – horaceT
    Commented Apr 15, 2017 at 19:18
  • $\begingroup$ @horaceT That's right. $\endgroup$
    – KevinKim
    Commented Apr 15, 2017 at 22:13

2 Answers 2

Reset to default
2
$\begingroup$

Simply creating this extra variable does not fix the missing data issue, but it is an important step in understanding the pattern and mechanism of missingness. The mechanism of missingness is particularly important because different missing data techniques (e.g., listwise deletion, multiple imputation, or maximum likelihood) assume different mechanisms and will be biased if those assumptions are violated. The indicator variable you are asking about can be used to conduct sensitivity analyses and to identify auxiliary variables to include in missing data techniques. For instance, the missing-completely-at-random (MCAR) mechanism states that the cases with missingness and without missingness were drawn from the same population and thus have the same means and variances on all relevant variables. You can test these assumptions by comparing the means and variances between cases that have missingness and those that don't. This is a broad topic area and I recommend you read more about the topic to understand it further. I recommend the book "Applied Missing Data Analysis" by Craig Enders.

Improve this answer
$\endgroup$
-3
$\begingroup$

you can replace the missing value with the mean,median or mode of the existing values. In the following example i chose to use mean.

data$variablename[is.na(data$variable)]<-mean(data$variablename,na.rm=T)

Improve this answer
$\endgroup$
3
  • 2
    $\begingroup$ That doesn't answer OP's question. He's asking why one would create an extra variable to flag the missingness. $\endgroup$
    – horaceT
    Commented Apr 15, 2017 at 5:02
  • $\begingroup$ This approach of single imputation is also likely to bias standard errors. $\endgroup$
    – Jeffrey Girard
    Commented Apr 15, 2017 at 16:26
  • $\begingroup$ Note that if you delete this answer, reputation lost due to downvotes will be restored. $\endgroup$
    – GeoMatt22
    Commented Apr 16, 2017 at 4:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.