1
$\begingroup$

When dealing with real world data, we often see some of the columns have missing values. I frequently see that people using the following way to deal with missing values.

Assume column Z has 50% of NaNs, then they just leave those NaNs there and add a new column denoted by Z_indicator which contains only 0 and 1 such that for the entry in column Z with NaN, then the corresponding entry in Z_indicator is 1 and if the entry in column Z is not NaN, then the corresponding entry in Z_indicator is 0.

I don't understand why this way of dealing with missing value is "better" than other ways? Is there any theoretically justification for that?

$\endgroup$
  • 3
    $\begingroup$ Schafer, J. L. (1999). Multiple imputation: a primer. Statistical Methods in Medical Research, 8:3–15. $\endgroup$ – Alexis Apr 15 '17 at 4:46
  • $\begingroup$ @KevinKim You have to clarify how this data matrix is used. I presume the columns are your predictors that feed into a model to explain some response, correct? $\endgroup$ – horaceT Apr 15 '17 at 19:18
  • $\begingroup$ @horaceT That's right. $\endgroup$ – KevinKim Apr 15 '17 at 22:13
1
$\begingroup$

Simply creating this extra variable does not fix the missing data issue, but it is an important step in understanding the pattern and mechanism of missingness. The mechanism of missingness is particularly important because different missing data techniques (e.g., listwise deletion, multiple imputation, or maximum likelihood) assume different mechanisms and will be biased if those assumptions are violated. The indicator variable you are asking about can be used to conduct sensitivity analyses and to identify auxiliary variables to include in missing data techniques. For instance, the missing-completely-at-random (MCAR) mechanism states that the cases with missingness and without missingness were drawn from the same population and thus have the same means and variances on all relevant variables. You can test these assumptions by comparing the means and variances between cases that have missingness and those that don't. This is a broad topic area and I recommend you read more about the topic to understand it further. I recommend the book "Applied Missing Data Analysis" by Craig Enders.

$\endgroup$
-3
$\begingroup$

you can replace the missing value with the mean,median or mode of the existing values. In the following example i chose to use mean.

data$variablename[is.na(data$variable)]<-mean(data$variablename,na.rm=T)

$\endgroup$
  • 2
    $\begingroup$ That doesn't answer OP's question. He's asking why one would create an extra variable to flag the missingness. $\endgroup$ – horaceT Apr 15 '17 at 5:02
  • $\begingroup$ This approach of single imputation is also likely to bias standard errors. $\endgroup$ – Jeffrey Girard Apr 15 '17 at 16:26
  • $\begingroup$ Note that if you delete this answer, reputation lost due to downvotes will be restored. $\endgroup$ – GeoMatt22 Apr 16 '17 at 4:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.