I have downloaded the sample codes from kaggle for the randomForest benchmark [URL?] and there's this part that I don't understand.

  appendNAs <- function(dataset, cols) {
  append_these = data.frame( is.na(dataset[, cols] ))
  names(append_these) = paste(names(append_these), "NA", sep = "_")
  dataset = cbind(dataset, append_these)
  dataset[is.na(dataset)] = -1

I understand that randomForest does not accept NAs and I usually just impute them with median or rfImpute. Adding extra columns as NAs indicator is new to me. I have looked at the help file for randomForest but didn't anything useful.

Would be great if someone can explain how and why the extra columns work.


You somehow have to model the the missingness (NAs). You can either model the variables that contain NAs in order to impute values, or you can include NA indicator values that become a part of your overall model.

When you think about it that way, simply choosing the median of a variable to fill in its NAs is probably not a great model. It's convenient, since it does not involve any of the other variables and it's easy to calculate, but is there any real justification that the particular NA values really would fall close to the median?

Using indicator variables lets you attempt to model the missingness in terms of the other variables.

  • $\begingroup$ Will the choice of '-1' affect the prediction? or can I basically put any unrealistic number of my choice? e.g. -999 $\endgroup$ – King May 21 '12 at 13:44
  • $\begingroup$ I've heard of replacing NA's with 0 and adding the indicator variable. I believe you want something that's actually near-reasonable, but I don't really know what options are best. Perhaps in the case you mention, they are using 1 for TRUE and -1 for FALSE or something like that. $\endgroup$ – Wayne May 21 '12 at 16:32
  • 2
    $\begingroup$ I rarely work outside the regression context but I can't imagine why you would replace NAs with 0 instead of mean/median. If want to do better, consider multiple imputation with cross-validation. And always ask yourself why data is missing. $\endgroup$ – Michael Bishop May 21 '12 at 17:26
  • $\begingroup$ @MichaelBishop: +1 for the "always ask yourself" tip. I thought I'd read replacing with zero in a couple of examples, but can't google up anything, so I guess it was very case-specific. $\endgroup$ – Wayne May 21 '12 at 19:01
  • $\begingroup$ @MichaelBishop: All the data given are just Quan_#, Cat_#; so there's no way for me to find out why they are missing! $\endgroup$ – King May 22 '12 at 8:20

In many cases, the presence of NA is non-random, and can be used as a predictive variable in its own right. For example, knowing whether survey participants chose to disclose a certain fact can be as important as the fact itself. Imputation discards this information, which can reduce predictive performance.

  • 1
    $\begingroup$ I like your answer too, but Wayne's similar explanation was 2 minutes faster, so I ticked his. Thank you for helping me understand this step!! $\endgroup$ – King May 21 '12 at 12:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.