using random forest for missing data imputation in categorical variables ( in R)

Question

I have following type of associated data. The following example step to generate associated variable. p number of variables and n is number of observations.

p = 500
n = 200
mat <- matrix(NA, ncol = 500, nrow = 200)

for (i in 1:p){
if(i ==1){
     fs <- sample (c("AA", "AB", "AB", "BB"), n, replace=TRUE) 
     mat[,i] <- fs 
     fs1 <- fs 
     } 
rechr <- sample(1:n, 1)
fs1[rechr] <- sample (c("AA", "AB", "AB", "BB"), 1)
mat[,i] <- fs1 
}

dim(mat)
mat[1:10,1:20]

The above data is complete, but in real dataset I have missing values (which are randomly distributed within each variables).

I would like to predict them using random forest (or any other appropriate algorithms). Let's randomly put 10% missing values (the approximate number of variables) to the above data.

rowind <- sample(1:n, 20)
colind <- sample(1:p, 20)

for (i in 1:length(rowind)){
                mat[rowind[i],colind[i]] <- NA
}
mat[1:20,1:10]

How can do this and what issues I need to think up in doing so - considering categorical variables, number of observations and variables ?

Edit: As all variables are correlated and none are response (but can serve as response) while prediction. for example mat[,1] missing value prediction can be based on rest of variables, mat[,2] prediction can be based on prediction of rest of variables. So the objective here is prediction all missing values in all variables (all of them are categorical).

Edit 2

I am interested in multiple imputation of this data so that I can perform other analysis that requires complete dataset without missing value.

Answer 1

In looks like you are interested in multiple imputations. See this link on ways you can impute / handle categorical data. The link discuss on details and how to do this in SAS.

The R package mice can handle categorical data for univariate cases using logistic regression and discriminant function analysis (see the link). If you use SAS proc mi is way to go [see link].

Edit:

You can use the function rfunsuper used in my answer for the another question.

p = 500
n = 200
mat <- matrix(NA, ncol = 500, nrow = 200)

for (i in 1:p){
if(i ==1){
     fs <- sample (c("AA", "AB", "AB", "BB"), n, replace=TRUE) 
     mat[,i] <- fs 
     fs1 <- fs 
     } 
rechr <- sample(1:n, 1)
fs1[rechr] <- sample (c("AA", "AB", "AB", "BB"), 1)
mat[,i] <- fs1 
}

dim(mat)

rowind <- sample(1:n, 20)
colind <- sample(1:p, 20)

for (i in 1:length(rowind)){
                mat[rowind[i],colind[i]] <- NA
}
mat[1:20,1:10]

out1 <- rfunsuper (data.frame(mat), iter=5, ntree=100)
diff.rel = 1.555556 / 50587.33 = 3.07499e-05 
diff.rel = 0.1111111 / 50590.67 = 2.196277e-06 

Call:
 randomForest(x = x.roughfixed, ntree = ntree) 
               Type of random forest: unsupervised
                     Number of trees: 100
No. of variables tried at each split: 22

Answer 2

dfife's answer is good though it may be informative to re-read Breiman's description of how random forests can handle missing values in the training data while trying to predict a variable without missing values: http://www.stat.berkeley.edu/~breiman/RandomForests/cc_home.htm#missing1

Another approach I've had good luck with to predict or get importance scores from data sets with missing values with out imputing is three way splitting which splits the missing value onto a third branch at each node. This is particularly useful if the fact something is NA may actually be informative (ie in a survey where a question being left blank may be significant).

This method originates (to my knowledge) in Timo Erkkila's rf-ace: https://code.google.com/p/rf-ace/ and can also be found in my implementation, CloudForest: https://github.com/ryanbressler/CloudForest.

rf-ace has a (difficult to build) r-package, cloudforest doesn't yet but is faster and more memory efficient and can also handle missing values via a bias correction without three way splitting or imputation as in some variants of CART.

If you have missing values in the thing you are trying to predict you may want to look into methods for semi supervised learning. If you want an unsupervised method for imputing missing values while you aren't trying to predict thing you could start with nearest neighbor imputation or impute to the variable mean or mode (as Breiman suggests to initialize his method for RF imputation).

Answer 3

There's two ways of doing it that depends on what you're trying to do. You can impute the missing values using the proximity matrix (the rfImpute function in the randomForest package).

If you're only interested in computing variable importance, you can use the cforest function in the party package then compute variable importance via the varimp() function. This approach is based on this article, which suggests a very clever way of bypassing the missing data (and handling it in an elegant way) in order to compute variable importance.