What are theoretical reasons to not handle missing values? Gradient boosting machines, regression trees handle missing values. Why doesn't Random Forest do that?
- Anybody can ask a question
- Anybody can answer
- The best answers are voted up and rise to the top
Gradient Boosting Trees uses CART trees (in a standard setup, as it was proposed by its authors). CART trees are also used in Random Forests. What @user777 said is true, that RF trees handles missing values either by imputation with average, either by rough average/mode, either by an averaging/mode based on proximities. These methods were proposed by Breiman and Cutler and are used for RF. This is a reference from authors Missing values in training set.
However, one can build a GBM or RF with other type of decision trees. The usual replacement for CART is C4.5 proposed by Quinlan. In C4.5 the missing values are not replaced on data set. Instead, the impurity function computed takes into account the missing values by penalizing the impurity score with the ration of missing values. On test set the evaluation in a node which has a test with missing value, the prediction is built for each child node and aggregated later (by weighting).
Now, in many implementations C4.5 is used instead of CART. The main reason is to avoid expensive computation (CART has more rigorous statistical approaches, which require more computation), the results seems to be similar, the resulted trees are often smaller (since CART is binary and C4.5 not). I know that Weka uses this approach. I do not know other libraries, but I expect it to not be a singular situation. If that is the case with your GBM implementation, than this would be an answer.
RF does handle missing values, just not in the same way that CART and other similar decision tree algorithms do. User777 correctly describes the two methods used by RF to handle missing data (median imputation and/or proximity based measure), whereas Frank Harrell correctly describes how missing values are handled in CART (surrogate splits). For more info, see links on missing data handling for CART (or it's FOSS cousin: RPART) and RF.
An answer to your actual question is covered clearly, IMHO, in Ishwaran et al's 2008 paper entitled Random Survival Forests. They provide the following plausible explanation for why RF does not handle missing data in the same way as CART or similar single decision tree classifiers:
This is an aside, but for me, this calls into question those who claim that RF uses an ensemble of CART models. I've seen this claim made in many articles, but I've never seen such statements sourced to any authoritative text on RF. For one, the trees in a RF are grown without pruning, which is usually not the standard approach when building a CART model. Another reason would be the one you allude to in your question: CART and other ensembles of decision trees handle missing values, whereas [the original] RF does not, at least not internally like CART does.
With those caveats in mind, I think you could say that RF uses an ensemble of CART-like decision trees (i.e., a bunch of unpruned trees, grown to their maximum extent, without the ability to handle missing data through surrogate splitting). Perhaps this is one of those punctilious semantic differences, but it's one I think worth noting.
EDIT: On my side note, which is unrelated to the actual question asked, I stated that "I've never seen such statements sourced to any authoritative text on RF". Turns out Breiman DID specifically state that CART decision trees are used in the original RF algorithm:
Source: p.9 of Random Forests. Breiman (2001)
However, I still stand (albeit more precariously) on the notion that these are CART-like decision trees in that they are grown without pruning, whereas a CART is normally never run in this configuration as it will almost certainly over-fit your data (hence the pruning in the first place).
Random Forest has two methods for handling missing values, according to Leo Breiman and Adele Cutler, who invented it.
The first is quick and dirty: it just fills in the median value for continuous variables, or the most common non-missing value by class.
The second method fills in missing values, then runs RF, then for missing continuous values, RF computes the proximity-weighted average of the missing values. Then this process is repeated several times. Then the model is trained a final time using the RF-imputed data set.
Recursive partitioning uses surrogate splits based on non-missing predictors that are correlated with the predictor possessing the missing value for an observation. It would seem possible in theory for random forests to be implemented that use the same idea. I don't know if any random forest software has done so.
Random forest does handle missing data and there are two distinct ways it does so:
1) Without imputation of missing data, but providing inference. 2) Imputing the data. Imputed data is then used for inference.
Both methods are implemented in my R-package randomForestSRC (co-written with Udaya Kogalur). First, it is important to remember that because random forests employs random feature selection, traditional missing data methods used by single trees (CART and the like) do not apply. This point was made in Ishwaran et al. (2008), "Random Survival Forests", Annals of Applied Statistics, 2, 3, and nicely articulated by one of the commenters.
Method (1) is an "on the fly imputation" (OTFI) method. Prior to splitting a node, missing data for a variable is imputed by randomly drawing values from non-missing in-bag data. The purpose of this imputed data is to make it possible to assign cases to daughter nodes in the event the node is split on a variable with missing data. Imputed data is however not used to calculate the split-statistic which uses non-missing data only. Following a node split, imputed data are reset to missing and the process is repeated until terminal nodes are reached. OTFI preserves the integrity of out-of-bag data and therefore performance values such as variable importance (VIMP) remain unbiased. The OTFI algorithm was described in Ishwaran et al (2008) and implemented in the retired randomSurvivalForest package, and has now been extended to randomForestSRC to apply to all families (i.e. not just survival).
Method (2) is implemented using the "impute" function in randomForestSRC. Unsupervised, randomized, and multivariate splitting methods for imputing data are available. For example, multivariate splitting generalizes the highly successful missForest imputation method (Stekhoven & Bühlmann (2012), "MissForest—non-parametric missing value imputation for mixed-type data", Bioinformatics, 28, 1). Calling the impute function with missing data will return an imputed data frame which can be fit using the primary forest function "rfsrc".
A detailed comparison of the different forest missing data algorithms implemented using "impute" was described in a recent paper with Fei Tang ("Random forest missing data algorithms", 2015). I recommend consulting the help files of "rfsrc" and "impute" from randomForestSRC for more details about imputation and OTFI.