Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

What are theoretical reasons to not handle missing values? Gradient boosting machines, regression trees handle missing values. Why doesn't Random Forest do that?

share|improve this question

3 Answers 3

up vote 3 down vote accepted

Gradient Boosting Trees uses CART trees (in a standard setup, as it was proposed by it's authors). CART trees are also used in Random Forests. What @user777 said is true, that CART trees handles missing values either by imputation with average, either by rough average/mode, either by an averaging/mode based on proximities. These methods were proposed by Breiman and Cutler and are used for CART.

However, one can build a GBM or RF with other type of decision trees. The usual replacement for CART is C4.5 proposed by Quinlan. In C4.5 the missing values are not replaced on data set. Instead, the impurity function computed takes into account the missing values by penalizing the impurity score with the ration of missing values. On test set the evaluation in a node which has a test with missing value, the prediction is build for each child node and aggregated later (by weighting).

Now, in many implementations C4.5 is used instead of CART. The main reason is to avoid expensive computation (CART has more rigorous statistical approaches, which require more computation), the results seems to be similar, the resulted trees are often smaller (since CART is binary and C4.5 not). I know that Weka uses this approach. I do not know other libraries, but I expect to not be a singular situation. If that is the case with your GBM implementation, than this would be an answer.

share|improve this answer

Random Forest has two methods for handling missing values, according to Leo Breiman and Adele Cutler, who invented it.

The first is quick and dirty: it just fills in the median value for continuous variables, or the most common non-missing value by class.

The second method fills in missing values, then runs RF, then for missing continuous values, RF computes the proximity-weighted average of the missing values. Then this process is repeated several times. Then the model is trained a final time using the RF-imputed data set.

share|improve this answer
    
Thank you for your answer! But, both this methods are replacing missing values. But in GBM or regression trees missing values don't replace for anything. What is theoretical difference between, for example GBM and RF in this sense? –  Kristina May 16 at 13:22
    
I'm not an expert on GBM, but the RF handling of missing values appears to be rooted in the idea of imputation, en.wikipedia.org/wiki/Imputation_(statistics) In cases where missing values are not missing at random, your results can be biased due to missingness. Imputation attempts to recover this missing values and reduce bias. –  user777 May 16 at 14:09

Recursive partitioning uses surrogate splits based on non-missing predictors that are correlated with the predictor possessing the missing value for an observation. It would seem possible in theory for random forests to be implemented that use the same idea. I don't know if any random forest software has done so.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.