random forests for optimal variable selection/feature selection

Question

Gurus, I just came across this tutorial (http://blog.datadive.net/selecting-good-features-part-iii-random-forests/) about using "random forests" for optimal variable selection/feature selection. The first method is to use" mean decrease impurity" to measure the importance of each feature. The code is here.

from sklearn.datasets import load_boston
from sklearn.ensemble import RandomForestRegressor
import numpy as np
#Load boston housing dataset as an example
boston = load_boston()
X = boston["data"]   # All other variables besides the boston housing price
Y = boston["target"] # The boston housing price column
names = boston["feature_names"]  # The column name of all variables
rf = RandomForestRegressor()
rf.fit(X, Y)
print "Features sorted by their score:"
print sorted(zip(map(lambda x: round(x, 4), rf.feature_importances_), names), reverse=True)

In the following result, the sum of feature scores is 0. However, I am still not sure how to choose optimal number and best features/variables from the outcome. For example:

Features sorted by their score: (scores, 'Column/variable name')
[(0.5298, 'LSTAT'), (0.4116, 'RM'), (0.0252, 'DIS'), (0.0172, 'CRIM'), (0.0065, 'NOX'),
 (0.0035, 'PTRATIO'), (0.0021, 'TAX'), (0.0017, 'AGE'), (0.0012, 'B'), (0.0008, 'INDUS'), (0.0004, 'RAD'),
 (0.0001, 'CHAS'), (0.0, 'ZN')]

In the above case, it is obvious that "LSTAT" and "RM" are the most important ones. However, the important features are not as obvious in the following case IMHO

Features sorted by their score: (scores, 'Column/variable name')
[(0.1461, 'revol_bal'), (0.1386, 'dti'), (0.124, 'annual_inc'), (0.099,     'desc_len'), 
(0.0963, 'total_acc'), (0.0953, 'funded_amnt_inv'), (0.0745,     'open_acc'), (0.0513, 'funded_amnt'), 
(0.0508, 'loan_amnt'), (0.0377, 'term'), (0.0283, 'inq_last_6mths'), (0.0167, 'home_ownership'), 
(0.0159, 'delinq_2yrs'), (0.0144, 'verification_status'), (0.011, 'pub_rec')]

May I know how to read the outcome in the second case?

In PCA, we choose the smallest number of PCs, so, for example, 95% of variance is retained. Is there any similar optimal number of features selection method in random forest? Thank you!

Answer 1

There is no standard variable selection method for random forests (RF). The absolute variable importance values have no meaning, but their relative sizes can be useful to comparing different predictors. Deciding how many variables to include can be a little subjective, so many authors have proposed several variable selection algorithms. A few articles are given below:

For microarray data, Diaz-Uriarte and Alvarez de Andres [1] suggest reiteratively fitting RFs discarding 20% of variables with the smallest variable importance and choosing the variables that give the smallest out-of-bag (OOB) error rate. Genuer et al. [2] recommend a preliminary elimination that removes variables whose importance is below the minimum prediction value given by a CART model. After a preliminary elimination, a nested collection of RF models or a sequence of RF models is used to select the variables (see paper). Ishwaran et al. [3] propose a new metric called minimal depth which can select variables since the exact distribution is known. The three papers aforementioned have R packages called varSelRF, VSURF, and randomForestSRC, respectively. These articles are a small subset of the literature addressing variable selection using RFs.

As a side note, I believe the blog does not use the standard approach to calculate permuted variable importance. I do not know Python that well, but it seems the code permutes each variable in the training sample and compares the permuted and non-permuted prediction error from the random forest. The standard approach is to permute the variables in the OOB sample and compare the permuted and non-permuted prediction error in each tree. The final permuted variable importance is the average difference in prediction error. I personally would suggest using R as there are more tools already available for variable selection using RFs

[1] R. Diaz-Uriarte and S. Alvarez de Andres (2006) Gene selection and classification of microarray data using random forest. BMC Bioinformatics

[2] R. Geneur, J.-M. Poggi, C. Tuleau-Malot (2010) Variable selection using Random Forests. Pattern Recognition Letters

[3] H. Ishwaran, U.B. Kogalur, E.Z. Gorodeski, A.J. Minn, M.S. Lauer (2010) High-dimensional variable selection for survival data. Journal of the American Statistical Association

Answer 2

If you read further down in blog you find Mean decrease accuracy a.k.a. permutation importance or simply variable importance(when not specified further). Gini is unstable and should not be used for regular variable selection.

Below is one suggestion on how to perform variable selection with variable importance:

• Outer CV loop
• • pre RF training
• • get variable importance (computed on OOB samples, scaled/unscaled)
• • rank variables
• • drop e.g. 70% of lowest variables
• • Retrain on the remaining variables
• • predict outer test fold
• Compare outer.preds vs outer.true and compute a estimated predictive performance