Gurus, I just came across this tutorial (http://blog.datadive.net/selecting-good-features-part-iii-random-forests/) about using "random forests" for optimal variable selection/feature selection. The first method is to use" mean decrease impurity" to measure the importance of each feature. The code is here.
from sklearn.datasets import load_boston from sklearn.ensemble import RandomForestRegressor import numpy as np #Load boston housing dataset as an example boston = load_boston() X = boston["data"] # All other variables besides the boston housing price Y = boston["target"] # The boston housing price column names = boston["feature_names"] # The column name of all variables rf = RandomForestRegressor() rf.fit(X, Y) print "Features sorted by their score:" print sorted(zip(map(lambda x: round(x, 4), rf.feature_importances_), names), reverse=True)
In the following result, the sum of feature scores is 0. However, I am still not sure how to choose optimal number and best features/variables from the outcome. For example:
Features sorted by their score: (scores, 'Column/variable name') [(0.5298, 'LSTAT'), (0.4116, 'RM'), (0.0252, 'DIS'), (0.0172, 'CRIM'), (0.0065, 'NOX'), (0.0035, 'PTRATIO'), (0.0021, 'TAX'), (0.0017, 'AGE'), (0.0012, 'B'), (0.0008, 'INDUS'), (0.0004, 'RAD'), (0.0001, 'CHAS'), (0.0, 'ZN')]
In the above case, it is obvious that "LSTAT" and "RM" are the most important ones. However, the important features are not as obvious in the following case IMHO
Features sorted by their score: (scores, 'Column/variable name') [(0.1461, 'revol_bal'), (0.1386, 'dti'), (0.124, 'annual_inc'), (0.099, 'desc_len'), (0.0963, 'total_acc'), (0.0953, 'funded_amnt_inv'), (0.0745, 'open_acc'), (0.0513, 'funded_amnt'), (0.0508, 'loan_amnt'), (0.0377, 'term'), (0.0283, 'inq_last_6mths'), (0.0167, 'home_ownership'), (0.0159, 'delinq_2yrs'), (0.0144, 'verification_status'), (0.011, 'pub_rec')]
May I know how to read the outcome in the second case?
In PCA, we choose the smallest number of PCs, so, for example, 95% of variance is retained. Is there any similar optimal number of features selection method in random forest? Thank you!