Feature selection when bagging trees/random forest

Question

I want to get a better understanding of feature selection and how the number of features affect performance when bagging trees. I am using Matlab's treebagger and I am working on classification into two classes (0 & 1) for Y

The probability of belonging to class 1 (prob1) is given by the second column of the 'scores' return value. When using cross-validated forward selection (add one feature at the time), is it best to minimize the classification error or to minimize abs((prob1) - Y)? It would seem that the latter makes more sense as we are using the mean response of all grown trees?

Another thing. It seems like the accuracy in the predictions does not decrease when using many features? I have added 40 features and keep seeing improvements on the held out data.

Finally, would you recommend any other feature-selection algorithm for bagged trees than cross-validated forward selection?

Answer 1

In general random forest are able to dial with an high number of variables. I personally think that it is not required a dimensionality reduction step unless you are not in a particular situation.

With the random forest function you can compute the importance of each variable and this could be a way to perform dimensionality reduction.

If you are interested only in classification I suggest you to do PCA on your data and to use all the variables after PCA transformation.