I want to get a better understanding of feature selection and how the number of features affect performance when bagging trees. I am using Matlab's
treebagger and I am working on classification into two classes (0 & 1) for Y
The probability of belonging to class 1 (prob1) is given by the second column of the 'scores' return value. When using cross-validated forward selection (add one feature at the time), is it best to minimize the classification error or to minimize abs((prob1) - Y)? It would seem that the latter makes more sense as we are using the mean response of all grown trees?
Another thing. It seems like the accuracy in the predictions does not decrease when using many features? I have added 40 features and keep seeing improvements on the held out data.
Finally, would you recommend any other feature-selection algorithm for bagged trees than cross-validated forward selection?