I'm working with a dataset of approximately 150,000 observations and 50 features, using SVM for the final model. To trim down the feature count, I decided to look into using RF so SVM optimization doesn't take too long. I'm currently using the TreeBagger implementation in Matlab and had a few questions.
- When investigating feature importances, should the RF be tuned for the highest CV performance? Does the accuracy of the model play into the accuracy of the reported predictor importances?
- What is the best way to deal with one of two correlated feature having it's importance underreported? Can this be cancelled out by training the RF multiple times and averaging the feature rankings?
- There doesn't seem to be any way to manually select split criterion in TreeBagger, nor could I find any documentation on what the default is. Does anyone know? If not, would it be safe to assume it's using Gini?
- How do the feature importances from TreeBagger compare to those generated by Matlab's fitensemble? This has support for bagging and different boosting algorithms, as well as different split criteria. But, as far as I know, these don't invoke Breiman's RF algorithm. The only Matlab function which does is TreeBagger, when specifying a number of features to sample. Please correct me if I'm wrong. As it stands, fitensemble is looking more attractive due to more options and better documentation.