If random forests gives me a bad cross-validation score, should I trust it for feature selection?

I get an R^2 value of about 0.22 when I 10-fold cross-validate with my entire dataset.

My main use for random forests is to analyze feature interactions. But should I trust the feature importances given, despite it's low R^2 value?

Thanks in advacne

• I'd recommend using a better measure of future prediction error rate, like mean squared prediction error when doing 10-fold CV. $R^2$ is not a good measure. Dec 1 '14 at 0:04

1 Answer

The feature importances that come out of RF are relative to the forest it built, regardless of the model performance. They are also relative to each other. It is quite possible that your data do not support a predictive model very well, but RF will still compute their relative effects and give importances. I would first try tweaking the RF parameters (minimum per node, minimum to split, maximum depth, etc.) to get a good model. You can also remove the less important variables to see if RF performance improves.

You should also run the data through another method, perhaps SVM, and see if you get good prediction. If another model can give you decent prediction, then go back to your RF importances, cut out the low end, and rebuild your SVM model.

In general, you always want to validate model output by another method if you can.

You mention "feature interactions", but feature importances won't give you that. You need to create dummy features that represent any interactions you want to look at and see if RF evaluates them with high importance.

• I have a follow-up question about "can also remove the less important variables" - I thought that with a large enough ntree compared to the number of predictors that should not matter as much? Is this not true? Nov 30 '14 at 22:44
• If variables do not hold much predictive value, then the splits are nearly random. This dilutes the overall consensus. In general, though, you can build things so that dilution doesn't affect the overall prediction much when the other variables are present, as you suggest. However, if your data contains mostly those variables, you'll see poor performance. Also see my edit addressing feature interaction. Nov 30 '14 at 23:08
• +1 thank you! good pt on dummy features for interactions, too. Nov 30 '14 at 23:37