I am trying to model a binary response using a 500+ dataset. I already removed many non useful features in order to reduce dimensionality and improve my model.

I am wondering whether in general removing features with correlation of around 0.5, 0.6 could improve the model performance when predicting new values. In particular I would like to know if in better predictive perfomarce can be achieved following this approach with Random Forests and SVM.

If the case above is not true, which models are more sensitive to correlated features?



There could both be situations where correlation pruning would improve - and deteriorate prediction performance. RF and SVM are both not sensitive to collinearity, but to small data-sets of few hundreds of samples it could show to be an advantage to perform some kind feature selection. Remember feature selection (grid-search also) will make inner prediction performance e.g. OOB over-optimistic. Therefore, the entire modelling should be wrapped in an outer validation loop. Removing features having a correlation(pearson/spearmann) = 0.6 would probably be unnecessary. Consider to select features with 'variable importance' instead.

@which models are more sensitive to correlated features? - Classifiers/regressors with low or no regularization e.g. LDA or MLR.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.