0
$\begingroup$

By trial and error I found that my random forest regressor from sklearn is giving higher RMSE when I add features to training. Also the importance of features is changing in each addition of feature. When does this happen? I expect the model to give same RMSE or improved RMSE with each addition. Is there a way to find which combination of features provide the lowest RMSE?

$\endgroup$
0
$\begingroup$

Intuition might tell you that adding more variables should make the model better, and to some degree it is true, but after a certain number of variables this does not hold anymore and adding more variables only increases the complexity of the model.

Finding the right combination of variables which minimizes your loss functions (such as RMSE) and minimizes the used variables is a hard problem and I do not know of many procedure besides the kind of trial and error, as in try different combinations and see what you get.

An alternative is to use some sort of regularization in your trees, such as regularized random forest, which penalizes the variables based on some crieria.

$\endgroup$
2
  • $\begingroup$ This was indeed useful information. But I just wonder that just more than two variables is making RMSE lower. Is this because the bad choice of variables? I assumed that Random forest doesn't consider that variable if it is making the model worse than better $\endgroup$ – Chinti Mar 31 '20 at 9:21
  • 1
    $\begingroup$ @Chinti That is not strictly true. RF uses feature bagging during model building, meaning it samples a random subset of variables during each node in the tree, and selects the best variable to split upon based on this subsample, so it will happen that you will select a "bad" variable just by chance. $\endgroup$ – user2974951 Mar 31 '20 at 9:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.