By trial and error I found that my random forest regressor from sklearn is giving higher RMSE when I add features to training. Also the importance of features is changing in each addition of feature. When does this happen? I expect the model to give same RMSE or improved RMSE with each addition. Is there a way to find which combination of features provide the lowest RMSE?
Intuition might tell you that adding more variables should make the model better, and to some degree it is true, but after a certain number of variables this does not hold anymore and adding more variables only increases the complexity of the model.
Finding the right combination of variables which minimizes your loss functions (such as RMSE) and minimizes the used variables is a hard problem and I do not know of many procedure besides the kind of trial and error, as in try different combinations and see what you get.
An alternative is to use some sort of regularization in your trees, such as regularized random forest, which penalizes the variables based on some crieria.