1
$\begingroup$

I'm creating a random forest model (classification) in caret:

model <- train(formula,
               data = training.data,
               method = "rf")

I can get the variable importance list which ranks all the variables from most important to least:

varImp(model)

I can use that to report the top 20 predictors, but I'm interested in finding out at what point additional predictors don't significantly increase model fit. Twenty is somewhat arbitrary. Is there some statistical test to find out which predictors are necessary and a way to incorporate it into caret syntax?

$\endgroup$
  • 1
    $\begingroup$ you might want to look into the fscaret package. $\endgroup$ – phiver Feb 9 '18 at 17:06
1
$\begingroup$

I am not sure about a statistical test. But, one way you could try is a process of iteration. Since you know predictors and their ranks, you can add predictors in increments to your model. Make sure to keep track of your performance measure. In the end you should have 20 iterations. You can plot your performance measure and using elbow technique you can identify the point at which, adding a new predictor doesn't make to much of a difference.

$\endgroup$
1
$\begingroup$

You can add features until the increase is not more than some error bars. More importantly, make sure you make this decision before you evaluate on your final test set (e.g. with either nested cross-validation or a separate validation set). Otherwise you risk overfitting.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.