How to use a varImp function to select features from training set?

Question

Till now I have used a following flow for training a random forest model.

create 10 folds of data.
for each fold i:
    - use ith fold as validation data
    - use remaining 9 folds as training data
    - apply normalization on training and validation data
    - # apply feature selection on training data
    - # select same features from validation data
    - train random forest on training data
    - predict values for validation data
combine all predictions.

Now I want to do feature selection using varImp() function. I am confused as it is said that varImp itself trains a model on training data to find out best set of features.

How should I use varImp to get important features (say using partial least squares) and then again apply training model on training data?

Answer 1

This sounds a lot like recursive feature elimination. See the caret help page for feature selection.

Answer 2

I'd suggest removing the least important/lowest varImp features using this criterion, then re-runing test and validation sets. In some cases, but not always, this would increase your success. Note that normalization is generally not recommended for this approach and may in fact result in a loss of valuable signals.