I would like to eventually use the PIMP-Algorithm (Permutation Variable Importance Measure) in order to get p values for the variables' importance. However, the formula

          "PIMP"(X, y, rForest, S = 100, parallel = FALSE, ncores=0, seed = 123, ...)

requires rForest which is an object of class randomForest.

I can carry out the 5 times repeated 10 fold cross-validation fine using caret.

   rf.fit <- train(T2DS ~ ., 
            data = mod_train.new, 
            method = "rf",     
            importance = TRUE, 
            trControl = trainControl(method = "repeatedcv", 
                                     number = 10, 
                                     repeats = 5))

However, I cannot seem to find any examples of documentation as to how to implement this using randomForest. The below is incorrect.

   rf.fit.try <- randomForest(T2DS ~., data=mod_train.new, importance=TRUE, 
      trControl=trainControl(method="repeatedcv", number=10, repeats=5))

Please could anybody suggest how the repeated measures cross-validation can be done using the randomForest package, or an alternative way I can calculate p values for my variable importances following permutation?

  • $\begingroup$ I don't fully understand your question. So you want to pass a randomForest object into PIMP-Algorithm, and this object, you would like the parameters to be optimized by CV? $\endgroup$
    – StupidWolf
    Commented May 22, 2020 at 23:23
  • $\begingroup$ @StupidWolf, from my recent reading, I can see that caret can only be used to fine-tune accuracy (this can't be done via randomForest). However, I am assuming I can input the rf.fit$finalModel into the PIMP algorithm? I can see that the p values generated are more significant using this model from caret versus the randomForest package. $\endgroup$
    – Willow9898
    Commented May 22, 2020 at 23:34
  • $\begingroup$ you are more or less correct. It's not fine-tuning. caret test a series of mtry for the randomForest model and chooses the best mtry. If you run randomForest, and don't specify mtry, the default is floor(ncol(x)/3) for classification and floor(sqrt(ncol(x)) for regression $\endgroup$
    – StupidWolf
    Commented May 22, 2020 at 23:37
  • $\begingroup$ answer is yes, you can plug in the finalModel. it should be ok $\endgroup$
    – StupidWolf
    Commented May 22, 2020 at 23:37
  • $\begingroup$ @StupidWolf, thanks for your help. Was just wondering ( if you're familiar with this) what your opinion is on SMOTE (Synthetic Minority Oversampling Technique To Handle Class Imbalancy In Binary Classification). I have an unbalanced data set with an excess of controls in an initial training set (69 vs 26). Would you randomly select ~ 29 controls from this so they are balanced or use SMOTE? $\endgroup$
    – Willow9898
    Commented May 22, 2020 at 23:43

1 Answer 1


You can always provide the final model obtained from training a randomForest with caret:


rf.fit <- train(Class ~ ., data = Sonar,method = "rf",importance=TRUE,
          trControl = trainControl(method = "repeatedcv",number = 10,repeats=5))

[1] "train"         "train.formula"

[1] "randomForest"

res = "PIMP"(Sonar[,-ncol(Sonar)], Sonar$Class, rf.fit$finalModel, S = 10)
  • 1
    $\begingroup$ Thank you very much, this answers my comment above too! I have used rf.fit$finalModel in mine. $\endgroup$
    – Willow9898
    Commented May 22, 2020 at 23:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.