I am faced with a regression problem which I am addressing using a Random Forest Regressor. I would like to use k folds cross validation to tune the parameters and estimate the out-of-set (i.e. test set) root mean squared error. However, I am a little confused as to what the right order of operations is.

To do this would I simply:

1) split my training data using a train/test split (ex:80/20)

2) run GridSearchCV on the 80 to tune the parameters

3) refit my parameter tuned model on the entire 80 set

4) run k folds on the 20 set to find my out-of-set RMSE for my parameter tuned model?

If so, how does this approach differ from nested k folds cross validation in terms of the bias variance tradeoff ?

  • $\begingroup$ How would you run $k$ folds on the test split (step 4), given that there is only one such split and you already have a model trained on the full train split (based on the current description)? If you would make $K$ 80/20 splits and run steps 1 to 4 as you describe $K$ times, you end up with something very similar to nested cross-validation. Nested cross-validation would be slightly better, though, as a simple resample approach in the outer loop may result in correlated test splits. $\endgroup$ – Marc Claesen Mar 22 '16 at 7:14
  • $\begingroup$ I am not sure if I was not clear or if I am misunderstanding your comment so let me try to clarify what I meant in step 4; GridSearchCV would run a k folds CV on 80% of the data, select the optimal parameters and then refit the optimal model on the 80% of the data (instead of the k folds). Then I would take this tuned model and run it on the remaining and unseen 20% using k folds in order to estimate an out-of-sample RMSE $\endgroup$ – Jay Karimi Mar 22 '16 at 7:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.