0
$\begingroup$

I have a training set, and a test set. I think it is important for the model to be tested always on the same test set. Therefore, I cannot mix the training and test sets and perform k-fold cross validation on the whole dataset. (please correct me if I am wrong)

Nevertheless, I need to tune the parameters of my model. I have seen many papers saying, "we tuned the parameters using k-fold cross validation". I know that if I have a single validation set, I can use that to tune the data and then report the results for the test set. How about k-fold cross validation? Let's say k=5. I will have 5 different models and 5 different final results on my test set.

Now I have two questions:

  1. If I want to report the overall result of my model on the test set, should I average the performance of those 5 models and report it? Or should I pick one of them?
  2. If I want to pick a final parameter set (or trained model). Should I pick one of them? Or average (if possible) them and have a final parameter set? If the averaging is done, then I can report the final result on the test set using the averaged parameter set, right?
$\endgroup$
  • 2
    $\begingroup$ Use k-fold on the training set only. After identifying your model run it on your test set. $\endgroup$ – Arun Jose Sep 29 '16 at 9:59
  • $\begingroup$ @ArunJose Thanks. How should I identify the model? By averaging or something? $\endgroup$ – Moh Sep 29 '16 at 10:00
  • 1
    $\begingroup$ Think you are misunderstanding role of cross validation. Please refer: stats.stackexchange.com/questions/52274/… $\endgroup$ – Arun Jose Sep 29 '16 at 10:02
  • $\begingroup$ @ArunJose The reason I'm asking is in many papers I see they report their results of k-fold cross validation and also report standard deviation. I do not understand it. $\endgroup$ – Moh Sep 29 '16 at 10:07
  • 1
    $\begingroup$ The standard deviation tells you about how stable your model is across each fold. When the standard deviation is effectively low, you would go ahead and build a model on your entire training set and achieve your model. Test this on your holdout set to get a final estimate of model performance. Any other approach of averaging is going to mean an ensemble model. That is a different approach outside of scope of this question. $\endgroup$ – Arun Jose Sep 29 '16 at 10:39
1
$\begingroup$

I'm adding this as an answer as the discussion was getting too long for chat.

Your original question has to do about how to use k-fold cross-validation. As the name suggests, it is a technique for validation of a specific model.

If as you require you want to combine the five models to produce an "average" model this is independent of k-fold and moves into the realm of ensemble models.

Use k-fold to assess stability of your model based on standard deviation.

Build a model using the entire training set.

Predict on test set and assess model performance.

The question of reporting "average performance" is a personal choice. There is no right or wrong approach, however doing so you would be advised on researching ensemble models instead of k-fold cross validation. In fact, you could end up doing a k-fold cross validation of an ensemble model too!

$\endgroup$
  • $\begingroup$ Ensemble models are more stable than the individual "submodel" in the ensemble. Thus, they also help only if the problem is instability. (In)stability can be measured using repeated or iterated cross validation (for some types of models also between the surrogate models / folds of a single run/iteration/repetion = normal CV). Cross validation on an ensemble can be done in analogy to out-of-bag performance estimation within the ensemble by sorting out which models are independent of which cases - no additional cross validation loop needed. $\endgroup$ – cbeleites Sep 29 '16 at 12:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.