Doubt about k-Fold CrossValidation [duplicate]

Question

In k-fold crossvalidation, data is divided in k folds, then k-1 folds are taken for training and 1 fold is taken for validation.

This process is repeated k times, taking a different fold for validation each time, so we end with k different models and k different results (although the results should be very similar if everything is OK)

This is very useful to have an idea of the average performance of a classifier. Some validation folds may contain outliers which will give extreme results, but repeating for different validation sets solves this problem.

Now my question:

At the end I want a final model which is the one that I'll use to get the final results on the testing set, and the one that I'll end up using in production. Since in k-fold crossval I trained k different models, which one to I use as my "final" model?

• The one with better results?
• Do I train again a model this time using both train and validation sets as training?

How can I go from a crossvalidation to a final model?

Answer 1

You would do the cross validation to compare performance of a few algorithms on the same folds of the same data-set. Then you take the best one* and retrain it on the entire data-set. So your second option.

*The best one is typically the one with the highest average performance, but you often have some discretion in choosing. If there aren't statistically significant differences between some of the algorithms, you can also take the least computationally complex one, the most intuitive one or the one with the least parameters to tune.

Note also that this procedures only works if all the algorithms you compare either have few or no parameters to tune or, if they have parameters, provide good default values from the literature.

If you have to tune model parameters, you need to do a nested cross validation which makes it less straightforward to have a final model to apply to unseen data.