# Training with the full dataset after cross-validation?

Is it always a good idea to train with the full dataset after k-fold cross-validation? Or is it better instead to stick with one of the models learned in one of the cross-validation splits for $\alpha_{best}$? Is there any way to tell?

Perhaps most importantly, how can I train with all the samples in my dataset and still fight overfitting?

Some background on the problem:

Say I have a family of models parametrized by $\vec\alpha = \alpha_1, \ldots \alpha_n$. Say also that I have a set of $N$ data points and that I do model selection with k-fold cross-validation to choose the model that best generalizes the data.

For model selection, I can do a grid search on the parameter space of $\vec\alpha$ running k-fold cross-validation for each $\vec\alpha$ candidate. In each of the folds in cross-validation, the set of $N$ data points is split into $N_{train}$ points for training and $N_{validation}$ for validation, and in each of these splits, training with the parameters $\vec\alpha$ along with its corresponding training set $N_{train}$ outputs a learned model $\beta_\alpha$.

The point of cross-validation is that for each of these folds I can check if the learned model overfits on the validation set, and based on this I can choose the model $\beta_\text{best}$ learned for the parameters $\vec\alpha_\text{best}$ that generalized best during cross validation in the grid search.

Now, say that after model selection, I would like to use all the $N$ points in my dataset and hopefully learn a better model. For this I could use the parameters $\vec\alpha_{best}$ corresponding to the model that I chose during model selection, and then after training on the full dataset, I would a get a new learned model $\beta_{full}$.

The problem is that, if I use all points in my dataset for training, I can't check if this new learned model $\beta_{full}$ overfits!

-

The way to think of cross-validation is as estimating the performance obtained using a method for building a model, rather than for estimating the performance of a model.

If you use cross-validation to estimate the hyperparameters of a model (the $\alpha$s) and then use those hyper-parameters to fit a model to the whole dataset, then that is fine, provided that you recognise that the cross-validation estimate of performance is likely to be (possibly substantially) optimistically biased. This is because part of the model (the hyper-parameters) have been selected to minimise the cross-validation performance, so if the cross-valdation statistic has a non-zero variance (and it will) there is the possibility of over-fitting the model selection criterion.

If you want to choose the hyper-parameters and estimate the performance of the resulting model then you need to perform a nested cross-validation, where the outer cross-validation is used to assess the performance of the model, and in each fold cross-validation is used to determine the hyper-paremeters separately in each fold. You build the final model by using cross-validation on the whole set to choose the hyper-parameters and then build the classifier on the whole dataset using the optimized hyper-parameters.

This is of course computationally expensive, but worth it as the bias introduced by improper performance estimation can be large. See my paper

G. C. Cawley and N. L. C. Talbot, Over-fitting in model selection and subsequent selection bias in performance evaluation, Journal of Machine Learning Research, 2010. Research, vol. 11, pp. 2079-2107, July 2010. (www, pdf)

However, it is still possible to have over-fitting in model selection (nested cross-validation just allows you to test for it). A method I have found useful is to add a regularisation term to the cross-validation error that penalises hyper-parameter values likely to result in overly-complex models, see

G. C. Cawley and N. L. C. Talbot, Preventing over-fitting in model selection via Bayesian regularisation of the hyper-parameters, Journal of Machine Learning Research, volume 8, pages 841-861, April 2007. (www,pdf)

So the answers to your question are (i) yes, you should use the full dataset to produce your final model as the more data you use the more likely it is to generalise well but (ii) make sure you obtain an unbiased performance estimate via nested cross-validation and potentially consider penalising the cross-validation statistic to further avoid over-fitting in model selection.

-
+1: Answers the question: "If you use cross-validation to estimate the hyperparameters of a model (the αs) and then use those hyper-parameters to fit a model to the whole dataset, then that is fine…" –  Neil G Apr 19 '12 at 19:09
What could be a different approach from a nested cross-validation? Can we run some initial experiments and observe a proper setting for the hyper-parameter and then use cross-validation for the parameters? –  soufanom Dec 15 '12 at 11:13
@soufanom, no, the use of "initial experiments" to make choices regarding the model is likely to result in over-fitting and almost certainly will introduce an optimistic bias into the performance analysis. The cross-validation used for performance analysis must repeat EVERY step used in fitting the model independently in each fold. The experiments in my paper show that kernel models can be very sensitive to this sort of bias, so it is vital to perform the model selection and performance evaluation with all possible rigour. –  Dikran Marsupial Dec 15 '12 at 12:58
Thanks for your direct response. I want to avoid nested-cross validation because of its high computational time cost. Lets say I have an alpha and lambda parameters. Can I loop over values of alpha and within every value we apply cross-validation for lambda? This might still more optimistic than the nested-cross validation but is it acceptable? –  soufanom Dec 15 '12 at 18:53
For kernel methods, such as the SVM, it is often possible to perform leave-one-out cross-validation at almost no computational cost (see the papers listed in my answer). I use this "virtual" leave-one-out cross-validation for tuning the hyper-parameters, nested in k-fold cross-validation for performance evaluation. The cost is then quite reasonable. In my opinion it is not acceptable to use any procedure where the performance evaluation is biased in any way by the tuning of the hyper-parameters. It is worth the computational expense to get a reliable estimate. –  Dikran Marsupial Dec 15 '12 at 19:08
show 2 more comments

Just to add to the answer by @mark999, Max Kuhn's caret package (Classification and Regression Training) is the most comprehensive source in R for model selection based on bootstrap cross validation or N-fold CV and some other schemes as well.

Not to disregard the greatness of the rms package, but caret lets you fit pretty much every learning method available in R, whereas validate only works with rms methods (I think).

The caret package is a single infrastructure to pre process data, fit and evaluate any popular model, hence it is simple to use for all methods and provides graphical assessment of many performance measures (something that next to the overfit problem might influence the model selection considerably as well) over your grid and variable importance.

See the package vignettes to get started (it is very simple to use)
Data Preprocessing
Variable Selection with caret
Model Building with caret
Variable Importance

-
Thanks. Do you know if, after model selection (which is done by calling train), there is a way in caret to train with the full dataset? –  user023472 Feb 15 at 20:59
Not sure if that is a good idea or why you would want to that, but you can just fit the final model returned by train to the full data set. –  Momo Feb 17 at 15:24

I believe that Frank Harrell would recommend bootstrap validation rather than cross validation. Bootstrap validation would allow you to validate the model fitted on the full data set, and is more stable than cross validation. You can do it in R using validate in Harrell's rms package.

See the book "Regression Modeling Strategies" by Harrell and/or "An Introduction to the Bootstrap" by Efron and Tibshirani for more information.

-
To omit a next myth about "bad CV", this is a terminology problem -- Harrell's "cross validation" means N-fold CV, and "bootstrap validation" means resampling CV. Obviously I agree that this second flavor is more stable and overall nicer, but this is also a type of cross validation. –  mbq Jun 5 '11 at 20:29
mark999 or @mbq, would you mind elaborating on how bootstrap would allow one to validate a model fitted on the full dataset? –  user023472 Feb 7 at 19:19
@user27915816 Well, in principle nohow; the idea behind cross-validation is that you tests whether given training method is reliably making good models on a sets very similar to the final one, and, if so, generalise this observation to the full set with a silent assumptions that nothing strange will happen and that CV method you used is not somehow biased. This is of course almost always good enough, still you can never be sure that the model built on all the data you have is not overfitted. –  mbq Feb 7 at 23:10

What you do is not a cross validation, rather some kind of stochastic optimization.

The idea of CV is to simulate a performance on unseen data by performing several rounds of building the model on a subset of objects and testing on the remaining ones. The somewhat averaged results of all rounds are the approximation of performance of a model trained on the whole set.

In your case of model selection, you should perform a full CV for each parameter set, and thus get a on-full-set performance approximation for each setup, so seemingly the thing you wanted to have.

However, note that it is not at all guaranteed that the model with best approximated accuracy will be the best in fact -- you may cross-validate the whole model selection procedure to see that there exist some range in parameter space for which the differences in model accuracies are not significant.

-
Thanks @mbq, but I'm not sure I follow. I do N-fold cross-validation for each point value of my grid search in the hyperparameter space. The average result of the N-folds gives me the approximation that you mention, which I use to compare models and do model selection by selecting the model the best fits the validation set. My question is about what happens when I train with the full dataset. I think the learned model changes (the $\vec\beta$ parameters of the learned model change), and in principle I have no way to know if I suffer from overfitting. –  user023472 Jun 5 '11 at 22:05
@AmV If so, ok -- as I wrote, CV already tests full set scenario, you cannot say more without new data. Again, you can at most do a nested CV to see if there is no overfitting imposed by the model selection itself (if the selection gives very good improvement or the data is noisy the risk of this is quite big). –  mbq Jun 5 '11 at 22:15

I think you have a bunch of different questions here:

The problem is that, if I use all points in my dataset for training, I can't check if this new learned model βfull overfits!

The thing is, you can use (one) validation step only for one thing: either for parameter optimization, (x)or for estimating generalization performance.

So, if you do parameter optimization by cross validation (or any other kind of data-driven parameter determination), you need test samples that are independent of those training and optimization samples. Dikran calls it nested cross validation, another name is double cross validation. Or, of course, an independent test set.

So here's the question for this post : Is it a good idea to train with the full dataset after k-fold cross-validation? Or is it better instead to stick with one of the models learned in one of the cross-validation splits for αbest?

Using one of the cross validation models usually is worse than training on the full set (at least if your learning curve performance = f (nsamples) is still increasing. In fact, it is: if it wasn't, you would probably have set aside an independent test set.)

If you observe a large variation between the cross validation models (with the same parameters), then your models are unstable. In that case, aggregating the models can help and actually be better than using the one model trained on the whole data.

Perhaps most importantly, how can I train with all points in my dataset and still fight overfitting?

By being very conservative with the degrees of freedom allowed for the "best" model, i.e. by taking into account the (random) uncertainty on the optimization cross validation results. If the d.f. are actually appropriate for the cross validation models, chances are good that they are not too many for the larger training set. The pitfall is that parameter optimization is actually multiple testing. You need to guard against accidentally good looking parameter sets.

-