3
$\begingroup$

When a dataset is needed to be modeled, the process is to take a part of it out as holdout set which is "unseen" by the training method and is used to test the performance of models created using various techniques. Also while training, cross validation divides the data into parts for training and evaluation.

My question is that after a model has been selected using its performance in k-fold cross validation as well as on the holdout set which training methods have not seen at all, is there any need to train a new model on the entire set (including the holdout data) with the selected technique when putting it in actual use?

For example if, after evaluation of different techniques, logistic regression model is selected with some subset of variables. When putting it to use, it is better to use the selected model as it is, or is it better to train a new model using logistic regression (which produced the best model) with same variables set and same other parameters, but with entire data (training and holdout sets combined)?

The reason against training a new model are:

  • The selected model has gone through cross validation and other selection processes which the new model has not, and it might be overfitted. Its performance on holdout set is unknown becuase now there is no holdout set.

  • Depending on modeling technique, the new model might be different than the selected model. For example if lasso is used, on the larger set it might give a different set of variables in the final model, putting more doubts on its real world performance.

On the other hand, using the model that has been selected and then verified using holdout set have the advantage of an evaluated performance on holdout set, but

  • It might have less information because it has been trained on training set which is subset of the entire data. I understand if the dataset is large enough, there is less chance of this and new model being very different but it is still a possibility.

Another scenario that makes the choice even less clear is if the dataset is huge, beyond the capability of the development machine, and therefore a small sample is taken from it. And then the analyst uses multiple techniques, and selects the best technique and model. Now considerable time and effort is required to create a new model on the larger set to prepare it for actual use. Is this step of training a new model on entire set necessary, or the previously selected model should be put to final use.

A small additional question from this question that comes to my mind is: According to bias-variance tradeoff concept, does training on a larger set compared to a smaller sample add more variance and reduces bias from the model, and does this become a factor to consider in my original question?

$\endgroup$
2
$\begingroup$

According to bias-variance tradeoff concept, does training on a larger set compared to a smaller sample add more variance and reduces bias from the model, and does this become a factor to consider in my original question?

Citing Hastie et al. (2001) section 7.3 "The Bias-Variance Decomposition" (as of 2nd edition, 2009), if we assume

$$ Y = f(X) + \varepsilon $$

with $\mathbb{E}(\varepsilon)=0$ and $\text{Var}(\varepsilon)=\sigma_{\varepsilon}^2$, the expected prediction error of a regression fit $\hat f(X)$ at an input point $X=x_0$ using squared error loss will be

$$ \begin{aligned} \text{Err}(x_0) &= \mathbb{E}[((Y-\hat f(x_0))^2|X=x_0] \\ &= [\mathbb{E}\hat f(x_0)-f(x_0)]^2 + \mathbb{E}[\hat f(x_0)-\mathbb{E}\hat f(x_0)]^2 + \sigma_{\varepsilon}^2 \\ &= \text{Bias}^2(\hat f(x_0)) + \text{Var}(\hat f(x_0)) + \sigma_{\varepsilon}^2 \\ &= \text{Bias}^2 + \text{Variance} + \text{Irreducible Error}. \end{aligned} $$

For a given model (a functional form), changing the sample size will only affect $\text{Var}(\hat f(x_0))$; namely, increasing the sample will diminish it. Meanwhile, $\text{Bias}^2(\hat f(x_0))$ will stay the same as the functional form $\hat f(\cdot)$ is fixed. (Clearly, the irreducible error also stays the same.)

So in your original question, you reduce the expected squared error by reestimating the chosen model on the full sample as compared to having estimated it on just the training sample.

References:

  • Friedman, Jerome, Trevor Hastie, and Robert Tibshirani. The Elements of Statistical Learning. Vol. 1. Springer, Berlin: Springer series in statistics, 2001.
| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

Yes, you should do that. It will be a lot faster than the cross-validation, anyway, since you're running the modeling procedure only once instead of k times.

When you do cross-validation, recall, you aren't looking at the model with just one training set; you're trying out a bunch of slightly different training sets. Refitting the model to all your data is similar in this way, and the resulting fitted model differs only from the one you created for each fold in that it has a little more training data, and hence, in all likelihood, it is a little more accurate.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Negative voter please explain. $\endgroup$ – SpeedBirdNine Jul 27 '16 at 9:19
  • 1
    $\begingroup$ This is correct, I don't get the downvote either. @Kodiologist: maybe mention that the purpose of CV in model selection is obtaining information about how well different models would perform, while the final training on all data is just for obtaining the final model. $\endgroup$ – geekoverdose Jul 27 '16 at 9:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.