When a dataset is needed to be modeled, the process is to take a part of it out as holdout set which is "unseen" by the training method and is used to test the performance of models created using various techniques. Also while training, cross validation divides the data into parts for training and evaluation.
My question is that after a model has been selected using its performance in k-fold cross validation as well as on the holdout set which training methods have not seen at all, is there any need to train a new model on the entire set (including the holdout data) with the selected technique when putting it in actual use?
For example if, after evaluation of different techniques, logistic regression model is selected with some subset of variables. When putting it to use, it is better to use the selected model as it is, or is it better to train a new model using logistic regression (which produced the best model) with same variables set and same other parameters, but with entire data (training and holdout sets combined)?
The reason against training a new model are:
The selected model has gone through cross validation and other selection processes which the new model has not, and it might be overfitted. Its performance on holdout set is unknown becuase now there is no holdout set.
Depending on modeling technique, the new model might be different than the selected model. For example if lasso is used, on the larger set it might give a different set of variables in the final model, putting more doubts on its real world performance.
On the other hand, using the model that has been selected and then verified using holdout set have the advantage of an evaluated performance on holdout set, but
- It might have less information because it has been trained on training set which is subset of the entire data. I understand if the dataset is large enough, there is less chance of this and new model being very different but it is still a possibility.
Another scenario that makes the choice even less clear is if the dataset is huge, beyond the capability of the development machine, and therefore a small sample is taken from it. And then the analyst uses multiple techniques, and selects the best technique and model. Now considerable time and effort is required to create a new model on the larger set to prepare it for actual use. Is this step of training a new model on entire set necessary, or the previously selected model should be put to final use.
A small additional question from this question that comes to my mind is: According to bias-variance tradeoff concept, does training on a larger set compared to a smaller sample add more variance and reduces bias from the model, and does this become a factor to consider in my original question?