Should final (production ready) model be trained on complete data or just on training set?

Question

Suppose I trained several models on training set, choose best one using cross validation set and measured performance on test set. So now I have one final best model. Should I retrain it on my all available data or ship solution trained only on training set? If latter, then why?

UPDATE: As @P.Windridge noted, shipping a retrained model basically means shipping a model without validation. But we can report test set performance and after that retrain the model on complete data righteously expecting the performance to be better - because we use our best model plus more data. What problems may arise from such methodology?

Answer 1

You will almost always get a better model after refitting on the whole sample. But as others have said you have no validation. This is a fundamental flaw in the data splitting approach. Not only is data splitting a lost opportunity to directly model sample differences in an overall model, but it is unstable unless your whole sample is perhaps larger than 15,000 subjects. This is why 100 repeats of 10-fold cross-validation is necessary (depending on the sample size) to achieve precision and stability, and why the bootstrap for strong internal validation is even better. The bootstrap also exposes how difficult and arbitrary is the task of feature selection.

I have described the problems with 'external' validation in more detail in BBR Chapter 10.

Answer 2

Unless you're limiting yourself to a simple class of convex models/loss functions, you're considerably better off keeping a final test split. Here's why:

Let's say you collect iid sample pairs from your data generating distribution, some set of (x, y). You then split this up into a training and test set, and train a model on the training set. Out of that training process you get a model instance, f(x; w). Where w denotes the model parameters.

Let's say you have N observations in the test set. When you validate this model on that test set you form the set of test predictions, {f(x_i, w) : i=1,2,...,N} and compare it to the set of test labels {y_i : i=1,2,...,N} using a performance metric.

What you're able to say using N independent observations is how you expect that model instance, i.e. the function given a specific w, will generalize to other iid data from the same distribution. Importantly, you only really have one observation (that w you found) to comment on your process for determining f(x, w), i.e. the training process. You can say a little more using something like k-fold cross validation, but unless you're willing to do exhaustive cross-validation (which is not really feasible in a computer vision or NLP context), you'll always have less data on the reliability of your training process.

Take a pathological example, where you draw the model parameters at random, and you don't train them at all. You obtain some model instance f(x, w_a). Despite the absurdity of your (lack of) training process, your test set performance is still indicative of how that model instance will generalize to unseen data. Those N observations are still perfectly valid to use. Maybe you'll have gotten lucky and have landed on a pretty good w_a. However, if you combine the test and training set, then "retrain" the model to obtaining a w_b, you're in trouble. The results of your previous test performance amounts to basically a point estimate of how well your next random parameter draw will fare.

There are statistical results that you can use to comment on the reliability of the entire training process. But they require some assumptions about your model class, loss function, and your ability to find the best f(x, w) from within that class for any given set of training observation. With all that, you can get some bounds on the probability that your performance on unseen data will deviate by more that a certain amount from what you measured on the training data. However, those results do not carry over (in a useful way) to overparameterized and non-convex models like modern neural networks.

The pathological example above is a little over the top. But as an ML researcher and consultant, I have seen neural network training pipelines that occasionally latch on to terrible local minima, but otherwise perform great. Without a final test split, you'd have no way of being sure that hadn't happened on your final retraining.

More generally, in a modern machine learning context, you cannot treat the models coming out of your training process as interchangeable. Even if they do perform similarly on a validation set. In fact, you may see considerable variation from one model to the next when using the full bag of stochastic optimization tricks. (For more details on that, check out this work on underspecification.)

Answer 3

You dont need to re-train again. When you report your results, you always report test data results because they give much better understanding. By test data set we can more accurately see how well a model is likely to perform on out-of-sample data.