35
$\begingroup$

Suppose I trained several models on training set, choose best one using cross validation set and measured performance on test set. So now I have one final best model. Should I retrain it on my all available data or ship solution trained only on training set? If latter, then why?

UPDATE: As @P.Windridge noted, shipping a retrained model basically means shipping a model without validation. But we can report test set performance and after that retrain the model on complete data righteously expecting the performance to be better - because we use our best model plus more data. What problems may arise from such methodology?

$\endgroup$
7
  • $\begingroup$ Are you working in an externally regulated environment? (i.e. possibly you must ship the validated model, and your question is only hypothetical, but it's worth discussing anyway :)). Edit: ok I see you edited your post. $\endgroup$ Commented Nov 29, 2015 at 11:55
  • $\begingroup$ Do you believe that your test data is representative of the population/cover a part of the population not in the dev sample? Is your original development sample deficient in some way? $\endgroup$ Commented Nov 29, 2015 at 11:57
  • $\begingroup$ @P.Windridge well, my question is just hypothetical. About your second comment I believe no one should expect an engineer to train a good model while giving him unrepresentative data. $\endgroup$
    – Yurii
    Commented Nov 29, 2015 at 12:00
  • 1
    $\begingroup$ I can't imagine many situations where you'd ship a model without validation. I'd rather look into decreasing the size of the test sample (subject to it still being large enough to validate on!). A possibly more interesting discussion is about the pros/cons of /selecting/ the model based on /all/ the data, and then training it using a sub-sample, and then validating on the rest. $\endgroup$ Commented Nov 29, 2015 at 12:18
  • 1
    $\begingroup$ Similar question= stats.stackexchange.com/questions/174026/… , although I think it could use more discussion $\endgroup$ Commented Nov 29, 2015 at 12:24

3 Answers 3

24
$\begingroup$

You will almost always get a better model after refitting on the whole sample. But as others have said you have no validation. This is a fundamental flaw in the data splitting approach. Not only is data splitting a lost opportunity to directly model sample differences in an overall model, but it is unstable unless your whole sample is perhaps larger than 15,000 subjects. This is why 100 repeats of 10-fold cross-validation is necessary (depending on the sample size) to achieve precision and stability, and why the bootstrap for strong internal validation is even better. The bootstrap also exposes how difficult and arbitrary is the task of feature selection.

I have described the problems with 'external' validation in more detail in BBR Chapter 10.

$\endgroup$
1
  • 1
    $\begingroup$ Terminology in my field (analytical chemistry) would consider any splitting of the data you do at (before) beginning the training very much an internal validation. External validation would begin somewhere between doing a dedicated validation study and ring trials. $\endgroup$
    – cbeleites
    Commented Aug 14, 2019 at 8:39
2
$\begingroup$

Unless you're limiting yourself to a simple class of convex models/loss functions, you're considerably better off keeping a final test split. Here's why:

Let's say you collect iid sample pairs from your data generating distribution, some set of (x, y). You then split this up into a training and test set, and train a model on the training set. Out of that training process you get a model instance, f(x; w). Where w denotes the model parameters.

Let's say you have N observations in the test set. When you validate this model on that test set you form the set of test predictions, {f(x_i, w) : i=1,2,...,N} and compare it to the set of test labels {y_i : i=1,2,...,N} using a performance metric.

What you're able to say using N independent observations is how you expect that model instance, i.e. the function given a specific w, will generalize to other iid data from the same distribution. Importantly, you only really have one observation (that w you found) to comment on your process for determining f(x, w), i.e. the training process. You can say a little more using something like k-fold cross validation, but unless you're willing to do exhaustive cross-validation (which is not really feasible in a computer vision or NLP context), you'll always have less data on the reliability of your training process.

Take a pathological example, where you draw the model parameters at random, and you don't train them at all. You obtain some model instance f(x, w_a). Despite the absurdity of your (lack of) training process, your test set performance is still indicative of how that model instance will generalize to unseen data. Those N observations are still perfectly valid to use. Maybe you'll have gotten lucky and have landed on a pretty good w_a. However, if you combine the test and training set, then "retrain" the model to obtaining a w_b, you're in trouble. The results of your previous test performance amounts to basically a point estimate of how well your next random parameter draw will fare.

There are statistical results that you can use to comment on the reliability of the entire training process. But they require some assumptions about your model class, loss function, and your ability to find the best f(x, w) from within that class for any given set of training observation. With all that, you can get some bounds on the probability that your performance on unseen data will deviate by more that a certain amount from what you measured on the training data. However, those results do not carry over (in a useful way) to overparameterized and non-convex models like modern neural networks.

The pathological example above is a little over the top. But as an ML researcher and consultant, I have seen neural network training pipelines that occasionally latch on to terrible local minima, but otherwise perform great. Without a final test split, you'd have no way of being sure that hadn't happened on your final retraining.

More generally, in a modern machine learning context, you cannot treat the models coming out of your training process as interchangeable. Even if they do perform similarly on a validation set. In fact, you may see considerable variation from one model to the next when using the full bag of stochastic optimization tricks. (For more details on that, check out this work on underspecification.)

$\endgroup$
-1
$\begingroup$

You dont need to re-train again. When you report your results, you always report test data results because they give much better understanding. By test data set we can more accurately see how well a model is likely to perform on out-of-sample data.

$\endgroup$
5
  • 5
    $\begingroup$ We can report test set performance and after that retrain the model on complete data righteously expecting the performance to be better - because we use best mode plus more data. Is there a flaw in my reasoning? $\endgroup$
    – Yurii
    Commented Nov 29, 2015 at 12:08
  • $\begingroup$ Well if after testing, u collect more data then then u can re-split the data, re-train it again, then re-test it and then report the test result from the re-test. $\endgroup$
    – Umar
    Commented Nov 29, 2015 at 12:24
  • 6
    $\begingroup$ By not estimating on the whole sample you forego the opportunity of higher efficiency. This is not justified. I also agree with Yurii's comment above. $\endgroup$ Commented Nov 29, 2015 at 15:16
  • $\begingroup$ @RichardHardy, whats wrong in my comment ? $\endgroup$
    – Umar
    Commented Nov 29, 2015 at 15:29
  • $\begingroup$ It's spelled out in my last comment. By not utilizing all the data for estimating the model you are foregoing the highest available efficiency. Why do that? $\endgroup$ Commented Nov 29, 2015 at 16:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.