Suppose I trained several models on training set, choose best one using cross validation set and measured performance on test set. So now I have one final best model. Should I retrain it on my all available data or ship solution trained only on training set? If latter, then why?

UPDATE: As @P.Windridge noted, shipping a retrained model basically means shipping a model without validation. But we can report test set performance and after that retrain the model on complete data righteously expecting the performance to be better - because we use our best model plus more data. What problems may arise from such methodology?

  • $\begingroup$ Are you working in an externally regulated environment? (i.e. possibly you must ship the validated model, and your question is only hypothetical, but it's worth discussing anyway :)). Edit: ok I see you edited your post. $\endgroup$ Nov 29 '15 at 11:55
  • $\begingroup$ Do you believe that your test data is representative of the population/cover a part of the population not in the dev sample? Is your original development sample deficient in some way? $\endgroup$ Nov 29 '15 at 11:57
  • $\begingroup$ @P.Windridge well, my question is just hypothetical. About your second comment I believe no one should expect an engineer to train a good model while giving him unrepresentative data. $\endgroup$
    – Yurii
    Nov 29 '15 at 12:00
  • 1
    $\begingroup$ I can't imagine many situations where you'd ship a model without validation. I'd rather look into decreasing the size of the test sample (subject to it still being large enough to validate on!). A possibly more interesting discussion is about the pros/cons of /selecting/ the model based on /all/ the data, and then training it using a sub-sample, and then validating on the rest. $\endgroup$ Nov 29 '15 at 12:18
  • 1
    $\begingroup$ Similar question= stats.stackexchange.com/questions/174026/… , although I think it could use more discussion $\endgroup$ Nov 29 '15 at 12:24

You will almost always get a better model after refitting on the whole sample. But as others have said you have no validation. This is a fundamental flaw in the data splitting approach. Not only is data splitting a lost opportunity to directly model sample differences in an overall model, but it is unstable unless your whole sample is perhaps larger than 15,000 subjects. This is why 100 repeats of 10-fold cross-validation is necessary (depending on the sample size) to achieve precision and stability, and why the bootstrap for strong internal validation is even better. The bootstrap also exposes how difficult and arbitrary is the task of feature selection.

I have described the problems with 'external' validation in more detail at Biostatistics in Biomedical Research Section 10.11.

  • $\begingroup$ Terminology in my field (analytical chemistry) would consider any splitting of the data you do at (before) beginning the training very much an internal validation. External validation would begin somewhere between doing a dedicated validation study and ring trials. $\endgroup$ Aug 14 '19 at 8:39

You dont need to re-train again. When you report your results, you always report test data results because they give much better understanding. By test data set we can more accurately see how well a model is likely to perform on out-of-sample data.

  • 5
    $\begingroup$ We can report test set performance and after that retrain the model on complete data righteously expecting the performance to be better - because we use best mode plus more data. Is there a flaw in my reasoning? $\endgroup$
    – Yurii
    Nov 29 '15 at 12:08
  • $\begingroup$ Well if after testing, u collect more data then then u can re-split the data, re-train it again, then re-test it and then report the test result from the re-test. $\endgroup$
    – Umar
    Nov 29 '15 at 12:24
  • 6
    $\begingroup$ By not estimating on the whole sample you forego the opportunity of higher efficiency. This is not justified. I also agree with Yurii's comment above. $\endgroup$ Nov 29 '15 at 15:16
  • $\begingroup$ @RichardHardy, whats wrong in my comment ? $\endgroup$
    – Umar
    Nov 29 '15 at 15:29
  • $\begingroup$ It's spelled out in my last comment. By not utilizing all the data for estimating the model you are foregoing the highest available efficiency. Why do that? $\endgroup$ Nov 29 '15 at 16:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.