Refitting on the whole dataset after train-validation split? [duplicate]

Question

Suppose I am facing a problem that needs hyper tuning, say batch learning with different batch sizes. I am splitting the training data into (80% train + 20% validation) and will choose the optimal value of batch size according to the lowest validation error.

Here is my question: do I need to refit the model (with the optimal batch size) on the entire 100% dataset?

I think it's a reasonable choice by intuition, there should be no harm in doing so (except for extra training time). But I'm not sure if it's useful in practice. I've asked several friends doing research in machine learning and got two opposite responses:

• It is incorrect to NOT do so, you always want to use all the data;

• It is not necessary to do so, results from a train-validation split is close enough;

Can anyone tell me which one I should follow? Thanks!

Answer 1

Once you have chosen a model, you want to fit it to as much data as possible, so, yes, refitting it is the right thing to do.

This is of course more important for smaller than for more extensive data sizes. If you have sizes like e.g. that of MNIST (60,000 + 10,000), the model fitted to the complete dataset of 70,000 cases often doesn't change in any relevant way. And with millions of cases, if they are indeed all from the same population, refitting won't change anything.

As a very rough rule of thumb, the error of many estimations goes approximately with $\sigma/\sqrt{n}$, where $\sigma$ is the correct standard deviation and $n$ is the sample size. You can use this as a coarse guide to what improvement you can expect from increasing the dataset size.

And always keep an eye on the effect size of the parameter: if the parameter has only a negligible effect on your results, you don't have to bother anyway.