Should I join train and validation sets for final NN model training? If yes, when to stop training the final model?

Question

Normally we divide our dataset into 3 sets:

• train set,
• validation set,
• test set.

We use train set to find optimal parameters (weights and biases of NN) and validation set to find optimal NN architecture (e.g. # of hidden layers, # of neurons in each hidden layer...).

Here is my question: after we find optimal architecture of the model (using validation set) and optimal number of training epochs (for the optimal model architecture) using early stopping and validation set, which of the following is more appropriate:

• test final model (trained only on train set) on test set,
• join train and validation set (do not waste validation data) and re-train the model (with previously found optimal architecture) on joined set (train + validation)?

Is there any kind of rule of thumb for this, does it depend on application, size of train set or something else?

If second option is more appropriate (train + validation for final model), when do we stop training? Should we use optimal number of epochs from early stopping (because it was optimal for train set, but not for train + validation) or stopping criteria should be something else?

Thanks!

Answer 1

In general, the second option is better b/c it theoretically results in a model with lower variance than the one trained only on the train dataset, even if you recycle hyperparameters. And as long as the test set predictions you'll make are independent of the test set labels, you'll get an unbiased estimate of the train + val model's error by evaluating it on test.

Before going ahead, you must assume that your NN training procedure is stable. If you can't assume this, you'd have to assess stability by running your training procedure many times. But at that point, you might as well run k-fold CV on train + val and re-optimize hyperparameters.

If second option is more appropriate (train + validation for final model), when do we stop training? Should we use optimal number of epochs from early stopping (because it was optimal for train set, but not for train + validation) or stopping criteria should be something else?

Of course, it depends on how much your selected model changes as its training data increases, and the sizes of the train and val datasets. The optimal hyperparameter settings are likely different for a model trained on the train + val dataset—greater model complexity, e.g., more layers, more epochs, is allowed as there's more training data. But if you did something like 70% train 20% val, I'd err towards simply using the same hyperparameter settings (which you found by cross-validating on the val dataset). This model will underfit. But it's probably not worth addressing.

You could even go as far as deploying the model trained on the entire dataset (train + val + test) assuming (1) your training procedure is stable, and (2) you're okay with slightly overestimating error (think about the model's application).

One extra (and slightly unnecessary) note re

Is there any kind of rule of thumb for this, does it depend on application, size of train set or something else?

In my limited experience, I've found that rules of thumb in ML often turn out to come from a game of telephone. Be wary of rules which lack specificity and repeatable results. Running your own careful simulation experiments can be expensive, but they're usually worth it.