When using k-fold cross validation in a neural network, do you also need a separate validation set? Or is the use of the k-fold on its own good enough to minimise the possibility of over-training?
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
It depends on what you want to do. There are two cases:
- You want to convince someone that your method is good.
- You want to use your method in production.
In the case of 1., you should have a test set that you never even looked at in your model bulding phase. Otherwise information from the test set might leak into the model building process via your brain.
In the case of 2., the best approach is probably to use the full data for model selection and train on the full data afterwards. To battle overfitting, you can use many folds.
A more recent and exotic approach is to do the following:
- Split data into train, validation and test.
- Optimize model on train performing early stopping on the validation data. Remember training error $e_{\text{train}}$.
- Further optimize model on training and validation, until $e_{\text{validation}} = e_{\text{train}}$.
- Evaluate $e_{\text{test}}$
I don't know of any theoretical justification for this, but it seems to work quite well.
