0
$\begingroup$

This question already has an answer here:

Consider that I use k-fold cross validation and select the best model based on the smallest mean error or using some heuristic to choose the best model. After I chose the best model, if I use the full training data to train this model, would it be guaranteed to overcome overfitting (or less overfitting) in deep neural networks?

Is it a good idea to train with the full training dataset after k-fold cross-validation, or again do another set of cross validation (split training dataset to training and validation) and then train the best model on the part of training data?

I read other questions regarding k-fold cross validation but none of them fully convinced me that when selected deep network model trained on whole training dataset will not overfit. As deep neural network, I mean the number of parameters to train P are far more than number of full training samples N (P>>N).

$\endgroup$

marked as duplicate by user20160, Peter Flom Feb 9 '18 at 11:50

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • $\begingroup$ I have already read that, my question is mostly if you have complex model with lots of parameter P>>N, does it still hold to train with the full dataset after k-fold cross validation? (Is it going to hurt the performance or improve it) $\endgroup$ – karaspd Feb 8 '18 at 18:25
2
$\begingroup$

Cross validation would never overcome over-fitting issue, instead, CV can only control it.

The purpose of cross validation in neural network:

  1. helping choosing optimal number of epochs (controlling over training weights in neural network )
  2. robust metrics to measure the performance under the current parameter settings

It is also a good practise to train your model using full data with chosen epochs (by cross validation). Cross validation will always leaves a proportion of data aside for prediction (hence model metrics), however, training model will full data use all info without leaving any info on the table.

There is an idea to avoid training on full data again: with cross validation, you will have 5 models (say you have done 5-fold cross validation). And your prediction for new data would be the average of 5 model predictions. This is so called bagging in machine learning.

There is a good discussion on cross validation, and hopefully this is useful:

Training with the full dataset after cross-validation?

$\endgroup$
  • $\begingroup$ When you want to train on big networks with more parameters than your dataset, your chance of overfitting is higher if you use all the training data. Consider that I used k-fold cross validation, and I chose my network (parameters, architecture, number of epochs, ...), later for testing a model with test dataset, one way is to train the selected network with full training data. My assumption is that I can get overfitted by using the full training data in this complex model. $\endgroup$ – karaspd Feb 8 '18 at 18:23

Not the answer you're looking for? Browse other questions tagged or ask your own question.