Consider that I use k-fold cross validation and select the best model based on the smallest mean error or using some heuristic to choose the best model. After I chose the best model, if I use the full training data to train this model, would it be guaranteed to overcome overfitting (or less overfitting) in deep neural networks?

Is it a good idea to train with the full training dataset after k-fold cross-validation, or again do another set of cross validation (split training dataset to training and validation) and then train the best model on the part of training data?

I read other questions regarding k-fold cross validation but none of them fully convinced me that when selected deep network model trained on whole training dataset will not overfit. As deep neural network, I mean the number of parameters to train P are far more than number of full training samples N (P>>N).

  • $\begingroup$ I have already read that, my question is mostly if you have complex model with lots of parameter P>>N, does it still hold to train with the full dataset after k-fold cross validation? (Is it going to hurt the performance or improve it) $\endgroup$
    – karaspd
    Feb 8, 2018 at 18:25

1 Answer 1


Cross validation would never overcome over-fitting issue, instead, CV can only control it.

The purpose of cross validation in neural network:

  1. helping choosing optimal number of epochs (controlling over training weights in neural network )
  2. robust metrics to measure the performance under the current parameter settings

It is also a good practise to train your model using full data with chosen epochs (by cross validation). Cross validation will always leaves a proportion of data aside for prediction (hence model metrics), however, training model will full data use all info without leaving any info on the table.

There is an idea to avoid training on full data again: with cross validation, you will have 5 models (say you have done 5-fold cross validation). And your prediction for new data would be the average of 5 model predictions. This is so called bagging in machine learning.

There is a good discussion on cross validation, and hopefully this is useful:

Training with the full dataset after cross-validation?

  • $\begingroup$ When you want to train on big networks with more parameters than your dataset, your chance of overfitting is higher if you use all the training data. Consider that I used k-fold cross validation, and I chose my network (parameters, architecture, number of epochs, ...), later for testing a model with test dataset, one way is to train the selected network with full training data. My assumption is that I can get overfitted by using the full training data in this complex model. $\endgroup$
    – karaspd
    Feb 8, 2018 at 18:23

Not the answer you're looking for? Browse other questions tagged or ask your own question.