How does k-fold cross validation overcome overfitting in deep neural networks? [duplicate]

Question

Consider that I use k-fold cross validation and select the best model based on the smallest mean error or using some heuristic to choose the best model. After I chose the best model, if I use the full training data to train this model, would it be guaranteed to overcome overfitting (or less overfitting) in deep neural networks?

Is it a good idea to train with the full training dataset after k-fold cross-validation, or again do another set of cross validation (split training dataset to training and validation) and then train the best model on the part of training data?

I read other questions regarding k-fold cross validation but none of them fully convinced me that when selected deep network model trained on whole training dataset will not overfit. As deep neural network, I mean the number of parameters to train P are far more than number of full training samples N (P>>N).

Answer 1

Cross validation would never overcome over-fitting issue, instead, CV can only control it.

The purpose of cross validation in neural network:

1. helping choosing optimal number of epochs (controlling over training weights in neural network )
2. robust metrics to measure the performance under the current parameter settings

It is also a good practise to train your model using full data with chosen epochs (by cross validation). Cross validation will always leaves a proportion of data aside for prediction (hence model metrics), however, training model will full data use all info without leaving any info on the table.

There is an idea to avoid training on full data again: with cross validation, you will have 5 models (say you have done 5-fold cross validation). And your prediction for new data would be the average of 5 model predictions. This is so called bagging in machine learning.

There is a good discussion on cross validation, and hopefully this is useful:

Training with the full dataset after cross-validation?