# Can K-fold cross validation cause overfitting?

I am learning $$k$$-fold cross validation. Since each fold will be used to train the model (in $$k$$ iterations), won't that cause overfitting?

• Please keep in mind that training restarts after every iteration, with the model "forgetting everything it knew about the data" – David Jul 9 '19 at 7:19
• – Franck Dernoncourt Jul 19 '20 at 23:05
• "No. Each fold is used to train a new model from scratch, predict the accuracy, and then the model is discarded. You don't use any of the models trained during CV." But when ı used k fold cv, ı see that accuracy increasing each fold, smallest one is initial fold, is it just a coincidicence? – zafer kovancı Mar 11 at 11:27

K-fold cross validation is a standard technique to detect overfitting. It cannot "cause" overfitting in the sense of causality.

However, there is no guarantee that k-fold cross-validation removes overfitting. People are using it as a magic cure for overfitting, but it isn't. It may not be enough.

The proper way to apply cross-validation is as a method to detect overfitting. If you do CV, and if there is a big difference between the test and the training error then you know you are overfitting and need to get more diverse data or choose simpler models and stronger regularization. The contrary does not hold: no big difference between test and train error does not mean you haven't been overfitting.

It's not a magic cure, but the best method to detect overfitting we have (when used right).

Some examples when cross-validation can fail:

• data is ordered, and not shuffled prior to splitting
• unbalanced data (try stratified cross-validation)
• duplicates in different folds
• natural groups (e.g., data from the same user) shuffled into multiple folds

There are other cases where it cannot detect information leakage and overtitting even when used perfectly right. For example when analyzing time series, people like to standardize the data, split it into past and future data, then train a model to predict the future development of these stocks. The subtle information leakage was in the preprocessing: standardization prior to temporal splitting leaks information about the average of the remainder. Similar leaks can occur in other preprocessing. In outlier detection, if you scale the data to 0:1, a model can learn that values close to 0 and 1 are the most extreme values you can observe etc.

Since each fold will be used to train the model (in  iterations), won't that cause overfitting?

No. Each fold is used to train a new model from scratch, predict the accuracy, and then the model is discarded. You don't use any of the models trained during CV.

You use validation (such as CV) for two purposes:

1. Estimate how good your model will (hopefully) work in practise when you deploy it, without risking a real A-B-test in production yet. You only want to go live with models that are expected to work better than you current approach, or this may cost your company millions.
2. Find the "best" parameters for train your final model (which you want to train on the entire training data). Tuning hyperparameters is when you have a high risk of overfitting if you are not careful.

CV is not a way of "training" a model by feeding 10 batches of data.

On the contrary, cross-validation is a good way to combat overfitting!

## Why $$k$$-fold CV?

Suppose you have a model and you want an estimate of its out-of-sample performance...

You could assess the prediction error on the same data used to fit the model (i.e. the training error), but this is obviously not a good indicator of out-of-sample performance. If the model is indeed overfitting, it will perform poorly on new observations, but you will still observe a low training error.

Alternatively, you could split your data into two part (train/test) and only use the train set to fit the model. The rest of the data, never seen by the model in any way, is then used to get an estimate of the out-of-sample performance. Great! But what if we had used a different split? As it turns out the variance between results obtained from different splits can be quite large... so large in fact, that data splitting is only reliable for really large $$n$$.

This is what $$k$$-fold CV attempts to tackle, by doing the following repeatedly:

• Fit your model with $$n - \frac{n}{k}$$ observations;
• Observe its performance on the remaining $$\frac{n}{k}$$ observations, which were not used to fit your model.

You repeat this process $$k$$ times, each time leaving out the next $$\frac{n}{k}$$ observations for testing, until all observations have been used once as a test set. You then sum the errors on the test set of each fold (or compute a weighted average), and you have an estimate of out-of-sample performance that is less sensitive to the particular splits used, because there are now $$k$$ of them.$$^\dagger$$

## Can this cause overfitting?

Since each fold will be used to train the model (in $$k$$ iterations), won't that cause overfitting?
$$\dagger$$: Even better, if you can afford it computationally, is to repeat $$k$$-fold CV multiple times.