Confusion with k-fold cross validation

I did k-fold CV on some data, but there is something which is confusing me which I don't really understand. I read in a book that for e.g. 3-fold cv I have to do 3 iterations so I will get 3 $MSE$'s. My mentor told me I can iterate many times and get many $MSE$'s. I briefly describe here how I understand it and how he told me I should do it:

For 3 iterations I shuffle the data and I split the data into e.g. 3-folds.

1. In the test fold I calculate $MSE_1$.
2. Again I shuffle the data and calculate $MSE_2$ on the test data.
3. Again I shuffle the data and calculate $MSE_3$ on the test data.

I calculate the CV error by averaging the $MSE_1, MSE_2, MSE_3$.

How my mentor told me I should do CV:

For $n$ iterations e.g. $n=100$

1. I shuffle the data and I split the data into 3 folds and In the test fold I calculate $MSE_1$.
2. Again I shuffle the data and calculate $MSE_2$ on the test data.
3. ...

100.. Again I shuffle the data and calculate $MSE_{100}$ on the test data.

I calculate the CV error by adding $MSE_1,MSE_2,... MSE_n$ and dividing them by the number of folds,3.

I would very much appreciate any clarification.

• You are just doing well, except that you must shuffle the data once, not three times. – pythinker May 25 '18 at 18:40

I think the confusion arises from the term "iterations".

Here's the process in the terminology I'm familiar with:

3-fold CV:

1. shuffle data
2. split into k = 3 segments (aka folds)
3. for each fold: calculate surrogate model without the fold in question and test the cases of the left-out fold.
4. after all k = 3 folds are done, calculate MSE as average squared error over all cases (= of all 3 folds).

Iterated k-fold CV (aka repeated k-fold CV) repeats/iterates that whole process 1. - 4. e.g. 100 times. You then have 100 MSEs, each of them calculated on all cases. (Only difference is that the surrogate models were trained with slightly different training sets).

Unfortunately, there's a whole lot of confusion about terminology for various flavors of cross validation. For the moment, I recommend to always describe in detail what you did.

I've met your procedures that shuffle freshly for each surrogate model under the name of set validation. In your case with 1/3 of all cases set aside for testing, and yes, also that procedure can be iterated or repeated - whether you do 3 or 100 repetitions is up to you, but there's no connection (unlike in k-fold CV) between the fraction of cases reserved for testing and the number of repetitions (aka iterations).

• ok now I understand it, I actually thought that the number of repetitions(eka iterations) is connected with the number of folds. I have still a small question, after I get the 100 MSEs, and I want to calculate the CV-error should then I sum all the 100 MSEs and divide them by 3 ? Based on this formula : $CV_{(k)}=\frac{1}{k}\sum_{i=1}^{k}MSE_i$ where my number of folds=3 – Ville May 25 '18 at 21:34
• Usually not. The usual way of calculating MSE is the average over all cases (i.e. if your folds have slightly varying numbers of cases e.g. due to rounding, you'd need to weight the fold-MSEs). – cbeleites unhappy with SX May 25 '18 at 21:41
• "Unfortunately, there's a whole lot of confusion about terminology for various flavors of cross validation." True story. (+1 obviously) – usεr11852 May 25 '18 at 22:46