Confusion with k-fold cross validation I did k-fold CV on some data, but there is something which is confusing me which I don't really understand. I read in a book that for e.g. 3-fold cv I have to do 3 iterations so I will get 3 $MSE$'s. My mentor told me I can iterate many times and get many $MSE$'s. I briefly describe here how I understand it and how he told me I should do it: 
For 3 iterations
I shuffle the data and I split the data into e.g. 3-folds. 


*

*In the test fold I calculate $MSE_1$.   

*Again I shuffle the data and calculate $MSE_2$ on the test data.

*Again I shuffle the data and calculate $MSE_3$ on the test data.


I calculate the CV error by averaging the $MSE_1, MSE_2, MSE_3$.
How my mentor told me I should do CV:
For $n$ iterations e.g. $n=100$


*

*I shuffle the data and I split the data into 3 folds and In the test fold I calculate $MSE_1$. 

*Again I shuffle the data and calculate $MSE_2$ on the test data.

*...


100.. Again I shuffle the data and calculate $MSE_{100}$ on the test data.
I calculate the CV error by adding $MSE_1,MSE_2,... MSE_n$ and dividing them by the number of folds,3.
I would very much appreciate any clarification. 
 A: I think the confusion arises from the term "iterations". 
Here's the process in the terminology I'm familiar with:
3-fold CV:


*

*shuffle data

*split into k = 3 segments (aka folds)

*for each fold: calculate surrogate model without the fold in question and test the cases of the left-out fold. 

*after all k = 3 folds are done, calculate MSE as average squared error over all cases (= of all 3 folds). 


Iterated k-fold CV (aka repeated k-fold CV) repeats/iterates that whole process 1. - 4. e.g. 100 times. You then have 100 MSEs, each of them calculated on all cases. (Only difference is that the surrogate models were trained with slightly different training sets). 

Unfortunately, there's a whole lot of confusion about terminology for various flavors of cross validation. For the moment, I recommend to always describe in detail what you did. 

I've met your procedures that shuffle freshly for each surrogate model under the name of set validation. In your case with 1/3 of all cases set aside for testing, and yes, also that procedure can be iterated or repeated - whether you do 3 or 100 repetitions is up to you, but there's no connection (unlike in k-fold CV) between the fraction of cases reserved for testing and the number of repetitions (aka iterations).
