Suppose we have only four observations in a dataset. Let's called them a,b,c and d.

If we perform a cross-validation in a k-fold, with k=2, we would get the following :

We get two groups of data, (a,b) and (c,d). We first learn on (a,b) then validate our machine learning model on (c,d). Then we learn on (c,d) and check with (a,b).

Is this really a complete 2-fold procedure?

Because, if we shuffle the data, we could get two other groups, let's say (a,c) and (b,d).

So this time, we would need to do another 2-fold cross-validation.

So my first question is, do we really need to perform multiple k-fold cross-validation, with shuffeling the data at every stage, in order to get a really good estimate of he performances of our model?

With k=number of observation, so a leave-one-out procedure, of course there is no need to do so, since shuffeling the data will give use the same groups.

Finally, if the answer of the abose question is yes, what is the advantage of (multiple) croos-validation compared to just multiple train-test splitting?

For example, we could shuffle the dataset, then take the first 80% of the observations to train and the last 20% to test. And do this again multiple times.

Am I totally wrong or is a single k-fold not enough to assess the performance of a model? And if so, what's the difference between it and doing multiple train-test?

Thanks guys


Your original design is in fact a 2-fold cross validation.

As to why you do or do not need to repeat k-fold cross validation, I believe your question is a repeat of this one: How many times should we repeat a K-fold CV?

  • $\begingroup$ Yes it's a 2-fold cross validation procedure. It's a approximated/non exhaustive version of a leave 2 out procedure (in this case because we have only 4 obs), which is a exhaustive CV method. But, with k-fold CV, if we do again the same procedure, we could get (slightly) different results, because it's only an approximation of a complete cross validation. My question is (I don't really know if this is really clear...) : Is there an advantage of doing a k-fold cross validation (or even several runs of it) against doing multiple runs of Holdout procedure (so with different train/test ensembles)? $\endgroup$ – hellowolrd Oct 9 '19 at 13:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.