How should I perform cross-validation of 10-fold?

Question

Suppose that the circles in red are negative samples and the circles in blue are positive samples and that the green boxes are the validation set and all the blank boxes are the training set. In addition, suppose I charge the following data:

The loaded data is sorting, first the positive samples and then the negative samples. If I now perform cross-validation, taking disjoint but consecutive folds (and with ordered positive samples and ordered negative samples):

If instead of performing the previous cross-validation, I do it in the following way, taking disjoint but not consecutive folds (and with ordered positive samples and ordered negative samples) which gives rise to random folds for both training and testing:

These are the first two forms of cross-validation that I think can be implemented.

Or, if I decide to load the data in the following way:

The loaded data is cluttered, that is, loaded randomly. If I now perform cross-validation, taking disjoint but consecutive folds (and with disordered positive samples and disordered negative samples) which still gives rise to random folds for both training and testing:

Finally, if I perform cross-validation by taking disjoint but not consecutive folds (and with positive samples disordered and disordered negative samples) which also results in random folds for both training and testing:

Of these 4 ways in which I think it can be implemented, what is the "correct" way to perform cross-validation? What is the advantage / disadvantage between one and the other? Is there any criteria to make this choice? Thanks in advance

Answer 1

Your procedures 2 - 4 are equivalent:

• Randomly selecting disjoint sets of n/k cases is equivalent to first randomly shuffling all cases and then selecting consecutive sets of n/k cases.
• After shuffling the order of cases, also randomly selecting n/k cases doesn't make the situation any more random (nor less).

As for selecting the folds in a systematic manner from sorted cases (which is a generalization of your case 1): this is sometimes done, but you should know why you do so if you do so E.g.

• sorting cases according to some characteristic, leaving out the lowest n/k cases and highest n/k cases does yield an estimate for slight extrapolation.
• whereas systematically taking every kth case into set 1, k+1th case into set 2, ... will give a situation where the k surrogate models and k test sets are as similar as possible for the given data. (This is called Venetian blinds)
• For one-class classifiers, one may even leave out all samples of a class and test whether they are predicted as unknown or are confused with one of the other classes.

An intermediate procedure having both systematic and random elements that is fequently used is randomly assigning folds separately for each class, aka stratified random sampling. This ensures that the relative freqency of the classes is constant across folds.

One way to do this is:

1. shuffle randomly all blue cases, shuffle randomly all red cases, and so on for further classes/groups/strata
2. put the shuffled cases behind each other => you'll have blue block, red block and so on.

3. assign folds: case x => fold x mod k

   case 1     => fold 1
   case 2     => fold 2
   ...
   case k     => fold 0 (or k)
   case k + 1 => fold 1
   ...
   

Answer 2

It's recommended that you always shuffle all the data you have and then divide the shuffled dataset into Training/Cross-Validation(Dev)/Test Sets. Doing so you can be sure that the data is distributed randomly between all these different sets.

The reason this is considered a good practice is that, the model doesn't develop bias for some examples and gets enough different examples of all varieties in the dataset.

Consider working on a cat classifier and collecting the dataset, say your dataset includes 100 images of cats and non-cats. It has first 40 images of white cats, next 20 of black ones, and rest of non-cats. When you simply divide the dataset into Training and Dev. set, say in ratio 70:30, you get 40 white cats, 20 black ones and only 10 negative examples in Training set, whereas Dev set all Negative examples. And you desire to have a fair distribution of varied examples in both the sets (for reasons described in the above para.), a simple way is just shuffle it and then take top 70 for Training set and rest for Dev set, or according to the ratio you decide for data distribution.

Now, as per the distribution you described above, it depends on how the data is distributed in the DataLoaded above. I'd prefer the random ones.