How exactly to partition training-set for k-fold cross validation on multi-class dataset?

Cross validation is one of the most important tools because it gives us an honest assessment of the true accuracy of our system. In other words, the cross-validation process provides a much more accurate picture of our system's true accuracy.

If we have for example a dataset that contains one class (lets consider a face dataset):

In this case, we divide our dataset into k folds (or k portions). A common value of k is 10, so in that case, we would divide our dataset into 10 parts. We will run k rounds of cross validation. In each round, we use one the folds for validation, and the remaining folds for training. After training our classifier, we measure its accuracy on the validation data. Average the accuracy over k rounds to get a final cross-validation accuracy.

Prepare our dataset
Divide it into 10 folds.
for i=1:10 % ten times
fold(i) for testing
the remaining for training
end
Final accuracy = Average(Round1, Round2, ...., Round10).


Else if we have a dataset that contains multiple class (lets consider 3 classes : faces, airplanes and strawberry categories):

I don't know if my opinion is correct or not : each of the three categories is split into 10 folds. Does we measure the final accuracy as above? for example:

For the first round: take the first folds from the first, second and third categories and use them as testing and all the remaining folds (from all the categories) as training.

For the second round: take the second folds from the first, second and third categories and use them as testing and all the remaining folds (from all the categories) as training.

etc. until the round 10...


Does that is correct ? Does my opinion is correct? please I need your help and explanation.

Any help will be very appreciated.

• In your first example, if the dataset contains only one class, then what is the classifier supposed to classify? Commented Feb 26, 2014 at 18:01
• I am using the Support Vector Machine. But I have a data that contains multiple classes not only one. Commented Feb 26, 2014 at 18:09
• Okay, but what is your first example supposed to mean? You wrote "If we have for example a dataset that contains one class" -- what is there to classify then? Commented Feb 26, 2014 at 19:26

First you need to decide whether you need model/parameter selection, or just model. Once your model is fixed, bootstrap seems make more sense to determine how your modeling procedure performs.

If you are implementing cross validation on multiple dataset, just randomly partition the data without considering their labels. It is possible sometimes that one label in test data set does not even gets trained, and it counts into the validation error. Usually a 10-fold cross-validation is highly recommended to repeat 50-100 times for stability.

You may try to avoid class imbalance issue (thus indirectly reduce the odds of the excluded label event mentioned above), but if your data really suffers from this problem, there are several re-sampling strategies in my previous answer in this post.

• you are very welcome :) Commented Mar 5, 2014 at 20:41
• Dear lennon, if we consider that we have a dataset that contains 400 images of the same class (lets consider a face database). If we use the 10-cross validation. Therefore the data is split into 10 chunks. And each chunk contains 40 images split into 90% (36 images) training and 10% (4 images) as testing. If I want to also use the negative training and testing . How to take the number of negative training and testing ? Commented Mar 6, 2014 at 13:55
• you can choose another 400 images of non-face. And split all the 800 to 10 chunks. I think the 10 chunks should be enough for the cross validation, and you don't need to split each chunk again. Commented Mar 6, 2014 at 13:57
• Ah no but I want to train my classifier on positive and negative training, then classify the positive and negative testing. That is why I need to take a fix number of negative training and testing in a way to get always a balanced dataa Commented Mar 6, 2014 at 14:04
• Do you think that it is good to take 36 negative training and testing ? In this case, we are taking an equal number of positive and negative training. Commented Mar 6, 2014 at 14:08

I think you would generally not want to incorporate the known classifications into the selection of training and test samples. If you do that, the proportion of each class in the testing sample will always be the same as the proportions in the sample used to train the machine you are, but you actually want sampling variation between training and test to see how the classifier performs in real situations like that. Basically, just randomly partition your data set into 10 folds ignoring the classification variable.

• Firstly thank you for your answer. But unfortunately, I think that you didn't understand well my question :) Commented Feb 26, 2014 at 18:30
• @Christina: vafisher is correct: in general you partition your data independently of its label (classification), so that it might add some variability to your calculations, reflecting possible future scenarios. The only reason I'd see for stratifying your data in this way is if the size of the classes is very unequal. You may also want to look at bootstrap validation. Commented Feb 26, 2014 at 18:59