5
$\begingroup$

Cross validation is one of the most important tools because it gives us an honest assessment of the true accuracy of our system. In other words, the cross-validation process provides a much more accurate picture of our system's true accuracy.

If we have for example a dataset that contains one class (lets consider a face dataset):

In this case, we divide our dataset into k folds (or k portions). A common value of k is 10, so in that case, we would divide our dataset into 10 parts. We will run k rounds of cross validation. In each round, we use one the folds for validation, and the remaining folds for training. After training our classifier, we measure its accuracy on the validation data. Average the accuracy over k rounds to get a final cross-validation accuracy.

Prepare our dataset
Divide it into 10 folds.
for i=1:10 % ten times
fold(i) for testing
the remaining for training 
end
Final accuracy = Average(Round1, Round2, ...., Round10). 

Else if we have a dataset that contains multiple class (lets consider 3 classes : faces, airplanes and strawberry categories):

I don't know if my opinion is correct or not : each of the three categories is split into 10 folds. Does we measure the final accuracy as above? for example:

For the first round: take the first folds from the first, second and third categories and use them as testing and all the remaining folds (from all the categories) as training.  

For the second round: take the second folds from the first, second and third categories and use them as testing and all the remaining folds (from all the categories) as training.

etc. until the round 10...

Does that is correct ? Does my opinion is correct? please I need your help and explanation.

Any help will be very appreciated.

$\endgroup$
  • $\begingroup$ In your first example, if the dataset contains only one class, then what is the classifier supposed to classify? $\endgroup$ – amoeba says Reinstate Monica Feb 26 '14 at 18:01
  • $\begingroup$ I am using the Support Vector Machine. But I have a data that contains multiple classes not only one. $\endgroup$ – Christina Feb 26 '14 at 18:09
  • $\begingroup$ Okay, but what is your first example supposed to mean? You wrote "If we have for example a dataset that contains one class" -- what is there to classify then? $\endgroup$ – amoeba says Reinstate Monica Feb 26 '14 at 19:26
  • $\begingroup$ jajaja yeah you are right :D $\endgroup$ – Christina Feb 28 '14 at 12:48
5
$\begingroup$

First you need to decide whether you need model/parameter selection, or just model. Once your model is fixed, bootstrap seems make more sense to determine how your modeling procedure performs.

If you are implementing cross validation on multiple dataset, just randomly partition the data without considering their labels. It is possible sometimes that one label in test data set does not even gets trained, and it counts into the validation error. Usually a 10-fold cross-validation is highly recommended to repeat 50-100 times for stability.

You may try to avoid class imbalance issue (thus indirectly reduce the odds of the excluded label event mentioned above), but if your data really suffers from this problem, there are several re-sampling strategies in my previous answer in this post.

$\endgroup$
  • $\begingroup$ you are very welcome :) $\endgroup$ – lennon310 Mar 5 '14 at 20:41
  • $\begingroup$ Dear lennon, if we consider that we have a dataset that contains 400 images of the same class (lets consider a face database). If we use the 10-cross validation. Therefore the data is split into 10 chunks. And each chunk contains 40 images split into 90% (36 images) training and 10% (4 images) as testing. If I want to also use the negative training and testing . How to take the number of negative training and testing ? $\endgroup$ – Christina Mar 6 '14 at 13:55
  • $\begingroup$ you can choose another 400 images of non-face. And split all the 800 to 10 chunks. I think the 10 chunks should be enough for the cross validation, and you don't need to split each chunk again. $\endgroup$ – lennon310 Mar 6 '14 at 13:57
  • $\begingroup$ Ah no but I want to train my classifier on positive and negative training, then classify the positive and negative testing. That is why I need to take a fix number of negative training and testing in a way to get always a balanced dataa $\endgroup$ – Christina Mar 6 '14 at 14:04
  • $\begingroup$ Do you think that it is good to take 36 negative training and testing ? In this case, we are taking an equal number of positive and negative training. $\endgroup$ – Christina Mar 6 '14 at 14:08
4
$\begingroup$

I think you would generally not want to incorporate the known classifications into the selection of training and test samples. If you do that, the proportion of each class in the testing sample will always be the same as the proportions in the sample used to train the machine you are, but you actually want sampling variation between training and test to see how the classifier performs in real situations like that. Basically, just randomly partition your data set into 10 folds ignoring the classification variable.

$\endgroup$
  • $\begingroup$ Firstly thank you for your answer. But unfortunately, I think that you didn't understand well my question :) $\endgroup$ – Christina Feb 26 '14 at 18:30
  • 3
    $\begingroup$ @Christina: vafisher is correct: in general you partition your data independently of its label (classification), so that it might add some variability to your calculations, reflecting possible future scenarios. The only reason I'd see for stratifying your data in this way is if the size of the classes is very unequal. You may also want to look at bootstrap validation. $\endgroup$ – Wayne Feb 26 '14 at 18:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.