1
$\begingroup$

What is the difference? How can I know when to use each one? For example if I have a dataset of 21 patients and I want to identify each of them by their features; which method do I have to use?

Thank you

$\endgroup$
1
$\begingroup$

Such experimental setups (cross validation, bootstrap or the repeated training test splits) are a way to gather multiple performance metrics when you want to compare algorithms with statistical tests.

This means that you already need to have the correct class labels of all 21 records. You then selectively ignore a few labels, predict them, compare predictions with the true labels, ignore some other labels etc.

If your main goal is to predict the particular data-set that you already have, you don't need such setups. If however you want to use the 21 records to select and train an algorithm that can extend to unseen data, you can use leave one out cross validation with such a tiny data-set.

$\endgroup$
  • $\begingroup$ So there is no advantage between stratified and leave-one-out? For example with one of them I need to balance data meanwhile with the other one no. $\endgroup$ – Aizzaac Feb 5 '17 at 20:43
  • $\begingroup$ With such low numbers you can't very well stratify. With high numbers, leave one out is very costly. $\endgroup$ – David Ernst Feb 6 '17 at 2:51
0
$\begingroup$

They can be quite important and are different, here is an example for each:

Suppose you want to learn to recognize handwritten letters of people. Suppose you have 100000 samples from 10 humans, 10000 for each subject, taken from their copying part of 'The cat in the hat'.

  • n-subjects out: If you just mix everything and split it equally, your model will over-fit for those particular persons. Every sample in the validation set will be closer to some in the test set - the difference between the model trying to distinguish e.g. my letter 'a' after it has seen me writing vs. it has seen a bunch of other people than are not myself. Iff we want to train for any random person, it makes sense to leave the samples of say one person completely out. That is each person goes into train or validation as a whole

  • stratified cross validation: The samples for each person will belong to one of the 26 classes-alphabet letters. If you feed the samples as is, your model will learn that the frequency of each letter is that of 'the cat in the hat'. This means that probably the letters 'c' and 'h' will be over-represented. What one could do is mark each sample with its label and make sure that the batches provided to the model have the expected frequencies, or depending on the application that each class is equally represented. This decision can have great importance on the model's performance, especially when there is class imbalance. Stratified cross validation can be the solution to some of these problems as it equally represents each class. It also makes sure that batches are more representable.

For the samples of each person you just mix everything and split it equally, your model will learn that the frequencies of each letter are more or less the same, but not exactly the same.

In your case, the question is are you going for classification vs authentication. Do you want to separate between your subjects, and are sure that the sample is from one of them, or separate between them, but mainly search if the subject is someone else entirely? These are two separate problems, both important. For authentication, leave one subject out should be used.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.