0
$\begingroup$

I have a training set of 50 subjects with about 550-600 measurements each. One measurement consists of 24 features and one class label (1 or 0). So my data looks like this (simplified):

Subject  F1  F2  F3  ...  Class
1        1   3   2   ...  1
1        1   4   7   ...  0
...
2        2   3   2   ...  1
2        1   1   1   ...  1
...

I want to train my classification model (artificial neural network) with all measurements of all 50 subjects. Now I want to know if I can just use leave-one-out-cross-validation on the subject level (training with all measurements of all but one subject and then testing on the measurements of the remaining subject) or is validation on measurement level also needed (k-fold cross-validation within the measurements of each single subject)?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

To answer this question, you should ask what you want your classifier to be able to do.

If I understand correctly, when you train your classifier on the 'measurement level' you would 'teach' the classifier to distinguish (classify) a set of features of a single subject. This is different from training it to classify any set of features independent of what subject it came from.

Assuming you want your classifier to be able to classify any set of features, independent of what subject those features came from, you should not do any cross-validation on the 'measurement level'.

In this same setting. I do not exactly understand why you would consider all the measurements of a single subject as 'one' (In the context of the leave-one-out cross-validation). Did you consider a single set of features (a single measurement), independent of what subject it came from, as being 'one' thing?

[EDIT]: I just discussed this problem with a colleague. Doing cross-validation on the entire dataset (independent of subject) will leak information.

If part of the measurements of a subject are included in the training set and if the other measurements of that same subject are in the testing set, this will overestimate the performance! Although, if you do not get any measurements from new subjects, this is not a problem.

In the case you do get measurements from new subjects (more likely, I think), then it is a good idea to include all the measurements of a subject in the testing set, if that subject was selected to be in that set.

Improve this answer
$\endgroup$
6
  • $\begingroup$ I want a general classifier to classify any set of features, independent of it's subject. I considered all measurements of one subject as 'one', because the measurements are some what biased by their subject. When considering a single measurement as being 'one' thing, you mean that I should put all measurements together (not considering their subject at all)? So this way a simple k-fold cross-validation would do the job? $\endgroup$
    – Robomatix
    Commented May 20, 2016 at 7:19
  • $\begingroup$ Cross-validation is a simulation in which you want to simulate a real world situation. So if you are interested in classifying a single measurement (independent of the subject) at the end, then the answer is yes. $\endgroup$
    – Rick
    Commented May 20, 2016 at 7:26
  • $\begingroup$ [cont'd] yes, you can do a 'simple' k-fold cross-validation on the entire dataset (put all measurements together). If you want the Neural Network to classify a single measurement of a distinct subject, then you can consider it as 50 separate (one for every subject) datasets, maybe, depending on what you want. $\endgroup$
    – Rick
    Commented May 20, 2016 at 7:33
  • $\begingroup$ The classifier will almost exclusively classify measurements of new subjects. So following your EDIT, after all, you would suggest validation on subject level. Therefore leave-one-out cross-validation (on subject level) would be a good choice again, right? In fact the class distribution of my training measurements is almost equal among all 50 subjects. So this way I would get a leave-one-out cross-validation that is somewhat stratified. $\endgroup$
    – Robomatix
    Commented May 20, 2016 at 8:55
  • $\begingroup$ Yes, then doing it on the subject level would be a better representation of the real world application. If you would not do it on subject level you will have a biased model and overestimate the generalisability. $\endgroup$
    – Rick
    Commented May 20, 2016 at 9:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.