3
$\begingroup$

I am implementing an EEG classifier with 15 subjects (patients), specifically a support vector machine classifier.

I randomly choose the training and testing sets, but I was faced by a question "how did you choose subjects in each set?". I looked for the response but I couldn't find a good one (cross validation wouldn't be the best solution in my case).

Could you please help me with this problem?

Improve this question
$\endgroup$
8
  • 1
    $\begingroup$ Why wouldn't cross validation be if not the best at least a better solution than a random one time split of the data? Just splitting once is a highly inefficient use of the data. And why couldn't you answer the question? If you assigned randomly isn't that the answer to the question? $\endgroup$
    – Erik
    Nov 6, 2013 at 13:52
  • 1
    $\begingroup$ I did not suggest to use it. I was wondering if you think about it when you say CV is not feasible because your data set is too large. As I cannot really imagine a scenario where building a model once (twice in your case) is fast enough but doing it ten times (10-fold CV) suddenly becomes impossible. $\endgroup$
    – Erik
    Nov 6, 2013 at 14:27
  • 2
    $\begingroup$ Suppose you have $n$ observations. LOO-CV takes 1 observation out of your data set and uses all the others to train your model. Measures how well it predicted on the left out(LO) observation. Then it takes another observation wich is not the previous one and uses all the others to train your model. Measures how well this model predicts on this LO observation. Does this $n$ times. The error would be the average of all these measures. 10-fold separates your observations into 10 groups $|g1|+|g2|+...+|g10|=n$. Uses the groups 1 to 9 to train, tests on $g10$. Does this 10 times. A shorter process. $\endgroup$
    – JEquihua
    Nov 6, 2013 at 18:52
  • 1
    $\begingroup$ As you say that you have 15 subjects, but a large data set: do you have many repeated measurements per subject/patient? In that case, the question may have been aimed at whether you did split (randomly) patients [correct] or measurements [optimistically biased]. $\endgroup$
    – cbeleites unhappy with SX
    Nov 7, 2013 at 17:02
  • 1
    $\begingroup$ i have an 8 hour record for each patient and i have to dived data into 30sec epochs, so the result is relatively big. i'm trying to implement the LOO procedure now i will tell you about what i found $\endgroup$
    – Tarek
    Nov 7, 2013 at 20:00

1 Answer 1

Reset to default
1
$\begingroup$

I am assuming you are seeking to classify the EEG data into one or more disease states e.g. seizure/non-seizure, pathological/non-pathological etc.

The best way to validate a classifier model for an application like this is to implement Leave One Out cross validation.

What I mean by this is to start with all data for patient 1 as the test set and all data for patients 2-15 as the training set and store the results. Next, set the data for patient 2 as the test set and the remainder as the training set. Do this for each patient's data in turn so that you have 15 classification results, one for each patient. The take the mean of these 15 values and you have an estimate for the classification performance of your classifier model on unseen data.

Improve this answer
$\endgroup$
2
  • 3
    $\begingroup$ IMHO leave-one-out is certainly not the best way to validate such data. Iterated/repeated $k$-fold or leave-$n$-out (with $k$ < no of cases / $n$ > 1) is certainly better, as is out-of-bootstrap. Nevertheless, LOO is a sensible starting point for someone not familiar with resampling validation. $\endgroup$
    – cbeleites unhappy with SX
    Nov 7, 2013 at 16:49
  • 2
    $\begingroup$ @cbeleites, agreed, but was trying to provide a solid starting point for a newcomer to the area $\endgroup$
    – BGreene
    Nov 7, 2013 at 17:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.