Data split into training and test

I am implementing an EEG classifier with 15 subjects (patients), specifically a support vector machine classifier.

I randomly choose the training and testing sets, but I was faced by a question "how did you choose subjects in each set?". I looked for the response but I couldn't find a good one (cross validation wouldn't be the best solution in my case).

• Why wouldn't cross validation be if not the best at least a better solution than a random one time split of the data? Just splitting once is a highly inefficient use of the data. And why couldn't you answer the question? If you assigned randomly isn't that the answer to the question? – Erik Nov 6 '13 at 13:52
• I did not suggest to use it. I was wondering if you think about it when you say CV is not feasible because your data set is too large. As I cannot really imagine a scenario where building a model once (twice in your case) is fast enough but doing it ten times (10-fold CV) suddenly becomes impossible. – Erik Nov 6 '13 at 14:27
• Suppose you have $n$ observations. LOO-CV takes 1 observation out of your data set and uses all the others to train your model. Measures how well it predicted on the left out(LO) observation. Then it takes another observation wich is not the previous one and uses all the others to train your model. Measures how well this model predicts on this LO observation. Does this $n$ times. The error would be the average of all these measures. 10-fold separates your observations into 10 groups $|g1|+|g2|+...+|g10|=n$. Uses the groups 1 to 9 to train, tests on $g10$. Does this 10 times. A shorter process. – JEquihua Nov 6 '13 at 18:52
• As you say that you have 15 subjects, but a large data set: do you have many repeated measurements per subject/patient? In that case, the question may have been aimed at whether you did split (randomly) patients [correct] or measurements [optimistically biased]. – cbeleites supports Monica Nov 7 '13 at 17:02
• i have an 8 hour record for each patient and i have to dived data into 30sec epochs, so the result is relatively big. i'm trying to implement the LOO procedure now i will tell you about what i found – Tarek Nov 7 '13 at 20:00

• IMHO leave-one-out is certainly not the best way to validate such data. Iterated/repeated $k$-fold or leave-$n$-out (with $k$ < no of cases / $n$ > 1) is certainly better, as is out-of-bootstrap. Nevertheless, LOO is a sensible starting point for someone not familiar with resampling validation. – cbeleites supports Monica Nov 7 '13 at 16:49