Classification and multiple hypothesis testing I have a set of EEG data from 80 sick patients, and I know the outcomes of 6 of these patients (i.e. if they survived or died). I have discovered multiple measures that can be extracted from the EEG data that can be used to classify these 6 patients. 
I would like to test how effective each measure is in predicting the outcome of the other 74 patients (the outcomes are known by the research group, but not me) and to be able to compare these methods of prediction. 
I understand that the patients have to be split into multiple groups to deal with the problem of multiple hypothesis testing, but how can I go about this? I know that I can use a Support Vector Classifier to classify the data, but how does that account for multiple hypothesis testing? Does anyone have some resources or literature they can recommend on this topic? 
 A: One possible approach could be:


*

*Perform a Bootstrap estimation of classification accuracy (Duda, Hart, Stork, 2nd edition, pg 485)

*Multiple hypothesis testing (for ex., Bonferroni), using a parametric or non-parametric (for ex., bootstrap) approach. In this step, you should assume a target accuracy, say, 80%.

A: What I understood from the problem is that using 6 patients you discovered let say K ways/algorithms of determining the patient outcome. The "ways" is your "measures": if the entropy of the EEG is above x, or if the power in the 2-3Hz range is below y, and so on.
Now using the 74 remaining patients you want to fiure out which of the ways is the best one. 
If YOU came up with the K ways then YOU DO NOT HAVE TO MAKE ANY STATISTICAL TESTS. If they are your algorithms and you want to to find out which one is the best, just measure the accuracy (or other quality metric you are interested) in the 74 patients and pick the best. let say algorithm A has an accuracy 81% and B has an accuracy of 80% but the difference is not significant - you still will pick algorithm A as your best bet!!!
You have to perform a statistical test only if some of the algorithms you are comparing are not yours, and you want to to show that one of your algorithms is "better" then these others, and should be selected as the new algorithm to be used in practice, or to be considered the "state of the art" and so on.
If it indeed the case that some of the algorithm are not yours, and you must perform a statistical test, then leave a comment and I will edit this answer to include the test - but it is not simple. You will have to do pairwise McNemar tests (which only work with binary outcomes)  and perform some p-value adjustment.
