I have a set of EEG data from 80 sick patients, and I know the outcomes of 6 of these patients (i.e. if they survived or died). I have discovered multiple measures that can be extracted from the EEG data that can be used to classify these 6 patients.

I would like to test how effective each measure is in predicting the outcome of the other 74 patients (the outcomes are known by the research group, but not me) and to be able to compare these methods of prediction.

I understand that the patients have to be split into multiple groups to deal with the problem of multiple hypothesis testing, but how can I go about this? I know that I can use a Support Vector Classifier to classify the data, but how does that account for multiple hypothesis testing? Does anyone have some resources or literature they can recommend on this topic?

  • $\begingroup$ Does "measure" mean "feature"? $\endgroup$ – Yurii Sep 30 '15 at 14:38
  • $\begingroup$ Yes, I believe so! For example, I'd like to classify patients by the entropy of their EEG signals, and separately also classify them according to the spectral power in a frequency band (for example) and then compare how well each feature (?) predicted outcomes. $\endgroup$ – Beena61 Sep 30 '15 at 20:22
  • $\begingroup$ Well, then I don't see a problem. You just train your models on your whole data and then compare their performance. You don't need to split patients into groups unless I misunderstood a question. $\endgroup$ – Yurii Oct 1 '15 at 8:48
  • $\begingroup$ Thanks for your input! But, isn't there a problem with multiple testing? For example, if I test the performance of 15 features, then the likelihood of one feature being a good predictor is considerable, no? Also, I would only have one value for each prediction (so, like 70% of patients are classified correctly using a particular feature) how would I statistically compare this? $\endgroup$ – Beena61 Oct 1 '15 at 17:18

One possible approach could be:

  1. Perform a Bootstrap estimation of classification accuracy (Duda, Hart, Stork, 2nd edition, pg 485)
  2. Multiple hypothesis testing (for ex., Bonferroni), using a parametric or non-parametric (for ex., bootstrap) approach. In this step, you should assume a target accuracy, say, 80%.
  • $\begingroup$ Welcome to the site, @BernardoAflalo. Note that CV is a pure Q&A site, not a discussion forum. Since you're new here, why not take our tour which has information for new users. $\endgroup$ – gung - Reinstate Monica Jan 5 '16 at 12:50

What I understood from the problem is that using 6 patients you discovered let say K ways/algorithms of determining the patient outcome. The "ways" is your "measures": if the entropy of the EEG is above x, or if the power in the 2-3Hz range is below y, and so on.

Now using the 74 remaining patients you want to fiure out which of the ways is the best one.

If YOU came up with the K ways then YOU DO NOT HAVE TO MAKE ANY STATISTICAL TESTS. If they are your algorithms and you want to to find out which one is the best, just measure the accuracy (or other quality metric you are interested) in the 74 patients and pick the best. let say algorithm A has an accuracy 81% and B has an accuracy of 80% but the difference is not significant - you still will pick algorithm A as your best bet!!!

You have to perform a statistical test only if some of the algorithms you are comparing are not yours, and you want to to show that one of your algorithms is "better" then these others, and should be selected as the new algorithm to be used in practice, or to be considered the "state of the art" and so on.

If it indeed the case that some of the algorithm are not yours, and you must perform a statistical test, then leave a comment and I will edit this answer to include the test - but it is not simple. You will have to do pairwise McNemar tests (which only work with binary outcomes) and perform some p-value adjustment.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.