4
$\begingroup$

I have a set of EEG data from 80 sick patients, and I know the outcomes of 6 of these patients (i.e. if they survived or died). I have discovered multiple measures that can be extracted from the EEG data that can be used to classify these 6 patients.

I would like to test how effective each measure is in predicting the outcome of the other 74 patients (the outcomes are known by the research group, but not me) and to be able to compare these methods of prediction.

I understand that the patients have to be split into multiple groups to deal with the problem of multiple hypothesis testing, but how can I go about this? I know that I can use a Support Vector Classifier to classify the data, but how does that account for multiple hypothesis testing? Does anyone have some resources or literature they can recommend on this topic?

Improve this question
$\endgroup$
4
  • $\begingroup$ Does "measure" mean "feature"? $\endgroup$
    – Yurii
    Commented Sep 30, 2015 at 14:38
  • $\begingroup$ Yes, I believe so! For example, I'd like to classify patients by the entropy of their EEG signals, and separately also classify them according to the spectral power in a frequency band (for example) and then compare how well each feature (?) predicted outcomes. $\endgroup$
    – Beena61
    Commented Sep 30, 2015 at 20:22
  • $\begingroup$ Well, then I don't see a problem. You just train your models on your whole data and then compare their performance. You don't need to split patients into groups unless I misunderstood a question. $\endgroup$
    – Yurii
    Commented Oct 1, 2015 at 8:48
  • $\begingroup$ Thanks for your input! But, isn't there a problem with multiple testing? For example, if I test the performance of 15 features, then the likelihood of one feature being a good predictor is considerable, no? Also, I would only have one value for each prediction (so, like 70% of patients are classified correctly using a particular feature) how would I statistically compare this? $\endgroup$
    – Beena61
    Commented Oct 1, 2015 at 17:18

2 Answers 2

Reset to default
1
$\begingroup$

One possible approach could be:

  1. Perform a Bootstrap estimation of classification accuracy (Duda, Hart, Stork, 2nd edition, pg 485)
  2. Multiple hypothesis testing (for ex., Bonferroni), using a parametric or non-parametric (for ex., bootstrap) approach. In this step, you should assume a target accuracy, say, 80%.
Improve this answer
$\endgroup$
1
  • $\begingroup$ Welcome to the site, @BernardoAflalo. Note that CV is a pure Q&A site, not a discussion forum. Since you're new here, why not take our tour which has information for new users. $\endgroup$
    – gung - Reinstate Monica
    Commented Jan 5, 2016 at 12:50
1
$\begingroup$

What I understood from the problem is that using 6 patients you discovered let say K ways/algorithms of determining the patient outcome. The "ways" is your "measures": if the entropy of the EEG is above x, or if the power in the 2-3Hz range is below y, and so on.

Now using the 74 remaining patients you want to fiure out which of the ways is the best one.

If YOU came up with the K ways then YOU DO NOT HAVE TO MAKE ANY STATISTICAL TESTS. If they are your algorithms and you want to to find out which one is the best, just measure the accuracy (or other quality metric you are interested) in the 74 patients and pick the best. let say algorithm A has an accuracy 81% and B has an accuracy of 80% but the difference is not significant - you still will pick algorithm A as your best bet!!!

You have to perform a statistical test only if some of the algorithms you are comparing are not yours, and you want to to show that one of your algorithms is "better" then these others, and should be selected as the new algorithm to be used in practice, or to be considered the "state of the art" and so on.

If it indeed the case that some of the algorithm are not yours, and you must perform a statistical test, then leave a comment and I will edit this answer to include the test - but it is not simple. You will have to do pairwise McNemar tests (which only work with binary outcomes) and perform some p-value adjustment.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.