5
$\begingroup$

I have developed a new predictor based on neural networks for a specific problem in bioinformatics. This predictor takes as inputs several features and returns a boolean target value. Additionally i have runned my dataset through other already published predictors (none of which based on neural networks). From all these results i have generated 9 contingency tables (one per predictor) based on the target value and the predictor response like the one below.

                  Target
              |    0     1 |
           ----------------|
Prediction  0 |  123    10 |
            1 |    5   171 |
            ----------------

Is there any way to compare these statistical tables in such a manner that i can state that my predictor is better or worse than any of the other predictors supported by a significant p-value? Note that i have the results table for all cases (Ei) in my dataset for all the predictors (Pj), like:

   P1 P2 P3 P4 P5 P6 P7 P8 P9
E1  0  0  1  0  1  0  0  1  1
E2  0  1  1  0  0  0  0  1  1
E3  0  1  0  0  0  0  0  0  0
$\endgroup$

1 Answer 1

4
$\begingroup$

I think it's important first to define what is important in this particular problem. Are you looking for best overall accuracy, specificity, sensitivity, precision, AUC, etc? I think if you know the measure you want to use then the results of repeated cross validation runs would provide you a sample of measures for each classifier, you could then use a simple ANOVA to determine if the means of the measure for each run were different between your classifier and the control classifiers. I'm guessing since you said this is a specific bioinformatics problem that you probably have a measure of classifier strength in mind, but if not I'd recommend just going with AUC as it's a little more fine grained than accuracy. I would point you towards

http://arion.csd.uwo.ca/faculty/ling/papers/ijcai03.pdf

If you were curious why I say that. Hope that helps.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.