1
$\begingroup$

Suppose I have N methods and M benchmarks. I have an AUC statistic (and some other similar statistics) for each combination of method with benchmark. What test should I use to test if one method is better than the rest? I have seen some authors do pairwise comparisons using a one-sided Wilcoxon signed-rank test but I would prefer to test all methods at once. In any case I'm not sure the assumptions for the one-sided Wilcoxon signed-rank test hold. If the average AUC for each benchmark varies widely can you say the samples are from the same population? Also I'm not sure the distribution of the AUCs is symmetric around the median. Any advice would be welcome.

$\endgroup$

1 Answer 1

0
$\begingroup$

If the M benchmarks are supposed to yield score identically distributed score estimates (e.g. cross-validation folds) then maybe you can estimate confidence intervals for the mean AUC score for each method by bootstrapping on the M benchmarks of that method and then compare methods by considering non-overlapping confidence intervals. As bootstrapped confidence interval is a non-parametric method, you do not make any assumption on the symmetry of AUCs around the median.

$\endgroup$
2
  • $\begingroup$ Thanks for the answer but I'm afraid the AUC statistics are expected to vary depending on the benchmark. Sorry I didn't make that clear. $\endgroup$
    – Epimetheus
    Oct 10, 2013 at 12:19
  • 2
    $\begingroup$ Not that the concordance probability (AUROC) is often not sensitive enough to compare methods. It is best for describing pure predictive discrimination for a single model. It is better to use the gold standard (deviance-based methods such as generalized $R^2$ or likelihood ratio $\chi^2$) for comparing two or more predictive methods. $\endgroup$ May 15, 2014 at 12:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.