# How to compare classification methods in terms of performance?

I'd like to compare logistic regression to classification trees. In a first step, I compared the theoretical framework of the two classifiers. In a second step, I compared the performance using a rather unbalanced data set containing two classes. I therefore compared confusion-matrices, balanced-accuracy, sensitivity and specificity. Moreover, I compared the ROC curves and derived therefrom the AUC values. Are there any other value adding measures to compare the classifiers? How would you compare the efficiency in terms of running time?

You seem to be using a mish-mash of methods. Focus on getting predicted risks and using proper accuracy scores such as the Brier score or logarithmic scoring rule (log likelihood; related to pseudo $R^2$). Things started going south when you chose to use classifiers rather than predictors. And note that regression trees are highly unstable when the sample size is $<100,000$ subjects for example. That's why people use bagging, boosting, and random forests instead of single trees.
Proper accuracy scores are not destroyed by imbalanced $Y$.
• The concordance probability (AUROC) is less sensitive. Think of a pair of predicted probabilities 0.2 and 0.8 corresponding to a non-event and an event. Then think of a pair 0.1 and 0.9. Both get one point in the concordance calculation even though the second goes out on a limb. You can also think about this from the standpoint that the concordance probability is essentially the Wilcoxon-Mann-Whitney statistic for comparing two groups ($Y=0, Y=1$). Comparing 2 $c$-indexes is essentially the same as subtracting one Wilcoxon stat from another, which no one does. – Frank Harrell Dec 31 '15 at 21:06