I'd like to compare logistic regression to classification trees. In a first step, I compared the theoretical framework of the two classifiers. In a second step, I compared the performance using a rather unbalanced data set containing two classes. I therefore compared confusion-matrices, balanced-accuracy, sensitivity and specificity. Moreover, I compared the ROC curves and derived therefrom the AUC values. Are there any other value adding measures to compare the classifiers? How would you compare the efficiency in terms of running time?


1 Answer 1


You seem to be using a mish-mash of methods. Focus on getting predicted risks and using proper accuracy scores such as the Brier score or logarithmic scoring rule (log likelihood; related to pseudo $R^2$). Things started going south when you chose to use classifiers rather than predictors. And note that regression trees are highly unstable when the sample size is $<100,000$ subjects for example. That's why people use bagging, boosting, and random forests instead of single trees.

Proper accuracy scores are not destroyed by imbalanced $Y$.

  • $\begingroup$ @ Frank Harrell Thank you for the insightful comment! So you would for example recommend to use Pseudo R^2 to compare the performance? This would be rather straightforward since I guess I can derive them directly from the confusion matrix. This would be a great measure since I have an economics background and R^2 is often used there. $\endgroup$ Dec 30, 2015 at 12:14
  • $\begingroup$ And if I may ask. Why do you call it a "proper" score? $\endgroup$ Dec 30, 2015 at 15:26
  • $\begingroup$ A proper accuracy scoring rule is one that gives the right rewards, i.e., it is optimized by a correct model. An improper accuracy score is optimized by choosing the wrong features and giving them the wrong weights. $\endgroup$ Dec 30, 2015 at 15:41
  • 1
    $\begingroup$ My course notes go into this a bit especially in the logistic regression chapter. See biostat.mc.vanderbilt.edu/rms and look under course materials. $\endgroup$ Dec 30, 2015 at 17:32
  • 1
    $\begingroup$ The concordance probability (AUROC) is less sensitive. Think of a pair of predicted probabilities 0.2 and 0.8 corresponding to a non-event and an event. Then think of a pair 0.1 and 0.9. Both get one point in the concordance calculation even though the second goes out on a limb. You can also think about this from the standpoint that the concordance probability is essentially the Wilcoxon-Mann-Whitney statistic for comparing two groups ($Y=0, Y=1$). Comparing 2 $c$-indexes is essentially the same as subtracting one Wilcoxon stat from another, which no one does. $\endgroup$ Dec 31, 2015 at 21:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.