0
$\begingroup$

I'd like to compare logistic regression to classification trees. In a first step, I compared the theoretical framework of the two classifiers. In a second step, I compared the performance using a rather unbalanced data set containing two classes. I therefore compared confusion-matrices, balanced-accuracy, sensitivity and specificity. Moreover, I compared the ROC curves and derived therefrom the AUC values. Are there any other value adding measures to compare the classifiers? How would you compare the efficiency in terms of running time?

$\endgroup$
1
$\begingroup$

You seem to be using a mish-mash of methods. Focus on getting predicted risks and using proper accuracy scores such as the Brier score or logarithmic scoring rule (log likelihood; related to pseudo $R^2$). Things started going south when you chose to use classifiers rather than predictors. And note that regression trees are highly unstable when the sample size is $<100,000$ subjects for example. That's why people use bagging, boosting, and random forests instead of single trees.

Proper accuracy scores are not destroyed by imbalanced $Y$.

$\endgroup$
  • $\begingroup$ @ Frank Harrell Thank you for the insightful comment! So you would for example recommend to use Pseudo R^2 to compare the performance? This would be rather straightforward since I guess I can derive them directly from the confusion matrix. This would be a great measure since I have an economics background and R^2 is often used there. $\endgroup$ – Patrick Balada Dec 30 '15 at 12:14
  • $\begingroup$ And if I may ask. Why do you call it a "proper" score? $\endgroup$ – Patrick Balada Dec 30 '15 at 15:26
  • $\begingroup$ A proper accuracy scoring rule is one that gives the right rewards, i.e., it is optimized by a correct model. An improper accuracy score is optimized by choosing the wrong features and giving them the wrong weights. $\endgroup$ – Frank Harrell Dec 30 '15 at 15:41
  • 1
    $\begingroup$ My course notes go into this a bit especially in the logistic regression chapter. See biostat.mc.vanderbilt.edu/rms and look under course materials. $\endgroup$ – Frank Harrell Dec 30 '15 at 17:32
  • 1
    $\begingroup$ The concordance probability (AUROC) is less sensitive. Think of a pair of predicted probabilities 0.2 and 0.8 corresponding to a non-event and an event. Then think of a pair 0.1 and 0.9. Both get one point in the concordance calculation even though the second goes out on a limb. You can also think about this from the standpoint that the concordance probability is essentially the Wilcoxon-Mann-Whitney statistic for comparing two groups ($Y=0, Y=1$). Comparing 2 $c$-indexes is essentially the same as subtracting one Wilcoxon stat from another, which no one does. $\endgroup$ – Frank Harrell Dec 31 '15 at 21:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.