I'd like to compare logistic regression to classification trees. In a first step, I compared the theoretical framework of the two classifiers. In a second step, I compared the performance using a rather unbalanced data set containing two classes. I therefore compared confusion-matrices, balanced-accuracy, sensitivity and specificity. Moreover, I compared the ROC curves and derived therefrom the AUC values. Are there any other value adding measures to compare the classifiers? How would you compare the efficiency in terms of running time?
$\begingroup$
$\endgroup$
asked Dec 30, 2015 at 9:58
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
15
You seem to be using a mish-mash of methods. Focus on getting predicted risks and using proper accuracy scores such as the Brier score or logarithmic scoring rule (log likelihood; related to pseudo $R^2$). Things started going south when you chose to use classifiers rather than predictors. And note that regression trees are highly unstable when the sample size is $<100,000$ subjects for example. That's why people use bagging, boosting, and random forests instead of single trees.
Proper accuracy scores are not destroyed by imbalanced $Y$.
answered Dec 30, 2015 at 11:25
-
$\begingroup$ @ Frank Harrell Thank you for the insightful comment! So you would for example recommend to use Pseudo R^2 to compare the performance? This would be rather straightforward since I guess I can derive them directly from the confusion matrix. This would be a great measure since I have an economics background and R^2 is often used there. $\endgroup$ Commented Dec 30, 2015 at 12:14
-
$\begingroup$ And if I may ask. Why do you call it a "proper" score? $\endgroup$ Commented Dec 30, 2015 at 15:26
-
$\begingroup$ A proper accuracy scoring rule is one that gives the right rewards, i.e., it is optimized by a correct model. An improper accuracy score is optimized by choosing the wrong features and giving them the wrong weights. $\endgroup$ Commented Dec 30, 2015 at 15:41
-
1$\begingroup$ My course notes go into this a bit especially in the logistic regression chapter. See biostat.mc.vanderbilt.edu/rms and look under course materials. $\endgroup$ Commented Dec 30, 2015 at 17:32
-
1$\begingroup$ The concordance probability (AUROC) is less sensitive. Think of a pair of predicted probabilities 0.2 and 0.8 corresponding to a non-event and an event. Then think of a pair 0.1 and 0.9. Both get one point in the concordance calculation even though the second goes out on a limb. You can also think about this from the standpoint that the concordance probability is essentially the Wilcoxon-Mann-Whitney statistic for comparing two groups ($Y=0, Y=1$). Comparing 2 $c$-indexes is essentially the same as subtracting one Wilcoxon stat from another, which no one does. $\endgroup$ Commented Dec 31, 2015 at 21:06