Is there any other measure of the performance of a classifier than the area under the ROC curve? I am trying to draw an ROC curve for a classifier and wondered to know if there is any other measure for the performance of the classifier than the AUC. And is there any free software that I can use to draw either the histograms or the probability density functions for the noise and the (signal+noise) in one graph?
 A: You cannot make an ROC curve for a classifier.  By definition a classifier returns only 2 values when $Y$ is binary.  You can create an ROC curve for a continuous predicted value. But to you question there are many approaches such as


*

*smooth nonparametric calibration curve to show absolute predictive accuracy

*generalized $R^2$ based on log-likelihood

*Brier score - the quadratic error score which is a proper scoring rule like measures based on log-likelihood


IMHO these are all more useful than sensitivity and specificity, which are improper accuracy scoring rules that make up the ROC curve.
A: There exist several alternative ways for the performance evaluation of different classification methods with different thresholds, e. g. 


*

*ROC gap: highest vertical distance between the ROC curve and the angle bisector.

*The Lorenz Curve and the corresponding Gini coefficient.

*The divergence could also be measured by $ \frac{(m_A - m_B)^2}{\sigma^2_A + \sigma^2_B} $ where $m$ denotes the mean and $\sigma^2$ the variance of group $A$/$B$

*Another measure for the discriminatory power is the area where the distributions of both classes intersect

*One can also use a two sample t-test or non parametric versions of the Mann-Whitney-U test, Kolmogorow-Smirnow test and Wilcoxon rank sum test. Here, $H_0$ is equal in saying there is no difference between both distributions. Hence, the result of the better classifier will likely get lower p values.
A: Scoring rules are very useful in evaluating categorical models. I would rank them higher that AUC because AUC is open to subjectiveness in where you want to cut off the threshold values for the ROC.
For binary models, I prefer the logarithmic rule, mainly because it's the easiest to explain to PHB types, although Briers isn't that far behind.
There are also precision-recall curves and the area under them; these are used instead of ROC when the distribution of the data is highly skewed.
