1
$\begingroup$

I am having some trouble with two different implementations of a classification problem giving different results. Me and my college who did the other implementation has narrowed the problem down to the way we calculate the area under the receiver operating characteristic curve (AUC). One solution is derived from a formula appearing at least at one location: [1]

$$AUC_1 = \frac{1}{mn}\sum_{i=1}^{m}\sum_{j=1}^{n}\mathbf{1}_{p_i<p_j}$$

I have ported the implementation into R and compared it with the result $AUC_2$ from the R-package pROC:

auc1 <- function(p) {
  values <- p[,1]
  positives <- values[ p[,2]==1 ]
  negatives <- values[ p[,2]==0 ]
  count <- 0
  for ( p in positives ) {
    for ( n in negatives ) {
      if ( p>n ) {
        count <- count + 1
      }
    }
  }
  return(count/(length(positives) * length(negatives)))
}

auc2 <- function(p) {
  library(pROC)
  c <- roc(p[,2], p[,1], print.auc=TRUE, ci=F, of="auc")
  return (auc(c))
}

predicted <- c(0, 1, 0, 1, 1, 1, 1, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1)
real      <- c(0, 1, 0, 1, 1, 1, 1, 1, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 1, 1)
p <- cbind(predicted, real)

auc1(p)
auc2(p)

Can someone shed some light on why I get:

$$AUC_1 = 0.9090909 \mbox{ and } AUC_2 = 0.9545\mbox{?}$$

$\endgroup$
3
$\begingroup$

Go back to basics. The AUROC is mainly a good measure because of a coincidence: it equals the concordance probability ($c$-index; $U$-statistic) commonly used in rank correlation measures and the Wilcoxon-Mann-Whitney statistic. Concordance is an excellent measure of separation or predictive discrimination. So calculate it efficiently:

require(Hmisc)
somers2(predicted, real)
         C        Dxy          n    Missing 
 0.9545455  0.9090909 20.0000000  0.0000000 

The efficient calculation is essentially a one-liner in somers2:

c.index <- (mean(rank(x)[y == 1]) - (n1 + 1)/2)/(n - n1)

But be clear on why you are using AUROC in the first place. It is a nice supplement to log-likelihood-based measures but not a substitute for the gold standard.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.