# How to test the statistical significance of AUC?

(Step 1) Using my predictive model, I predicted 1000 scores for my sample dataset.

(Step 2) I then calculate the random score using the same method for a randomized dataset. I firstly fit the distribution of the random score.

(Step 3) For each of my predictive score (1000 scores, in step 1), I calculated the p-values of getting a score larger than my predictive score for my sample sample dataset. Thus, 1000 p-values for my sample dataset are obtained.

(Step 4) As the real classification is known, by looking at the enrichment of true positive, I found when filter the sample dataset by p-values < 0.05. The best true positive values is obtained, which represents about 150 data from my sample dataset.

I then, want to test the predictive power of my model by doing AUC of the ROC (sensitivity vs 1- specificity plot).

However, I am facing a problem now, should I include all 1000 data for ROC plot to get the AUC, or should I only include those 150 data (p < 0.05) for my AUC analysis?

When I said p < 0.05, the p-value to obtain a score higher than my predictive score by random. In general speaking, does it means that 50% of my data are obtained by chance?

## Edit

Thanks for the comments from @AlefSin, @steffen and @Frank Harrell.

For easier to discuss, I have prepare a sample dataset (x) as follows:

• My model predicted score (assume it is normal distributed with mean=1, sd=1)
• random set (assume also has mean=1, sd=1)
• Probability for each predicted score
• class prediction are listed as below, as listed in four columns

x <- data.frame (predict_score=c(rnorm(50,m=1, sd=1)))
x$random <- rnorm(50, m=1, sd=1) x$probability <- pnorm(x$predict_score, m=mean(x$random),sd=sd(x$random)) x$class <- c(1,1,1,1,2,1,2,1,2,2,1,1,1,1,2,1,2,1,2,2,1,1,1,1,2,1,2,1,2,2,1,1,2,1,2,1,2,2,1,1,1,1,1,1,2,2,2,2,1,1)


I then did AUC for all data as follows for all data points:

library(caTools)
colAUC(x$predict_score, x$class, plotROC=T, alg=c("Wilcoxon","ROC"))


result:

       [,1]
1 vs. 2  0.6


Let's said if the enrichment of true positive is higher (Your runs may be differnt from mine, as rnorm give different results everytime) when I filter the dataset by p < 0.5, I did the AUC for a subset of the data as folows:

b <- subset(x, x$probability < 0.5) colAUC(b$predict_score, b\$class, plotROC=T, alg=c("Wilcoxon","ROC"))


result:

             [,1]
1 vs. 2 0.7401961


My question is: When I do AUC analysis, is it a must to do the analysis with the whole dataset, or should we do filter the dataset first based on enrichment of true positive or what ever criteria before doing AUC?

-
Try this: E.R. DeLong, D.M. DeLong, and D.L. Clarke-Pearson (1988), "Comparing the Areas Under Two or More Correlated Receiver Operating Characteristic Curves: A Nonparametric Approach," Biometrics, 44, 837-845. It explains how ROC curve and the U-statistics are related, how to calculate the significance of AUC and also how to say if difference between two AUCs is statistically significant. –  AlefSin Jul 8 '11 at 8:06
@AlefSin why don't you make an answer ;) ? @anyone: Can someone verify that the procedure performed by the OP is correct ? @evdstat: (+1) a really interesting question –  steffen Jul 8 '11 at 8:24
@AlefSin, @steffen, Yes, please kindly comments if I have done it correctly or not. Many thanks again for your help. –  evdstat Jul 8 '11 at 9:52
It would be better to state your ultimate goals and to describe the candidate features (predictors) and their dimensionality, and also what is the dependent variable in its rawest form (e.g., before any dichotomization). Then we can better help you. –  Frank Harrell Jul 8 '11 at 12:35
@evdstat I merged your second post as an edit to this because it wasn't a new question and relied heavily on this one. –  whuber Jul 8 '11 at 15:19