Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I struggle to find a way to compute the p-value for the area under a receiver operator characteristic (ROC). I have a continuous variable and a diagnostic test result. I want to see if AUROC is statistically significant.
I found many packages dealing with ROC curves: pROC, ROCR, caTools, verification, Epi. But even after many hours spent reading the documentation and testing, I couldn't find how. I think I've just missed it.
In your situation it would be fine to plot a ROC curve, and to calculate the area under that curve, but this should be thought of as supplemental to your main analysis, rather than the primary analysis itself. Instead, you want to fit a logistic regression model.
The logistic regression model will come standard with a test of the model as a whole. (Actually, since you have only one variable, that p-value will be the same as the p-value for your test result variable.) That p-value is the one you are after. The model will allow you to calculate the predicted probability of an observation being diseased. A Receiver Operating Characteristic tells you how the sensitivity and specificity will trade off, if you use different thresholds to convert the predicted probability into a predicted classification. Since the predicted probability will be a function of your test result variable, it is also telling you how they trade off if you use different test result values as your threshold.
If you are not terribly familiar with logistic regression there are some resources available on the internet (besides the Wikipedia page linked above):
R, the UCLA stats help website is generally excellent and has a relevant page here.
summary(GLM.1) should give you what you need, & I think anova(GLM.1) will test it against the null model w/o your actually having to fit it as well. But your way will definitely work, yes.
$\endgroup$
Nov 13 '13 at 21:05
Basically you want to test H0 = "The AUC is equal to 0.5".
This is in fact equivalent as saying H0 = "The distribution of the ranks in the two groups are equal".
The latter is the null hypothesis of the Mann-Whitney (Wilcoxon) test (see for instance Gold, 1999).
In other words, you can safely use a Mann-Whitney-Wilcoxon test to answer your question (see for instance Mason & Graham, 2002). This is exactly what the verification package mentioned by Franck Dernoncourt does.
You can use roc.area() from the package verification:
install.packages("verification")
library("verification")
# Data used from Mason and Graham (2002).
a<- c(1981, 1982, 1983, 1984, 1985, 1986, 1987, 1988, 1989, 1990,
1991, 1992, 1993, 1994, 1995)
d<- c(.928,.576, .008, .944, .832, .816, .136, .584, .032, .016, .28, .024, 0, .984, .952)
A<- data.frame(a,d)
names(A)<- c("year", "p2")
# For model without ties
roc.area(A$event, A$p2)
It will return $p.value
[1] 0.0069930071
Two ROC curves can be compared in pROC using roc.test(). This also produces a p-value. In addition, using roc(..., auc=TRUE, ci=TRUE) will give you the lower and higher confidence intervals along with the AUC in the output while creating the ROC object, which may be useful.
The following is working example code that tests whether the miles per gallon or the weight of a car is a better predictor of kind of transmission it comes equipped with (automatic or manual):
library(pROC)
roc_object_1 <- roc(mtcars$am, mtcars$mpg, auc=T, ci=T) #gives AUC and CI
roc_object_2 <- roc(mtcars$am, mtcars$wt, auc=T, ci=T) #gives AUC and CI
roc.test(roc_object_1, roc_object_2) #gives p-value
The weight is a significantly better predictor than the fuel consumption, it seems. However, this is comparing two curves, and not a single curve against a number such as 0.5. Looking at the confidence interval to see whether it contains the number 0.5 tells us whether it is significantly different, but it doesn't produce a p-value.
