6
$\begingroup$

(Step 1) Using my predictive model, I predicted 1000 scores for my sample dataset.

(Step 2) I then calculate the random score using the same method for a randomized dataset. I firstly fit the distribution of the random score.

(Step 3) For each of my predictive score (1000 scores, in step 1), I calculated the p-values of getting a score larger than my predictive score for my sample sample dataset. Thus, 1000 p-values for my sample dataset are obtained.

(Step 4) As the real classification is known, by looking at the enrichment of true positive, I found when filter the sample dataset by p-values < 0.05. The best true positive values is obtained, which represents about 150 data from my sample dataset.

I then, want to test the predictive power of my model by doing AUC of the ROC (sensitivity vs 1- specificity plot).

However, I am facing a problem now, should I include all 1000 data for ROC plot to get the AUC, or should I only include those 150 data (p < 0.05) for my AUC analysis?

When I said p < 0.05, the p-value to obtain a score higher than my predictive score by random. In general speaking, does it means that 50% of my data are obtained by chance?


Edit

Thanks for the comments from @AlefSin, @steffen and @Frank Harrell.

For easier to discuss, I have prepare a sample dataset (x) as follows:

  • My model predicted score (assume it is normal distributed with mean=1, sd=1)
  • random set (assume also has mean=1, sd=1)
  • Probability for each predicted score
  • class prediction are listed as below, as listed in four columns

x <- data.frame (predict_score=c(rnorm(50,m=1, sd=1)))
x$random <- rnorm(50, m=1, sd=1)
x$probability <- pnorm(x$predict_score, m=mean(x$random),sd=sd(x$random))
x$class <- c(1,1,1,1,2,1,2,1,2,2,1,1,1,1,2,1,2,1,2,2,1,1,1,1,2,1,2,1,2,2,1,1,2,1,2,1,2,2,1,1,1,1,1,1,2,2,2,2,1,1)

I then did AUC for all data as follows for all data points:

library(caTools)
colAUC(x$predict_score, x$class, plotROC=T, alg=c("Wilcoxon","ROC"))

result:

       [,1]
1 vs. 2  0.6

Let's said if the enrichment of true positive is higher (Your runs may be differnt from mine, as rnorm give different results everytime) when I filter the dataset by p < 0.5, I did the AUC for a subset of the data as folows:

b <- subset(x, x$probability < 0.5)
colAUC(b$predict_score, b$class, plotROC=T, alg=c("Wilcoxon","ROC"))

result:

             [,1]
1 vs. 2 0.7401961

My question is: When I do AUC analysis, is it a must to do the analysis with the whole dataset, or should we do filter the dataset first based on enrichment of true positive or what ever criteria before doing AUC?

$\endgroup$
  • 1
    $\begingroup$ Try this: E.R. DeLong, D.M. DeLong, and D.L. Clarke-Pearson (1988), "Comparing the Areas Under Two or More Correlated Receiver Operating Characteristic Curves: A Nonparametric Approach," Biometrics, 44, 837-845. It explains how ROC curve and the U-statistics are related, how to calculate the significance of AUC and also how to say if difference between two AUCs is statistically significant. $\endgroup$ – AlefSin Jul 8 '11 at 8:06
  • 1
    $\begingroup$ @AlefSin why don't you make an answer ;) ? @anyone: Can someone verify that the procedure performed by the OP is correct ? @evdstat: (+1) a really interesting question $\endgroup$ – steffen Jul 8 '11 at 8:24
  • $\begingroup$ @AlefSin, @steffen, Yes, please kindly comments if I have done it correctly or not. Many thanks again for your help. $\endgroup$ – evdstat Jul 8 '11 at 9:52
  • $\begingroup$ It would be better to state your ultimate goals and to describe the candidate features (predictors) and their dimensionality, and also what is the dependent variable in its rawest form (e.g., before any dichotomization). Then we can better help you. $\endgroup$ – Frank Harrell Jul 8 '11 at 12:35
  • $\begingroup$ @evdstat I merged your second post as an edit to this because it wasn't a new question and relied heavily on this one. $\endgroup$ – whuber Jul 8 '11 at 15:19
1
$\begingroup$

ROC analysis answers the following question (in short): is your predictor different in the two groups?

You are confident that the subset obtained at step 4 is highly enriched in true positives. However this result won't answer the question: "If I submit a new data point, is my predictive method going to get it right?" which is the typical question in ROC analysis. Instead, it will tell you "If this has a p < 0.05, am I highly confident this is a positive?". That is not the point of ROC analysis. While this type of analysis might be relevant in your case (please note I said might, I have no idea if it is actually the case here), it is clearly not a standard ROC analysis and you'd have to make this clear.

To convince yourself that your procedure is not correct, repeat the procedure you showed above (generating a random dataset and computing the AUCs) multiple times. If this is random data, so you expect to obtain an AUC of 0.5 on average, right? Is it what you get? I bet not!


A few more comments:

  • If you are especially interested in the positives, and you want to ensure that all your positive observations are predicted in the positive group, you may be interested in the partial area under the ROC curve that focuses on the high specificity region. See this paper by McClish.
  • If you are interested in the enrichment of true positives, you want to check the positive predictive value rather than the specificity / ROC.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.