Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

It is possible to calculate the c-index for time dependent outcomes (such as disease) using the survivalROC package in R. My question is : is it possible to produce a p-value for the c-index that is calculated (at a specific point in time)?

If I were to use a standard ROC I understand that I can use Wilcoxon test to calculate the p-value of the AUC (here's a simple example). However, because this is survival data I don't think I can use quite the same approach. I have thought of just calculating the concordance index using the survcomp package and getting the p-value from that, however survivalROC predicts the c-index at a specific time, whereas the concordance index doesn't seem to.

Any suggestions?

share|improve this question
    
What about a confidence interval obtained through bootstrapping? –  andrea Mar 28 '12 at 7:57
2  
This is what I did some time ago using the packages risksetROC (by P.J. Heagerty) and boot. CIs are obtained, as said, trough bootstrapping (x-axis is time in months). i.imgur.com/Gf0g5.png –  andrea Mar 28 '12 at 8:26
    
I am definitively going to try these and report back : ) Thanks for the suggestions –  user4673 Mar 28 '12 at 16:18
add comment

1 Answer 1

up vote 0 down vote accepted

This is what I did in the end to obtain the confidence interval and a p-value:

For the CI, I did indeed used bootstrapping method suggested by Andrea. I used boot and survivalROC, it worked very nicely (thanks very much ).

For the p-value, I randomly permuted the scores assigned to the labels approximately 10,000 times (I ran this in parallel, because it take quite a bit of time doing it serially).

After 10,000 iterations I had a distribution of AUCs which was centered on approximately 0.5. To get a p-value I summed the total number of AUCs that were larger than the AUC I observed, i.e. 124 permutations had and AUC larger than the AUC I observed with the "correct" associate to labels and scores (what my classifier spit out). Thus my p-values was 124/10,000. I suppose, given the nature of AUC I can also do it two sided and see how many AUCs are fewer than 1-observedAUC.

For some of the other methods I was comparing against, I found I needed much more than 10,000 iterations to get AUCs that were higher than what I observed. That means that p-values were always zero. To help out with that, I also calculated the extreme AUCs I could get (so, if I gave all my highest scores to label X), so at least there was always at least one values larger than what I observed.

In the end, I do find the CI to be the most informative, however I was asked to get a specific p-value for the AUC.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.