It is possible to calculate the c-index for time dependent outcomes (such as disease) using the survivalROC package in R. My question is : is it possible to produce a p-value for the c-index that is calculated (at a specific point in time)?

If I were to use a standard ROC I understand that I can use Wilcoxon test to calculate the p-value of the AUC (here's a simple example). However, because this is survival data I don't think I can use quite the same approach. I have thought of just calculating the concordance index using the survcomp package and getting the p-value from that, however survivalROC predicts the c-index at a specific time, whereas the concordance index doesn't seem to.

Any suggestions?

  • $\begingroup$ What about a confidence interval obtained through bootstrapping? $\endgroup$
    – boscovich
    Commented Mar 28, 2012 at 7:57
  • 2
    $\begingroup$ This is what I did some time ago using the packages risksetROC (by P.J. Heagerty) and boot. CIs are obtained, as said, trough bootstrapping (x-axis is time in months). i.imgur.com/Gf0g5.png $\endgroup$
    – boscovich
    Commented Mar 28, 2012 at 8:26
  • $\begingroup$ I am definitively going to try these and report back : ) Thanks for the suggestions $\endgroup$
    – user4673
    Commented Mar 28, 2012 at 16:18

2 Answers 2


This is what I did in the end to obtain the confidence interval and a p-value:

For the CI, I did indeed used bootstrapping method suggested by Andrea. I used boot and survivalROC, it worked very nicely (thanks very much ).

For the p-value, I randomly permuted the scores assigned to the labels approximately 10,000 times (I ran this in parallel, because it take quite a bit of time doing it serially).

After 10,000 iterations I had a distribution of AUCs which was centered on approximately 0.5. To get a p-value I summed the total number of AUCs that were larger than the AUC I observed, i.e. 124 permutations had and AUC larger than the AUC I observed with the "correct" associate to labels and scores (what my classifier spit out). Thus my p-values was 124/10,000. I suppose, given the nature of AUC I can also do it two sided and see how many AUCs are fewer than 1-observedAUC.

For some of the other methods I was comparing against, I found I needed much more than 10,000 iterations to get AUCs that were higher than what I observed. That means that p-values were always zero. To help out with that, I also calculated the extreme AUCs I could get (so, if I gave all my highest scores to label X), so at least there was always at least one values larger than what I observed.

In the end, I do find the CI to be the most informative, however I was asked to get a specific p-value for the AUC.


With survcomp you can indeed obtain a p-value, just change the default method (e.g. to method="noether"). Actually, example(survcomp::concordance.index) shows that.

Also, I would be careful bootstrapping around 0.5 and obtaining a (one-sided) p-value from counting of how many bootstrapped values are higher than your test value x. You are creating a bias, because you neglect values lower than 0.5 as being non-informative, but they may actually turn out to be significantly different form 0.5. I suggest transforming your bootstrapped values xx by .5+abs(xx-.5) to flip them around 0.5, as a value of 0.1 is equivalent of 0.9 when changing the signs of the predictors. Then, p=(n_higher+1)/(n+1), where n_higher are the number of bootstrap values higher than your x and n your number of resamples.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.