Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

It is possible to calculate the c-index for time dependent outcomes (such as disease) using the survivalROC package in R. My question is : is it possible to produce a p-value for the c-index that is calculated (at a specific point in time)?

If I were to use a standard ROC I understand that I can use Wilcoxon test to calculate the p-value of the AUC (here's a simple example). However, because this is survival data I don't think I can use quite the same approach. I have thought of just calculating the concordance index using the survcomp package and getting the p-value from that, however survivalROC predicts the c-index at a specific time, whereas the concordance index doesn't seem to.

Any suggestions?

share|improve this question
What about a confidence interval obtained through bootstrapping? – boscovich Mar 28 '12 at 7:57
This is what I did some time ago using the packages risksetROC (by P.J. Heagerty) and boot. CIs are obtained, as said, trough bootstrapping (x-axis is time in months). – boscovich Mar 28 '12 at 8:26
I am definitively going to try these and report back : ) Thanks for the suggestions – user4673 Mar 28 '12 at 16:18
up vote 0 down vote accepted

This is what I did in the end to obtain the confidence interval and a p-value:

For the CI, I did indeed used bootstrapping method suggested by Andrea. I used boot and survivalROC, it worked very nicely (thanks very much ).

For the p-value, I randomly permuted the scores assigned to the labels approximately 10,000 times (I ran this in parallel, because it take quite a bit of time doing it serially).

After 10,000 iterations I had a distribution of AUCs which was centered on approximately 0.5. To get a p-value I summed the total number of AUCs that were larger than the AUC I observed, i.e. 124 permutations had and AUC larger than the AUC I observed with the "correct" associate to labels and scores (what my classifier spit out). Thus my p-values was 124/10,000. I suppose, given the nature of AUC I can also do it two sided and see how many AUCs are fewer than 1-observedAUC.

For some of the other methods I was comparing against, I found I needed much more than 10,000 iterations to get AUCs that were higher than what I observed. That means that p-values were always zero. To help out with that, I also calculated the extreme AUCs I could get (so, if I gave all my highest scores to label X), so at least there was always at least one values larger than what I observed.

In the end, I do find the CI to be the most informative, however I was asked to get a specific p-value for the AUC.

share|improve this answer

With survcomp you can indeed obtain a p-value, just change the default method (e.g. to method="noether"). Actually, example(survcomp::concordance.index) shows that.

Also, I would be careful bootstrapping around 0.5 and obtaining a (one-sided) p-value from counting of how many bootstrapped values are higher than your test value x. You are creating a bias, because you neglect values lower than 0.5 as being non-informative, but they may actually turn out to be significantly different form 0.5. I suggest transforming your bootstrapped values xx by .5+abs(xx-.5) to flip them around 0.5, as a value of 0.1 is equivalent of 0.9 when changing the signs of the predictors. Then, p=(n_higher+1)/(n+1), where n_higher are the number of bootstrap values higher than your x and n your number of resamples.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.