P-value of a survival ROC c-index

Question

It is possible to calculate the c-index for time dependent outcomes (such as disease) using the survivalROC package in R. My question is : is it possible to produce a p-value for the c-index that is calculated (at a specific point in time)?

If I were to use a standard ROC I understand that I can use Wilcoxon test to calculate the p-value of the AUC (here's a simple example). However, because this is survival data I don't think I can use quite the same approach. I have thought of just calculating the concordance index using the survcomp package and getting the p-value from that, however survivalROC predicts the c-index at a specific time, whereas the concordance index doesn't seem to.

Any suggestions?

Answer 1

This is what I did in the end to obtain the confidence interval and a p-value:

For the CI, I did indeed used bootstrapping method suggested by Andrea. I used boot and survivalROC, it worked very nicely (thanks very much ).

For the p-value, I randomly permuted the scores assigned to the labels approximately 10,000 times (I ran this in parallel, because it take quite a bit of time doing it serially).

After 10,000 iterations I had a distribution of AUCs which was centered on approximately 0.5. To get a p-value I summed the total number of AUCs that were larger than the AUC I observed, i.e. 124 permutations had and AUC larger than the AUC I observed with the "correct" associate to labels and scores (what my classifier spit out). Thus my p-values was 124/10,000. I suppose, given the nature of AUC I can also do it two sided and see how many AUCs are fewer than 1-observedAUC.

For some of the other methods I was comparing against, I found I needed much more than 10,000 iterations to get AUCs that were higher than what I observed. That means that p-values were always zero. To help out with that, I also calculated the extreme AUCs I could get (so, if I gave all my highest scores to label X), so at least there was always at least one values larger than what I observed.

In the end, I do find the CI to be the most informative, however I was asked to get a specific p-value for the AUC.

Answer 2

With survcomp you can indeed obtain a p-value, just change the default method (e.g. to method="noether"). Actually, example(survcomp::concordance.index) shows that.

Also, I would be careful bootstrapping around 0.5 and obtaining a (one-sided) p-value from counting of how many bootstrapped values are higher than your test value x. You are creating a bias, because you neglect values lower than 0.5 as being non-informative, but they may actually turn out to be significantly different form 0.5. I suggest transforming your bootstrapped values xx by .5+abs(xx-.5) to flip them around 0.5, as a value of 0.1 is equivalent of 0.9 when changing the signs of the predictors. Then, p=(n_higher+1)/(n+1), where n_higher are the number of bootstrap values higher than your x and n your number of resamples.