Estimate optimal cutoff for time-dependent ROC

Question

Recently I am working with survival data and trying to fit time-dependent ROC.

And I have difficulty estimating the optimal cutoff for time-dependent ROC.

Here is the process I do.

library(survival)
library(timeROC)
library(survivalROC)

data(pbc)
head(pbc)
pbc<-pbc[!is.na(pbc$trt),] 
pbc$status<-as.numeric(pbc$status==2) # create event indicator: 1 for death, 0 for censored

ROC.bili1<-timeROC(T=pbc$time,
                  delta=pbc$status,marker=pbc$bili,
                  cause=1,weighting="marginal",
                  times=c(365,365*3,365*5,365*10),
                  iid=TRUE)
ROC.bili1

And here are the results

Time-dependent-Roc curve estimated using IPCW  (n=312, without competing risks). 
       Cases Survivors Censored AUC (%)   se
t=365     22       290        0   85.59 3.51
t=1095    59       240       13   85.02 2.68
t=1825    85       159       68   87.58 2.29
t=3650   120        32      160   81.57 3.85

Method used for estimating IPCW:marginal 

Total computation time : 0.29  secs.

Next step, I would like to estimate the optimal cutoff in 10 years (time = 365*10)

However, it seems that timeROC does not return the results I want.

And it is reported that survivalROC does.

ROC.bili2<-survivalROC(Stime=pbc$time,
                       status=pbc$status,
                       marker=pbc$bili,
                       predict.time=365*10,
                       method = 'KM')

Estimate the optimal cutoff

ROC.bili2$cut.values[which.max(ROC.bili2$TP-ROC.bili2$FP)]
[1] 1.9

And calculate the AUC

ROC.bili2[["AUC"]]
[1] 0.8394563

I got the optimal cutoff is 1.9, but the AUC is 0.8394 which is different from using timeROC (0.8157)

So my question is

1. If I can use 1.9 as the optimal cutoff?

2. Why is the AUC difference between timeROC and survivalROC

3. Is there any way to calculate the optimal cutoff using timeROC

Any insight is appreciated.

Answer 1

Defining time-dependent receiver operating characteristic (ROC) curves for censored survival data is not a trivial problem. For an introduction to the issues, I'd suggest reading "Survival Model Predictive Accuracy and ROC Curves" by Heagerty and Zheng, Biometrics 61: 92–105 (2005). That describes a more recent approach, implemented in the risksetROC package, than that of Heagerty et al. implemented in the survivalROC package.

Those approaches differ from the inverse probability of censoring weighting method used in the timeROC package. So it's not surprising that you get slightly different AUC estimates for the two methods that you tried.

Your approach to determining an "optimal cutoff" might be highly flawed. Any "optimal" point on an ROC curve depends on the relative costs and benefits of different types of true and false predictions. Unless you take those costs into account you don't have an "optimal" cutoff. And even if an "optimal" cutoff on that basis could be available, you might want to use further information to guide practical decisions.

You chose to use the point on the curve with the highest difference between true-positive and false-positive predictions as "optimal," but that only would make sense if (1) you can completely ignore the costs/benefits of true-negative and false-negative results and (2) the cost of 1 false-positive result exactly balances the gain from 1 true-positive result. It's hard to think of a practical situation where those conditions would hold.

Software-specific questions are off-topic on this site devoted to statistics. Presumably, timeROC() provides the details of its ROC curves (sensitivity/specificity as a function of biomarker value) somewhere in the object it returns. If you have a valid criterion for an "optimal" cutoff, you should be able to apply that criterion to the ROC with a bit of manipulation.