Estimate optimal cutoff for time-dependent ROC

Recently I am working with survival data and trying to fit time-dependent ROC.

And I have difficulty estimating the optimal cutoff for time-dependent ROC.

Here is the process I do.

library(survival)
library(timeROC)
library(survivalROC)

data(pbc)
pbc<-pbc[!is.na(pbc$$trt),] pbc$$status<-as.numeric(pbc$status==2) # create event indicator: 1 for death, 0 for censored ROC.bili1<-timeROC(T=pbc$$time, delta=pbc$$status,marker=pbc$bili,
cause=1,weighting="marginal",
times=c(365,365*3,365*5,365*10),
iid=TRUE)
ROC.bili1


And here are the results

Time-dependent-Roc curve estimated using IPCW  (n=312, without competing risks).
Cases Survivors Censored AUC (%)   se
t=365     22       290        0   85.59 3.51
t=1095    59       240       13   85.02 2.68
t=1825    85       159       68   87.58 2.29
t=3650   120        32      160   81.57 3.85

Method used for estimating IPCW:marginal

Total computation time : 0.29  secs.


Next step, I would like to estimate the optimal cutoff in 10 years (time = 365*10)

However, it seems that timeROC does not return the results I want.

And it is reported that survivalROC does.

ROC.bili2<-survivalROC(Stime=pbc$time, status=pbc$status,
marker=pbc$bili, predict.time=365*10, method = 'KM')  Estimate the optimal cutoff ROC.bili2$cut.values[which.max(ROC.bili2$TP-ROC.bili2$FP)]
[1] 1.9


And calculate the AUC

ROC.bili2[["AUC"]]
[1] 0.8394563


I got the optimal cutoff is 1.9, but the AUC is 0.8394 which is different from using timeROC (0.8157)

So my question is

1. If I can use 1.9 as the optimal cutoff?

2. Why is the AUC difference between timeROC and survivalROC

3. Is there any way to calculate the optimal cutoff using timeROC

Any insight is appreciated.

Defining time-dependent receiver operating characteristic (ROC) curves for censored survival data is not a trivial problem. For an introduction to the issues, I'd suggest reading "Survival Model Predictive Accuracy and ROC Curves" by Heagerty and Zheng, Biometrics 61: 92–105 (2005). That describes a more recent approach, implemented in the risksetROC package, than that of Heagerty et al. implemented in the survivalROC package.

Those approaches differ from the inverse probability of censoring weighting method used in the timeROC package. So it's not surprising that you get slightly different AUC estimates for the two methods that you tried.

Your approach to determining an "optimal cutoff" might be highly flawed. Any "optimal" point on an ROC curve depends on the relative costs and benefits of different types of true and false predictions. Unless you take those costs into account you don't have an "optimal" cutoff. And even if an "optimal" cutoff on that basis could be available, you might want to use further information to guide practical decisions.

You chose to use the point on the curve with the highest difference between true-positive and false-positive predictions as "optimal," but that only would make sense if (1) you can completely ignore the costs/benefits of true-negative and false-negative results and (2) the cost of 1 false-positive result exactly balances the gain from 1 true-positive result. It's hard to think of a practical situation where those conditions would hold.

Software-specific questions are off-topic on this site devoted to statistics. Presumably, timeROC() provides the details of its ROC curves (sensitivity/specificity as a function of biomarker value) somewhere in the object it returns. If you have a valid criterion for an "optimal" cutoff, you should be able to apply that criterion to the ROC with a bit of manipulation.

• Thank you for your detailed answers. As for the method I used to select the optimal cutoffs, the maximum Youden index was used, As the Youden index = Sen + Spec -1, it also can be calculated by TP - FP (TP = Sen and FP = 1 - Spec). I hope it's right. Actually， I can calculate the maximum (TP-FP) in timeROC, but it didn't provide the value (bili in this case) accordingly.
– Lyn
Dec 4, 2022 at 4:20
• @Lyn the problem with the Youden Index is that "[t]he index gives equal weight to false positive and false negative values." That's not how you would typically evaluate a clinical test, for example. The matrices of TP and FP rates returned by thetimeROC() function are based on the set of marker values and times that you specified in the call to the function, so the bilivalue would be the value in your marker set that corresponds to the position of the maximum (TP-FP) at the time in question, if you want to use the Youden index.
– EdM
Dec 4, 2022 at 15:03