Prediction metrics for left-truncated and right-censored data

Question

I am trying to assess (out-of-sample) predictive performance of survival analysis models with left-truncated and right-censored data. Assume the training and test data, respectively, consist of measurements $$ D_{train} = \{(y_i,\ell_i,\delta_i,x_i)\}_{i=1}^m, \qquad D_{test} = \{(y_i, \ell_i,\delta_i,x_i)\}_{i=1}^n, $$ where $y_i$ denotes age of the $i$th subject at death or censoring, $\ell_i$ denotes age at entry to the study (left truncation time), $\delta_i\in\{0,1\}$ indicates death (1) or censoring (0), and $x_i$ are subject-specific covariates.

The main metrics I am interested in are the Brier score and AUC (area under (ROC) curve). However, the formulas/software implementations of these metrics that I can find usually only account for right-censoring and not left-truncation.

Brier Score

For example, the time-dependent Brier score for right-censored data is defined as $$BS(t) = \frac{1}{n}\sum_{i=1}^n w_i(t)[\hat{S}(t|x_i) - I(y_i > t)]^2,$$ where $\hat{S}(t|x_i)$ is the predicted survival probability for the $i$th subject in the test data $D_{test}$ conditional on their covariates $x_i$, and $w_i(t)$ is the inverse probability of censoring weight (IPCW): $$w_i(t) = \frac{I(y_i\le t, \delta_i=1)}{\hat{G}(y_i)}+\frac{I(y_i>t)}{\hat{G}(t)}.$$ Here, $\hat{G}$ is the Kaplan-Meier estimator of the censoring time distribution estimated from the training data $D_{train}$. (See the scikit-survival implementation in Python.)

I haven't found a formula for left-truncated data, and the only software implementation I have found is the sbrier_ltrc() function from the LTRCforests package in R. In the paper accompanying the package, the authors provide the same formula as given above only accounting for right-censoring. However, upon looking at the package's source code, it appears that they modify the IPCW via the substitution $$ \hat{G}(y_i) \to \hat{G}_{LT}(y_i)/\hat{G}_{LT}(\ell_i), $$ and similarly for $\hat{G}(t)$, where $\hat{G}_{LT}$ is now the Kaplan-Meier estimator accounting for left truncation. Is dividing by $\hat{G}_{LT}(\ell_i)$ necessary if $\hat{G}_{LT}(y_i)$ already accounts for the left truncation?

Also, I would guess that we also need to modify the IPCW via the substitution $$ I(y_i>t) \to I(y_i>t \ge \ell_i), $$ i.e., we only include those subjects actually observed at time $t$ (the risk set) in the calculation.

AUC

The situation is much the same for the AUC, although here I have not found any formulas or software that account for left truncation. As with the Brier score, the formula for the time-dependent AUC with right-censoring includes IPCWs $\omega_i$ that account for the censoring distribution: $$ AUC(t) = \frac{\sum_{i=1}^n\sum_{j=1}^n I(y_j>t)I(y_i\le t)\omega_i I(\hat{f}(x_j)\le\hat{f}(x_i))}{\left(\sum_{i=1}^n I(y_i>t)\right)\left(\sum_{i=1}^n I(y_i\le t)\omega_i\right) }. $$ Here $\hat{f}(x_i)$ is the estimated risk score of the $i$th subject. In this case, I am not sure if the IPCW $\omega_i$ is the same as $w_i(t)$ for the Brier score above. (See the scikit-survival implementation.)

I am looking for any guidance extending these formulas for left-truncated data. Essentially, I think this boils down to calculating IPCW for left-truncated and right-censored data, which I haven't found in the literature. Thanks.

Answer 1

Left truncation with right censoring is how time-varying covariate values are handled in Cox models. The counting process form (in R) of Surv(startTime, stopTime, event)is interpreted as left truncation at startTime and right censoring or event at stopTime.

A straightforward approach would be to use the concordance index, closely related to AUC, as your performance measure. That's the fraction of comparable pairs of cases for which the model got the correct order of survival times. The R coxph models allow for predictions and thus concordance index values based on new left truncated, right censored survival data. There are also methods for time-dependent AUC survival measures; you could check the risksetROC package. According to the manual, its risksetROC() function handles left truncation and right censoring to evaluate a fixed outcome marker score at a desired survival time. That marker score could be the linear predictor from your model for baseline covariate values.

That said, I'd be very wary of just what's being evaluated. With left truncation, proposing new data for predictions implies that you already know that an individual has survived beyond that left-truncation time. What about individuals who didn't survive that long? That runs a risk of survivorship bias; the Python lifelines package won't even allow predictions on new left-truncated/right-censored data, for reasons explained here.

Finally, unless you have tens of thousands of cases, separate train and test sets will lose precision in training and power in testing. See this post, for example. You are otherwise better off building the model on the entire data set, followed by internal validation via resampling.