# Prediction metrics for left-truncated and right-censored data

I am trying to assess (out-of-sample) predictive performance of survival analysis models with left-truncated and right-censored data. Assume the training and test data, respectively, consist of measurements $$D_{train} = \{(y_i,\ell_i,\delta_i,x_i)\}_{i=1}^m, \qquad D_{test} = \{(y_i, \ell_i,\delta_i,x_i)\}_{i=1}^n,$$ where $$y_i$$ denotes age of the $$i$$th subject at death or censoring, $$\ell_i$$ denotes age at entry to the study (left truncation time), $$\delta_i\in\{0,1\}$$ indicates death (1) or censoring (0), and $$x_i$$ are subject-specific covariates.

The main metrics I am interested in are the Brier score and AUC (area under (ROC) curve). However, the formulas/software implementations of these metrics that I can find usually only account for right-censoring and not left-truncation.

Brier Score

For example, the time-dependent Brier score for right-censored data is defined as $$BS(t) = \frac{1}{n}\sum_{i=1}^n w_i(t)[\hat{S}(t|x_i) - I(y_i > t)]^2,$$ where $$\hat{S}(t|x_i)$$ is the predicted survival probability for the $$i$$th subject in the test data $$D_{test}$$ conditional on their covariates $$x_i$$, and $$w_i(t)$$ is the inverse probability of censoring weight (IPCW): $$w_i(t) = \frac{I(y_i\le t, \delta_i=1)}{\hat{G}(y_i)}+\frac{I(y_i>t)}{\hat{G}(t)}.$$ Here, $$\hat{G}$$ is the Kaplan-Meier estimator of the censoring time distribution estimated from the training data $$D_{train}$$. (See the scikit-survival implementation in Python.)

I haven't found a formula for left-truncated data, and the only software implementation I have found is the sbrier_ltrc() function from the LTRCforests package in R. In the paper accompanying the package, the authors provide the same formula as given above only accounting for right-censoring. However, upon looking at the package's source code, it appears that they modify the IPCW via the substitution $$\hat{G}(y_i) \to \hat{G}_{LT}(y_i)/\hat{G}_{LT}(\ell_i),$$ and similarly for $$\hat{G}(t)$$, where $$\hat{G}_{LT}$$ is now the Kaplan-Meier estimator accounting for left truncation. Is dividing by $$\hat{G}_{LT}(\ell_i)$$ necessary if $$\hat{G}_{LT}(y_i)$$ already accounts for the left truncation?

Also, I would guess that we also need to modify the IPCW via the substitution $$I(y_i>t) \to I(y_i>t \ge \ell_i),$$ i.e., we only include those subjects actually observed at time $$t$$ (the risk set) in the calculation.

AUC

The situation is much the same for the AUC, although here I have not found any formulas or software that account for left truncation. As with the Brier score, the formula for the time-dependent AUC with right-censoring includes IPCWs $$\omega_i$$ that account for the censoring distribution: $$AUC(t) = \frac{\sum_{i=1}^n\sum_{j=1}^n I(y_j>t)I(y_i\le t)\omega_i I(\hat{f}(x_j)\le\hat{f}(x_i))}{\left(\sum_{i=1}^n I(y_i>t)\right)\left(\sum_{i=1}^n I(y_i\le t)\omega_i\right) }.$$ Here $$\hat{f}(x_i)$$ is the estimated risk score of the $$i$$th subject. In this case, I am not sure if the IPCW $$\omega_i$$ is the same as $$w_i(t)$$ for the Brier score above. (See the scikit-survival implementation.)

I am looking for any guidance extending these formulas for left-truncated data. Essentially, I think this boils down to calculating IPCW for left-truncated and right-censored data, which I haven't found in the literature. Thanks.

• Please edit the question to say more about the nature and size (number of events) of the data, both training and test, and your study. Left truncation means that you have no information about survival times prior to the truncation time, so it's not always clear how to "validate" fairly in that case: if you have left-truncated data for an individual you already know that the individual survived at least that long. How about members of the underlying population that didn't survive that long and whose data are thus unavailable?
– EdM
Commented Jul 12, 2022 at 13:10
• Thanks for your comment. I made some clarifying edits. Commented Jul 12, 2022 at 15:46

Left truncation with right censoring is how time-varying covariate values are handled in Cox models. The counting process form (in R) of Surv(startTime, stopTime, event)is interpreted as left truncation at startTime and right censoring or event at stopTime.

A straightforward approach would be to use the concordance index, closely related to AUC, as your performance measure. That's the fraction of comparable pairs of cases for which the model got the correct order of survival times. The R coxph models allow for predictions and thus concordance index values based on new left truncated, right censored survival data. There are also methods for time-dependent AUC survival measures; you could check the risksetROC package. According to the manual, its risksetROC() function handles left truncation and right censoring to evaluate a fixed outcome marker score at a desired survival time. That marker score could be the linear predictor from your model for baseline covariate values.

That said, I'd be very wary of just what's being evaluated. With left truncation, proposing new data for predictions implies that you already know that an individual has survived beyond that left-truncation time. What about individuals who didn't survive that long? That runs a risk of survivorship bias; the Python lifelines package won't even allow predictions on new left-truncated/right-censored data, for reasons explained here.

Finally, unless you have tens of thousands of cases, separate train and test sets will lose precision in training and power in testing. See this post, for example. You are otherwise better off building the model on the entire data set, followed by internal validation via resampling.

• Thanks for your suggestions. I am considering a range of parametric and non-parametric models, including CoxPH. I have landed on AUC because the c-index is not proper for predicting t-year risks. I have tried to use risksetROC, but without success when handling time-varying risk scores. I understand the epistemological problem of prediction with time-varying covariates. However, I do not see any issue when using baseline covariates and adjusting the risk set (as described in my OP) to account for left truncation. Commented Jul 13, 2022 at 14:53
• @njirons you can estimate probability of censoring from data, as you have full information on times to events and to right censoring. It's not clear to me how you can estimate a probability of left truncation as you seem to need, when observations prior to left-truncation times aren't available (by definition). That epistemological problem isn't fundamentally different from what's involved with time-varying covariates; think of "case is under observation and at risk" as being a time-varying covariate.
– EdM
Commented Jul 13, 2022 at 16:34
• I think the standard (or at least classical) approach is to assume independence of truncation and failure times. This is how the Kaplan-Meier estimator for left-truncated and right-censored data is derived. If we make this assumption, does the issue you are describing persist? Commented Jul 13, 2022 at 16:57
• @njirons yes, the Kaplan-Meier and Cox model estimates are valid if event and left-truncation times are independent. See Sections 4.6 and 9.4 of Klein and Moeschberger. If truncation is random, it's possible to evaluate that assumption; see Tsai, Biometrika 77:169, 1990. The question is how well such assumptions work in your case. If you have time-varying risk scores, however, then the epistemological problem arises again.
– EdM
Commented Jul 14, 2022 at 16:10
• Just trying to wrap my head around this. Say I am using a random survival forest with no time-varying covariates. RSF does not make the PH assumption, so I still obtain time-varying risk scores for each subject (i.e., log CHF curves are not parallel as in CoxPH). The epistemological problem still applies here? (One caveat being that RSF does not account for left truncation.) Commented Jul 15, 2022 at 9:29