1
$\begingroup$

Assume a pure prediction problem.

Say I want to evaluate a prediction-focused survival model in the context of actual dates instead of at a specific survival time or integration thereof.

What would be the correct way to approach this/which metrics would be the correct measures of calibration/discrimination in this case?

An Example:

Assume a survival model based on only right-censored data. Time-since-entry is measured in months. I obtain $\hat{S_i}(t_i|X_i)$. Calculating this survival probability for each $i$ with respect to a specific date (e.g. january year X), makes it necessary to use different $t_i$ for each $i$ (for the population still at risk at this specific date). Of course, a little rounding bias gets introduced due to calculating survival times and then translating them back to real time.

1. How to calculate the (time-dependent) AUC and the Brier/Graf Score in this setting?

2. Specifically, does one even need weights of the censoring distribution for the brier score/AUC? (see e.g. here)

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

It makes the most sense to evaluate the model with respect its original time origin, with time = 0 as study entry date. If the model's performance differs depending on the calendar date of time = 0 for an individual, you probably haven't incorporated calendar date of entry appropriately into your model. Check that first, and fix if necessary instead.

The need for inverse probability of censoring weights depends on the nature of the data and the use of the model. Weighting can help with causal inference, as explained by Hernán and Robins in Causal Inference: What If. Weighting won't necessarily work as hoped, however, if assumptions aren't met. See for example Howe et al, Limitation of Inverse Probability-of-Censoring Weights in Estimating Survival in the Presence of Strong Selection Bias, Am J Epidemiol 173: 569-77, 2011.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks for the answer! Let me clarify things a bit (will also edit for clarification above). The problem as stated above is a pure prediction problem. I am not interested at all in estimating effects of individual features in the face of confounding etc. The weigthing question referred to the calculation of the time-dependent AUC/Graf-Score. With repspect to your first paragraph: So youre suggesting to do: for all calendar dates where a subject entered: for all i: set t_i to the respective survival time for that calendar date: do AUC/Brier? $\endgroup$
    – Rob G.
    Sep 16, 2022 at 9:01
  • $\begingroup$ And then compare these scores for all calendar dates of entry? Including this into the model would then necessitate one-hot encoded indicators for all entry dates, right? $\endgroup$
    – Rob G.
    Sep 16, 2022 at 9:06
  • $\begingroup$ @RobG. define time = 0 for survival as the calendar entry date and the survival times as times relative to that date. Include the actual entry_date (e.g., months since January 2000) as a continuous predictor in the model. Model entry_date flexibly, for example with a regression spline. That will adjust for any calendar-date-related changes in survival behavior (as seen in clinical studies where therapy improves over time). Do your AUC/Brier evaluations on that model, in its own time scale. $\endgroup$
    – EdM
    Sep 16, 2022 at 14:01
  • $\begingroup$ @RobG. when you do a prediction for survival of an individual through a particular calendar date, enter the corresponding entry_date value as a covariate along with the other covariates in your model, and make a prediction for the number of months between that entry_date and the calendar date of interest. $\endgroup$
    – EdM
    Sep 16, 2022 at 14:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.