1
$\begingroup$

I am using Deep Recurrent Survival Machines (RDSM) model from auton-survival library for survival analysis with time-varying covariates. I am using a longitudinal survey dataset (Health and Retirement Study), where every respondent has variables from 13 waves at max.

I first tried to understand how RDSM works (link to the paper http://proceedings.mlr.press/v146/nagpal21a.html). From what I understood, they assume that a time to event follows a mixture of parametric distributions (like Weibull, Lognormal). Then the shape and scale of the distributions are implemented as a function of the input covariates using RNNs.

  1. My question is: how predictions are usually made in time-varying survival analysis?

In my case, RDSM treats each sample at every time point separately. What I mean is, for example if I have repondent's covariates from 2 waves, predictions at wave 1 and wave 2 are made separately, as if they come from different respondents. I am not sure whether this is correct. I think the model should know that the next set of covariates are coming from the same respondent. Or is this temporal relationship already captured while obtaining feature representations from RNN?

To be more clear, my test set is 4691, there are 13 time steps, and 206 covariates. I am expecting 4691 survival curves from my model, but it gives me 4691*13 = 60983 survival curves. Is this how it should be?

Any help or thought will be appreciated.

$\endgroup$
2
  • $\begingroup$ Is this the link to the "Health and Retirement Study" that you are using? $\endgroup$
    – EdM
    Sep 2 at 14:09
  • $\begingroup$ Hi, thank you so much for the elaborated answer. Yes, that's the correct link. $\endgroup$ Sep 5 at 8:14

1 Answer 1

2
$\begingroup$

These are panel data, with the same individuals evaluated at a set of discrete points in time, 13 time points in this case. The Health and Retirement Study supplements that with "Exit" questionnaires, in which survivors of those who die between interviews provide information about date and cause of death. If your outcome of interest is death, then you can model this with continuous-time survival methods, incorporating time-varying covariate values available at interview dates prior to death.

There are two questions here, one about the modeling method and another about predictions.

Modeling

It seems that there was some difficulty in preparing the panel data for the survival modeling software, as somehow you got 13 separate sets of models. Although you might have 13 separate "waves" of data from which to draw, you need to combine all the data together for analysis. You should take advantage of the unique individual ID value for each participant that is maintained throughout the Health and Retirement Study to label the data, even if the modeling might not ultimately require that.

For standard survival modeling with time-varying covariate values, each participant you would have separate data rows for all time periods between successive "waves" while in the study. Each row would have the ID, the start time and stop time of the interval between the waves, and the covariate values in place at the beginning of the interval. For those that die, the stop time is set instead to the time corresponding to death. Each row also has an event indicator, say 0 for still alive and 1 for died.

Those data rows should all be collected together into a single file for analysis. The start time of each row thus represents a left-truncated survival time, with either an event (if died) or right censoring (if alive) at the stop time. That's the standard "counting process" data format explained, for example, in this survival vignette.

In a comment, you say "time to event is defined as the time from the start of the survey till the death of a respondent." I'm not sure that "start of the survey" is a good choice for time = 0 for survival analysis, although if you use age as a covariate I suppose you could argue for that choice. Consider whether you should be using the date when age reached 51 years as time = 0 instead, as that's the earliest age when interviews start.

The method that you cite is based on parametric continuous-time modeling of covariates and survival. It seems designed to incorporate the past history of covariates into the current event risk, so that you would need to keep track of each individual's identity and history of covariate values.

I don't know enough about that method to say whether it's appropriate for your type of data. The example in the linked paper used hourly covariate values during stays in an intensive care unit, values that might reasonably be considered to be continuous. I don't know whether it can properly handle highly discretized panel data like yours or the associated "counting-process" data format.

Some survival models, like proportional-hazards models, only evaluate covariate values that are in place at event times and don't depend on past covariate history (except insofar as it might be incorporated into a value current at an event time, like a cumulative value). In modeling a terminal event like death that way, you can treat all of the observations as independent and don't need to keep track of individual IDs. The time-dependence vignette of the R survival package explains this in section 2.

Predictions

Regardless of how you ultimately model the data, your question about making predictions still stands. The problem with survival predictions when covariates are varying in time is that you have to assume sets of time-varying covariate values for the predictions.

That poses two potential problems.

First, if the event is death, then assuming time-varying covariate values implicitly assumes that the individual is still alive through those times. The author of the Python lifelines package for survival analysis thus won't allow the software to do such predictions. See this answer.

Second, even if you are willing to make such assumptions to represent the survival of a cohort of patients having a particular trajectory of the covariate values over time, there's a risk of assuming a set of values that could not realistically occur together. See this answer.

It might be possible to model future covariate values for participants as a guide to potentially realistic predictions. There is a field of study devoted to joint modeling of covariates and event times. The methods are implemented in the R JM package and the frailtypack package, among others you can find in the R survival view. That is beyond my expertise. This page and its links might help.

$\endgroup$
5
  • $\begingroup$ Does the instantaneous hazard estimate suffer from the same problem: one must be alive at time x to interpret the risk at time x+1? $\endgroup$
    – Todd D
    Sep 3 at 0:32
  • $\begingroup$ @EdM, thanks a lot for your reply. I think my data is continuous in time, since time to event is defined as the time from the start of the survey till the death of a respondent. And this may take some values like 25.5 etc. $\endgroup$ Sep 5 at 8:29
  • $\begingroup$ @EdM, I understood the concerns about predicting survival curves for people who have time-varying data, because that implies that they are alive. But then, do you know what might be the best method to model the data that I am using? I want the model to take respondent's covariates from 13 waves, and make predictions for the next 1 or 5 years? $\endgroup$ Sep 5 at 8:37
  • $\begingroup$ @ToddD instantaneous hazard is defined as the event probability (density) conditional upon having survived to the time in question, so there's no survivorship bias. It's just a definition. For predictions with time varying covariates, I'm willing to accept the approach of Therneau and Grambsch, outlined here, to assume a cohort of individuals with a set of time-varying covariates--provided that those values are realistic. $\endgroup$
    – EdM
    Sep 5 at 12:48
  • $\begingroup$ @ElnuraZhalieva I expanded the answer a bit, based on what I have since learned about the data and the method that you tried to implement. $\endgroup$
    – EdM
    Sep 5 at 16:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.