Skip to main content
Cross Validated

Return to Answer

expanded in response to comments and with more details on data and proposed method
Source Link
EdM
EdM
  • 101.5k
  • 11
  • 102
  • 303

Data like theseThese are typically referred to as panel data, with the same individuals evaluated at a set of discrete points in time, 13 time points in this case. The Health and Retirement Study supplements that with "Exit" questionnaires, in which survivors of those who die between interviews provide information about date and cause of death. If your outcome of interest is an event of some typedeath, then thisyou can be thought of as binomial regression that incorporatesmodel this with continuous-time survival methods, incorporating time-varying covariate values available at interview dates prior to death. You evaluate

There are two questions here, one about the probabilitymodeling method and another about predictions.

Modeling

It seems that there was some difficulty in preparing the panel data for the survival modeling software, as somehow you got 13 separate sets of an event atmodels. Although you might have 13 separate "waves" of data from which to draw, you need to combine all the data together for analysis. You should take advantage of the unique individual ID value for each participant that is maintained throughout the Health and Retirement Study to label the data, even if the modeling might not ultimately require that.

For standard survival modeling with time as a function-varying covariate values, each participant you would have separate data rows for all time periods between successive "waves" while in the study. Each row would have the ID, the start time and stop time of the currentinterval between the waves, and the covariate values in place at the beginning of the interval. For those that die, the stop time is set instead to the time corresponding to death. Each row also has an event indicator, say 0 for still alive and 1 for died.

Those data rows should all be collected together into a single file for analysis. The start time of each row thus represents a left-truncated survival time, with either an event (if died) or right censoring (if alive) at the stop time. That's the standard "counting process" data format explained, for example, in this survival vignette.

AlthoughIn a comment, you say "time to event is defined as the time from the start of the survey till the death of a respondent." I'm not familiar withsure that "start of the survey" is a good choice for time = 0 for survival analysis, although if you use age as a covariate I suppose you could argue for that choice. Consider whether you should be using the date when age reached 51 years as time = 0 instead, as that's the earliest age when interviews start.

The method that you cite, it is based on parametric continuous-time modeling of covariates and survival. It seems designed for analysisto incorporate the past history of outcomes in continuous timecovariates into the current event risk, so that you would need to keep track of each individual's identity and history of covariate values. It's not clear

I don't know enough about that this method isto say whether it's appropriate for discrete-timeyour type of data. If it isn't and you still want to model with neural networksThe example in the linked paper used hourly covariate values during stays in an intensive care unit, investigate methods for handling binomial outcomes at fixed times insteadvalues that might reasonably be considered to be continuous. I don't know whether it can properly handle highly discretized panel data like yours or the associated "counting-process" data format.

The answer to your question about repeated measures on the same individuals dependsSome survival models, like proportional-hazards models, only evaluate covariate values that are in place at event times and don't depend on the nature of thepast covariate history (except insofar as it might be incorporated into a value current at an event time, like a cumulative value). If it'sIn modeling a terminal event (likelike death) and you are modeling survival as a function of current covariate values at event times that way, then you can treat all of the observations as independent. If an individual can have multiple events (like infections), you should incorporate information about the and don't need to keep track of individual IDs. The time-dependence vignette of the R survival package explains this in section 2.

Predictions

Regardless of how you ultimately model the data, your question about making predictions still stands. The problem with survival predictions when covariates are varying in time is that you have to assume sets of time-varying covariate values for the predictions.

Second, even if you are willing to make such assumptions to represent the survival of a cohort of patients having a particular trajectory of the covariate values over time, there's a risk of assuming a set of values that could not realistically occur together. See this answer.

It might be possible to model future covariate values for participants as a guide to potentially realistic predictions. There is a field of study devoted to joint modeling of covariates and event times. The methods are implemented in the R JM package and the frailtypack package, among others you can find in the R survival view. That is beyond my expertise. This page and its links might help.

Data like these are typically referred to as panel data, with the same individuals evaluated at a set of discrete points in time, 13 time points in this case. If your outcome of interest is an event of some type, then this can be thought of as binomial regression that incorporates time. You evaluate the probability of an event at each time as a function of the current covariate values and time.

Although I'm not familiar with the method that you cite, it seems designed for analysis of outcomes in continuous time. It's not clear that this method is appropriate for discrete-time data. If it isn't and you still want to model with neural networks, investigate methods for handling binomial outcomes at fixed times instead.

The answer to your question about repeated measures on the same individuals depends on the nature of the event. If it's a terminal event (like death) and you are modeling survival as a function of current covariate values at event times, then you can treat all of the observations as independent. If an individual can have multiple events (like infections), you should incorporate information about the individual. The time-dependence vignette of the R survival package explains this in section 2.

Regardless of how you ultimately model the data, your question about making predictions still stands. The problem with survival predictions when covariates are varying in time is that you have to assume sets of time-varying covariate values for the predictions.

Second, even if you are willing to make such assumptions to represent the survival of a cohort of patients having a particular trajectory of the covariate values over time, there's a risk of assuming a set of values that could not realistically occur together. See this answer.

These are panel data, with the same individuals evaluated at a set of discrete points in time, 13 time points in this case. The Health and Retirement Study supplements that with "Exit" questionnaires, in which survivors of those who die between interviews provide information about date and cause of death. If your outcome of interest is death, then you can model this with continuous-time survival methods, incorporating time-varying covariate values available at interview dates prior to death.

There are two questions here, one about the modeling method and another about predictions.

Modeling

It seems that there was some difficulty in preparing the panel data for the survival modeling software, as somehow you got 13 separate sets of models. Although you might have 13 separate "waves" of data from which to draw, you need to combine all the data together for analysis. You should take advantage of the unique individual ID value for each participant that is maintained throughout the Health and Retirement Study to label the data, even if the modeling might not ultimately require that.

For standard survival modeling with time-varying covariate values, each participant you would have separate data rows for all time periods between successive "waves" while in the study. Each row would have the ID, the start time and stop time of the interval between the waves, and the covariate values in place at the beginning of the interval. For those that die, the stop time is set instead to the time corresponding to death. Each row also has an event indicator, say 0 for still alive and 1 for died.

Those data rows should all be collected together into a single file for analysis. The start time of each row thus represents a left-truncated survival time, with either an event (if died) or right censoring (if alive) at the stop time. That's the standard "counting process" data format explained, for example, in this survival vignette.

In a comment, you say "time to event is defined as the time from the start of the survey till the death of a respondent." I'm not sure that "start of the survey" is a good choice for time = 0 for survival analysis, although if you use age as a covariate I suppose you could argue for that choice. Consider whether you should be using the date when age reached 51 years as time = 0 instead, as that's the earliest age when interviews start.

The method that you cite is based on parametric continuous-time modeling of covariates and survival. It seems designed to incorporate the past history of covariates into the current event risk, so that you would need to keep track of each individual's identity and history of covariate values.

I don't know enough about that method to say whether it's appropriate for your type of data. The example in the linked paper used hourly covariate values during stays in an intensive care unit, values that might reasonably be considered to be continuous. I don't know whether it can properly handle highly discretized panel data like yours or the associated "counting-process" data format.

Some survival models, like proportional-hazards models, only evaluate covariate values that are in place at event times and don't depend on past covariate history (except insofar as it might be incorporated into a value current at an event time, like a cumulative value). In modeling a terminal event like death that way, you can treat all of the observations as independent and don't need to keep track of individual IDs. The time-dependence vignette of the R survival package explains this in section 2.

Predictions

Regardless of how you ultimately model the data, your question about making predictions still stands. The problem with survival predictions when covariates are varying in time is that you have to assume sets of time-varying covariate values for the predictions.

Second, even if you are willing to make such assumptions to represent the survival of a cohort of patients having a particular trajectory of the covariate values over time, there's a risk of assuming a set of values that could not realistically occur together. See this answer.

It might be possible to model future covariate values for participants as a guide to potentially realistic predictions. There is a field of study devoted to joint modeling of covariates and event times. The methods are implemented in the R JM package and the frailtypack package, among others you can find in the R survival view. That is beyond my expertise. This page and its links might help.

Source Link
EdM
EdM
  • 101.5k
  • 11
  • 102
  • 303

Data like these are typically referred to as panel data, with the same individuals evaluated at a set of discrete points in time, 13 time points in this case. If your outcome of interest is an event of some type, then this can be thought of as binomial regression that incorporates time. You evaluate the probability of an event at each time as a function of the current covariate values and time.

Although I'm not familiar with the method that you cite, it seems designed for analysis of outcomes in continuous time. It's not clear that this method is appropriate for discrete-time data. If it isn't and you still want to model with neural networks, investigate methods for handling binomial outcomes at fixed times instead.

The answer to your question about repeated measures on the same individuals depends on the nature of the event. If it's a terminal event (like death) and you are modeling survival as a function of current covariate values at event times, then you can treat all of the observations as independent. If an individual can have multiple events (like infections), you should incorporate information about the individual. The time-dependence vignette of the R survival package explains this in section 2.

Regardless of how you ultimately model the data, your question about making predictions still stands. The problem with survival predictions when covariates are varying in time is that you have to assume sets of time-varying covariate values for the predictions.

That poses two potential problems.

First, if the event is death, then assuming time-varying covariate values implicitly assumes that the individual is still alive through those times. The author of the Python lifelines package for survival analysis thus won't allow the software to do such predictions. See this answer.

Second, even if you are willing to make such assumptions to represent the survival of a cohort of patients having a particular trajectory of the covariate values over time, there's a risk of assuming a set of values that could not realistically occur together. See this answer.