I am starting out with survival analysis, and am confused about some things specifically in Cox regression. I have not found a tutorial that explains these things clearly, that's why I'm posting this question here, hoping for some clarity.
CONFUSION #1:
Say, I'm trying to predict time to death of some individuals. Specifically, I want to know the probability of death of each individual in the data set who is not yet dead after t
days/months/years from the current day. The data set contains info about a number of people like age, ethnicity, income, date of birth, date of death and some other fields. If I want to use the Cox proportional hazards model for this, I would need to derive the time-to-event field, in this case TIME_TO_DEATH
, right? How should this be done? For the people who have already died, I can calculate this as the difference between the DATE_OF_BIRTH
and DATE_OF_DEATH
columns; but how do I calculate this for the alive individuals whose DATE_OF_DEATH
values are empty?
- Do I even need to calculate this - should it be left empty for the alive individuals?
- Do I assume the
DATE_OF_DEATH
for the alive individuals as the current day (or the last observation day for the dataset), in which case theTIME_TO_DEATH
becomes identical to theAGE
column? - If these approaches are wrong (which I suspect they are), then what's the correct approach to calculate the
TIME_TO_DEATH
for the alive individuals? I have not found any article on Cox regression talk about this.
CONFUSION #2:
Assuming we have somehow resolved confusion #1, I would do something like this for the regression, right?
import pandas as pd
import lifelines
data = pd.read_csv('Dataset.csv') # contains age, ethnicity, income, date of birth, etc
# the event duration column is 'TIME_TO_DEATH' - how to calculate this?
# the event column is 'DEATH' - 1 if death occurred, 0 if still alive
cox = lifelines.CoxPHFitter()
cox.fit(df=data, duration_col='TIME_TO_DEATH', event_col='DEATH', show_progress=True)
Now, I would need to validate the model to know its accuracy, right? How do I do that?
If I split the data into training and testing sets:
- should I do a random split, so that the training and testing sets contain info about both dead and alive individuals, and then do
cox.fit(df=data_train, ...)
? In that case, how do I validate the model for the alive individuals (as I don't have the ground truthTIME_TO_DEATH
values for them)? - should I take the dead individuals as the training set and predict for the alive ones? This way, I can at least validate the model (since I have the
TIME_TO_DEATH
values for the dead individuals), but if the number of dead individuals in the data set is, say, only 20-30%, then it'd probably not be a good model (as typically we take 70-80% of the data as training data for most machine learning models).
- should I do a random split, so that the training and testing sets contain info about both dead and alive individuals, and then do
Should I do something similar to A/B testing, where I construct the training data set by filtering the data to contain info up to
t
time, and the testing data set to contain info aftert
time until now? What I mean is, say t=3 months. Then, should I filter out the records in the data set that happened in the last 3 months and train the model on that, and then evaluate the model on the records that happened in the last 3 months?Another idea that I can think of, although it won't be Cox regression anymore, is to divide the time period into
n
divisions, and doing logistic regression on each of these divisions to answer with a yes/no to the question, "will this individual die in thei*n
th time interval, where i={0,1,2,...}?" That is, if say n=6 months, will this individual die in the coming 0 months (i.e., immediately), 6 months, 12 months, 18 months, etc? The only problem with this method that I can think of so far, apart from havingn
models for each of then
time divisions instead of 1 model for all the data, is sampling noise - if there is a very less number of records in one or more of these time intervals, then the model for those time intervals wouldn't be too robust. Consequently, we wouldn't be able to maken
too small. For example, we couldn't haven
as 1 week in my data set, as there are only 5-10 people who die each week.