Creating Survival Dataset with Pythons lifelines using add_covariate_to_timeline creates NaN rows

Question

I'd like to perform a survival analysis on how fast companies achieve a certain Event after their foundation. I have a dataframe with several static and non static covariates over a 5 years observation period. The companies are getting founded within the first 2 Years of observation. I tried to create the input data for lifelines CoxTimeVaryingFitter using to_long_format and add_covariate_to_timeline. Here is some example df:

Year id time_to_event event CV_stat1 CV_stat2 CV_dyn1 CV_dyn2 foundingyear
2010 1  2             0     1        0        3       4       2011
2011 1  2             1     1        0        5       7       2011
2012 1  2             0     1        0        3       8       2011
2013 1  2             0     1        0        6       9       2011
2014 1  2             0     1        0        8       10      2011

Here is my code:

df['Year'] = df.Year - df.foundingyear
df = df[df.Year >= 0]
df['time_to_event'] = df.time_to_event - df.foundingyear +1

base_df = df[['id','time_to_event', 'event','CV_stat1', 'CV_stat2']]

base_df = to_long_format(base_df, duration_col="time_to_event")

cv = df[['id', 'Year', 'CV_dyn1']]

surv_df = add_covariate_to_timeline(base_df, cv, duration_col="Year", id_col="id", event_col="event")

Unfortunately that results in several NaN rows for most of the companies, which does not make sense

start   CV_stat1    CV_stat2    CV_dyn1     stop    id   event
0       1           0           NaN         0       27   FALSE
0       1           0           NaN         0       27   FALSE
0       1           0           NaN         0       27   FALSE
0       1           0           NaN         1       27   FALSE
1       1           0           33,16667    2       27   FALSE

The event should be True for period 1 for that id. What am I doing wrong so that the NaN rows are included?

Answer 1

I think there is more than one problem with the design.

1. I first noticed that you have duplicate (start, stop, id) rows in the final dataframe, which should not be there. These should be unique (an subject can only pass through a time period once!).

2. This suggests your "times" are off, originally. Forget the concept of time_to_event - instead model the "age" of the company. So

Year id age event CV_stat1 CV_stat2 CV_dyn1 CV_dyn2 foundingyear
2010 1  1       0     1        0        3       4       2011
2011 1  2       1     1        0        5       7       2011
2012 1  3       0     1        0        3       8       2011
2013 1  4       0     1        0        6       9       2011
2014 1  5       0     1        0        8       10      2011

makes more sense (and we can drop Year and foundingyear).

3. But looking at this dataframe, we have the "event" occur in year 2, but we have observations in the years after - how can this be?

4. Finally, your original dataframe is almost already in a time-varying format (after a few edits), so you may not need to use add_covariate_to_timeline, which is designed to introduce a new column into the dataframe.

Let me know if this helps or you need more clarification!