Predictions using CoxTimeVaryingFitter for survival analysis in Python

Question

For a customer churn analysis , i am building a time varying cox model in Python (available under lifelines package) to predict survival probabilities. The model object CoxTimeVaryingFitter() currently does not support or include functions to predict survival probability directly.

On the contrary, they have baseline_cumulative_hazard_ that shows baselime cumulative hazard across tenure and predict_partial_hazard() to predict partial hazard rate $\exp\{(\mathbf{x}−\overline{\mathbf{x}})^T\mathbf{β}\}$.

Can anyone advise me on how i can use these two outputs from model object to calculate survival rate for all the customers.

Does this the below approach makes sense?

$$S(t) = \{\exp(-H(t))\}^{exp(\mathbf{\beta}^T\mathbf{x})}$$

where $H(t)$ = baseline cumulative hazard and $\mathbf{\beta}$ = coefficient and $\mathbf{x}$ = co variate

If so , does this mean , i have to compute survival rate for all customers across each tenures based on baseline and covariate value?

Thanks

Reference: https://lifelines.readthedocs.io/en/latest/fitters/regression/CoxTimeVaryingFitter.html?highlight=predict_partial_hazard#lifelines.fitters.cox_time_varying_fitter.CoxTimeVaryingFitter.predict_partial_hazard

Answer 1

Simply put: you can't predict for epistemological reasons. Why is that? To predict, you must have time-varying $X$, that is, $X(t)$. But, if you have $X(t)$, then you must be making measurements on a customer, so they are not dead! Therefore, the fact that you have or don't have $X(t)$ tells you if the customer is alive or not, making any "survival" prediction moot.

Answer 2

I generally agree with the sentiment voiced in the answer from @Cam.Davidson.Pilon (which I upvoted). It's too easy to introduce survivorship bias that way.

Nevertheless, there are circumstances in which it can make sense to use time-varying covariates for predictions from Cox models. Therneau and Grambsch, in "Modeling Survival Data: Extending the Cox Model" (Springer, 2000), show how to do so with the survival package (S-Plus/R) in Section 10.2.4, then discuss (pages 271-2):

Some authors have argued that although a hazard function may be mathematically defined for a model such as the one above [with time-varying covariates], that "this hazard bears no relationship to a survivor function"... We tend to disagree, and could imagine some hypothetical "cohort" of subjects who follow a given path, losing members to death along the way.

This seems safe when the time-varying covariate is predictable and could be defined even if an event had occurred (e.g., a non-linear effect of age, a time-transformed baseline covariate to deal with non-proportional hazards). Therneau and Grambsch do go on to warn:

Examples where the covariate path is guaranteed are the exception, however. A major concern ... is whether the hypothetical path represents any patient at all. Survival curves based on a time-dependent covariate must be used with extreme caution.

Answer 3

This seems fairly straightforward, if I'm understanding your question correctly. Just make predictions like a non-time-varying model once its fit.

If you're making predictions with a Time-Varying Cox Regression model, you obviously don't know what the time-varying coefficients will be in the future. But you do know what the most recent value is, and that the patient was alive when you last interacted with them.

So why not just assume the last measured value of the coefficient is what it will stay for the foreseeable future? This at least gives a starting point for a prediction.

You can just use the predict_cumulative_hazard method from SemiParametricPHFitter directly:

from lifelines import CoxTimeVaryingFitter
from lifelines.fitters.coxph_fitter import SemiParametricPHFitter

# Create and fit model
ctv = CoxTimeVaryingFitter()
ctv.fit(...)

# Set method
f = SemiParametricPHFitter.__dict__["predict_cumulative_hazard"]

# Make predictions
cum_hazard_preds = f(ctv, X, times, conditional_after)
survival_preds = np.exp(-1 * cum_hazard_preds)

No guarantee this continues to work in future versions, but it works now.