7
$\begingroup$

For a customer churn analysis , i am building a time varying cox model in Python (available under lifelines package) to predict survival probabilities. The model object CoxTimeVaryingFitter() currently does not support or include functions to predict survival probability directly.

On the contrary, they have baseline_cumulative_hazard_ that shows baselime cumulative hazard across tenure and predict_partial_hazard() to predict partial hazard rate $\exp\{(\mathbf{x}−\overline{\mathbf{x}})^T\mathbf{β}\}$.

Can anyone advise me on how i can use these two outputs from model object to calculate survival rate for all the customers.

Does this the below approach makes sense?

$$S(t) = \{\exp(-H(t))\}^{exp(\mathbf{\beta}^T\mathbf{x})}$$

where $H(t)$ = baseline cumulative hazard and $\mathbf{\beta}$ = coefficient and $\mathbf{x}$ = co variate

If so , does this mean , i have to compute survival rate for all customers across each tenures based on baseline and covariate value?

Thanks

Reference: https://lifelines.readthedocs.io/en/latest/fitters/regression/CoxTimeVaryingFitter.html?highlight=predict_partial_hazard#lifelines.fitters.cox_time_varying_fitter.CoxTimeVaryingFitter.predict_partial_hazard

$\endgroup$

3 Answers 3

6
$\begingroup$

Simply put: you can't predict for epistemological reasons. Why is that? To predict, you must have time-varying $X$, that is, $X(t)$. But, if you have $X(t)$, then you must be making measurements on a customer, so they are not dead! Therefore, the fact that you have or don't have $X(t)$ tells you if the customer is alive or not, making any "survival" prediction moot.

$\endgroup$
3
  • $\begingroup$ Thanks for your reply!! In that case , which use cases will require a time varying model if predictions cannot be carried on unseen data. Apologies , if the questions sounds very basic. $\endgroup$
    – swat
    Commented Sep 29, 2020 at 8:32
  • 1
    $\begingroup$ Thinking on my above question , i could use this model to understand in how many time periods n a customer is hiighly likely to churn given that the i know that the customer is active today and i know his covariates. I would like to understand how a TV Cox can be used for predictions here? Thanks $\endgroup$
    – swat
    Commented Sep 29, 2020 at 9:00
  • $\begingroup$ Am I interpreting this answer overly broadly? The poster's question was about generating survival curves and it would seem prima facie that it is moot to ask about a survival rate for a patient whom we know to be alive. However, is there a reason why we couldn't predict a hazard rate for such a patient? If so, it would seem there are things that we can predict using a fitted time-varying Cox PH model. $\endgroup$
    – matmat
    Commented Jan 3, 2022 at 23:22
4
$\begingroup$

I generally agree with the sentiment voiced in the answer from @Cam.Davidson.Pilon (which I upvoted). It's too easy to introduce survivorship bias that way.

Nevertheless, there are circumstances in which it can make sense to use time-varying covariates for predictions from Cox models. Therneau and Grambsch, in "Modeling Survival Data: Extending the Cox Model" (Springer, 2000), show how to do so with the survival package (S-Plus/R) in Section 10.2.4, then discuss (pages 271-2):

Some authors have argued that although a hazard function may be mathematically defined for a model such as the one above [with time-varying covariates], that "this hazard bears no relationship to a survivor function"... We tend to disagree, and could imagine some hypothetical "cohort" of subjects who follow a given path, losing members to death along the way.

This seems safe when the time-varying covariate is predictable and could be defined even if an event had occurred (e.g., a non-linear effect of age, a time-transformed baseline covariate to deal with non-proportional hazards). Therneau and Grambsch do go on to warn:

Examples where the covariate path is guaranteed are the exception, however. A major concern ... is whether the hypothetical path represents any patient at all. Survival curves based on a time-dependent covariate must be used with extreme caution.

$\endgroup$
5
  • $\begingroup$ If I understood your answer correctly, could you give an example of introducing survivorship bias by using a fitted time-varying Cox PH model to predict hazard rate as a function of covariates? $\endgroup$
    – matmat
    Commented Jan 3, 2022 at 23:26
  • $\begingroup$ I'm not sure I entirely follow Therneau and Grambsch's warning. Is there anything fundamentally wrong with generating survival curves from a fitted time-varying Cox PH model? I can see a risk with using the model to extrapolate from the training data, but if this risk is understood, are there other dangers? Why is it important that a particular survival curve represents any real patient? $\endgroup$
    – matmat
    Commented Jan 3, 2022 at 23:33
  • $\begingroup$ @matmat there's nothing wrong with fitting such a model, provided that you understand that associations of covariates with outcome are implicitly assumed to be instantaneous under PH. T&G used bilirubin, prothrombin time and albumin as predictors in their time-varying primary biliary cirrhosis (PBC) example. These tend to change together as liver function declines in PBC. If you specify prothrombin time increasing while the others stayed constant, such a patient probably has a clotting problem independent of liver failure, a disease that isn't the modeled PBC. $\endgroup$
    – EdM
    Commented Jan 4, 2022 at 16:36
  • 1
    $\begingroup$ @matmat the potential survivorship bias is expressed nicely in the other answer, from the author of lifelines. Or, quoting from T&G, "A basic concern is that in order to have a rising bilirubin the subject must be remeasured and ergo, still alive." They suggest their hypothetical-cohort approach to overcome such objections, but you must be careful to make sure that assumed covariate paths through time represent the population that was originally modeled. $\endgroup$
    – EdM
    Commented Jan 4, 2022 at 16:46
  • $\begingroup$ Thank you for your well-written explanations, @EdM. I am modeling the survival time of parts in the field, so, I was at first skeptical that I needed to be quite this rigorous, but your points have given me food for thought. $\endgroup$
    – matmat
    Commented Jan 4, 2022 at 18:23
0
$\begingroup$

This seems fairly straightforward, if I'm understanding your question correctly. Just make predictions like a non-time-varying model once its fit.

If you're making predictions with a Time-Varying Cox Regression model, you obviously don't know what the time-varying coefficients will be in the future. But you do know what the most recent value is, and that the patient was alive when you last interacted with them.

So why not just assume the last measured value of the coefficient is what it will stay for the foreseeable future? This at least gives a starting point for a prediction.

You can just use the predict_cumulative_hazard method from SemiParametricPHFitter directly:

from lifelines import CoxTimeVaryingFitter
from lifelines.fitters.coxph_fitter import SemiParametricPHFitter

# Create and fit model
ctv = CoxTimeVaryingFitter()
ctv.fit(...)

# Set method
f = SemiParametricPHFitter.__dict__["predict_cumulative_hazard"]

# Make predictions
cum_hazard_preds = f(ctv, X, times, conditional_after)
survival_preds = np.exp(-1 * cum_hazard_preds)

No guarantee this continues to work in future versions, but it works now.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.