# Testing the validity of a Cox Time-Varying regression model in Python Lifelines

Using the lifelines library for python, I've fitted a Cox Time-varying regression to some customer data, to see which coefficients have an effect on customer churn. The dataset is a combination of time-independent and time-dependent variables. The time-dependent variables are measured at the end of each month, for 6 years.

Inspecting the model output, almost 50% of all the coefficients are significant at the 0.05% level, which to me sounds like there are at least some false positives.

While the lifelines library offers a great function for checking the general assumptions for ordinary cox proportional hazard, the CoxTimeVaryingFitter has no such solution. I have also not found any good sources for checking time-varying models in the lifelines documentation or elsewhere.

Is there some general solution or function I could use? If there is a potential solution in R or other statistical software that would also be very helpful!

First, don't worry too much about having a lot of "statistically significant" predictors if you have a large data set (as you presumably do with customer churn analysis). With a large enough data set almost any predictor might show a "statistically significant" association with outcome, even if it isn't of much practical significance. That's like the problem with normality testing: with a large enough real-world data set you will in practice tend to find "statistically significant" deviations from normality that don't fundamentally matter.

In terms of validating a model with time-varying covariate values, I don't know many details of what's possible in lifelines, but in principle the tools for evaluating Cox models can be applied to models with time-varying covariates. The problem is that some of those tools implicitly involve making predictions. For example, an optimism bootstrap builds the model on multiple bootstrapped samples of the data and evaluates the models' predictions on the full data set. Predictions with time-varying covariates, however, aren't straightforward and can easily lead to circular reasoning and violations of causality.

The lifelines author has put this quite clearly on another page on this site:

you can't predict for epistemological reasons... To predict, you must have time-varying $$X$$, that is, $$𝑋(𝑡)$$. But, if you have $$𝑋(𝑡)$$, then you must be making measurements on a customer, so they are not dead!

That said, some but not all survival tools in R can nevertheless make predictions from a time-varying covariate model. For example, the Predict() function in the rms package won't handle them, but its survest() function and the standard survival package survfit() function can. The rms validate() function will perform an optimism bootstrap on such a model, although I couldn't readily get its calibrate() function to work.

Standard tools for checking proportional hazards (PH) work with these packages (as I suspect they might in lifelines, as the usual scaled Schoenfeld residuals test for PH just uses model-expected versus observed covariate values at event times and thus doesn't involve event predictions). I don't immediately see a reason why you couldn't use martingale residuals to evaluate the functional form of a continuous time-varying covariate (although I haven't done that myself and might be missing something important).

The question is just what is being predicted or validated. With data and analysis in the (startTime, stopTime, event) counting-process format, these functions evidently treat each line of data as a separate observation, left truncated at startTime with an event or censoring at stopTime. That makes sense from the way that the data are modeled in situations with at most a single event per individual. As the R survival time-dependence vignette (a superb introduction to handling different types of time dependence in Cox models) says near the end of Section 2, in that case:

The likelihood equations at any time point use only one copy of any subject, the program picks out the correct row of data at each time.

So in that situation it makes sense to think about separate data lines as independent (albeit time-limited) observations.

It's possible (if you pay close attention to some data-formatting quirks) to get per-subject predictions from the predict.coxph() function in the survival package. Those seem to be based on predicting for all data rows and then summing all rows associated with a subject, however, so they might not make sense for all types of predictions. I'm not at all sure how well these methods will work with multi-state models involving time-varying covariates, in which intra-subject correlations need to be taken into account.

Finally, be very careful in applying these models to customer churn. A Cox model evaluates covariate values precisely at event times. A covariate associated with the specific time of loss of a customer thus might be like the following reverse-causality situation in clinical survival analysis (Section 3.5 of the vignette):

It is sometimes the case that a covariate measured soon before death is not a predictor of death but rather is simply a marker for an event that is already in progress... A simple example would [be] the time dependent covariate “have called the family for a final visit”.

If you are modeling customer churn in an attempt to prevent it in the future, you don't want to be modeling something like the "last straw" that finally pushed the customer to abandon your enterprise. With time-varying covariates, think carefully about what you are modeling, what you are validating, and what you are predicting.

• Wow! This is some really good information, thanks a lot! We decided to make the switch from lifelines to R with the survival library, which works great since we have both Schoenfeld residuals available for the proportional test and Martingale for linearity. The survival time-dependence vignette as well as this tutorial was really helpful. Commented Jun 6, 2021 at 15:03
• It would seem that if a prediction is ill-advised, then inference from the same models would be suspect as well. Don’t these cautions on time-dependent covariates undermine the entire idea of time-dependent covariates? Commented Jun 1, 2022 at 3:07
• @ToddD if you accept the premise underlying Cox regression--only the instantaneous values of covariates at event times matter with respect to outcome--then there's no problem with inference based on time-dependent covariates. If anything, time-dependent covariate values are better than baseline values at study entry. With time-dependent covariates your model is based on the covariate values that were actually in place at each event time. Problems come when, for prediction, you start imagining trajectories of future covariate values that aren't possible or violate causality.
– EdM
Commented Jun 1, 2022 at 12:28