I want to evaluate my Cox model using cross validation for which lifelines package does not support. So I must use the sklearn adapter. However, when I fit my cox model using the sklearn adapter, I get a convergence error. Inexplicably, I do not get a convergence error when I fit my cox model on the exact same data with the exact same parameters using the CoxPHfitter directly.

X = test_data.drop('DxToFollowup', axis=1) 
Y = test_data['DxToFollowup']
CoxRegression = sklearn_adapter(CoxPHFitter, event_col='IsDead')
sk_cph = CoxRegression(penalizer=.1)

scores = cross_val_score(sk_cph, X, Y, cv=3) 

Give me the error

ConvergenceError: Convergence halted due to matrix inversion problems. 
Suspicion is high `collinearity. Please see the following tips in the lifelines documentation: 
https://lifelines.readthedocs.io/en/latest/Examples.html#problems-with-convergence-in-the-cox-proportional-hazard-modelMatrix is singular.

However when I run

cph = CoxPHFitter(penalizer=0.1)
cph.fit(test_data, duration_col='DxToFollowup', event_col='IsDead', show_progress=True)

It converges with no issues. I have even tried reducing the penalty to 0.000001 but this does not help.

  • $\begingroup$ Likely the reason is the cv=3 - it's splitting the original dataset into smaller datasets, and it's possible that training on these smaller datasets produce a singularity. How large is your dataset? $\endgroup$ Commented May 16, 2020 at 2:52
  • $\begingroup$ 119 observations 104 events. $\endgroup$
    – Mattreex
    Commented May 16, 2020 at 3:51
  • 1
    $\begingroup$ Try a higher cv. How many columns are there? $\endgroup$ Commented May 16, 2020 at 13:35
  • $\begingroup$ That's interesting. Choosing cv=5 seems to work. There are 23 columns. I have 6 continuous variables and 5 categorical variables. I found that one variable in particular was causing the issue. I am not sure why 5 works, but with cv=5 I can keep all the variables $\endgroup$
    – Mattreex
    Commented May 16, 2020 at 17:10
  • $\begingroup$ Are you dropping one of the categories in each categorical variable? Another problem could be i) a rare category that isn't present in the training df (i.e. all 0s), or ii) because there are 17 categorical variables, and 78 rows (in cv=3), it's possible that two of the encoded variables become co-linear. $\endgroup$ Commented May 16, 2020 at 18:54


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.