Cross-validation not converging on Cox PH due to dummies

Question

I'm running a Cox PH model on Python using lifelines.

For some categoric variables (like purpose of loan) my go-to approach was to make it into dummies and use n-1 of those on the model. I have 6 categoric variables like this one.

I have 14090 observations in total.

If I run the model with all observations it does so without a problem, however, when I use the kfold cross validation approach (which is included on the lifelines package), it gives a convergence error.

I tried removing all variables that had a high correlation with each other, but it didn't work. Then, the error output said something like this, for many of the dummies:

ConvergenceWarning: Column(s) ['number_arrears_so_far_15'] have very low variance. This may harm convergence. 1) Are you using formula's? Did you mean to add '-1' to the end. 2) Try dropping this redundant column before fitting if convergence fails.

ConvergenceWarning: Column number_arrears_so_far_12 have very low variance when conditioned on death event present or not. This may harm convergence. This could be a form of 'complete separation'. For example, try the following code:

events = df['cured'].astype(bool)
print(df.loc[events, 'number_arrears_so_far_12'].var())
print(df.loc[~events, 'number_arrears_so_far_12'].var())

A very low variance means that the column number_arrears_so_far_12 completely determines whether a subject dies or not.

So I tried removing all those variables that showed up in these errors. It finally worked, however I find it very strange to have "gaps" in a variable (i.e. I have all categories in "number arrears so far" except 12 and 15, the same for some other variables) and I'm not sure it's correct to do it.

Finally, I thought maybe it's not necessary to make the categoric variables into dummies for survival analysis, for any reason (if this is true, could someone confirm it?) left the categoric variables as they are and tried to run it, and it worked.

Now, the difference in performance between the cross validation without some dummies and the cross validation with the categoric variables as-is is quite large:

## WITHOUT SOME DUMMIES
scores = k_fold_cross_validation(cph, daten, 'length_of_arrears', event_col='cured', k=5, scoring_method='concordance_index')

scores
Out[205]: 
[0.6983024508915319,
 0.6821232019611653,
 0.6818250610284223,
 0.6912730691207758,
 0.6923946433989651]

## WITH THE CATEGORIC VARIABLES (no dummies)
scores = k_fold_cross_validation(cph, daten, 'length_of_arrears', event_col='cured', k=5, scoring_method='concordance_index')

print(scores)
[0.9097667326739138, 0.9150128684751608, 0.9155633135874914, 0.9127594168524475, 0.9093496576107671]

Is there any explanation for this? Could someone point me in the right direction please? I'm very new to the topic and I'm trying this by myself.

Answer 1

Rather than a problem with multicollinearity, I suspect that in your cross-validation (CV) you end up with some data folds that lack events for at least one level of one or more categorical predictors in the fold's training set. In that case the Cox model will not converge for that fold.

One way to proceed in general would be to build a try/catch step into the CV and discard folds for which the model doesn't converge. That's probably not a good choice for a single k-fold CV, but might be acceptable with repeated k-fold CV based on multiple different random assignments of folds, or with bootstrap resampling.

In this particular case, however, I think that @DWin, in a comment, has put a finger upon a big part of the problem: you seem to be categorizing at least one predictor (number_arrears_so_far) that might better be handled as continuous. That is not generally a good choice; in this situation, it probably leads to folds that lack events for certain predictor categories in some fold training sets. Furthermore, if you just broke that predictor up into dummies as a multi-category predictor, you are throwing away the information provided by the natural ordering within the predictor. If it's reasonable to treat that predictor as continuous, doing so should minimize the problem you show here.