I'm running a Cox PH model on Python using
For some categoric variables (like purpose of loan) my go-to approach was to make it into dummies and use n-1 of those on the model. I have 6 categoric variables like this one.
I have 14090 observations in total.
If I run the model with all observations it does so without a problem, however, when I use the kfold cross validation approach (which is included on the
lifelines package), it gives a convergence error.
I tried removing all variables that had a high correlation with each other, but it didn't work. Then, the error output said something like this, for many of the dummies:
ConvergenceWarning: Column(s) ['number_arrears_so_far_15'] have very low variance. This may harm convergence. 1) Are you using formula's? Did you mean to add '-1' to the end. 2) Try dropping this redundant column before fitting if convergence fails. ConvergenceWarning: Column number_arrears_so_far_12 have very low variance when conditioned on death event present or not. This may harm convergence. This could be a form of 'complete separation'. For example, try the following code: events = df['cured'].astype(bool) print(df.loc[events, 'number_arrears_so_far_12'].var()) print(df.loc[~events, 'number_arrears_so_far_12'].var()) A very low variance means that the column number_arrears_so_far_12 completely determines whether a subject dies or not.
So I tried removing all those variables that showed up in these errors. It finally worked, however I find it very strange to have "gaps" in a variable (i.e. I have all categories in "number arrears so far" except 12 and 15, the same for some other variables) and I'm not sure it's correct to do it.
Finally, I thought maybe it's not necessary to make the categoric variables into dummies for survival analysis, for any reason (if this is true, could someone confirm it?) left the categoric variables as they are and tried to run it, and it worked.
Now, the difference in performance between the cross validation without some dummies and the cross validation with the categoric variables as-is is quite large:
## WITHOUT SOME DUMMIES scores = k_fold_cross_validation(cph, daten, 'length_of_arrears', event_col='cured', k=5, scoring_method='concordance_index') scores Out: [0.6983024508915319, 0.6821232019611653, 0.6818250610284223, 0.6912730691207758, 0.6923946433989651] ## WITH THE CATEGORIC VARIABLES (no dummies) scores = k_fold_cross_validation(cph, daten, 'length_of_arrears', event_col='cured', k=5, scoring_method='concordance_index') print(scores) [0.9097667326739138, 0.9150128684751608, 0.9155633135874914, 0.9127594168524475, 0.9093496576107671]
Is there any explanation for this? Could someone point me in the right direction please? I'm very new to the topic and I'm trying this by myself.