1
$\begingroup$

I am trying to fit a Cox regression model to my time-to event data and have a categorical variable with 5 different levels. I leave one level out as the 'reference'. I also have 2 levels with small number of subjects, and I am getting a warning about these variables (levels) having low variance as follows:

ConvergenceWarning: Column(s) ['xxx'] have very low variance. This may harm convergence. 1) Are you using formula's? Did you mean to add '-1' to the end. 2) Try dropping this redundant column before fitting if convergence fails.

And my model does not converge. How should I handle this case?

I am using CoxTimeVaryingFitter from lifelines library in Python.

Improve this question
$\endgroup$
2
  • $\begingroup$ First thing I would do is answer the question the program asked ""did you mean to add -1 at the end?" $\endgroup$
    – Peter Flom
    Commented Jul 9 at 23:13
  • $\begingroup$ I am not using a formula @PeterFlom $\endgroup$
    – smgtkn
    Commented Jul 10 at 14:58

1 Answer 1

Reset to default
1
$\begingroup$

In a Cox regression, categories with small N may have "no event" at all, so all (the few) cases in the category are censored. In that case you could conclude that the hazard is infinitesimally small compared to other hazards or the one in the reference category. So check if this is so first! This phenomenon is called "separation".

Improve this answer
$\endgroup$
3
  • $\begingroup$ This is not the case, in fact they have more 'events' than the other categories. There are very little number of elements from those categories compared to others. For reference: There are around 100K subjects in other categories and less than 1K subjects in the 'low variance' columns $\endgroup$
    – smgtkn
    Commented Jul 10 at 15:01
  • $\begingroup$ What is "xxx" in the message? Is this generated by the software or a real column name you used as independent? If real, then you probably have checked its variance already. $\endgroup$
    – BenP
    Commented Jul 11 at 7:01
  • $\begingroup$ It is the level of the categorical variable's name, I didn't use the real column name. Think of it as 'level_0'. Sorry for the confusion. That column has indeed low variance since most of the subjects have 0 (binary encoded) on the column for that level as they are from different categories. However, there are still subjects that are from level_0, and it is logically different than the other categories so I cannot simply merge it with other categories. Also, subjects from level_0 experience the event 'proportionally' more than the other categories. @BenP $\endgroup$
    – smgtkn
    Commented Jul 11 at 9:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.