64
$\begingroup$

I'm trying to predict a binary outcome using 50 continuous explanatory variables (the range of most of the variables is $-\infty$ to $\infty$). My data set has almost 24,000 rows. When I run glm in R, I get:

Warning messages:  
1: glm.fit: algorithm did not converge  
2: glm.fit: fitted probabilities numerically 0 or 1 occurred

I've read the other responses that suggest perfect separation might be occurring, but I'm confident that isn't the case in my data (though quasi-complete separation could exist; how can I test to see if that's the case?). If I remove some variables, the "did not converge" error might go away. But that's not always what happens.

I tried using the same variables in a bayesglm function and got the same errors.

What steps would you take to figure out exactly what's going on here? How do you figure out which variables are causing the problems?

Improve this question
$\endgroup$
7
  • 7
    $\begingroup$ Why are you confident that separation isn't occurring? In the bayesglm paper, they argue that separation is "a common problem, even when the sample size is large and the number of predictors is small" $\endgroup$
    – David J. Harris
    Commented Dec 13, 2012 at 5:26
  • 2
    $\begingroup$ Another thought: bayesglm tries to avoid separation by adding a prior, but with 24,000 rows, the prior is probably getting swamped by the likelihood. Try shrinking prior.scale, possibly by a large amount. Also consider increasing the prior's degrees of freedom, which will help rule out large values associated with separation. $\endgroup$
    – David J. Harris
    Commented Dec 13, 2012 at 5:31
  • $\begingroup$ Thanks for the suggestions David. I don't think separation is occurring because when I sort each of the explanatory variables, the dependent variable isn't always true or false for high or low values of the explanatory variables. Unless this is considered separation: the dependent variable is true for all x7 > 32 but x7 is only > 32 in 10 cases. Is there a way to verify the separation outside of a logistic regression? Or see which variable is causing the separation? I tried your bayesglm suggestions (I set prior.scale to 1 and prior.df to Inf) and still got the Hauck Donner errors. $\endgroup$
    – Dcook
    Commented Dec 13, 2012 at 7:27
  • 1
    $\begingroup$ related questions $\endgroup$
    – user603
    Commented Dec 13, 2012 at 18:23
  • $\begingroup$ "How do you figure out which variables are causing the problems?" Binary-search is always a good fallback. You only have 50 variables, so if it's perfectly separated by one individual variable, 6 iterations will find the culprit. If it's two variables, at most 49+6=55 iterations will find it, worst-case. $\endgroup$
    – smci
    Commented Feb 2, 2017 at 9:03

1 Answer 1

Reset to default
60
$\begingroup$

With such a large design space ($\mathbb{R}^{50}$!) it is possible to get perfect separation without having separation in any of the variable taken individually. I would even second David J. Harris's comment in saying that this is likely.

You can easily test whether your classes are perfectly separated in your design space. This boils down to solving a linear programming problem. An R implementation of this 'test' (not a test in the statistical sense of the term) is implemented in the safeBinaryRegression package.

If it turns out that separation is indeed the issue, and if you are only interested in a plain vanilla use of glm (e.g. glm is not called by a higher level function but by you), then there is an R implementation of an algorithms that slightly modifies the classical one to make it 'robust' against separation. It is implemented in the hlr package

Improve this answer
$\endgroup$
11
  • 5
    $\begingroup$ Very cool and useful answer! I'll have to look into those packages. (+1) $\endgroup$
    – Peter Flom
    Commented Dec 13, 2012 at 11:40
  • 1
    $\begingroup$ FWIW here is a description of another robust algorithm: win-vector.com/blog/2012/10/rudie-cant-fail-if-majorized $\endgroup$
    – Alex
    Commented Dec 13, 2012 at 17:59
  • 2
    $\begingroup$ @Alex: thanks for the link. If glm is not converging because of bad starts then I can see how this method will help with that. On the other hand, if the problem is caused by perfect separation it is not clear to me how the MM idea would address that. I was wondering whether you could comment on this (i can eventually post this as a separate question). $\endgroup$
    – user603
    Commented Dec 13, 2012 at 18:09
  • 1
    $\begingroup$ Thanks for the answer @user603! I used safeBinaryRegression and separation was indeed occurring with several of the variables. Then I tried using MEL in the hlr package to build a model robust to this separation. However, the coefficients are huge (as they would be when separation occurs in normal glm) and here are the df and deviance numbers: Degrees of Freedom: 19112 Total (i.e. Null); 19063 Residual Null Deviance: 24990 Residual Deviance: 626000 AIC: 626000 Do you think I did something wrong? $\endgroup$
    – Dcook
    Commented Dec 14, 2012 at 3:59
  • 2
    $\begingroup$ The question is here for the record. $\endgroup$
    – Arnaud Mortier
    Commented May 27, 2021 at 16:48

Not the answer you're looking for? Browse other questions tagged or ask your own question.