56
$\begingroup$

I'm trying to predict a binary outcome using 50 continuous explanatory variables (the range of most of the variables is $-\infty$ to $\infty$). My data set has almost 24,000 rows. When I run glm in R, I get:

Warning messages:  
1: glm.fit: algorithm did not converge  
2: glm.fit: fitted probabilities numerically 0 or 1 occurred 

I've read the other responses that suggest perfect separation might be occurring, but I'm confident that isn't the case in my data (though quasi-complete separation could exist; how can I test to see if that's the case?). If I remove some variables, the "did not converge" error might go away. But that's not always what happens.

I tried using the same variables in a bayesglm function and got the same errors.

What steps would you take to figure out exactly what's going on here? How do you figure out which variables are causing the problems?

$\endgroup$
  • 5
    $\begingroup$ Why are you confident that separation isn't occurring? In the bayesglm paper, they argue that separation is "a common problem, even when the sample size is large and the number of predictors is small" $\endgroup$ – David J. Harris Dec 13 '12 at 5:26
  • 2
    $\begingroup$ Another thought: bayesglm tries to avoid separation by adding a prior, but with 24,000 rows, the prior is probably getting swamped by the likelihood. Try shrinking prior.scale, possibly by a large amount. Also consider increasing the prior's degrees of freedom, which will help rule out large values associated with separation. $\endgroup$ – David J. Harris Dec 13 '12 at 5:31
  • $\begingroup$ Thanks for the suggestions David. I don't think separation is occurring because when I sort each of the explanatory variables, the dependent variable isn't always true or false for high or low values of the explanatory variables. Unless this is considered separation: the dependent variable is true for all x7 > 32 but x7 is only > 32 in 10 cases. Is there a way to verify the separation outside of a logistic regression? Or see which variable is causing the separation? I tried your bayesglm suggestions (I set prior.scale to 1 and prior.df to Inf) and still got the Hauck Donner errors. $\endgroup$ – Dcook Dec 13 '12 at 7:27
  • 1
    $\begingroup$ related questions $\endgroup$ – user603 Dec 13 '12 at 18:23
  • $\begingroup$ "How do you figure out which variables are causing the problems?" Binary-search is always a good fallback. You only have 50 variables, so if it's perfectly separated by one individual variable, 6 iterations will find the culprit. If it's two variables, at most 49+6=55 iterations will find it, worst-case. $\endgroup$ – smci Feb 2 '17 at 9:03
53
$\begingroup$

With such a large design space ($\mathbb{R}^{50}$!) it is possible to get perfect separation without having separation in any of the variable taken individually. I would even second David J. Harris's comment in saying that this is likely.

You can easily test whether your classes are perfectly separated in your design space. This boils down to solving a linear programming problem. An R implementation of this 'test' (not a test in the statistical sense of the term) is implemented in the safeBinaryRegression package.

If it turns out that separation is indeed the issue, and if you are only interested in a plain vanilla use of glm (e.g. glm is not called by a higher level function but by you), then there is an R implementation of an algorithms that slightly modifies the classical one to make it 'robust' against separation. It is implemented in the hlr package

$\endgroup$
  • 4
    $\begingroup$ Very cool and useful answer! I'll have to look into those packages. (+1) $\endgroup$ – Peter Flom Dec 13 '12 at 11:40
  • 1
    $\begingroup$ FWIW here is a description of another robust algorithm: win-vector.com/blog/2012/10/rudie-cant-fail-if-majorized $\endgroup$ – Alex Dec 13 '12 at 17:59
  • 2
    $\begingroup$ @Alex: thanks for the link. If glm is not converging because of bad starts then I can see how this method will help with that. On the other hand, if the problem is caused by perfect separation it is not clear to me how the MM idea would address that. I was wondering whether you could comment on this (i can eventually post this as a separate question). $\endgroup$ – user603 Dec 13 '12 at 18:09
  • $\begingroup$ Excellent (+1)! I, too, will have to look into those packages. $\endgroup$ – jbowman Dec 13 '12 at 19:41
  • 1
    $\begingroup$ Thanks for the answer @user603! I used safeBinaryRegression and separation was indeed occurring with several of the variables. Then I tried using MEL in the hlr package to build a model robust to this separation. However, the coefficients are huge (as they would be when separation occurs in normal glm) and here are the df and deviance numbers: Degrees of Freedom: 19112 Total (i.e. Null); 19063 Residual Null Deviance: 24990 Residual Deviance: 626000 AIC: 626000 Do you think I did something wrong? $\endgroup$ – Dcook Dec 14 '12 at 3:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.