# Unstable logistic regression when data not well separated

There are some good answers discussing convergence issues of logistic regression when the data are well separated here and here. I am wondering what can cause convergence issues when the data are not well separated.

As an example, I have the following data, df

   y       x1         x2
1  0 66.06402 -1.0264739
2  1 58.40813  0.2887934
3  1 58.58011  0.2626232
4  0 59.05929 -0.5286438
5  0 55.81817 -1.3184894
6  0 58.00018 -0.8445602
7  1 69.53926 -1.1018149
8  0 55.73621 -0.9000901
9  1 79.80170  0.6690657
10 0 55.40042  0.6600415
11 0 57.42124 -0.7237973
12 1 78.22012 -0.8121816
13 0 53.54296  0.2265636
14 1 56.14096  0.4216436
15 1 66.90146  0.6189839
16 0 50.40008  0.4311339


Fitting a logistic regression in R, I am getting a glm.fit: fitted probabilities numerically 0 or 1 occurred warning message even though the data are non-separable

> attach(df)
> safeBinaryRegression::glm(y ~ x1 + x2, family=binomial)

Call:  safeBinaryRegression::glm(formula = y ~ x1 + x2, family = binomial)

Coefficients:
(Intercept)           x1           x2
-82.930        1.395       10.255

Degrees of Freedom: 15 Total (i.e. Null);  13 Residual
NullDeviance:       21.93
Residual Deviance: 5.927    AIC: 11.93
Warning message:
glm.fit: fitted probabilities numerically 0 or 1 occurred


A visual confirmation that the data are in fact non-separable is also included Removing the red point seems resolve the convergence issues, however I am at a bit of a loss for why this is.

> df2 <- df[-c(9),]
> detach(df)
> attach(df2)
> safeBinaryRegression::glm(y ~ x1 + x2, family=binomial)

Call:  safeBinaryRegression::glm(formula = y ~ x1 + x2, family = binomial)

Coefficients:
(Intercept)           x1           x2
-82.930        1.395       10.255

Degrees of Freedom: 14 Total (i.e. Null);  12 Residual
Null Deviance:      20.19
Residual Deviance: 5.927    AIC: 11.93

• You go back and forth between alluding to your data as "not well separated" and "separable": are you sure you have used these terms consistently? Regardless, this is not instability or a "convergence issue." It literally is what the warning message states: some of the fitted values are exactly (within floating point error) equal to 1. Check the fit at the red point! If you would use stats::glm, then the object it returns will have elements converged, boundary, and control that supply relevant details of how the algorithm terminated. Perhaps safeBinaryRegression::glm does the same. – whuber Dec 28 '17 at 15:19
• @whuber thank you, I was wrongly referring to data as separable in several places where I meant non separable. Fixed inline. – mgilbert Dec 28 '17 at 15:46

The warning about "fitted probabilities numerically 0 or 1" might be useful for diagnosing separability, but these issues are only indirectly related.

Here is a dataset and a binomial GLM fit (in gray) where there is enough overlap among the $x$ values for the two response classes that there is little concern about separability. In particular, the estimate of the $x$ coefficient of $2.35$ is modest and significant: its standard error is only $1.1$ $(p=0.03)$. The gray curve shows the fit. Corresponding to values on this curve are their log odds, or "link" function. Those I have indicated with colors; the legend gives the common (base-10) logs. The software flags fitted values that are within $2.22\times 10^{-15}$ of either $0$ or $1$. Such points have white halos around them. All that's going on here is there's such a wide range of $x$ values that for some points, the fit is very, very close to $0$ (for very negative $x$) or very, very close to $1$ (for the most positive $x$). This isn't a problem in this case.

It might be a problem in the next example. Now a single outlying value of $x$ triggers the warning message. How can we assess this? Simply delete the datum and re-fit the model. In this example, it makes almost no difference: the coefficient estimate does not change, nor does the p-value.

Finally, to check a multiple regression, first form the linear combinations of the coefficient estimates and the variables, $x_i\hat\beta$: this is the link function. Plot the responses against these values exactly as above and study the patterns, looking at (a) the degree to which the 1's overlap the 0's (which assesses separability) and (b) the points with extreme values of the link.

Here is the plot for your data: The point at the far right corresponds to the red dot in your figure: the fitted value is $1$ because that dot is far from the area where 0's transition to 1's. If you remove it from the data, nothing changes. Thus, it's not influencing the results. This graph indicates you have obtained a reasonable fit.

You can also see that slight changes in the values of $x_1$ or $x_2$ at a couple of critical points (those near $0$) could create perfect separation. But is this really a problem? It would only mean that the software could no longer distinguish between this fit and other fits with arbitrarily sharp transitions near $x\beta=0$. However, all would produce similar predictions at all points sufficiently far from the transition line and the location of that line would still be fairly well estimated.

• Thank you for the excellent illustration of why "fitted probabilities numerically 0 or 1" may or may not be problematic, particularly the multiple regression diagnostic plot. This is implicitly what I was misunderstanding above, although I didn't necessarily know that at the time of asking. – mgilbert Dec 28 '17 at 19:24
• @mgilbert why not accept this answer? – Haitao Du Dec 28 '17 at 21:03
• I usually wait around a day before accepting an answer (provided a suitable one is provided). My rationale for this is that accepting an answer often stifles the conversation, so I find it best to wait until interest has naturally moved to other questions. The thinking behind this is influenced by the discussion here: meta.stackexchange.com/a/28557 – mgilbert Dec 28 '17 at 21:21
• (+1) Thanks for explaining more clearly how separation and near-perfect prediction are distinct phenomena. I agree that data manipulation is warranted if its sole purpose is to ensure the stability of the fitting algorithm without modifying the statistical results. I also agree that diagnostic statistics play a key role in detecting these cases. I wonder if multivariate instances of this problem are sure to be detected by fitted vs. residual plots, especially when three or more covariates have a mixture of strong negative/positive correlations & effects on Y. – AdamO Dec 28 '17 at 23:34
• @AdamO Good question. Considering that the predicted probabilities have log odds given by $x^\prime\hat\beta$, we see that this warning about "fitted probabilities numerically 0 or 1" is the result of comparing $|x^\prime\hat\beta|$ to an interval $[-a,a]$ for some largish $a\gt 0$. Consequently, any diagnostic that reveals the amplitude of $x^\prime\hat\beta$ will be useful--and completely accurate--for identifying individual observations that trigger this warning. Including some kind of residuals in such plots will help determine which of these observations really matter. – whuber Dec 29 '17 at 13:45

Perfect seperation will cause the optimization not converge, not converge will cause the coefficients to be very large, and the very large coefficient will cause "fitted probabilities numerically 0 or 1".

On the reverse side, "fitted probabilities numerically 0 or 1" does not mean the fitting does not converge. It just means with finite precision system IEEE754, the fitted number is very close to 0 or 1, and the computer cannot differentiate it.

This is very likely to happen, if we have some "outliers" (as demonstrated in @whuber's answer, second figure), here is a simpler example by adding one outlier to mtcars data.

Note the fit1 is fine, but fit2 has the warning.

> d1=mtcars[,c("am","mpg")]
> fit1=glm(am~mpg,d1, family="binomial")

> d2=rbind(d1,c(0,-100))
> fit2=glm(am~mpg,d2, family="binomial")
Warning message:
glm.fit: fitted probabilities numerically 0 or 1 occurred


You are having trouble picturing multidimensional separation. While neither X1 nor X2 separately perfectly predict the Y outcome, together they do. Make use of the coplot to avoid this problem in the future

coplot(y ~ x1 | x2, data=l, panel=panel.smooth) The same recommendations apply that have been described elsewhere for handling such an issue.

• This is not correct, because the data definitely are not linearly separable and applying a GLM shows that convergence does occur. (Your analysis implicitly includes an interaction in the model.) – whuber Dec 28 '17 at 16:47
• @whuber Linear separability applies to linear discriminant analysis. The classifier for logistic regression is quite a bit different as a line in twospace. Also, you can see that it barely just meets the non-separability condition in that case. I think that "fitted probabilities of 0 or 1" warns of separability even if the regression converges. Lastly the coplot does not imply interaction, I am not checking whether the slope of the sigmoid changes in each panel, but rather that the sigmoid in each panel is (practically) perfectly separated which it is. – AdamO Dec 28 '17 at 17:00
• @whuber the "averaged" sigmoid slope (infinity in nearly every panel) is a way of eyeballing the Mantel Haenszel estimator which is a one-step estimate from the logistic regression model. – AdamO Dec 28 '17 at 17:02
• Again: if you would actually fit a GLM to these data, you will find that it does converge. Thus, there really isn't any support for your claims of "barely" separable. Separability in logistic regression is mathematically equivalent to a separating hyperplane for the two classes, so the reference to LDA really isn't germane. – whuber Dec 28 '17 at 17:18
• @whuber I have not disagreed that the model converges. The OP is correct in complaining that the estimates are unstable. Our disagreement may be on whether there is such a thing as "gray" separation. Am I right in saying we agree that while these data are not mathematically separated, the estimated odds ratio of 22,000 is because the data very nearly meet an official designation of separated? I think it can be shown that such cases generally result in unstable ORs. – AdamO Dec 28 '17 at 17:47