I've got some data about airline flights (in a data frame called flights) and I would like to see if the flight time has any effect on the probability of a significantly delayed arrival (meaning 10 or more minutes). I figured I'd use logistic regression, with the flight time as the predictor and whether or not each flight was significantly delayed (a bunch of Bernoullis) as the response. I used the following code...

flights$BigDelay <- flights$ArrDelay >= 10
delay.model <- glm(BigDelay ~ ArrDelay, data=flights, family=binomial(link="logit"))
summary(delay.model)

...but got the following output.

> flights$BigDelay <- flights$ArrDelay >= 10
> delay.model <- glm(BigDelay ~ ArrDelay, data=flights, family=binomial(link="logit"))
Warning messages:
1: In glm.fit(x = X, y = Y, weights = weights, start = start, etastart = etastart,  :
  algorithm did not converge
2: In glm.fit(x = X, y = Y, weights = weights, start = start, etastart = etastart,  :
  fitted probabilities numerically 0 or 1 occurred
> summary(delay.model)

Call:
glm(formula = BigDelay ~ ArrDelay, family = binomial(link = "logit"),
    data = flights)

Deviance Residuals:
       Min          1Q      Median          3Q         Max
-3.843e-04  -2.107e-08  -2.107e-08   2.107e-08   3.814e-04

Coefficients:
            Estimate Std. Error z value Pr(>|z|)
(Intercept)  -312.14     170.26  -1.833   0.0668 .
ArrDelay       32.86      17.92   1.833   0.0668 .
---
Signif. codes:  0 â***â 0.001 â**â 0.01 â*â 0.05 â.â 0.1 â â 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 2.8375e+06  on 2291292  degrees of freedom
Residual deviance: 9.1675e-03  on 2291291  degrees of freedom
AIC: 4.0092

Number of Fisher Scoring iterations: 25

What does it mean that the algorithm did not converge? I thought it be because the BigDelay values were TRUE and FALSE instead of 0 and 1, but I got the same error after I converted everything. Any ideas?

  • 1
    not sure I deserve the "accept". @Conjugate Prior's answer explained what was wrong with your model. I thought it worth explaining the warning you mentioned in terms of the algorithm. – Gavin Simpson Dec 10 '10 at 17:11
  • 3
    If you have the actual delay times, you are likely to get better information by modeling them, rather than reducing them to a binary variable. – whuber Dec 10 '10 at 17:18
  • related question – user603 Dec 13 '12 at 18:24
  • you can try glm1() function. It overcome the problem converge – user36030 Dec 11 '13 at 12:31
up vote 34 down vote accepted

First thought: Perfect separation, meaning the predictor is 'too good', the logits go to +/- infinity and everything falls over.

Second thought: Does the code do what you think it does? Your variable names don't seem to quite match your description. You might elaborate what the data is more precisely, since it looks like you might be trying to predict something with itself.

  • Would the ridiculously small residual deviance support the idea of perfect separation? When doing a lack-of-fit test, you expect this value to follow a chi-square distribution with 2291291 degrees of freedom, right? – Daniel Standage Dec 10 '10 at 16:57
  • 3
    @Daniel; yes, given the Null device you have perfectly explained the response. Notice too the large standard errors, so that even though you predict the data perfectly, the Wald test is suggesting that the covariate is not significant. I believe the latter phenomenon is known as the Hauck Donner effect after the authors who noted this issue with the binomial GLM. – Gavin Simpson Dec 10 '10 at 17:02
  • Prior Both of your thoughts were correct, but your second thought gets at the real problem. I should have seen it when you told me the first time, but seeing it again from Gavin made me look closer. The formula should be BigDelay ~ AirTime. Silly mistake. – Daniel Standage Dec 10 '10 at 17:07
  • Nice catch! – whuber Dec 10 '10 at 17:19

glm() uses an iterative re-weighted least squares algorithm. The algorithm hit the maximum number of allowed iterations before signalling convergence. The default, documented in ?glm.control is 25. You pass control parameters as a list in the glm call:

delay.model <- glm(BigDelay ~ ArrDelay, data=flights, family=binomial,
                   control = list(maxit = 50))

As @Conjugate Prior says, you seem to be predicting the response with the data used to generate it. You have complete separation as any ArrDelay < 10 will predict FALSE and any ArrDelay >= 10 will predict TRUE. The other warning message tells you that the fitted probabilities for some observations were effectively 0 or 1 and that is a good indicator you have something wrong with the model.

The two can warnings go hand in hand. The likelihood function can be quite flat when some $\hat{\beta}_i$ get large, as in your example. If you allow more iterations, the model coefficients will diverge further if you have a separation issue.

  • Could you explain what exactly do you mean by model convergence here? – Bach Feb 6 '16 at 21:50
  • By convergence I mean that the parameters being estimated in the model don't change (or only change less than some small tolerance) between iterations. Here the parameters get increasingly large and fitting stops because of the limit on iterations but the parameter estimates changed a lot between the penultimate and the last iterations and as such haven't converged. – Gavin Simpson Feb 7 '16 at 17:28

You could try to check if Firth's bias reduction works with your dataset. It is a penalized likelihood approach that can be useful for datasets which produce divergences using the standard glm package. Sometimes it can be used instead of eliminating that variable which produces complete/almost complete separation.

For the formulation of the bias reduction (the $O(n^{-1})$-term in the asymptotic expansion of the bias of the maximum likelihood estimator is removed using classical cumulants expansion as motivating example) please check http://biomet.oxfordjournals.org/content/80/1/27.abstract

Firth's bias reduction is implemented in the R-package logistf: http://cran.r-project.org/web/packages/logistf/logistf.pdf

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.