57
$\begingroup$

I'm quite new on this with binomial data tests, but needed to do one and now I´m not sure how to interpret the outcome. The y-variable, the response variable, is binomial and the explanatory factors are continuous. This is what I got when summarizing the outcome:

glm(formula = leaves.presence ~ Area, family = binomial, data = n)

Deviance Residuals: 
Min      1Q  Median      3Q     Max  
-1.213  -1.044  -1.023   1.312   1.344  

Coefficients:
                        Estimate Std. Error z value Pr(>|z|) 
(Intercept)           -0.3877697  0.0282178 -13.742  < 2e-16 ***
leaves.presence        0.0008166  0.0002472   3.303 0.000956 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 
(Dispersion parameter for binomial family taken to be 1)

Null deviance: 16662  on 12237  degrees of freedom
Residual deviance: 16651  on 12236  degrees of freedom
(314 observations deleted due to missingness)
AIC: 16655
Number of Fisher Scoring iterations: 4

There's a number of things I don't get here, what does this really say:

                        Estimate Std. Error z value Pr(>|z|) 
(Intercept)           -0.3877697  0.0282178 -13.742  < 2e-16 ***
leaves.presence        0.0008166  0.0002472   3.303 0.000956 ***

And what does AIC and Number of Fisher Scoring iterations mean?

> fit
Call:  glm(formula = Lövförekomst ~ Areal, family = binomial, data = n)

Coefficients:
(Intercept)        Areal  
-0.3877697    0.0008166  

Degrees of Freedom: 12237 Total (i.e. Null);  12236 Residual
(314 observations deleted due to missingness)
Null Deviance:      16660 
Residual Deviance: 16650        AIC: 16650

And here what does this mean:

Coefficients:
(Intercept)        Areal  
-0.3877697    0.0008166 
$\endgroup$
8
  • 7
    $\begingroup$ Since your question is very broad -- "how does one interpret a binomial regression?" -- I would suggest picking up an introductory text on the topic. Agresti's An Introduction to Categorical Data Analysis is very approachable. $\endgroup$
    – Sycorax
    Feb 12, 2014 at 15:17
  • 2
    $\begingroup$ This may be too broad to answer here; as @user777 said, consulting a good text might be in order. Agresti is good, I agree. Hosmer & Lemeshow is also good. If you want something brief and free (self plug alert) see my introduction to logistic regression but it may be too basic for your needs. $\endgroup$
    – Peter Flom
    Feb 12, 2014 at 15:41
  • $\begingroup$ Ok, thank for you´re quick answers, i will try Agresti and see if it helps :) $\endgroup$
    – user40116
    Feb 12, 2014 at 15:49
  • 6
    $\begingroup$ I don't think this question is too broad to be answerable. It seems to me it is essentially the logistic regression version of interpretation-of-rs-lm-output, which has consistently been considered on-topic. $\endgroup$ Feb 12, 2014 at 15:49
  • 2
    $\begingroup$ I'm with @gung on this one, if the question is about interpreting what R squirted onto the screen. Where there is ambiguity is what is meant by "mean"? If the OP is happy to be told that the coefficients are the estimated values of the model with values on the scale of the log odds, then this Q is OK. If the OP is not satisfied with this and requires an explanation of their meaning in terms of the data, model etc, then that would be too broad a question given that this is but one of several questions asked. $\endgroup$ Feb 12, 2014 at 15:59

2 Answers 2

92
$\begingroup$

What you have done is logistic regression. This can be done in basically any statistical software, and the output will be similar (at least in content, albeit the presentation may differ). There is a guide to logistic regression with R on UCLA's excellent statistics help website. If you are unfamiliar with this, my answer here: difference between logit and probit models, may help you understand what LR is about (although it is written in a different context).

You seem to have two models presented, I will primarily focus on the top one. In addition, there seems to have been an error in copying and pasting the model or output, so I will swap leaves.presence with Area in the output to make it consistent with the model. Here is the model I'm referring to (notice that I added (link="logit"), which is implied by family=binomial; see ?glm and ?family):

glm(formula = leaves.presence ~ Area, family = binomial(link="logit"), data = n)

Let's walk through this output (notice that I changed the name of the variable in the second line under Coefficients):

Deviance Residuals: 
   Min      1Q  Median      3Q     Max  
-1.213  -1.044  -1.023   1.312   1.344  

Coefficients:
                        Estimate Std. Error z value Pr(>|z|) 
(Intercept)           -0.3877697  0.0282178 -13.742  < 2e-16 ***
Area                   0.0008166  0.0002472   3.303 0.000956 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 
(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 16662  on 12237  degrees of freedom
Residual deviance: 16651  on 12236  degrees of freedom
(314 observations deleted due to missingness)
AIC: 16655
Number of Fisher Scoring iterations: 4

Just as there are residuals in linear (OLS) regression, there can be residuals in logistic regression and other generalized linear models. They are more complicated when the response variable is not continuous, however. GLiMs can have five different types of residuals, but what comes listed standard are the deviance residuals. (Deviance and deviance residuals are more advanced, so I'll be brief here; if this discussion is somewhat hard to follow, I wouldn't worry too much, you can skip it):

Deviance Residuals: 
   Min      1Q  Median      3Q     Max  
-1.213  -1.044  -1.023   1.312   1.344  

For every data point used in your model, the deviance associated with that point is calculated. Having done this for each point, you have a set of such residuals, and the above output is simply a non-parametric description of their distribution.


Next we see the information about the covariates, which is what people typically are primarily interested in:

Coefficients:
                        Estimate Std. Error z value Pr(>|z|) 
(Intercept)           -0.3877697  0.0282178 -13.742  < 2e-16 ***
Area                   0.0008166  0.0002472   3.303 0.000956 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 

For a simple logistic regression model like this one, there is only one covariate (Area here) and the intercept (also sometimes called the 'constant'). If you had a multiple logistic regression, there would be additional covariates listed below these, but the interpretation of the output would be the same. Under Estimate in the second row is the coefficient associated with the variable listed to the left. It is the estimated amount by which the log odds of leaves.presence would increase if Area were one unit higher. The log odds of leaves.presence when Area is $0$ is just above in the first row. (If you are not sufficiently familiar with log odds, it may help you to read my answer here: interpretation of simple predictions to odds ratios in logistic regression.) In the next column, we see the standard error associated with these estimates. That is, they are an estimate of how much, on average, these estimates would bounce around if the study were re-run identically, but with new data, over and over. (If you are not very familiar with the idea of a standard error, it may help you to read my answer here: how to interpret coefficient standard errors in linear regression.) If we were to divide the estimate by the standard error, we would get a quotient which is assumed to be normally distributed with large enough samples. This value is listed in under z value. Below Pr(>|z|) are listed the two-tailed p-values that correspond to those z-values in a standard normal distribution. Lastly, there are the traditional significance stars (and note the key below the coefficients table).


The Dispersion line is printed by default with GLiMs, but doesn't add much information here (it is more important with count models, e.g.). We can ignore this.


Lastly, we get information about the model and its goodness of fit:

    Null deviance: 16662  on 12237  degrees of freedom
Residual deviance: 16651  on 12236  degrees of freedom
(314 observations deleted due to missingness)
AIC: 16655
Number of Fisher Scoring iterations: 4

The line about missingness is often, um, missing. It shows up here because you had 314 observations for which either leaves.presence, Area, or both were missing. Those partial observations were not used in fitting the model.

The Residual deviance is a measure of the lack of fit of your model taken as a whole, whereas the Null deviance is such a measure for a reduced model that only includes the intercept. Notice that the degrees of freedom associated with these two differs by only one. Since your model has only one covariate, only one additional parameter has been estimated (the Estimate for Area), and thus only one additional degree of freedom has been consumed. These two values can be used in conducting a test of the model as a whole, which would be analogous to the global $F$-test that comes with a multiple linear regression model. Since you have only one covariate, such a test would be uninteresting in this case.

The AIC is another measure of goodness of fit that takes into account the ability of the model to fit the data. This is very useful when comparing two models where one may fit better but perhaps only by virtue of being more flexible and thus better able to fit any data. Since you have only one model, this is uninformative.

The reference to Fisher scoring iterations has to do with how the model was estimated. A linear model can be fit by solving closed form equations. Unfortunately, that cannot be done with most GLiMs including logistic regression. Instead, an iterative approach (the Newton-Raphson algorithm by default) is used. Loosely, the model is fit based on a guess about what the estimates might be. The algorithm then looks around to see if the fit would be improved by using different estimates instead. If so, it moves in that direction (say, using a higher value for the estimate) and then fits the model again. The algorithm stops when it doesn't perceive that moving again would yield much additional improvement. This line tells you how many iterations there were before the process stopped and output the results.



Regarding the second model and output you list, this is just a different way of displaying results. Specifically, these

Coefficients:
(Intercept)       Areal  
-0.3877697    0.0008166

are the same kind of estimates discussed above (albeit from a different model and presented with less supplementary information).

$\endgroup$
0
2
$\begingroup$

Call: This is just the call that you made to the function. It will be the exact same code you typed into R. This can be helpful for seeing if you made any typos.

(Deviance) Residuals: You can pretty much ignore these for logistic regression. For Poisson or linear regression, you want these to be more-or-less normally distributed (which is the same thing the top two diagnostic plots are checking). You can check this by seeing if the absolute value of 1Q and 3Q are close(ish) to each other, and if the median is close to 0. The mean is not shown because it's always 0. If any of these are super off then you probably have some weird skew in your data. (This will also show up in your diagnostic plots!)

Coefficients: This is the meat of the output.

  • Intercept: For Poisson and linear regression, this is the predicted output when all our inputs are 0. For logistic regression, this value will be further away from 0 the bigger the difference between the number of observation in each class.. The standard error represents how uncertain we are about this (lower is better). In this case, because our intercept is far from 0 and our standard error is much smaller than the intercept, we can be pretty sure that one of our classes (failed or didn't fail) has a lot of more observations in it. (In this case it's "didn't fail", thankfully!)

  • Various inputs (each input will be on a different line): This estimate represents how much we think the output will change each time we increase this input by 1. The bigger the estimate, the bigger the effect of this input variable on the output. The standard error is how certain about it we are. Usually, we can be pretty sure an input is informative is the standard error is 1/10 of the estimate. So in this case we're pretty sure the intercept is important.

  • Signif. Codes: This is a key to the significance of each :input and the intercept. These are only correct if you only ever fit one model to your data. (In other words, they’re great for experimental data if you from the start which variables you’re interested in and not as informative for data analysis or variable selection.)

    Wait, why can't we use statistical significance? You can, I just wouldn't generally recommend it. In data science you'll often be fitting multiple models using the same dataset to try and pick the best model. If you ever run more than one test for statistical significance on the same dataset, you need to adust your p-value to make up for it. You can think about it this way: if you decide that you'll accept results that are below p = 0.05, you're basically saying that you're ok with being wrong one in twenty times. If you then do five tests, however, and for each one there's a 1/20 chance that you'll be wrong, you now have a 1/4 chance of having been wrong on at least one of those tests... but you don't know which one. You can correct for it (by multiplying the p-value you'll accept as significant by the number of tests you'll preform) but in practice I find it's generally easier to avoid using p-values altogether.

(Dispersion parameter for binomial family taken to be 1): You'll only see this for Poisson and binomial (logistic) regression. It's just letting you know that there has been an additional scaling parameter added to help fit the model. You can ignore it.

Null deviance: The null deviance tells us how well we can predict our output only using the intercept. Smaller is better.

Residual deviance: The residual deviance tells us how well we can predict our output using the intercept and our inputs. Smaller is better. The bigger the difference between the null deviance and residual deviance is, the more helpful our input variables were for predicting the output variable.

AIC: The AIC is the "Akaike information criterion" and it's an estimate of how well your model is describing the patterns in your data. It's mainly used for comparing models trained on the same dataset. If you need to pick between models, the model with the lower AIC is doing a better job describing the variance in the data.

Number of Fisher Scoring iterations: This is just a measure of how long it took to fit you model. You can safely ignore it.

I suggest this toturial to learn more. https://www.kaggle.com/rtatman/regression-challenge-day-5

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.