12
$\begingroup$

Here is a list of logistic regression coefficients (first one is an intercept)

-1059.61966694592
-1.23890500515482
-8.57185269220438
-7.50413155570413
 0
 1.03152408392552
 1.19874787949191
-4.88083274930613
-5.77172565873336
-1.00610998453393

I find it weird how the intercept is so low and I have a coefficient that is actually equal to 0. I am not fully sure how I would interpret this. Does the 0 indicate that the specific variable has no affect at all on the model? But the intercept that is made by inputting a column of one's is suddenly really important? Or is my data just crap and the model is unable to properly fit to it.

$\endgroup$
  • 2
    $\begingroup$ What is the range or standard deviation of your other variables? Is there a large difference between the standard deviation of the variable with zero estimate compared to the others? You may expect a coefficient of zero if the standard deviation is small compared to the others (numerical precision). Also intercept basically means you have variables which have large averages (away from zero). Centering your variables would give a more interpretable intercept, and won't change the betas for the other variables (iterative algorithm error aside). $\endgroup$ – probabilityislogic Mar 9 '12 at 20:10
  • 1
    $\begingroup$ If you were to subtract 1027 from all the values of the sixth variable, your intercept would be quite close to 0. Would that make you feel better? :-) $\endgroup$ – whuber Mar 9 '12 at 21:01
  • 3
    $\begingroup$ Showing a list of coefficients like this, with no context at all, is likely saying "Joe has 31, isn't that a lot?" without saying 31 what. 31 cars? A lot. 31 kids? One heck of a lot! 31 dollars? Not much. $\endgroup$ – Peter Flom - Reinstate Monica Mar 9 '12 at 23:38
  • 1
    $\begingroup$ Regarding the coefficient of zero: I could see this happening as an artifact of putting all your coefficients into XL before pasting them here--something that seems consistent with the high number of decimal places we're generally seeing. Maybe one of those XL cells was set to round to integers, yielding the zero. I've had things like this happen. $\endgroup$ – rolando2 Mar 10 '12 at 1:26
  • $\begingroup$ Thanks all of you for your input! I really appreciate each and every one of you! A lot of my questions were answered $\endgroup$ – shiu6rewgu Mar 14 '12 at 18:12
16
$\begingroup$

You are getting some very good information in the comments, in my opinion. I wonder if some basic facts about logistic regression would help make these things more comprehensible, so with that in mind, let me state a couple of things. In logistic regression, coefficients are on the logistic scale (hence the name...). If you were to plug in your covariate values for an observation, multiply them by the coefficients, and sum them, you would get a logit.
$$ \text{logit}=\beta_0+\beta_1x_1+\beta_2x_2+...+\beta_kx_k $$ A logit is a number that makes no intuitive sense to anyone, so it is very difficult to know what to do with a number looks funny (e.g., very high or very low). The best way to understand these things is to convert them from their original scale (logits) to one that you can understand, specifically probabilities. To do that, you take your logit and exponentiate it. That means you take the number e ($e\approx 2.718281828$) and raise it to the power of the logit. Imagine your logit were 2:
$$ e^2=7.389056 $$ This will give you the odds. You can convert the odds to a probability by dividing the odds by one plus the odds: $$ \frac{7.389056}{1+7.389056}=0.880797 $$ People typically find the probability much easier to deal with.

For your model, imagine you had an observation in which the value of all of your variables is exactly 0, then all of your coefficients would drop out and you would be left with only your intercept value. If we exponentiate your value, we get 0 as the odds (if it were -700, the odds would be $9.8\times 10^{-305}$, but I can't get my computer to give me a value for -1060, it is too small given the numerical limits of my software). Converting those odds to a probability, ( $0/(1+0)$ ), gives us 0 again. Thus, what your output is telling you is that your event (whatever it is) simply does not occur when all of your variables are equal to 0. Of course, it depends on what we are talking about, but I find nothing too remarkable about this. A standard logistic regression equation (say, without a squared term, for instance) necessarily assumes that the relationship between a covariate and the probability of success is either monotonically increasing or monotonically decreasing. That means that it always gets larger and larger (or smaller and smaller), and so, if you go far enough in one direction, you will get to numbers so small my computer can't tell them apart from 0. That is just the nature of the beast. As it happens, for your model, going really far is going to where your covariate vales equal 0.

As for the coefficient of 0, it does mean that that variable has no effect, as you suggest. Now, it is quite reasonable that a variable will not have an effect, nonetheless, you will basically never get a coefficient of exactly 0. I don't know why it occurred in this case; the comments offer some possible suggestions. I can offer another, which is that there may be no variation in that variable. For example, if you had a variable that coded for sex, but only women in your sample. I don't know if that's the real answer (R, for example, returns NA in that case, but software differ)--it's just another suggestion.

$\endgroup$
  • 2
    $\begingroup$ Note you can get the odds to base ten by multiply the natural log odds by $\frac{306}{700}\approx\frac{3}{7}$. The answer you get is roughly $-460$. and for small odds, the probability is equal to the odds. So we have roughly a probability of $10^{-460}$ when all covariates are zero. $\endgroup$ – probabilityislogic Mar 10 '12 at 9:46
10
$\begingroup$

Interpreting the intercept

You can think of logistic regression as giving you a posterior probability of being a '1'. The intercept represents a prior on categories derived from the dataset: specifically, it is the empirical estimate of log(p(Y=1)/p(Y=0), by itself when the model has only an intercept, for the cases in the 'reference' classes when there are categorical covariates, and for cases when the covariates are at 0 more generally (but less interpretably). So your strongly negative number is probably telling you that '1's are rare among the cases in your sample characterised by having all covariates at 0. Again, there may be no observations there, so it's not worth worrying about the intercept value. This discussion is fairly clear.

Because of this handy separation of concerns among the parameters, you can correct for category imbalance by training on a better balanced sample and only adjusting the intercept. See King and Zeng for a thorough discussion.

$\endgroup$
  • $\begingroup$ The link to "this discussion" seems to have died. Any chance to recover this link? $\endgroup$ – Alexey Grigorev Aug 22 '17 at 14:31
  • 1
    $\begingroup$ @alexey-grigorev I updated the UCLA link $\endgroup$ – conjugateprior Sep 28 '17 at 19:36
  • $\begingroup$ and garnered a downvote. Very odd. $\endgroup$ – conjugateprior Dec 21 '17 at 4:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.