8
$\begingroup$

Theoretically, does the intercept term in a logistic regression model capture all unobserved effects?

In other words, in a logistic regression model with a perfect fit (i.e. all relevant variables are included), the intercept term should be zero right?

$\endgroup$
6
$\begingroup$

Theoretically, does the intercept term in a logistic regression model capture all unobserved effects?

This is an interesting question, and I can see how with some simple experiments, one might think that this is the case. In fact, in my first attempt to set this up, I actually created a demonstration would only incorrectly estimate the intercept when I mis-specified the model -- otherwise, all coefficient estimates were fine!

In an OLS regression, the error term is where we would like for all of the effects for which we have not accounted to go... but if there are effects for which we have not accounted (i.e., the model is mis-specified) then they will tend to rear their heads in other features of the model, particularly if there are confounding relationships among variables. This is also true of all other conventional regression methods -- if the model is mis-specified, the coefficient estimates are not trustworthy (but perhaps the predictions will be helpful or the model serves some other useful purpose).

For example, here is a binomial model where there are just two features, and some dependence between them. I've rigged it such that the coefficients should be $\beta_0=10, \beta_1=-5, \beta_2=5.$ But if we omit $x_2$ from the model estimation, all of our coefficients are estimated incorrectly -- and wildly so!

set.seed(13)
N <- 100

inv_logit <- function(x){
    ifelse(x< -20, -20, x)
    out <- 1/(1+exp(-x))
    return(out)
}

x0 <- rep(1, N)
x1 <- rnorm(N)
x2 <- rnorm(N, mean=10+3*x1-0.5*x1^2)
zTransform <- cbind(x0, x1, x2)%*%c(-10,-5,1)
summary(zTransform)

yObs <- rbinom(N, size=1, prob=inv_logit(zTransform))

badModel <- glm(yObs~x1, family=binomial(link="logit"))
summary(badModel)

Coefficients:
            Estimate Std. Error z value Pr(>|z|)    
(Intercept)  -0.1404     0.2327  -0.604    0.546    
x1           -1.3417     0.3041  -4.412 1.02e-05 ***

But if we correctly-specify the model, we get our coefficients back, but with some estimate error.

goodModel <- glm(yObs~x1+x2, family=binomial(link="logit"))
summary(goodModel)

Coefficients:
            Estimate Std. Error z value Pr(>|z|)    
(Intercept)  -9.9512     2.9331  -3.393 0.000692 ***
x1           -4.8657     1.1918  -4.083 4.45e-05 ***
x2            0.9970     0.2948   3.382 0.000720 ***

In other words, in a logistic regression model with a perfect fit (i.e. all relevant variables are included), the intercept term should be zero right?

Why would this be the case? Suppose you're performing a logistic regression and you have no covariates -- for example, your experiment is rolling a die and every 6 is a "success", and every other outcome is a failure (perhaps you're doing quality assurance for a casino). If we assume that the dice are fair, you'd estimate the coefficient at some nonzero value purely because there are more unfavorable outcomes than favorable outcomes in your data.

It's important to understand that you've asked two different questions in your post. The first asks whether the intercept captures un-modeled effects (it doesn't! All coefficient estimates are wrong when the model is mis-specified!) The second question asks whether the intercept should be zero -- and the answer is also no, because the intercept term is fixed by the ratio of "successes" to "failures".

$\endgroup$
  • $\begingroup$ Thank you, your answer actually helped alot! So basically the unobserved effects are only caputed in the difference between the maximum lnLikelihood value (=0) and the lnLikelihood function that takes all independent variables into account, right? $\endgroup$ – student_of_life Oct 29 '15 at 22:18
  • 2
    $\begingroup$ I don't understand your question. The correct model specification includes all relevant features-- which is a bit of a dilemma, because the world is complicated and accounting for all effects is generally impossible. Omitted effects can mean that coefficient estimates are quite incorrect! $\endgroup$ – Sycorax says Reinstate Monica Oct 29 '15 at 23:32
  • $\begingroup$ As you said, since a model can rarely capture all effects, there will be always omitted effects. I was wondering if there can be found 'an indicator' within a standard binary logistic regression model that indicates the size of these omitted effects. $\endgroup$ – student_of_life Oct 30 '15 at 0:27
  • $\begingroup$ Not that I'm aware: you can't fit data you don't have. $\endgroup$ – Sycorax says Reinstate Monica Oct 30 '15 at 0:51
  • 1
    $\begingroup$ @student_of_life: Any metric comparing model fit to a perfect fit - that of a model predicting a success probability of 1 for all "successes" & 0 for all "failures" - might be taken to indicate the size of the omitted effects in a deterministic universe. $\endgroup$ – Scortchi - Reinstate Monica Oct 30 '15 at 10:39
2
$\begingroup$

I am not sure if any model, even a linear one, with a 'perfect' fit implies that the intercept term should be 0. It helps in these cases to think of a simple linear regression. The way I understand the intercept is that it fixes some reasonable value for the y-variable. It just shows what value the y-variable takes even if all the x's are 0. There must be a good reason to think why this should be 0. I dont think it has anything to do with unobservables. In a linear model, it allows a) a better fit and b) ensures that the residuals sum to 1.

$\endgroup$
1
$\begingroup$

in a logistic regression model with a perfect fit (i.e. all relevant variables are included), the intercept term should be zero right?

No. The intercept captures the constant part of the hazard.

$\endgroup$
1
$\begingroup$

The intercept allows the linear hyperplane to move "sideways". For instance, in one dimension it moves the sigmoid left and right, effectively changing the place the regression activates.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.