33
$\begingroup$

I have a question about omitted variable bias in logistic and linear regression.

Say I omit some variables from a linear regression model. Pretend that those omitted variables are uncorrelated with the variables I included in my model. Those omitted variables do not bias the coefficients in my model.

But in logistic regression, I just learned that this isn't true. Omitted variables will bias the coefficients on included variables even if the omitted variables are uncorrelated with the included variables. I found a paper on this topic, but I can't make heads or tails of it.

Here's the paper and some powerpoint slides.

The bias, apparently, is always towards zero. Can anyone explain how this works?

$\endgroup$
6
  • $\begingroup$ Are you familiar with how the logistic regression model emerges from an underlying "latent-variable" linear regression model? $\endgroup$ Commented Aug 30, 2014 at 4:57
  • $\begingroup$ @AlecosPapadopoulos I for one am not. What's the dish? $\endgroup$
    – Alexis
    Commented Aug 30, 2014 at 5:36
  • $\begingroup$ There are other articles that discuss this, but the one you linked to is the easiest I know. So I don't think I can improve on it. $\endgroup$ Commented Aug 30, 2014 at 11:39
  • $\begingroup$ Dear Mr. Papadopoulos: I've read up on the latent-variable idea. Why do you ask? $\endgroup$ Commented Aug 30, 2014 at 14:42
  • $\begingroup$ @ Alexis See e.g. this post, stats.stackexchange.com/questions/80611/…, and the wikipedia article, en.wikipedia.org/wiki/…. This approach clarifies also that it is the assumption we make on the error term of the underlying model that determines what model we will obtain at the Probabilities level. For another example, if we assume that the underlying error follows a uniform, we obtain the Linear Probability Model, see, stats.stackexchange.com/questions/81789 $\endgroup$ Commented Aug 30, 2014 at 14:53

1 Answer 1

36
$\begingroup$

The case of "attenuation bias" can be more clearly presented if we examine the "probit" model -but the result carry over to the logistic regression also.

Underneath the Conditional Probability Models (Logistic (logit), "probit", and "Linear Probability" models) we can postulate a latent (unobservable) linear regression model:

$$y^* = X\beta + u$$

where $y^*$ is a continuous unobservable variable (and $X$ is the regressor matrix). The error term is assumed to be independent from the regressors, and to follow a distribution that has a density symmetric around zero, and in our case, the standard normal distribution $F_U(u)= \Phi(u)$.

We assume that what we observe, i.e. the binary variable $y$, is an Indicator function of the unobservable $y^*$:

$$ y = 1 \;\;\text{if} \;\;y^*>0,\qquad y = 0 \;\;\text{if}\;\; y^*\le 0$$

Then we ask "what is the probability that $y$ will take the value $1$ given the regressors?" (i.e. we are looking at a conditional probability). This is

$$P(y =1\mid X ) = P(y^*>0\mid X) = P(X\beta + u>0\mid X) = P(u> - X\beta\mid X) \\= 1- \Phi (-Χ\beta) = \Phi (X\beta) $$

the last equality due to the "reflective" property of the standard cumulative distribution function, which comes from the symmetry of the density function around zero. Note that although we have assumed that $u$ is independent of $X$, conditioning on $X$ is needed in order to treat the quantity $X\beta$ as non-random.

If we assume that $X\beta = b_0+b_1X_1 + b_2X_2$, then we obtain the theoretical model

$$P(y =1\mid X ) = \Phi (b_0+b_1X_1 + b_2X_2) \tag{1}$$

Let now $X_2$ be independent of $X_1$ and erroneously excluded from the specification of the underlying regression. So we specify

$$y^* = b_0+b_1X_1 + \epsilon$$ Assume further that $X_2$ is also a normal random variable $X_2 \sim N(\mu_2,\sigma_2^2)$. But this means that

$$\epsilon = u + b_2X_2 \sim N(b_2\mu_2, 1+b_2^2\sigma_2^2)$$

due to the closure-under-addition of the normal distribution (and the independence assumption). Applying the same logic as before, here we have

$$P(y =1\mid X_1 ) = P(y^*>0\mid X_1) = P(b_0+b_1X_1 + \epsilon>0\mid X_1) = P(\epsilon> - b_0-b_1X_1\mid X_1) $$

Standardizing the $\epsilon$ variable we have

$$P(y =1\mid X_1 )= 1- P\left(\frac{\epsilon-b_2\mu_2}{\sqrt {1+b_2^2\sigma_2^2}}\leq - \frac {(b_0 + b_2\mu_2)}{\sqrt {1+b_2^2\sigma_2^2}}- \frac {b_1}{\sqrt {1+b_2^2\sigma_2^2}}X_1\mid X_1\right)$$

$$\Rightarrow P(y =1\mid X_1) = \Phi\left(\frac {(b_0 + b_2\mu_2)}{\sqrt {1+b_2^2\sigma_2^2}}+ \frac {b_1}{\sqrt {1+b_2^2\sigma_2^2}}X_1\right) \tag{2}$$

and one can compare models $(1)$ and $(2)$.

The above theoretical expression, tells us where our maximum likelihood estimator of $b_1$ is going to converge, since it remains a consistent estimator, in the sense that it will converge to the theoretical quantity that really exists in the model (and of course, not in the sense that it will find the "truth" in any case):

$$\hat b_1 \xrightarrow{p} \frac {b_1}{\sqrt {1+b_2^2\sigma_2^2}} \implies |\hat b_1|< |b_1|$$

which is the "bias towards zero" result.

We used the probit model, and not the logit (logistic regression), because only under normality can we derive the distribution of $\epsilon$. The logistic distribution is not closed under addition. This means that if we omit a relevant variable in logistic regression, we also create distributional misspecification, because the error term (that now includes the omitted variable) no longer follows a logistic distribution. But this does not change the bias result (see footnote 6 in the paper linked to by the OP).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.