I fit a binary logistic regression model with a single categorical variable, for which I received a coefficient. When I added further categorical predictor variables, the coefficient of the original categorical variable changed. What is the reason for this? That is, why is the coefficient of a predictor variable sensitive to the presence of other predictor variables in the model?

  • $\begingroup$ Why was this question voted down? $\endgroup$ – Namenlos Jul 8 '18 at 3:56
  • 3
    $\begingroup$ Incidentally, I did not downvote, but I imagine it was because "this question does not show any research effort", to quote the mouseover text. $\endgroup$ – Stephan Kolassa Jul 8 '18 at 4:58

Actually, this question deserves special consideration. This is because even if your predictors are orthogonal, addition of more predictors to a model that are predictive of the outcome will increase the logistic regression coefficients of variables that are already present in the model. By increase, I mean move them further away from zero.

The reason for this is simple. In logistic regression, the error is fixed and assumed to have a variance of $\pi^2/3$. If you create a better model for the outcome, the better model doesn't reduce the variance of the error, it increases the variance of the linear predictor on the logit scale. This causes an inflation in your coefficients.

Mathematically, in logistic regression, you're estimating:

\begin{equation} y^*/\sigma = X\frac{\beta}{\sigma} + e \end{equation}

$y^*$ is the continuous latent variable underlying the observed binary 0,1 and $e \sim \mathrm{Logistic}(0, 1)$. If all the determinants of $y^*$ are in the model, $\sigma$, the scaling factor of the logistic distribution is 1. If any of the predictors of $y^*$ are missing, $\sigma$ is less than one such that as you include more determinants of the underlying latent variable, your coefficients will be guaranteed to increase if they are orthogonal and non-zero. If they are not orthogonal, how they will change also depends on the traditional mechanics of omitted variable bias.

A simple simulation of the problem:

coefs <- t(replicate(1500, {
  # generate two independent random variables
  xb <- rbinom(200, 1, .5)
  xc <- rnorm(200)
  # generate outcome from both predictors
  # we care about xb, so give xc a large coefficient
  # to simulate the effect of many missing predictors
  y <- rbinom(200, 1, 1 / (1 + exp(-(.25 * xb + xc))))
  # retrieve binary coefficient in model with:
    coef(glm(y ~ xb, binomial))["xb"], # just binary predictor
    coef(glm(y ~ xb + xc, binomial))["xb"] # both predictors

# get average coefficient from each set of models

       xb        xb
0.2083103 0.2562395 

As one can see, the coefficient from the model without the continuous predictor is deflated while the one in the model with both predictors is close to the .25 value we assigned it.

This problem is described in the econometrics/sociology literature as unobserved heterogeneity. It plagues logistic and probit regression models. With these models, the extent to which the change in coefficients is caused by confounding is not clear.

Based on a quick glance, traditional omitted variable bias is explained simply on Wikipedia. See this article for a simple to read explanation of the unobserved heterogeneity process in logistic regression.

  • 1
    $\begingroup$ +1, see also this blog post I wrote not too long ago, the second half of which is all about this phenomenon (also called non-collapsibility): jakewestfall.org/blog/index.php/2018/03/12/… $\endgroup$ – Jake Westfall Jul 9 '18 at 23:32
  • $\begingroup$ @JakeWestfall I favor linear probability models or even Poisson which is a recommendation in the epidemiology literature because you get collapsible risk ratios. Personally, it's interesting to identify the conditions under which we may trust OLS on binary data and how to handle situations where OLS is clearly problematic. $\endgroup$ – Heteroskedastic Jim Jul 9 '18 at 23:47
  • $\begingroup$ @JakeWestfall also, I read your post some months back when I first came across the concept :). It led me to the WB page which led me to the very helpful Horace and Oaxaca paper. $\endgroup$ – Heteroskedastic Jim Jul 9 '18 at 23:52

Unless your predictors are orthogonal, adding a new predictor to a model will always change the coefficient estimates on existing predictors. This holds for any model.

  • $\begingroup$ Why is this? Or can you recommend literature that will lay out why this is the case? $\endgroup$ – Namenlos Jul 8 '18 at 4:31
  • 2
    $\begingroup$ Fitting a model means estimating parameter coefficients that give the best fit to your response. If you add predictors, then these optimizing parameter estimates will almost always change on predictors in the smaller model. Put another way: it would be surprising if this didn't happen. You may want to revisit elementary textbooks on modeling. $\endgroup$ – Stephan Kolassa Jul 8 '18 at 4:34
  • $\begingroup$ Namenlos, you might want to look up confounding variables to get intuitive understanding.eg predict mortality by gender, then add smoker non smoker. The gender 'effect' will change because smoking is correlated with gende. $\endgroup$ – seanv507 Jul 8 '18 at 15:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.