3
$\begingroup$

I am a bit puzzled about the behavior of uncorrelated predictors in logistic regression. As in OLS, I thought that if two predictors (rv1 and rv2) are uncorrelated, then the regression weights of rv1 will not change from a regression that only includes rv1 to one that includes rv1 and rv2. However, it seems to be the case that this is not true in logistic regression and coefficients change between the two regression models, even if the predictors are uncorrelated.

I have pasted some R syntax below that demonstrates this behavior.

Why is this the case and how do the regression weights from the two regressions (the one with only rv1 and the other one with rv1 and rv2) relate to each other? Is there a way to know what the regression weight of rv1 will be if one knows the regression weight of rv1 in the regression that includes both predictors?

Thanks! P.S. This post is crossposted at another unrelated stat answer site.

library(MASS)

#generate lots of data (a little bit weird data handling, I know)
n <- 10000
rdta <- as.data.frame(mvrnorm(n=n,c(0,0),matrix(c(1,0,0,1),2,2),empirical=TRUE))
names(rdta) <- c("rv1","rv2")

#confirm that preds are uncorrelated
cov(rdta$rv1,rdta$rv2)

rv1 <- rdta$rv1
    rv2 <- rdta$rv2

rv1ry <- 1
rv2ry <- 1

#generate binary data from known regression coefficients
ylinp <- (1 / (1+exp(-(-1 + rv1*rv1ry + rv2*rv2ry))))
y <- rbinom(n,1,ylinp) 
glm(y~rv1+rv2,family=binomial(link='logit'))
glm(y~rv1,family=binomial(link='logit'))
glm(y~rv2,family=binomial(link='logit'))

#confirm that OLS regression works as expected (regression weights do not change)
rv1y <- .222
rv2y <- .333
y <- rv1y * rv1 + rv2y * rv2 + rnorm(n,0,.5)
lm(y~rv1+rv2)
lm(y~rv1)
lm(y~rv2)   

I am not sure if it is expected to also paste relevant output here, but here goes: OLS results

lm(y~rv1+rv2)

Call:
lm(formula = y ~ rv1 + rv2)

Coefficients:
(Intercept)          rv1          rv2  
 0.001096     0.220051     0.333072  

lm(y~rv1)

Call:
lm(formula = y ~ rv1)

Coefficients:
(Intercept)          rv1  
 0.001096     0.220051  

lm(y~rv2)

Call:
lm(formula = y ~ rv2)

Coefficients:
(Intercept)          rv2  
 0.001096     0.333072  

Logistic regression results

glm(ry~rv1+rv2,family=binomial(link='logit'))

Call:  glm(formula = ry ~ rv1 + rv2, family = binomial(link = "logit"))

Coefficients:
(Intercept)          rv1          rv2  
     -1.001        1.916        2.469  

glm(ry~rv1,family=binomial(link='logit'))

Call:  glm(formula = ry ~ rv1, family = binomial(link = "logit"))

Coefficients:
(Intercept)          rv1  
    -0.5495       1.0535  

glm(ry~rv2,family=binomial(link='logit'))

Call:  glm(formula = ry ~ rv2, family = binomial(link = "logit"))

Coefficients:
(Intercept)          rv2  
-0.6538       1.6140  
$\endgroup$
1
$\begingroup$

I don't have a proof of this, but I think this relates to the link function in the context of a generalized linear model. A model with normal errors and identity link (ie. an OLS regression) will have the property you describe. A model with a non-linear link function (like a logistic regression) will have differences in regression coefficients for rv1 between Y~rv1 + rv2 and Y~rv1 because you are averaging over a non-linear function.

$\endgroup$
0
$\begingroup$

I don't think this is peculiar and it is not restricted to logistic regression. It can happen with linear regression as well. The least squares estimates of the regression coefficients will depend on which variable you include to fit in the model. This will happen even if covariates $X_1$ and $X_2$ are independent or even just uncorrelated.

$\endgroup$
  • 1
    $\begingroup$ Appreciate the answer, but I think if two predictors are uncorrelated, then the regression weight of a predictor in linear OLS regression do not change from a regression that includes only one or both of the uncorrelated predictors. This also follows from the formulas for regression weights in multiple regression which can be expressed as: beta = (r_y1 - r_y2*r_12) / (1-r_12) where r_y1 is the correlation of rv1 with outcome y r_y2 is the correlation of rv2 with outcome y and r_12 is the correlation among the predictors. In the case of uncorrelated predictors, r_12 is zero $\endgroup$ – Felix Thoemmes Jul 3 '12 at 21:03
  • $\begingroup$ So you think that if you have a given set of data with variables X1 and X2 uncorrelated that fitting Y=b1 X1 +b2 X2 will give the same b1 as fitting Y=b1 X1 +0 X2? $\endgroup$ – Michael Chernick Jul 3 '12 at 21:11
  • 1
    $\begingroup$ At least that is what my simple example (see R code in question that I just added) seem to suggest. If I am missing something, feel free to point that out. $\endgroup$ – Felix Thoemmes Jul 3 '12 at 21:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.