4
$\begingroup$

I am curious about the consequences of changing the order of the explanatory variables in a binary logistic regression. In a recent series of logistic regressions I ran in SPSS, I found that changing the order of the explanatory variables ($z, y, x$ instead of $x, y, z$) resulted in different coefficient values and significance levels. Investigating further, I got the same results in R using the same orders. Clearly, shifting the predictors around matters in terms of the results—my question is how? (And yes, I checked to make sure I wasn't entering the variables stepwise.)

$\endgroup$
6
$\begingroup$

The order of the explanatory does not matter in logistic regression. The symptoms you describe, i.e. the unstable results, may hint at multicollinearity problems or problems of quasi-complete separation in your data. Do you also have large standard errors? As your independent variables are factors, you can tabulate them one against each other and inspect the relation between them. You should also tabulate the covariate patterns against your dependent variable. You can also start by fitting the model stepwise. Then you will see at which point the problem arises.

$\endgroup$
  • $\begingroup$ Even if there was multicollinearity, the coefficients wouldn't change based on what order you entered them into the formula. $\endgroup$ – Macro Sep 8 '11 at 11:15
  • 2
    $\begingroup$ OK, but in the presence of multicollinearity, the maximisation routine can have problems to converge, or may even fail to converge. These problems can translate into dicely parameter estimates. $\endgroup$ – user5644 Sep 8 '11 at 14:03
4
$\begingroup$

The order in which you enter the variables (into a single model) does not matter. The logistic regression model of a binary response, $Y$, with three predictors $X_{1},X_{2},X_{3}$ -

$$ \log \left( \frac{ P(Y=1) }{ P(Y=0) } \right) = \beta_{0} + \beta_{1} X_{1} + \beta_{2} X_{2} + \beta_{3} X_{3} $$

will give identical results to the "re-arranged" logistic regression model

$$ \log \left( \frac{ P(Y=1) }{ P(Y=0) } \right) = \alpha_{0} + \alpha_{1} X_{2} + \alpha_{2} X_{3} + \alpha_{3} X_{1} $$

with the coefficients re-arranged appropriately. Here some R code that verifies this:

x1=rnorm(100)
x2=rnorm(100)
x3=rnorm(100)
y=-.3 + .5*x1 + 1.2*x2 - .75*x3 + rlogis(100)
y=(y>0)
g1=glm(y~x1+x2+x3,family="binomial")
g2=glm(y~x2+x3+x1,family="binomial")
summary(g1)
summary(g2)

The code generates from the model

$$ \log \left( \frac{ P(Y=1) }{ P(Y=0) } \right) = -.3 + .5 X_{1} + 1.2 X_{2} - .75 X_{3} $$

The estimates for the coefficients of $X_{1}, X_{2}, X_{3}$ (and their significance) is the same regardless of what order you put the variables in. I'm not sure why you'd be finding results that disagree with this. A coding error may be the culprit.

Edit: Based on the comment discussion, it appears that perhaps you are using a categorical predictor and the variables you're including are dummy variables, indicating levels of the predictor. In that case, the order does matter because, in R, the first one entered will be chosen as the "reference" level of the variable and all comparisons are made with respect to it. So, if you change the reference level, you also change the thing the coefficients are being compared to, therefore the estimates and $p$-values can certainly change.

$\endgroup$
  • $\begingroup$ thanks for the response. Can you think of any reason why the following two equations might produce different results? g1 <- glm(y1~ as.factor(x1) + as.factor(x2) + as.factor(x3), family=binomial(link = "logit")) vs. g2 <- glm(y1~ as.factor(x2) + as.factor(x1) + as.factor(x3), family=binomial(link = "logit")) ? Because they did for me. All I did was swap the first two terms. (Sorry for the crappy formatting; can't figure out how to make it look better.) $\endgroup$ – darkfaculties Sep 8 '11 at 5:05
  • $\begingroup$ Another clue—one of the models kicks out one of the categorical variables with the message "Coefficients: (1 not defined because of singularities)," but the other model doesn't do this. I don't understand why the order of the terms would cause R to boot that variable in one model but not the other. $\endgroup$ – darkfaculties Sep 8 '11 at 5:18
  • $\begingroup$ Normally that error message indicates that some parameters are non-identified. For example, if level '1' of x1 always perfectly coincided with level '1' of x2, the two effects can not be separated from each other. It is mysterious why this would depend on the order in which you enter the variables. Are you sure there is no other code entered between the two calls to glm()? $\endgroup$ – Macro Sep 8 '11 at 11:21
  • $\begingroup$ Yeah, the order issue is perplexing. But the same thing happened in both R and SPSS, which makes me think it's more than just a simple bug. Same code both times; I promise. $\endgroup$ – darkfaculties Sep 8 '11 at 12:57
  • 1
    $\begingroup$ Are you sure that they are the same coefficients? When you include factors, one level from each is dropped. Maybe a different level is dropped in each of orderings? $\endgroup$ – Charlie Sep 8 '11 at 14:35
1
$\begingroup$

You could try the following in SPSS and see whether it sheds any light:

graph/scatter (matrix) outcome factor1 factor2 factor3.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.