Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am interested in the combined effects of 12 continuous/interval predictors on a dichotomous/binomial outcome and thus am using logistic regression.

First I ran 12 separate bivariate logistic regressions. All had positive B's. Then when I ran the multiple logistic regression (enter method), some of these predictors have now become negatively associated with the outcome.

I'm not sure what to make of this result. Is this unusual? What does this mean when this happens?

Thanks so much for any guidance! Bonnie

share|improve this question

Welcome to the site.

To keep it simple, let's go to a case with two IVs (the principle is the same). If the $\beta$s are positive in the bivariate regressions it means that as IV 1 increases the odds of "success" (however that is coded) on the DV go up. Similarly for IV 2. If one of the signs (say, that on IV2) switches when including both IVs, it means that the odds of success go down as IV 2 increases, after controlling for IV 1. That is, at constant levels of IV 1, increasing IV 2 lowers the odds of success.

If you tell us what the IVs and the DV are, we will be able to answer more precisely.

share|improve this answer
Thank you Peter Flom. – Bonnie Dec 31 '12 at 19:15

This sort of thing usually happens when you have correlated predictors. Apart from examining the correlation matrix of you predictors, one way to check for high correlations is too look at the Principal Components of your predictors, and the amount of common variation in the first few components.

Another thing to note is that the coefficients can only be interpretted as a rate of change by assuming that all other predictors in the model stay unchanged. However, if all of you variables are positively related to the outcome variable (as shown by the single variable regressions), then it is likely that they are also going to be positively related to each other. So this means that if you increase (decrease) one predictor, the other predictors are likely to also increase (decrease) as well. This wrecks the usual interpretation of the betas, because it makes the values of all the other betas important.

I assume you want to use your univariate regressions to validate or understand your multivariate regression better. One way you can do this is to plot the predicted probabilities against each predictor variable. This should still show a positive relationship with each dependent variable.

From a technical perspective, the multivariate regression satisfies additional constraints compared to the regression with just one variable. Assuming that you're using the logistic link function, these constraints take the form $$\sum_{i=1}^{n}n_i(f_i-p_i)x_{ij}=0\;\;\;\;\; j=0,\dots,q$$ Where $f_i$ is the observed proportion of successes for the ith data point; $n_i$ the number of units the proportion is based on for the ith data point; $x_{ij}$ is the value of the jth predictor for the ith data point; $q$ is the number of predictors; and $p_i$ is the predicted probability for the ith data point, which is a function of the predictors $x_{i0},x_{i1},\dots,x_{iq}$ and their coefficients $\beta_0,\beta_1,\dots,\beta_q$. I have used $x_{i0}$ and $\beta_0$ to denote the intercept term of the model, so $x_{i0}=1$.

These are the set of equations that define the regression coefficients (i.e. betas). There is one equation for each predictor variable. If we define a residual as the difference between the observed and predicted counts $r_i=n_i(f_i-p_i)$, then what the equations are doing is forcing the correlation between the $r_i$ and $x_{ij}$ to be equal to zero.

The regression with only one predictor has two constraint equations: one for the intercept term, and one for the single predictor variable - call it $x_{i1}$. For your case you say all of the betas are positive, which means that the predicted probability must be an increasing function $x_{i1}$. We can use this and the constraint equation to deduce that the observed count $n_{i}f_{i}$ must also tend to increase as $x_{i1}$ increases.

However, note that adding an additional variable to the model, say $x_{i2}$, does not change the constraint which must be satisfyied from using $x_{i1}$. Now we now from the above paragraph that $n_if_i$ tends to increase as $x_{i1}$ increases. This means that the predicted probability $p_i$ will also need to increase as $x_{i1}$ increases as well. The negative value for the coefficient makes it seem like maybe this isn't true. In fact, what the negative coefficient actually means here is that $p_i$ is increasing with $x_{i1}$ too much when only $x_{i2}$ is included in the model.

share|improve this answer
Thank you probabilityislogic. – Bonnie Dec 31 '12 at 19:15

Here is a simple idea to help with understanding (it is linear rather than logistic regression, but the concept is the same).

Imagine a dataset that is based on the coins in the pockets of several people (I will use US coins, but others would work), y is the total value of the coins, x1 is the total number of the coins, and x2 is the number of coins that are not quarters (or whatever the highest denomination common coin is). You can see that using one x at a time increasing x will increase y, but if we look at both x's at the same time then if we hold x1 (total number of coins) constant, but increase x2 (the interpretation of the slope) then y would decrease because that means replacing quarters with smaller valued coins.

The same general concept will hold for other datasets, including yours.

share|improve this answer
Thank you Greg Snow. – Bonnie Dec 31 '12 at 19:16

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.