3
$\begingroup$

I have a multiple regression model where I have nearly 20 independent variables. These variables are modestly correlated with each other (e.g., the maximum VIF is around 4 with most of them in the 2s).

One of the coefficients is statistically significant and is negative when I expected that it would be positive. I know that 'wrong signs' can be because of several reasons such as multi-collinearity, missing data, omitted variables etc but I am wondering if there is a simpler explanation for the 'wrong sign'.

The usual interpretation of the coefficients is that it represents the impact on the dependent variable when we change the independent variable by 1 unit holding everything else constant.

However, the above interpretation is accurate only if the independent variables are completely uncorrelated with one another. In the presence of modest correlations among the independent variables, when we increase one of them by 1 unit the others are also bound to go up/down by a modest amount (depending on the sign of the correlation) and hence the only way to predict the impact of a unit change of an independent variable is to evaluate its impact on the other independent variables and then assess the overall impact on the dependent variable. When we do such an analysis we may well discover that the 'wrong sign' is a non-issue as increasing that variable by 1 unit may result in an increase in the dependent variable via the changes in the other independent variables in the model.

Does the above explanation make sense or am I missing something?

$\endgroup$
  • 2
    $\begingroup$ Not your question, but your regression model is multiple (not multivariate) if you have just one response (outcome, regressand, dependent) variable. $\endgroup$ – Nick Cox Oct 23 '13 at 23:44
3
$\begingroup$

This is not an answer, but it is too long for a comment.

I would say the interpretation is accurate even with multicollinearity, but the ceteris paribus coefficient is not the quantity you care about. If you believe that the multicollinearity arises from an approximate linear relationship among some of the regressors, that relationship could be formalized either through some constraint on the parameters (such as dropping a variable or something more) or with a simultaneous equation approach. Without more details about the nature of your problem, it's hard to be more specific. There are some examples (28, 29 and 5) in Peter Kennedy's paper Oh No! I Got the Wrong Sign! What Should I Do?.

$\endgroup$
2
$\begingroup$

If you hold everything else constant, you assume that it is constant. So it does not matter that the independent variables are correlated and they might change when you change your variable of interest. The assumption was that everything else is constant. It is perfectly ok to question this assumption, but the interpretation holds nevertheless. Whether you should care about it is another matter.

$\endgroup$
1
$\begingroup$

Super late reply but I'll weigh in anyway.

You aren't quite right about this:

the above interpretation is accurate only if the independent variables are completely uncorrelated with one another. In the presence of modest correlations among the independent variables, when we increase one of them by 1 unit the others are also bound to go up/down

Whether or not changes in one predictor must cause another to go up or down depends on whether there is some kind of causal relationship between them. For instance, consider a strange, contrived model in which ice cream sales and number of drownings per week are both predictors of some response. They will be collinear (both will be high in hot periods and low in cold periods) but there isn't a causal relationship between them. In this case the ice cream coefficient will accurately measure the independent effect of ice cream sales on your response after controlling for any confounding by drowning, because changing ice cream sales doesn't causally affect drownings.

All of this stuff is the realm of path analysis, which deals with what the causal relationships are between a group of variables, and whether negative 'direct effects' (which is what multiple regression coefficients represent) are mitigated by positive indirect effects (your scenario where an increase in one predictor causes a decrease in a second predictor which causes a positive effect on the response).

$\endgroup$
0
$\begingroup$

I think you're right. You're basically interpreting the statistical term "interaction" when you say that the change in sign might be a non-issue because the interrelated variables may be impacting one another. However, when interpreting the sign change, you should not assume that it's some naturally-explicible process rather than statistical multicollinearity that's causing the sign change. You cannot reason your way out of a multicollinearity problem without running some statistics because either a natural process or multicollinearity can cause an interaction like you've described. You can check if you still have a sign change on the main effect in the model when you add interactions between the variables to your model (y = a + b + a:b rather than just y = a + b). It also helps to standardize all of your explanatory variables and interactions when you run such an interaction model in the presence of multicollinearity (a becomes a_std = (a-mean(a))/std(a), and a:b becomes a_std:b_std). This process will often, but not always, change the sign back to that of the original simple regression.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.