"Wrong Sign" On Regression Coefficients - Hierarchical Multiple Linear Regression

Question

I am analyzing my data on the relationship between spirituality and negative emotional states (depression, anxiety, and stress) using a hierarchical multiple linear regression. Everything seemed to be "okay" with the results until I checked the signs of the regression coefficients!! The results are showing a positive relationship between depression and a spirituality factor which includes having a reason for living. Virtually all of the research in this area shows that there should be a negative relationship between these two factors, i.e. that higher levels of depression are correlated with lower levels of having a reason for living (without any interaction terms or moderating variables).

Here are the steps that I have completed so far in this three-part data analysis (1st data analysis - Principal components analysis / 2nd Part - EFA (Exploratory Factor Analysis) / 2nd data analysis - hierarchical multiple linear regression):

1. Created spirituality questionnaire and collected data from sample (n = 189).

2. Used a principal components analysis to identify how many factors to extract in EFA.

3. Derived four factors from EFA.

4. Factors included log transformed questions which changed the directionality of the questionnaire (i.e. questions which were "positively" worded became negatively worded).

5. All questions on questionnaire were standardized (transformed to z-scores) to correct for directionality problem.

6. Negative emotional states were examined for normality. Two of the three negative emotional states had to be log transformed.

7. Ran 3 hierarchical multiple linear regressions with DV = Either log depression, log anxiety, or stress and IVs = Age (first "block"), four spirituality factors (second "block") using enter method.

8. Noticed that signs were incorrect, so I compared log transformed versions of DVs to non-transformed versions AND standardized DV (z-score) versions (see this article: The Independent Sign Bias: Gaining Insight from Multiple Linear Regression [PDF] --> pg. 1)

9. IV having to do with meaning in life had correct sign in non-transformed version of depression HOWEVER it looks like there is a collinearity issue between two of the spirituality IVs with the non-transformed version which may affect the signs!!! (Condition index > 15 and Variance proportions: .81 and .40)

10. Looked at bivariate correlations to see if the signs were correct, and oddly enough the signs were accurate (this was only true for the non-transformed versions of the hierarchical multiple linear regressions). None of the correlations were above .6.

11. Removed Age as an IV to see if that made a difference. It did not.

12. Removed spirituality IVs one at a time with stepwise method rather than enter method and this only increased the amount of collinearity between the variables that remained in the final model.

I would greatly appreciate any suggestions on how to correct for the wrong signs and/or if I did anything incorrectly in my three-part data analysis.

Answer 1

There are several things in your description that are a bit confusing, for example you state that taking the log transform reverses the direction of the coding, but the log by itself does not reverse coding.

Your main question seems to be that when you look at individual pairwise correlations the sign of the correlation is as expected, but some of the signs of the slopes in a multiple regression are opposite what you expect. This is not uncommon since the interpretation of slopes is much more complex in multiple regression models.

Consider this example (I read this recently, I don't get the credit for thinking of it): Collect data on the change in various peoples pockets, the variables to collect are the total value of the change (y), the total number of coins (x1), and the total number of coins that are not quarters, or if using non-US coins then number of coins not the highest common coin carried (x2). Generally x1 and x2 will both be positivly correlated with y, but if you do a multiple regression using both x1 and x2 then the slope on x2 will be negative because to increase the number of non-quarters without changing the total number of coins we need to trade quarters for other coins of lesser value which decreases y. You could have something similar happening with your data, does it really make sense to increase the religeous variable without the others changing? What is often more meaningful is to compare predicted outcomes for what would be considered common combinations of your predictor variables.

Answer 2

Factors included log transformed questions which changed the directionality of the questionnaire (i.e. questions which were "positively" worded became negatively worded).

How does taking the log do this? Also, a questionnaire does not have directionality, questions do. For a factor anlaysis, though, it is fine to have some negatively worded and some positively worded questions.

All questions on questionnaire were standardized (transformed to z-scores) to correct for directionality problem.

Standardizing via z-scores does nothing about directionality

Negative emotional states were examined for normality. Two of the three negative emotional states had to be log transformed.

Why? The independent variables do not need to be normally distributed.

IV having to do with meaning in life had correct sign in non-transformed version of depression HOWEVER it looks like there is a collinearity issue between two of the spirituality IVs with the non-transformed version which may affect the signs!!! (Condition index

15 and Variance proportions: .81 and .40)

Collinearity can certainly do this.

Looked at bivariate correlations to see if the signs were correct, and oddly enough the signs were accurate (this was only true for the non-transformed versions of the hierarchical multiple linear regressions). None of the correlations were above .6.

What? The regressions were not transformed, the variables were. It's unclear why you transformed the variables or how this changed the signs. Also, high correlations are not necessary for collinearity.

Removed Age as an IV to see if that made a difference. It did not.

Removed spirituality IVs one at a time with stepwise method rather than enter method and this only increased the amount of collinearity between the variables that remained in the final model.

Don't use stepwise, please. If some variables are collinear there are various things you can do: 1) Remove one or more of them 2) Use a biased but lower variance method of regression, such as ridge regression 3) Combine the variables into a smaller number of variables

Answer 3

If you visualized the data and it confirms what you found in the regression then why do you think the signs are wrong? Your study may merely have found the opposite result. Consider your likelihood of being published in your particular field and then ponder whether you now believe that the effect in the literature you read is unbiased.

Alternatively, I'm not sure how you're measuring spirituality and whether prior research directly measured having a reason for living but it seems to me that if a component of spirituality is believing things might be far more grand in some sort of an after life then you might have a poor proxy for a reason for living.