I have a total relationship quality variable constructed as the mean of three other scales. Each of these three (family, friend, and spouse) scales is significantly negatively associated with an outcome in a simple regression; however, when I combine them into the total relationship quality, the total relationship quality is non-significantly positively associated with the outcome in a simple regression. What gives?

              Estimate Std. Error t value Pr(>|t|)    
(Intercept)         17.386      3.638   4.779 3.02e-06 ***
family/friend qual   -2.648      1.251  -2.116   0.0354 *  
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

            Estimate Std. Error t value Pr(>|t|)
(Intercept)        5.599      3.983   1.406    0.161
above plus spouse    1.289      1.383   0.932    0.352
  • 1
    $\begingroup$ Most surprising things can happen when you sum or average raw (not centered) variables in one scale. Did you do centering or not before averaging? $\endgroup$
    – ttnphns
    Dec 6 '17 at 12:48
  • 2
    $\begingroup$ en.wikipedia.org/wiki/Ecological_fallacy $\endgroup$
    – whuber
    Dec 6 '17 at 16:11

Case 1: You looked at 3 separate univariate models for each score, the signs differed from the effect of the average score.

Explanation: This can't happen. The sign of the regression coefficient is the same as the sign of the covariance between the regressor and the outcome. Note that $Cov(X+W, Y) = Cov(X, Y) + Cov(W, Y)$. So the sum of a bunch of positive things is a positive thing.

Case 2: You inspected the coefficient signs in a multivariate model adjusting for all 3 scores.

Explanation: Two or more scores are strongly negatively collinear, one has low variance, the other is strongly positively correlated with the outcome. Call one score X and the other W, so that the data frame [Y,X,W] has covariance matrix $$\Sigma = \left[ \begin{array}{ccc} \sigma_{11} & \sigma_{12} & \sigma_{13} \\ & \sigma_{22} & \sigma_{23} \\ & & \sigma_{33} \end{array} \right]$$ the W adjusted coefficient of X is $$\beta_{X|W} = \dfrac{\sigma_{12}\sigma_{33} -\sigma_{23}\sigma_{13}}{\sigma_{22} \sigma_{33} - \sigma_{23}^2}$$.

Note I'm not using squared terms.

We've established that $\sigma_{12} + \sigma_{13} > 0$ since the sum-score has a positive coefficient. One of those then has to be positive and greater than or equal to the other, let it be $\sigma_{12} \ge |\sigma_{13}|$ WLOG. But the adjusted effects being negative requires $\sigma_{12}\sigma_{33} < \sigma_{23}\sigma_{13}$ and $\sigma_{13}\sigma_{22} < \sigma_{23}\sigma_{12}$. Via neat algebra, the condition is met when the univariate regression of X (the more predictive covariate) on W has a coefficient greater in magnitude than the inverse of the univariate regression of W on X.

EDIT: Why does this matter?

Subscales and total scores have different measurement properties. Fitting many models rarely does much to elucidate these differences. Correctly interpreting adjusted analyses is essential to understanding these differences when doing so is warranted by the present investigation. We have already proven that the scenario in which this occurs is when a single model adjusts simultaneously for each of the subscales, and the model coefficients for the subscales are of opposite sign than is found in the univariate model for the total score. The interpretation of a model coefficient in the multivariate model is an expected difference in the outcome holding each of the other scales fixed. Inconsistencies in findings from these two approaches have been well described in a variety of scenarios: Simpson's paradox is an example, as is the ecological fallacy as noted above. In your case, the other scales are not intrinsic stratifying variables, thus notion of holding another variable fixed is not appropriate to the scientific question. I would disregard the results of the stratified analysis.

  • $\begingroup$ Here's the output of the four separate regressions: spouse -0.1168 0.6279 -0.186 0.853 ( int=9.40) family -1.4774 0.8879 -1.664 0.0974 . (int=13.95) friend -1.5253 0.9661 -1.579 0.115 (int=14.2) total 1.289 1.383 0.932 0.352 (int=5.6) $\endgroup$
    – Benji
    Dec 11 '17 at 0:06
  • $\begingroup$ Does that make sense? I wasn't really able to understand your explanation because I'm not great at stats, but it seems a lot like Case 1 to me. $\endgroup$
    – Benji
    Dec 11 '17 at 0:10
  • $\begingroup$ @BenjiKaveladze to highlight my answer: Case 1 is not possible, Case 2 is when two or more scales are very negatively correlated. Did you inspect this? $\endgroup$
    – AdamO
    Dec 11 '17 at 14:38
  • $\begingroup$ Each scale is correlated around .1-.2 with the other scales, and correlated .5-.75 with the combined (friend+family+spouse) scale. $\endgroup$
    – Benji
    Dec 11 '17 at 20:05
  • $\begingroup$ @BenjiKaveladze in your first comment, presumably a set of model coefficients: have you adjusted for total score and the subscales in the same model? $\endgroup$
    – AdamO
    Dec 11 '17 at 20:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.