3
$\begingroup$

I have a study where I pair two people up and have them play a behavioral game together. I measure some change score before and after the game. I create a four-level factor variable called treatment that has the following values: MM, MF, FM, FF which describe your biological sex, and the biological sex of your partner.

Let's say I also have some index moderator_idx where I believe, for people in the MF and FM category, high levels of moderator_idx will be associated with high levels of the outcome. So I model the following:

> m1 <- lm_robust(outcome ~ treatment / moderator_idx -1,
              cluster = team_id,
              se = "stata",
              data = data_full)



                             Estimate  Std. Error   t value    Pr(>|t|)      CI Lower     CI Upper  DF
treatmentFF                 9.6698108  4.31703853  2.239918 0.025776158   1.176747161 18.162874409 323
treatmentFM                -2.4975363  6.76639155 -0.369109 0.712288286 -15.809299377 10.814226686 323
treatmentMF                -6.5241575  5.42255272 -1.203152 0.229798459 -17.192138536  4.143823553 323
treatmentMM               -20.0332461 13.66149199 -1.466403 0.143511539 -46.909985777  6.843493496 323
treatmentFF:moderator_idx  -0.1041088  0.05135792 -2.027123 0.043470833  -0.205147088 -0.003070541 323
treatmentFM:moderator_idx   0.1580438  0.07965394  1.984131 0.048087366   0.001337787  0.314749868 323
treatmentMF:moderator_idx   0.1845383  0.06667904  2.767562 0.005973407   0.053358306  0.315718388 323
treatmentMM:moderator_idx   0.2405057  0.15315809  1.570310 0.117322055  -0.060807677  0.541819060 323

Examining only the interaction effects (e.g. treatmentFM:moderator_idx), I get a result that I suspected: for people in different-sex conditions (FM, MF), the effect of an increase in the moderator is associated with significant increases in the outcome of interest.

However, I can re-cast this analysis as a mediation analysis like so (where different_sex is a dummy variable set to 1 if you're paired with someone in the opposite sex, and sex is your own biological sex). Note that this is equivalent to a four-level treatment factor above...

med.fit <- lm(moderator_idx ~ different_sex * sex, data = data_full)
out.fit <- lm(outgroup_feelings_diff ~ sex * different_sex * moderator_idx, data = data_full)
med.out <- mediation::mediate(med.fit, out.fit, treat = "different_sex", mediator = "moderator_idx", robustSE = TRUE, sims = 1000)
summary(med.out)

Quasi-Bayesian Confidence Intervals

                         Estimate 95% CI Lower 95% CI Upper p-value    
ACME (control)            -0.0484      -0.5168         0.37    0.84    
ACME (treated)            -0.6666      -1.7546         0.20    0.14    
ADE (control)             11.2765       7.7503        14.58  <2e-16 ***
ADE (treated)             10.6583       7.2103        13.88  <2e-16 ***
Total Effect              10.6099       7.2444        13.87  <2e-16 ***
Prop. Mediated (control)  -0.0032      -0.0525         0.04    0.84    
Prop. Mediated (treated)  -0.0620      -0.1759         0.02    0.14    
ACME (average)            -0.3575      -0.9851         0.13    0.15    
ADE (average)             10.9674       7.5714        14.19  <2e-16 ***
Prop. Mediated (average)  -0.0326      -0.1000         0.01    0.15    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

And I get that ACME of the treated is non-significant. My question is: How can I get in the first analysis large and significant effects of moderation, but non-significant effects of mediation. What are the substantive differences between the two results, and which should I trust?

To be clear: I understand that mediation and moderation analyses are fundamentally different. I'm wondering the circumstances under which moderation and mediation analysis would produce effects in different directions (the effect of the moderator is positive for the FM and MF conditions in the moderation analysis, but it's negative in the mediation analysis).

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

The results are different because moderation and mediation are two fundamentally different phenomena. (more details here: http://davidakenny.net/cm/mediate.htm)

In your mediation code, you're stating that moderator_idx is a function of the interaction between different_sex and sex (this makes no sense to me...) and that outgroup_feelings_diff is a function of the three way interaction between sex, diff_sex, and moderator_idx. Altogether, these are very very different regression from the given moderation example.

Ultimately, the model you should run depends on your research question. Are you trying to test some intervening mechanism? If so, then figure out how to code your mediation model properly. If this is not what you were aiming to test, then just stick to the moderation part.

Improve this answer
$\endgroup$
2
  • $\begingroup$ interacting sex * different_sex will get you the effects of FM and MF (male paired with female partner and female paired with male partner). It's formally equivalent to having the four-level factor treatment variable in the moderator analysis. In the moderator analysis: treatmentMF*moderator_idx IS equivalent to a three-way interaction. (sex * other_sex * moderator_idx) My main question was about the direction of results. How can the moderation analysis show a positive effect of moderator_idx within treatmentMF and treatmentFM, but mediation analysis shows the opposite? $\endgroup$
    – Parseltongue
    Sep 21, 2020 at 13:24
  • 1
    $\begingroup$ I think part of the issue is that you're not testing a pure mediation model here. The mediation step where you add a new predictor is actually adding other new predictors: moderator_idx, moderator_idxsex, moderator_idxdifferent_sex, and moderator_idxsexdifferent_sex. You're then telling the mediation function that only moderator_idx's unique effect is a mediator while there are new moderators. Additionally, the sex variable is new to the mediation model as it's not in the moderation model. As @Rak observed, these are two different models being tests, so different results are not surprising $\endgroup$
    – Billy
    Sep 25, 2020 at 15:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.