5
$\begingroup$

I'm about to open the door to a very thorny issue in the social sciences. How does one correctly model and test hypotheses about mediating variables using observational data?

I'm familiar with the Baron-Kenny approach to mediation (see previous answer here), and also with structural equation modeling. However, I've heard both approaches disparaged by more quantitative-minded social scientists than myself -- especially when one is using observational rather than experimental data.

So, let's say that I'm trying to resolve the following:

Y is a behavioral outcome. Both X and Z are observed characteristics of subjects that cannot be manipulated by an experiment. X is an attitude (something that can be changed over the long term) and Z is an unchangeable characteristic such as age, race, etc.

I hypothesize that X is a mediating variable, thus Z affects Y through the pathway of X. While it's reasonable to suggest that Z is in some way correlated with X, my theory argues that it has no impact on Y (other than through **X).

How would one best test these hypotheses using best practices in current research?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ There is a huge literature on this. One good book is MacKinnon $\endgroup$
    – Peter Flom
    May 22, 2013 at 22:59
  • 1
    $\begingroup$ Did you read the paper? $\endgroup$
    – Glen_b
    Aug 20, 2014 at 8:54
  • 1
    $\begingroup$ Please give complete references, as you would be expected to do in a dissertation or paper submitted to a journal. $\endgroup$
    – Nick Cox
    Aug 20, 2014 at 9:26

5 Answers 5

Reset to default
8
$\begingroup$

Baron and Kenny are indeed outdated, though that does not make them wrong in all cases.

The concerns divide into broadly statistical limitations and assumptions which are discussed in the reference your reviewer suggests and in the literature alluded to by @PeterFlom, and broadly non-statistical concerns about the definition and causal identification of mediation effects.

Following this order it might be helpful to start your reading with MacKinnon et al.'s 2007 review, or with the reviewers suggested reading. Then move on to Imai et al. 2010a (or Imai et al. 2010b) These last papers are dense, but repay study. that should bring you more or less up to speed on how mediation analysis is being thought of lately.

Improve this answer
$\endgroup$
5
  • $\begingroup$ I am not good in statistic, and I'm wondering what is the steps now? According to Boron and Kenny using Multiple Regression there are 4 steps. Now I am confuse after reading the Rucker, Preacher, Tormala & Petty (2011) paper??. What is the appropriate steps? This paper stated that partial and fully mediated is meaningless. So the result only state yes-no mediation. $\endgroup$
    – Minn
    Aug 21, 2014 at 1:52
  • $\begingroup$ I think that's not what Rucker et al. are saying. They say "Given their dependence on sample size, the meaningfulness and utility of the ‘full’ and ‘partial’ mediation labels is limited in our view." This is a point about reporting results, not about fitting models. $\endgroup$
    – conjugateprior
    Aug 21, 2014 at 11:37
  • $\begingroup$ I understand you're looking for a list of steps but the reviewer's comment rather forces you to look into your method a bit more deeply than just finding a new list of steps. @PeterFlom's advice about MacKinnon's website is well worth following here. $\endgroup$
    – conjugateprior
    Aug 21, 2014 at 11:44
  • $\begingroup$ Just curious, I still can see some authors still used B&K method in 2014 publication even in A* journal? If it is really outdated why some authors is using it??? $\endgroup$
    – Minn
    Aug 22, 2014 at 3:05
  • $\begingroup$ As I believe William Gibson noted, the future is already here. It's just not very evenly distributed. $\endgroup$
    – conjugateprior
    Aug 22, 2014 at 8:32
7
$\begingroup$

This is more a discussion of concerns I have firstly with the approach of Baron and Kenny (which has some bearing on your question), and with a number of more recent papers (I haven't seen them all, so my comments may not apply to everything). It may also relate to the 2011 paper you mention, which I have only had the chance to skim through just now.

From what I've seen, the idea of measuring/establishing mediation mostly seems to suffer a basic problem* that I haven't seen adequately dealt with. (I've just taken a fairly quick look at the 2011 paper you mention, so maybe I missed something. The example in figure 2 of the paper seems to be related, which is encouraging in the sense that at least some possibilities are being mentioned in some parts of the literature now.)

* The first time I ever heard of mediation and read a copy of Baron & Kenny, I saw this would be an issue. It seemed to be a surprise to most people I mentioned it to.

The problem is this - to establish that $M$ actually mediates $X\to Y$ (at least partly), as below:

$\quad $ enter image description here

(the dashed arrow indicates a reduced level of relationship), it is necessary (for example) to rule out all feasible explanations like these in place of the second diagram:

$\quad $ enter image description here

(The grey variables might be latent, unknown, unaccounted for - or in some other sense 'hidden' from the model, or the researcher, or perhaps even anyone. There may also be some direct relationship between $X$ and $Y$ as well, it makes no real difference to this issue.)

Many papers (at least many of the ones I have seen) which deal with mediation, when they follow the recipe that is supposed to establish whether mediation happens, immediately start saying things along the lines of "$M$ mediates $X\to Y$" and discuss the size of this effect, when unless they have eliminated essentially any possibility of such hidden variables actually driving the relationship in any number of arrangements and variations, they really haven't established such a thing at all, and any measures of the size of the mediation effect rely heavily on those other possibilities not being present, at least not to any substantive degree.

An additional issue is that methods such as regression don't "put heads on arrows". To do so with such methods requires careful experiments, or very careful argument; if both are missing, generally speaking all really we have is correlation, and correlation is not the same thing as causation.

I am hoping one of the very good quality statisticians here will be able to school me on why my concerns are mostly unfounded; such an education would be most welcome.

Improve this answer
$\endgroup$
7
  • $\begingroup$ This seems pretty reasonable to me. In many cases (although not all) $X$ is exogenous due to experimental intervention (e.g. MacKinnon and Dwyer, 1993), but $M$ is not. In social research there is often (theoretically) steps in-between a feasible intervention and the causes of the outcome, or the intervention has several intermediate effects that cause the outcome. Reductionists would say there is pretty much always steps in-between X and Y (or equivalently hidden variables). $\endgroup$
    – Andy W
    Aug 20, 2014 at 13:04
  • $\begingroup$ Clearly nobody but the analyst 'puts heads on arrows'. Certainly there isn't any non-intervention related way to do that generally, or here to establish some mediation relationships here in particular. So reasonable mediation analyses have to start with rather than try to establish the graph structure. They're then just ways to estimate various different effect sizes and directions, under, it turns out from the readings I noted, more or less weird and restrictive assumptions. $\endgroup$
    – conjugateprior
    Aug 20, 2014 at 16:09
  • 1
    $\begingroup$ @PeterFlom In observational studies, the "hidden/missing" variable problem certainly applies to regression more broadly. Indeed I plan to post a question someone asked me that relates to this issue. There are ways to at least partly deal with it, but they're often not done, at least not in a lot of treatments I see (it varies by area, naturally). As far as arrows go, not all regression is regarded as causal, so no, I don't think it's a problem with all regression --- but yes, it's quite a common issue. $\endgroup$
    – Glen_b
    Aug 21, 2014 at 11:04
  • 1
    $\begingroup$ (ctd) ... however, the particular thing with mediation is it is quite explicitly about establishing not only a graphical model (heads on arrows) but the changing magnitudes of effects, and yet the care required to even begin to establish what is claimed generally seems to be absent. At least in regression on observational studies it's not hard to find people making it very clear why it's usually just establishing association and why things like Simpson's paradox must be kept in mind. With mediation studies, there doesn't seem to have been the same density of caveats in what I've seen. $\endgroup$
    – Glen_b
    Aug 21, 2014 at 11:11
  • 1
    $\begingroup$ @Glen_b see those Imai papers' discussion of 'sequential ignorability' for more caveats than you can shake a stick at. It's amazing anyone signs up for that assumption once it's spelled out... $\endgroup$
    – conjugateprior
    Aug 21, 2014 at 11:47
6
$\begingroup$

Here are some places to look. I'd especially recommend the work by Kosuke Imai and colleagues.

Bullock, John G., and Shang E. Ha. 2011. Mediation Analysis is Harder Than it Looks. In Cambridge Handbook of Experimental Political Science, ed. James N. Druckman, Donald P. Green, James H. Kuklinski, and Arthur Lupia. New York: Cambridge University Press.

Bullock, John G., Donald P. Green, and Shang E. Ha. 2010. Yes, But What’s the Mechanism? (Don’t Expect an Easy Answer). Journal of Personality and Social Psychology 98 (April): 550-58.

Fiedler, Klaus, Malte Schott, and Thorsten Meiser. 2011. “What Mediation Analysis Can (Not) Do.” Journal of Experimental Social Psychology. doi:10.1016/j.jesp.2011.05.007. http://dx.doi.org/10.1016/j.jesp.2011.05.007.

Green, Donald P., Shang E. Ha, and John G. Bullock. 2009. “Enough Already about `Black Box’ Experiments: Studying Mediation is More Difficult than Most Scholars Suppose.” The ANNALS of the American Academy of Political and Social Science 628 (March) (January): 200-208.

Imai, Kosuke, Luke Keele, Dustin Tingley, and Teppei Yamamoto. 2011. “Unpacking the Black Box: Learning about Causal Mechanisms from Experimental and Observational Studies.” American Political Science Review 105 (4) (November 10): 765-789. http://imai.princeton.edu/talk/files/ISM10.pdf.

Imai, Kosuke, and Teppei Yamamoto. 2011. "Identification and Sensitivity Analysis for Multiple Causal Mechanisms: Revisiting Evidence from Framing Experiments."

MacKinnon, David P., Amanda J. Fairchild, and Matthew S. Fritz. 2007. “Mediation Analysis.” Annual review of psychology 58 (January): 593-614. doi:10.1146/annurev.psych.58.110405.085542. http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=2819368&tool=pmcentrez&rendertype=abstract.

Spencer, Steven J., Mark P. Zanna, and Geoffrey T Fong. 2005. “Establishing a Causal Chain: Why Experiments Are Often More Effective than Mediational Analyses in Examining Psychological Processes.” Journal of Personality and Social Psychology 89 (6) (December): 845-51. doi:10.1037/0022-3514.89.6.845. http://www.ncbi.nlm.nih.gov/pubmed/16393019.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks Thomas...Bullock et al were exactly the scholars I was thinking of when I asked this question. Somehow I completely forgot about Imai though! $\endgroup$
    – roody
    May 23, 2013 at 12:07
4
$\begingroup$

Baron and Kenny is distinctly old fashioned these days. They see mediation as a "yes-no" "present-absent" quality; more recent approaches (lots of work by MacKinnon and others) treats it as a continuum. This makes more sense to me.

Improve this answer
$\endgroup$
7
  • $\begingroup$ I am not good in statistic, and I'm wondering what is the steps now? According to Boron and Kenny using Multiple Regression: 1. IV -> DV (must be significant) 2. IV -> M (must be significant). 3. M -> DV (must be significant) and 4. IV + M -> DV (must be insignificant) = mediation effect. $\endgroup$
    – Minn
    Aug 21, 2014 at 1:37
  • $\begingroup$ My result shows that my IV -> DV IS SIGNIFICANT but when I include IV + M -> DV it is in significant. However the result shows that IV-> M is significant. How do I interpret this? Pls help. $\endgroup$
    – Minn
    Aug 21, 2014 at 8:52
  • 1
    $\begingroup$ The mediation effect is really the difference between 1 and 4. That is, how much does adding the mediator change the parameter estimate in 1.? Rather than say "it does" or "it does not" the new view is to say "it does it this much". MacKinnon has a website that has a lot of material and some step by step instructions. A lot of this is referred to in other answers you've already gotten $\endgroup$
    – Peter Flom
    Aug 21, 2014 at 10:56
  • $\begingroup$ "Rather than say "it does" or "it does not" the new view is to say "it does it this much"." This is a good point. Where and how I can quote this? $\endgroup$
    – Minn
    Aug 23, 2014 at 0:25
  • $\begingroup$ Feel free to quote me, if you like. As far as I know, the line itself is original to me. Or, if you look through MacKinnon's materials you may find it in his words. $\endgroup$
    – Peter Flom
    Aug 23, 2014 at 10:52
0
$\begingroup$

I agree with the above answer, and I would like to add more information in a form of a succinct summary.

Baron and Kenny's (1986) method of testing mediation has been extensively applied, but there are many papers discussing severe limitations of this approach, which broadly include:

1) Not directly testing the significance of an indirect effect

2) Low statistical power

3) Inability to accommodate models with inconsistent mediation

*Note: see Memon, Cheah, Ramayah, Ting, and Chuah (2018) for an overview.

Considering these limitations, a new typology of mediation was developed by Zhao, Lynch and Chen (2010). As of Oct 2019, it has over 5,000 citations, so it is gaining greater popularity.

As a brief summary, and taking a three-variable causal model as an example, thee types of mediation exist.

Complementary mediation: Mediated effect (a x b) and direct effect (c) both exist and point at the same direction.

Competitive mediation: Mediated effect (a x b) and direct effect (c) both exist and point in opposite directions.

Indirect-only mediation: Mediated effect (a x b) exists, but no direct effect (c).

Further, two non-mediation types were proposed:

Direct-only non-mediation: Direct effect (c) exists, but no indirect effect.

No-effect non-mediation: Nether direct effect (c), nor indirect effect exists.

References

Memon, M. A., Cheah, J., Ramayah, T., Ting, H., & Chuah, F. (2018). Mediation Analysis Issues and Recommendations. Journal of Applied Structural Equation Modeling, 2(1), 1-9.

Zhao, X., Lynch Jr, J. G., & Chen, Q. (2010). Reconsidering Baron and Kenny: Myths and truths about mediation analysis. Journal of Consumer Research, 37(2), 197-206.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.