Evaluating effect sizes of interactions in multiple regression

Question

I have been running 3-level multilevel models with HLM, and my main interest is in some cross-level interaction effects that I am finding. My concern is that the effect sizes of these interactions appear to be small – I am wondering whether they are really meaningful.

I am turning to you to ask whether anyone could advice me on how to evaluate the size of this kind of effect. Are there any benchmarks for interaction effects in regression analyses? Would these be appropriate for multilevel models, too? I believe that it is not uncommon that the explained variance does not increase much when one adds in interaction terms in multiple regression, and that this would be even more the case for multilevel models when the variance that one is trying to explain is the level 1 variance. Is that right?

Answer 1

The effect sizes of interactions in a multivariate regression can be assessed in same way as the effect sizes of any other predictor. The common thing is to look at the incremental contribution to R^2 (semi-partial R^2), but there are other possibilities, including Cohen's f^2 for nested models (this is a likelihood ratio test). Chapter 9 of Cohen's Statistical Power Analysis for the Behavioral Sciences has a very good discussion. It is true, too, that the effect sizes of product-term interactions tend to be "small" in terms of incremental addition to R^2. But the practical effect of such an interaction can be very large. This point is especially important to bear in mind when the interaction involves some sort of treatment or intervention -- e.g., in a drug trial, the practical effect of an interaction between the treatment & some individual characteristic of a patient might contribute only a small amount to mode R^2 but have a very appreciable effect on the clinical outcomes. See Rosenthal, R. & Rubin, D.B. A Note on Percent Variance Explained as A Measure of the Importance of Effects. J Appl Soc Psychol 9, 395-396 (1979); Abelson, R.P. A Variance Explanation Paradox: When a Little is a Lot. Psychological Bulletin 97, 129-133 (1985). Both this possibility & the challenge of trying to interpret (or explicate) the simultaneous importance of the coefficients for the predictor, moderator & product-interaction in a regression output, tend to make reporting of the interaction's "effect size" uninformative; better, I'd say, is to illustrate (graphically) the effect size of the interaction in practical terms -- that is, by showing how changes in meaningful levels of the predictor ("high exposure vs. low exposure") and moderator ("being a man vs. being a woman") affect the outcome variable expressed in units that make sense given the context ("additional yrs of life"). I don't have as much experience w/ multilvel modeling, but I do know that the strategy I'm describing is the basic philosophy of Gelman, A. & Hill, J. Data Analysis Using Regression and Multilevel/Hierarchical Models. (Cambridge University Press, Cambridge ; New York; 2007)-- the greatest work on regression, in my opinion, after Cohen & Cohen!

Answer 2

Pursuant to my discussion on the conceptual overlap between effects size and likelihood ratios here, I wonder if the likelihood ratio for each effect against its respective null might serve as a useful metric to achieve the aims sought by those who conventionally employ effect size measures.