My question is simple:
How do you determine the overall significance of an interaction (i.e. the marginal effect of $X$ on $Y$ for different values of $Z$)?
But the background is a bit long-winded, so please bear with me. Consider this example (data available here along with some details, taken from Berry, Golder and Milton 2012):
library("foreign")
library("lmtest")
dta <- read.dta("alexseev.dta")
slvote.mod <- lm(xenovote ~ slavicshare * changenonslav +
inc9903 + eduhi02 + unemp02 + apt9200 + vsall03 + brdcont, data=dta)
coeftest(slvote.mod)
Which will output:
> coeftest(slvote.mod) ##same as in Berry et al. 2012, but without clustered SEs
t test of coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 8.94287844 2.71084994 3.2989 0.0016110 **
slavicshare 0.03148617 0.02089537 1.5068 0.1369279
changenonslav -0.85110769 0.35210573 -2.4172 0.0185955 *
[..]
slavicshare:changenonslav 0.00822591 0.00527380 1.5598 0.1239045
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
However Brambor, Clark and Golder (2006) (which comes with an internet appendix) caution that:
Although we still want to know about the marginal effect of some independent variable X on Y in an interaction model ($\beta_1 + \beta_3Z$), typical results tables will report only the marginal effect of X when the conditioning variable is zero, i.e., $\beta_1$. Similarly, these tables report only the standard error for this particular effect. As a result, the only inference that can be drawn in this situation is whether $X$ has a significant effect on $Y$ for the unique case in which $Z = 0$. [..] There is simply no way of knowing from the typical results table if $X$ has a significant effect on $Y$ when the conditioning variable is not zero.
The analyst cannot even infer whether $X$ has a meaningful conditional effect on $Y$ from the magnitude and significance of the coefficient on the interaction term either. As we showed earlier, it is perfectly possible for the marginal effect of $X$ on $Y$ to be significant for substantively relevant values of the modifying variable $Z$ even if the coefficient on the interaction term is insignificant. [..] It means that one cannot determine whether a model should include an interaction term simply by looking at the significance of the coefficient on the interaction term. [..]
The point here is that the typical results table often conveys very little information of interest because the analyst is not concerned with model parameters per se; he or she is primarily interested in the marginal effect of $X$ on $Y$ ($\beta_1 + \beta_3Z$) for substantively meaningful values of the conditioning variable $Z$.
So what they propose is a marginal-effect plot. Following How can I explain a continuous by continuous interaction, I can roughly replicate it in R as follows:
##marginal effect plot
at.changenonslav <- seq(min(dta$changenonslav, na.rm=T),
max(dta$changenonslav, na.rm=T), 1)
sl_slopes <- coef(slvote.mod)['slavicshare'] +
coef(slvote.mod)['slavicshare:changenonslav'] * at.changenonslav
library(msm)
sl_estmean<-coef(slvote.mod)
sl_var<-vcov(slvote.mod)
SEs <- rep(NA, length(at.changenonslav))
for (i in 1:length(at.changenonslav)){
j <- at.changenonslav[i]
SEs[i] <- deltamethod(~ (x2) + (x10)*j, sl_estmean, sl_var)
}
sl_upper <- sl_slopes + 1.96*SEs
sl_lower <- sl_slopes - 1.96*SEs
plot(at.changenonslav, sl_slopes, ylim=c(-.05, .25), type='l',
ylab='Marginal effect of X ("slavicshare") on Y',
xlab='Level of Z ("changenonslav")')
points(at.changenonslav, sl_upper, type = "l", lty = 2)
points(at.changenonslav, sl_lower, type = "l", lty = 2)
abline(h=0, col="grey")
Which outputs this (not unlike the graph in Stata by the authors of the paper):
Brambor, Clark and Golder (2006) go on to suggest that (slightly paraphrased):
95% confidence intervals around the line allow us to determine the conditions under which $X$ has a statistically significant effect on $Y$—they have a statistically significant effect whenever the upper and lower bounds of the confidence interval are both above (or below) the zero line.
There are also the effect displays from effects package in R:
##effects display
require(effects)
plot(effect("slavicshare:changenonslav", slvote.mod, xlevels=4))
Which outputs the following graph, although it seems to me that this is a different (more detailed) presentation of the marginal-effect plot above, and that it doesn't help directly wrt statistical significance (i.e. the interpretation is less intuitive and more subjective):
Another way to determine significance of interactions seems to be suggested (although not explicitly) in Fox 2003. There each linear model is followed by an Anova table which, if my understanding is correct, indicates if the interaction effect is statistically significant overall.
##Anova
require(car)
Anova(slvote.mod)
Which outputs:
> Anova(slvote.mod)
Anova Table (Type II tests)
Response: xenovote
Sum Sq Df F value Pr(>F)
slavicshare 41.374 1 8.8471 0.0041787 **
changenonslav 34.098 1 7.2913 0.0089209 **
[..]
slavicshare:changenonslav 11.377 1 2.4329 0.1239045
Residuals 289.945 62
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
But the two methods seemingly contradict each other in this case, the marginal-effect plot indicating significance, while the Anova indicating lack of significance for the interaction...
So to return to my original question, how can I determine if the marginal effect of $X$ on $Y$ (for different values of $Z$) is overall statistically significant?
Are both the marginal-effect plot and the Anova (as presented above) valid tools for determining the significance of an interaction? Is one preferred over the other? Are there other statistical tools that can assist in this case? I'm interested in statistical approaches more generally, although solutions in R are very much appreciated...
