Comparing interaction effects using different variables. Possible?

Question

I want to compare which are the "most important" interaction effects (in a data driven way, I realize that it is has downsides). I realize that for substantial researchers this does not make sense, yet am curious to the procedure.

Imagine we have the following variables: $X_1, X_2, X_3, X_4$, $y$ and $z$ (all equal length).

I'm going to assume that we are constantly interested in change between the main-effects model and the interaction model (right?).

Two cases:

1) What happens when we compare:

$$y | X_1B_1 + X_2B_2$$

$$y | X_1B_1 + X_2B_2 + X_1X_2B_{12}$$

against

$$y | X_3B_3 + X_4B_4$$

$$y | X_3B_3 + X_4B_4 + X_3X_4B_{34}$$

Can I safely assume that the one with the higher R-squared change would be more interesting?

2) Imagine that, to save my life, I have to make a best guess to advise someone what is the "most interesting" / "strongest" interaction effect. Could we compare:

$$y | X_1B_1 + X_2B_2$$

$$y | X_1B_1 + X_2B_2 + X_1X_2B_{12}$$

with

$$z | X_3B_3 + X_4B_4$$

$$z | X_3B_3 + X_4B_4 + X_3X_4B_{34}$$

Might it still be possible to just simply compare the R-square change? Should we perhaps look at relative R-squared change difference? What about the F-test? I also wonder if the "interestingness" of an interaction effect would depend on the strength of the main effect (I guess that last one is the most subjective). Thanks.

Answer 1

I just going to go down the line and try to answer your questions.

Might it still be possible to just simply compare the R-square change? Should we perhaps look at relative R-squared change difference?

• Looking at a change in $R^2$, relative $R^2$, or AIC is one way that people measure the importance of a particular predictor in general linear models. The pitfalls of this approach can be found here. When I use the vocabulary predictor this also includes an interaction effect. An interaction effect of two variables $X_1$ and $X_2$ can be statistically significant even if the individual main effects of $X_1$ and/or $X_2$ is not significant. So you can use the $R^2$, relative $R^2$, or $AIC$ change of the interaction term ($X_1$ * $X_2$) as a measure of importance

Can I safely assume that the one with the higher R-squared change would be more interesting?

• I'm not sure exactly what you mean by interesting. I'm going to use "interesting" as a measure of predictive power. Another way to assess the predictive power of including an interaction effect is to build 2 models in a cross validated fashion. The first model does not include the interaction effect and the second model does include the interaction effect. Cross validation repeatedly breaks you dataset into training and testing sets. You can derive the SSE for a regression model or Accuracy for a classification model from each testing set. You can then look at the difference between $SSE$ or $Accuracy$ between the two models and see if there is a statistical difference. If there is a statistical difference you can say the interaction effect increased predictive performance

What about the F-test?

An F-test is simply an overall model test. If an F-test is statistically significant it says that one [not any one in paritcular] of the $Beta$ values on your predictors is statistically different than 0

I also wonder if the "interestingness" of an interaction effect would depend on the strength of the main effect (I guess that last one is the most subjective).

It is possible to have a completely insignificant main effect on both variables but still have a significant interaction effect

Answer 2

I am a little confused by what exactly you mean by "more interesting". I think what you are asking is, "is including the interaction term more interesting in this model". If this is what you are asking then to your question 1 I would say, all else equal the model which is more interesting in terms of interaction effect is the one in which there is the greatest change in the R-squared as a result of including the interaction term.

As for question 2. I would have to give the same answer.

In addition I believe that the relative R-squared change difference might be interesting since an interaction model which explains x% more of the data in terms of R-squared might be a useful exploration. Finally, I think you should definitely think about testing the change in explanatory power as measured by the F-stat (which is particularly easy in the case of only adding an interaction term since the p-value on the t-stat on the interaction term is identical to the p-value on the f-stat of the difference between the model not including the interaction term and that including the interaction term). This statistic is useful since it indicates whether including the interaction term is statistically significant.