Interpretation of a model with interactions of a treatment variable with more than one dummy

Question

I'm struggling with the interpretation of a regression model. My advisor wants me to check whether the effect of a treatment dummy varies with several dummy variables in one model (female dummy, high education dummy and high income dummy).

The regression model looks like this:

$$ y = b_0 +b_1 treatment+b_2 female+b_3 higheduc+ b_4 highincome+b_5 treatment \cdot female +b_6 treatment \cdot higheduc +b_7 treatment \cdot highincome + errorterm $$

I know that in a model with only one interaction (e.g. of female and treatment), the interaction term shows me how much the treatment effect differs among females relative to the basecategory of males. But how is it when I have more than one interaction of the treatment dummy with demographics? Is this still interpretable as the average difference in the treatment effect between gender groups?

I do not think so because the treatment dummy should then show the treatment effect for all interaction dummies equal to zero, i.e. for males having no high income and no high education?

In my view I would have to run 3 regressions, one for each check whether the treatment effect varies with the demographics. But maybe you can correct me :)

Best regards!

Answer 1

It's almost never a good idea to run multiple separate sub-models on the same data. You run a risk of omitted-variable bias in a sub-model that leaves out an outcome-associated predictor. You are implicitly averaging over the cases that should be distinguished based on the values of the omitted predictors, but aren't because you left those predictors out. If the distribution of those cases in your data set isn't representative of the underlying population of interest, you can get into even more trouble.

The most efficient use of your data is to build a single model with all predictors and interactions of interest, provided that you have enough data to avoid overfitting. You then can get the most reliable predictions for any specified set of predictor values, and proceed accordingly to generalize to the underlying population.

As you note, there is a problem is in interpreting interaction coefficients, for example $b_5$ for the female by treatment interaction, in this situation. It is "how much the treatment effect differs among females relative to the base category of males" when all other predictors are held constant. In your model there are no other interactions involving female, so that's the same treatment-effect difference for female within any class of high education or high income.

That coefficient, however, is not necessarily an "average difference in the treatment effect between gender groups." Say that there is a significant $b_6$ coefficient for the high education by treatment interaction, and that female cases are relatively over-represented in the high education class. Then the "average treatment effect between gender groups" has to take into account the interaction coefficients and gender imbalances associated with all the predictors interacting with treatment.

The solution to getting "averages," however, is not to use separate sub-models with individual interaction terms, for the reasons outlined in the first paragraph. Get a full model, think carefully about what types of "averages" you want to examine, and evaluate the reliability of those "average" values by taking into account the (co)variance matrix of the coefficient estimates.