2
$\begingroup$

I have an experiment I have run, and I am testing for heterogeneous treatment effects (pre-registered and not fishing for any particular result!). Let's call the outcome $Y$ and the treatment variable $W$. I have two variables, each of which I think could be responsible for heterogeneous treatment effects, let's call them $A$ and $B$. These two are correlated with each other but not so much as to pose a problem with multicollinearity, and $A$ is dichotomous while $B$ is continuous. Lastly, Let's say I have a set of control variables $X$. I am using linear regression to analyze the results.

The most basic model with no heterogeneous treatment effects would be:

$Y = \beta_0 + \beta_W * W$

Another model would be using $A$ as an interaction term for heterogeneous treatment effects, and including a set of control variables I think might confound the relationship when including this interaction term:

$Y = \beta_0 + \beta_W * W + \beta_A * A + \beta_{INT} * W * A + \beta_X * X$

Testing for heterogeneous treatment effects would then be a matter of testing for a significant difference between the case where $A = 0$ and $A = 1$.

However, I am worried that any significant interaction effect I find might be due to $B$ instead of $A$. Additionally, it seems wrong to include $B$ in the control variables ($X$) because that wouldn't appropriately take the potential interaction into account. What is the best way to test for whether $B$ or $A$ is responsible for the interaction? Would an F-test be appropriate here, i.e. constructing a new model with $B$ instead of $A$, or would controlling for the other variable in the regression model be sufficient? When I do settle on one of these variables for heterogeneous treatment effects, should I include the other as a control variable?

Edit: To further clarify the question, I am mainly interested in adjudicating between A and B as potential sources of effect heterogeneity, but I'm unsure of the appropriate statistical test for doing so.

$\endgroup$
0
$\begingroup$

The problem you have to deal with it seems is one of multicollinearity since you stated that both $A$ and $B$ are correlated. If $A$ and $B$ are not strongly correlated, you may be able to safely ignore the multicollinearity and simply include a model that includes $A$, $B$, any other relevant relevant predictors and relevant interaction terms. If $A$ and $B$ are highly correlated, then you have some work to do and there are a few options on how to proceed.

If it's not terribly important to distinguish unique effects between $A$ and $B$, then you could consider forming a composite variable from $A$ and $B$ and then include that in your model. For example, if it made sense to form an average or weighted average of $A$ and $B$, say $C = (A+B)/2$, then you could include $C$ and the interaction between $C$ and $W$ in your model. Then $C$ would account for the affect of both $A$ and $B$ on $W$. Alternatively, you could perform a principal components analysis on $A$ and $B$ and use several principal components in your model instead of $A$ and $B$ - but this may make interpretation difficult.

Another approach would be to perform an analysis that is not sensitive to multicollinearity. For example, you could consider performing Ridge Regression. This methods allows you to handle multicollinearity problems by creating biased --but potentially more precise -- estimators of the regression coefficients. I won't go into details about this method here as you can easily search for Ridge Regression on CV or check out the text Collinearity and Weak Data in Regression which describes these and other "multicollinearity-proof" methods in detail.

UPDATE BASED ON YOUR ADDITIONAL COMMENT

If I understand your comment correctly, you are interested in determining if there is an interaction effects between $A$ and $B$. You can include interaction terms between and $A$ and $B$ in your model. Your model would then look like this:

$Y = \beta_0 + \beta_WW + \beta_AA + \beta_XX + \beta_{INT_{WA}}WA + \beta_{INT_{WX}}WX + \beta_{INT_{AX}}AX + \beta_{INT_{WAX}}WAX $

Then you can test to determine if there are interaction effects by testing if the estimated coefficient(s) associated with the relevant interaction(s) are (statistically) significantly different from zero.

$\endgroup$
  • $\begingroup$ Thank you for your feedback! A and B are correlated, but not so highly that multicollinearity would be a problem. Additionally, it is important to me to be able to distinguish between A and B - they are competing explanations that I want to be able to adjudicate between. The question is more of a modeling question - if you have two possible sources of heterogeneous treatment effects from an experiment, and you want to adjudicate which of these two variables produces significant effects, what is the appropriate statistical test for that? I will clarify my original post to provide more detail. $\endgroup$ – lish Jan 25 at 16:38
  • $\begingroup$ @lish do you think you could update your question with a statement of your hypothesis? What is your hypothesis? Are you simply interested in testing for interaction effects? To help us understand what you are really after, perhaps you could write out, in plain what A and B are (e.g. a new level of a drug or a two levels of a new treatment, etc.) and then what your hypothesis that you are trying to test is? Once we get a better understanding of that, I think we can provide some more insight. $\endgroup$ – StatsStudent Jan 25 at 18:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.