1
$\begingroup$

I am analysing the effect of a randomised treatment on several outcome variables.

First i am interested in whether the treatment changes the first outcome (non-pecuniary value) by controlling for demographics characteristics X. $$ Value = \beta_0 + \beta_1*Treatment_i + \beta_2*X_i + \epsilon_i $$

However, I am also interested if it affects another outcome (pecuniary value) that is according to the literature connected to the first variable $Value$. Therefor, I originally intended to control for it to avoid omitted variable bias.

$$ Money_i = \alpha_0 + \alpha_1*Treatment_i + \alpha_2*Value_i + \alpha_3*X_i + \epsilon_i$$

Now lastly, I am interested in the effect of the treatment on a third outcome variable (Probability to go to school) that is according to the literature related to the variables $Money_i$ and $Value_i$. Hence, therefor I thought to control for both of them in the regressions.

$$ Probschooling_i = \gamma_0 + \gamma_1*Treatment_i + \gamma_2*Value_i+\gamma_3*Money_i + \gamma_4*X_i + \epsilon_i$$

However, now I came across the concept of "bad" control variables that bias inference. In my case, as the variables $Value_i$ and $Money_i$ are themselves dependent variables in the other regressions, I thought that therefore they can be called bad control variables (some kind of dependencies). My question now is if that is true? Are they bad control variables so that i do need to take them out (and to not bias the treatment effect) ? Or can I just leave it as I have it (and so to avoid omitted variable bias)?

$\endgroup$
3
  • $\begingroup$ The $X_i$ surely needs a coefficient as well. $\endgroup$ Jul 6 at 11:43
  • $\begingroup$ sorry, I have added it now $\endgroup$
    – Laura
    Jul 6 at 11:44
  • $\begingroup$ Being "bad" is not a technical term, however what will happen if you involve a variable that is itself a linear combination of the other variables you are interested in plus some residual term is that it isn't possible to tell apart the influence of those other variables directly and through that variable modelled earlier, i.e., Treatment will affect Money on its own and through Value, and to what extent it is one or the other may be hard to identify from the data. Because of this results may be unstable and hard to interpret. $\endgroup$ Jul 6 at 11:48
1
$\begingroup$

The answer would probably depend on what causal structure you are assuming. To take your example with the Money variable: if you assume that treatment affects both Value and Money, and Value also affects Money, then whether to control for Value or not depends on your question!

e.g.:

dag <- dagitty::dagitty("dag {
                 Treatment -> Value -> Money
                 Treatment -> Money
                 }")

plot(dag)

If you are interested in the total effect of Treatment on Money -- i.e., does Treatment increase Money in general -- then controlling for Value will invalidate your inference, because you will be "cancelling" one important way in which Treatment does cause and increase in Money, namely because it increases Value, which then itself increases Money.

If you are interested in the direct effect of Treatment on Money -- i.e., does Treatment increase Money "directly", without counting the path that goes through Value? -- then you should control for value.

I find McElreath's explanations quite enlightening, in this lecture and the next: https://www.youtube.com/watch?v=e0tO64mtYMU&list=PLDcUM9US4XdNM4Edgs7weiyIguLSToZRI&index=5

Also there is a paper, geared more towards psychologists, that would likely help you wrap your head around this: https://journals.sagepub.com/doi/10.1177/2515245917745629

$\endgroup$
5
  • 1
    $\begingroup$ I agree that this is an important aspect of whether you would like to control for value. But, if there are unobserved confounders for value (variables affecting both value and money) then value is a bad control regardless of whetther you would like to control for it or not. Regressing an outcome on treatment alone however is guaranteed to yield something unbiased, treatment being randomised. Mastering Metrics has a good section on this $\endgroup$
    – Jonathan
    Jul 6 at 12:18
  • $\begingroup$ In case of when my explanatory variable is not a randomised treatment but rather just an observable variable, does then the controlling/not controlling situation change? Is then the omitted variable bias more present such that it is better to add e.g. value as control? $\endgroup$
    – Laura
    Jul 6 at 13:29
  • $\begingroup$ @Laura Yes. When treatment is randomised there can be no OVB but conditioning on an outcome is quite likely to introduce bias. When treatment is not randomised, you have to worry about OVB. Adding a bad control may not introduce a lot of bias since it was already there to begin with $\endgroup$
    – Jonathan
    Jul 6 at 14:57
  • $\begingroup$ Okey - so in that case it sound to me that with the randomised treatment I will only leave in the demographic control variables. Whereas for a observable explanatory variable, adding value/money and demographic control variable seems to be more appropriate. Does that make sense? $\endgroup$
    – Laura
    Jul 6 at 15:37
  • $\begingroup$ @Laura if your goal is causal inference, in the absence of an experiment, there's really no way out, you have to go through the process of thinking about your assumptions and the causal structure you are testing. I highly recommend the Julia Rohrer paper I mentioned in my answer. I think it would really help you get started. $\endgroup$ Jul 8 at 9:37
0
$\begingroup$

As the other answer states, if you are interested in the effect of the treatment on money that does not work via value (the direct effect), then you would, in principle, like to control for value. If it Is the total effect (direct+indirect) you are after, you should not be controlling for variables that are (potentially) outcomes of the treatment. It is correct that you need to think about which effect you actually want to estimate.

However, value is a bad control and you will not be able to get an estimate of the direct effect that is as credible as one of the total effect. This is because there are probably other variables that cause both value and money (confounders) that you have not observed. Controlling for value would therefore introduce bias unless you are able to also control for these confounders.

It is usually very hard to make a credible case that you have controlled for all confounders. The beauty of randomised treatment assignment is that there are no confounders (for treatment), and you can therefore get a very credible estimate of the (total) causal effect of the treatment on the outcome. If you decide to try and estimate the direct effect, then you are back to worrying about confounding and the point of having run an experiment is kind of lost. Similarly, you would not want to throw away a good quasi-experiment by introducing bad controls.

Bad controls are less bad with a non-experimental research design. In that case you would have to worry about confounding to begin with and the “bad” controls would only mean additional potential confounders to take into account.

It is fine to use “demographic control variables” if they are determined before the treatment. However, you would more typically use these to demonstrate that randomisation has produced a treatment and control group that are balanced with respect to observed characteristics; e.g, a table with the means for both groups. If you have balance, the point of controlling for these variables would be to improve precision.

If you do, it should be in addition to your estimate without any control variables at all which is the main result of your experiment (unless you had pre-specified otherwise)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.