6
$\begingroup$

I'm running a randomized controlled trial which has good balance in the co-variates. I'm unsure whether to use:

  • OLS: $P(Y_i=1) = \beta_0 + \beta_1 \text{Treat} + \epsilon_i$.

This is problematic because the variance of binary $Y$ is not homoskedastic, thus invalidating the OLS assumption. Note that the right-hand side (RHS) only has the 0-1 indicator $\text{Treat}$, so I don't have the problem of the RHS going beyond the 0-1 range.

$\beta_1$ in this case is simply the difference in sample mean between the control and the treatment group.

  • Logistic: $\text{logit}(P(Y_i=1)) = \beta_0 + \beta_1 \text{Treat}$

This is problematic because the correct model needs to include other co-variates (despite balance) and not just the $\text{Treat}$ indicator. The substantive effect of $\beta_1$ on $P$ then depends on the value of these other co-variates. However, then I no longer see the benefit of my experimental design, which allows me to not control for any other co-variates in the OLS case.

SUMMARY of the answers I've got so far:

  • @glen_b suggests that I should include predictors in both OLS and logistic despite experimental design in order to improve precision. In this case of binary outcome, including other predictors means that OLS is not an option anymore (because the RHS now can go outside the 0-1 range)

  • @AMD suggests how to interpret the treatment effect in logistic regression, regardless of whether other predictors are included: $$ \frac{p}{1-p} = \exp(\beta_0 + \beta_1 \text{Treat} + \beta_2 X) $$

When $\text{Treat} = 0: \frac{p}{1-p} = \exp(\beta_0 + \beta_2 X)$

When $\text{Treat} = 1: \frac{p}{1-p} = \exp(\beta_0 + \beta_2 X) \times \exp(\beta_1)$

Therefore, the treatment effect is that it multiplies the odd-ratio ($\frac{p}{1-p}$) by $\exp(\beta_1)$. Pro: the size of this effect does not depend on other $X$. Con: This is not directly about $p$, which some may be interested in.

REMAINING QUESTION due to several conflicting advices: If I don't care so much about precision and just want to prove the causal effect of my treatment, is it okay to not include other predictors in the logistic given the experimental design? If I fail to include some interactions (with observable and/or unobservable), is $\beta_1$ of $\text{Treat}$ still a consistent estimate of the causal effect?

$\endgroup$
  • 2
    $\begingroup$ Maybe I am being a little dense right now, but I don't see how you arrive at the conclusion that you have an argument that would allow you to omit variables for OLS that wouldn't also apply to logistic regression. How does the argument work for one and fail for the other? $\endgroup$ – Glen_b -Reinstate Monica Dec 17 '14 at 21:29
  • $\begingroup$ Sorry if I'm not clear. Since I'm running RCT, my treatment is orthogonal to everything else. So for OLS, I can just shove all predictors into the error term (which is still orthogonal to the treatment). OLS is then simply a comparison of outcome mean between treatment and control group. I can't do that with logistic, because when I interpret the substantive effect of the treatment, it depends on the value of the other predictors. So I'm not sure whether to include predictors in logistic. $\endgroup$ – Heisenberg Dec 17 '14 at 21:36
  • 1
    $\begingroup$ When you run a logit, you're saying that the expectation of y is a logit function of $X'\beta$. Marginal effects do indeed depend on the values of the other predictors, which is why logit models are typically interpreted in terms of odds ratios, or marginal effects of one variable at a particular value of the other variables (usually the mean). $\endgroup$ – generic_user Dec 17 '14 at 21:41
  • 1
    $\begingroup$ In an RCT with binary treatment and binary outcome, the OLS estimator is the same as the difference in sample means of treatment group and control group. The average treatment effect is identified, you can omit other predictors. But of course, most treatment effects have interactions. Are you interested in those? $\endgroup$ – CloseToC Dec 17 '14 at 21:42
  • 1
    $\begingroup$ I'm not sure what he means by that. Maybe he means the following: say you've got smoking, cancer, and gender. The expectation of how much smoking would change your odds of getting cancer is mediated by your gender, so modeling without the interaction doesn't get you the expectation of what would happen to person $i$. Does this mean that you shouldn't include variables if you don't know perfectly what interactions are important? Sort of a subtle question. $\endgroup$ – generic_user Dec 18 '14 at 17:49
6
$\begingroup$

Edit for clarity: It looks like my responses here have led to some clarifying additions to the question or additional information in comments, which make parts of my answer now at least partially obsolete. However, I plan to leave my answer as is, partly for context and partly because I believe the points raised may be relevant to later readers.

Changing the order a little:

Logistic: ... This is problematic because the correct model needs to include other co-variates (despite balance) and not just the Treat indicator.

Both models should include predictors that would be likely to have a substantive effect, even if the design is perfectly balanced and there are no interactions between variables. For example, to omit them if you have them would reduce power - for example, in OLS it inflates error variance by incorporating their effect in the error.

[Further, if there can be interactions between variables, you won't get the expectation in the model right. You should be consider diagnostic checks for potential interactions with such variables, included or not.]

OLS: ... This is problematic because the variance of binary Y is not homoskedastic,

That's not even the worst problem with OLS on this. The even more serious problem is that once you include the other covariates*, the relationship cannot be linear -- you will - necessarily have a model that predicts probabilities that are negative and others that are greater than 1 (predicted rather than fitted).

*(which I strongly believe you should, unless you are confident they are actually unrelated to $Y$)

$\endgroup$
  • $\begingroup$ 1) If I only have 0-1 $\text{Treat}$ on the right-hand side, doesn't it always stay within the 0-1 range? 2) If I need to include the other predictors in logistic, is the model wrong if there's some unobservable predictors? In OLS + RCT, unobservable predictors are not a problem since they are orthogonal to treatment. But in logistic, the substantive effect of treatment depends on the value of other predictors. $\endgroup$ – Heisenberg Dec 17 '14 at 21:33
  • $\begingroup$ If I need to include other predictors in the logistic, what's the benefit of the experimental design? Indded, if there is some interaction with unobservable predictors (which I can't include obviously), the expectation in my model will STILL be wrong, despite the experimental design? $\endgroup$ – Heisenberg Dec 18 '14 at 0:40
  • $\begingroup$ The existence of unobservable predictors is why we randomize to treatment (and why logistic regression tends to focus on odds ratios). The possibility of interactions with missing variables might lead to conditionally biased mean estimates (conditional on the missing variable, that is). We can't do anything about the effect on variance of missing predictors, so we have to accept loss of power. $\endgroup$ – Glen_b -Reinstate Monica Dec 18 '14 at 0:46
  • $\begingroup$ Why would he care about heteroscedasticity and predictions out of [0,1] if he is only interested in the treatment effect? Heteroscedasticity can be corrected and the [0,1] thing is only problematic if he wants out of sample predictions. $\beta_1$ will capture the average treatment effect nonetheless. $\endgroup$ – Andy Dec 18 '14 at 18:09
  • $\begingroup$ Given given the more recent information in comments and additions to the question, the OP probably doesn't care all that much about those two things. $\endgroup$ – Glen_b -Reinstate Monica Dec 18 '14 at 23:19
3
$\begingroup$

A lot of economists use linear probability models, arguing that LPM provides the linear approximation of the conditional expectation function, which is often considered "good enough." Consistent (in large samples) standard errors can be gotten by using ``robust'' variance-covariance matrix estimators.

This is an OK argument if you really just want $\beta$, and want it to be interpretable as a conditional expectation in the larger group. You don't want to do this if you have any interest in prediction.

In reality though, arguing that $\beta$ increases a probability by a certain amount can only make sense on average (hence conditional expectation in the sample, which you generalize to the population). It can't be a description of what you would expect to happen to unit $i$ if you treat them. Because if $i$ has covariates that push them up or down, then adding $\beta$ to the effect of those covariates could lead to probabilities outside of 0/1, which wouldn't make any sense.

That said, logit models involve assuming that the link between the predictors and the outcome is a logit. This can be restrictive.

But you can interpret a simple logit coefficient as an odds ratio by exponentiating it. For example, if $\hat\beta=1$, then you're estimating that the treatment leads to a $e^1=2.7$-times more likely odds of $y$ equaling 1.

$\endgroup$
  • 1
    $\begingroup$ I definitely think it would help if the perceived problem that led to that downvote was explained. All we know is someone thinks something is wrong, but not what. $\endgroup$ – Glen_b -Reinstate Monica Dec 17 '14 at 22:28
  • $\begingroup$ So I agree with most of @ACD said, but the last paragraph is the concern that prompts me to ask in the first place. Is it really valid to run a logit with only treatment and interpret its effect as $e^\beta$? If we include other predictors, 1) $\beta$ may change slightly, and 2) the marginal effect of $\beta$ then depends on the values of other predictors. So we get two very different conclusions regarding the marginal effect of $\beta$ when we do and do not include other predictors. $\endgroup$ – Heisenberg Dec 18 '14 at 0:30
  • $\begingroup$ Not as a marginal effect, but as a multiplicative odds ratio. Irrespective of the other variables in the model, someone who is treated has a $e^\beta$ times more likely chance of $y==1$ than had they not been treated. It says nothing about the baseline probability, for which you need something like a predicted probability at some chosen value of $X_{-T}$. So like, smoking makes you $e^\beta$ times more likely to get cancer. Says nothing directly about the marginal probability. $\endgroup$ – generic_user Dec 18 '14 at 1:03
  • 3
    $\begingroup$ Careful. With a logit link, $e^{\beta}$ gives you the odds ratio. I don't know what a chance is. You can use a log link if you wish to interpret the coefficients as relative risks or probabilities. $\endgroup$ – Andrew M Dec 18 '14 at 1:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.