3
$\begingroup$

I have data from an experiment in which participants were randomly assigned to one of two groups and asked a series of opinion questions. One group, the control, was not presented with any additional context beyond a brief introduction and the questions themselves. The other group, the treatment group, was given a textual stimulus providing more information.

There is a single key question, let's call it CHOICE, which serves as the main question that I am trying to answer. CHOICE is always one of two answers, A or B. I want to know if being in the treatment group had any statistically significant effect on the answer that respondents gave for CHOICE.

I have read through chapter 8 of Experimental and quasi-experimental designs for generalized causal inference (Shadish, Cook, and Campbell) for some guidance on how to analyze such a randomized assignment experiment. On page 251, they write:

Randomization equates groups on expectations of every variable before treatment, whether observed or not.... Randomization ensures that confounding variables are unlikely to be correlated with the treatment condition a unit receives.

As I understand this (and the rest of the chapter), when I run my logistic regressions on this data, I should not be controlling for other factors because I risk biasing my estimates. I.e. I should be using the formula CHOICE ~ GROUP rather than CHOICE ~ GROUP + AGE + INCOME + GENDER + ....

If this is correct, can someone explain why in more detail? If not, how should I be addressing this? Thanks!

P.S. In case it's relevant, I am using R.

$\endgroup$
  • 1
    $\begingroup$ GROUP will be uncorrelated with other regressors, so AGE, INCOME, etc. will be clearer effects of these variables on CHOICE (with all the caveats about collinearity omnipresent in social science data). However, the effect of GROUP will be free of these confounders, and the coefficient/its significance should not change much between these two specifications. (If it does, this is indicative of a violation of the pure randomization, such as contamination of the treatment arms or differential response rates in two treatments groups.) $\endgroup$ – StasK Apr 23 '13 at 12:11
1
$\begingroup$

Randomization ensures that covariates are in expectation balanced between the two groups. Balance of covariates implies independence (which is stronger than uncorrelatedness) between treatment assignment and covariates. Given that this independence holds in expectation, we expect a-priori that the inclusion or exclusion of the covariate should not alter the observed treatment effect. If two regressors are uncorrelated with each other, then the multiple regression coefficients and the zero-order regression coefficients are identical. In short, inclusion of the covariates is not expected to change your treatment effect estimate that you observe from the short regression, and it most certainly does not bias your results (assuming that these are actual pre-treatment covariates). The only caveat comes with the in expectation part. In finite samples, randomization may not perfectly balance covariates, and hence residual imbalances may remain.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.