I have data from an experiment in which participants were randomly assigned to one of two groups and asked a series of opinion questions. One group, the control, was not presented with any additional context beyond a brief introduction and the questions themselves. The other group, the treatment group, was given a textual stimulus providing more information.
There is a single key question, let's call it CHOICE
, which serves as the main question that I am trying to answer. CHOICE
is always one of two answers, A
or B
. I want to know if being in the treatment group had any statistically significant effect on the answer that respondents gave for CHOICE
.
I have read through chapter 8 of Experimental and quasi-experimental designs for generalized causal inference (Shadish, Cook, and Campbell) for some guidance on how to analyze such a randomized assignment experiment. On page 251, they write:
Randomization equates groups on expectations of every variable before treatment, whether observed or not.... Randomization ensures that confounding variables are unlikely to be correlated with the treatment condition a unit receives.
As I understand this (and the rest of the chapter), when I run my logistic regressions on this data, I should not be controlling for other factors because I risk biasing my estimates. I.e. I should be using the formula CHOICE ~ GROUP
rather than CHOICE ~ GROUP + AGE + INCOME + GENDER + ....
If this is correct, can someone explain why in more detail? If not, how should I be addressing this? Thanks!
P.S. In case it's relevant, I am using R
.
GROUP
will be uncorrelated with other regressors, soAGE
,INCOME
, etc. will be clearer effects of these variables onCHOICE
(with all the caveats about collinearity omnipresent in social science data). However, the effect ofGROUP
will be free of these confounders, and the coefficient/its significance should not change much between these two specifications. (If it does, this is indicative of a violation of the pure randomization, such as contamination of the treatment arms or differential response rates in two treatments groups.) $\endgroup$