Setting: Experimental data and control-variables
I want to evalute the (average) treatment effect in a randomized controlled trial. Individuals $i$ in one group received a treatment ($D_i=1$) and in another group a placebo ($D_i=0$).
In order to reduce estimation uncertainty I control for pre-treatment variables which predict the outcome (e.g. survival is prediced by age, this holds regardless of treatment). The regression equation I estimate is: $$Y_i = \alpha + \beta D_i + \gamma X_i,$$where $X_i$ is a vector of covariates.
Selecting $X_i$
There are several aspects to be considered when deciding on $X_i$. By design (randomization) the covariates will be uncorrelated (in expectation) with treatment status. Their inclusion may increases the precision of the $\beta$-estimation if they "suck-up" noise in the outcome variable, but they might also increase estimation uncertainty by costing degrees of freedom.
The choice of covariates is hence non-trivial and of course also leaves room for p-hacking.
My question: Do stepwise procedures (such as the stepwise elimination of variables with insignificant coefficients) invalidate inference regarding $\beta$?
My understanding is that such proceedures generally lead to overfitting. Yet, if I only apply it to variables within the vector $X_i$, and the only inference I care about is regarding $\beta$ I fail to see how this would become a problem. What would be an example for a data generating process illustrating this?
Edit: Schematic example for a stepwise proceedure.
- Let $X_i$ contain all base-line variables, transformations of those (e.g. logs squares), and their interactions.
- Estimate $Y_i = \alpha + \beta D_i + \gamma X_i$, using OLS
- Drop from $X_i$ all entries variables for which the coefficient estimate in 2 had a p-value below, say, 5%
- If any variables were dropped in 3., goto 2, if not continue to 5
- Report the estimate for $\beta$ obtained in the last iteration of 2.
My intuition is that this would yield unbiased estimates of $\beta$, but biased estimates of $SE[\hat\beta]$. But I might be wrong and happy to be proven wrong. If estimates for $\beta$ would be biased I would be interested to hear why/see an example for such a D.G.P.
EDIT2: I will select the answer to this question, which provides at least a scematic example of a d.g.p. with which I would get undersized test restults for the ATE, under random treatment assignment (even if that only works in small samples).