Is stepwise elimination of insignifcant variables invalid if applied to experimental data?

Question

Setting: Experimental data and control-variables

I want to evalute the (average) treatment effect in a randomized controlled trial. Individuals $i$ in one group received a treatment ($D_i=1$) and in another group a placebo ($D_i=0$).

In order to reduce estimation uncertainty I control for pre-treatment variables which predict the outcome (e.g. survival is prediced by age, this holds regardless of treatment). The regression equation I estimate is: $$Y_i = \alpha + \beta D_i + \gamma X_i,$$where $X_i$ is a vector of covariates.

Selecting $X_i$

There are several aspects to be considered when deciding on $X_i$. By design (randomization) the covariates will be uncorrelated (in expectation) with treatment status. Their inclusion may increases the precision of the $\beta$-estimation if they "suck-up" noise in the outcome variable, but they might also increase estimation uncertainty by costing degrees of freedom.

The choice of covariates is hence non-trivial and of course also leaves room for p-hacking.

My question: Do stepwise procedures (such as the stepwise elimination of variables with insignificant coefficients) invalidate inference regarding $\beta$?

My understanding is that such proceedures generally lead to overfitting. Yet, if I only apply it to variables within the vector $X_i$, and the only inference I care about is regarding $\beta$ I fail to see how this would become a problem. What would be an example for a data generating process illustrating this?

Edit: Schematic example for a stepwise proceedure.

1. Let $X_i$ contain all base-line variables, transformations of those (e.g. logs squares), and their interactions.
2. Estimate $Y_i = \alpha + \beta D_i + \gamma X_i$, using OLS
3. Drop from $X_i$ all entries variables for which the coefficient estimate in 2 had a p-value below, say, 5%
4. If any variables were dropped in 3., goto 2, if not continue to 5
5. Report the estimate for $\beta$ obtained in the last iteration of 2.

My intuition is that this would yield unbiased estimates of $\beta$, but biased estimates of $SE[\hat\beta]$. But I might be wrong and happy to be proven wrong. If estimates for $\beta$ would be biased I would be interested to hear why/see an example for such a D.G.P.

EDIT2: I will select the answer to this question, which provides at least a scematic example of a d.g.p. with which I would get undersized test restults for the ATE, under random treatment assignment (even if that only works in small samples).

Answer 1

Yes, stepwise methods invalidate inference in this setting. Variables are retained because either (1) they are truly strong or (2) their effects are mis-estimated to be too far from zero. This creates a selection ("publication") bias. Even more clearly, variable selection results in a biased-low estimate of $\sigma^2$ which you can almost see from just looking at the formula for $\hat{\sigma}^2$ which is the sum of squared residuals divided by $n - p - 1$ where $p$ is the number of slopes in the model. With stepwise variable selection, one could say that $p$ is dishonestly low. Analytsts take $p$ to be the number of retained variables and not the number of candidate variables. The formula mandates that $p$ be non-stochastic, i.e., pre-specified.

Generally speaking, it is a mistake to think that stepwise variable selection improves estimation of $\beta$. Using outside knowledge to specify variables to include in the model can improve everything, but using the same dataset to select which parameters to estimate does not. Some of the information in the data is diverted to vainly try to answer the question "which predictors are important?". This information is better used in estimation of effects.