3
$\begingroup$

Setting: Experimental data and control-variables

I want to evalute the (average) treatment effect in a randomized controlled trial. Individuals $i$ in one group received a treatment ($D_i=1$) and in another group a placebo ($D_i=0$).

In order to reduce estimation uncertainty I control for pre-treatment variables which predict the outcome (e.g. survival is prediced by age, this holds regardless of treatment). The regression equation I estimate is: $$Y_i = \alpha + \beta D_i + \gamma X_i,$$where $X_i$ is a vector of covariates.

Selecting $X_i$

There are several aspects to be considered when deciding on $X_i$. By design (randomization) the covariates will be uncorrelated (in expectation) with treatment status. Their inclusion may increases the precision of the $\beta$-estimation if they "suck-up" noise in the outcome variable, but they might also increase estimation uncertainty by costing degrees of freedom.

The choice of covariates is hence non-trivial and of course also leaves room for p-hacking.

My question: Do stepwise procedures (such as the stepwise elimination of variables with insignificant coefficients) invalidate inference regarding $\beta$?

My understanding is that such proceedures generally lead to overfitting. Yet, if I only apply it to variables within the vector $X_i$, and the only inference I care about is regarding $\beta$ I fail to see how this would become a problem. What would be an example for a data generating process illustrating this?

Edit: Schematic example for a stepwise proceedure.

  1. Let $X_i$ contain all base-line variables, transformations of those (e.g. logs squares), and their interactions.
  2. Estimate $Y_i = \alpha + \beta D_i + \gamma X_i$, using OLS
  3. Drop from $X_i$ all entries variables for which the coefficient estimate in 2 had a p-value below, say, 5%
  4. If any variables were dropped in 3., goto 2, if not continue to 5
  5. Report the estimate for $\beta$ obtained in the last iteration of 2.

My intuition is that this would yield unbiased estimates of $\beta$, but biased estimates of $SE[\hat\beta]$. But I might be wrong and happy to be proven wrong. If estimates for $\beta$ would be biased I would be interested to hear why/see an example for such a D.G.P.

EDIT2: I will select the answer to this question, which provides at least a scematic example of a d.g.p. with which I would get undersized test restults for the ATE, under random treatment assignment (even if that only works in small samples).

$\endgroup$
  • $\begingroup$ Before this question derails into something to specific, I posted the example-coding follow-up question somewhere else: statalist.org/forums/forum/general-stata-discussion/general/… $\endgroup$ – sheß Jul 19 '16 at 9:52
  • $\begingroup$ Stepwise selection does not improve any aspect of estimation. What made you think it did? $\endgroup$ – Frank Harrell Jul 20 '16 at 14:04
  • 2
    $\begingroup$ I'm struggling to appreciate the question. If there indeed is an effect $\beta\ne 0$, then the null hypothesis is false. Thus the appropriate sense of "bias" must be conditioned on the assignment into treatment and control. If that assignment happens to be orthogonal to the covariates, you will be fine; but if it is not, then there's nothing special or different about your data than any other dataset and we can apply what is generally known about stepwise procedures. $\endgroup$ – whuber Jul 20 '16 at 14:46
  • $\begingroup$ @Frank, the idea would be that it improves something (regarding the point estimates) if little is known about which controls would improve estimation. Then the alternatives (i) using no controls or (ii) using all controls, could both give less precise point estimates, no? Of course, using the right controls is always best, but they are unknown in most settings. $\endgroup$ – sheß Jul 20 '16 at 15:20
8
$\begingroup$

Yes, stepwise methods invalidate inference in this setting. Variables are retained because either (1) they are truly strong or (2) their effects are mis-estimated to be too far from zero. This creates a selection ("publication") bias. Even more clearly, variable selection results in a biased-low estimate of $\sigma^2$ which you can almost see from just looking at the formula for $\hat{\sigma}^2$ which is the sum of squared residuals divided by $n - p - 1$ where $p$ is the number of slopes in the model. With stepwise variable selection, one could say that $p$ is dishonestly low. Analytsts take $p$ to be the number of retained variables and not the number of candidate variables. The formula mandates that $p$ be non-stochastic, i.e., pre-specified.

Generally speaking, it is a mistake to think that stepwise variable selection improves estimation of $\beta$. Using outside knowledge to specify variables to include in the model can improve everything, but using the same dataset to select which parameters to estimate does not. Some of the information in the data is diverted to vainly try to answer the question "which predictors are important?". This information is better used in estimation of effects.

$\endgroup$
  • 2
    $\begingroup$ I don't think it matters but would need you to write out a proposed step-by-step algorithm to know for sure. $\endgroup$ – Frank Harrell Jul 14 '16 at 12:56
  • 1
    $\begingroup$ The algorithm you described is a classic stepwise variable selection approach where a few of the variables are mandated. It suffers from all the problems of stepwise variable selection. $\endgroup$ – Frank Harrell Jul 14 '16 at 13:45
  • 4
    $\begingroup$ For models that are nonlinear in $\beta$ even orthogonality doesn't protect from bias. $\endgroup$ – Frank Harrell Jul 16 '16 at 12:15
  • 2
    $\begingroup$ The main idea is that things like odds ratios and hazard ratios are "non-collapsible" meaning that in nonlinear models that do not have an error variance, model misspecification cannot just inflate $\sigma^2$; it spills over into modifying the $\beta$s. $\endgroup$ – Frank Harrell Jul 18 '16 at 18:38
  • 2
    $\begingroup$ Destruction of the error variance, which affects every aspect of inference (except for $\beta$ point estimates perhaps) is plenty of reason not to use stepwise methods in this context. $\endgroup$ – Frank Harrell Jul 20 '16 at 11:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.