14
$\begingroup$

In psychology and other fields a form of stepwise regression is often employed that involves the following:

  1. Look at remaining predictors (there are none in the model at first) and identify the predictor that results in the largest r-square change;
  2. If the p-value of the r-square change is less than alpha (typically .05), then include that predictor and go back to step 1, otherwise stop.

For example, see this procedure in SPSS.

The procedure is routinely critiqued for a wide range of reasons (see this discussion on the Stata website with references).

In particular, the Stata website summarises several comments by Frank Harrell. I'm interested in the claim:

[stepwise regression] yields R-squared values that are badly biased to be high.

Specifically, some of my current research focuses on estimating population r-square. By population r-square I refer to the percentage of variance explained by the population data generating equation in the population. Much of the existing literature I am reviewing has used stepwise regression procedures and I want to know whether the estimates provided are biased and if so by how much. In particular, a typical study would have 30 predictors, n = 200, alpha of entry of .05, and r-square estimates around .50.

What I do know:

  • Asymptotically, any predictor with a non-zero coefficient would be a statistically significant predictor, and r-square would equal adjusted r-square. Thus, asymptotically stepwise regression should estimate the true regression equation and the true population r-square.
  • With smaller sample sizes, the possible omission of some predictors will result in a smaller r-square than had all predictors been included in the model. But also the usual bias of r-square to sample data would increase the r-square. Thus, my naive thought is that potentially, these two opposing forces could under certain conditions result in an unbiased r-square. And more generally, the direction of the bias would be contingent on various features of the data and the alpha inclusion criteria.
  • Setting a more stringent alpha inclusion criterion (e.g., .01, .001, etc.) should lower expected estimated r-square because the probability of including any predictor in any generation of the data will be less.
  • In general, r-square is an upwardly biased estimate of population r-square and the degree of this bias increases with more predictors and smaller sample sizes.

Question

So finally, my question:

  • To what extent does the r-square from stepwise regression result in a biased estimate of population r-square?
  • To what extent is this bias related to sample size, number of predictors, alpha inclusion criterion or properties of the data?
  • Are there any references on this topic?
$\endgroup$
  • 3
    $\begingroup$ The main version of the Stata FAQ you cite predated Frank Harrell's 2001 book Regression modeling strategies. New York: Springer, which is the reference I would start from here. $\endgroup$ – Nick Cox Jun 15 '13 at 8:09
  • 3
    $\begingroup$ I highly recommend reading the book by @FrankHarrell that Nick Cox mentions there; I regularly assign my postgrad students and honors students reading from it (particularly chapter 4). That the R^2 is biased in the presence of variable selection is fairly easy to see by simulating many data sets (e.g. n=100, p=50) that has population correlation of zero and then doing whatever variable selection procedure you wish to show it on. $\endgroup$ – Glen_b -Reinstate Monica Jun 15 '13 at 8:46
  • 5
    $\begingroup$ As the comments note, simulation can show that, in a known situation, stepwise regression will overestimate $R^2$, and they can show how much. But it cannot show how much inflation there is in a situation where you don't know what the population values ought to be. That is, not only are results from stepwise biased, they are biased in ways that are very hard (if not impossible) to estimate. $\endgroup$ – Peter Flom - Reinstate Monica Jun 15 '13 at 11:44
  • 3
    $\begingroup$ If the population R^2 is zero, I would think that the sample R^2 is biased even if you don't use stepwise selection. I suspect (but am not sure) that it would also be biased if the population R^2 is non-zero. $\endgroup$ – mark999 Jun 15 '13 at 22:00
  • 2
    $\begingroup$ To say that the population has an $R^2$ and that we're doing some form of model selection is to make some particular assumptions about the sampling situation - that there's a 'true' model, and that it has a particular size, that there are more variables to be considered than are in the true model (at least potentially more), and so on. I think there's even an implication that not all variables are equally strongly related to the response. To that end, I think any simulations we do to investigate the properties would need to respect all the implications it sets up. $\endgroup$ – Glen_b -Reinstate Monica Jun 19 '13 at 12:20
5
$\begingroup$

Referenced in my book, there is a literature showing that to get a nearly unbiased estimate of $R^2$ when doing variable selection, one needs to insert into the formula for adjusted $R^2$ the number of candidate predictors, not the number of "selected" predictors. Therefore, biases caused by variable selection are substantial. Perhaps more importantly, variable selection results in worse real $R^2$ and an inability to actually find the "right" variables.

$\endgroup$
  • $\begingroup$ Particularly when the number of candidate predictors exceeds the number of observations! $\endgroup$ – Alexis Dec 21 '19 at 17:49
2
$\begingroup$

Overview

Many researchers have discussed the many problems with stepwise regression (e.g., @FrankHarrell (2001) in section 4.3). In particular Harrell notes that "it yields $R^2$ values that are biased high" (p.56). There are several possible interpretations of this statement, based on what you assume is the estimand. If you assume the estimate is some form of $\rho^2$, then the following can be said: While this is true for some combinations of data generating process, sample size, set of predictors and p-value criterion of predictor entry, it is not true in all cases.

Specifically, $R^2$ from stepwise regression is not inherently biased in a particular direction when estimating $\rho^2$. The p-value criterion for entry of predictors in the stepwise regression can be used to modulate the expected value of stepwise $R^2$ (i.e., the estimator of $\rho^2$). Specifically, as the p-value of entry approaches zero, then the probability of any predictor being included in the final model approaches zero, and the expected value of stepwise $R^2$ will approach zero. With a p-value of entry of one, all predictors will be retained, and stepwise $R^2$ will display the same bias that $R^2$ shows with all predictors. The bias is monotonically related to the p-value of entry. Thus, there will be a p-value of entry which results in an unbiased estimate of $\rho^2$.

I've run a few simulations under different conditions. The p-value of predictor entry which yielded an approximately unbiased estimate often ranged between .05 and .0001. However, I haven't yet read any simulations which explicitly explore this or provide advice on what kind of bias to expect from published stepwise $R^2$ values using a given p-value of entry and given the features of the data.

That said, for practical purposes, adjusted $R^2$ is specifically designed to estimate $\rho^2$. Thus, it is more suited to estimating $\rho^2$ than merely hoping that the p-value of entry in a stepwise regression happens to be correct in order to result in an approximately unbiased estimate.

Simulation

The following simulation has four uncorrelated predictors where population r-square is 40%. Two of the predictors explain 20% each, and the other two predictors explain 0%. The simulation generates a 1000 datasets and estimates stepwise regression r-square as a percentage for each dataset.

# source("http://bioconductor.org/biocLite.R")
# biocLite("maSigPro") # provides stepwise regression function two.ways.stepfor 
library(maSigPro)
get_data <- function(n=100) {
    x1 <- rnorm(n, 0, 1)
    x2 <- rnorm(n, 0, 1)
    x3 <- rnorm(n, 0, 1)
    x4 <- rnorm(n, 0, 1)
    e  <- rnorm(n, 0, 1)
    y <- 1 * x1 + 1 * x2 + sqrt(3) * e
    data <- data.frame(y, x1, x2, x3, x4)
    data
}

get_rsquare <- function(x, alpha=.05) {
    fit <- two.ways.stepfor(x$y, subset(x, select=-y),  alfa=alpha)
        class(fit) <-'lm'
        summary.lm(fit)$r.square * 100
}

The following code returns the r-square with an alpha for entry of .01, .001, .0001, and .00001.

set.seed(1234)
simulations <- 1000
datasets <- lapply(seq(simulations), function(X) get_data(n=100))
rsquares01 <- sapply(datasets, function(X) get_rsquare(X, alpha=.01))
rsquares001 <- sapply(datasets, function(X) get_rsquare(X, alpha=.001))
rsquares0001 <- sapply(datasets, function(X) get_rsquare(X, alpha=.0001))
rsquares00001 <- sapply(datasets, function(X) get_rsquare(X, alpha=.00001))

The following results indicate the bias for each of the five alpha of entries. Note that I've multiplied r-square by 100 to make it easier to see the differences.

mean(rsquares01) - 40 
mean(rsquares001) - 40 
mean(rsquares0001) - 40 
mean(rsquares00001) - 40 
sd(rsquares01)/sqrt(simulations) # approximate standard error in estimate of bias 

The results suggest that alpha of entries of .01 and .001 result in positive bias and alpha of entries of .0001 and .00001 result in negative bias. So presumably an alpha of entry around .0005 would result in an unbiased stepwise regression.

> mean(rsquares01) - 40 
[1] 1.128996
> mean(rsquares001) - 40 
[1] 0.8238992
> mean(rsquares0001) - 40 
[1] -0.9681992
> mean(rsquares00001) - 40 
[1] -5.126225
> sd(rsquares01)/sqrt(simulations) # approximate standard error in estimate of bias
[1] 0.2329339

The main conclusion I take from this is that stepwise regression is not inherently biased in a particular direction. That said, it will be at least somewhat biased for all but one p-value of predictor entry. I take @Peter Flom's point that in the real world we don't know the data generating process. However, I imagine a more detailed exploration of how this bias varies over, n, alpha of entry, data generating processes, and stepwise regression procedure (e.g., including backwards pass) could substantially inform an understanding of such bias.

References

  • Harrell, F. E. (2001). Regression modeling strategies: with applications to linear models, logistic regression, and survival analysis. Springer.
$\endgroup$
  • $\begingroup$ It's still biased (I'd say), you've just reduced the bias a lot. $\endgroup$ – Jeremy Miles Jun 18 '13 at 19:16
  • $\begingroup$ @JeremyMiles Yes. But it is not inherently biased in a particular direction. $\endgroup$ – Jeromy Anglim Jun 19 '13 at 0:10
  • $\begingroup$ I'd be very interested to see @FrankHarrell 's take on this. $\endgroup$ – Glen_b -Reinstate Monica Jun 19 '13 at 12:21
  • 1
    $\begingroup$ +1 Nice work! But shouldn't your conclusion be different? Let "$SW(p)$" be estimation of $R^2$ using stepwise regression with p-to-enter of $p$. You have agreed (and partially demonstrated) that for any given population there exists at least one $p$ for which $SW(p)$ is unbiased. Fine: but how do you select that $p$? If you don't know what it is, then it seems you're right back where you started, but this time you know that $SW(p)$ is "inherently" biased unless you made a lucky guess of $p$. $\endgroup$ – whuber Jul 3 '13 at 13:55
  • 1
    $\begingroup$ @whuber I tweaked the final paragraph to hopefully make a few of the points you mention clearer. $\endgroup$ – Jeromy Anglim Jul 5 '13 at 0:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.