3
$\begingroup$

One of the prime objections to best-subset and stepwise regression techniques (forward selection and/or backward elimination) is that multiple hypothesis tests are conducted on the same dataset, leading to biased regression coefficient estimates and standard errors.

Can the limitations of stepwise regression be avoided if:

1) The data is split into multiple subsets, with each step of the forward selection or backward elimination conducted on an independent dataset, or

2) Each step of the forward selection or backward elimination is conducted on a bootstrapped sample from the original data?

Solution (1) would require a sufficiently large dataset, hence (2) may be more practical.

Assume for this question that we're interested in building an explanatory or inferential model containing unbiased coefficient estimates, hence predictive modeling techniques like the Lasso and ridge regression are off the table.

$\endgroup$
4
  • 2
    $\begingroup$ Why do you think that LASSO can't be used for inference? There's a fair amount of work on ways to do so. See Statistical Learning with Sparsity, Chapter 6 in particular, for a start. $\endgroup$
    – EdM
    Apr 25, 2019 at 17:54
  • $\begingroup$ @EdM Thanks,, I'll take a look at the text. But my understanding is that the Lasso introduces a penalty parameter, deliberately biasing the coefficient estimates, in order to minimize MSE. An explanatory regression model starts with the assumption you have best linear unbiased estimates. I think they're apples and oranges. $\endgroup$
    – RobertF
    Apr 25, 2019 at 18:01
  • $\begingroup$ See Section 6.4 of the linked book, "Inference via a Debiased Lasso." $\endgroup$
    – EdM
    Apr 25, 2019 at 18:07
  • $\begingroup$ @EdM The debiased Lasso is interesting - but the Lasso is still minimizing MSE, is that right? If I'm looking for the optimal comb. of variables that maximize adjusted R-square I'm asking a different question. $\endgroup$
    – RobertF
    Apr 25, 2019 at 18:28

1 Answer 1

2
$\begingroup$

A fundamental problem with your approach, as you have now elaborated it in one of your comments, is that there is no way to calculate a reliable adjusted-$R^2$ when you have used the outcome to select predictors in a standard stepwise regression. The problem with best-subset or stepwise regressions is not (just) about multiple hypothesis tests.

For standard OLS regression with a pre-specified model, minimizing mean-square error and maximizing $R^2$ are obviously equivalent. The adjusted $R^2$ uses the degrees of freedom spent, the effective number of fitted predictors, to account for the improvement in $R^2$ inherent in adding additional predictors to a model. But if the choice of predictors to add was itself based on their relations to outcome, in unpenalized regression you have used up more effective degrees of freedom than is simply expressed by the number of predictors.

Statistical Learning with Sparsity puts the case quite clearly early on (pp. 17-18):

Suppose we have $p$ predictors, and fit a linear regression model using only a subset of $k$ of these predictors. Then if these $k$ predictors were chosen without regard to the response variable, the fitting procedure “spends” $k$ degrees of freedom. ... However if the $k$ predictors were chosen using knowledge of the response variable, for example to yield the smallest training error among all subsets of size $k$, then we would expect that the fitting procedure spends more than $k$ degrees of freedom. ... Similarly, a forward-stepwise procedure in which we sequentially add the predictor that most decreases the training error is adaptive, and we would expect that the resulting model uses more than $k$ degrees of freedom after $k$ steps. For these reasons and in general, one cannot simply count as degrees of freedom the number of nonzero coefficients in the fitted model.

So your hope to use an adjusted-$R^2$ to find a best stepwise model does not have a reliable theoretical foundation. There is no way in standard stepwise regression to adjust the degrees of freedom for the use of the outcome to select the predictors. So there is no way to obtain a reliable adjusted $R^2$ for model selection with standard stepwise approaches.

The next sentence shows a way out of the problem:

However, it turns out that for the lasso, one can count degrees of freedom by the number of nonzero coefficients ...

The penalization of coefficients in lasso effectively makes up for the use of the outcome to select the predictors. Furthermore, as lasso is based on a defined optimization procedure rather than a highly data-dependent algorithm, a wide range of theoretical supports for the method have been developed that now extend to use of lasso for inference (Chapter 6 of the book).

There are some other problems with your proposed implementations. For example, the standard approach to use bootstrapping to estimate and correct for bias, the optimism bootstrap, uses the reverse of your approach. The idea is that the relationship of the data sample at hand to the population of interest is akin to that of the bootstrap samples to the data sample at hand. So to see how well your model based on your data set might generalize to the full population of interest, you don't develop a model on the full data set and then test it on bootstrap samples, as you propose. Rather, after you have developed a model on the full data set, you develop models from multiple bootstrap samples while testing each of them on the full original sample. The difference between the average of the results for the bootstrap-based models and that for the full model provides an estimate of bias for the modeling procedure, which is then used to correct the model developed on the full data set so that it lessens the bias when applied to the population of interest.

The major issue, however, is that there is now a firm theoretical basis for using lasso even for inference, while there is none for unpenalized stepwise regression.

$\endgroup$
6
  • $\begingroup$ Thank you, this is helpful. If I randomly split my original dataset into 10 subsets, each 1/10 the size of the original, in your opinion would it be a valid approach to fit step 1 (only intercept) of forward selection on the 1st subset, add one variable that maximizes R^2_adj, then fit step 2 regression (intercept + 1 variable) on the 2nd subset and find the 2nd variable that maximizes R^2_adj, and so on for up to 10 steps. Each subset is decoupled from the rest of the data so that the degrees of freedom counter is "reset" for each new subset - but at the cost of sacrificing sample size. $\endgroup$
    – RobertF
    Apr 26, 2019 at 14:07
  • 1
    $\begingroup$ @RobertF I'm not sure that this approach would really "reset" the degrees of freedom at each step, but even if it did this approach would seem to provide unstable selection of predictors that would be highly dependent not only on the original data sample (as is the case even for LASSO) but also on the specific subsetting of the sample. That's particularly a problem if your intent is inference. A useful guide, unless you have tens of thousands of cases, is to use as much of your data as possible to build a model and then test it by extensive repeated cross-validation or bootstrapping. $\endgroup$
    – EdM
    Apr 26, 2019 at 14:36
  • $\begingroup$ Sounds like part of the problem is defining exactly how many degrees of freedom are used when we snoop around in our data. If we could nail down a more rigorous definition I think that would open up more possibilities in modeling. $\endgroup$
    – RobertF
    Apr 26, 2019 at 14:50
  • 1
    $\begingroup$ @RobertF that's a big advantage of LASSO over the other approaches at variable selection: it directly provides the equivalent of how many degrees of freedom have been used. The SLS book in Section 6.3.2 does provide "A General Scheme for Post-Selection Inference" that can be applied to forward stepwise regression, but it's not clear that such would outperform LASSO. Unlike unpenalized forward stepwise regression, LASSO allows predictors to leave, not just enter, the model as the penalty is relaxed. $\endgroup$
    – EdM
    Apr 26, 2019 at 15:04
  • 1
    $\begingroup$ @RobertF I guess that would come down to how to define a properly penalized $R^2$, as the adjusted $R^2$ as usually defined only includes the number of predictors, not the degrees of freedom used up in choosing them. $\endgroup$
    – EdM
    Apr 26, 2019 at 15:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.