0
$\begingroup$

I am trying to get a better grasp of BIC and AIC scores. I know BIC has a harsher penalty than AIC regarding model size (it prefers smaller, less complex models).

Suppose there is a situation where I am working with my friend to predict "score" from 8 predictor variables.

Using best-subset regression based on BIC, I find out that my best model (lowest BIC)is one that has 2 predictors- "hours_studied" and "number_of_questions_practiced". I also find the best one-predictor model just for fun and it turns out the predictor there is "hours_of_sleep_before_exam".

My friend sees my results but now wants to try another method. She wants to do forward selection based on AIC.

Is there any chance she have the same model as mine?

I know that forward selection starts with an intercept only model. It falls under stepwise selection and so it means that my friend would start by checking all potential one predictor models, select the one with lowest AIC and then add a second predictor and so on. I think this process would lead to her having the same one-predictor model as mine because she would see my result too and she would agree that "score" predicted using "hours_of_sleep_before_exam" is the best one-predictor model. Once she decides on this, "hours_of_sleep_before_exam" is going to be part of her model no matter what and so no matter what second predictor she chooses, she wont get the same two predictor model as mine. However, is there any scenario that she might get the same two predictor model as mine?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

As Hastie et al. say on page 86 of Statistical Learning with Sparsity (SLS):

Forward stepwise is a greedy algorithm—at each step fixing the identity of the terms already in the model, and finding the best variable to include among those remaining.

If the first step in forward stepwise identifies hours_of_sleep_before_exam as the best single predictor, then it will be included in all subsequent steps--you can't get rid of it. Your 2-predictor best-subset model based on BIC doesn't include that predictor, so in the scenario you posit those two approaches can't give the same 2-predictor model.

The result, as James et al. say in An introduction to Statistical Learning (p. 208) is:

Though forward stepwise tends to do well in practice, it is not guaranteed to find the best possible model out of all $2^p$ models containing subsets of the $p$ predictors.

That should give you pause to think about whether either of these predictor-selection methods make sense. Extensive discussion on this page explores the limitations of these methods.

One problem is the risk of overfitting. You might get the "best" model for your particular data set, but it might not work well on another sample from your population. Certainly the p-values as usually calculated won't be valid, as they don't take the use of the data in selecting the model into account.*

Another problem is variability in the selected predictors: if you repeated the analysis on a new data sample from the same population with the same method (either best-subset or forward-stepwise), you would probably find different "best" predictors. You can see that for yourself by repeating your modeling on multiple bootstrap samples from your own data set. That might not be a big problem if you are just trying to find a model the "works" for prediction, but people are often tempted to treat the selected predictors as the "most important" ones. That can be a big mistake.

I'd suggest studying the freely available texts linked above for other approaches to model selection. Penalized methods like LASSO, ridge regression, and their hybrid elastic net get around the practical limitation in using best-subset with large numbers of predictors and the irreversible greediness of forward stepwise, while having solid theoretical bases as principled approaches.

*In some circumstances there are ways to account for the use of the data to select the model, as discussed in Chapter 6 of SLS and implemented in the R selectiveInference package.

Improve this answer
$\endgroup$
3
  • $\begingroup$ LASSO is a greedy algorithm that suffers from variability in selected predictors (Frank Harrell likes to remind us of the latter point). In fact, LASSO is fundamentally fairly similar to stepwise regression. $\endgroup$
    – Richard Hardy
    Nov 22, 2021 at 6:20
  • $\begingroup$ @RichardHardy LASSO isn't irreversibly greedy in the way that forward stepwise is. In principle, as the LASSO penalty is decreased it's possible for an initially selected predictor to be removed from the model, so you're not necessarily stuck with your initial choice as you are with forward stepwise. Yes, you still have the problem with variability in selected predictors; how big that problem is depends on the intended use of the model. $\endgroup$
    – EdM
    Nov 22, 2021 at 14:15
  • $\begingroup$ The point that LASSO can drop a previously selected variable is a good one. Otherwise, I do not see a clear qualitative difference between forward stepwise and LASSO. Just wanted to note that the alternative you have suggested is not very different from the original approach. (The idea is not mine; I have borrowed it from some treatments of LASSO, though I do not remember which ones. Perhaps Tibshirani's original paper, perhaps Elements of Statistical Learning, or perhaps another one.) $\endgroup$
    – Richard Hardy
    Nov 22, 2021 at 15:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.