Why isn't stepwise regression, like backward elimination, used for Bayesian models? What is generally used to find insignificant variables in bayesian methods? Or does one simply not worry about insignificant explanatory variables, when doing bayesian inference? if so, why?
5
-
$\begingroup$ Search for "spike and slab priors" and "Bayesian model averaging" - they are probably the closest methods to stepwise algorithms $\endgroup$– probabilityislogicCommented Aug 19, 2016 at 13:37
-
5$\begingroup$ Stepwise regression model-building is garbage, malarkey, hooey, poppycock… it's just pants. It is one of the 'academic' versions of waving goat entrails over a fire whilst wearing a funny hat in order to give one's pronouncements the (slim) veneer of 'being scientific.' Bayesians have more important things to do with their time. $\endgroup$– AlexisCommented Jun 8, 2022 at 20:09
-
2$\begingroup$ Wow, @Alexis - I'm bookmarking the linked answer, as I can foresee a day in the near future when I'm going to want to pass it on to somebody! $\endgroup$– jbowmanCommented Jun 14, 2022 at 0:01
-
$\begingroup$ @Alexis I can’t say that I’ve ever heard a regression strategy called “pants”, but I’ve also saved this for future use! $\endgroup$– DaveCommented Jun 14, 2022 at 22:37
-
1$\begingroup$ @Dave See the UK slang definition under Noun #3. :) $\endgroup$– AlexisCommented Jun 14, 2022 at 23:46
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
2
If I could hazard a guess as to the unpopularity of stepwise regression among Bayesians.
- Different stepping patterns can produce inconsistent results.
- It can't explore the full space of configurations easily because of the one-at-a-time choice of variables.
- The choice of model is being explored, and given the model coefficients being estimated etc but the model choice is made ad hoc and not probabilistically as it would be in a fully Bayesian treatment.
- Bayes has gotten more popular with MCMC in the late 80s onwards. I believe stepwise regression is an older method and the bad points above were known at that time.
- Availability of alternatives, e.g. the Bayesian Lasso, sparse models.
-
4$\begingroup$ On this site you will find unpopularity of stepwise models among Frequentists as well. $\endgroup$– EdMCommented Jun 9, 2015 at 14:01
-
$\begingroup$ True, but the question was about Bayesians and why it isn't really discussed in that sort of literature. $\endgroup$ Commented Jun 9, 2015 at 15:37
$\begingroup$
$\endgroup$
Because it isn't compatible with Bayesian thinking.
Bayesian inference centers around belief functions and the updating of beliefs. Given certain inputs (prior beliefs and new information) what new beliefs necessarily follow?
Stepwise regression doesn't really fit in that way of thinking. Stepwise regression, especially when combined with cross-validation, can be a great way to generate predictive models. However, there is no principled reason to think the output of stepwise regression is actually representative of the true model. I.e. there is no sensible way to update my underlying beliefs about the true model and therefore there isn't a sensible Bayesian interpretation of the output of stepwise regression (at least not one that I'm familiar with). As was already mentioned, stepwise regression doesn't examine the entire possible model space, which means it can't update your beliefs about that model space.
Stepwise regression also doesn't take prior beliefs into consideration, and as a consequence is totally unbiased between simple and complex models which naturally leads to over-fitting. There are two reasons to be biased against complex models: 1.) given that complexity has no upper bound (you can always make a model more complex), there are always more complex models than simple models no matter where you set the dividing line and 2.) complex models have more ways of going wrong than simple models, even if the ground truth is very complex.
That isn't to say you can't mix and match. You can use stepwise regression in model selection and Bayesian inference in parameter fitting. After all pure Bayesian model selection/specification is very difficult and very often actually impossible (what should your prior belief be concerning models that have never even occurred to you?). You'll just have to accept that you're betting that your chosen model is the right one or right enough.
Edit: I should say great with the caveat that much better model selection methods exist. So not great... Actually pretty terrible.
