Stepwise regression for Bayesian models Why isn't stepwise regression, like backward elimination, used for Bayesian models?
What is generally used to find insignificant variables in bayesian methods?
Or does one simply not worry about insignificant explanatory variables, when doing bayesian inference? if so, why?
 A: If I could hazard a guess as to the unpopularity of stepwise regression among Bayesians. 


*

*Different stepping patterns can produce inconsistent results. 

*It can't explore the full space of configurations easily because of the one-at-a-time choice of variables. 

*The choice of model is being explored, and given the model coefficients being estimated etc but the model choice is made ad hoc and not probabilistically as it would be in a fully Bayesian treatment.

*Bayes has gotten more popular with MCMC in the late 80s onwards. I believe stepwise regression is an older method and the bad points above were known at that time.

*Availability of alternatives, e.g. the Bayesian Lasso, sparse models.

A: Because it isn't compatible with Bayesian thinking.
Bayesian inference centers around belief functions and the updating of beliefs. Given certain inputs (prior beliefs and new information) what new beliefs necessarily follow?
Stepwise regression doesn't really fit in that way of thinking. Stepwise regression, especially when combined with cross-validation, can be a great way to generate predictive models. However, there is no principled reason to think the output of stepwise regression is actually representative of the true model. I.e. there is no sensible way to update my underlying beliefs about the true model and therefore there isn't a sensible Bayesian interpretation of the output of stepwise regression (at least not one that I'm familiar with). As was already mentioned, stepwise regression doesn't examine the entire possible model space, which means it can't update your beliefs about that model space.
Stepwise regression also doesn't take prior beliefs into consideration, and as a consequence is totally unbiased between simple and complex models which naturally leads to over-fitting. There are two reasons to be biased against complex models: 1.) given that complexity has no upper bound (you can always make a model more complex), there are always more complex models than simple models no matter where you set the dividing line and 2.) complex models have more ways of going wrong than simple models, even if the ground truth is very complex.
That isn't to say you can't mix and match. You can use stepwise regression in model selection and Bayesian inference in parameter fitting. After all pure Bayesian model selection/specification is very difficult and very often actually impossible (what should your prior belief be concerning models that have never even occurred to you?). You'll just have to accept that you're betting that your chosen model is the right one or right enough.
Edit: I should say great with the caveat that much better model selection methods exist. So not great... Actually pretty terrible.
