I am fitting a linear regression model using shrinkage priors (Horseshoe and Laplace/LASSO). This shrinks many of the variables close to zero, but I would like to select the variables. Can I use the posterior samples to select the variables? If so, what approaches have been proposed to select variables based on a posterior sample using shrinkage priors?





Using a Bayesian method, you would treat each possible model as a parameter and marginalize out the parameters. You would need to set priors over the model space as well.

In the most extreme case, it would be an entirely combinatoric exercise. If you can logically restrict the combinations, you can reduce the number of hypotheses.

Let me give you an example, the Current Ratio, and the Quick Ratio are almost identical in finance. If you were testing financial ratios as an element of your model, they should not both be in your model as there is little extra independent information being generated by having them both. Their shared information is so high that even if having both were a better fit the added computational cost of having them both is high because Bayesian posteriors won't double count information while Frequentist estimators will.

So if you are not going to solve it in a combinatoric manner, then you should be thinking in terms of shared information and not variables. The only difficulty with logically reducing your set is that your inference cannot be considered complete because there will be possible ways in which the world is constructed that you have ignored.

  • $\begingroup$ This does not seem to address my question about how to conduct variable selection with shrinkage priors. $\endgroup$
    – Shrimp
    Nov 18 '19 at 9:10
  • 1
    $\begingroup$ That is how you conduct Bayesian variable selection. For non-analytic posteriors, you would do posterior sampling for the integration to marginalize out each model. In some cases, marginalization won't matter because the differences in the likelihoods will be so great that the result is obvious. $\endgroup$ Nov 18 '19 at 15:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.