# Bayesian machine learning for supervised learning

I began to read about Bayesian machine learning. I have an expertise in using algorithms such as gradient boosting or random forests for supervised learning problems. In order to understand the difference of the Bayesian approach I would like to look at an example application of the Bayesian approach for a classical supervised learning problem such as house price prediction (such as the one given in this Kaggle competition) However, I could not find such an example application. Is it because Bayesian approach not suitable for this problem? If it is not then to what kind of problems it is suitable. If it is, how can I solve this problem using the Bayesian approach? Thanks

• One of the simplest Bayesian supervised learning algorithms is (Bayesian) linear regression. I don't have a Kaggle account so can't see the data but I'd imagine linear regression should give OK results here and is a good starting point for Bayesian analysis. Lots on Bayesian regression here: stats.stackexchange.com/questions/tagged/bayesian+regression Nov 8, 2021 at 13:50

It depends what do you mean by "Bayesian machine learning". For example, using Lasso regression is equivalent to using Bayesian regression with Laplace priors for the parameters, for ridge regression, it is equivalent to having Gaussian priors, using Naive Bayes with Laplace smoothing is like assuming uniform prior for the smoothing, etc. Many machine learning models can be interpreted in as special cases of Bayesian models. In scikit-learn you can find the BayesianRidge regressor that works nearly the same as the Ridge regressor.

The above examples refer to maximum a posteriori (MAP) estimation, i.e. finding only the mode of the posterior distribution, rather than full Bayesian estimation, where you would learn the posterior distribution. Learning the full posterior distribution, it would be more complicated and you could use several ways to approximate it, for example Laplace approximation, variational inference, or Markov Chain Monte Carlo sampling. Notice that those methods would be more computationally intensive, would not be available out-of-the-box in the standard machine learning software, and in many cases would need deeper mathematical understanding of the model, what makes them less popular.

Using the example of linear regression, the Bayesian equivalent would not differ very much from the frequentist counterpart. In every case where you could use linear regression, you could use Bayesian flavor as well. The main differences would be that in Bayesian case, you need to decide on priors for the parameters. it will give you uncertainty estimates for free, and it will be more computationally demanding to train than the non-Bayesian model.

• Thank you for this answer. I think understanding the difference between "mode of the posterior distribution" vs. "full posterior distribution" is crucial here. And I think Bayesian methods are useful when you need the full posterior distribution. Nov 8, 2021 at 16:38
• @SanyoMn yes it’s crucial because with mode is just optimization so it’s equally complicated as maximum likelihood or minimizing some loss. Full posterior is useful if you want to learn about uncertainty and that’s one of the reasons you use Bayesian approach.
– Tim
Nov 8, 2021 at 16:57
• I wonder what are some practical cases where you need a full posterior distribution instead of just the mode. Nov 13, 2021 at 19:52
• @SanyoMn every time you need to know uncertainty about the estimates. Many statisticians would say that’s “always”. Moreover, even if you want a point estimate, mode is just one of the possibilities, you might want instead expected value and in such case you need the full distribution.
– Tim
Nov 13, 2021 at 19:58