# What is meant by Bayesian Machine Learning in Regression?

Suppose I have a classification task and I assume a Gaussian discriminative model: $$P(y|x,\theta)= N(y|\mu_x,\sigma_x)$$ where $$x\in \{0,1\}$$ are the features (1 for Company A, 0 for Company B) and $$y\in R$$ are the delivery time. The book "Probabilistic Machine Learning: An Introduction" (Murphy, 2022) said that there are two ways to model the parameters: $$\mu_x,\sigma_x$$

1. Use MLE which solves the parameters as the empirical mean and variance respectively.
2. Do a Bayesian approach, utilizing $$P(\theta|y,x)$$

I fully understand the derivation and reasoning for using choice 1. However, I can't wrap my head around choice 2.

Suppose I use a full Bayesian approach and I used a Gaussian prior $$N(\mu_x|\mu_0, \sigma_0)$$ to model (assuming that $$\sigma_x$$ is given for simplicity): $$P(\theta|y,x,\sigma_x)=N(\mu_x|\hat{\mu},\hat{\sigma})$$ where $$\hat{\sigma}$$ and $$\hat{\mu}$$ are linear weighted combinations of the prior parameters and the parameters that arrived from using MLE.

After computing, in a fully Bayesian manner, the parameters $$\hat{\sigma},\hat{\mu}$$ of $$P(\mu_x|y,x,\sigma_x)$$, how can I use these to solve the earlier prediction (regression) task ?

• How is $y$ related to the classification task? What is the binary variable? Commented Jun 1, 2022 at 8:39
• If $y$ is the label how can it have a normal distribution ? Commented Jun 1, 2022 at 9:32
• @J.Delaney sorry, I had mistaken $x$ for $y$. I had corrected the mistake thank you! Commented Jun 1, 2022 at 9:53
• Do you mean to put a Gaussian prior distribution on a parameter in, say, a logistic regression?
– Dave
Commented Jun 1, 2022 at 9:56
• If you are doing classification you need a latent variable that captures class probabilities, you then compute posterior probabilities of that variable which gives you the probability of that class given the observed data. edit: it's still unclear from the question if this is regression or classification. If you meant regression, then the e.g. you get distribution of posterior mean (c.f. confidence interval in frequentist analysis).
– Lulu
Commented Jun 1, 2022 at 10:21

A Bayesian computation provides not just point estimates of the unknown parameters (as in "standard" regression) but a full probability distribution of those parameters.

$$y|x,\theta \sim f(x,\theta)$$

where $$\theta$$ represents the unknown parameters of the model, then the Bayesian calculation gives the posterior probability distribution of $$\theta$$,

$$\hat P(\theta) \equiv P(\theta | x,y) \propto f(x,\theta)\pi(\theta)$$

from which you can calculate the prediction for a new data point $$x^*$$, by integrating over all possible values of $$\theta$$ given its posterior distribution

$$P(y^*|x^*) = \int d\theta P(y^*|x^*,\theta) \hat P(\theta)$$

which is again a probability distribution for $$y^*$$ (called the posterior predictive distribution).

You can use this distribution to calculate, for example, the mean of $$y^*$$ as well as intervals having a particular probability of containing $$y^*$$ (Credible Intervals) , as demonstrated for example by this plot (taken from this blog )