# Bayesian Regression- Expectation Maximization

In Bayesian regression, we have $$y_i=x_i^{T}w+\epsilon_i$$ where $$w \sim \mathcal{N}(0,\alpha)$$ and $$\epsilon_i \sim \mathcal{N}(0,\frac{1}{\beta})$$. Inference of $$\alpha$$ and $$\beta$$ is done by maximizing the likelihood(or marginal likelihood) given the data. This paper(Appendix A.1) explains that maximization can be done by expectation-maximization(EM).

My questions is that why do we need EM to do the maximization. I understand that EM can be used by treating the $$w$$ as the hidden variable. However, $$w$$ can be integrated out and we can apply gradient descent(GD). Why don't we use GD?

• Maybe i didnt understand well the paper you mention. But it seems like the author show the update when you do gradient descent and EM update. And he explains that EM update gives faster convergence. (Again i read it very fast, so i can be wrong). I focus only on Appendix A to say that :D – PauZen Jan 20 at 9:14
• He doesn't compare gradient descent with EM. He compares EM with another update rule that is similar to EM, but not the same. The other update rule converges faster than EM but its convergence to a local minimum is not guaranteed contrary to EM. – arke Jan 21 at 14:17
• why would you do inference on a hyper-parameter such as $\alpha$? – Carlos Llosa Jan 22 at 0:16
• @CarlosLlosa I don't know much about how to determine the hyper-parameters in Bayesian methods. That is how they are determined in every paper I read. Note that I also know that the same type of inference is done for gaussian process regression too. – arke Jan 22 at 16:53
• @CarlosLlosa Inference in $\alpha$ is very useful when it is actually a vector of $\alpha_i$. In that way, we may find that many $\alpha_i$ are very small, and that means that the associated weight $w_i$ is 0. Consequently, we can remove it from the model training and reduce the dimensionality of the problem by 1. This is especially important if we are dealing with a kernel based method in which the dimensionality is actually the number of data points, which can be very big. – kastellane Jan 23 at 12:22