3
$\begingroup$

In Bayesian regression, we have $y_i=x_i^{T}w+\epsilon_i$ where $w \sim \mathcal{N}(0,\alpha)$ and $\epsilon_i \sim \mathcal{N}(0,\frac{1}{\beta})$. Inference of $\alpha$ and $\beta$ is done by maximizing the likelihood(or marginal likelihood) given the data. This paper(Appendix A.1) explains that maximization can be done by expectation-maximization(EM).

My questions is that why do we need EM to do the maximization. I understand that EM can be used by treating the $w$ as the hidden variable. However, $w$ can be integrated out and we can apply gradient descent(GD). Why don't we use GD?

Improve this question
$\endgroup$
5
  • $\begingroup$ Maybe i didnt understand well the paper you mention. But it seems like the author show the update when you do gradient descent and EM update. And he explains that EM update gives faster convergence. (Again i read it very fast, so i can be wrong). I focus only on Appendix A to say that :D $\endgroup$ – PauZen Jan 20 at 9:14
  • $\begingroup$ He doesn't compare gradient descent with EM. He compares EM with another update rule that is similar to EM, but not the same. The other update rule converges faster than EM but its convergence to a local minimum is not guaranteed contrary to EM. $\endgroup$ – arke Jan 21 at 14:17
  • $\begingroup$ why would you do inference on a hyper-parameter such as $\alpha$? $\endgroup$ – Carlos Llosa Jan 22 at 0:16
  • $\begingroup$ @CarlosLlosa I don't know much about how to determine the hyper-parameters in Bayesian methods. That is how they are determined in every paper I read. Note that I also know that the same type of inference is done for gaussian process regression too. $\endgroup$ – arke Jan 22 at 16:53
  • $\begingroup$ @CarlosLlosa Inference in $\alpha$ is very useful when it is actually a vector of $\alpha_i$. In that way, we may find that many $\alpha_i$ are very small, and that means that the associated weight $w_i$ is 0. Consequently, we can remove it from the model training and reduce the dimensionality of the problem by 1. This is especially important if we are dealing with a kernel based method in which the dimensionality is actually the number of data points, which can be very big. $\endgroup$ – kastellane Jan 23 at 12:22
0
$\begingroup$

You ask "why do we need EM to do the maximization", but the author says "Another strategy to maximize (36) is to exploit an EM formulation". Clearly, we don't need EM. However, it is very interesting content for an appendix, since under some circumstances EM might come in handy. For example, gradient descent is not guaranteed to increase the likelihood in each iteration, while EM is. Also, although not in this problem, sometimes you need to estimate parameters that must fulfill a constraint (e.g. the sum of the weights must add to one) and EM can include such constraints in a straightforward manner while gradient descent can step into regions where the constraint is not fulfilled. Moreover, EM algorithms tend to be numerically more stable than gradient based ones. In my opinion, EM can be a superior alternative to gradient-based methods, depending on the specific details of your problem.

UPDATE In this book, section 9.3.4 the same problem is solved using both methods. However, the author does not address the superiority of any of them over the other which suggests that, at least in this case, there's no obvious superiority of any. Even more, he claims that "these re-estimation equations are formally equivalent to those obtained by direct maximization."

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks for your answer. Regarding your update, I don't see any mention of GD. The other method(the one that is not EM) is not EM but it similar to EM in the way the parameters updated in the iterations. $\endgroup$ – arke Jan 22 at 21:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.