5
$\begingroup$

In cases such as Gaussian mixture models, there's is no closed-term solution for the original likelihood maximization. Maximizing the ELBO, however, does have analytical update formulas (i.e. formulas for the E and M steps). I understand why in this case maximizing the ELBO is a useful approximation.

However, in more complex models, such as VAE, the E & M steps themselves don't have a closed solution, and ELBO maximization is done with SGD. In this scenario, what's the advantage of optimizing the ELBO with SGD over maximizing the original likelihood with SGD?

Thanks

Improve this question
$\endgroup$

1 Answer 1

Reset to default
6
$\begingroup$

I think you confuse the purpose of the two methods.

Maximizing the ELBO leads to a parameterized class of densities that approximates closely the true distribution, in terms of Kullback-Leibler divergence. If you instead just do SGD on the target, what you will achieve is just a (local) maximum of parameters, but no approximate probability distribution.

In other words, working with variational inference allows full approximate posterior inference (calculation of probabilities, intervals, expectations etc.), whereas SGD on the target just allows for point estimates of parameters, but no uncertainty quantification of these.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Yet, there are some results showing that using SGD-based methods may have it's merits arxiv.org/abs/2002.02405 $\endgroup$
    – Tim
    Commented Feb 12, 2020 at 13:54
  • $\begingroup$ Of course 'pure' SGD methods have merit (its just an optimization algorithm for big data sets!), posterior inference is useful only so far as the integral estimates can be used to either improve understanding or, in machine learning, quantify uncertainty of predictions. This is, as the paper discusses, much less clear for the Bayesian neural net class of models. I'm just pointing out the difference between two inference goals and associated algorithms; I can't exactly see the relevance with respect to the article which addresses a much more narrow, specific research question. $\endgroup$
    – Forgottenscience
    Commented Feb 12, 2020 at 13:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.