Does the use of "variational" always refer to optimization via variational inference?


  • "Variational auto-encoder"
  • "Variational Bayesian methods"
  • "Variational renormalization group"
  • $\begingroup$ For understanding VAEs, you might also refer to the nice youtube videos here and here. They answered all the questions I had to the topic. $\endgroup$ – André Sep 16 '19 at 10:58

It means using variational inference (at least for the first two).

In short, it's an method to approximate maximum likelihood when the probability density is complicated (and thus MLE is hard).

It uses Evidence Lower Bound (ELBO) as a proxy to ML:

$log(p(x)) \geq \mathbb{E}_q[log(p, Z)] - \mathbb{E}_q[log(q(Z))]$

Where $q$ is simpler distribution on hidden variables (denoted by $Z$) - for example variational autoencoders use normal distribution on encoder's output.

The name 'variational' comes most likely from the fact that it searches for distribution $q$ that optimizes ELBO, and this setup is kind of like in calculus of variations, a field that studies optimization over functions (for example, problems like: given a family of curves in 2D between two points, find one with smallest length).

There's a nice tutorial on variational inference by David Blei that you can check out if you want more concrete description.


Actually what I described is one type of VI: in general you could use different divergence (the one I described corresponds to using KL divergence $KL(q, p)$). For details see this paper, section 5.2 (VI with alternative divergences).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.