16
$\begingroup$

Does the use of "variational" always refer to optimization via variational inference?

Examples:

  • "Variational auto-encoder"
  • "Variational Bayesian methods"
  • "Variational renormalization group"
$\endgroup$
2
  • 2
    $\begingroup$ For understanding VAEs, you might also refer to the nice youtube videos here and here. They answered all the questions I had to the topic. $\endgroup$
    – André
    Sep 16, 2019 at 10:58
  • $\begingroup$ I think the 'variational inference' should be called 'optimization inference', since it basically uses 'optimization' to conduct 'inference'. $\endgroup$
    – xiawenwen
    Aug 12, 2021 at 13:49

2 Answers 2

13
$\begingroup$

It means using variational inference (at least for the first two).

In short, it's an method to approximate maximum likelihood when the probability density is complicated (and thus MLE is hard).

It uses Evidence Lower Bound (ELBO) as a proxy to ML:

$log(p(x)) \geq \mathbb{E}_q[log(p, Z)] - \mathbb{E}_q[log(q(Z))]$

Where $q$ is simpler distribution on hidden variables (denoted by $Z$) - for example variational autoencoders use normal distribution on encoder's output.

The name 'variational' comes most likely from the fact that it searches for distribution $q$ that optimizes ELBO, and this setup is kind of like in calculus of variations, a field that studies optimization over functions (for example, problems like: given a family of curves in 2D between two points, find one with smallest length).

There's a nice tutorial on variational inference by David Blei that you can check out if you want more concrete description.

EDIT:

Actually what I described is one type of VI: in general you could use different divergence (the one I described corresponds to using KL divergence $KL(q, p)$). For details see this paper, section 5.2 (VI with alternative divergences).

$\endgroup$
1
  • $\begingroup$ Shouldn't log only take one argument? $\endgroup$ Feb 7 at 14:26
0
$\begingroup$

You can find a good explanation in this source by Jason Eisner, where he cites:

The term variational is used because you pick the best q in Q -- the term derives from the "calculus of variations", which deals with optimization problems that pick the best function (in this case, a distribution q).

One way it occurs is when you try to optimize a Functional (a function $F$ that receives a function $q$ and returns a value, e.g. Entropy), so you try to find the best $q$ in a set of functions $Q$ that optimizes $F$.

$\endgroup$
1
  • $\begingroup$ Hi, what information does this add over the previous answer? $\endgroup$ Oct 26, 2021 at 13:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.