What does "variational" mean?

Question

Does the use of "variational" always refer to optimization via variational inference?

Examples:

• "Variational auto-encoder"
• "Variational Bayesian methods"
• "Variational renormalization group"

Answer 1

It means using variational inference (at least for the first two).

In short, it's an method to approximate maximum likelihood when the probability density is complicated (and thus MLE is hard).

It uses Evidence Lower Bound (ELBO) as a proxy to ML:

$log(p(x)) \geq \mathbb{E}_q[log(p, Z)] - \mathbb{E}_q[log(q(Z))]$

Where $q$ is simpler distribution on hidden variables (denoted by $Z$) - for example variational autoencoders use normal distribution on encoder's output.

The name 'variational' comes most likely from the fact that it searches for distribution $q$ that optimizes ELBO, and this setup is kind of like in calculus of variations, a field that studies optimization over functions (for example, problems like: given a family of curves in 2D between two points, find one with smallest length).

There's a nice tutorial on variational inference by David Blei that you can check out if you want more concrete description.

EDIT:

Actually what I described is one type of VI: in general you could use different divergence (the one I described corresponds to using KL divergence $KL(q, p)$). For details see this paper, section 5.2 (VI with alternative divergences).

Answer 2

You can find a good explanation in this source by Jason Eisner, where he cites:

The term variational is used because you pick the best q in Q -- the term derives from the "calculus of variations", which deals with optimization problems that pick the best function (in this case, a distribution q).

One way it occurs is when you try to optimize a Functional (a function $F$ that receives a function $q$ and returns a value, e.g. Entropy), so you try to find the best $q$ in a set of functions $Q$ that optimizes $F$.