What does "variational" mean?

Question

Does the use of "variational" always refer to optimization via variational inference?

Examples:

• "Variational auto-encoder"
• "Variational Bayesian methods"
• "Variational renormalization group"

Answer 1

It means using variational inference (at least for the first two).

In short, it's an method to approximate maximum likelihood when the probability density is complicated (and thus MLE is hard).

It uses Evidence Lower Bound (ELBO) as a proxy to ML:

$log(p(x)) \geq \mathbb{E}_q[log(p, Z)] - \mathbb{E}_q[log(q(Z))]$

Where $q$ is simpler distribution on hidden variables (denoted by $Z$) - for example variational autoencoders use normal distribution on encoder's output.

The name 'variational' comes most likely from the fact that it searches for distribution $q$ that optimizes ELBO, and this setup is kind of like in calculus of variations, a field that studies optimization over functions (for example, problems like: given a family of curves in 2D between two points, find one with smallest length).

There's a nice tutorial on variational inference by David Blei that you can check out if you want more concrete description.

EDIT:

Actually what I described is one type of VI: in general you could use different divergence (the one I described corresponds to using KL divergence $KL(q, p)$). For details see this paper, section 5.2 (VI with alternative divergences).

Answer 2

To precisely answer the question what does "variational" mean, we first review the origins of variational inference. By this approach, we gain a broader understanding of the term's meaning.

Variational inference originated in the 18th century with the work of Euler, Lagrange and others studying the field of calculus. In calculus of variations, a function maps an input value to an output value, and the function’s derivative describes how the output changes with respect to the input fed into the same function. In the same spirit, imagine defining a functional as a mapping from a function to a value of the functional. The definition of entropy $\mathcal{H}_{P}$ can be used as such an example. The input is a probability distribution $p_{data}(x)$ and the output is the value of the entropy.

$\mathcal{H}_{P} \: = \: - \: \int \: p(x) \cdot \log p(x) \: dx$

The derivative, usually referred to as the $\it functional \: derivative$, is the idea that the value of the $functional$ updates in response to infinitesimal changes to the input function. Logical thinking suggests that if we were to search the input space for all functions, might we discover a set of functions that maximize (or minimize) the functional. By adopting this viewpoint, we've modified and re-framed this idea as an optimization problem, whereby the quantity being optimized is the functional.

Then when thinking about the meaning of "variational", it can be regarded to be the refinement of something. And relating back to the topic of variational inference once more, we can think of the method as restricting the set of functions over which the optimization is performed.

To the aforementioned examples, a variational autoencoder (VAE) optimizes for a set of functions in the latent space. Variational bayesian methods optimizes the values of distributional parameters and hyperparameters of a bayesian model. In both cases, this is traditionally performed using the Evidence of Lower Bound (ELBO).

The following excerpts are taken from my book on variational inference. Learn more on the topic by visiting https://www.thevariationalbook.com/

Answer 3

You can find a good explanation in this source by Jason Eisner, where he cites:

The term variational is used because you pick the best q in Q -- the term derives from the "calculus of variations", which deals with optimization problems that pick the best function (in this case, a distribution q).

One way it occurs is when you try to optimize a Functional (a function $F$ that receives a function $q$ and returns a value, e.g. Entropy), so you try to find the best $q$ in a set of functions $Q$ that optimizes $F$.