How does variational inference fit in the big picture of inference?

Question

Apologise for the clickbaity title, but it is difficult to frame this question in a single sentence. Also, the practicality of variational inference is very clear: intractable posteriors; intractable marginals get the chance to be approximated quickly using this technique.

The distinctions between Bayesian and non-Bayesian techniques I believe have been already discussed here in extensive detail. Still, I believe to some extent that variational inference begs the question - is it really a Bayesian technique?

How I see, a Bayesian technique has three characteristics:

• it has a prior quantity,
• it has a posterior quantity,
• a Bayesian technique maximises posterior probabilities.

From the first two perspective I believe that, variational inference is a Bayesian technique. The third condition is not met in my opinion.

Again, while it is practical, it still feels odd that VI is maximising the lower bound on the evidence. The evidence function in most techniques are simply not utilised, and when it is, then it is used solely for model selection.

Does that make variational inference a new "evidence-based" school, distinct from the "likelihood-based" and "posterior-based" schools?

Answer 1

VI is an approximate Bayesian technique, I think just because it has the word “inference” in its name you shouldn’t confuse it as a whole new school of thought.

I would first say that I disagree with you saying that a “Bayesian technique maximises posterior probability” as really all Bayesian techniques are doing is evaluating the posterior distribution over parameters given the data. Once you have a posterior distribution you can maximise what you like, for example my choosing a MAP estimate of the parameter, but there’s plenty of other things you can do that people might consider more principled like intergrating out the parameter which has nothing to do with maximisation.

Secondly if you aren’t satisfied by “maximising the lower bound of the evidence” you might be better off thinking about it as equivalently minimising the KL divergence between the variational posterior and the true one. This way you can see that VI is explicitly Bayesian, as you are trying to make an approximation that is as close to the true posterior distribution as possible.

Answer 2

The central equation of inference defines the integral for computing the expected (average) of a function $F(z)$ with a probability density $\Pi(z)$ as

$E_{\: \Pi(z)} \: \big\lbrack F(z) \big\rbrack \: = \: \int F(z) \cdot \Pi(z)$

Variational inference considers the case when the probability density $\Pi(z)$ is intractable, and instead utilizes approximate inference for approximation of the distribution. There are multiple approximate inference algorithms to choose from (Junction tree, MCMC, VI), however, it is optimization-based approaches for estimation of the target density that are commonly used in the machine learning literature.

The general framework is the following: Variational inference fits a surrogate and tractable distribution $q_{\phi}(z)$ to the true posterior $\Pi(z)$ distribution. In the foregoing notation, we used $\phi$ to denote a common family of probability distributions from which the surrogate distribution is chosen. The variational idea is detailed as

$E_{\Pi(z)} \: \big\lbrack \: F(z) \: \big\rbrack \: \approx \: E_{q_{\phi}(z)} \: \big\lbrack \: F(z) \: \big\rbrack$

We can think of the method as restricting the set of functions over which the optimization is performed. The minimization of the KL-divergence or synonymous, the maximization of the Evidence of Lower Bound (ELBO) is a common objective for performing optimization.

In summary, the technique enables Bayesian inference to scale to large datasets and complex models with intricate posterior geometries. While variational inference does not provide guarantees on the bounds, the stochastic optimization and distributed optimization speed up inference and enables the exploration and comparison of many different models.

The following excerpts are taken from my book on Variational inference. Learn more by visiting https://www.thevariationalbook.com/