disadvantages variational inference

Question

A lot of methods utilize variational inference for hyperparameter calculation.

What are the advantages and disadvantages of variational inference ?

(ex: does it guarantee a global optimal ? )

Answer 1

Further disadvantages:

1. The outcome tends to depend heavily on the starting point for the optimization. Example: this paper which is heavily cited but known to have severe problems (software packages based on it were later withdrawn, etc.)
2. The calculations required to figure out what you are optimizing are often very complicated. (See any paper on variational inference.)

On the plus side, there is an excellent introduction to the subject in Mackay's textbook Information Theory, Inference and Learning Algorithms.

Answer 2

Briefly:

• Disadvantages: approximate, very little theory around it
• Advantages: speed, scalability, novelty

There isn't much theory around variational inference. However you define "optimal" (qv below), you probably can't expect to obtain it.

VI is a method for approximating a difficult-to-compute probability density, $p$, by optimization. This is done by suggesting a family of distributions $\mathcal Q$ and finding the member $q \in \mathcal Q$ that has the lowest Kullback–Leibler divergence $KL(q \|p)$. How well you can approximate $p$ naturally depends on your choice of $\mathcal Q$, but you can assume that some aspect of $p$ is lost when substituting it by $q$.

VI doesn't guarantee you find the globally optimal member $q \in \mathcal Q$ either. A common choice is to use what's called the mean-field variational family and find $q$ by coordinate ascent. You can find a local optimum.

A big advantage is that VI is very fast and scales well to large datasets. It is natural to compare with MCMC methods as these solve the same problem, see the answer to this related question, which compares the two.

---

Reading:

David M. Blei, Alp Kucukelbir, Jon D. McAuliffe Variational Inference: A Review for Statisticians