A lot of methods utilize variational inference for hyperparameter calculation.
What are the advantages and disadvantages of variational inference ?
(ex: does it guarantee a global optimal ? )
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this communityFurther disadvantages:
On the plus side, there is an excellent introduction to the subject in Mackay's textbook Information Theory, Inference and Learning Algorithms.
Briefly:
There isn't much theory around variational inference. However you define "optimal" (qv below), you probably can't expect to obtain it.
VI is a method for approximating a difficult-to-compute probability density, $p$, by optimization. This is done by suggesting a family of distributions $\mathcal Q$ and finding the member $q \in \mathcal Q$ that has the lowest Kullback–Leibler divergence $KL(q \|p)$. How well you can approximate $p$ naturally depends on your choice of $\mathcal Q$, but you can assume that some aspect of $p$ is lost when substituting it by $q$.
VI doesn't guarantee you find the globally optimal member $q \in \mathcal Q$ either. A common choice is to use what's called the mean-field variational family and find $q$ by coordinate ascent. You can find a local optimum.
A big advantage is that VI is very fast and scales well to large datasets. It is natural to compare with MCMC methods as these solve the same problem, see the answer to this related question, which compares the two.
Reading:
David M. Blei, Alp Kucukelbir, Jon D. McAuliffe Variational Inference: A Review for Statisticians