What is meant by 'Black box variational inference'? I'm aware of the topic of variational inference (VI) however I'm not really sure what Black box VI is?
In particular I am watching a video by David Blei titled Black box variational inference and on this slide it mentions "black box criteria".
I would really appreciate an example as well.
 A: So he’s referring the technique introduced in this paper: https://arxiv.org/abs/1401.0118
The idea behind black box VI is that normally in VI it takes a significant amount of work to decide on a variational posterior and derive the ELBO and it’s gradients. As such there was room for a more general algorithm that can be easily implemented and doesn’t require the practitioner to derive these forms every time. The “black box” part of the name is just from the fact it’s a general algorithm that works and you don’t need to think about what’s going on inside.
Essentially black box VI is a method that yields an estimator for the gradient of the ELBO with respect to the variational parameters with very little constraint on the form of the posterior or the variational distribution. These constraints (the black box criteria you mention) are only that you can evaluate the first derivatives of the log of the variational distribution with respect to its parameters (which you should be able to as you would normally pick a relatively simple distribution for the variational) and that you can evaluate the log of the joint of the data and latent variables (again very standard in the setting of probabilistic modelling).
A: The main algorithm used in VI before was the Coordinate Ascent VI (CAVI), which does a "variational" coordinate ascent on the ELBO. An alternative to doing Coordinate Ascent is doing Gradient Ascent.
Why should we do that?

*

*Gradient Ascent / Descent can also be done stochastically (SGA / SGD), by sampling only part of the data. As the data size becomes larger each iteration of CAVI becomes more expensive.

*The CAVI update rule might be too complicated for some problems, while the gradient may be simple. For certain problems where we can assume the distributions are from the Exponential Family, the gradient is actually easy to compute (even more so if we use the “natural” gradient instead of the regular gradient).

*For other problems, we can still use Monte-Carlo (MC) integration and calculate the gradients. Two known algorithms that do so are Automatic-Differentiation VI (ADVI) and Black-Box VI (BBVI).

BBVI computes an MC estimator of the gradient, using what is known as the "log-derivative" or the "REINFORCE" trick. The "vanilla" version is quite simple, with just a little bit of derivation you get to:
$$ \hat {\nabla_{\phi}\text{ELBO}} ^{(t+1)} = \frac{1}{m}\sum_{i=1}^m \nabla_{\phi} \log q(\theta_{i}) \cdot (\log p(\theta_{i},x)-\log q(\theta_{i}))) \\
\phi ^{(t+1)} = \phi^{(t)}+\alpha \cdot \hat {\nabla_{\phi}\text{ELBO}} ^{(t+1)}$$
Where $q$ is the variational family parameterized by $\phi$, and $p(\theta,x)$ is the joint.
Unfortunately, this suffers from high variance (the ELBO-derivative can give extremely different results depending on the sample of θ’s at each iteration), and so some variance reduction techniques are used, which complicate things a bit.
You can check my medium article on the topic, or my YouTube video, for more information.
