I'm aware of the topic of variational inference (VI) however I'm not really sure what Black box VI is?

In particular I am watching a video by David Blei titled Black box variational inference and on this slide it mentions "black box criteria".

I would really appreciate an example as well.


So he’s referring the technique introduced in this paper: https://arxiv.org/abs/1401.0118

The idea behind black box VI is that normally in VI it takes a significant amount of work to decide on a variational posterior and derive the ELBO and it’s gradients. As such there was room for a more general algorithm that can be easily implemented and doesn’t require the practitioner to derive these forms every time. The “black box” part of the name is just from the fact it’s a general algorithm that works and you don’t need to think about what’s going on inside.

Essentially black box VI is a method that yields an estimator for the gradient of the ELBO with respect to the variational parameters with very little constraint on the form of the posterior or the variational distribution. These constraints (the black box criteria you mention) are only that you can evaluate the first derivatives of the log of the variational distribution with respect to its parameters (which you should be able to as you would normally pick a relatively simple distribution for the variational) and that you can evaluate the log of the joint of the data and latent variables (again very standard in the setting of probabilistic modelling).

  • $\begingroup$ This is a good start to answer. But it's missing the most crucial constraint: that sampling from the variational distribution is possible (assuming that by VI we mean "VI using the reverse KL cost functional"). Sampling is required in order to form an estimator of the KL divergence (or at least, it is for the most common Monte Carlo estimator). This is a considerably more strict constraint than assuming that derivatives of the variational distribution's log density are available. $\endgroup$ 13 hours ago

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.