1
$\begingroup$

we posit a family of approximate densities Q. This is a set of densities over the latent variables. Then, we try to find the member of that family that minimizes the Kullback-Leibler(KL) divergence to the exact posterior,

Blei et al.

Why is it called "family"? Why don't we call it "assumptions" on the joint distribution of latent variables.
For example, Why don't we say:
"In the mean field approximation we assume that the latent variables are independent and we try to find the best approximation of the joint distribution of latent variables to minimizes the $KL$ divergence"

$\endgroup$
1
$\begingroup$

There's no special technical meaning (as far as I know) to the word "family" here. It just means a set of probability distributions that are all intuitively related in some way. For example, all the 1d normal distributions with zero mean and any positive variance might be collectively termed a family. This set of distributions is an exponential family, the combination of which has a particular technical meaning, but "family" alone doesn't.

The Blei et al. quote is basically saying:

  1. We pick a bunch of potential distributions $Q$, for example the set of joint normal distributions in which the latent parameters are all independent.
  2. We try to find the particular distribution in $Q$ which is the best approximation to the distribution we actually care about, the true posterior.

This is essentially the same as your proposed quote, except that yours is a little fuzzier about what your assumption actually is: everything is independent, but what are the set of allowable marginal distributions on each latent? Can we have one be Cauchy and another Poisson?

$\endgroup$
2
  • $\begingroup$ But How can we have a choice over the distributions of the variables? for example: in the latent dirichlet allocation(LDA), the variables have dirichlet and multinomial distributions and we didn't pick anything other than the independence assumption. $\endgroup$
    – floyd
    Sep 6 '19 at 19:10
  • 1
    $\begingroup$ That's the usual thing you do: take a family the posterior lives inside and choose only some easy-to-work-with elements of it. But you don't have to: you could definitely use a variational approximation of a Laplace distribution inside a normal family, or whatever. $\endgroup$
    – Danica
    Sep 6 '19 at 19:22
0
$\begingroup$

Please read my recent paper titled "Gibbs sampler and coordinate ascent variational inference: A set-theoretical review". Visit link https://www.tandfonline.com/doi/full/10.1080/03610926.2021.1921214

Mean field assumption is NOT a modeling assumption. Rather, it is some imposition on top to a Bayesian model to implement coordinate ascent variational inference algorithm.

Here, what I am saying that it is NOT a modeling assumption is that, essentially, even without mean field assumptions, we can very nicely approximate our target density, which is normally, posterior density.

That being said, mean field assumption would be better to be commented or discussed from an implementational point of view.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.