3
$\begingroup$

Apologise for the clickbaity title, but it is difficult to frame this question in a single sentence. Also, the practicality of variational inference is very clear: intractable posteriors; intractable marginals get the chance to be approximated quickly using this technique.

The distinctions between Bayesian and non-Bayesian techniques I believe have been already discussed here in extensive detail. Still, I believe to some extent that variational inference begs the question - is it really a Bayesian technique?

How I see, a Bayesian technique has three characteristics:

  • it has a prior quantity,
  • it has a posterior quantity,
  • a Bayesian technique maximises posterior probabilities.

From the first two perspective I believe that, variational inference is a Bayesian technique. The third condition is not met in my opinion.

Again, while it is practical, it still feels odd that VI is maximising the lower bound on the evidence. The evidence function in most techniques are simply not utilised, and when it is, then it is used solely for model selection.

Does that make variational inference a new "evidence-based" school, distinct from the "likelihood-based" and "posterior-based" schools?

$\endgroup$
2
$\begingroup$

VI is an approximate Bayesian technique, I think just because it has the word “inference” in its name you shouldn’t confuse it as a whole new school of thought.

I would first say that I disagree with you saying that a “Bayesian technique maximises posterior probability” as really all Bayesian techniques are doing is evaluating the posterior distribution over parameters given the data. Once you have a posterior distribution you can maximise what you like, for example my choosing a MAP estimate of the parameter, but there’s plenty of other things you can do that people might consider more principled like intergrating out the parameter which has nothing to do with maximisation.

Secondly if you aren’t satisfied by “maximising the lower bound of the evidence” you might be better off thinking about it as equivalently minimising the KL divergence between the variational posterior and the true one. This way you can see that VI is explicitly Bayesian, as you are trying to make an approximation that is as close to the true posterior distribution as possible.

$\endgroup$
  • $\begingroup$ Yes, this seems logical to a certain extent. The only objection I have is when we meet methods like Automatic Relevance Determination which "learns the prior" and is optimised using variational inference, does not feel Bayesian from your perspective, but people often mention that it is not really Bayesian, though. $\endgroup$ – boomkin Aug 6 at 8:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.