2
$\begingroup$

Is it possible to do ELBO maximization using stochastic gradient estimates (i.e. iteratively apply variational updates using (3) in http://proceedings.mlr.press/v33/ranganath14.pdf), when it's cheap to sample from the variational posterior $ q(z|\mathbf{\lambda})$, however $ q(z|\mathbf{\lambda})$ and the score function $ \nabla_{\mathbf{\lambda}}\log q(z|\mathbf{\lambda}) $ are analytically intractable?

I know that the motivation for applying VI is exactly relying on $q$s with easy to compute statistics, however let's say we are interested in an application where our variational approximation should live in a constrained family of distributions.

$\endgroup$
5
  • $\begingroup$ Can you expand on what you mean by "where our variational approximation should live in a constrained family of distributions"? Since this is pretty much always the case. $\endgroup$
    – XanderJC
    Commented Sep 26, 2019 at 15:46
  • $\begingroup$ I mean that we cannot analytically compute $ q(z|\lambda)$ $\endgroup$
    – Dion
    Commented Sep 26, 2019 at 16:21
  • $\begingroup$ Okay, can you give me an example of a parameterised family of distributions that are analytically intractable. yet we can still sample from? $\endgroup$
    – XanderJC
    Commented Sep 26, 2019 at 21:42
  • $\begingroup$ MCMC is an asymptotically exact method, so I suppose using that and the marginal distribution of the Markov chain as the variational distribution would constitute perfect ELBO maximisation in the limit. I also don't think it's very fair to say MCMC constitutes an efficient sampling method, its computational cost is one of the main reasons for using VI. $\endgroup$
    – XanderJC
    Commented Sep 26, 2019 at 21:59
  • $\begingroup$ Let us continue this discussion in chat. $\endgroup$
    – Dion
    Commented Sep 26, 2019 at 22:11

1 Answer 1

2
$\begingroup$

The only reason Variational Inference (VI) requires knowing the density $q(z|\lambda)$ is if you seek to maximize the ELBO w.r.t. $\lambda$, which is equivalent to minimizing $\text{KL}(q(z|\lambda) \mid\mid p(z|\mathcal{D}, \theta))$ – a KL divergence to the true posterior.

If instead, you're interested in optimizing the ELBO w.r.t. model parameters $\theta$ (as in VAEs), then you only need samples. So let's focus on the posterior approximation part.

In general, the approximation's objective can be equivalently rewritten as a difference of a cross-entropy and entropy of $q$: $$ \text{KL}(q(z|\lambda) \mid\mid p(z|\mathcal{D})) = \text{CE}[q(z|\lambda), p(z|\mathcal{D})] -\text{H}[q(z|\lambda)] $$ The density $q(z|\lambda)$ is needed for the latter. In fact, it's known that estimating the entropy from the samples only is very hard, things are much easier when the density is known.

However, you might know something about the density $q(z|\lambda)$. For example, if $q(z|\lambda)$ is a marginal distribution of some hierarchical process $q(z|\lambda) = \int q(z, \varepsilon |\lambda) d\varepsilon$, then an efficient variational bound can be given on the entropy.

A black-box alternative is to learn the density of $q(z|\lambda)$. In practice, you won't capture all the quirks of $q(z|\lambda)$ exactly, so this method lacks theoretical guarantees.


But is KL-divergence the only way to measure discrepancy between the true posterior and its approximation? Certainly, no, and other options might be more convenient. For example, Ruiz and Titsias have leveraged the fact that MCMC is guranteed to improve upon proposal to cancel out chain's marginal distribution.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.