# Choice of approximate posterior in variational inference with positive support

I have a simple probabilistic graphical model: $$z \longrightarrow x$$ where $$z_i \sim Exp\left(\lambda_i\right)$$ where subscript $$i$$ denotes the $$i$$th dimension and $$x|z \sim \mathcal{N}\left(f\left(z\right), \sigma^2\right)$$ where $$f$$ is some real valued function (usually non-linear). I'm using variational inference to approximate the posterior $$p\left(z|x\right)$$. Let's say $$q_{\phi}\left(z|x\right)$$ is the approximate posterior with $$\theta$$ being the parameters.

What are some well-informed choices for $$q$$?

Since $$z$$ is a positive random variable, I cannot use the prototypical normal distribution assumption for $$q$$. Furthermore I also want the posterior to have some correlation structure like in a multivariate normal distribution.

My idea is to indeed use a multivariate normal distribution but only as the base distribution and add a positive, bijective transformation, $$T$$ on top such that my target distribution has positive support, retains a correlation structure and the bijective transformation renders a full likelihood computation. Basically: $$q_{\phi}\left(z|x\right) \equiv \mathcal{N}\left(z|x\right)\left|\operatorname{det} \frac{\partial T}{\partial \mathbf{z}}\right|^{-1}$$.

What are some good choices for $$T$$? And what are some other distributions I can use in general in that allow dependencies between dimensions, and have positive support? I believe I can also just choose univariate distributions as long as I also use copulas but I lack the knowledge on that and some advice will be much appreciated.

So far I have tried the exponential function as a transformation on top of multivariate normal, which is essentially a multivariate lognormal.

Finally, disclaimer: sorry that I'm cross posting this from mathoverflow where I posted the same question (https://mathoverflow.net/questions/431744/choice-of-approximate-posterior-in-variational-inference-with-positive-support), but I believe Cross Validated is a better forum for discussion for this topic. Please feel free to let me know otherwise if you strongly think so.

• I have had difficulty finding practical advice for how to choose variational distributions, so I look forward to answers to this question. I'll just note real quickly that the "Gaussian-bijection" approach is quite popular, and is the approach implemented by Stan. One benefit of the lognormal is that it's very easy to reparameterize for autodiff; I've had issues using, for example, Gamma variational distributions. One practical piece of advice: life is easier for us if we choose a variational distribution that has a closed form KL divergence with the prior (exponential in your case). Oct 4, 2022 at 20:02
• Thanks for your comment! Cool to know I'm not completely off with the "Gaussian-bijection" idea. I am using PyTorch to perform the optimization and torch.distributions's TransformedDistribution has been handy for parameterizing as well as likelihood computation. Re KL computation, sigh yes at the moment I'm computing it numerically... Oct 4, 2022 at 20:11
• Oh I just noticed your correlated variational distribution, I'm way more used to working with the mean-field variety (but they're totally rad) :) You expect important posterior correlation between parameters? Oct 4, 2022 at 22:46
• I do expect the posterior to encode correlation among dimensions of my latent variable. So yeah, mean-field approximation would be limiting. Oct 5, 2022 at 8:15
• Word, keep in mind that just cause there's posterior correlation doesn't mean the mean-field variationals will be useless; we just expect them to be a little more certain than they should be. Oct 5, 2022 at 11:57

Going somewhat along this logic, it may make sense to use a $$\Gamma$$ distribution, a distribution which has a well known KL divergence with itself (keeping in mind that the exponential distribution can also be thought of as a gamma and using this CV post).