In the Automatic Differentiation Variational Inference (ADVI) paper, the authors claim to solve the VI problem in a transformed parameter space, which is over $\mathbb{R}$, in order to simplify the complexities of optimising over a constrained set of values.

How would one transform a parameter vector of arbitrary uniform distributions to the real number line, in order for ADVI to proceed? How would the Jacobian look for this transformation?

Paper for reference: https://arxiv.org/abs/1603.00788


In STAN, the main implementation of ADVI that I am aware of, constraints are removed when the model is parametrized, rather than when it is optimized. Chapter 35 of the STAN manual has all the details. I recount it below.

If $X$ is a variable with a lower bound constraint $a$ so that $X>a$

The model is reparametrized into $$ Y = f(X) = \log(X-a), $$ thus $X = f^{-1}(Y) = \exp(Y) + a$ and the change of variables gives $$ \begin{align*} p_y(y) & = p_x(f^{-1}(Y)) \lvert \frac{d}{dy} (f^{-1}(y)) \rvert \\ & = p_x(\exp(y) + a) \exp(y) \end{align*} $$

For an upper bound constrain, negate the argument to the logarithm in the above.

If $X$ has a box constraint on (a, b)

Then $Y = f(X) = \text{logit} \frac{X-a}{b-a}$ where $\text{logit}(u) = \log u/(1-u)$. Thus $$X = f^{-1}(Y) = a + (b-a) \text{logit}^{-1}(Y),$$ where $\text{logit}^{-1}(v) = (1+ \exp(-v))^{-1}.$ Thus $$ \begin{align*} p_y(y) = p_x\left(a + (b-a) \text{logit}^{-1}(y) \right) (b-a) \text{logit}^{-1}(y) \left(1 - \text{logit}^{-1}(y) \right). \end{align*} $$

  • $\begingroup$ beautiful. Thanks! $\endgroup$ – pche8701 Aug 7 at 13:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.