Derivation of ELBO in ADVI Paper, Jacobian of Elliptical Transformation

Question

I've been following the ELBO derivations in the paper Automatic Differentiation Variational Inference and have a few questions. With the model $p(x,\theta)$, they first transform $\theta$ so that it lies on the real coordinate plane. Let $\zeta = T(\theta)$ be the transformed variable. Once they do that transformation, they have that $p(x,\zeta) = p\big(x,T^{-1}(\zeta)\big)|\det J_{T^{-1}}(\zeta)|$. Next, they apply a second transformation to standardize $\zeta$. Let $\eta = S_\phi(\zeta) = L^{-1}\zeta-\mu$. Here, $L$ and $\mu$ come from the variational distribution $q(\zeta;\mu,L) = N(\zeta;\mu,L)$. $L$ is the upper triangular matrix from the Cholesky Decomposition of the covariance matrix.

This is where my confusion arises. In the the paper, the authors state "The Jacobian of elliptical standardization evaluates to one, because the Gaussian distribution is a member of the location-scale family: standardizing a Gaussian gives another Gaussian distribution."
This means that $p(x,\eta) = p\bigg(x,T^{-1}(S^{-1}(\eta))\bigg)|\det J_{T^{-1}}(S^{-1}(\eta))|$ and not
$p\bigg(x,T^{-1}(S^{-1}(\eta))\bigg)|\det J_{T^{-1}}(S^{-1}(\eta))||\det J_{S^{-1}}(\eta)|$

Why does the Jacobian evaluate to one? I'm not sure what "standardizing the gaussian gives another gaussian" has to do with the Jacobian being one. Doesn't $J_{S^{-1}}(\eta) = L$, which means $|\det J_{S^{-1}}(\eta)| = |\det(L)|$?

They say this at the top of page ten by the way.

Answer 1

The transformation of the formula from (5) in the paper to the formula at the bottom of page 9 is a consequence of the Law of the Unconscious Statistician (LOTUS) -see (28) in Monte Carlo Gradient Estimation in Machine Learning for this form - and is a special case of the reparameterization trick described there applied to standardizing a Gaussian distribution.

The proof of the LOTUS effectively involves showing that the Jacobian determinant ($|\det J_{S^{-1}}|$) arising in the modified density for $\eta$ as a result of the change of variables cancels out with the Jacobian determinant arising in the integral corresponding to the expectation as a result of the same change of variables (i.e. we need to replace $d\zeta$ by $|\det J_S| d\eta$). This doesn't happen in integrals in general, but happens in expectations because of the co-occurence of the density and the differential.

So I agree that the Jacobian is not one; however, the result specified is correct.

Answer 2

I don't think this has anything to do with LOTUS, instead it's just a quality of the Normal distribution:

Suppose $x\sim N(\mu, \Sigma)$. And suppose $g(x)$ is some function of $x$ we want to take the expectation of. Then:

$$\mathbb E_x[g(x)]=\int (2\pi)^{-d/2}\det(\Sigma)^{-0.5} \exp[-\frac{1}{2}(x-\mu)^T\Sigma ^{-1}(x-\mu)]\cdot g(x) dx = *$$

Transforming from $x$ to $z=\Sigma^{-0.5}(x-\mu)$ has the inverse transform $x=\Sigma^{0.5}z+\mu$ the Jacobian equal to $\Sigma^{0.5}$ and the det(Jacobian) equal to $\det(\Sigma^{0.5})=\det(\Sigma)^{0.5}$. So overall we would get:

$$\require{cancel} *= \int (2\pi)^{-d/2}\cancel{\det(\Sigma)^{-0.5}} \exp[-\frac{1}{2}z^Tz]\cdot g(\Sigma^{0.5}z+\mu) \cancel{\det(\Sigma)^{0.5}} dz $$ $$=\int (2\pi)^{-d/2} \exp[-\frac{1}{2}z^Tz]\cdot g(\Sigma^{0.5}z+\mu) dz=\mathbb E_z[g(\Sigma^{0.5}z+\mu)]$$

That is, the determinant Jacobian was canceled with part of the density, it was "absorbed". So, we can use the standard (multivariate) normal distribution in the expectation, and only change the function variable, without the Jacobian.