0
$\begingroup$

According to Burda et al (2015) number of active units is computed as:
$ Cov_x(E_{z \sim q_\phi(z|x)}) > \delta $

for some particular delta. In the paper it is set to 0.02 empirically. But this only measures units in vector z that are mostly constant. Another way of prior collapse for VAE is random units in z, i.e. - units that are independent from input x and those that decoder part of VAE learns to ignore. Is there a way to measure it?

One way I can think of is to set units in z to random value and measure its effect on decoded input. But this is rather slow process for high-dimensional z. Is there a better way?

$\endgroup$

1 Answer 1

1
$\begingroup$

A latent unit $z_i$ having low covariance with $x$ doesn't mean that $z_i$ is constant, it means it varies independently of $x$, i.e. $p(x,z_i) = p(x)p(z_i)\ ^{**}$. Simple proof:

$$\text{Covar}(x, z_i) = \int_{x,z} \!\!\!p(x,z) (x-\bar{x})(z - \bar{z}) \ \overset{**}{=} \int_{x,z} \!\!\!p(x)p(z) (x-\bar{x})(z - \bar{z}) \\\qquad\quad\ \ = \int_{x} p(x)(x-\bar{x}) \int_zp(z) (z - \bar{z}) = (\bar{x}-\bar{x}) (\bar{z} - \bar{z}) = 0$$

So low covariance (not variance) identifies $z$ components that are independent of $x$, i.e. carry no information, or are "dead" from a useful representation perspective.

The same measure is used when considering posterior collapse (e.g. in Variational Autoencoders (VAEs)), which refers to when all dimensions of $z$ are independent of $x$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.