# VAE active units

According to Burda et al (2015) number of active units is computed as:
$$Cov_x(E_{z \sim q_\phi(z|x)}) > \delta$$

for some particular delta. In the paper it is set to 0.02 empirically. But this only measures units in vector z that are mostly constant. Another way of prior collapse for VAE is random units in z, i.e. - units that are independent from input x and those that decoder part of VAE learns to ignore. Is there a way to measure it?

One way I can think of is to set units in z to random value and measure its effect on decoded input. But this is rather slow process for high-dimensional z. Is there a better way?

A latent unit $$z_i$$ having low covariance with $$x$$ doesn't mean that $$z_i$$ is constant, it means it varies independently of $$x$$, i.e. $$p(x,z_i) = p(x)p(z_i)\ ^{**}$$. Simple proof:
$$\text{Covar}(x, z_i) = \int_{x,z} \!\!\!p(x,z) (x-\bar{x})(z - \bar{z}) \ \overset{**}{=} \int_{x,z} \!\!\!p(x)p(z) (x-\bar{x})(z - \bar{z}) \\\qquad\quad\ \ = \int_{x} p(x)(x-\bar{x}) \int_zp(z) (z - \bar{z}) = (\bar{x}-\bar{x}) (\bar{z} - \bar{z}) = 0$$
So low covariance (not variance) identifies $$z$$ components that are independent of $$x$$, i.e. carry no information, or are "dead" from a useful representation perspective.
The same measure is used when considering posterior collapse (e.g. in Variational Autoencoders (VAEs)), which refers to when all dimensions of $$z$$ are independent of $$x$$.