Why does enforcing a prior distribution create semantic latent variables in variational autoencoders?

Variational autoencoders create latent variables that have a known distribution (e.g., Gaussian with zero mean and unit variance), and so do adversarial autoencoders. I understand why this turns the autoencoder into a generative model, but somehow this also causes the latent variables to have semantic meaning. For example, when "walking in latent space" of an encoder trained on face images, one of the coordinates of the latent vector might gradually transition the generated images from male to female, and another coordinate might gradually change the age, another might affect the appearance of sunglasses. What is the theoretical explanation for this?

• Because I haven't found anything on this yet: do we know that ordinary non-variational autoencoders have a latent space which is not interpretable in the same way? May 15, 2018 at 19:41
• Classic AEs have a discontinuous and generally highly multidimensional latent space - you simply cannot walk on like you can on VAEs. The closest you can get is by projecting those with a t-SNE or something into a lower-dimensional space. For MNIST at least there's clustering, but it's ugly and messy.
– jkm
Jun 9, 2018 at 22:19

It doesn't. Not on its own. That property is a product of the interaction between the elements that comprise the model.

First there's the assumption of (normal) distribution of each variable and reconstruction from sampling that distribution, which means encoded values represent N real-valued features (N being the dimensionality of the latent space).

Leave that out altogether and of course, you no longer have a VAE - just a bunch of arbitrary encodings that happen to represent the true values... somehow.

Reconstruction loss, of course, is what enforces your encoded feature is meaningful - without it, you're not really learning anything.

Second comes the constraint on the values of the distribution parametrs - typically KLD between the parameters and a zero-mean, unit-stddev Gaussian. It's effectively a form of centripetal force that anchors the representations together.

That's what gives you an interpolatable latent space. If you used it as the only loss, the optimization will output all the encodings as random means close to zero, so you'll arrange them into something like an N-dimensional bubble in the latent space.

Leave it out on the other hand and chances are the values will drift apart across the manifold. Why?

You're training the model on samples where features vary in some way - for example, you're showing the model samples of the same face at different rotations, or two identically positioned faces of two different people, or a combination of both...

This means the representation of each face needs to somehow account for the variation in the facial features in the training set in the bottleneck, and won't you know it - we've already set up the encoding to be something perfect for that purpose.

The differences between samples that cannot be explained away as noise push the means apart to allow sampling distinct subclasses - if the means are close together, the sampled feature shows up equally likely in the decoded data in both classes, so it'd be a waste of precious bottleneck neurons as far as the optimizer cares.

So, the semantic features come from trying to best represent N most semantically salient variables in the training data (as extracted by the preceding encoder layers).

Encoding as mean+stddev ensures the features are interpolatable, and the KL-Div term limits the space between feature encodings to minimize the gaps in interpolation.

• How does this explain the ability to perform vector arithmetic with the latent features? For example, (man with glasses) - (man) + (woman) = (woman with glasses) Jun 10, 2018 at 15:08
• It simply directly follows from the fact that the encodings are semantically meaningful and is not exclusive to VAEs. If you managed to teach the encoder to assign glasses-havey-ness to a value on neuron 1 and gender to value on neuron 2, you can change one value without affecting the other more or less by definition, and it will give you the same output feature given that value - again by definition. VAEs simply facilitate the process by constraining the space where you may find those values and their magnitude.
– jkm
Jun 10, 2018 at 15:36
• Also, in a VAE, the feature arithmetic is a bit different - assuming a 2D space with gender (M/W) and glasses (G/N) being the features on the x/y axes of the space, WG = -1*MN, (M/W)G = [1,-1] * (M/W)N, and M(G/N) = [-1,1]*W(G/N). Simply adding means will give you ambiguous cases, e.g. MG+WG will result in an androgynous face wearing glasses.
– jkm
Jun 10, 2018 at 15:47
• Why is there a KL term only for the encoder distribution, and not also for the decoder distribution? Based on the motivation for the KL term as regularization encouraging interpolability, one would imagine -- by symmetry, and the fact that for a fixed $\hat{z}$ sampled in the latent space, the $\hat{x}$ generated are normally distributed) -- that there should be a KL penalty on the decoder distribution, too? Why isn't there one? Jul 7 at 16:16
• It doesn't follow that the generated x^ will be normally distributed - unless your decoder is very simple, the normal samples will go through a bunch of nonlinearities. In fact, the outputs probably shouldn't be normally distributed, because whatever you're generating is usually not going to be normally distributed
– jkm
Jul 9 at 11:02