0
$\begingroup$

I'm trying to understand further how a variational autoencoder works beyond the conceptual level. However I'm still confused as to what the "vector of mean and variances" can look like and to digest it in a simplistic way.

For example, I understand that the latent variables in an autoencoder represents the compressed features of some input X and in the context of a variational autoencoder, you try to get the probabilistic distribution represented by mean and variance of the latent variable. So does this mean that:

a. If I have 5 latent variables in an autoencoder, in the context of a variational autoencoder, I should have 10 parameters (2 sets of mean and variances for each latent variables) represented as 2 vectors (1 vector of size 5 for means and 1 vector of size 5 for variances). In sampling/decoding, I can pass 5 means and 5 variances to generate an output.

OR

b. Based on code examples, the representation of mean and variances are always 2 values (during sampling, you can randomize just a single mean and variance).

Thanks in advance. My goal is to be able to simplify the explanation since most examples online always give just 2 variables for decoding (just a single mean and variance value).

$\endgroup$
2
$\begingroup$

It's a). In VAE you assume that distribution over latent variables is multivariate normal with diagonal covariance matrix, and penalize using KL divergence from standard normal distribution. This KL divergence can be calculated using mean and covariance matrix of the distribution that is being sampled.

$\endgroup$
  • $\begingroup$ Got it. Thanks. So when sampling, I just pass in values for latent variables (let's say 5 values) from (0, 1) right? The closer the value it is to 0 the less likely or farther the sample is from the distribution for that latent variable? $\endgroup$ – user1668926 Jan 5 at 17:07
  • $\begingroup$ I don't know what 'pass in values for latent variables' means. And where does (0,1) come from $\endgroup$ – Jakub Bartczuk Jan 5 at 17:21
  • $\begingroup$ I mean when sampling from a trained autoencoder. So let's say I have an autoencoder with an architecture of 10 as my input vector and 5 as my latent space vector. (10 input x, 5 latent z and 10 output y). If I were to create a variational autoencoder, this means I would want to sample base off of the 5 latent variables right? So each of those latent variables would have some mean and variance. Does this mean that given the trained autoencoder, I would have to pass 5 values in the decoding process where each value is from (0, 1)? Hope it made sense. $\endgroup$ – user1668926 Jan 5 at 17:32
  • $\begingroup$ I don't get where you get values from (0,1) from. When you have an autoencoder it contains parameters for the distribution on latent variable that can be used for sampling. $\endgroup$ – Jakub Bartczuk Jan 8 at 21:37
  • $\begingroup$ The (0,1) values are for sampling with a trained variational autoencoder. Check the bottom part of this article (wiseodd.github.io/techblog/2016/12/10/variational-autoencoder) for reference. Sorry I can't explain it as clearly. $\endgroup$ – user1668926 Jan 9 at 1:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.