2
$\begingroup$

I'm not an ML scientist, but I'm trying to understand how variational autoencoder works.

I'll take as reference the following diagram, which it couldn't be used for backpropagation as includes a sampling process but it captures anyway what I don't understand. The diagram is taken from this link.

enter image description here

I'm specifically going to focus on the encoder part. My understanding is that $x_1,\ldots, x_6$ are real values (the features) and in the second layer each of $a_1, \ldots\ a_4$ is another real value.

No we have these functions $\mu_1,\mu_2,\sigma_1,\sigma_2$, from that diagram again it seems that both $\mu_1$ and $\mu_2$ compute the mean of the vector $a = (a_1,\ldots,a_4)$ but if this is the case then $\mu_1(a) = \mu_2(a)$ and I don't see the point of this, I'd make a similar observation for the $\sigma_1, \sigma_2$ functions.

The question is, w.r.t. that diagram, how exactly are $\mu_1$ and $\mu_2$ computed?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

The point of a variational autoencoder is to have an encoder that produces a probability distribution for a given input. In this model, the latent probability distribution is 2 independent normals, equivalently a bivariate normal distribution with mean vector $\begin{bmatrix}\mu_1 \\ \mu_2 \end{bmatrix}$ and covariance matrix $\begin{bmatrix} \sigma_1^2 & 0 \\ 0 & \sigma_2^2 \end{bmatrix}$. Each input is mapped to its own probability distribution. Then you sample from that distribution, and the decoder reconstructs the input given that random draw from the distribution.

Importantly, $\mu$ and $\sigma$ are not parameters of the network. They are the outputs of the encoder. The way that the model finds good values of $\mu$ and $\sigma$ is by updating the parameters (weights and biases) of the network.

With this in mind, it's important to recognize that $\mu_i$ doesn't compute the mean of $a$; it's an estimate of the mean parameter $\mu_i$ for that observation. Likewise, $\sigma$ is an estimate of the covariance matrix for the latent probability distribution. When your model learns a disentangled latent representation, each component $i$ corresponds to a different feature of that latent representation, so they will not be equal in general.

More details and general information about VAEs are available in this thread: What are variational autoencoders and to what learning tasks are they used?

Improve this answer
$\endgroup$
11
  • $\begingroup$ So wait... are these $\mu$'s and $\sigma$'s learnt as well? $\endgroup$
    – user8469759
    Commented Aug 26, 2019 at 19:00
  • $\begingroup$ @user8469759 $\mu$ and $\sigma$ are not parameters of the network. They are the outputs of the encoder. The way that the model finds good values of $\mu$ and $\sigma$ is by updating the weights and biases of the network. $\endgroup$
    – Sycorax ♦
    Commented Aug 26, 2019 at 19:22
  • $\begingroup$ So they're literally the output of some activation function, it's just how we interpret them then, is that right? $\endgroup$
    – user8469759
    Commented Aug 27, 2019 at 8:32
  • $\begingroup$ It's a matter of interpretation and use. The re-parameterization trick uses $\mu$ and $\sigma$ so that you can draw a random normal deviate and still apply back-prop to the encoder. $\endgroup$
    – Sycorax ♦
    Commented Aug 27, 2019 at 11:40
  • $\begingroup$ But am I correct in saying that in a VAE we want to learn a distribution such that when we sample from such distribution using a Gaussian we get a sample of the distribution representing the data? (I'm reading through the link you provided). $\endgroup$
    – user8469759
    Commented Aug 27, 2019 at 13:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.