2
$\begingroup$

I'm probably missing something obvious, but if we're only looking to generate images and are not interested in the latent space, why do we even need the encoder in VAEs?

In my understanding, the second term of the VAE loss mainly ensures that the encoder distribution approaches $N(0,1)$ which then makes it easier to generate new outputs, as we only need to sample from a standard normal distribution and do not have to know the encoder distribution explicitly.

Why can't we just start by sampling from a normal distribution and train a generator using the reconstruction loss, without the encoder?

Or is in only a nice side effect that we also get an encoder "for free", as in BiGANs for example?

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ Intuitively, you let the encoder construct your latent space. Without it you could manually impose which parts of the noise shall map to which labels.... But how do you know which labels should go together closely? $\endgroup$
    – Ggjj11
    Commented May 23 at 20:38
  • $\begingroup$ I guess that makes sense.. But isn't this basically what a GAN does? As far as I know, we also just sample noise without any knowledge about the relationship between different noise vectors and it seems to be working in that case right? $\endgroup$
    – Jannik
    Commented May 23 at 21:09
  • 2
    $\begingroup$ But a GAN is trained differently without needing to reconstruct an image from noise: you have a generator (similar to the decoder) and a discriminator. Certainly your generator generates data from noise, but only such that the generator cannot tell it apart from the training data. In GANs the training is more indirect (and also much less stable ...). So in a GAN you do not care about any structure in the noise space and most importantly there is no reconstruction loss in a GAN. $\endgroup$
    – Ggjj11
    Commented May 23 at 22:04
  • $\begingroup$ Alright, thanks! $\endgroup$
    – Jannik
    Commented May 23 at 22:39

1 Answer 1

Reset to default
2
$\begingroup$

This decoder can exist theoretically, but you "can't train" to arrive to it. Imagine we have a space ( Z ) distributed as a normal distribution from which we sample; then we apply a decoder to obtain an image. The problem is that during training, we would sample ( z )'s and real-looking images ( x )'s randomly. This can lead to samples like ( (z, 1) ) and ( (z, 2) ). (This is not a mathematical function) Basically, the model won't know which exact number you want to generate for each point of the latent space because during training, you will provide misleading (or will lead the model in different directions) samples. Therefore, you can do two things here:

  1. Add an encoder, which will enforce, when sampling ( x ), some kind of specific ( z )'s; therefore, you won't have these misleading training pairs ( (z, 1) ), ( (z, 2) ).
  2. Add a discriminator, for which you ask the model to just generate realistic samples, and get rid of the encoder. Now you won't have training pairs; you will have just ( z )'s, and the decoder would be able to give the desired structure to the latent space.
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.