1
$\begingroup$

I have some experience with machine learning, but no background in deep learning at all. The idea of a GAN looks so cool, and there are so many sources out there that talks about the idea, but there is no good source that I could find which can convey how to train GANs to a deep learning beginner.

Here are my detailed questions:

  1. Why is there a need for the vector $z$? And why $p(z) = N(0,1)$ initially?

  2. Is the latent vector $z$ learned in the training step of the generator network?

  3. Do we ever learn the distributions $p(x)$ and $p(z)$? If so, when?

  4. According to here, the discriminator network is trained with the real samples and the fake samples in separate steps. When there is only one class of samples (either real or fake), how come you can train a "discriminator" (there is nothing to discriminate)?

$\endgroup$
4
$\begingroup$

First off, if you are new to deep learning, a good idea is to start from scratch, like a "linear GAN", and deep learning is just replacing the linear models with neural networks. They do the same thing, just performing better on images/texts etc.

Suppose you have some 1D data $X = \{x_1, \dots, x_n\}$, and you want your generator function $G$ to produce some new data different from $X$ but look like $X$ (one purpose of GAN is to create labeled data because they are often expensive), so you cannot cheat by taking the average $\bar{x}$ as that's just one data. You will need some randomness, for example, $z \sim N(0,1)$ as the seed for your next generated data. Say you decide to use $G(z) = w_1z + b_1$ as your generator, and initialize your $W, b$ randomly.

Now your $G$ produces random data $x \sim N(b_1, w_1)$ different from $X$, but how to achieve the second goal (look like $X$)? The answer is to use a classifier. Say your classifier is logistic $D(x) = \sigma(w_2x + b_2)$, and you 1) sample some data from $X$; 2) generate some data from $G(z)$. You label them with $1$ for real data from $X$ and $0$ for fake data from $G(z)$, and train your logistic classifier as usual.

Your discriminator works nicely now, and then you want your generator to be good, too. So you generate some data from $G$, and for the data that successfully fooled $D$, you want $G$ to generate more, and less otherwise. You tune $w_1, b_1$ to achieve this.

Then you go back to train your discriminator again. The process will continue until the Nash equilibrium, where your generator works perfectly and your discriminator decides randomly.

Regarding your questions, IMPO:

1) Why is there a need for the vector z? And why p(z) = N(0,1) initially?

Goodfellow calls the distribution a "noise prior" in the original paper, which explains a lot. Think of $z$ as the random number seed. Since we want to generate fake data, we must give the generator some seed to start with. The choice of $N(0,1)$ is for convenience, I think.

2) Is the latent vector z learned in the training step of the generator network?

No, we don't want to learn the seed. We want to learn the function that processes the seed, i.e. the parameters that define the generator network.

3) Do we ever learn the distributions p(x) and p(z)? If so, when?

In the end we want that given $z \sim p(z)$, we have $G(z) \sim p(x)$. So in some sense, we do want the generator to learn $p(x)$ through training. but there is no point to learn $p(z)$.

4) According to here, the discriminator network is trained with the real samples and the fake samples in separate steps. When there is only one class of samples (either real or fake), how come you can train a "discriminator" (there is nothing to discriminate)?

The link to "here" seems to be dead, but at least according to Goodfellow's original paper, you do sample two classes of examples during discriminator training (using the cross-entropy loss). As the comment points out, SGD confuses people a bit by only taking some data in each loop, but the stochastic gradient does have the same gradient in expectation as the regular GD. The discussion is beyond the range of this question, though.

original algorithm

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ To clarify 4 a bit: in practice, you likely update the discriminator with a batch of all real samples and then with a batch of all generated samples for convenience. It's just an SGD step, and it still "remembers" the previous updates, so this should be basically the same as mixing the two batches together and then taking a training step on the mixed batch. $\endgroup$ – djs Sep 15 '17 at 23:55
  • $\begingroup$ I don't fully understand the training process in the link you provided. Maybe take a look at Goodfellow's actual code if you are not confident: github.com/goodfeli/adversarial $\endgroup$ – Jiāgěng Sep 16 '17 at 5:27
  • $\begingroup$ @Jiāgěng You said "z is sampled from our seed generator p(z)". Is not p(z) the standard Gaussian? Is it standard Gaussian only in the beginning and changing over time after the generator has a good sense of the data? $\endgroup$ – user5054 Sep 16 '17 at 5:48
  • $\begingroup$ @user5054 I have clarified my answer a bit. $p(z)$ is the standard Gaussian and does not change. The only thing $p(z)$ does in the model is to provide some randomness to our generated data. The job of "having a good sense of the data" goes to the generator network, not the seed it is fed with. $\endgroup$ – Jiāgěng Sep 16 '17 at 19:50
  • $\begingroup$ @Jiāgěng Thanks! 2 more questions and I really appreciate your response. 1) So, we do not learn a latent structure for the data at all. I at first was thinking that we learn the latent vector $z$, which would be a low-dimensional representation of the data. Are you aware of a GAN extension which can do that? 2) Do you think 600 samples is too few to train a GAN in a setting where I want to generate fake samples from $p(x)$ which is 42000-dimensional? This is a genomic dataset but I think the setting resembles generating a fake image where there are 600 training images each with 42000 pixels. $\endgroup$ – user5054 Sep 19 '17 at 0:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.