I implemented a generative adversarial network. The generator and the discriminator both seem to learn independently. I put them together in the following manner :

pre-train the discriminator to distinguish between the untrained generator output and real output samples.

I then iteratively

  1. get half real data samples, half samples from the generator

  2. train the discriminator

  3. train the entire GAN ( with the discriminator turned static ) by feeding in random numbers and telling the GAN that the output is all of the correct class.

At first, the loss of the entire GAN is nearly 1 ( 100 % error ) and after a few iterations, it reaches about .2 or less at which point the discriminator and the GAN start to fight off so that one has a high loss while the other has a low loss. Then at some point after 50-100 iterations the GAN has a very high loss (0.99998) and the discriminator has very low loss ( 1e-5). The GAN makes some small comebacks but in general, the remainder of the training looks like that.

Am I not waiting long enough for the GAN to beat the discriminator? Should I also pre-train the generator? Is there something else critical I am missing like needing the generator to have lot more parameters than the discriminator or learning rate differences...etc? Should I allow the GAN to learn a few iterations for every iteration that the discriminator gets?

Any knowhow/ experience you can share is welcome. Thank you

  • $\begingroup$ "Should I allow the GAN to learn a few iterations for every iteration that the discriminator gets?" Isn't it exactly what minibatch algorithm does? I think you have answered it yourselves. $\endgroup$ – Failed Scientist Dec 11 '18 at 8:07

I've never trained a GAN, so this answer is incomplete. However, the mixup paper, "mixup: Beyond Empirical Risk Minimization" (Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, David Lopez-Paz) claims that mixup stabilizes the training of GANs.

Mixup works by training a model using an augmented data set which is composed of random, convex combinations of the original data. For random pairs of indices $i,j$, mixup makes new data points and the loss is computed as a convex combination of the two labels. These indices refer to the samples comprising the data, so if your data set is 100 images, then the indices are 1, 2, 3, ..., 100. $$ \begin{align} \lambda &\sim \text{Beta}(\alpha, \alpha) \\ \tilde{x} &= \lambda x_i + (1-\lambda)x_j \\ \mathcal{L}(\tilde{x}|\lambda,y_i,y_j, f) &= \lambda L(f(\tilde{x}), y_i) + (1-\lambda)L(f(\tilde{x}),y_j) \end{align} $$ where $f(\cdot)$ is the model prediction and $L$ is a standard loss function (e.g. cross-entropy loss) and $\alpha$ is a tuning hyperparameter.

This helps training by enforcing smoother interpolation between training data, instead of abrupt shifts.

There are probably more tricks that could help, this just happens to be one that I learned "by accident" in the course of doing some other research.

| cite | improve this answer | |
  • $\begingroup$ Thanks sycorax. I am not sure I understood. Let's say we are using the GAN to generate images from random noise. Instead of using only random noise. I would input real images where some of the pixels are random noise. I then label the image based on the ratio of real to fake pixels ( instead of 0 or 1 )? $\endgroup$ – Dan Erez Dec 20 '18 at 13:40
  • $\begingroup$ @DanErez No, that's not correct. The equations mean what they say: take some image (perhaps a matrix or a vector) $x_i$ and multiply it by $\lambda$, and then add it to $(1 - \lambda) x_j$. This creates two superimposed images, kind of like double-exposure photography. The mixup article is mostly about images, but in principle can be applied to generic inputs. $\endgroup$ – Sycorax Feb 12 '19 at 15:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.