# how to stabalize GAN learning

I implemented a generative adversarial network. The generator and the discriminator both seem to learn independently. I put them together in the following manner :

pre-train the discriminator to distinguish between the untrained generator output and real output samples.

I then iteratively

1. get half real data samples, half samples from the generator

2. train the discriminator

3. train the entire GAN ( with the discriminator turned static ) by feeding in random numbers and telling the GAN that the output is all of the correct class.

At first, the loss of the entire GAN is nearly 1 ( 100 % error ) and after a few iterations, it reaches about .2 or less at which point the discriminator and the GAN start to fight off so that one has a high loss while the other has a low loss. Then at some point after 50-100 iterations the GAN has a very high loss (0.99998) and the discriminator has very low loss ( 1e-5). The GAN makes some small comebacks but in general, the remainder of the training looks like that.

Am I not waiting long enough for the GAN to beat the discriminator? Should I also pre-train the generator? Is there something else critical I am missing like needing the generator to have lot more parameters than the discriminator or learning rate differences...etc? Should I allow the GAN to learn a few iterations for every iteration that the discriminator gets?

Any knowhow/ experience you can share is welcome. Thank you

• "Should I allow the GAN to learn a few iterations for every iteration that the discriminator gets?" Isn't it exactly what minibatch algorithm does? I think you have answered it yourselves. Dec 11, 2018 at 8:07

Mixup works by training a model using an augmented data set which is composed of random, convex combinations of the original data. For random pairs of indices $$i,j$$, mixup makes new data points and the loss is computed as a convex combination of the two labels. These indices refer to the samples comprising the data, so if your data set is 100 images, then the indices are 1, 2, 3, ..., 100. \begin{align} \lambda &\sim \text{Beta}(\alpha, \alpha) \\ \tilde{x} &= \lambda x_i + (1-\lambda)x_j \\ \mathcal{L}(\tilde{x}|\lambda,y_i,y_j, f) &= \lambda L(f(\tilde{x}), y_i) + (1-\lambda)L(f(\tilde{x}),y_j) \end{align} where $$f(\cdot)$$ is the model prediction and $$L$$ is a standard loss function (e.g. cross-entropy loss) and $$\alpha$$ is a tuning hyperparameter.
• @DanErez No, that's not correct. The equations mean what they say: take some image (perhaps a matrix or a vector) $x_i$ and multiply it by $\lambda$, and then add it to $(1 - \lambda) x_j$. This creates two superimposed images, kind of like double-exposure photography. The mixup article is mostly about images, but in principle can be applied to generic inputs.