how to stabalize GAN learning

Question

I implemented a generative adversarial network. The generator and the discriminator both seem to learn independently. I put them together in the following manner :

pre-train the discriminator to distinguish between the untrained generator output and real output samples.

I then iteratively

1. get half real data samples, half samples from the generator

2. train the discriminator

3. train the entire GAN ( with the discriminator turned static ) by feeding in random numbers and telling the GAN that the output is all of the correct class.

At first, the loss of the entire GAN is nearly 1 ( 100 % error ) and after a few iterations, it reaches about .2 or less at which point the discriminator and the GAN start to fight off so that one has a high loss while the other has a low loss. Then at some point after 50-100 iterations the GAN has a very high loss (0.99998) and the discriminator has very low loss ( 1e-5). The GAN makes some small comebacks but in general, the remainder of the training looks like that.

Am I not waiting long enough for the GAN to beat the discriminator? Should I also pre-train the generator? Is there something else critical I am missing like needing the generator to have lot more parameters than the discriminator or learning rate differences...etc? Should I allow the GAN to learn a few iterations for every iteration that the discriminator gets?

Any knowhow/ experience you can share is welcome. Thank you

Answer 1

I've never trained a GAN, so this answer is incomplete. However, the mixup paper, "mixup: Beyond Empirical Risk Minimization" (Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, David Lopez-Paz) claims that mixup stabilizes the training of GANs.

Mixup works by training a model using an augmented data set which is composed of random, convex combinations of the original data. For random pairs of indices $i,j$, mixup makes new data points and the loss is computed as a convex combination of the two labels. These indices refer to the samples comprising the data, so if your data set is 100 images, then the indices are 1, 2, 3, ..., 100. $$ \begin{align} \lambda &\sim \text{Beta}(\alpha, \alpha) \\ \tilde{x} &= \lambda x_i + (1-\lambda)x_j \\ \mathcal{L}(\tilde{x}|\lambda,y_i,y_j, f) &= \lambda L(f(\tilde{x}), y_i) + (1-\lambda)L(f(\tilde{x}),y_j) \end{align} $$ where $f(\cdot)$ is the model prediction and $L$ is a standard loss function (e.g. cross-entropy loss) and $\alpha$ is a tuning hyperparameter.

This helps training by enforcing smoother interpolation between training data, instead of abrupt shifts.

There are probably more tricks that could help, this just happens to be one that I learned "by accident" in the course of doing some other research.