What are the current methods to check for GAN overfitting?

Question

In generative modeling, the goal is to find a way for a model to output samples of some distribution $p_X$ given a lot of samples $x_1, \ldots, x_n$. In particular, we want sampling from our model $G$ to satisfy

• $G(z)$ is a new example
• $G(z)$ looks like it was sampled from $p_X$.

GAN's approach this by finding a Nash equilibrium where $p_g=p_X$, where $p_g$ is the distribution implicitly defined by mapping the latent noise $z$ under $G$. How do we know that $G$ does not simply memorize the input data?

For example, if I train a GAN to output pictures of cats, how do I know that the output isn't just a modified picture of one of the cats that was used to train $G$? In the original Deep Convolutional GAN (DCGAN) paper, they have the following explanation which I don't find particularly convincing.

Answer 1

To my knowledge, there aren't any truly robust methods. One strategy that I have seen in involved for the nearest neighbor (often using Euclidean distance). From there, the practitioner conducts visual inspection to check that overfitting is not occuring. This tends to be a poor indicator though because nearest-neighbor approaches tend not to work in such high-dimensional space. Also, see here for failures of pixel-wise loss. Another strategy involves looking at the nearest-neighbors of embeddings of the images (both synthesized and original data) into some space that is designed to extract features. This sounds better but doesn't seem to have much empirical or rigorous theoretical justification, with only a heuristic explanation.

Answer 2

What you're interested is GAN mode collapse and mode dropping. (You can call it overfitting too, it's just that the community has adopted these names). There are literally thousands of GAN papers devoted to solving the problem with varying success, but checking for mode collapse/dropping is still an area of active research.

Mode collapse means that the generator only learns to output a small number of distinct images / classes and very obviously fails to learn the distribution in any sense. Mode dropping means that the generator may appear to learn the distribution, but still drops modes (for example, suppose you train a GAN to generate images 1000 different species of birds... you're unlikely to notice if the GAN only produces images of 500 species and forgets the rest). We can also talk about inter-class mode collapse (GAN fails to learn all classes) and inter-class collapse (GAN can generate images from all classes, but only generates one / a few distinct images for each class).

---

General purpose metrics for GAN evaluation

The most popular metrics for measuring GAN performance are Inception Score and Frechét Inception Distance. A one sentence summary of Inception Score: do the images look like they're being drawn from many different classes? A one sentence summary of FID: does the distribution of perceptual features from generated images match that of ground truth images?

IS can detect inter-class mode collapse, whereas FID can also detect intra-class mode collapse. However these metrics aren't perfect by any means, and a high score / low distance isn't a guarantee that no collapse/dropping has happened.

Another metric is Classifier Augmentation Score, which basically uses GAN outputs to train a classifier, and measures the performance of the classifier. Intuitively, any form of overfitting / mode dropping would result in a poor classifier. However CAS is pretty expensive to compute.

Specific methods to detect mode collapse/dropping

In GLO, the authors propose to examine mode dropping in GANs by a "reconstruction" test. Basically, for some held out ground truth image $x$, we find the best noise vector $z$ which when passed through the GAN, produces something like $x$, and measure the cost by $\text{min}_z\ ||f(z)-x||_2^2$. The idea is that if your GAN can only produce one image of a cat, it will be unable to reconstruct the held out cat image. A similar idea was explored in more detail recently in Seeing what a GAN cannot generate.

The birthday paradox can be rephrased a bit more abstractly as: "If there are $n$ distinct outcomes, it's likely that you'll come across the same outcome twice after sampling only $\Theta(\sqrt{n})$ times". So, if a GAN only copies the 10,000 images from the training set, we can figure this out by just looking at a ~100 of them, as done by Arora and Zhang.

PRD proposes some direct method to measure the precision and recall of a GAN. (precision meaning high quality, non blurry samples, and recall meaning that we haven't dropped any modes in the data, and are fully covering the distribution). It's too complicated to summarize neatly, but the results look promising.