My understanding of convergence of GANs (at least in theory) is that when the discriminator can no longer distinguish between real and fake examples the GANs have converged. Given a real example the discriminator is only 50% sure whether it is real or fake (same for when it is given a fake) i.e. it is resorting to a coin toss! That means that the generator has successfully "learnt" the density function so that anything it creates looks like a real example. Is that correct? Thx.
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
Theoretically what you said is right, that a GAN converges only when the discriminator cannot distinguish generated image from sampled image, but you can just think about how impossible it is. Images lie in $\mathbb{R}^n$, where $n$ is at least 784 (for MNIST). Just think about it, how can you capture a distribution in $\mathbb{R}^{784}$?
In fact it has been proven that $p_{G}$ have no intersection with $p_{data}$ almost everywhere in a 2017 paper.
Empirically, GAN convergence is hard to capture because GAN loss are always vibrating. What I use is to see how the generated samples look like and that may give some sense (after a sufficiently long time, say 20 epochs). Of course it is not an ideal method, but that is currently the only way.
$\begingroup$
$\endgroup$
Your understanding seems to be based on Proposition 1 of Goodfellow et al (2014) https://arxiv.org/abs/1406.2661, that is, the optimal discriminator result $$D_G^*(x)=\frac{{\rm p}_d(x)}{{\rm p}_d(x)+{\rm p}_g(x)}$$ where ${\rm p}_d(x)$ is the data PDF and ${\rm p}_g(x)$ is the generator output PDF. Clearly if ${\rm p}_d(x)={\rm p}_g(x)$ (almost everywhere in $x$), then $D_G^*(x)=\frac{1}{2}$, which is where the 50% figure comes from. Recent research by myself and a collaborator has shown that the optimal discriminator result requires ${\rm dim}(z)\geq{\rm dim}(x)$, that is, the dimension of the latent variable is at least equal to that of the data. This is however not the case that applies in practice: most of the time for GANs we have ${\rm dim}(z)<{\rm dim}(x)$. In this case it can be shown that the generator output PDF contains ${\rm dim}(x)-{\rm dim}(z)$ delta functions. This invalidates the application of variational calculus used in the proof of the optimal discriminator result. The proof of these claims is contained in a peer reviewed IEEE Access paper: Convergence & Optimality of low dimensional GANs.
The same paper also has a quasi-analytical counter-example for the simplest case of ${\rm dim}(z)={\rm dim}(x)=1$, i.e., a one-dimensional LSGAN, which uses a least-squares loss function. (Here quasi-analytical means that the results can be obtained to high accuracy via a simple 1-D Monte Carlo integration that does not require approximation of the expectations in the GAN loss function.) The convergence of this 1-D LSGAN generally does not achieve ${\rm p}_d(x)={\rm p}_g(x)$ with the parameters reaching a saddle point. Instead the parameters converge to a plateau where the gradients of the loss function are zero. The exact convergence point depends on initialisation, generator & data PDF and on the optimisation step size. For certain parameter values, divergence results even in this simple case. This paper has some interesting visualisations to illustrate these points.
Note that none of this invalidates the utility of GANs, but the theoretical claims need further qualification.
