0
$\begingroup$

In WGAN(https://arxiv.org/abs/1701.07875), we have both gradient clipping and gradient penalty. How both differ and why does standard GAN doesn't have any kind of gradient penalty or gradient clipping.

Can we have gradient penalty for standard GAN?

Any Kind of reference is helpful.

$\endgroup$

1 Answer 1

1
$\begingroup$

In your standard GAN, the discriminator loss is only an indication of how well the discriminator is doing. However in a WGAN, the discriminator loss is proportional to the negative Wasserstein distance between the generator distribution and the true distribution. However, this is only true if the discriminator is Lipschitz (among other conditions). But how can we control the Lipschitzness of a neural network? As it turns out, two ways are

  1. Clip the weights of the network. It shouldn't be hard to see that this makes each layer lipschitz, and therefore the entire network is lipschitz.

  2. Use a loss function to discourage the gradient from being too far from 1. This doesn't strictly constrain the network to be lipschitz, but empirically, it's a good enough approximation.

Since your standard GAN, unlike WGAN, is not trying to minimize Wasserstein distance, there's no need for these tricks. However, constraining a similar measure -- the "spectral norm" of a neural network, has been found to improve standard GAN performance. For a linear layer, the spectral norm and the lipschitzness are the same. However, from what I've heard and seen, spectral normalization doesn't work particularly well in WGANs.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.