Differentiability in Generative Adversarial Networks

Question

I've got some questions about the differentiability condition of GAN's, i.e. both G and D need to be differentiable wrt. their inputs and the parameters describing them. It's of more mathematical nature and I just couldn't find a clear answer to that anywhere else and I want to make sure I understand these things correctly before presenting them in a seminar. I'm always referring to goodfellows paper / tutorial in the following questions.

https://arxiv.org/abs/1406.2661

https://arxiv.org/abs/1701.00160

1) Is differentiability of G and D wrt. their parameters only required to train them by gradient descent or are there other reasons?

2) Why do we need differentiability wrt. the inputs of the generator? I don't think we need that to apply gradient descent algorithms. (Please correct me if I'm wrong there.) Differentiability of the Discriminator wrt it's inputs is required for gradient descent (having a look at the objective function and applying the chain rule).

3) Somewhat related to the other two questions: When Good fellow et al talk about probability distribution in their paper, $p_{data}$, $p_z$ and $p_g$, are they talking about discrete or continuous probability distributions? I don't think it's made clear in the paper and in the latter case it would imply, for instance, that both $p_{data}$ and $p_g$ be differentiable since the optimal Discriminator is essentially a function of their ratio. Also, the existence of a continuous $p_g$ is non-trivial by just assuming a continuous $p_z$.

4) In the case that the answer to 3 is discrete distributions: Differentiability of G implies continuous outputs of the generator. How can this work together with a discrete distribution $p_g$ of its outputs? Does the answer have something to do with the fact that we can only represent a finite set of numbers with computers anyway?

Thank you very much!

Best

EDIT: updated link for tutorial by goodfellow

Answer 1

1) Is differentiability of G and D wrt. their parameters only required to train them by gradient descent or are there other reasons?

I believe it's also assumed in a lot of GAN convergence proofs. example. Granted, this is mostly of theoretical interest rather than practical.

2) Why do we need differentiability wrt. the inputs of the generator?

Yeah, you're right, wrt. the parameters should be good enough.

are they talking about discrete or continuous probability distributions?

Usually continuous. However there is work which explores discrete domain GANs (which have interesting applications in language or molecule generation). The discrete case usually requires some sort of trick like gumbel-softmax or Reinforce.

Also, the existence of a continuous $p_g$ is non-trivial by just assuming a continuous $p_z$.

I'm not familiar with theory here, but the paper I linked above does analyze convergence on absolutely continuous generator distributions versus the general case.