This is the value function from the GANs paper:

enter image description here

The authors explain that this equation "may not provide sufficient gradient for $G$ to learn well", because early in the learning process the discriminator $D$ can reject the generated samples from $G$ with high probability. This leads to the saturation of $\log(1-D(G(z)))$.

My question: I know that saturation in this context means, that the learning is slowed down, because the gradient is so small. I would like to understand why.

My idea is that because the generated sample $G(z)$ is basically just random noise, $D(G(z))$ (probability of $G(z)$ coming from the training data) is close to 0.

Thus the influence of $G(z)$ on $\log(1-D(G(z)))$ is small, meaning the partial derivative $\frac{\partial \log(1-D(G(z)))}{\partial G(z)}$ is very small. This will lead to a small gradient, because the gradient of $G$, the partial derivatives of $\log(1-D(G(z)))$ with respect to the weights and biases, are multiplied with $\frac{\partial \log(1-D(G(z)))}{\partial G(z)}$ via the chain rule. And as I stated above a gradient close to 0 will lead to slow learning/saturation.

Is this reasoning correct?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.