2
$\begingroup$

This is the value function from the GANs paper:

enter image description here

The authors explain that this equation "may not provide sufficient gradient for $G$ to learn well", because early in the learning process the discriminator $D$ can reject the generated samples from $G$ with high probability. This leads to the saturation of $\log(1-D(G(z)))$.

My question: I know that saturation in this context means, that the learning is slowed down, because the gradient is so small. I would like to understand why.

My idea is that because the generated sample $G(z)$ is basically just random noise, $D(G(z))$ (probability of $G(z)$ coming from the training data) is close to 0.

Thus the influence of $G(z)$ on $\log(1-D(G(z)))$ is small, meaning the partial derivative $\frac{\partial \log(1-D(G(z)))}{\partial G(z)}$ is very small. This will lead to a small gradient, because the gradient of $G$, the partial derivatives of $\log(1-D(G(z)))$ with respect to the weights and biases, are multiplied with $\frac{\partial \log(1-D(G(z)))}{\partial G(z)}$ via the chain rule. And as I stated above a gradient close to 0 will lead to slow learning/saturation.

Is this reasoning correct?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.