3
$\begingroup$

The adversarial loss function and training algorithm for GANs are more or less intuitive to me, but I feel I do not entirely understand the Generator's domain. Specifically, the following line from the original paper (Generative Adversarial Nets, Goodfellow et al. 2014)

To learn the generator's distribution $p_g$ over data $x$, we define a prior on input noise variable $p_z(z)$, then represent a mapping to data space as $G(z;\theta_g)$, where $G$ is a differentiable function represented by a multilayer perceptron with parameters $\theta_g$.

Where this is the first step in the training algorithm:

  • Sample minibatch of $m$ noise samples $[{z^{(1)},...,z^{(m)}}]$ from noise prior $p_g(z)$

I am not sure I understand probability distribution and random sampling enough to understand this part. Could someone explain what is going on? It seems like everyone skips over the very beginning of the GAN algorithm (given random vector Z, do stuff) and just talk about the adversarial loss.

Does the generator sample from a probability distribution that was assigned in the beginning, or is it simply doing transformations with its weights $W$ on a vector of random numbers generated upon each iteration?

$\endgroup$
3
$\begingroup$

I am not sure I understand probability distribution and random sampling enough to understand this part. Could someone explain what is going on?

In non-math words: the generator $G$ is a network and as such needs some kind of input to produce an image, we call this input $z$ and so the generated images can be seen as $G(z)$, or more explicitly $G(z, \theta_G)$.

Does the generator sample from a probability distribution that was assigned in the beginning, or is it simply doing transformations with its weights $W$ on a vector of random numbers generated upon each iteration?

In math words, $z$ is drawn from a prior probability distribution $p_z$ and then used to condition on another probability distribution $p_G$ so that samples drawn from it are similar to real images.
So, to answer your question: yes and yes. Yes because the probability distribution is chosen from the beginning and never changes, and yes because every time we feed an input to the generator we sample a new vector from that distribution, i.e. we feed a vector of randomly generated number.

Now, how do people select the prior distribution probability $p_z$?
A common and sensible choice is to draw $z$ from a zero-centered unit-variance Gaussian distribution. This gives some nice properties once the network is trained:

  1. The unit variance means that each element in the random vector might end up representing a feature in the generated image uncorrelated to the other features (e.g. $z_2$ might be "the face has glasses" and $z_5$ might be "length of the hairs")
  2. The zero-centered Gaussian means that it will be easy to draw samples and even interpolate between two values and see a progressive change in the image. In fact, because the density is all around the center and not scattered, we can pick any point in the latent space and be sure that the generated image will make sense, rather than being a mess of pixels. In other words, there are no "holes" in the prior where the network was not trained enough to generate something meaningful.

Hope this helps ;)

$\endgroup$
  • $\begingroup$ By "unit Gaussian distribution" do you mean that we generate unit vectors that have a random direction? So the points will be on a hypersphere? $\endgroup$ – Bloc97 Mar 21 '18 at 15:51
  • $\begingroup$ The vectors you sample do not need to have unit length. So in 2D you would not be constrained to the circumference of a circle, in 3D you would not be constrained to the surface of a spere etc. $\endgroup$ – baldassarreFe Mar 21 '18 at 22:29
  • $\begingroup$ What I mean with "zero-centered" and "unit-variance" is that the mean of the vectors you sample will be 0 (i.e. no preferred direction, i.e. random direction) and the variance of the vectors you sample will be 1 (i.e. the length of the vector in one direction does not affect the length in other directions, i.e. the single components of the vector are independent) $\endgroup$ – baldassarreFe Mar 21 '18 at 22:36
  • $\begingroup$ I understand now, but I still don't get how [eg. z2 might be "the face has glasses" and z5 might be "length of the hairs"] would occur in such a network since all vectors are randomized. In fact wouldn't the network learn that all the prior components affect the hair and eye color? $\endgroup$ – Bloc97 Mar 22 '18 at 17:45
  • 1
    $\begingroup$ If you don't apply explicit conditioning you won't get that clear separation of attributes (check Conditional GANs and similar ones). However, there is one thing you can be sure of: given that during training you fed the network with randomized vectors that span all across the input/noise/prior space, you can be sure that for every point you sample at inference time you are going to generate some believable face. In other words, you can interpolate between two noise vector and be sure that there will be no "gap" in the quality of the generated faces, all should be equally realistic (kind of). $\endgroup$ – baldassarreFe Mar 23 '18 at 20:09
1
$\begingroup$

The following idea from this awesome Stanford lecture about generative models was very helpful to me in understanding the sample generations in GANs:

Deep Neural Networks (DNNs) are great in learning mappings from an input X to an output Y but are deterministic, hence cannot generate new samples. To overcome this, we sample random noise from a simple probability distribution and use a DNN to learn a transformation from the simple noise distribution to the complex (training) data distribution.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.