What does log-likelihood mean in the context of generative models like GANs?

Question

I understand the general notion of likelihood as "probability to generate the data given parameters" (like here). But people use (log-)likelihood as a measure of "goodness" of a generative model.

But, e.g., let's take a look at Generative Adversarial Networks model. It takes some random noise and deterministically (using a neural network) transforms it to a sample. If we take a look at a particular (test) sample with a couple of thousands of pixels, isn't the probability that it will be generated 0? As it is very improbable to get the exact values of each pixel in each place and could possibly happen to at most one setting of "noise" anyway?

How is the likelihood defined in terms of GANs and other generative models?

Answer 1

You're absolutely correct. The log likelihood of most GAN models on most datasets or true distributions is $-\infty$, and the probability density under the trained model for any actual image is almost always zero.

This is easy to see. Suppose a GAN has a 200-dimensional latent variable and is generating 200x200 grayscale images. Then the space of all images is 40,000-dimensional, and the GAN can only ever generate images on a 200-dimensional submanifold of this 40,000-dimensional space. Real images will almost always lie off this submanifold, and so have a probability density of zero under the GAN. This argument holds whenever the output space has higher dimension than the latent space, which is typically the case (e.g. Nvidia's recent progressive GANs used a 512-dimensional latent space to generate 1024x1024x3 images).

Whether this is a problem or not depends on what you want the generative model for; GANs certainly generate visually attractive samples in many cases.

Answer 2

The theory of maximum likelihood is generally not tractable for generative models, as seen here. Instead, methods such as VAEs and GANs, the likelihood is approximated by a KL Divergense, on VAEs, and JS Divergense on GANs.

Such functions are a measure of how much two distribution probabilities diverges, also known as relative entropy.

Roughly, although two distribution of the same shape but different means have same entropy, you can think that while such functions adapt the shape learned by the network, the Discriminator (in case of GANs), when optimal, decide where is the mean.

For example, if a GAN is trying to learn a Gaussian distribution of mean -1 and standard deviation 2, the discriminator, which reaches its optimum state over time, is responsible for locate that mean, while the divergence function learn its shape.

Tecnically, it is possible because, according to the proof in the article, the optimal Discriminator should achieve a cost of -log(4). So the Generator indirectly approximates its actual distribution to reach this value in Discriminator's cost, ensuring the convergence to the mean and the shape.