What does log-likelihood mean in the context of generative models like GANs?

I understand the general notion of likelihood as "probability to generate the data given parameters" (like here). But people use (log-)likelihood as a measure of "goodness" of a generative model.

But, e.g., let's take a look at Generative Adversarial Networks model. It takes some random noise and deterministically (using a neural network) transforms it to a sample. If we take a look at a particular (test) sample with a couple of thousands of pixels, isn't the probability that it will be generated 0? As it is very improbable to get the exact values of each pixel in each place and could possibly happen to at most one setting of "noise" anyway?

How is the likelihood defined in terms of GANs and other generative models?

• The likelihood is not a probability in the mathematical sense of the term. It is the value of the probability distribution function at the point x, given the probability function parameters. Does it answer your question? :) Jan 12, 2017 at 16:56
• Also people use the log-likelihood because it is easier to manipulate than the likelihood in some cases. Multiplications become additions, and if high value arise, the log keeps them small (this is especially useful when the probability distribution functions embed a ratio of Gamma functions for instance). Jan 12, 2017 at 16:58
• I don't understand your answer. How is it defined off it's not probability? And what is wrong in the above argument that it would be 0 all the time?
– sygi
Jan 12, 2017 at 17:54
• The likelihood of a discrete variable X to be equal to x for a discrete probability mass function is L = P(X = x|params). In the continuous case, it is defined as the value that the probability distribution function takes at value x. This value can be bigger than 1 for instance as the only requirement for it is: (1) to be positive and (2) that the integral over the support is equal to 1. In informal contexts people often use the word "probability" for "likelihood" hence the confusion. Jan 12, 2017 at 18:37

You're absolutely correct. The log likelihood of most GAN models on most datasets or true distributions is $-\infty$, and the probability density under the trained model for any actual image is almost always zero.

This is easy to see. Suppose a GAN has a 200-dimensional latent variable and is generating 200x200 grayscale images. Then the space of all images is 40,000-dimensional, and the GAN can only ever generate images on a 200-dimensional submanifold of this 40,000-dimensional space. Real images will almost always lie off this submanifold, and so have a probability density of zero under the GAN. This argument holds whenever the output space has higher dimension than the latent space, which is typically the case (e.g. Nvidia's recent progressive GANs used a 512-dimensional latent space to generate 1024x1024x3 images).

Whether this is a problem or not depends on what you want the generative model for; GANs certainly generate visually attractive samples in many cases.

The theory of maximum likelihood is generally not tractable for generative models, as seen here. Instead, methods such as VAEs and GANs, the likelihood is approximated by a KL Divergense, on VAEs, and JS Divergense on GANs.

Such functions are a measure of how much two distribution probabilities diverges, also known as relative entropy.

Roughly, although two distribution of the same shape but different means have same entropy, you can think that while such functions adapt the shape learned by the network, the Discriminator (in case of GANs), when optimal, decide where is the mean.

For example, if a GAN is trying to learn a Gaussian distribution of mean -1 and standard deviation 2, the discriminator, which reaches its optimum state over time, is responsible for locate that mean, while the divergence function learn its shape.

Tecnically, it is possible because, according to the proof in the article, the optimal Discriminator should achieve a cost of -log(4). So the Generator indirectly approximates its actual distribution to reach this value in Discriminator's cost, ensuring the convergence to the mean and the shape.