Question: Optimal D notation in Generative Adversarial Network (GANs) I am completely new to Computer Vision and how Deep Neural Networks work on images in general. In particular, I have questions on the Discriminator component of Adversarial Generative Network (GANs). There are some things that are left unexplained to me:
Optimal Discriminator is given by:
$$
D^*(x) = \frac{p_{data}(x)}{p_{data}(x) + p_G(x)}
$$
where $p_{data}(x)$ and $p_G(x)$ are the probability distribution of samples of the original and generated dataset, respectively. However, assuming that $x$ is a $2\times2$ RGB image, how is $p(x)$ calculated in this case? In other words:
$$
p(x) = p([R,G,B]) = p([[1,2,3,4],[4,5,6,8],[9,10,11,12]]) = ???.
$$
And how is this different for $p_{data}(x)$ and $p_G(x)$? Assuming that both $p_{data}(x)$ and $p_G(x)$ returns a single value, resulting in optimal $D^*(x) = 1/2$.
If anyone could also give me a source that explains this well, that would be very much appreciated!
 A: $p(x)$ is some probability distribution or density over images $x$. Of course, there can be many different distributions over $x$, so it can make sense to give them names like $p_\text{data}(x)$ and $p_G(x)$. Without more context, I can't tell what exactly your $p(x)$ refers to.
$p_G$ is the distribution of images generated by the generator network - and while you can sample from this distribution, it's generally intractable to actually compute $p_G(x) = \int p(x|z;\theta)p(z) dz$, where $p(x|z;\theta) = \mathcal{N}(G(z;\theta), \sigma^2)$.
$p_\text{data}$ is the distribution of $x$ in your dataset. Depending on what you're doing, this could be almost anything, but in the context of training a GAN to produce natural images, you could imagine the process by which the universe, and then the earth was created, leading to a photographer to point their camera at a subset, and photons traveling down the lens and onto each photosite of a camera sensor, forming the images you have in your dataset.
Obviously, this can't be practically computed, but if in theory you knew all the possible initial states of the universe and the true laws of physics, and had some measure over the initial states, you'd know exactly what the distribution over observed images is.
Luckily, when training GANs, we don't really need to worry about the optimal discriminator -- it turns out that gradient descent does a good enough job.
