I'm trying to understand if the latest ML methods associated with GANs can estimate the pdf of complex joint distributions from data (as opposed to the generator model).

So far I've found from Ian Goodfellow's paper, pg 29:

"For GANs, the generator is $P_{model}$, while for NCE and MLE, $P_{model}$ is part of the discriminator. Beyond this, the differences between the methods lie in the update strategy. GANs learn both players with gradient descent. MLE learns the discriminator using gradient descent, but has a heuristic update rule for the generator. Specifically, after each discriminator update step, MLE copies the density model learned inside the discriminator and converts it into a sampler to be used as the generator. NCE never updates the generator; it is just a fixed source of noise."

Does anyone know further details of this process he refers to? The closest thing I've found is this, which uses the well-known discriminator relationship

$D(x)=\frac{P_{data}}{P_{data} + P_{model}}$ and inverts to get $P'_{data}=P_{model}\frac{D(x)}{1-D(x)}$ where $P'_{data}$ is the updated $P_{model}$ for the next iteration. Then the generator on the next iteration is $P'_{data}$ via rejection sampling. By starting with a known pdf this has reliably converged in my experience.

The problem here is that the model continues to grow each iteration. It's not especially scale-able.

Does anyone know if Ian refers to a method that is trained more similarly to the GANs, with a "static" sized model in which the pdf can be extracted from the discriminator at the end of training, and the generator can be more like what we see in GANs (as opposed to rejection sampling)?


1 Answer 1


Found the answer in chapter 2 of another paper by Ian: ON DISTINGUISHABILITY CRITERIA FOR ESTIMATING GENERATIVE MODELS. It is based on the concept of noise-contrastive estimation and he calls it self-contrastive estimation as would be expected. In this case instead of static noise for the discriminator to work against, the samples are generated from the model as it evolves. The paper linked above [2011] by Italian researchers is amazingly simple and achieves similar goals via very similar methods, and was developed before Goodfellow's work on GANs, so he really should have given credit but he probably never knew about it.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.