In the paper on Noise Contrastive Estimation, the authors define a parameterized density function $p_m^0\left(x;\alpha\right)$ to estimate the unnormalized PDF of the data, and then further define a function $p_m\left(x;\theta\right) = ln\left(p_m^0\left(x;\alpha\right)\right) + C$, where $C$ is the logarithm of the normalization constant.

Then, under the assumption a noise PDF $p_n\left(x\right)$ over the support of x is employed to generate contrastive samples, they go on to define a cost function matching typical logistic regression (cross entropy) under a nonlinear logistic regression model with the posterior probability $P(data;\theta) = h(x,\theta)$ where $h(x,\theta)$ is a logistic function with the argument of the exponential function being $ln\left(p_m(x;\theta)\right) - ln\left(p_n(x)\right)$

This is all fine and good -- and they go on to discuss associated theorems and show resulting comparisons to MLE and contrastive divergence, etc. HOWEVER, they only claim [several times] that the $C$ parameter will just automatically be optimized such that it normalizes $p_m^0\left(x;\alpha\right)$ to integrate to 1. Literally from the article "With our objective function, no such constraints are necessary. The maximizing pdf is found to have unit integral automatically. ".

I don't see a proof of this (and it's not obvious to me that it's true). The have a starred note for the following theorem that states "proofs are omitted due to a lack of space":

In Theorem 1 of their paper, they make this claim that "$J$" (the likelihood) attains a maximum when $p_m\left(x;\theta\right) = p_d$ (when the model equals the actual pdf of the data)

I'm not seeing how this is going to be true (i.e., that C will normalize $p_m^0\left(x;\alpha\right)$) as the cost function stands defined.

  • 1
    $\begingroup$ This was first proposed by Charlie Geyer in 1992 in an unpublished technical report. $\endgroup$
    – Xi'an
    Commented Nov 29, 2017 at 20:58
  • $\begingroup$ The proofs of the theorems can be found in the journal version of the paper. I'm also not very clear on why the normalization constant falls out of the optimization process, but I assume it's a consequence of theorem 1: if the estimated distribution $p_m$ converges to the true distribution $p_d$ then $p_m$ should be normalized as data distribution $p_d$ is normalized by definition. $\endgroup$ Commented Mar 20, 2020 at 10:39


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.