I'd like to know if there is a way anyone knows of for doing information theory with unnormalized densities.

Specifically, I hav two log likelihoods $\phi(x), \psi(x)$ and so I can write:

$p(x) = \frac{\exp(\phi(x))}{Z(\phi)}, q(x) = \frac{\exp(\psi(x))}{Z(\psi)}$.

I would like to have a measure of discrepancy between these densities that is similar (maybe in spirit) to the KL divergence $KL(p || q)$ that does not require me to know $Z(\phi) ,Z(\psi)$, since I cannot integrate them analytically and monte carlo is too expensive.

I know this is a bit vague, any pointers would be appreciated.

$\textbf{edit:}$ let me stress that I am not necessarily looking for a cheap way to calculate KL divergence, but rather, a cheap way to estimate discrepancy between unnormalized probablities.

  • $\begingroup$ You'd have to integrate at some point, me thinks. $\endgroup$
    – Aksakal
    Dec 15, 2014 at 19:15
  • $\begingroup$ I am facing a similar problem like yours. Do you have an answer? $\endgroup$
    – heroxbd
    Mar 26, 2021 at 15:10

1 Answer 1


Just as unnormalized probabilities (likelihoods) can be compared but not turned into probabilities (without normalizing) — similarly, given log-likelihoods $\phi$ and $\psi$, you cannot calculate the KL divergence, but you can compare KL divergences.

For example, if you were trying to select a predictive unnormalized distribution $p$ out of $p_1, p_2, \dotsc$, given an observed unnormalized distribution $q$. Then you would want to choose $\text{arg min}_i D(q; p_i)$.

You can estimate this by sampling $x$s and taking a weighted average of $-\log p_i(x)$ weighted according $q(x)$. This requires no integration.

  • $\begingroup$ Thanks for the input. Sampling according to q is just like calculating a normalization constant - it does the job via monte carlo (importance sampling). The computational difficulty I am trying to overcome is exactly that - avoiding MC and\or calculating normalization constans. $\endgroup$
    – Yair Daon
    Dec 30, 2014 at 14:46
  • $\begingroup$ @YairDaon: I don't know if it's "just like calculating a normalization constant". Calculating a normalization constant can be really expensive. Sampling costs you as many samples as you take. If you don't take enough samples, you pay for it in inaccuracy. Seems fair to me. Anyway, you won't be able to do better since likelihoods are scale-invariant and so any distance between likelihoods has to respect that, which is impossible without some kind of normalization (like importance sampling). $\endgroup$
    – Neil G
    Dec 30, 2014 at 15:05
  • $\begingroup$ I agree that sampling costs as many samples as you want. I said these two concepts are the same since in order to be able to make good decisions based on these samples, you would need reasonable accuracy, which might be costly. If you have any idea that does not require sampling and\or calculating normalization constants, I would be extremely happy to hear. $\endgroup$
    – Yair Daon
    Dec 30, 2014 at 15:13
  • $\begingroup$ @YairDaon: I would also be interested in that since I don't think it's possible :) $\endgroup$
    – Neil G
    Dec 30, 2014 at 16:10
  • $\begingroup$ It seems your pessimism is in place :( $\endgroup$
    – Yair Daon
    Dec 30, 2014 at 19:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.