# Information theory without normalization

I'd like to know if there is a way anyone knows of for doing information theory with unnormalized densities.

Specifically, I hav two log likelihoods $\phi(x), \psi(x)$ and so I can write:

$p(x) = \frac{\exp(\phi(x))}{Z(\phi)}, q(x) = \frac{\exp(\psi(x))}{Z(\psi)}$.

I would like to have a measure of discrepancy between these densities that is similar (maybe in spirit) to the KL divergence $KL(p || q)$ that does not require me to know $Z(\phi) ,Z(\psi)$, since I cannot integrate them analytically and monte carlo is too expensive.

I know this is a bit vague, any pointers would be appreciated.

$\textbf{edit:}$ let me stress that I am not necessarily looking for a cheap way to calculate KL divergence, but rather, a cheap way to estimate discrepancy between unnormalized probablities.

• You'd have to integrate at some point, me thinks. Dec 15, 2014 at 19:15
• I am facing a similar problem like yours. Do you have an answer? Mar 26, 2021 at 15:10

Just as unnormalized probabilities (likelihoods) can be compared but not turned into probabilities (without normalizing) — similarly, given log-likelihoods $\phi$ and $\psi$, you cannot calculate the KL divergence, but you can compare KL divergences.
For example, if you were trying to select a predictive unnormalized distribution $p$ out of $p_1, p_2, \dotsc$, given an observed unnormalized distribution $q$. Then you would want to choose $\text{arg min}_i D(q; p_i)$.
You can estimate this by sampling $x$s and taking a weighted average of $-\log p_i(x)$ weighted according $q(x)$. This requires no integration.