Long-time listener, first-time caller. I'm reading the Wikipedia pages on exponential families and maximum entropy probability distributions, and trying to wrap my head round the role of the base measure in an exponential family (to steal a phrase from the title of this question.) Specifically, I'm trying to fit the entropy-maximizing property of an exponential family into my existing (spotty) understanding of KL divergence and maximum entropy.

To fix notation, let an exponential family of distributions be given by $$ p(x|\theta) = h(x) \exp (\eta(\theta) \cdot T(x) - A(\eta) ) $$

where $A$ is the log-partition function, $\eta$ is the natural parameter, $T$ is the sufficient statistic, and $h$ is the base measure. Let $h$ and $p$ have common support $\Omega$. It's not hard to show that $p(\cdot | \theta)$ has maximum entropy relative to $h$: that is, $p(\cdot | \theta)$ maximizes the KL-divergence or relative entropy $$ S[p | h] = \int_{\Omega} p(x) \log \frac{h(x)}{p(x)} \, dx, $$ subject to a constraint on the sufficient statistic. The KL divergence also has an interpretation as the amount of information gained in a Bayesian update from $h$ to $p$, up to (I think) an additive constant that comes from normalizing $h$. In this case, though, since we're trying to maximize the entropy of $p$ relative to $h$, it seems like the interpretation is closer to, "How much of the information in $h$ can we lose (rather than gain)?"

My question is how far this analogy goes. For example, in the binomial distribution with $n$ trials, the base measure is $h(x) = \binom{n}{x}$ for $x = 0, \dots, n$. Is there any value in taking $\binom{n}{x}$ as a "starting measure", then obtaining the binomial($n,p$) distribution by requiring maximum entropy subject to $np$ expected successes?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.