As the author Christian Robert asks about reference measures that are absolutely continuous to one another, and from what I can gather this just means they have the same null set? But would that not be true for any continuous distribution? Since from what I can gather, Lebesgue is zero over points, non zero over intervals or such. Similar to continuous distributions which are zero at any given point.

Some people pointed out that not all distributions have this , for example the cantor distribution, because it does not have a density.

So the author pointed out, it is really about having a density not just a distribution.

I am trying to work through some of the examples and problems. One talks about that if we have absolutely continuous measures, then we can construct example where corresponds maximum entropy distributions are the same.

So this leads me to think, that the reference measure does not uniquely determine the maximum distribution? Ie, if they are absolutely continuous, then it may be ( or will be?), the same. And I am wondering why this would be true.

I get the formula, but wouldn't they normalise differently? How would it end up being the same distribution?

An example given is Lebesgue measure and standard normal. Since standard normal has a density, I think this is an example of absolute continuity. I am thinking they may give the same when the two measure have no distance between them? with known expected value and variance as constraints, we could find the Lagrange multipliers. But we will still have $\frac{1}{\sqrt{2\pi}}exp(\frac{-\theta^{2}}{2})$ But since its a density, wont it simply integrate to 1 as well with the normal, so even if both can be normalised I have trouble seeing how it gives the same max entropy. That is, I am wanting to see how different reference priors can lead to the same max entropy.

More my thoughts:

So, for a standard normal reference prior, since we know mean and variance I assume we could write it as

$\pi^{*}(\theta)$ $\alpha$ $\exp(\lambda_{1}\theta+\lambda_{2}\theta^{2})\exp(\frac{-\theta^{2}}{2})$

Which I guess has the same form with a factor, would it still give the same normal though?


And as seen previously , for Lebesgue measure with known mean and variance,

$\pi^{*}(\theta)$ $\alpha$ $\exp(\lambda_{1}\theta+\lambda_{2}\theta^{2})$

So both are normalise, but I have trouble seeing how they will correspond to same distribution. The second one is a normal with mean 0 and variance $\sigma^{2}$ unless the factor in the first just would change the Lagrange, and leave with same normal

  • $\begingroup$ @Xi'an . My apologies. I will note that and do my best to be clear in any further questions. $\endgroup$ – Learning Nov 15 '18 at 21:04

The point I am making in my book and in the previous question is not original but worth repeating. For a dominating measure $\text{d}\mu$, the maximum entropy prior is defined as maximising$$\int_\mathcal{X}\log p(x) \text{d}\mu(x)$$in the density $p$ under a set of constraints$$\mathbb{E}_p[g_k(X)]=\int_\mathcal{X} g_k(x) p(x) \text{d}\mu(x)=\omega_k\qquad k=1,\ldots,K$$including$$\int_\mathcal{X} p(x) \text{d}\mu(x)=1$$ If there is such a probability distribution, then it is given by $$p(x) = \exp\{\lambda_1 g_1(x)+\cdots+\lambda_K g_K(x)\}\Big/ \int_\mathcal{X} \exp\{\lambda_1 g_1(x)+\cdots+\lambda_K g_K(x)\} \text{d}\mu(x)$$whose notation is simplified into $$p(x) \propto \exp\{\lambda_1 g_1(x)+\cdots+\lambda_K g_K(x)\}$$ to avoid dragging the denominator into every equation.

This being a density against $\text{d}\mu$, changing the measure changes the resulting density and the resulting distribution. Except in your example where you unfortunately chose to move from the measure $\text{d}\mu$ to the measure $e^{-x^2/2}\text{d}\mu$ and a constraint involving $g_2(x)=x^2$: in this case the maximum entropy prior distribution is the same for both measures. Indeed, they both have the same dominating measure $\text{d}\mu$ and the same defining equations: \begin{align*} p_1(x)\text{d}\mu &\propto\exp\{\lambda_1 x+\lambda_2 x^2\}\text{d}\mu\\ p_2(x)e^{-x²/2}\text{d}\mu &\propto\exp\{\lambda_1^\prime x+\lambda_2^\prime x^2\}e^{-x²/2}\text{d}\mu\\ &=\exp\{\lambda_1^\prime x+(\lambda_2^\prime -1/2)x^2\}\text{d}\mu\\ \end{align*} Hence$$\lambda_1=\lambda_1^\prime\qquad\lambda_2=\lambda_2^\prime -1/2$$and the probability distributions are the same distributions even though $p_1(\cdot)\ne p_2(\cdot)$.

If, for this example of the first two moments [meaning $g_1(x)=x$ and $g_2(x)=x^2$], I choose instead to replace the Lebesgue measure $\text{d}x$ with the new measure $|x|^{-1/2}\text{d}x$, which is absolutely continuous wrt the Lebesgue measure, the corresponding maximum entropy prior distributions will differ since their density wrt the Lebesgue measure will be \begin{align*} p_1(x)\text{d}\mu &\propto\exp\{\lambda_1 x+\lambda_2 x^2\}\text{d}\mu\\ p_2(x)|x|^{-1/2}\text{d}\mu &\propto\exp\{\lambda_1^\prime x+\lambda_2^\prime x^2\}|x|^{-1/2}\text{d}\mu\\ \end{align*} For $\omega_1=0$ and $\omega_2=1$, the first solution $p_1(\cdot)$ is the density of the standard Normal distribution, while the second one satisfies $$\int_{-\infty}^\infty \frac{x}{\sqrt{|x|}} \exp\{\lambda_1'x+\lambda_2'x^2\}\text{d}x=0$$ which is achieved when taking $\lambda_1'=0$ and implies that $$p_2(x)\propto \exp\{ \lambda_2^\prime x^2\}|x|^{-1/2}$$The normalising constant is thus given by $$\int_{-\infty}^\infty \exp\{ \lambda_2^\prime x^2\}|x|^{-1/2}\text{d}x \overbrace{=}^{y=x^2}2\int_0^\infty \exp\{ \lambda_2^\prime y\}|y|^{-1/4}\underbrace{y^{-1/2}\text{d}y/2}_\text{Jacobian}=\Gamma(1/4)(-\lambda_2^\prime)^{-1/4}$$ The second parameter is determined by the constraint $$\int_{-\infty}^\infty x^2 \exp\{ \lambda_2^\prime x^2\}|x|^{-1/2}\text{d}x =1 \times \Gamma(1/4)(-\lambda_2^\prime)^{-1/4}$$ or $$2\int_0^\infty \exp\{ \lambda_2^\prime y\}|y|^{1+1/4-1}\text{d}y/2= \Gamma(5/4)(-\lambda_2^\prime)^{-5/4}= \Gamma(1/4)(-\lambda_2^\prime)^{-1/4}$$which simplifies as $$\frac{1}{4}(-\lambda_2^\prime)^{-1}=1\quad\text{i.e.}\quad\lambda_2=-1/4$$ These two distributions share the same first two moments but are clearly not identical distributions $$p_1(x)=\dfrac{\exp\{-x^2/2\}}{\sqrt{2\pi}}\qquad p_2(x)=\dfrac{\exp\{-x^2/4\}}{\Gamma(1/4)(1/4)^{-1/4}}$$

enter image description here

  • 1
    $\begingroup$ (+1) Nice answer, Christian. Apart from something related on p. 376 on Jaynes' book, I haven't seem much discussion about these points on the literature. $\endgroup$ – Zen Nov 20 '18 at 21:05
  • $\begingroup$ Thanks! I am uncertain the issue can be "solved" but it is important to be aware of this often implicit choice of a dominating measure. $\endgroup$ – Xi'an Nov 20 '18 at 21:07

Not the answer you're looking for? Browse other questions tagged or ask your own question.