Some Questions about reference measures and maximum entropy priors (from The Bayesian Choice)

Question

I am relatively new to statistics and Bayesian theory but I am trying to understand it by working through a few books. There are some things I am confused about. ( As I believe my analysis and such is not up to the ideal level for some of the material)

Firstly, the author writes, "If some characteristics of the prior distribution (moments, quantiles,etc) are known, assuming that they can be written as prior expectation, $E^{\pi}[g_{k}(\theta)]=w_{k} , k=(1,2,...K)"$

I am wondering if this is just another way of talking about moments? For example for the n'th moment $g_{k}(\theta)=\theta^{n}$

I am having trouble understanding what is meant by reference priors, and especially what is meant by saying "if the reference measure $\pi_{0}$ is the Lebesgue measure on $\mathbb{R}$"

For example, the author has an example where he writes ( example 3.2.4 )

"Consider $\theta$ , a real parameter such that $E^{\pi}[\theta]=\mu$. If the reference measure $\pi_{0}$ is the Lebesgue measure on $\mathbb{R}$, the maximum entropy prior satisfies $\pi^{*}(\theta)$ $\alpha$ $exp(\lambda \theta)$ and cannot be normalised into a probability distribution, On the contrary if in addition it is known that $var(\theta)=\sigma^{2}$, the corresponding maximum entropy prior is $\pi^{*}(\theta)$ $\alpha$ $exp(\theta \lambda_{1}+\theta^{2}\lambda_{2})$, ie the normal distribution.."

So few questions. One is what I asked above about Lebesgue measure. Second about the characteristic thing. How do we know to include a $\theta^{2}$ in the final form? I get that we know it from the definition of variance of $E[\theta^{2}]-(E[\theta])^{2}$ and we have the second term. But how do we know that is the g term we use? Or is that just that we know it so we can use it? Because I am not sure if we know that the g is moment, quantile, etc.

Finally, on the same page the author writes that when "characteristics" are related with quantiles, that we will not be able to derive a distribution. Why? Just because quantiles are not enough? For example he has "when characteristics are related with .., $I_{[b_{k},\infty)}(\theta)$. I.e just an indicator of if $\theta$ lies in that interval or does not.But is that the reason that the maximum entropy wont exist? Does this mean we would know the associated probability of being in that interval and we would know $b_{k}$. I get that the expected value of it would be the probability we are in that interval. But is that enough to conclude we can not define a distribution in such a way? What if one of the intervals told us that the probability of being in such was 1. I guess because we are on $\mathbb{R}$ this couldn't happen for a distinct value?

Someone had mentioned "any minimal, summarizing sequence of statistics is recursively computable" as a reason , but I don't understand that.

Again, I apologise if there are obvious things I am missing. I am trying to catch up on my analysis and such as well, so hopefully that will make it easier for me in the future.

Thank you all for your time and patience.

Answer 1

Let the author try to answer these questions about maximum entropy priors:

"If some characteristics of the prior distribution (moments, quantiles,etc) are known, assuming that they can be written as prior expectation, $\mathbb{E}^π[g_k(θ)]=w_k,\, k=(1,2,...K)$"

By this sentence, I mean that some functions have known expectations under the prior, i.e., that while our prior beliefs or prior information are not enough to come up with a complete prior probability distribution, we can make informed guesses about the expectation of some functions of $\theta$. Moments are mentioned in the parenthesis, so this is a special case, but there is no reason to restrict the functions to standard moments. The book contains an example take from a capture-recapture experiment where the prior information is expressed in terms of the mean and a 95% confidence interval of the prior distribution. There is a large degree of arbitrariness in the selection of these functions $g_k$, with each choice leading to a different maximum entropy prior. (The same criticism applies to empirical likelihood estimation and to the various forms of the method of moments.)

trouble understanding what is meant by reference priors, and especially what is meant by saying "if the reference measure $π_0$ is the Lebesgue measure on ℝ"

This is admittedly a more advanced point. When maximising the entropy to construct the prior, the criterion function is $$\mathfrak E(\pi)=-\int_\Theta \log\{\pi(\theta)\}\,\mathrm dx$$ But this choice of criterion depends on the choice of the measure $\mathrm dx$ in the integral.

One can argue about a special case of measure, the right Haar measure, in cases when invariance or equivariance is present and relevant, but this is a rare occurrence. In general, there is no default choice for the measure $\mathrm dx$ and each different choice leads to a different maximum entropy prior, although all share the same moments (i.e., expectations of the $g_k$'s). The Lebesgue measure on $\Theta$ is a default choice, but this is not coherent when considering a new parametrisation of the model $f_X(\cdot|\theta)$, which is the standard criticism against Laplace's uniform priors. This also relates to the general difficulty (I would say impossibility) of Bayesian analysis to define the most noninformative prior. This is why I prefer to call the dominating measure a reference measure, in acknowledgement of Bernardo's (1979) theory of non-informative priors.

when "characteristics" are related with quantiles, that we will not be able to derive a distribution. Why?

The reason why is that the resulting maximum entropy prior cannot be normalised into a probability density over a unbounded parameter space, $\mathbb R$ say. If $g_k(\theta)=\mathbb{I}_{(b_k,\infty)}(\theta)$, the density of the maximum entropy prior against the Lebesgue measure would be $$\pi^\star(\theta) \propto \exp \sum_{k=1}^K \lambda_k \mathbb{I}_{(b_k,\infty)}(\theta)$$where the $\lambda_k$'s are determined by the coverage constraints associated with the quantiles, except that the function $$\exp \sum_{k=1}^K \lambda_k \mathbb{I}_{(b_k,\infty)}(\theta)$$integrates to infinity over the real line. There cannot be a probability density proportional to this function. (This is similar to the other example you quote of $\mathbb E[\theta]$ being insufficient to set a maximum entropy prior.)