Consider a coin $X\sim\operatorname{Bernoulli}(p)$. Its Fisher information is given by $J=\frac{1}{p(1-p)}$.

Now suppose we are in a Bayesian setting and our prior on $p$ is $\pi=\operatorname{Beta}(s,s)$ for some known $s$. This has mean value $\frac12$ and variance $\frac{1}{4+8s}$. For reference, Jeffrey's prior is $s=\frac12$ and a uniform prior is $s=1$.

The inequality of van Trees (aka the Bayesian Cramer-Rao bound) has a lower bound $1/(J_D+J_P)$ where $J_D=\int J d\pi(p)$ is the Fisher information integrated over the prior, and $J_P$ is the information of the prior. See page 84 of his textbook (sorry for the paywall), or equation (3) in this paper. In our example we get, using Mathematica, $$ J_D=4+\frac{2}{s-1}, $$ $$ J_P=4+8s+\frac{12}{s-2}. $$

The first integral, $J_D$, converges only for $s>1$, and the second integral, $J_P$, converges only for $s>2$.

To me, a uniform prior on a coin (i.e. $s=1$) seems perfectly reasonable. My question is why does the van Trees inequality not give a finite bound for $s\leq2$? Is it a limitation of the inquality, being rooted in the idea of likelihood curvature? Or is $s\leq2$ somehow a bad prior? I am, more than anything, looking for insight, if there is any to be had.

As an extra tidbit for comparison, it is easy to compute the MSE (risk) for the Bayes estimator, $$ r=\frac{s}{2(2s+1)^2}, $$ which holds for any choice of our prior, $s>0$. Both are plotted below, where van Trees' inequality indeed holds for $s>2$.

A plot of inverse Bayesian info and risk.

  • $\begingroup$ How do you define the information of a prior $\pi(p)$? $\endgroup$ – Xi'an Jan 8 '18 at 17:47
  • $\begingroup$ @Xi'an van Trees defines it as the expected value of $\left(\frac{d}{dp}\pi(p)\right)^2$, which results in a function of only $s$. $\endgroup$ – Ian Hincks Jan 8 '18 at 17:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.