# Cramer-Rao type bound for Information Gain

I am interested in the Bayes risk of some distribution $\pi$ $$r(\pi) = \mathbb{E}_{\pi(x)}[ \mathbb{E}_{\Pr(y|d,x)}[L(x,\hat x(y|d))]],$$ where $L$ is some loss function and $\hat x$ is the posterior mean (here: Bayes estimator). Here $d$ is some experimental design parameter.

If $L(x,\hat x) = |x-\hat x|^2$ (squared error), then the Posterior Cramer-Rao bound gives a lower bound on the Bayes risk. The Bayes risk is also identical to the expected posterior variance. So to optimally design experiments, I can use negative expected variance as a utility function to maximize.

Now I would like to reverse this logic to find the effective loss function and a lower bound for information gain as an alternative utility function. That is, suppose the utility function I use is information gain. Then,

1. What is the loss function which this is optimizing?
2. What is a lower bound on the Bayes risk for this loss function?
• Generalization bounds are in this spirit: videolectures.net/mlss05us_langford_gb – Tristan Apr 24 '12 at 7:50
• @Tristan - Thanks. Interesting and certainly seems relevant. Although, I'm not sure yet whether it answers my specific question--maybe I just need a better question, though. – Chris Ferrie May 4 '12 at 11:59