Biased estimator obtained by optimal experiment design

Question

I am using a model-based approach to infer the parameters of a given system. Namely, I represent my system by a model $\mathcal{M}$ with parameters $\theta$. To estimate the true value of $\theta$, I record the output $\mathcal{D}$ of my system to a given input, and use the likelihood of the data $p(\mathcal{D}|\mathcal{M},\theta)$ to compute the posterior distribution of my parameters $p(\theta|\mathcal{D},\mathcal{M})$.

My goal is to maximize the information I can get about $\theta$, and hence to obtain a posterior distribution $p(\theta|\mathcal{D},\mathcal{M})$ as peaky as possible. I use Bayesian optimal experiment design to find the experimental protocol (i.e. the input to my system) which will maximize the information about $\theta$.

As explained in this article, the utility of a given experiment design can be defined

• Either as the gain in Shannon information about $\theta$, that is to say the difference between the entropy of my posterior distribution and the entropy of my prior distribution;
• Or as the Kullback-Leibler divergence between the prior and the posterior.

In each case, the optimal experiment design is the one that maximized the sharpness of my posterior $p(\theta|\mathcal{D},\mathcal{M})$. But, by focusing only on minimizing the variance of my estimator, do I risk to maximize its bias ? With optimal experiment design techniques, I obtain a sharp and very informative posterior, but I have no guarantee that it will be close to the true value of $\theta$. Is it possible to have degenerate cases in which estimators $\hat{\theta}$ obtained by optimal design have a low variance but a very high bias ?

The literature seems to be only focused on minimizing its variance; any reference would be much welcome.

Answer 1

A theorem, demonstrated by Liam Paninski in 2005, provides an answer to my question. Namely, it provides the conditions under which an Optimal Experiment Design (OED) will not only reduce the variance of the posterior distribution, but also converge towards an unbiased estimate.

Under some conditions, which are described in the paper (namely, that the prior over $\theta$ is not degenerate, that the space of parameters is compact, and that the log likelihood of the model is sufficiently smooth in $\theta$):

• The posterior distribution $p(\theta|\mathcal{D},\mathcal{M})$ will converge towards the ground truth value of $\theta$ (i.e. the solution will be unbiased)
• and its variance will decrease as $\frac{1}{T}$ ($T$ being the number of recorded data points), the exact rate depending on the Fisher Information.

In short, under some conditions, the posterior distribution will be asymptotically normal, with a mean converging to the ground truth value of $\theta$ and a variance decreasing as more data points are acquired.

This result holds for non-optimal experiment designs (i.e. if the stimuli are chosen randomly), but the rate of convergence will be higher for optimal designs.

Finally, I am not sure if this result still holds for non-i.i.d. models (that is to say if data points are correlated). In that case, the information contained in the data might saturate.

Paninski, Liam. "Asymptotic theory of information-theoretic experimental design." Neural Computation 17.7 (2005): 1480-1507.

Answer 2

The Wikipedia article you cite does not seem good or clear to me, so find some other source. Let us look first at the basics of Bayesian experimental design. We want to choose a design (maybe choose which observations to take from some experimental region, with regression-like models.) Denote the design by $\xi$, the response variable by $y$, with a distribution (density) governed by some unknown parameter $\theta$, density is $p(y \mid \theta)=p_\xi(y \mid \theta)$. The model function have subscript $\xi$ because choosing the design is choosing a distribution for $y$. The prior distribution is $p(\theta)$, without subscript because the prior information do not depend on the design.

If the goal of the experiment is inference on $\theta$, we need a criterion function reflecting that. If, after doing the experiment, the posterior distribution is equal to (or close to) the prior, we didn't learn much! So we want an experiment $\xi$ that we can expect to give data that surprises us and so changes our opinion on $\theta$, see Statistical interpretation of Maximum Entropy Distribution. So a natural measure is the Kullback-Leibler divergence $$ \tag{*}\label{*} \newcommand{\KL}[2]{KL\left(#1 || #2\right)} \newcommand{\KLint}[3]{\int #1 \log\left(\frac{#1}{#2}\right)\;d #3} \KL{p_\xi(\theta | y)}{p(\theta)} = \KLint{p_\xi(\theta | y)}{p(\theta)}{\theta} $$ This can never be negative, and will be zero only if the posterior equals the prior, in which case we have learnt nothing, out state of knowledge is exactly as before doing the experiment. But, this cannot be used directly, it depends on $y$, which is unknown when we are planning the experiment. So, we need its expectation, relative to the marginal distribution of $y$, $m(y)$. From Bayes theorem $p(\theta | y)= p(y | \theta)p(\theta) / m(y)$ we can write $$ m(y)=\frac{p(y | \theta)p(\theta)}{p(\theta | y)} $$ Calculating the expectation of $\eqref{*}$ over $m(y)$: (leaving out subscript $\xi$) $$ \int m(y) \cdot \KLint{p_\xi(\theta | y)}{p(\theta)}{\theta} \; dy = \\ \int \int \frac{p(y | \theta) p(\theta)}{p(\theta | y)} \cdot p(\theta | y) \log\left( \frac{p(\theta | y)}{p(\theta)} \right) \; d\theta \; dy = \\ \int\int p(y,\theta) \log\left( \frac{p(\theta | y)}{p(\theta)} \right) \; d\theta \; dy = \\ \int\int p(y,\theta) \log p(\theta | y) \;d\theta\; dy - \int\int p(y,\theta) \log p(\theta)\;d\theta\;dy $$ where we can see that the last term do not depend on the design, so we can omit it and use the first term to define the utility function we want to maximize: $$ U(\xi) = \int\int p(y,\theta) \log p(\theta | y) \;d\theta\; dy $$ which we can see is the expected Shannon (or differential) entropy of the posterior. As this is calculated before observing $y$, it is sometimes called a preposterior expectation. But for understanding it must be better to look at the earlier expressions. What is maximized here is emphatically not

sharpness of my posterior

,it is the surprise value of the experiment!