# Biased estimator obtained by optimal experiment design

I am using a model-based approach to infer the parameters of a given system. Namely, I represent my system by a model $$\mathcal{M}$$ with parameters $$\theta$$. To estimate the true value of $$\theta$$, I record the output $$\mathcal{D}$$ of my system to a given input, and use the likelihood of the data $$p(\mathcal{D}|\mathcal{M},\theta)$$ to compute the posterior distribution of my parameters $$p(\theta|\mathcal{D},\mathcal{M})$$.

My goal is to maximize the information I can get about $$\theta$$, and hence to obtain a posterior distribution $$p(\theta|\mathcal{D},\mathcal{M})$$ as peaky as possible. I use Bayesian optimal experiment design to find the experimental protocol (i.e. the input to my system) which will maximize the information about $$\theta$$.

As explained in this article, the utility of a given experiment design can be defined

• Either as the gain in Shannon information about $$\theta$$, that is to say the difference between the entropy of my posterior distribution and the entropy of my prior distribution;
• Or as the Kullback-Leibler divergence between the prior and the posterior.

In each case, the optimal experiment design is the one that maximized the sharpness of my posterior $$p(\theta|\mathcal{D},\mathcal{M})$$. But, by focusing only on minimizing the variance of my estimator, do I risk to maximize its bias ? With optimal experiment design techniques, I obtain a sharp and very informative posterior, but I have no guarantee that it will be close to the true value of $$\theta$$. Is it possible to have degenerate cases in which estimators $$\hat{\theta}$$ obtained by optimal design have a low variance but a very high bias ?

The literature seems to be only focused on minimizing its variance; any reference would be much welcome.

• I'm not sure that "maximizing sharpness of the posterior" is the correct conclusion about a Bayes optimal design. Bayes optimal designs minimize the MSE, which is variance and squared bias. In this way you can find an estimator with very low variance, but the bias cannot be arbitrarily large. Apr 7, 2020 at 15:11
• Well, that's precisely what I'm not sure about. All the papers I found on OED where about maximizing the gain in information (i.e. reducing the entropy) between the posterior and prior distributions. Do you have any reference where the MSE is mentioned ? Apr 7, 2020 at 15:17
• You have conflated three things - maximizing the information gain, obtaining a posterior distribution that is as "peaky" as possible, and minimizing the posterior variance - that are actually not the same, and additionally are throwing bias into the mix. The importance of understanding the differences between these concepts makes this an excellent question (+1)! Apr 7, 2020 at 15:38
• To the best of my knowledge, all the papers I could find considered these three notions as somewhat equivalent, and did not seem to consider bias minimization in the utility function. Are there papers clarifying this matter ? I would be equally happy if someone confirms that it is not the case, because it would mean that we still have plenty to discover ^^ I have especially studied this reference :Huan, X., & Marzouk, Y. M. (2013). Simulation-based optimal Bayesian experimental design for nonlinear systems. Journal of Computational Physics, 232(1), 288-317. Apr 7, 2020 at 15:51
• Could you please tell us some details of the model/system you want to optimize? Apr 7, 2020 at 18:10

A theorem, demonstrated by Liam Paninski in 2005, provides an answer to my question. Namely, it provides the conditions under which an Optimal Experiment Design (OED) will not only reduce the variance of the posterior distribution, but also converge towards an unbiased estimate.

Under some conditions, which are described in the paper (namely, that the prior over $$\theta$$ is not degenerate, that the space of parameters is compact, and that the log likelihood of the model is sufficiently smooth in $$\theta$$):

• The posterior distribution $$p(\theta|\mathcal{D},\mathcal{M})$$ will converge towards the ground truth value of $$\theta$$ (i.e. the solution will be unbiased)
• and its variance will decrease as $$\frac{1}{T}$$ ($$T$$ being the number of recorded data points), the exact rate depending on the Fisher Information.

In short, under some conditions, the posterior distribution will be asymptotically normal, with a mean converging to the ground truth value of $$\theta$$ and a variance decreasing as more data points are acquired.

This result holds for non-optimal experiment designs (i.e. if the stimuli are chosen randomly), but the rate of convergence will be higher for optimal designs.

Finally, I am not sure if this result still holds for non-i.i.d. models (that is to say if data points are correlated). In that case, the information contained in the data might saturate.

Paninski, Liam. "Asymptotic theory of information-theoretic experimental design." Neural Computation 17.7 (2005): 1480-1507.

The Wikipedia article you cite does not seem good or clear to me, so find some other source. Let us look first at the basics of Bayesian experimental design. We want to choose a design (maybe choose which observations to take from some experimental region, with regression-like models.) Denote the design by $$\xi$$, the response variable by $$y$$, with a distribution (density) governed by some unknown parameter $$\theta$$, density is $$p(y \mid \theta)=p_\xi(y \mid \theta)$$. The model function have subscript $$\xi$$ because choosing the design is choosing a distribution for $$y$$. The prior distribution is $$p(\theta)$$, without subscript because the prior information do not depend on the design.

If the goal of the experiment is inference on $$\theta$$, we need a criterion function reflecting that. If, after doing the experiment, the posterior distribution is equal to (or close to) the prior, we didn't learn much! So we want an experiment $$\xi$$ that we can expect to give data that surprises us and so changes our opinion on $$\theta$$, see Statistical interpretation of Maximum Entropy Distribution. So a natural measure is the Kullback-Leibler divergence $$\tag{*}\label{*} \newcommand{\KL}[2]{KL\left(#1 || #2\right)} \newcommand{\KLint}[3]{\int #1 \log\left(\frac{#1}{#2}\right)\;d #3} \KL{p_\xi(\theta | y)}{p(\theta)} = \KLint{p_\xi(\theta | y)}{p(\theta)}{\theta}$$ This can never be negative, and will be zero only if the posterior equals the prior, in which case we have learnt nothing, out state of knowledge is exactly as before doing the experiment. But, this cannot be used directly, it depends on $$y$$, which is unknown when we are planning the experiment. So, we need its expectation, relative to the marginal distribution of $$y$$, $$m(y)$$. From Bayes theorem $$p(\theta | y)= p(y | \theta)p(\theta) / m(y)$$ we can write $$m(y)=\frac{p(y | \theta)p(\theta)}{p(\theta | y)}$$ Calculating the expectation of $$\eqref{*}$$ over $$m(y)$$: (leaving out subscript $$\xi$$) $$\int m(y) \cdot \KLint{p_\xi(\theta | y)}{p(\theta)}{\theta} \; dy = \\ \int \int \frac{p(y | \theta) p(\theta)}{p(\theta | y)} \cdot p(\theta | y) \log\left( \frac{p(\theta | y)}{p(\theta)} \right) \; d\theta \; dy = \\ \int\int p(y,\theta) \log\left( \frac{p(\theta | y)}{p(\theta)} \right) \; d\theta \; dy = \\ \int\int p(y,\theta) \log p(\theta | y) \;d\theta\; dy - \int\int p(y,\theta) \log p(\theta)\;d\theta\;dy$$ where we can see that the last term do not depend on the design, so we can omit it and use the first term to define the utility function we want to maximize: $$U(\xi) = \int\int p(y,\theta) \log p(\theta | y) \;d\theta\; dy$$ which we can see is the expected Shannon (or differential) entropy of the posterior. As this is calculated before observing $$y$$, it is sometimes called a preposterior expectation. But for understanding it must be better to look at the earlier expressions. What is maximized here is emphatically not

sharpness of my posterior

,it is the surprise value of the experiment!

• That's indeed how bayesian optimal experiment design is implemented in the literature : the utility function of a given protocol is the expected gain in Shannon information (or, equivalently, the expected KLD between the prior and the posterior). But that's not the point of my question. My point is that the bias of the estimator is not considered in this definition. An experiment might be informative, but lead to a posterior distribution of $\theta$ which mode is far away from the true value of $\theta$. Apr 9, 2020 at 9:27
• Frequentist optimal design criteria do not consider bias either en.wikipedia.org/wiki/…. Bias may not seem to be considered in the definition of the design because if it was known a priori, it would have been encoded in the prior distribution, and hence affect the design through the prior.
– JTH
Apr 9, 2020 at 13:43
• Another thought: the main weakness of optimal designs is that they assume $p(y|\theta)$ is correctly specified. This is true for Bayesian and Frequentist designs. If this is not the case, optimal designs are ... sub-optimal!
– JTH
Apr 9, 2020 at 18:28
• In Bayesian optimal design, what is optimized is the design, not any estimator! One is assuming that Bayesian analysis will be used, so any problems you have with the resulting estimators should be directed at the bayesian analysis, not the design. Apr 9, 2020 at 22:08
• @JTH you are absolutely right. Here, I'm only mentioning optimal design for inferring the parameters of a single (fixed model). But one might also want to perform optimal design for model selection, i.e. to maximize the discriminability between competing models.In that case, the utility of an experimental protocol is usually defined as the KLD between the models, like in this reference : Balietti, S., Klein, B., & Riedl, C. (2018). Fast Model-Selection through Adapting Design of Experiments Maximizing Information Gain. arXiv preprint arXiv:1807.07024. Apr 16, 2020 at 11:43