# Bayesian MMSE estimators from a transformation of the observations

Consider a random variable X whose value we want to estimate using a Bayesian MMSE estimator. Let $O_1(X)$ be a set of observations which depend on $X$ in some complex way (captured by $P(O_1|X)$) then the MMSE estimator is the conditional mean $\hat{X}_1=\mathbb{E}[X|O_1]$. Now consider another set of observation which is a possibly complex transformation of the first $O_2(O_1)$ and the corresponding MMSE estimator $\hat{X}_2=\mathbb{E}[X|O_2]$. Now it is obvious that if the mapping $O_2(O_1)$ is deterministic then $MSE[\hat{X}_1]\le MSE[\hat{X}_2]$ from the minimality of the MSE of $\hat{X}_1$.

My question is if this is true also when the mapping $O_2(O_1)$ is probabilistic, that is defined by a conditional distribution $P(O_2|O_1)$. Intuitively, it should be as any other stochasticity in the mapping just seems to introduce additional noise as it does not depend on $X$. But I wonder if one can show this explicitly.

EDITED: as the MMSE estimator is unbiased, indid $MSE[\hat{X}_i]=V[\hat{X}_i]$

## 1 Answer

This fact is called the data processing inequality. If the variables $X$, $O_1$ and $O_2$ form a Markov chain ($X \rightarrow O_1\rightarrow O_2$), then $Var [X|O_1] \leq Var[X|O_2]$, with equality iff $E[X|O_1] = E[X|O_2]$.

You can prove it by using the Markov property and the fact that conditioning decreases variance (which is itself the consequence of the law of total variance).

See Theorem 11 in "Functional Properties of Minimum Mean-Square Error and Mutual Information", Y. Wu, S. Verdú, IEEE Trans. Info. Theory, March 2012

http://www.princeton.edu/~verdu/reprints/WuVerIT2012f.pdf