Consider a random variable X whose value we want to estimate using a Bayesian MMSE estimator. Let $O_1(X)$ be a set of observations which depend on $X$ in some complex way (captured by $P(O_1|X)$) then the MMSE estimator is the conditional mean $\hat{X}_1=\mathbb{E}[X|O_1]$. Now consider another set of observation which is a possibly complex transformation of the first $O_2(O_1)$ and the corresponding MMSE estimator $\hat{X}_2=\mathbb{E}[X|O_2]$. Now it is obvious that if the mapping $O_2(O_1)$ is deterministic then $MSE[\hat{X}_1]\le MSE[\hat{X}_2]$ from the minimality of the MSE of $\hat{X}_1$.

My question is if this is true also when the mapping $O_2(O_1)$ is probabilistic, that is defined by a conditional distribution $P(O_2|O_1)$. Intuitively, it should be as any other stochasticity in the mapping just seems to introduce additional noise as it does not depend on $X$. But I wonder if one can show this explicitly.

EDITED: as the MMSE estimator is unbiased, indid $MSE[\hat{X}_i]=V[\hat{X}_i]$


This fact is called the data processing inequality. If the variables $X$, $O_1$ and $O_2$ form a Markov chain ($X \rightarrow O_1\rightarrow O_2$), then $Var [X|O_1] \leq Var[X|O_2] $, with equality iff $E[X|O_1] = E[X|O_2] $.

You can prove it by using the Markov property and the fact that conditioning decreases variance (which is itself the consequence of the law of total variance).

See Theorem 11 in "Functional Properties of Minimum Mean-Square Error and Mutual Information", Y. Wu, S. Verdú, IEEE Trans. Info. Theory, March 2012



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.