Let $X,Z_1,Z_2$ be 3 mutually independent RV's, with $Z_1, Z_2$ assuming $N(0,1)$ distribution. $X$ is constrained to have unit 2nd moment, i.e. $E[X^2] =1$, but may take arbitrary distribution. The question is: What's the maximum value of the following expectation of the conditional variance?

$$E \left[\operatorname{Var}(X+Z_1 \mid X+Z_2) \right]$$

A few remarks:

1. Recall that $$E \left[\operatorname{Var}(X+Z_1 \mid X+Z_2) \right] = E \left[(X+Z_1)^2 \right]-E \left[E[X+Z_1 \mid X+Z_2]^2 \right]$$

where $E[(X+Z_1)^2]$ is obviously equal to $E[X^2]+E[Z_1^2]=2$. So an equivalent question is: What's the minimum value of $E[E[X \mid X+Z_2]^2]$? (Note that $E[X+Z_1 \mid X+Z_2]=E[X \mid X+Z_2]+E[Z_1 \mid X+Z_2]$ and the 2nd term is zero.)

2. Note also that $E[X \mid X+Z_2]$ is the MMSE estimate of $X$ given $X+Z_2$, and the estimation error is $E[\operatorname{Var}(X\mid X+Z_2)]=E[X^2]-E[E[X\mid X+Z_2]^2]$. So effectively we are asking what distribution of X would result in the worst MMSE estimation error?

3. A few simple examples may shed some light on this : (1) If $X \sim N(0,1)$, then $X+Z_1$ and $X+Z_2$ are jointly Gaussian. It's known that $$\operatorname{Var}(X+Z_1\mid X+Z_2) = (1-\rho^2)Var(X+Z_1)=3/2,$$ $$E[X\mid X+Z_2]=(X+Z_2)/2,$$ and hence $E[E[X\mid X+Z_2]^2]=1/2.$
(2) If $X$ is a constant, say $1$ or $-1$, then obviously $$\operatorname{Var}(X+Z_1\mid X+Z_2)=\operatorname{Var}(X+Z_1)=1,$$ $$E[X\mid X+Z_2]=1.$$ So clearly $E[E[X\mid X+Z_2]^2]$ isn't constant, and I wonder if it is minimized by Gaussian distribution of $X$?

  • $\begingroup$ How you get this?$ E[Var(X+Z_1|X+Z_2)] = E[(X+Z_1)^2]-E[E[X+Z_1|X+Z_2]^2]$ $\endgroup$ – Deep North Aug 4 '15 at 3:42
  • $\begingroup$ $Var(U|V)=E[U^2|V]-E[U|V]^2$, and take the expectation on both sides. Note also that $E[E[U^2|V]]=E[U^2].$ $\endgroup$ – syeh_106 Aug 4 '15 at 3:50

Motivated by Prof. Guo's paper & Prof. Kim's course materials (see links in the comments below), I've found the answer to this question. I'll post it below in case other folks run into similar questions.

The maximum value of $E[Var(X+Z_1\mid X+Z_2)]$ is $3/2$ and is indeed achieved by $X\sim N(0,1)$, as speculated in Remark #3 in the question. The proof is outlined below.

  1. First, note that $E[Var(X+Z_1\mid X+Z_2)]$ is the (average) MMSE of estimating $X+Z_1$ based on observation of $X+Z_2$. Clearly, for any distribution of $X$, the MMSE cannot be greater than linear MMSE, or LMMSE.

  2. For any given distribution of $X$, the LMMSE $= \sigma_U^2-\frac{Cov(U,V)^2}{\sigma_V^2}=2-\frac{1}{\sigma_X^2+1}\leq 3/2$, independent of the distribution of $X$! ($U$ & $V$ denotes $X+Z_1$ & $X+Z_2$, respectively.)

  3. Therefore, $E[Var(X+Z_1\mid X+Z_2)]\leq 3/2$ for any distribution of $X$.

  4. Moreover, if $X\sim N(0,1)$, $X+Z_1$ & $X+Z_2$ will be jointly Gaussian. It's well known that in this case MMSE is identical to LMMSE. So the maximum expectation is achieved by Gaussian distribution of $X$.

  • $\begingroup$ This answer is motivated by Guo et al's paper (ieeexplore.ieee.org/xpl/…) and by Prof. Young-Han Kim's ECE 153 course materials. $\endgroup$ – syeh_106 Aug 11 '15 at 15:26
  • $\begingroup$ Nicely done. +1 $\endgroup$ – shadowtalker Sep 13 '15 at 22:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.