2
$\begingroup$

If $X=Y+Z$ where $Y$ and $Z$ are independent gaussians with zero means and known variances, what is the MMSE decision rule to estimate $Y$ for $X=x$?


I know that $h_{MMSE}(x)=$ posterior mean $= E[Y|X=x]=\int yf(y|x)dy$
and $f(y|x) = \frac{f(x,y)}{f(x)}$, but I don't have a good enough grasp of the material to put this information together. I found here a formula that shows it as:

enter image description here

However I can't follow the steps and don't know how this changes for my case of a linear combination of independent gaussians.

edit: Here's the original question I'm looking at, but I figured I'd just ask about the first part of it (MMSE). enter image description here

$\endgroup$

1 Answer 1

2
$\begingroup$

If $Y$ and $Z$ are independent Gaussian random variables, then $(X,Y) = (Y+Z,Y)$ enjoys a bivariate Gaussian distribution. Indeed, we nave that $$E[X] = E[Y]+E[Z], ~ \operatorname{var}(X) = \operatorname{var}(Y) + \operatorname{var}(Z), \\\operatorname{cov}(X,Y) = \operatorname{cov}(Y+Z,Y) = \operatorname{cov}(Y,Y) = \operatorname{var}(Y)$$ and we can use this to write down the joint density function explicitly if we choose to do so.

Now, the MMSE estimate of $Y$ given that $X = x_0$ is, as you know, $E[Y\mid X = x_0]$, that is, the mean of the conditional density of $Y$ conditioned on $X = x_0$. This is a general result that does not require any special conditions on $X$ and $Y$. But when $(X,Y)$ has a bivariate Gaussian distribution, the MMSE estimator coincides with the linear MMSE estimator of $Y$ given that $X = x_0$. The linear MMSE estimator is, of course, the estimator achieves the minimum mean-square error among all estimators that are constrained to be linear functions (actually, affine functions) of $x_0$: $\hat{Y} = ax_0+b$ where $a$ and $b$ are real numbers. So, what is the linear MMSE estimator of $Y$ given that the value of $X$ is $x_0$? If you have never seen this calculation done before, we want to find $a$ and $b$ such that $E[(Y=ax_0+b)^2]$ is as small as possible, and the usual calculus methods for finding minima tell us that $a$ and $b$ must be such that $$\hat{Y} =ax_0+b = E[Y] + \left.\left.\frac{\operatorname{cov}(X,Y)}{\operatorname{var}(X)}\right(x_0 - E[X]\right) = E[Y] + \left.\left.\frac{\rho\cdot\sigma_Y}{\sigma_X}\right(x_0 - E[X]\right)$$ where $\rho$ is the correlation coefficient of $X$ and $Y$, and $\sigma_Y$ and $\sigma_X$ are the standard deviations of $Y$ and $X$ respectively. Note that this result not require that $XY$ and $Y$ are jointly Gaussian; it is perfectly general. But the linear MMSE estimator is also, in general, not the MMSE estimator which is usually a nonlinear function of $x_0$. For jointly Gaussian random variables, we have the happy result that the linear MMSE estimator coincides with the MMSE estimator.

Truth be told, I can never remember the details this formula and prefer to keep it in mind in the more elegant form $$\left(\frac{\hat{Y}-E[Y]}{\sigma_Y}\right) = \rho \left(\frac{X-E[X]}{\sigma_X}\right)$$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.