1
$\begingroup$

The question is as follows, it's mainly part 3 that I was having problem with.

A discrete-valued parameter with the prior pdf $$p(x) = > \sum_{i=1}^2p_i\delta(x-i)$$ is measured with the additive noise $w ~ N(0, \sigma^2)$. The measurement z is given by: $$z = x + w$$

  1. Find the posterior pdf of the parameter.
  2. Find the MAP estimate and the associated MSE conditioned on z.
  3. Find its MMSE estimate and the associated variance.

For part 1, I used Bayes formula and the total probability theorem to get $$p(x|z) = \frac{f(z)}{2\pi\sigma^2}*e^{\frac{(z-x)^2}{2\sigma^2}}*\sum_{i=1}^2p_i\delta(x-i)$$ where $f(z) = e^{-\frac{(z-1)^2}{2\sigma^2}} + e^{-\frac{(z-2)^2}{2\sigma^2}}$

For part 2, I solved $$\frac{d}{dx}[ln(p(x|z)]= 0$$ by considering the dirac delta function as a constant and setting z-x =0, giving $\hat x^{MAP}= z$ and $MSE(\hat x^{MAP})= 0$

However, when it comes to part 3, I know that $\hat x^{MMSE} = E(x|z)$. I also noticed $p(x|z)$ can be divided into three parts, a function of z, a normal distribution of with mean x and variance $\sigma$, and p(x) which involves the dirac delta function. But I'm not sure how to calculate the expected value.

I thought about expanding the summation, and since $\delta(x-i) = 0$ everywhere except for when x = 1 & x = 2, I can just take x = 1 & x = 2 and have $p(x|z)$ as a function of z, which is esentially a constant. But it also seems a little sketchy.

I would really appreciate it if someone could give me a hint and point me in the right direction to solve this problem. Thank you in advance.

$\endgroup$
3
  • 1
    $\begingroup$ There are many issues here such as a sign error in what should be $e^{-\frac{(z-x)^2}{\sigma^2}}$ and the implausibility of the MAP being $z$ when that is almost never $1$ or $2$ $\endgroup$
    – Henry
    Sep 26, 2021 at 1:41
  • $\begingroup$ Sorry fixed the typo. $\endgroup$ Sep 26, 2021 at 2:25
  • 1
    $\begingroup$ The posterior is incorrect for at least two reasons, you cannot take derivatives when $x\in\{1,2\}$, the posterior mean is the sum of two terms and is not the MMSE when $x\in\{1,2\}$. $\endgroup$
    – Xi'an
    Sep 26, 2021 at 6:08

1 Answer 1

2
$\begingroup$

Ok, let's start with fixing up part 2, then go back to part 1.

Suppose the prior probabilities are equal. The prior is supported only on the set of points $\{1,2\}$, so the posterior will also be supported only on those points. The MAP estimator must also take values in $\{1,2\}$. The prior probability is equal on those two points. The loglikelihood for an observation $z$ given $x$ is proportional to $(x-z)^2$, so the posterior probability will be higher on whichever of 1 and 2 is closer to $z$. That gives you the MAP estimator.

To get question 1 (and to handle the non-equal case of question 2), you'd need to quantify how much more likely the closer point is. Bayes' Theorem says $p(x|z)\propto p(x)f(z|x)$, so the probabilities on $\{1,2\}$ are proportional to the $N(0,\sigma^2)$ densities at $z-1$ and $z-2$ respectively.

Now you have the posterior probabilities over the two points you should be able to calculate the posterior mean

$\endgroup$
2
  • $\begingroup$ Thank you so much for the answer! For question 2, how did you know the prior probability is equal for i = 1 & 2? Why can I assume p1 = p2 in p(x)? For question 1, could you elaborate a bit on what you mean by quantifying how much more likely the closer point is? Is the p(x|z) I had still not the final form? Thanks again! $\endgroup$ Sep 26, 2021 at 14:19
  • $\begingroup$ Ok, if it's not equal you do need to do part 2 before part 1 $\endgroup$ Sep 26, 2021 at 22:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.