I think after all the reading I've done I still don't fully understand MAP estimation. I came across a problem that's leaving me dumbfounded.

Suppose $A$ ~ $N(0,\sigma^2_1) $ and $\epsilon$ ~ $N(0,\sigma^2_2)$

Let's say $a$ is drawn from the distribution of $A$ and that we observe $b$, which is conditioned on $a$, coming from a random variable $B$:

$b = a + \epsilon$

The questions asks me to find a MAP estimate of $a$ given $b$.

This seems so simple and yet I'm confused - I thought the point of MAP estimation was to use Bayes rule to determine the posterior by multiplying the likelihood and the prior - but I don't see what the prior on $a$ is here. Am I missing something basic?

  • $\begingroup$ The prior on $A$ is a $N(0,\sigma^2_1)$ distribution, with $b$ a realisation of a $N(a,\sigma^2_2)$ distribution. $\endgroup$ – Xi'an Mar 14 '19 at 7:21

There seems to be some confusion about your understanding of the prior and posterior concepts. Having $X\sim f(x)$ is prior information. Because, without knowing extra details, you know how really $X$ is distributed. So, $A\sim N(0,\sigma_1^2)$ means $A$'s prior distribution is normal with mean $0$, and variance $\sigma_1^2$.

Posterior distibution is obtained after having some additional data. For example, here it is $f_{A|B}(a|b)$, i.e. the density of $A$ when we know that $B=b$. MAP estimates maximizes the posterior distribution, but commonly we don't derive the posterior and then maximize it. Posterior density is proportional to the multiplication of likelihood and the prior due to Bayes Rule, i.e. $f_{A|B}(a|b)\propto f_{B|A}(b|a)f_A(a)$. So, we maximize this multiplication and find $\hat{a}_{MAP}$.

You know the prior, $f_A(a)$. $f_{B|A}(b|a)$ is $N(a,\sigma_2^2)$ as in @Xi'an's comment, because if $a$ is given, it just acts as a shifter to $\epsilon$. A normal variable that is shifted towards some direction is also a normal RV, with shifted mean and the same variance.

You'll just multiply these two densities, and maximize.

  • $\begingroup$ This makes sense. As a follow up, if I was instead not explicitly told the distribution of $A$, but was told, for example, that $A$ can take values +1 or -1, with equal probabilities, then can I still infer the distribution of $A$ to get the prior information? $\endgroup$ – John Alperto Mar 14 '19 at 11:27
  • $\begingroup$ Yes, when you know how A is distributed, i.e. PDF or PMF, you know the prior. It doesn't matter if the distribution has a name or not. $\endgroup$ – gunes Mar 14 '19 at 11:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.