0
$\begingroup$

I am doing maximum a posteriori (MAP) to estimate $\mu$ and $\sigma$ with $N$ samples drawn from $\mathcal{N}(5, 1)$. The priors that I place are $\mu\sim\mathcal{N}(5, 1)$ and $\sigma\sim\mathcal{N}(1, 1)$.

Taking the derivatives of the posteriors and setting the derivatives to 0, I get \begin{align} -\mu+5+\frac{1}{\sigma^2}\left(\sum\limits_{n=1}^{N}x_n-N\mu\right) &= 0 \\ \ \\ -\sigma+1-\frac{N}{\sigma}+\frac{\sum_{n=1}^{N}(x_n-\mu)^2}{\sigma^3} &= 0, \end{align} which can be solved by plugging in $N$ data points $\{x_1,x_2,\ldots,x_N\}$.

My problem occurs when $N=1$. That is, I only have one data point available to do MAP. Say my that point is 5.1. Plugging in, I solve it in MATLAB by

syms m s;
% Assume \sigma is non-zero
S = solve([ ...
    s^2*(m-5)-5.1+m == 0, ...
    -s^4+s^3-s^2+(5.1-m)^2 == 0], ...
    [m, s]);

mus_hat = double(vpa(S.m));
sigmas_hat = double(vpa(S.s)); 

All the solutions are complex and hence cannot be correct.

I understand that a prior $\sigma\sim\mathcal{N}(1, 1)$ might be inappropriate. But how to explain it is this prior that causes all the solutions to be complex? I can't really see the link. Is there an intuitive explanation for this?

$\endgroup$
3
  • 1
    $\begingroup$ Instead of defacing your post, delete it if you want to delete it. $\endgroup$
    – J. Steen
    Mar 16, 2015 at 15:44
  • $\begingroup$ @J.Steen Crap. Sorry, new to the site. Didnt mean it. Great that it got rolled back. Thanks @gung! $\endgroup$ Mar 16, 2015 at 15:46
  • 3
    $\begingroup$ Please stop vandalizing your post. You can flag it to have your user ID disconnected from the Q, if you want. $\endgroup$ Mar 16, 2015 at 15:46

1 Answer 1

4
$\begingroup$

If we write down the posterior distribution on $(\mu,\sigma)$ associated with a single observation $x$, $$\pi(\mu,\sigma|x)\propto\sigma^{-1}\exp\frac{-1}{2}\left\{\sigma^{-2}(x-\mu)^2 +(\mu-5)^2+(\sigma-1)^2\right\}\mathbb{I}_{\mathbb{R}^+_-}(\sigma)$$ (as $\sigma$ is necessarily positive), this function takes the value $$\pi(x,\sigma|x)\propto\sigma^{-1}\exp\frac{-1}{2}\left\{(\mu-5)^2+(\sigma-1)^2\right\}\mathbb{I}_{\mathbb{R}^+_-}(\sigma)$$ when $\mu=x$ and $\pi(x,\sigma|x)$ is unbounded when $\sigma$ goes to zero: $$\lim_{\sigma\to 0^+} \pi(x,\sigma|x)=+\infty\,.$$ This demonstrates there is no MAP in this setting.

$\endgroup$
2
  • $\begingroup$ +1; Can I check this is what you meant: the posterior distribution given one sample shoots up to $+\infty$ at $(\mu, \sigma)=(x, 0)$; therefore, setting derivatives to 0 and solving for the maximum point will not return real solutions? $\endgroup$ Mar 17, 2015 at 14:37
  • $\begingroup$ When the maximum occurs at a boundary point, first order conditions do not have to be satisfied. $\endgroup$
    – Xi'an
    Mar 17, 2015 at 15:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.