0
$\begingroup$

I am doing maximum a posteriori (MAP) to estimate $\mu$ and $\sigma$ with $N$ samples drawn from $\mathcal{N}(5, 1)$. The priors that I place are $\mu\sim\mathcal{N}(5, 1)$ and $\sigma\sim\mathcal{N}(1, 1)$.

Taking the derivatives of the posteriors and setting the derivatives to 0, I get \begin{align} -\mu+5+\frac{1}{\sigma^2}\left(\sum\limits_{n=1}^{N}x_n-N\mu\right) &= 0 \\ \ \\ -\sigma+1-\frac{N}{\sigma}+\frac{\sum_{n=1}^{N}(x_n-\mu)^2}{\sigma^3} &= 0, \end{align} which can be solved by plugging in $N$ data points $\{x_1,x_2,\ldots,x_N\}$.

My problem occurs when $N=1$. That is, I only have one data point available to do MAP. Say my that point is 5.1. Plugging in, I solve it in MATLAB by

syms m s;
% Assume \sigma is non-zero
S = solve([ ...
    s^2*(m-5)-5.1+m == 0, ...
    -s^4+s^3-s^2+(5.1-m)^2 == 0], ...
    [m, s]);

mus_hat = double(vpa(S.m));
sigmas_hat = double(vpa(S.s)); 

All the solutions are complex and hence cannot be correct.

I understand that a prior $\sigma\sim\mathcal{N}(1, 1)$ might be inappropriate. But how to explain it is this prior that causes all the solutions to be complex? I can't really see the link. Is there an intuitive explanation for this?

$\endgroup$
  • 1
    $\begingroup$ Instead of defacing your post, delete it if you want to delete it. $\endgroup$ – J. Steen Mar 16 '15 at 15:44
  • $\begingroup$ @J.Steen Crap. Sorry, new to the site. Didnt mean it. Great that it got rolled back. Thanks @gung! $\endgroup$ – Stephnie Choi Mar 16 '15 at 15:46
  • 3
    $\begingroup$ Please stop vandalizing your post. You can flag it to have your user ID disconnected from the Q, if you want. $\endgroup$ – gung - Reinstate Monica Mar 16 '15 at 15:46
4
$\begingroup$

If we write down the posterior distribution on $(\mu,\sigma)$ associated with a single observation $x$, $$\pi(\mu,\sigma|x)\propto\sigma^{-1}\exp\frac{-1}{2}\left\{\sigma^{-2}(x-\mu)^2 +(\mu-5)^2+(\sigma-1)^2\right\}\mathbb{I}_{\mathbb{R}^+_-}(\sigma)$$ (as $\sigma$ is necessarily positive), this function takes the value $$\pi(x,\sigma|x)\propto\sigma^{-1}\exp\frac{-1}{2}\left\{(\mu-5)^2+(\sigma-1)^2\right\}\mathbb{I}_{\mathbb{R}^+_-}(\sigma)$$ when $\mu=x$ and $\pi(x,\sigma|x)$ is unbounded when $\sigma$ goes to zero: $$\lim_{\sigma\to 0^+} \pi(x,\sigma|x)=+\infty\,.$$ This demonstrates there is no MAP in this setting.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ +1; Can I check this is what you meant: the posterior distribution given one sample shoots up to $+\infty$ at $(\mu, \sigma)=(x, 0)$; therefore, setting derivatives to 0 and solving for the maximum point will not return real solutions? $\endgroup$ – Sibbs Gambling Mar 17 '15 at 14:37
  • $\begingroup$ When the maximum occurs at a boundary point, first order conditions do not have to be satisfied. $\endgroup$ – Xi'an Mar 17 '15 at 15:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.