0
$\begingroup$

Given a random variable $X$ which follows a normal distribution $X \sim (\mu,\sigma^2)$ with unknown mean and known variance. Say we have $n$ observations $y_i$ where $1\le i \le n$ and each $y_i$ is sampled from a normal distribution $Y_i \sim N(x,\sigma_i^2)$ (not i.i.d.) where $x$ is sampled from $X$ and it is the same for all $\{Y_i\}$, but $\sigma_i$ is different for different $Y_i$.

The question is: how to compute the posterior distribution $X$ given the observations p(X|{y_i})? The question is different from the answer listed in Wikipedia because $y_i$ is not sampled from i.i.d. distribution.

$\endgroup$
1
$\begingroup$

Here is the setup as I understand it. Let $y = (y_1, \ldots, y_n)$. The sampling distribution for $y$ can be expressed as \begin{equation} p(y|x) = \prod_{i=1}^n \textsf{N}(y_i|x,\sigma_i^2) . \end{equation} Once $y$ is observed, $p(y|x)$ becomes the likelihood for $x$. The prior distribution for $x$ is \begin{equation} p(x) = \textsf{N}(x|\mu,\sigma^2) . \end{equation}

Given this setup, the posterior distribution for $x$ is given by Bayes' rule: \begin{equation} p(x|y) = \frac{p(y|x)\,p(x)}{p(y)} = \textsf{N}(x|m,s^2) , \end{equation} where \begin{align} s^2 &= \left(\frac{1}{\sigma^2} + \sum_{i=1}^n \frac{1}{\sigma_i^2}\right)^{-1} \\ m &= s^2\left(\frac{\mu}{\sigma^2} + \sum_{i=1}^n \frac{y_i}{\sigma_i^2}\right) . \end{align} One may confirm the answer by checking that $p(y|x)\,p(x)/p(x|y)$ does not involve $x$ (after simplification).

The values for $m$ and $s^2$ may be found by using a property of the normal distribution. Given some $K > 0$, let \begin{equation} h(x) = \log\big(K\,\textsf{N}(x|m,s^2)\big) = -\frac{(x-m)^2}{2\,s^2} + C , \end{equation} where $C$ does not involve $x$. We may obtain $m$ by solving $h'(x) = 0$ for $x$. In addition, $-1/h''(x) = s^2$.

We can apply this approach as follows. Let $K = p(y)$. Then $h(x) = \log\big(p(y|x)\,p(x)\big)$.

$\endgroup$
  • $\begingroup$ Can you give the derivation or a reference of computing $s^2$ and $m$? $\endgroup$ – Yi Yang Dec 17 '17 at 3:23
  • $\begingroup$ I've added a derivation. $\endgroup$ – mef Dec 17 '17 at 10:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.