Consider the usual normal Bayesian model. The prior $X=\mu_0+\sigma_0\epsilon_0$, where $\epsilon_0$ follows a standard normal distribution. The data $Y=X+\sigma_1\epsilon_1$, where $\epsilon_1$ follows a standard normal distribution. We know that the posterior $(X|Y=y)$ is in the format of $a\mu_0+(1-a)y+b\epsilon_2$ for some $a$ and $b$, where $\epsilon_2$ follows a standard normal distribution. My question is: what is the relationship between $\epsilon_2$ and $\epsilon_0$? Are they independent, or are they the same with probability one?

All current books state that the distribution of $(X|Y=y)$ is $N(a\mu_0+(1-a)y,b)$. However, none of them discusses the relationship between $\epsilon_2$ and $\epsilon_0$.


This is an interesting question! However, you cannot use these representations for fear of falling into fiducial statistics and their difficulties. When you write [with an abuse of notations] \begin{align*} X&=\mu_0+\sigma_0\epsilon_0\\ Y|X&=X+\sigma_1\epsilon_1\\ Y&=\mu_0+\{\sigma_0^2+\sigma_1^2\}^{1/2}\epsilon_3\\ X|Y&=\{\sigma_0^{-2}\mu_0+\sigma_1^{-2}Y\}\{\sigma_0^{-2}\mu_0+\sigma_1^{-2}\}^{-1}+\sigma_2\epsilon_2\\ \end{align*} the random variables $\epsilon_i$ do not live in the same probabilistic universe (aka $\sigma$-algebra), even though they all are $\mathcal{N}(0,1)$ variates. Some are conditionally normal while others are marginally normals. For one thing, if you start substituting terms in these equations, you end up with the nonsensical $$X|Y = \{\sigma_0^{-2}\mu_0+\sigma_1^{-2} [X+\sigma_1\epsilon_1]\}\{\sigma_0^{-2}\mu_0+\sigma_1^{-2}\}^{-1}+\sigma_2\epsilon_2$$ which sees an X on both sides of the equal sign! For another thing, if you agree that $X$ has to be produced or generated before $Y$ is produced or generated conditional on $X$, $\epsilon_0$ has to be generated first, followed by $\epsilon_1$, while the order of generation is reverted for $\epsilon_3$ and $\epsilon_2$. For yet another thing, they cannot all be produced together. Either the pair $(\epsilon_0,\epsilon_1)$ or the pair $(\epsilon_3,\epsilon_2)$ is produced. But not both.

As an extra remark, note also that, since the pair $(X,Y)$ is a bivariate Normal vector, $$(X,Y) \sim \mathcal{N}_2\left((\mu_0,\mu_0), \left[\begin{matrix} \sigma_0^2 &-\sigma_1^2\\-\sigma_1^2 &\sigma_0^2+\sigma_1^2\end{matrix}\right]\right),$$it can also be written as $$(X,Y)=(\mu_0,\mu_0)+\left[\begin{matrix} \sigma_0^2 &-\sigma_1^2\\-\sigma_1^2 &\sigma_0^2+\sigma_1^2\end{matrix}\right]^{1/2}(\epsilon_4,\epsilon_5)$$where these new $\epsilon$'s live in yet another probabilistic space, while being independent Normal variates.

  • 1
    $\begingroup$ Thanks for answering the question. Based on your discussion of the order of generating the random variables, may I understand the Bayesian model in the following way: suppose we first generate $X$, say it is $x$, and we then generate $Y$ based on $Y|X=x$. By the time we observe the generated $Y$, $X$ is no longer a random variable. Instead, it is an unknown number $x$. The posterior is then not the usual probability -- un unknown fixed number does not have probability. Instead, it is a belief on the likelihood where the unknown number is located. $\endgroup$
    – Justin
    Oct 12 '17 at 23:08
  • $\begingroup$ @Justin: the posterior distribution in Bayesian analysis is a genuine [conditional] probability distribution that satisfies all axioms. This does not clash with the fact that a fixed [or true] value of the parameter $X$ led to the generation of the observed $Y=y$: $X=x$ has occurred but since we do not know the value of $x$, we provide the distribution of the possible values of $X$ given $Y=y$. $\endgroup$
    – Xi'an
    Oct 13 '17 at 8:49
  • $\begingroup$ @Xi'an: I understand that the posterior distribution in Bayesian analysis satisfies all axioms. However, the argument that an unknown fixed value has a distribution seems to contradict point estimation, where a usual view is that the unknown parameter does not have a distribution. For example, the confidence interval is using a random interval to cover a fixed point. To me, the posterior distribution seems more like likelihood, just as in the maximum likelihood estimation. $\endgroup$
    – Justin
    Oct 13 '17 at 16:54
  • $\begingroup$ @Justin: Take a simple case when $X$ takes only two values, -1 and 1. It is generated but hidden, while $Y$ is generated conditional on the realised value of $X$ and observed. Although $X$ has already taken a fixed (hence non-random) value, all I can produce about $X$ is its conditional distribution given this realisation of $Y$. That does not make $X$ random again. $\endgroup$
    – Xi'an
    Oct 14 '17 at 16:18
  • $\begingroup$ @Xi'an: I totally agree with what you said, but this is not what I meant. What I meant was that this is different from the usual view in point estimation, where an unknown parameter (e.g., an unknown population mean) is not considered to have a distribution. The unknown parameter in point estimation and the realized $X$ in the Bayesian analysis are both unknown and fixed. $\endgroup$
    – Justin
    Oct 14 '17 at 21:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.