# Are normal prior and posterior random variables dependent?

Consider the usual normal Bayesian model. The prior $X=\mu_0+\sigma_0\epsilon_0$, where $\epsilon_0$ follows a standard normal distribution. The data $Y=X+\sigma_1\epsilon_1$, where $\epsilon_1$ follows a standard normal distribution. We know that the posterior $(X|Y=y)$ is in the format of $a\mu_0+(1-a)y+b\epsilon_2$ for some $a$ and $b$, where $\epsilon_2$ follows a standard normal distribution. My question is: what is the relationship between $\epsilon_2$ and $\epsilon_0$? Are they independent, or are they the same with probability one?

All current books state that the distribution of $(X|Y=y)$ is $N(a\mu_0+(1-a)y,b)$. However, none of them discusses the relationship between $\epsilon_2$ and $\epsilon_0$.

## 1 Answer

This is an interesting question! However, you cannot use these representations for fear of falling into fiducial statistics and their difficulties. When you write [with an abuse of notations] \begin{align*} X&=\mu_0+\sigma_0\epsilon_0\\ Y|X&=X+\sigma_1\epsilon_1\\ Y&=\mu_0+\{\sigma_0^2+\sigma_1^2\}^{1/2}\epsilon_3\\ X|Y&=\{\sigma_0^{-2}\mu_0+\sigma_1^{-2}Y\}\{\sigma_0^{-2}\mu_0+\sigma_1^{-2}\}^{-1}+\sigma_2\epsilon_2\\ \end{align*} the random variables $\epsilon_i$ do not live in the same probabilistic universe (aka $\sigma$-algebra), even though they all are $\mathcal{N}(0,1)$ variates. Some are conditionally normal while others are marginally normals. For one thing, if you start substituting terms in these equations, you end up with the nonsensical $$X|Y = \{\sigma_0^{-2}\mu_0+\sigma_1^{-2} [X+\sigma_1\epsilon_1]\}\{\sigma_0^{-2}\mu_0+\sigma_1^{-2}\}^{-1}+\sigma_2\epsilon_2$$ which sees an X on both sides of the equal sign! For another thing, if you agree that $X$ has to be produced or generated before $Y$ is produced or generated conditional on $X$, $\epsilon_0$ has to be generated first, followed by $\epsilon_1$, while the order of generation is reverted for $\epsilon_3$ and $\epsilon_2$. For yet another thing, they cannot all be produced together. Either the pair $(\epsilon_0,\epsilon_1)$ or the pair $(\epsilon_3,\epsilon_2)$ is produced. But not both.

As an extra remark, note also that, since the pair $(X,Y)$ is a bivariate Normal vector, $$(X,Y) \sim \mathcal{N}_2\left((\mu_0,\mu_0), \left[\begin{matrix} \sigma_0^2 &-\sigma_1^2\\-\sigma_1^2 &\sigma_0^2+\sigma_1^2\end{matrix}\right]\right),$$it can also be written as $$(X,Y)=(\mu_0,\mu_0)+\left[\begin{matrix} \sigma_0^2 &-\sigma_1^2\\-\sigma_1^2 &\sigma_0^2+\sigma_1^2\end{matrix}\right]^{1/2}(\epsilon_4,\epsilon_5)$$where these new $\epsilon$'s live in yet another probabilistic space, while being independent Normal variates.

• Thanks for answering the question. Based on your discussion of the order of generating the random variables, may I understand the Bayesian model in the following way: suppose we first generate $X$, say it is $x$, and we then generate $Y$ based on $Y|X=x$. By the time we observe the generated $Y$, $X$ is no longer a random variable. Instead, it is an unknown number $x$. The posterior is then not the usual probability -- un unknown fixed number does not have probability. Instead, it is a belief on the likelihood where the unknown number is located. – Justin Oct 12 '17 at 23:08
• @Justin: the posterior distribution in Bayesian analysis is a genuine [conditional] probability distribution that satisfies all axioms. This does not clash with the fact that a fixed [or true] value of the parameter $X$ led to the generation of the observed $Y=y$: $X=x$ has occurred but since we do not know the value of $x$, we provide the distribution of the possible values of $X$ given $Y=y$. – Xi'an Oct 13 '17 at 8:49
• @Xi'an: I understand that the posterior distribution in Bayesian analysis satisfies all axioms. However, the argument that an unknown fixed value has a distribution seems to contradict point estimation, where a usual view is that the unknown parameter does not have a distribution. For example, the confidence interval is using a random interval to cover a fixed point. To me, the posterior distribution seems more like likelihood, just as in the maximum likelihood estimation. – Justin Oct 13 '17 at 16:54
• @Justin: Take a simple case when $X$ takes only two values, -1 and 1. It is generated but hidden, while $Y$ is generated conditional on the realised value of $X$ and observed. Although $X$ has already taken a fixed (hence non-random) value, all I can produce about $X$ is its conditional distribution given this realisation of $Y$. That does not make $X$ random again. – Xi'an Oct 14 '17 at 16:18
• @Xi'an: I totally agree with what you said, but this is not what I meant. What I meant was that this is different from the usual view in point estimation, where an unknown parameter (e.g., an unknown population mean) is not considered to have a distribution. The unknown parameter in point estimation and the realized $X$ in the Bayesian analysis are both unknown and fixed. – Justin Oct 14 '17 at 21:42