I want to find $p(x \vert y)$ given $p(y \vert x)$. I am aware of Bayesian formula $p(x \vert y) = \frac{p(y \vert x)p(x)}{\int_{-\infty}^{+\infty}p(y \vert x)p(x)dx}$ but I can not understand why the following logic is wrong.

Let $x$ be some random variable with pdf $p(x)$ and $\varepsilon \sim N(0,1)$ independent of $x$.

Define $$y(x) \equiv x + \varepsilon\tag{1},$$ then $(y\vert x) \sim N(x,1)$.

But it also should be true that $$x=y-\varepsilon\tag{2}$$ and because standard normal distribution is symmetric around zero, then it should follow that $(x \vert y) \sim N(y,1)$. Thus we didn't use the information about unconditional distribution $p(x)$, needed for Bayesian formula!

Please, point me where I am wrong.

The following analytical solution should prove that two distributions are indeed not equal.

Let $x \sim N(\mu,\sigma)$, then $p(x)=\frac{e^{-\frac{(x-\mu )^2}{2 \sigma ^2}}}{\sqrt{2 \pi } \sigma }$.

Let $y(x)=x+\varepsilon$ and $\varepsilon \sim N(0,1)$, then $p(y\vert x)=\frac{e^{-\frac{1}{2} (y-x)^2}}{\sqrt{2 \pi }}$. Hence, $\int_{-\infty}^{+\infty}p(y \vert x)p(x)dx=\frac{e^{-\frac{(y-\mu )^2}{2 \left(\sigma ^2+1\right)}}}{\sqrt{2 \pi } \sqrt{\frac{1}{\sigma ^2}+1} \sigma }$ and therefore by Bayesian formula $p(x\vert y)=\frac{\sqrt{\frac{1}{\sigma ^2}+1} \exp \left(-\frac{(x-\mu )^2}{2 \sigma ^2}-\frac{1}{2} (y-x)^2+\frac{(y-\mu )^2}{2 \left(\sigma ^2+1\right)}\right)}{\sqrt{2 \pi }}$. Finally, this is not equal to pdf of $N(y,1)$, which is $\frac{e^{-\frac{1}{2} (x-y)^2}}{\sqrt{2 \pi }}$.

  • 1
    $\begingroup$ This is the fiducial fallacy! $\endgroup$ – Xi'an Jan 10 '18 at 15:29

Your equation (1) implies that $y$ and $\varepsilon$ are dependent random variables. Hence, the distribution of $\varepsilon$ in equation (2) depends on $y$ and is no longer N(0,1) conditional on $y$ and $x=y-\varepsilon$ is not $N(y,1)$.


Your statement ε∼N(0,1) is the a priori distribution; that is, before you observe ε, your expectation for the distribution is that it's normal. Once you observe the value of ε, however, it would be silly to expect it to be normally distributed; its distribution is now a delta function at the value you observed. When you observe y, although you are not directly observing ε, you are getting information about it, and so you should update your expectation as to how ε is distributed based on that information. So once you observe y, it is no longer appropriate to model ε as normally distributed. Just as p(x|y) isn't going to be the same as p(x), p(ε|y) isn't going to be the same as p(ε)

Suppose X is a die roll, and y = .99. We can be pretty confident that x= 1 and ε = -.01. A less likely possibility is that x could be 2 and ε = -1.01. However, we know that "x = .99 and ε = 0 is zero" is impossible; x is an integer. So this example is a case where ignoring the a priori distribution of x would give an absurd result.

There is also regression to the mean to take into consideration. Suppose you're measuring people's heights, and your population has an average height of 1.7 m and std of 0.20 m, so X∼N(1.7,0.20). If y = 1.7, then everything is symmetric, so given any a, it's just as likely that ε = a as ε = -a. But what if y = 1.80? Consider the following two hypotheses: "The actual height was 1.79, but it was mismeasured as 1.80" versus "The actual height was 1.81, but it was mismeasured as 1.80"? Hopefully, it's clear that the first is more likely; given any extreme measured value, it's more likely that the true value was less extreme, than that it was more extreme. To make this even more clear, what if y = 2.7? Which is more likely: this person is 2.7 m tall, or you have a large error? As y gets farther and farther away from the mean of x, it becomes more and more likely that ε is large; as y gets larger and larger, the distribution of ε|y skews more and more negative. As y gets larger, the expected value of x|y increases, but the amount that it increases per unit of y decreases. The average height of people with measured height of 1.7 is 1.7. The average height of people with measured height of 1.8 is slightly less than 1.8. The average height of people with measured height of 2.7 isn't much larger than the average height of people with measure height of 2.8; at that point, most of the difference is due to error.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.