Is the posterior distribution of the difference of two parameters the same as the difference of the two parameters posteriors?

Question

I am reading the book Doing Bayesian Data Analysis by John K. Kruschke. In the chapter related to Gibbs sampling, the author suggests that we cannot understand the difference of two parameters by looking at their posterior distributions and that we should analyze the posterior of the parameters difference.

He also provides an example with two cases: θ₁ and θ₂ have the same posterior distributions across both cases but in the first one the parameters are positively correlated while in the second one they are negatively correlated. Then in the first case the distribution of the difference in narrow and far from 0 while in the second one it is much wider.

I always thought that there was no difference between sampling from two parameters posterior (and then taking the difference of the traces) and directly sampling from the difference posterior.

Are there cases in which the posteriors difference is not the same posterior of the difference?

Answer 1

I don't know the section of Kruschke you're thinking of, but I suspect the general point, is this. (I'll take some liberties with the math for the sake of intuition).

Assume that the complete posterior is $P(\theta_1, \theta_2, \theta_3 \mid X)$ from which you have $N$ samples, arranged as an N x 3 matrix, S.

Repeatedly choosing an element randomly from S[,1], another one randomly from S[,2], and then subtracting them (which is the sample equivalent of drawing from $P(\theta_1 \mid X) $ and then from $P(\theta_2 \mid X)$ and subtracting them), will generate $N$ new quantities whose distribution will not in general converge to $P(\theta_2 - \theta_1 \mid X)$.

However, repeatly sampling a row randomly, say the $k$th, from S[,1:2] and then subtracting S[k,2] from S[k,1], will generate $N$ quantities whose distribution will converge to $P(\theta_2 - \theta_1 \mid X)$, because this process is its sample analogue.

Kruschke altered the covariance to show a familiar form of the kind of dependency between $\theta_1$ and $\theta_2$ that will be preserved in samples from the second procedure but not in samples from the first.

In short, everything you want is available from a set of samples from the joint posterior, but if you only know the distribution of its marginals then all bets are off.

For a classical analogy, recall that group-specific confidence intervals may overlap while the test of a group difference (or equivalently, an interval for that difference) may still reject.

Answer 2

1. When you draw from the two posteriors, the draws are independent. Whereas when you draw from the posterior of the difference, you have already taken into account a possible relationship between the two random variables.
2. A correct way to draw from the two posteriors would be to draw from one of the posteriors and then draw from the conditional of the second posterior given the value drawn from the first posterior.

Answer 3

You want the pdf for the difference between two parameters, which may be written $$ p(\Delta\theta) = \int \delta\left[\Delta\theta - (\theta_2 - \theta_1)\right] \,p(\theta_1, \theta_2)\, \text{d}\theta_1 \text{d}\theta_2 $$ Evidently, this depends upon the joint pdf $p(\theta_1, \theta_2)$ and cannot be found from $p(\theta_1)$ and $p(\theta_2)$ only without a further assumption e.g. that $p(\theta_1, \theta_2) = p(\theta_1) p(\theta_2)$.