Confused with independence and Bayes theorem

Question

I am doing one calculation in two different ways and the result does not match.

Let two sets of random variables $\bar{y} = (y_1, y_2, \ldots, y_N)$ and $\bar{x} = (x_1, x_2, \ldots, x_N)$. Each $y_i$ depends only on $x_i$ and a parameter $\theta$ and all the $x_i$ are i.i.d from a parameter $\theta$.

If I try to calculate $p(\bar{x},\theta|\bar{y})$ we get

First way

First I apply Bayes theorem $$ p(\bar{x},\theta|\bar{y}) = p(\bar{y}|\bar{x},\theta) \frac{p(\bar{x},\theta)}{p(\bar{y})} \propto p(\bar{y}|\bar{x},\theta) p(\bar{x}|\theta) p(\theta) $$ Then I apply independence $$ p(\bar{x},\theta|\bar{y}) \propto \left( \prod_{i=1}^N p(y_i|x_i,\theta) \right) \left( \prod_{i=1}^N p(x_i|\theta) \right) p(\theta)$$

Second way

Independence applied first $$ p(\bar{x},\theta|\bar{y}) = \prod_{i=1}^N p(x_i,\theta|\bar{y}) = \prod_{i=1}^N p(x_i,\theta|y_i)$$

Then, Bayes theorem $$ p(\bar{x},\theta|\bar{y}) = \prod_{i=1}^N p(y_i|x_i,\theta) \frac{p(x_i,\theta)}{p(y_i)} \propto \prod_{i=1}^N p(y_i|x_i,\theta) {p(x_i|\theta)} p(\theta) $$

to make the inconsistency with the first way clearer, this could be written as $$ p(\bar{x},\theta|\bar{y}) \propto \left(\prod_{i=1}^N p(y_i|x_i,\theta) \right) \left(\prod_{i=1}^N {p(x_i|\theta)} \right) \left(p(\theta)\right)^N $$

In the first way, $p(\theta)$ contributes once. In the second it contributes as many times as data points. Can you spot what did I do wrong?

Thanks in advance

Answer 1

Your second way is incorrect because $\prod_{i=1}^n p(x_i,\theta|\overline{y})\neq\prod_{i=1}^n p(x_i,\theta|y_i)$. Perhaps this becomes more clear if you try it for two variables:

$$p(x_1,x_2,\theta|y_1,y_2) = p(x_1,\theta|x_2,y_1,y_2)p(x_2|y_1,y_2)=p(x_1,\theta|y_1)p(x_2|y_2)$$

This is different from $\prod_{i=1}^{n} p(x_i,\theta|y_i)$ for $n=2$ since

$$p(x_1,\theta|y_1)p(x_2|y_2)=p(x_1,\theta|y_1)\frac{p(x_2,\theta|y_2)}{p(\theta)}\\=\left(\prod_{i=1}^{n}p(x_i,\theta|y_i)\right)/p(\theta)^{n-1} \neq \prod_{i=1}^{n} p(x_i,\theta|y_i)$$

unless $p(\theta)=1$. If you insert the missing $\frac{1}{p(\theta)^{n-1}}$ term in your second way, then it becomes equivalent to your first way.