How to interpret marginal likelihood definition?

Question

Say we have a Beta-Bernoulli model where $X_i$ are i.i.d. Bernoulli variables, $p(X_i=1)=\theta$, and $\theta\sim\operatorname{Beta}(\alpha,\beta)$.

The marginal likelihood is defined as

$$ p(X)=\int p(X|\theta)p(\theta)d\theta $$

Consider a concrete $X=(1,1)$.

One way to compute $p(X)$, described in "Machine Learning. A Probabilistic Perspective" by Kevin P. Murphy, is

$$p(X)=p(X_1)p(X_2|X_1)=\frac{\alpha}{\alpha+\beta}\cdot\frac{\alpha+1}{\alpha+\beta+1}$$

On the other hand, since $X_i$ are independent, we should be able to say that $P(X_2|X_1)=p(X_2)$,

$$p(X)=p(X_1)p(X_2|X_1)=\left(\frac{\alpha}{\alpha+\beta}\right)^2$$

I assume that Kevin's interpretation is the correct one, but does it mean I can no longer apply the familiar facts from the probability theory such as $p(X_2|X_1)=p(X_2)$ for independent variables?

Answer 1

$X_1$ and $X_2$ are independent conditional on $\theta$: $$ p(X_1,X_2|\theta) = p(X_1|X_2,\theta)\,p(X_2|\theta) = p(X_1|\theta)\,p(X_2|\theta), $$ But they are not unconditionally independent: $$ \begin{split} p(X_1,X_2) &= \int p(X_1|\theta)\,p(X_2|\theta)\,p(\theta)\,d\theta \\ &= \begin{cases} \frac{\alpha (\alpha +1)}{(\alpha +\beta ) (\alpha +\beta +1)} & X_1=1\land X_2=1 \\ \frac{\alpha \beta }{(\alpha +\beta ) (\alpha +\beta +1)} & \left(X_1=0\land X_2=1\right)\lor \left(X_1=1\land X_2=0\right) \\ \frac{\beta (\beta +1)}{(\alpha +\beta ) (\alpha +\beta +1)} & X_1=0\land X_2=0 \end{cases}. \end{split} $$ Moral: it is important to keep track of the conditioning arguments.