I am self-studying and trying to understand the derivation of the posterior predictive distribution of a naieve bayes model from a graph perspective.

In an online course - the derivation of the probability $P(X[M+1]|x[1],...,x[M])$ is based on multiplying $P(x[M+1]|x[1],...,x[M],\theta)$ with $P(\theta|x[1],...,x[M])$ and integrating out $\theta$. (as shown in the diagram).

Intuitively, but potentially naievely, i would normally have approached the derivation by looking at the graph and first writing out the joint probability $P(X[1:M]),X[M+1],\theta)$ as $P(\theta)$ * $P(X[1:M]|\theta)$ * $P(X[M+1]|\theta)$, where i grouped the individual observations $X[1]$ to $X[M]$ in $X[1:M]$.

However, having started in this way, i am struggling to get from $P(\theta)$ * $P(X[1:M]|\theta)$ * $P(X[M+1]|\theta)$ to the desired probability $P(X[M+1]|x[1],...,x[M])$. Any tips much appreciated

enter image description here


Wouter, there is nothing naive about your statement about the joint probability distribution implied by the above graph. The three major ingredients to solving these kinds of problems are:

1) Knowledge of what you want: $P(X_{M+1} | X_1, \dots, x_M, \theta)$.

2) Understanding of assumptions specific to the model at hand: The factorization of the joint distribution you provided.

3) A basic tool box of definitions/theorems for manipulation of probabilities.

The equation shown in the above image is #1, #3. In fact, it has nothing to do with the particular model (other than the names of the variable). As I'm sure you know, the idea of marginalization is that:

$$P(A) = \int P(A|B) \cdot P(B) dB.$$

If we condition all of these probabilities on some other event $C$, we have the more general:

$$P(A|C) = \int P(A|B,C) \cdot P(B|C) dB.$$

This makes no assumptions about independence or conditional independence. From here the idea would be to use those assumptions to simplify the expression on the right hand side, for instance by noting that the first factor in the integral is equal to $P(X_{M+1} | \theta)$, or determining how do evaluate the second factor in the integral.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.