5
$\begingroup$

I am trying to understand where the formula used in Posterior Predictive Check comes from:

$$P(y^* | y) = \int p(y^*|\theta) p(\theta|y) d\theta$$

Where:

  • $y^*$ is the new data
  • $y$ is the original data
  • $\theta$ are the model parameters
  • $p(\theta |y)$ is the posterior probability distribution
  • $p(y^* |y)$ is the probability of observing new data based on the existing data
  • $p(y^* |\theta)$ is the probability of observing new data based on your model (i.e. the likelihood)

In the above equation, we first generate a random realization of $\theta_i$ from the posterior distribution, and then generate a random value of $y_i$ from the likelihood's distribution centered around $\theta_i$. If the model fits the data well, then in general :

$$P(y^* \mid y) \approx P(y)$$

Here is my attempt to derive the formula used in Posterior Predictive Check:

Step 1: For argument sake, let's write (using the Law of Conditional Probability):

$$P(y^*, \theta \mid y) = \frac{P(y^*, \theta, y)}{P(y)}$$

Step 2: Now, let's focus just on ${P(y^*, \theta, y)}$.

If we say that $A =y^*$ and $B=\theta, y$, then:

$$P(A \cap B) = P(A \mid B)P(B)$$ $$P(y^*, \theta , y) = {P(y^* | \theta, y)} \cdot P(\theta, y)$$

Step 3: Going back to Step 1, we can take the result from Step 2 and write:

$$P(y^*, \theta \mid y) = \frac{P(y^*, \theta, y)}{P(y)} = \frac{{P(y^* | \theta, y)} \cdot P(\theta, y)}{P(y)}$$

We can also see that:

$$P(\theta \mid y) = \frac{P(\theta, y)}{P(y)}$$

Thus, we can re-write it as:

$$P(y^*, \theta \mid y) = \frac{P(y^*, \theta, y)}{P(y)} = {P(y^* | \theta, y)} \cdot P(\theta | y)$$

Step 4: Now, we integrate both sides with respect to $\theta$:

$$\int P(y^*, \theta \mid y) \, d\theta = \int {P(y^* | \theta, y)} \cdot P(\theta | y) \, d\theta$$

From here, the Left Hand Side loses the integral - thus we can write:

$$P(y^* \mid y) = \int {P(y^* | \theta, y)} \cdot P(\theta | y) \, d\theta$$

Here is where I am stuck.

I don't know how I can further simplify :

$$P(y^* \mid y) = \int {P(y^* | \theta, y)} \cdot P(\theta | y) \, d\theta$$

$$P(y^* \mid y) = \int {P(y^* | \theta)} \cdot P(\theta | y) \, d\theta$$

I read this has to do with a conditional independence assumption, i.e. $P(y^* | \theta, y) = P(y^* | \theta)$ ... but I am not quite sure why this assumption is justified in this case.

Can someone please help me figure this out?

$\endgroup$

1 Answer 1

5
$\begingroup$

If you assume the model $\theta\to y$ and $\theta\to y^*$ then conditional on $\theta$, $y$ and $y^*$ are independent. Therefore $P(y^*|y,\theta)=P(y^*|\theta)$

Proof: $$ P(y,y^*,\theta)=P(y|\theta)P(y^*|\theta)P(\theta) $$ Conditioning on $\theta:$ $$ P(y,y^*|\theta)=\frac{P(y|\theta)P(y^*|\theta)P(\theta)}{P(\theta)}=P(y|\theta)P(y^*|\theta) $$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.