Prior/Posterior predictive distributions

Question

I have trouble understanding some notes I have from the lectures. We are in Bayesian linear regression and he explained how we can first introduce a prior probability distribution to the weights: $p(w∣α)$ where α is a hyperparameter. Then we can compute the posterior on the weights like this:$ p(w∣x,t,α,σ2) \propto p(t∣x,w,σ2)⋅$$p(w∣α)$, where x is our dataset and t our target variables. So we do Maximum A Posteriori for this posterior, and then he goes on to predictive distributions and gives us this formula:

$p(t_{new}∣x_{new},t,x,α,σ2)=∫((p(t_{new}∣x_{new},w,t,x,σ2)⋅$$p(w∣α))dw$

I can't understand the above formula. Does he use the prior predictive distribution? And if yes, why is he not using the earlier result of the posterior on the weights? Shouldn't we use exactly that to find the predictive distribution?

Answer 1

The prior predictive would have no dependence on $x,t$:

$$p(t_{new}|x_{new}, \sigma, \alpha) = \int p(t_{new}|x_{new}, w, \sigma)p(w|\alpha)dw$$

The posterior predictive would have such dependence. In principle, to find a posterior predictive, we use the data to learn about the parameters, and then integrate over our updated beliefs about those parameters. It would typically be expressed:

$$p(t_{new}|x_{new}, \sigma, x, t, \alpha) = \int p(t_{new}|x_{new}, w, \sigma)p(w|x,t,\alpha, \sigma)dw$$

(Assuming your example assumes $\sigma$ known, otherwise we would also integrate over the posterior for $\sigma$.)

So it would seem that by including $x,t$ in the first term, your professor wishes to stress that the posterior predictive depends—via $w$—on $x,t$. But if the equation you give is a valid expression of the posterior predictive, it would seem this must somehow hold:

$$ p(t_{new}∣x_{new},w,t,x,\sigma)p(w|\alpha) = p(t_{new}|x_{new}, w, \sigma)p(w|x,t,\alpha,\sigma)$$

$t_{new}$ is conditionally independent from $x,t | w, x_{new}$, and perhaps your professor's expression somehow leverages that fact, but it isn't immediately clear to me how.

This short tutorial might help approach the posterior predictive of linear regression from another angle.