Posterior predictive distribution, linear regression

Question

The following quotation is from Machine Learning: a Probabilistic Perspective Chapter 7, page 234

In machine learning, we often care more about predictive accuracy than about interpreting the parameters. Using Equation 4.126, we can easily show that the posterior predictive distribution at a test point $\mathbf{x}$ is also Gaussian: $$ \begin{aligned} p(y|\mathbf{x}, \mathcal{D}, \sigma^2) &= \int\mathcal{N}\left(y|\mathbf{x}^T\mathbf{w}, \sigma^2\right)\mathcal{N}\left(\mathbf{w}|\mathbf{w}_N, \mathbf{V}_N\right)d\mathbf{w}\\ &=\mathcal{N}\left(y|\mathbf{w}^T_N\mathbf{x}, \sigma^2_N(\mathbf{x})\right)\\ \sigma^2_N(\mathbf{x}) &= \sigma^2 + \mathbf{x}^T\mathbf{V}_N\mathbf{x} \end{aligned} $$ The variance in this prediction, $\sigma^2_N(x)$, depends on two terms: the variance of the observation noise, $\sigma^2$, and the variance in the parameters, $\mathbf{V}_N$. The latter translates into variance about observations in a way which depends on how close $\mathbf{x}$ is to the training data $\mathcal{D}$.

My question is in which way does "$\mathbf{V}_N$ translates into variance about observations in a way which depends on how close $\mathbf{x}$ is to the training data $\mathcal{D}$"? I mean, OK intuitively, but how can i prove it in a formal way?

Answer 1

Remember that $V_N$ is the covariance matrix for the model coefficients, so it is a positive definite matrix. That makes $d(x)=x^TV_Nx$ a vector norm, in the mathematical sense: it is not the usual euclidian norm, and it will give more weight to deviances in some directions than in others, but it still satisfies the definition of a norm function. Norms can be interpreted as the distance between a point and the origin, and since $X$ is centered, the origin is the mean of the input vectors. Therefore, $d(x)=x^TV_Nx$ measures the distance between the input vector $x$ and the mean of the input in the training set.

Hope it helps!

Answer 2

Its right there in line 7.60.

$y$ is drawn using $\mathbf{w}$, and $\mathbf{w}$ is drawn using $V_N$. The dependence of $y$ on $V_N$ is through $\mathbf{w}$.