1
$\begingroup$

The following quotation is from Machine Learning: a Probabilistic Perspective Chapter 7, page 234

In machine learning, we often care more about predictive accuracy than about interpreting the parameters. Using Equation 4.126, we can easily show that the posterior predictive distribution at a test point $\mathbf{x}$ is also Gaussian: $$ \begin{aligned} p(y|\mathbf{x}, \mathcal{D}, \sigma^2) &= \int\mathcal{N}\left(y|\mathbf{x}^T\mathbf{w}, \sigma^2\right)\mathcal{N}\left(\mathbf{w}|\mathbf{w}_N, \mathbf{V}_N\right)d\mathbf{w}\\ &=\mathcal{N}\left(y|\mathbf{w}^T_N\mathbf{x}, \sigma^2_N(\mathbf{x})\right)\\ \sigma^2_N(\mathbf{x}) &= \sigma^2 + \mathbf{x}^T\mathbf{V}_N\mathbf{x} \end{aligned} $$ The variance in this prediction, $\sigma^2_N(x)$, depends on two terms: the variance of the observation noise, $\sigma^2$, and the variance in the parameters, $\mathbf{V}_N$. The latter translates into variance about observations in a way which depends on how close $\mathbf{x}$ is to the training data $\mathcal{D}$.

My question is in which way does "$\mathbf{V}_N$ translates into variance about observations in a way which depends on how close $\mathbf{x}$ is to the training data $\mathcal{D}$"? I mean, OK intuitively, but how can i prove it in a formal way?

$\endgroup$
1

2 Answers 2

1
$\begingroup$

Remember that $V_N$ is the covariance matrix for the model coefficients, so it is a positive definite matrix. That makes $d(x)=x^TV_Nx$ a vector norm, in the mathematical sense: it is not the usual euclidian norm, and it will give more weight to deviances in some directions than in others, but it still satisfies the definition of a norm function. Norms can be interpreted as the distance between a point and the origin, and since $X$ is centered, the origin is the mean of the input vectors. Therefore, $d(x)=x^TV_Nx$ measures the distance between the input vector $x$ and the mean of the input in the training set.

Hope it helps!

$\endgroup$
0
0
$\begingroup$

Its right there in line 7.60.

$y$ is drawn using $\mathbf{w}$, and $\mathbf{w}$ is drawn using $V_N$. The dependence of $y$ on $V_N$ is through $\mathbf{w}$.

$\endgroup$
1
  • 2
    $\begingroup$ I believe the OP is trying to understand how $x^\prime V_N x$ expresses "how close $x$ is to the training data." $\endgroup$
    – whuber
    Jan 25, 2022 at 22:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.