Bayesian linear regression , variance

Question

I'm currently reading Gaussian processes for machine learning and I don't understand something (or need an intuitive explaination) in the following paragraph:

To make the above notation clearer, from pp. 8-9 of that book, the function we are predicting is the dependent variable $\mathrm{y_*} = f^*(\boldsymbol{\mathrm{x_*}}) = \boldsymbol{\mathrm{w^T x_*}}$ where $\boldsymbol{\mathrm{w}}$ is the vector of regression coefficients (usually denoted by beta in other books). The matrix $A = ( X X^{\mathrm{T}} + \mathbb{V}(\boldsymbol{\mathrm{w}}) ) / \mathbb{V}(\varepsilon)$ is the posterior covariance of the regression coefficient vector.

---

The predictive variance is quadratic form of the test input with the posterior covariance matrix, showing that the predictive uncertainties grow with the magnitude of the test input

In don't get why (intuitively) when fitting a linear regression the uncertainty grows with the magnitude of the input $\boldsymbol{\mathrm{x_*}}$. Let's imagine that naive example where I have this three points in my dataset:

$$(x_1, y_1) = (1, 1),$$ $$(x_2, y_2) = (2, 2),$$ $$(x_3, y_3) = (10, 10).$$

In my mind the uncertainty is lower at $x = 10.01$ than at $x = 6$ (for example), because we have a close training data point near $x = 10$.

Thanks per advance for your help !

Answer 1

The predictive variance is quadratic form of the test input with the posterior covariance matrix, showing that the predictive uncertainties grow with the magnitude of the test input.

The quoted sentence is referring to the variance $\mathrm{x_*^T} A^{-1} \mathrm{x_*}$, which is a quadratic form in $\mathrm{x_*}$. From this form we know that the standard deviation of the predictive distribution is proportional to the scale of the test input. This is unsurprising, given that we are predicting the unknown dependent variable for this value, which is a linear function $\mathrm{y_*} = \mathrm{w^T x_*}$. For any multiplier $k>0$ we have:

$$\mathbb{S}(\mathrm{w^T} (k \mathrm{x_*})) = \mathbb{S}(k \mathrm{w^T x_*}) = k \mathbb{S}(\mathrm{w^T x_*}).$$

To understand the intuition of this, remember what you are doing here. This is a situation where you observe the test input $\mathrm{x_*}$ and you want to predict the corresponding dependent variable $\mathrm{y_*}$. Your uncertainty about this prediction comes from the fact that you do not know the true value of the regression coefficient vector $\mathrm{w}$ and so you are using the posterior distribution of this quantity. If the magnitude of $\mathrm{x_*}$ is larger then there is a larger weighting on the unknown regression coefficient vector, and this leads to greater uncertainty about the dependent variable you are predicting. As you can see from the above equation, if how scale $\mathrm{w}$ up or down by a multiplicative factor $k$, then the predictive standard deviation for $\mathrm{y_*}$ goes up or down accordingly.

(Note: In your own explanation you are giving data points with the dependent variable known, so there is no prediction problem. It appear that you are talking about a different thing to what is being described in the section of the book you are referring to.)

Answer 2

They are using a Linear kernel here. A standard Radial Basis function kernel says that if 2 objects are close to each other then they are similar. In the linear kernel however this is not that case.