I'm currently reading Gaussian processes for machine learning and I don't understand something (or need an intuitive explaination) in the following paragraph:

enter image description here

To make the above notation clearer, from pp. 8-9 of that book, the function we are predicting is the dependent variable $\mathrm{y_*} = f^*(\boldsymbol{\mathrm{x_*}}) = \boldsymbol{\mathrm{w^T x_*}}$ where $\boldsymbol{\mathrm{w}}$ is the vector of regression coefficients (usually denoted by beta in other books). The matrix $A = ( X X^{\mathrm{T}} + \mathbb{V}(\boldsymbol{\mathrm{w}}) ) / \mathbb{V}(\varepsilon)$ is the posterior covariance of the regression coefficient vector.

The predictive variance is quadratic form of the test input with the posterior covariance matrix, showing that the predictive uncertainties grow with the magnitude of the test input

In don't get why (intuitively) when fitting a linear regression the uncertainty grows with the magnitude of the input $\boldsymbol{\mathrm{x_*}}$. Let's imagine that naive example where I have this three points in my dataset:

$$(x_1, y_1) = (1, 1),$$ $$(x_2, y_2) = (2, 2),$$ $$(x_3, y_3) = (10, 10).$$

In my mind the uncertainty is lower at $x = 10.01$ than at $x = 6$ (for example), because we have a close training data point near $x = 10$.

Thanks per advance for your help !

  • $\begingroup$ What is the function $f_*$ here? They must have had a specific function they are using to get from the general integral formula to the specific result. So what was the function? $\endgroup$
    – Ben
    Mar 1 '18 at 0:15
  • $\begingroup$ ...and what is its relationship to the matrix $A$? $\endgroup$
    – Ben
    Mar 1 '18 at 0:21
  • $\begingroup$ Hi Ben sorry, didn't put the whole section, its a very small section of the book (2.1.1) as you'll see $f(x) = w^T x$ , its a linear regression (starting at page 8) gaussianprocess.org/gpml/chapters/RW.pdf $\endgroup$
    – abcdaire
    Mar 1 '18 at 12:09
  • $\begingroup$ Page reference? $\endgroup$
    – Ben
    Mar 1 '18 at 12:35
  • $\begingroup$ The subsection 2.1.1 starts at the page 26/266 (Page 8 on the book) :) $\endgroup$
    – abcdaire
    Mar 1 '18 at 13:16

The predictive variance is quadratic form of the test input with the posterior covariance matrix, showing that the predictive uncertainties grow with the magnitude of the test input.

The quoted sentence is referring to the variance $\mathrm{x_*^T} A^{-1} \mathrm{x_*}$, which is a quadratic form in $\mathrm{x_*}$. From this form we know that the standard deviation of the predictive distribution is proportional to the scale of the test input. This is unsurprising, given that we are predicting the unknown dependent variable for this value, which is a linear function $\mathrm{y_*} = \mathrm{w^T x_*}$. For any multiplier $k>0$ we have:

$$\mathbb{S}(\mathrm{w^T} (k \mathrm{x_*})) = \mathbb{S}(k \mathrm{w^T x_*}) = k \mathbb{S}(\mathrm{w^T x_*}).$$

To understand the intuition of this, remember what you are doing here. This is a situation where you observe the test input $\mathrm{x_*}$ and you want to predict the corresponding dependent variable $\mathrm{y_*}$. Your uncertainty about this prediction comes from the fact that you do not know the true value of the regression coefficient vector $\mathrm{w}$ and so you are using the posterior distribution of this quantity. If the magnitude of $\mathrm{x_*}$ is larger then there is a larger weighting on the unknown regression coefficient vector, and this leads to greater uncertainty about the dependent variable you are predicting. As you can see from the above equation, if how scale $\mathrm{w}$ up or down by a multiplicative factor $k$, then the predictive standard deviation for $\mathrm{y_*}$ goes up or down accordingly.

(Note: In your own explanation you are giving data points with the dependent variable known, so there is no prediction problem. It appear that you are talking about a different thing to what is being described in the section of the book you are referring to.)

  • $\begingroup$ Hello Ben , thank you for your detailed response. However I'm still confused a little bit. My point is that you use the training points (with the dependent variable known) in your posterior distribution. The point is that I don't see why the magnitude enters into account. For me it's the fact that the test point is far away / close to several training points that makes the uncertainty bigger / smaller. I just found the same question today , maybe his formulation is better stats.stackexchange.com/questions/172903/… $\endgroup$
    – abcdaire
    Mar 1 '18 at 22:31
  • $\begingroup$ Maybe a figure will help , as we can see from this figure : s3.amazonaws.com/quantstart/media/images/… , we have less uncertainty around x=0.4 compared to x=0.2 .. Maybe I don't have the right definition of what magnitude is. $\endgroup$
    – abcdaire
    Mar 1 '18 at 23:04

They are using a Linear kernel here. A standard Radial Basis function kernel says that if 2 objects are close to each other then they are similar. In the linear kernel however this is not that case.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.