2
$\begingroup$

I am reading the chapter on regression in the Gaussian Process for machine learning book. The author introduces Bayesian linear regression and he describes that the distribution is normally distributed with some mean and variance.

He makes a statement like: "The predictive variance is a quadratic form of the test input with the posterior covariance matrix, showing that the predictive uncertainties grow with the magnitude of the test input, as one would expect for a linear model." This is related to eq (2.9) in the link.

This bit has me confused. Why should the posterior uncertainty be a function of the magnitude of the test input points? The test input points are where we want to make the predictions and if we have had some training data around these points, I can see that the uncertainty will be small but do not see what the relation with the magnitude of this input point is...

$\endgroup$

2 Answers 2

1
$\begingroup$

This post is quite old, but having read the same chapter of the book "Gaussian Processes for Machine Learning" of Rasmussen and Williams, I add my answer hoping to help someone in the future.

Short answer:
First, take a step back and consider what happens in the posterior distribution of the weights w.

The prior model of the weights assumes a gaussian around zero: $\mathcal{N}({\bf 0}, \Sigma_p)$. So, the output needs to be near zero according to this model and any output far from it is uncertain. This means that if we only consider the prior knowledge, the output of the regressor $y={\bf w^T x}$ will also be around zero with deviation $\Sigma_p {\bf x}$. As a result, the larger the magnitude of the test input x, the larger the deviation from zero. Since the a posteriori estimation of w is a combination of the prior model and the data, the uncertainty of w will grow with the magnitude of x. As a result, the prediction of a new test input, being the weighted average of all ${\bf w}$ multiplied by the input, will also be more and more uncertain when $||{\bf x}||$ is large.

Longer answer:
In this book, the authors almost arbitrarily choose the prior distribution of w to be zero. Let's assume a non-zero mean and for simplicity of the formulas let's consider the 1D case where our data is a single observation $(x_1, y_1)$. The prior is $p(w)=\mathcal{N}(w_0, \sigma_p^2)$ but again the deviation of $y$ from $w_0$ still depends on $\sigma_p x$. By "completing the square" the posterior distribution of $w$ becomes:

$ p(w | X, {\bf y}) = p(w | x_1, y_1) = \frac{p(y|x_1,y_1)p(w)}{p(y_1|x_1)} = c\mathcal{N}(wx_1, \sigma_n^2)\mathcal{N}(w_0, \sigma_p^2) = \mathcal{N}(\bar{w}, a^2) $

with $ \bar{w} = \frac{y}{x} \cdot \frac{1}{1 + \frac{\sigma_n^2}{x_1^2\sigma_p^2}} + w_0 \cdot \frac{1}{1 + \frac{\sigma_p^2}{x_1^2\sigma_n^2}} = \frac{y}{x} \kappa + w_0(1-\kappa) $

This equation is very important because it shows how our prior knowledge and data influence the final estimation of the weight $w$. Actually it is a weighted sum of the two estimations, since w_0 is the prior mean weight value and $\frac{y}{x}$ is what we expect the slope to be from a line passing through the origin. Let's now make some observations:

  • If $\sigma_n \rightarrow 0$ then $\kappa \rightarrow 1$ which means that we are perfectly sure about our observation ($\sigma_n$ is the uncertainty of the observation y) and therefore we drop the prior knowledge and we only keep the term $\frac{y}{x}$ as our final estimation of $w$.

  • If $\sigma_p \rightarrow 0$ then $\kappa \rightarrow 0$. Small $\sigma_p$ means that our prior knowledge is very solid and observed data must not affect it a lot. So the weight estimation is mostly based on the term $w_0$.

  • Finally, in all the other cases the $\kappa$ depends on the fraction $\lambda^2 = \frac{\sigma_n^2}{x^2\sigma_p^2}$ with $\kappa = \frac{1}{1+\lambda^2}$.
    If $x^2 >> 1$ then $\lambda << 1$ and so $\sigma_n^2 << x^2 \sigma_p^2$. So $\kappa \rightarrow 1$.
    This is the most important part: What this inequality says is that the uncertainty $\sigma_n$ of the observed output $y$ is much less than the uncertainty we would have had if we only used the prior model of $w$ and taking $w \cdot x$ as the output. Remember: $\sigma_n$ is the uncertainty of our measured observations and $(\sigma_p x)$ is the deviation of the observation from the prior model. Therefore, when $x$ is large, the deviation from the prior model is also large and the posterior estimation dictates that we have to mostly trust the observed output rather than the output coming from the prior model.

Similarly, if $x^2 << 1$ then $\lambda >> 1$ and so $\sigma_n^2 >> x^2 \sigma_p^2$ and $\kappa \rightarrow 0$. This means that since the magnitude of $x$ is small, so is the deviation of the output from the prior model. So the prior estimation is favored.

Finally, let's consider the 99.6% confidence intervals when we use the measured output and the output from the prior model respectively:

  • prior model:
    The weight $w$ is normally distributed with mean $w_0$ and standard deviation $ \sigma_p$. The confidence interval for the true output is:
    $(w_0 - 3\sigma_p)x \leq y_{true} \leq (w_0 + 3\sigma_p)x$
    or
    $|y_{true} - w_0x| \leq 3\sigma_p x$

  • data:
    The measurement error is normally distributed and therefore:
    $y -3\sigma_n \leq y_{true} \leq y + 3\sigma_n$
    or
    $|y_{true} - y| \leq 3\sigma_n$

From the above we can deduce which estimation is the best: the prior model $wx$ or the observed output $y$. The two estimations lie around different mean values with different uncertainties. What we care about is the uncertainty that is $3\sigma_p x$ or $3\sigma_n$.

$\endgroup$
0
$\begingroup$

I'm not familiar with the book, but I'm guessing the "test input points" are the explanatory variables. Prediction uncertainty is smallest when the explanatory variables are at their mean and grows larger as the explanatory variables go away from the mean.

$\endgroup$
1
  • $\begingroup$ The test inputs are the input points for which we want to predict the output for (equation 2.9) in my link. So, I do not see how this explanation works. I can see that there is a dot product involved but do not see how the intuition behind the uncertainties and the input data magnitude. $\endgroup$
    – Luca
    Commented Sep 17, 2015 at 14:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.