6
$\begingroup$

Let a distribution over functions be described by a Gaussian Process (GP) prior, following the notation of Rasmussen and Williams: $$ f(\mathbf{x})\sim\mathcal{GP}(m(\mathbf{x}), k(\mathbf{x},\mathbf{x}')) $$ then, considering the mean function $m(\mathbf{x})$ as zero and a set of points $X_*$, we can sample function points from the multivariate normal distribution: $$\mathcal{N}(\mathbf{0}, K(X_*, X_*))$$ where $K(X_*, X_*)$ is the covariance matrix corresponding to the kernel of choice.

Considering this, how should we refer to the interval $\pm (k\cdot\sigma_{\mathbf{x}})$, where $k$ is a positive constant multiplying the standard deviations (square root of the diagonal of $K(X_*, X_*)$), $\sigma_{\mathbf{x}}$?

Visually, this interval corresponds to the grey area of the following left subfigure (extracted from Rasmussen and Williams):

enter image description here

As can be seen in the caption, authors refer to this interval as "confidence region".

However, I do not understand some points related to this figure and its caption:

  1. Since the predicted function points are scalar values, shouldn't this interval be a "confidence interval" instead [link]?

  2. On the other hand, given that this interval does not correspond to any specific parameter but to the interval where we expect to observe function points, should this interval (for the left subfigure) instead be refered as "prediction interval"?

  3. Is it an approximation to consider just the marginalized $\sigma_{\mathbf{x}}$ when computing this interval, i.e., discarding correlation information?

  4. Finally, once we condition on observed training points to predict the the posterior predictive distribution (right subfigure), shouldn't the corresponding interval also be a prediction interval? For this last one, I have doubts if we should use credible interval instead, as suggested e.g. by this blog.

$\endgroup$

1 Answer 1

1
$\begingroup$

1. Since the predicted function points are scalar values, shouldn't this interval be a "confidence interval" instead?

It's a collection of pointwise confidence intervals. A confidence region is just a generalization of a confidence interval, so it's not technically wrong (even if it is a little confusing). Don't stress about this too much.


2. On the other hand, given that this interval does not correspond to any specific parameter but to the interval where we expect to observe function points, should this interval (for the left subfigure) instead be referred as "prediction interval"?

In this particular example, it is both. The confidence intervals are for the mean of the function. Since the function can be realized without noise, it is also a prediction interval.

If the data was generated with noise, then the covariance matrix becomes $\bf K+\tau\bf I$, where $\tau$ is called the "nugget". In this case, there would be a difference between the confidence and predictive intervals. In particular, the confidence interval (for the function mean) will look similar to above but the predictive interval must also account for the nugget effect.


3. Is it an approximation to consider just the marginalized $\sigma_{\mathbf{x}}$ when computing this interval, i.e., discarding correlation information?

I'm not entirely sure I understand this question. The left panel shows what we have before observing any data. We have nothing to condition on and thus no way to use any information about the correlation structure.

On the right, we have datapoints to condition on, and we use the correlation structure (plus the process variance $\sigma_X^2$) to create confidence intervals. This is what leads to the "football" shaped confidence intervals. Uncertainty is highest farthest away from observed data and is zero (for $\tau=0$, at least) at the observed data.


4. Finally, once we condition on observed training points to predict the the posterior predictive distribution (right subfigure), shouldn't the corresponding interval also be a prediction interval? For this last one, I have doubts if we should use credible interval?

I partially addressed this already in the answer to Q2. As for the term credible/confidence interval, I agree that it is confusing. You can call it a credible interval if you like, but both terms are fine in this case.

Many authors like to introduce GPs as a prior/posterior over functions, but in fact there is nothing inherently Bayesian about this analysis. It's based on simple facts of the Gaussian distribution.

To this point, all of this discussion so far is treating the correlation structure as fixed and known. If there are unknown correlation parameters $\phi$, then we can estimate them with both frequentist and Bayesian methods. After we account for uncertainty in the parameters, then the distinction between confidence/credible intervals becomes meaningful (although still largely pedantic, in my opinion).

$\endgroup$
6
  • $\begingroup$ Thank you for your time. I grasp it better now, but I'm still a bit confused: 1) Aren't confidence intervals (CI) for estimated parameters? Here, the mean is not estimated (0 mean) 2) Why is the CI the same as without noise $\pm k\cdot$diag$(K)$ but the PI is $\pm k\cdot$diag$(K+\tau I)$? 3) By correlation, I mean the non-diagonal terms of $K$. If we sample 3 points, we could draw ellipsoids for CIs|*PI*s, which isn't possible with only the diagonal of $K$ 4) My concern with credible intervals is similar to 1). Since a GP is non-parametric, isn't it unsuitable? $\endgroup$
    – abc
    Commented May 16 at 20:19
  • $\begingroup$ 1) $0$ is the prior mean. The mean of the posterior is directly in the middle of the confidence interval (not shown). The true mean is the function value. 2) Because the CI tries to capture the mean of the function, but if there is extra noise (such as measurement error) then predictions need to account for this extra noise. 3) So these are pointwise intervals. I wouldn't say its an "approximation" but yes you are losing some information. 4) As with $1$, the mean of the function is a parameter, so it makes sense to talk about confidence/credible intervals for it. $\endgroup$
    – knrumsey
    Commented May 16 at 21:14
  • $\begingroup$ Thank you so much for the clarifications. If I may ask two follow-up questions for 1) and 2): 1) I see your point about the posterior. However, the left subfigure corresponds to the prior distribution (with a fixed/assumed mean of 0), i.e., we are not predicting the mean. So in this case, for the prior, confidence intervals (CIs) should not be applicable, but prediction intervals (PIs) should be, is this correct? 2) Just to confirm: after conditioning with the covariance $K+\tau I$ of the observed samples, is modeling the noise in the test samples the difference between PIs and CIs? $\endgroup$
    – abc
    Commented May 16 at 21:51
  • $\begingroup$ @abc 1) Yes, these are "prior predictive intervals" and they are commonly used to check whether the model and prior are reasonable for the data. Focus one just vertical slice of both figures ($x=1$, for example) and it becomes easier. The prior for the mean $\mu(1)$ is $N(0, 1)$ and the posterior is described by the grey band at $x=1$. 2) It's a little bit more sophisticated than that, but that's the rough idea. If you want to learn about the mean at $x=1$, use a CI. If you want to predict the observed data at $x=1$, use a PI. Since there is no noise, these are the same in this example. $\endgroup$
    – knrumsey
    Commented May 16 at 22:00
  • $\begingroup$ @abc I highly recommend you check out Chapter 5 of Surrogates, by Bobby Gramacy. It's free online, and he does a really nice job of explaining the intuition behind GPs. It's even better if you can follow along with the R code he provides. $\endgroup$
    – knrumsey
    Commented May 16 at 22:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.