# Is Gaussian process regression a Bayesian method?

Actually I thought Gaussian Process is a kind of Bayesian method, since I read many tutorials in which GP is presented in Bayesian context, for example, in this tutorial, just pay attention to page 10.

Suppose the GP prior is $$\pmatrix{h\\ h^*} \sim N\left(0,\pmatrix{K(X,X)&K(X,X^*)\\ K(X^*,X)&K(X^*,X^*)}\right)$$, $(h,X)$ is for the observed training data, $(h^*,X^*)$ for the test data to be predicted. And the actually observed noisy output is $$Y=h+\epsilon$$, where $\epsilon$ is the noise, $$\epsilon\sim N(0,\sigma^2I)$$. And now as shown in the tutorial, we have $$\pmatrix{Y,Y^*}=\pmatrix{h\\ h^*}+\pmatrix{\epsilon\\ \epsilon^*}\sim N\left(0,\pmatrix{K(X,X)+\sigma^2I&K(X,X^*)\\ K(X^*,X)&K(X^*,X^*)+\sigma^*I}\right)$$, and finally by conditioning on $Y$, we could have $p(Y^*|Y)$, which is called as predictive distribution in some books or tutorials, but also called posterior in others.

QUESTION

1. According to many tutorials, the predictive distribution $p(Y^*|Y)$ is derived by conditioning on $Y$, if this is correct, I don't understand why GP Regression is Bayesian? Nothing about Bayesian is used in this conditional distribution derivation, right?

2. However, I don't actually think the predictive distribution should be just the conditional distribution, I think it should be $$p(Y^*|Y)=\int p(Y^*|h^*)p(h^*|h)p(h|Y)dh$$, in the above formula, $p(h|Y)$ is the posterior, right?

On your first question: GPs are Bayesian because they involve constructing a prior distribution (here over functions directly rather than over parameters) and updating this distribution by conditioning on the data. The Gaussian part just makes the resulting posterior friendlier to work with than it might be otherwise.

On your second question: you might ask how your last equation is realised by the 'even simpler approach' described in section 4.2. Things are indeed being integrated out there.

• I'm not sure what you exactly mean by "Gaussian part just makes the resulting posterior friendlier"? Do you mean, if it's not the Gaussian distribution, the derivation of posterior would be more complicated? Feb 3, 2014 at 9:21
• By my second question, I mean, to derive $p(Y^*|Y)$, I'm supposed to find out $p(h|Y)$ first, and this is the posterior of $h$ after observing $Y$. Then just like parametric Bayesian averaging, I compute the integral $\int p(Y^*|h^*)p(h^*|h)p(h|Y)dh$, which is the predictive distribution of $Y^*$, it averages over $p(h|Y)$. However, the tutorial I read doesn't do as I describe, they just go straight forward to compute the conditional distribution $p(Y^*|Y)$ from the joint $p(Y,Y^*)$, and this is why I think it's not Bayesian. I mean computing the Gaussian conditional probability is Bayesian? Feb 3, 2014 at 9:32
• by 'friendlier' I mean 'the marginal and conditional distributions of interest are available in closed form and moreover computable by straightforward matrix algebraic operations' Feb 3, 2014 at 10:32
• on the second question: Perhaps gaussianprocess.org/gpml/chapters/RW2.pdf (through section 2.2 at least) is helpful. This explicitly compares the normal parametric Bayesian regression approach to the GP one with no noise (your posterior over h) and then shows how it extends to a posterior over a new data point. Feb 3, 2014 at 10:48
• On the more general question: 'computing a Gaussian conditional probability' is in itself neither Bayesian nor anything else. The interpretation of a GP as a prior over functions any finite number of observations from which are jointly Normal is the Bayesian part. Feb 3, 2014 at 10:50

Seems the question is not settled totally. I was also frustrated about it until hit the below:

"Posterior probability is just the conditional probability that is outputted by the Bayes theorem. There is nothing special about it, it does not differ anyhow from any other conditional probability, it just has it's own name."