# Calculation of posterior distribution of a Gaussian process

Suppose I have a set of points $Y$, $X$ and I model them by using a Gaussian process $\mathcal{GP}(m,K)$. Let the noisy observations be given by

$$y^i = f^i + \sigma^2$$

$p(f)$ gives the prior on the Gaussian process. Then what will be the posterior $p(f|Y,Z)$ which is again a Gaussian process with a mean $(m',K')$. How can $(m',K')$ be derived?

• Can you clarify your notations ? In particular, you didn't defined Z which appears in your posterior p(f|Y,Z). Sep 1, 2013 at 1:08

In such a case the covariance function is changed. Let's fix notation. Suppose that we have a sample $$D = (X,\mathbf{y}) = \{(\mathbf{x}_i, y_i)\}_{i = 1}^N$$. The observations $$y_i$$ have the form $$y_i = y(\mathbf{x}_i) = f(\mathbf{x}_i) + \epsilon_i.$$ There $$f_i = f(\mathbf{x}_i)$$ is a true function value at the point $$\mathbf{x}_i$$, noise $$\epsilon_i$$ has an iid normal distribution with zero mean and variance $$\sigma^2$$. $$f$$ is a gaussian process with zero mean and covariance function $$cov(f(\mathbf{x}_i), f(\mathbf{x}_j)) = k(\mathbf{x}_i, \mathbf{x}_j).$$
Due to noise, the covariance function for $$y$$ is $$cov(y_i, y_j) = cov(f_i + \epsilon_i, f_j + \epsilon_j) = cov(f_i, f_j) + cov(\epsilon_i, \epsilon_j) = k(\mathbf{x}_i, \mathbf{x}_j) + \sigma^2 \delta(\mathbf{x}_i - \mathbf{x}_j).$$ There $$\delta(x_1 - x_2)$$ is a delta function: $$\delta(\mathbf{x})$$ equals $$1$$ for $$\mathbf{x} = 0$$ and zero for other $$\mathbf{x}$$; $$k(x_1, x_2)$$ is an old covariance function.
The posterior mean has the form: $$m = \mathbf{k} K^{-1} \mathbf{y}.$$ There $$K = \{k(\mathbf{x}_i, \mathbf{x}_j)\}_{i, j = 1}^N$$ is a covariance matrix. Due to change of the covariance function one has to use instead the new covariance matrix $$K' = K + \sigma^2 I$$ ($$I$$ is the identity matrix); the covariance vector $$\mathbf{k}$$ is unchanged, so the new equation is $$m = \mathbf{k} (K')^{-1} \mathbf{y}.$$ The formula for the posterior variance is derived in the same way.
• I am having a doubt. how can the true function value $f_i = f(\mathbf{x}_i)$ can be calculated? May 5, 2015 at 6:50
• If we set noise level to zero, then we get that $m(\mathbf{x}_i) = y_i$ as the product $\mathbf{k}$ and $K^{-1}$ in this case equals corresponding row of identity matrix. May 5, 2015 at 8:07
• @AlexeyZaytsev, your definition of Dirac delta is incorrect. You should probably switch to Kronecker notation $\delta_{ij}$ judging by the context of your answer. Dirac delta is usually defined via integral $\int_{-\infty}^\infty\delta(x)dx=1$ and $\delta(0)=\infty$ and $\delta(x\ne 0)=0$ May 14, 2019 at 20:30