1
$\begingroup$

Suppose I have a set of points $Y$, $X$ and I model them by using a Gaussian process $\mathcal{GP}(m,K)$. Let the noisy observations be given by

$$y^i = f^i + \sigma^2$$

$p(f)$ gives the prior on the Gaussian process. Then what will be the posterior $p(f|Y,Z)$ which is again a Gaussian process with a mean $(m',K')$. How can $(m',K')$ be derived?

$\endgroup$
1
  • $\begingroup$ Can you clarify your notations ? In particular, you didn't defined Z which appears in your posterior p(f|Y,Z). $\endgroup$
    – Emile
    Sep 1, 2013 at 1:08

1 Answer 1

5
$\begingroup$

In such a case the covariance function is changed. Let's fix notation. Suppose that we have a sample $D = (X,\mathbf{y}) = \{(\mathbf{x}_i, y_i)\}_{i = 1}^N$. The observations $y_i$ have the form $$ y_i = y(\mathbf{x}_i) = f(\mathbf{x}_i) + \epsilon_i. $$ There $f_i = f(\mathbf{x}_i)$ is a true function value at the point $\mathbf{x}_i$, noise $\epsilon_i$ has an iid normal distribution with zero mean and variance $\sigma^2$. $f$ is a gaussian process with zero mean and covariance function $$ cov(f(\mathbf{x}_i), f(\mathbf{x}_j)) = k(\mathbf{x}_i, \mathbf{x}_j). $$

Due to noise, the covariance function for $y$ is $$ cov(y_i, y_j) = cov(f_i + \epsilon_i, f_j + \epsilon_j) = cov(f_i, f_j) + cov(\epsilon_i, \epsilon_j) = k(\mathbf{x}_i, \mathbf{x}_j) + \sigma^2 \delta(\mathbf{x}_i - \mathbf{x}_j). $$ There $\delta(x_1 - x_2)$ is a delta function: $\delta(\mathbf{x})$ equals $1$ for $\mathbf{x} = 0$ and zero for other $\mathbf{x}$; $k(x_1, x_2)$ is an old covariance function.

The posterior mean has the form: $$ m = \mathbf{k} K^{-1} \mathbf{y}. $$ There $K = \{k(\mathbf{x}_i, \mathbf{x}_j)\}_{i, j = 1}^N$ is a covariance matrix. Due to change of the covariance function one has to use instead the new covariance matrix $K' = K + \sigma^2 I$ ($I$ is the identity matrix); the covariance vector $\mathbf{k}$ is unchanged, so the new equation is $$ m = \mathbf{k} (K')^{-1} \mathbf{y}. $$ The formula for the posterior variance is derived in the same way.

$\endgroup$
7
  • $\begingroup$ Very nice, clear answer. $\endgroup$
    – Placidia
    Jan 7, 2013 at 11:30
  • $\begingroup$ I am having a doubt. how can the true function value $f_i = f(\mathbf{x}_i)$ can be calculated? $\endgroup$ May 5, 2015 at 6:50
  • $\begingroup$ If we set noise level to zero, then we get that $m(\mathbf{x}_i) = y_i$ as the product $\mathbf{k}$ and $K^{-1}$ in this case equals corresponding row of identity matrix. $\endgroup$ May 5, 2015 at 8:07
  • $\begingroup$ @AlexeyZaytsev, your definition of Dirac delta is incorrect. You should probably switch to Kronecker notation $\delta_{ij}$ judging by the context of your answer. Dirac delta is usually defined via integral $\int_{-\infty}^\infty\delta(x)dx=1$ and $\delta(0)=\infty$ and $\delta(x\ne 0)=0$ $\endgroup$
    – Aksakal
    May 14, 2019 at 20:30
  • $\begingroup$ @Aksakal fixed, thx $\endgroup$ May 16, 2019 at 20:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.