Let's imagine I have the following equation $y_t=f(x_t)+e_t$, where $f(x)$ follows a gaussian process, and $e_t\sim N(0,\Sigma)$.

How does one go about to learn the hyperparameters, i.e., $\Sigma$ and those of of the gaussian process, in a Bayesian way?

More specifically, how would we write the $p(Y|X, \theta)$, where $\theta$ are the hyperparameters of the model, and $Y=(y_1,...,y_N)$ and $X=(x_1,...,x_N)$?


You could define a prior distribution on the hyperparameters and add the log-prior to the log-likelihood to obtain the log-posterior up to a constant.

You could then maximise the log-posterior just as you'd maximise the log-likelihood to obtain a MAP solution, or you could use MCMC methods to obtain samples from this log-posterior - you'd then have samples from the joint sampling distributions of the hyperparameters.

You'd then perhaps use the evidence (marginal likelihood) to compare different models/kernels, or if you're trying to fit more complex models, you could obtain lower bound approximations for the evidence and fine-tune parameters or model structures so that these lower bounds are maximised.

Is this roughly what you were looking for?

  • $\begingroup$ Could you elaborate a bit more on the MCMC method. How would you write the likelihood? I would like to draw from $p(\theta|X,Y)\propto p(X,Y|\theta)p(\theta)$. My problem is the $p(X,Y|\theta)$, where we've marginalized the function $f$ out... if we had $p(X,f(X), Y|\theta)$ instead, I would know how to find it's expression. $\endgroup$ Oct 12 '18 at 20:40
  • $\begingroup$ I'm not sure I understand. $Y$ is completely determined by $X$ and $e_t$ in your example. In that example, it's extremely easy to write the likelihood, as X is also an MVN, you'd just be adding $\Sigma$ to the GP's covariance matrix. So, if $e_t \sim N(0, \sigma_n^2 \mathcal I)$, you could simply add the $\sigma_n^2 1(x = x')$ to your kernel $\endgroup$
    – adityar
    Oct 12 '18 at 21:36
  • $\begingroup$ Thanks for the comment, but 2 things I don't understand. 1) When we fix X, we're not fixing Y, since f(x) follows a GP. Could you write the likelihood of the model in my question in your answer? 2) Why do you say that X is MVN (matrix-valued Normal?) ? $\endgroup$ Oct 12 '18 at 21:43
  • $\begingroup$ Sorry, I meant $f(X)$ is a multivariate normal, as it's a GP. X is a constant variable right? So, both $f(x_t)$ and $e_t$ are MVNs, hence their sum is as well. Their sum is a GP as well. $\endgroup$
    – adityar
    Oct 12 '18 at 21:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.