Gaussian Process Regression/ Classification How do we estimate parameters of the model while performing Gaussian Process Regression or Classification? While performing regression, we estimate parameters such that the model is the best fit to the data. Thus, we try to increase the likelihood of the model, given the data. However, how do we estimate the parameters of a Gaussian Process regression process, where we know that the posterior will pass through all observed data points? In that case, every model passes through all observed data points. How do we differentiate between models? 
 A: There are cases in which we observe noise free data, such as Bayesian quadrature or Bayesian optimisation. 
Two options that can be used to tune the (hyper-)parameters are:
Marginal Likelihood: We may tune the (hyper-)parameters by maximising the log marginal likelihood:
$$\log P(y|x,\theta) = -0.5y^TK^{-1}y - 0.5\log|K| - c$$
Where $c$ is a constant, the first term is the data fit term and the second is the complexity penalty. As long as you are not adding $\sigma I$ (you will usually see $K^+ = K + \sigma I$) to your covariance matrix there will be a range of values of (hyper-)parameters that will interpolate your data. They vary by the rate they drop back to the mean function. 
Warning: noiseless data can cause numerical issues in the inversion of $K$, so a little bit of 'jitter' may be unavoidable.
Leave One Out (LOO): You can also use LOO, simply work out the log predictive probability for each of your data points and find the (hyper-)parameters of your kernel to maximise this.
$$LOO(\theta | x,y) = \sum_{i=0}^n \log P(y_i|x,y_{-i},\theta)$$
Both of these methods are outlined in GPML Chapter 5. 
