How can you detect if a Gaussian process is over-fitting? I am training a Gaussian process with an ARD kernel with lots of parameters by maximizing the marginal lielihood of the data, instead of cross-validation. 
I suspect that it is over-fitting. How can I test this suspicion in a Bayesian context?
 A: The simplest thing to do would be to fit a Gaussian process with the non-ARD equivalent covariance function (usually the RBF) and compare the test error rates.  For many problems an ARD covariance function performs worse than a non-ARD covariance function because of over-fitting in tuning the hyper-parameters.  As the RBF covariance is a special case of the ARD covariance, if the RBF performs better, it is a strong indication that the ARD kernel is over-fitting (start optimising the ARD coefficients at the optimal values for the corresponding RBF covariance, this is faster, and also helps to ensure that the problem with the ARD covariance is not just due to local minima in the marginal likelihood). This is a much bigger problem than is generally appreciated.
I've written a couple of papers on this:
G. C. Cawley and N. L. C. Talbot, Preventing over-fitting during model selection via Bayesian regularisation of the hyper-parameters, Journal of Machine Learning Research, volume 8, pages 841-861, April 2007 (pdf)
and
G. C. Cawley and N. L. C. Talbot, Over-fitting in model selection and subsequent selection bias in performance evaluation, Journal of Machine Learning Research, 2010. Research, vol. 11, pp. 2079-2107, July 2010 (pdf)
The first includes some experiments with GPs, which show that over-fitting in model selection is also a problem for GPs with marginal likelihood maximisation based model selection.
A more thorough analysis would be to evaluate the test error of the GP at each step in the process of optimising the marginal likelihood.  It is highly likely that you will get the classic hall mark of over-fitting, where the model selection criterion is monotonically decreasing, but the test error initially decreases, but then starts to rise again as the model selection criterion is over-optimised (c.f. Figure 2a in the 2010 JMLR paper).  
