Without going to far into the details, I am using a Gaussian Process for the prediction of the posterior given by:

$$p(\mathbf{T}\vert\mathbf{X},\boldsymbol{\theta}) = \int p(\mathbf{T}\vert f)p(f\vert \mathbf{X},\boldsymbol{\theta})df$$

Above I am just performing a marginalization to merge the uncertainties in the likelihood and the prior, but I have an issue, the hyperparameters $\boldsymbol{\theta}$, what kind of condition do we have on these after the integral has been made? I have always considered these as parameters that can be rather tricky to learn and is more to be chosen with great care.

I mean, I have a hard time interpreting the question I'm supposed to answer, please, correct me if I'm wrong, but I suppose my goal is to choose hyperparameters in such a way that I maximize my posterior, but can we really interpret that as a condition on them?

  • $\begingroup$ You can actually learn the 'best' hyperparameter from the data before you predict, i.e. you maximize a likelihood function w.r.t. the data (see the slide "GP learning the kernel" in mlss2011.comp.nus.edu.sg/uploads/Site/lect1gp.pdf). However, you may overfit when doing so and the author claims that one can use Bayesian methods in order to cope for that but I have never done that before... You could also add some kind of regularization (which, however, introduces a new hyperparameter, namely the weight for the punishment factor in the likelihood) $\endgroup$ – Fabian Werner Dec 5 '18 at 7:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.