In GP regression we define a prior over functions in the form of a multivariate Gaussian distribution. After observing some data we can apply Bayes rule and we end up with a posterior distribution for predicting new inputs.

The prior distribution captures our initial beliefs via some function that computes the mean of our distribution and a kernel function through which we build the covariance matrix. This covariance matrix will capture any prior beliefs about how the random variables should behave in relation to one another.

How does one determine the dimensionality of this prior distribution? And how will the dimensionality of the prior affect our predictions? Will a larger covariance matrix result in assigning more weight to our prior beliefs since in higher dimensions we have more variables that behave in a similar way (defined via our kernel function)?


The prior distribution is always defined over the same set of random variables as the likelihood function and the posterior distribution. From this perspective, one doesn't technically "determine the dimensionality of the prior". There is only one set of random variables in the model; those that you aim to infer, and the prior must be defined on all such variables. This applies to Gaussian processes or any other model, whether it's parametric, non-parametric, or whether or not the set of random variables is finite or otherwise.

Of course, depending on your model, you may end up with a different set of unknowns and thus random variables (e.g. kernel coefficients in kernel regression will depend on the kernel type and number of kernels). Moreover, you may have a set of parameters governing the shape of the prior. Regardless, the higher the number of nuance parameters or variables in your full model, the higher the model complexity, and the higher the risk of overfitting if you don't have enough data.

So yes, you need to watch your model complexity, and you can make educated choices about what model (and thus parameters) is best for your problem. If you stick to the Bayesian methods a natural choice here is the Bayes factor, but cross validation and related techniques can also reveal whether or not you are overfitting and guide you through model choice.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.