7
$\begingroup$

I have an assignment and I'm kinda confused with the terminology of prior in the context of Gaussian processes.

Let $f$ be the function of interest. The matrix $\textbf{X}$ is also a set of $N$ $M$-dimensional input vectors.

According to the description I have been provided, we can formulate the prior over the output of the function f using a Gaussian Process in the following way:

$p(f|\textbf{X}, \textbf{θ}) = Ν(\textbf{0}, k(\textbf{X}, \textbf{X}))$

where $θ$ is the hyperparameters of the kernel $k$.

What troubles me is that this has been called a prior but it depends on the data $\textbf{X}$.

Shouldn't the prior be independent on the data, expressing our belief about some property of the function through the kernel?

Is the notation/terminology used wrong or is there something I haven't understood in the notation?

$\endgroup$
1
  • $\begingroup$ Short answer: The GP is called a prior over functions because before even seeing the underlying function, you assume that it can be modelled well by a gaussian process. For example, by using a GP implies that we can model only smooth functions. You can use the same analogy to other models. For example, if you use a linear function or a polynomial function instead of a GP, those also represent priors over the functions you can model. $\endgroup$ Sep 13 '21 at 19:55
7
$\begingroup$

A Gaussian Processes is considered a prior distribution on some unknown function $\mu(x)$ (in the context of regression). This is because you're assigning the GP a priori without exact knowledge as to the truth of $\mu(x)$. Learning a GP, and thus hyperparameters $\mathbf\theta$, is conditional on $\mathbf{X}$ in $k(\mathbf{x},\mathbf{x'})$.

It is worth noting that prior knowledge may drive the selection, or even engineering of kernel functions $k(\mathbf{x},\mathbf{x'})$ to particular model at hand.

If using a completely Bayesian formulation (such as fitting using MCMC rather than maximum liklihood), one may incorporate additional prior knowledge on hyperparameters $\mathbf\theta$ if such knowledge is available.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.