2
$\begingroup$

Having studies some youtube lectures and blogs, I kind of have an understand about Gaussian Process,however, I still wonder what prior and likelihood are in Gaussian Process. It is sort of confusing to me. If you have a clear understanding about it,please drop me an explanation.

$\endgroup$
5
$\begingroup$

Without knowing exactly what context you have in mind, Gaussian processes are commonly used as the prior in Bayesian nonparametric regression.

Linear regression

The ordinary linear regression model is $$y_i=\beta_0+\beta_1x_i+\varepsilon_i,\quad\varepsilon_i\sim\textrm{N}(0,\,\sigma^2).$$ So we are modeling the conditional distribution (likelihood) $p(y_i\,|\,x_i,\,\boldsymbol{\beta})$ as Gaussian $$y_i\,|\,x_i,\,\boldsymbol{\beta}\sim\textrm{N}(\beta_0+\beta_1x_i,\,\sigma^2),$$ and we are modeling the conditional mean as a linear function of $x_i$: $$\mathbb{E}[y_i\,|\,x_i,\,\boldsymbol{\beta}]=\beta_0+\beta_1x_i.$$ In this model the coefficients $\boldsymbol{\beta}$ are unknown, so to do inference from a Bayesian point of view, we place a prior on $\boldsymbol{\beta}$ and try to access its posterior distribution.

Nonparametric regression

An example of a nonparametric regression model is $$y_i=f(x_i)+\varepsilon_i,\quad\varepsilon_i\sim\textrm{N}(0,\,\sigma^2).$$ So we are still modeling the conditional distribution (likelihood) $p(y_i\,|\,x_i)$ as Gaussian $$y_i\,|\,x_i\sim\textrm{N}(f(x_i),\,\sigma^2),$$ but we are no longer making any hard assumptions about the functional form of the conditional mean. It's just "some function" $f(x)$: $$\mathbb{E}[y_i\,|\,x_i]=f(x_i).$$ In this model the entire conditional mean function $f$ is unknown, so to do inference from a Bayesian point of view we need to place a prior probability distribution on the space of functions $f:\mathbb{R}\to\mathbb{R}$.

Gaussian processes are a prior on the space of functions

A stochastic process is commonly defined as an indexed collection of random variables. So if $X$ is some indexing set, and for each $x\in X$, $Z_x$ is a random variable, then the collection $\{Z_x:x\in X\}$ is a stochastic process. But as noted in the wikipedia article, stochastic processes have an alternative interpretation as "random functions" -- that is, as probability distributions on the space of functions $f:X\to\mathbb{R}$. In nonparametric regression we want to place a probability distribution on the space of functions $f:\mathbb{R}\to\mathbb{R}$, so we can do that by writing down a stochastic process indexed by $\mathbb{R}$. A common choice of stochastic process for this is a Gaussian process, which is any stochastic process whose finite dimensional distributions are Gaussian.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.