Gaussian processes refer to stochastic processes whose realization consists of normally distributed random variables, with the additional property that any finite collection of these random variables have a multivariate normal distribution. The machinery of Gaussian processes can be employed in regression and classification problems.
Overview
Gaussian processes refer to stochastic processes whose realization consists of normally distributed random variables, with the additional property that any finite collection of these random variables have a multivariate normal distribution. The machinery of Gaussian processes can be employed in regression and classification problems.
Formulation
We consider a Gaussian process as an infinite version of a multivariate Gaussian distribution and has two parameters:
- mean function $\mu$ corresponding to the mean vector
- positive definite covariance or kernel function $K$ corresponding to a positive definite covariance matrix
Consider $x_1,...x_n$ as a finite collection of points in $\mathcal{X}$. For a Gaussian process over function $f \in \mathcal{H}$ the probability density of $\mathbf{f} = \\{f(x_1),...,f(x_n)\\}^T$ is a multivariate normal:
$$\mathbf{f} \sim MVN({\bf \mu}, {\bf \Sigma})$$
with:
- mean vector $\mathbf{\mu} = \\{\mu(x_1),...,\mu(x_n)\\}^T$
- covariance $\mathbf{\Sigma}_{ij} = K(x_i,x_j)$
- $\mu(x) = E[f(x)]$
- $K(x_i,x_j) = E[(f(x_i) - \mu(x_i))(f(x_j) - \mu(x_j)]$
Software Packages
mlegp,GPfitandCGPfor R.GPy,pygpandgaussian_processesfor Python
References
- Gaussian Processes Web Site is an extensive resource on the subject. It hosts a free copy of the book Gaussian Processes for Machine Learning (Carl Edward Rasmussen and Christopher K. I. Williams) for download.
- Introductory machine learning oriented video lectures by Nando de Freitas and John Cunningham.
