Maybe this is not an appropriate question to ask here, but for certain types of what I'll call "statistical objects" that are related to distributions and are built upon them you have some steps to follow to create that statistical object. For example, if I wanted to create a mixture of 3 Gaussians a "mixture model", I would define my 3 Gaussians with there receptive parameters and sample some N values from each according to some probabilties.

What I want to know is what are the steps required to create a "process" such as a Gaussian process or a Dirichlet process for example. From my limited understanding I know that a Gaussian process is a distribution of distributions (where those distributions are themselves Gaussian).

One way that I can think of doing this is to generate a set of parameters mean and variances to sample from the "outside" process which then gives you a Gaussian with that mean and variance as it's parameters. But then the question becomes, how do I know the set of parameters that should be available to sample from ? Or rather, how do I define what that set should be? Do I simply define a range over them?


1 Answer 1


In the sequel I will only talk about Gaussian processes. Let us begin by 'creating' some examples by hand and sampling elements from them.

All we need to know for the 'by hand' part is the following theorem: We let $N(\mu, \Sigma)$ denote the multivariate normal distribution and write $X \sim N(\mu, \Sigma)$ for the fact that a random variable $X$ has such a distribution. Notice that is to be read in the 'general' way as in https://en.wikipedia.org/wiki/Multivariate_normal_distribution, i.e. random variables might be just simple point loadings (if $\Sigma = 0$).

Theorem: Let $X$ be a random variable in $\mathbb{R}^d$ and let $X \sim N(\mu, \Sigma)$. Take any matrix $A \in \mathbb{R}^{m \times d}$ then $AX$ also follows a multivariate normal distribution, namely $AX \sim N(A\mu, A\Sigma A^T)$.

Example 1: Gaussian Lines We explicitly construct a Gaussian Process $(X_t)_{t \in T}$ on the index set $T = \mathbb{R}$. We start by taking any univariately normally distributed variable $Y \sim N(\mu, \sigma)$. Then we simply put $$X_t = t \cdot Y$$ This is a Gaussian Process: Take any $t_1, ..., t_n \in \mathbb{R}$. We need to show that $(X_{t_1}, ..., X_{t_n})$ has a multivariate normal distribution. Since $(X_{t_1}, ..., X_{t_n}) = (t_1, ..., t_n) \cdot Y$ we are in the situation of the Theorem with $m=n$ and $d=1$.

How do we sample from that process? We sample by sampling a single value from $N(\mu, \sigma)$ as the value for $y$ and then we draw a plot with $t$ on the x-axis and $X_t(\omega)$ on the y-axis. As $X_t(\omega) = tY(\omega) = ty$ this is just a line. Let us do this a single time:

enter image description here

Ok, nice. What happens if we select $\mu=0, \sigma=1$ and do this often?

enter image description here

Aha... every sampled 'member' of this Gaussian Process is a line and by sampling a single value $y$ we actually sample their slope and since this is $N(0,1)$, the slopes 'gather around' $\mu=0$ and lie mostly in $[-1,1]$. That is why this process is called Gaussian Lines.

Example 2: Gaussian Planes The first example carries over naturally to planes instead of lines, i.e. now the index set is $T = \mathbb{R}^d$ and we take any single fixed RV $Y \sim N(\mu, \Sigma)$ in $\mathbb{R}^d$ and then we simply define $$ X_v(\omega) = \langle v, Y(\omega) \rangle = \sum_{i=1}^d v_i Y_i(\omega)$$. This will then give planes whose 'slopes' in the different directions are given by the sampled value of the 'single root cause' $y = Y(\omega)$.

Exercise: Compute the covariance and mean functions of the Gaussian Lines and planes.

General 'sampling' procedure Assume we are given a Gaussian Process $(X_t)_{t \in T}$ [in the form of its mean function $\mu(t) = E[X_t]$ and covariance function $k(t, s) = \text{Cov}(X_t, X_s)$] and we want to sample from it or, at least, get an idea of how samples from it look like. Let us take $T = \mathbb{R}$ as an illustration, the process works with more general examples as well. Choose any area, for example, $[-1,1]$ on which you want to take a look at the samples. Now take any sequence $-1 \leq t_0 < ... < t_N \leq 1$ in that area. The more points you choose, the more 'exact' will the image of a single member be. Now that we know that $$(X_{t_0}, ..., X_{t_N}) \sim N(\mu, \Sigma)$$ where $\mu = (\mu(t_0), ..., \mu(t_N))$ and $\Sigma = (k(t_i, t_j))_{i,j=0,...,N}$ we can take any computer algebra program and just sample a single element $(x_{t_0}, ..., x_{t_N})$ from that distribution and then connect the points $(t_i, x_{t_i})$ with each other on a plot. When we do this for the mean function $\mu = 0$ and the kernel (or covariance) function $k(t,s) = e^{-(t-s)^2}$ (maybe I forgot a factor in here somewhere like $e^{-(t-s)^2/2}$ or so but thats a minor difference) then we get the following picture:

enter image description here

In that sense, every member is a wiggly (but at least continuous) function on $\mathbb{R}$.

Now for the theoretical part: Let us take $T=\mathbb{R}$ again although the whole thing can be extended. First of all we need to see that a Process is more than just its marginal descriptions. What people sometimes write is 'A Gaussian process is uniquely determined by its mean and covariance function'. What do people mean by that? If we are given two Gaussian Processes $(X_t)$ and $(Y_t)$ and they have the same mean and covariance functions then ... they are the same simply because people seem to 'define' a process solely in terms of the mean and kernel functions. But that is not the whole truth! A Gaussian Process actually is the following thing:

A Gaussian A set of random variables $(X_t)$ from a measure space $\Omega$ to $\mathbb{R}$ together with a single probability measure $P$ on $\Omega$ such that for every finite sequence $t_1, ..., t_n$ and every collection of sets $A_1, ..., A_n$ all being in the respective (Borel) sigma algebra of $\mathbb{R}$, we have $$P(X_{t_1} \in A_1 ~\text{and}~ ... ~\text{and}~ X_{t_n} \in A_n) = \int_{A_1 \times ... A_n} f(x_1, ..., x_n) d(x_1, ..., x_n)$$ where $f$ is the density function of some $N(\mu, \Sigma)$.

In that sense, if $X$ and $Y$ are GPs with potentially different probability measures $P$ and $Q$ such that their mean and covariance functions coincide then also $P$ and $Q$ coincide in some sense (more precisely, $P_X = Q_Y$ on the space $\prod_{t \in \mathbb{R}} \mathbb{R}$ that we endow with the sigma algebra generated by the "finite cubes" $\prod_{t \notin \{t_1, ..., t_n\}} \mathbb{R} \times A_{t_1} \times ... \times A_{t_n}$).

Now on the other direction: How to construct the whole process though? Let us assume that we are generally given not mean and kernel functions but rather a collection of prpbability measures, i.e. for every finite sequence $t_1, ..., t_n$ in $\mathbb{R}$ we are given a probability measure $P_{t_1, ..., t_n}$ on $\mathbb{R}^n$. Assume that

  1. $P_{t_1, ..., t_n}(A_1 \times ... \times A_n) = P_{t_{\tau(1)}, ..., t_{\tau(n)}}(A_{\tau(1)} \times ... \times A_{\tau(n)})$ for every permutation $\tau$ of the set $\{1,...,n\}$.

  2. $P_{t_1, ..., t_n}(A_1 \times ... A_{n-1} \times \mathbb{R}) = P_{t_1, ..., t_{n-1}}(A_1 \times ... A_{n-1})$

(i.e. two consistency assumptions meaning that the marginal probability measures do not contradict themselves in the thing they want the process to do) then we can always construct a process with that single complicated probability measure that satisfies

$$P(X_{t_1} \in A_1 ~\text{and}~ ... ~\text{and}~ X_{t_n} \in A_n) = P_{t_1, ..., t_n}(A_1 \times ... \times A_n)$$

(see https://fabricebaudoin.wordpress.com/2012/03/25/lecture-5-the-daniell-kolmogorov-existence-theorem/ or any notes on the so-called Daniell-Kolmogorov existence theorem).

Since we can turn mean and covariance functions into such a collection we always obtain a Gaussian Process on $\mathbb{R}$ for any mean and covariance function.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.