I'm reading this paper, and I've come to what seems to be a pretty crucial assumption:

Now, the n observations in an arbitrary data set, y = {y1, . . . , yn}, can always be imagined as a single point sampled from some multivariate (n-variate) Gaussian distribution, after enough thought.

I'm hoping someone can express the details of "enough thought".

Now, I can sort of see why this might be true. It seems somewhat vaguely related to the reason we use gaussian kernels in KDE and maybe even the Central Limit Theorem, but it's all a bit foggy to me.

Can anyone explain this?

  • 4
    $\begingroup$ "after enough thought" $\equiv$ handwaving $\endgroup$ – Cliff AB Apr 2 at 16:05
  • $\begingroup$ It's just saying that any vector valued observation can be thought as a realization of a multivariate gaussian processes. There's really not much to it. $\endgroup$ – Lucas Farias Apr 2 at 20:35
  • $\begingroup$ @LucasFarias I'm hoping someone can justify that statement. I could also think of it as a realization of a multivariate applesauce distribution, but why is it valid to think of it as gaussian? $\endgroup$ – timwiz Apr 2 at 20:40
  • $\begingroup$ The author is simply saying that, given you can see a set of observations as a single vector valued observation, it makes sense to think it might be generated by a multivariate applesauce distribution. $\endgroup$ – Lucas Farias Apr 2 at 20:46

By definition, a random process is a collection of random variables indexed by the elements of some set $\mathbb T$ which is typically $\mathbb R$ or $\mathbb Z$. Thus, the random process is the set $\{X(t)\colon t \in \mathbb T\}$ where $X(t)$ is the called the $t$-th random variable.

By definition, a Gaussian random process $\{X(t)\colon t \in \mathbb T\}$is a random process for which

For all choices of $n>0$ and all choices of time instants $t_1, t_2, \ldots, t_n \in \mathbb T$, $X(t_1), X(t_2,), \ldots, X(t_n)$ have a jointly Gaussian (also called a multivariate Gaussian) distribution.

Members of Nitpickers Anonymous please note that for $n = 1$, the sole random variable $X(t)$ (where $t\in \mathbb T$) has just a univariate Gaussian distribution and not a multivariate Gaussian distribution. So, the multivariate Gaussianity of $X(t_1), X(t_2,), \ldots, X(t_n)$ is actually baked into the definition of Gaussian random process.

"But, but, but," you splutter, "it says arbitrary data set, not a Gaussian random process." Well, the canonical model for arbitrary data sets is that they are (independent) samples from a Gaussian distribution and we don't abandon that unless someone beats us over the head and insists that it is not so. So, the data can be modeled as multivariate Gaussian (which, I remind those who march to the beat of a different drummer, includes independent Gaussian as a special case.)

Well, that's enough thought for today.

  • $\begingroup$ Clap clap, very amusing delivery! Is that first sentenced supposed to say "indexed by"? $\endgroup$ – timwiz Apr 2 at 21:32

This assumption is not universally valid (of course). Moreover in many cases it is not even necessary to make!

Relevant examples where it is obviously not valid are: strictly positive data (since a Gaussian has always a chance of being negative) or monotonic or convex data (same reason just for first and second derivatives).

That data is a realisation of a (stationary) Gaussian Field is a very strong assumption, which is not always necessary. Weaker assumptions will lead to weaker conclusions but in many cases these weaker conclusions are all you need.

Assumptions and possible conclusions in order of strength:

  1. Assumption: Data is from a stationary Gaussian Field. You are able to conclude: Hyperparameters from maximum likelihood and the full posterior/predictive distribution. Furthermore, the mean prediction is the best unbiased prediction in mean square.

  2. Assumption: Data is from a second order stationary process (i.e. mean and covariance function exist, full distribution not specified). Possible conclusions: Predictive variance, best linear(!) unbiased estimate for the mean function.

  3. Assumption: Data is deterministic, i.e. the problem is a pure interpolation problem. Possible conclusion: "Mean" or maybe better the interpolating function.

This explains why Gaussian Process Regression is applicable in fields (such as computer experiments or numerical analysis), where the normal or any other stochastic assumption does not make any sense.

For more details have a look at this nice overview: "Interpolation of Spatial Data - A Stochastic or a Deterministic Problem?"


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.