Gaussian Processes: A Crucial Assumption? I'm reading this paper, and I've come to what seems to be a pretty crucial assumption:

Now, the $n$ observations in an arbitrary data set, $y = \{y_1, \dots, y_n\}$,
can always be imagined as a single point sampled from some
multivariate ($n$-variate) Gaussian distribution, after enough thought.

I'm hoping someone can express the details of "enough thought".
Now, I can sort of see why this might be true. It seems somewhat vaguely related to the reason we use Gaussian kernels in KDE and maybe even the Central Limit Theorem, but it's all a bit foggy to me.
Can anyone explain this?
 A: This assumption is not universally valid (of course). Moreover, in many cases it is not even necessary to make!
Relevant examples where it is obviously not valid are: strictly positive data (since a Gaussian has always a chance of being negative) or monotonic or convex data (same reason just for first and second derivatives).
That data is a realisation of a (stationary) Gaussian Field is a very strong assumption, which is not always necessary.  Weaker assumptions will lead to weaker conclusions but in many cases these weaker conclusions are all you need.
Assumptions and possible conclusions in order of strength:

*

*Assumption: Data is from a stationary Gaussian Field.
You are able to conclude: Hyperparameters from maximum likelihood and the full posterior/predictive distribution. Furthermore, the mean prediction is the best unbiased prediction in mean square.


*Assumption: Data is from a second-order stationary process (i.e. mean and covariance function exist, full distribution not specified).
Possible conclusions: Predictive variance, best linear(!) unbiased estimate for the mean function.


*Assumption: Data is deterministic, i.e. the problem is a pure interpolation problem.
Possible conclusion: "Mean" or maybe better the interpolating function.
This explains why Gaussian Process Regression is applicable in fields (such as computer experiments or numerical analysis) in which the normal or any other stochastic assumption does not make any sense.
For more details have a look at this nice overview: "Interpolation of Spatial Data - A Stochastic or a Deterministic Problem?".
A: By definition, a random process is a collection of random variables indexed by the elements of some set $\mathbb T$ which is typically $\mathbb R$ or $\mathbb Z$. Thus, the random process is the set $\{X(t)\colon t \in \mathbb T\}$ where $X(t)$ is the called the $t$-th random variable.  
By definition, a Gaussian random process $\{X(t)\colon t \in \mathbb T\}$is a random process for which 

For all choices of $n>0$ and all choices of time instants $t_1, t_2, \ldots, t_n \in \mathbb T$, $X(t_1), X(t_2,), \ldots, X(t_n)$ have a jointly Gaussian (also called a multivariate Gaussian) distribution.

Members of Nitpickers Anonymous please note that for $n = 1$, the sole random variable $X(t)$ (where $t\in \mathbb T$) has just a univariate Gaussian distribution and not a multivariate Gaussian distribution.  So, the multivariate Gaussianity of $X(t_1), X(t_2,), \ldots, X(t_n)$ is actually baked into the definition of Gaussian random process. 
"But, but, but," you splutter, "it says arbitrary data set, not a Gaussian random process."  Well, the canonical model  for arbitrary data sets is that they are (independent) samples from a Gaussian distribution and we don't abandon that unless someone beats us over the head and insists that it is not so. So, the data can be modeled as multivariate Gaussian (which, I remind those who march to the beat of a different drummer, includes independent Gaussian as a special case.) 
Well, that's enough thought for today.
