What does the covariance matrix of a Gaussian Process look like? Let's assume $\{X_t\}_{t \geq0} $ is a Gaussian Process with a certain covariance matrix. In order for this process to be stationary the mean vector should not depend on $t$ and the covariance function should not depend on $t$. However, what additional restrictions should the covariance have?
In order for the process to be stationary, the variance of each realization should be the same. So,that means that the diago$nal elements of the covariance matrix should all be equal to the commn variance. Am I wrong? In general what does the covariance matrix of a stationary Gaussian process look like?
 A: Here is a somewhat informal explanation:
The covariance matrix for a Gaussian process is a gram matrix obtained by evaluating some kernel function $k(x, x')$ pairwise between a set of observations.
Stationary in the context of a Gaussian process implies that the covariance between two points, say $x$ and $x'$, would be identical to the covariance between the same two points perturbed by some $\Delta$, i.e:
$k(x, x') = k(x+\Delta, x' + \Delta)$
This implies that the hyper-parameters of $k$ (if they exist) do not vary across the index (here $x$). As an example, the popular exponetiated quadratic (also called the squared exponential, or "RBF") kernel is stationary:
$k(x,x') = \alpha^2 e^{ \frac{- (x - x')^2}{2 l^2} } + \delta_{ij} \sigma_0^2$
($\delta_{ij}$ being the kronecker delta)
Because the hyperparameters ($\alpha, l, \sigma_0$) have no dependency on the index, $x$. 
If, for example, the lengthscale $l$ would be permitted to vary over $x$, the covariance between any two points $k(x,x')$ would not necessarily be the same as $k(x + \Delta,x'+ \Delta)$.
If the covariance kernel is stationary, one can see that for the diagonal elements of the covariance matrix we will be evaluating $k(x,x)$. Since the pairwise distance between $x$ and itself is zero, in the case of the kernel above, the covariance collapses to:
$k(x,x) = \alpha^2 +  \sigma_0^2$
Implying all diagonal elements of the unconditional covariance matrix will be identical, as neither $\alpha$ nor $\sigma_0$ depend on $x$.
A: The variance covariance matrix of a stationary Gaussian process should have the same value for all its diagonal elements. Its auto-covariance should also be the same through time. In other words, the off-diagonal lines that are parallel to the diagonal should have the same values in each line. And obviously, the matrix needs to be positive semi-definite.
