# understanding exchangeability in GP regression

It is well known that exchangeability refers to the following property $$p(X_1,\dots, X_n) = p(X_{\pi(1)}, \dots, X_{\pi(n)})$$ for any finite $$n$$ and a permutation $$\pi$$ when we have an infinite sequence of random variables $$X_1, X_2, \dots$$. So, for instance, when $$n=2$$, I can write that $$p(X_1=1, X_2=0) = p(X_1=0, X_2=1)$$.

Gaussian processes are also exchangeable. Let's say we have two varibles $$X_1$$ and $$X_2$$ at time $$t=1$$ and $$t=2$$ respectively. A zero-mean GP with a kernel $$k$$ gives the following density:

$$p(X_1, X_2) = \mathcal N(\mathbf 0, \begin{bmatrix} k(X_1,X_1) & k(X_1,X_2) \\ k(X_2,X_1) & k(X_2,X_2) \end{bmatrix})$$

or, if the order is permuted:

$$p(X_2, X_1) = \mathcal N(\mathbf 0, \begin{bmatrix} k(X_2,X_2) & k(X_2,X_1) \\ k(X_1,X_2) & k(X_1,X_1) \end{bmatrix})$$

In this case $$p(X_1=1, X_2=0) \ne p(X_1=0, X_2=1)$$, so it must be wrong to think about exchangeability in GPs this way. What is the correct way of thinking then?

• Under a stationary gp (where the kernel only depends on the distance between $X$ and $Y$) $k(X,X) = \sigma^2$ for all $X$ and $k(X,Y) = k(Y,X)$. If you plug values into a stationary kernel you will see the covariance matrix will be the same regardless of the ordering of the $X_i$ – jcken Jun 14 at 16:42