Why is the best prediction for a gaussian process a linear prediction? I want to understand why, for Gaussian processes, the best prediction is linear. I do not understand its proof. 
For Gaussian process $X(t), t\in I $ the best prediction is linear.
Proof:
We only show the theorem for $ J \subset I $ with $\vert J \vert < \infty$. Let $J=\{ t_1,\dots t_m\},t_0 \in I \backslash J$ and $Y (t_0)= \sum_{i=1}^m a_i X (t_i) +a$ the best linear prediction of $X (t_0) $ given $ X (t_1),\dots X (t_m)$. Therefore $X (t_0)-Y (t_0) \perp 1,X(t_1),\dots ,X (t_m)$.
We orthogonalize $ X(t_1),\dots ,X (t_m) $ with result $1,X_1,\dots X_r$. Then $X (t_0)- Y (t_0),X_1,\dots X_r $ is a gaussian vector with pairwise uncorrelated components, i.e. orthogonal and with an other theorem follows that they are independent too.  This means that $X (t_0)-Y (t_0)$ and $g (1,X (t_1),\dots X (t_m))$ are independent for every measurable $g:R^{m+1}\to R, E g^2((1,X (t_1),\dots X (t_m))< \infty $. 
Here is what I do not get: Therefore $E( (X (t_0)-Y (t_0)g(1,X(t_1),\dots ,X (t_m))=0$ and with that it follows that $ E (X (t_0)\vert 1,X (t_1),\dots X (t_m))=Y (t_0) $
Could someone explain this proof, and especially why the last two lines hold true?
 A: The answer consists of multiple parts:


*

*Explanation of the proof

*Explanation of why $E[(X(t_0) - Y(t_0)) g(1,X(t_1), ..., X(t_n))] = 0$

*Explanation of why $E[X(t_0) | X(t_1), ..., X(t_n)] = Y(t_0)$



*

*There is a slight chance of misunderstanding of capital letters ($X, Y, ...$) and lowercase letters ($x, y, ...$). The latter ones means 'concrete values of random variables given a particular $\omega \in \Omega$ [the probability space] while the first notation refers to the random variables themselves as functions from $\Omega$ to $\mathbb{R}$. The proof is about the following: Suppose that we have decided to model some regression problem using Gaussian processes and we are given concrete data at the points $t_1, ..., t_n$. This could be stock prices at different times, it could be performance values of hyperparameters (when one uses them for optimizing the architecture of Neural Nets or so), it could be house prices at different parameters, ... i.e. whatever your regression problem is about, this is the training dataset. Now we are being asked to say what value we want to predict for the new data $t_0$. The proof sais that instead of doing complicated computations with common densities or so, one can simply run a linear regression on the data $x_{t_1}, ..., x_{t_n}$ in order to get the best possible prediction for $X(t_0)$. So now you should wonder: Ok, but the proof actually does not show that $Y(t_0)$ is the best prediction because it only shows that $E[X(t_0)|X(t_1), ..., X(t_n)] = Y(t_0)$, how does that fit together? One needs to know one thing from statistical learning theory: Given in general some regression problem (i.e. a target variable $X_{\text{target}} = f(X_{\text{features}})$) where we try to guess $f$ with respect to the error metric $(\text{prediction} - \text{actual value})^2$ from training data of the form $x_{\text{target}, i} = f(x_{\text{features}, i})$, the best possible prediction over ALL models (i.e. neural nets, svm/svr, gradient boosting, whatever) is actually just $E[X_{\text{target}}|x_{\text{features}}]$ (i.e. the factorization of the conditional expectation of the target variable evaluated at the new feature vector that we are given for predicting the new target value). You can find a proof here: Lecture notes in https://www.win.tue.nl/~rmcastro/2DI70/, Prop. 2.2.1.
Remark that we can hardly ever use this as usually, we do not know the underlying probability distribution. However, in the case of Gaussian Processes we explicitly modeled the underlying probability distribution, so this is one of the rare cases where we can actually say something about this best possible prediction.

*That is actually easy and more general: Given two random variables $X, Y$ that are independent, we have $E[XY] = E[X]E[Y]$. You can find the proof on the main Wikipedia page on expectations: click.

*What is $E[X(t_0) - Y(t_0) | X(t_1), ..., X(t_n)]$? Well, the general definition for a conditional expectation is the following: Any function $f : \Omega \to \mathbb{R}$ that satisfies $\int_C X(\omega) d\omega = \int_C f(\omega) d\omega$ for every set $C \in \sigma(Y)$ is called $E[X|Y]$. Here, $\sigma(Y)$ is the smallest sigma algebra that makes $Y$ measurable, i.e. $\sigma(Y) = \{Y^{-1}(B) : B \in \mathcal{B}\}$ where $\mathcal{B}$ is the sigma algebra on the space $Y$ maps into. Assume that we have that $E[X \cdot g(Y_1, ..., Y_n)] = 0$ for all measurable functions $g$ then it also follows that $E[X|Y_1, ..., Y_n] = 0$: We need to show that $0$ is a conditional expectation, i.e. $\int_C X d\omega = 0$ for all $C \in \sigma(Y_1, ..., Y_n)$. I strongly believe that this sigma algebra is generated by sets $C = C_1 \times ... \times C_n$ where $C_i \in \sigma(Y_i)$, i.e. $C_i = Y_i^{-1}(B_i)$. Then putting $g(y_1, ..., y_n) = \mathbf{1}_{y_1 \in C_1} \cdot ... \cdot \mathbf{1}_{y_n \in C_n}$ yields...
\begin{align*}
  0 &= E[X \cdot g(Y_1, ..., Y_n)] \\
    &= \int_\Omega X \cdot g(Y_1, ..., Y_n) d\omega \\
    &= \int_C X d\omega
\end{align*}
(because $g$ is $1$ iff. $Y_1(\omega) \in C_1$ and ... and $Y_n(\omega) \in C_n$).
That means: Since $E[X(t_0) - Y(t_0) | g(X(t_1), ..., X(t_n))] = 0$ for all $g$ measurable, we have $0 = E[X(t_0)|X(t_1), ..., X(t_n)] - E[Y(t_0)|X(t_1), ..., X(t_n)]$ but the latter term is nothing else than $Y(t_0)$ due to the linearity of the conditional expectation and due to the fact that $E[X_i|X_1, ..., X_n] = X_i$ (generally speaking for all random variables) because $X_i$ obviously is a function satisfying the weird universal property that conditional expectations need to have.
I think that one can conclude that a little easier (without so much measure theory) by just explicitly computing the conditional density $p(x_{t_0} | x_{t_1}, ..., x_{t_n})$ (which we can do because $X(t_0), ..., X(t_n)$ is a multivariate Gaussian but beware: its an ugly computation) and then using $E[g(X)|Y=y] = \int g(x) p(x|y)$...
