Relationship between Gaussian process and Regression by supervised learning model like SVR Recently, I often see studies about Gaussian Process, and started to learn it. 
I'm familiar with regression models using machine learning such as Support Vector Regression. SVR learns a training data $\{(x_i, y_i)\}_i$ and predict $y_t$ from $x_t$.
Could you show the relationship between Gaussian Process and regression model by supervised learning such as SVR?
 A: There exists a very strong link between Gaussian process regression and kernel Ridge regression (also called Tikhonov regularization). Indeed, the posterior expectation you compute using Bayesian inference with prior $\mathcal{GP}(0,k)$ and additive noise model $\mathcal{N}(0,\eta^2)$ gives exactly the same predictions as what obtained using a kernel Ridge regression in the RKHS $\mathcal{H}_k$ of kernel $k$ and regularization parameter $\eta^2$, that is the solution of:
$${\arg\!\min}_{f\in\mathcal{H}_k} \sum_{i=1}^n\big(y_i-f(x_i)\big)^2 + \eta^2 \lVert f \rVert^2_{\mathcal{H}_k}\,.$$
Note that the Bayesian inferance also computes the posterior variance, for which the links with kernel regression are less clear.
For SVR you'll find some differences between the two predictions since the least-squares fitting term is replaced by the $\epsilon$-insensitive error function:
$$g_\epsilon(z) = \begin{cases}
|z|-\epsilon &\text{if }|z|\ge \epsilon\,,\\
0&\text{otherwise.}
\end{cases}$$
The value of $\epsilon$ intuitively controls the "sparsity" of the solution. And using absolute values instead of squares may render the solution more robust to outliers.
You can find additional details in the Chapter 6 of the famous book from Rasmussen and Williams.
