# Equivalence between Gaussian Process Regression and Kernel Ridge Regression

Consider the model $$y(\mathbf{X}) = f(\mathbf{X}) + \epsilon,$$ where $$\mathbf{X}$$ is a given $$n\times D$$ matrix, and where $$\epsilon \sim \mathcal{N}(0, \sigma^{2}I_{N})$$ is iid Gaussian and is the noise.

First, by modeling the function $$f$$ as a GP, such that $$f(\cdot)\sim GP(0, K(\cdot, \cdot))$$ where K is a chosen covariance function. It can be shown (see for instance Rasmussen & Williams, 2006) that the prediction for a new point $$\mathbf{X}$$ is normal distributed so that $$f(\mathbf{x})|y(\mathbf{X}),\mathbf{X},\mathbf{x} \sim \mathcal{N}(K(\mathbf{x}, \mathbf{X})[K(\mathbf{X}, \mathbf{X}) + \sigma^{2}I_N]^{-1}\mathbf{Y}, K(\mathbf{x}, \mathbf{x})-K(\mathbf{x}, \mathbf{X})[K(\mathbf{X}, \mathbf{X}) + \sigma^{2}I_N]^{-1}K(\mathbf{x}, \mathbf{X})).$$

Now, instead of modeling the function with a GP, one wants to minimize the following Kernel Ridge Regression (KRR) equation: $$\min_{f} \sum\limits_{i=1}^{N}(f(\mathbf{X}_i) - y(\mathbf{X}_i))^{2} + \lambda||f||^{2}_{H_K},$$ where $$H_K$$ is the RKHS associated with the kernel $$K$$. The solution of this minimization is: $$f(\mathbf{x}) = K(\mathbf{x}, \mathbf{X})[K(\mathbf{X}, \mathbf{X}) + \lambda I_N]^{-1}\mathbf{Y},$$ which is equal to the predictive mean of the Gaussian process (under the assumption that $$\lambda = \sigma_N^{2}$$).

Now my questions:

1. It is shown in Rasmussen & Williams, 2006 that if one takes the exponential of minus the KKR equation given above, that is: $$exp(-\frac{1}{2\lambda}\sum\limits_{i=1}^{N}(f(\mathbf{X}_i) - y(\mathbf{X}_i))^{2}) \times exp(-\frac{1}{2}||f||^{2}_{H_K})$$ it is equal to the product of a Gaussian likelihood (the least squares term) times a Gaussian process prior on $$f$$. Why does the term $$\exp(-\frac{1}{2}||f||^{2}_{H_K})$$ correspond to a Gaussian process prior on $$f$$?

2. How do you show that $$\max_f exp(-\frac{1}{2\lambda}\sum\limits_{i=1}^{N}(f(\mathbf{X}_i) - y(\mathbf{X}_i))^{2}) \times exp(-\frac{1}{2}||f||^{2}_{H_K})$$ is equal to the predictive mean of the Gaussian process prior?

• Nice question(s). Looks like there are two questions. Commented Oct 27, 2022 at 19:04