How to prevent overfitting in Gaussian Process

I'm training Gaussian Process models on a relatively small data set, which have 8 input features and 75 input data.

I tried different kernels and find the following kernel (2 RBF + a white noise)works best.

from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import RBF, WhiteKernel

k1 = sigma_1**2 * RBF(length_scale=length_scale_1)
k2 = sigma_2**2 * RBF(length_scale=length_scale_2)
k3 = WhiteKernel(noise_level=sigma_3**2)  # noise terms

kernel = k1 + k2 + k3


I used 10-fold cv to calculate the R^2 score and find the averaged training R^2 is always > 0.999, but the averaged validation R^2 is about 0.65.

Looks like that the models are overfitted. I'm wondering what we could do to prevent overfit in Gaussian Process.

In linear regression, we can add regularization, and in neural network we can add regularization and dropout.

• You could increase the length scales or increase the sigma_3. That should help in my opinion – sega_sai Oct 24 '18 at 22:50

Gaussian processes are very flexible models and prone to overfit. For 75 samples I'd suggest that you move to a simpler model where you can better interpret the output.

If you'd like to learn about different ways to apply regularization to GPs, I'd recommend taking a look at this presentation and this related question.

• Thank you very much for the reference and suggestions! As you mentioned, 75 samples might be too small for Gaussian Process. I'm wondering, empirically speaking, how many sample are needed if we want to train a Gaussian process model with good performance, saying we have 8 features. – Rocco Oct 25 '18 at 20:23

Gaussian processes are sensible to overfitting when your datasets are too small, especially when you have a weak prior knowledge of the covariance structure (because the optimal set of hyperparameters for the covariance kernel often makes no sense).

Also, gaussian processes usually perform very poorly in cross-validation when the samples are small (especially when they were drawn from a space-filling design of experiment).

To limit overfitting:

• set the lower bounds of the RBF kernels hyperparameters to a value as high as reasonably possible regarding your prior knowledge
• try increasing (progressively) the noise kernel, or use sklearn's alpha parameter in GaussianProcessRegressor (increase the value corresponding to some training points where the GPR seems to overfit the most).
• Thanks Romain! I will try your suggestions and see how it works. Moreover, usually, what would be a good dataset size to use gaussian process? Could you offer some suggestions on this, too? – Rocco Oct 25 '18 at 20:27
• I think it mostly depends on the typical length scales and the size of the experimental space. A typical size (not necessarily enough) for a gaussian kernel with length scales $\sigma_i$ could be: $\prod\limits_{i=1}^{p}k\cdot (\frac{range(x_i)}{\sigma_i}+1)$ where $p$ is the number of dimensions, $range(x_i)$ is the feature space size in the $i$-th dimension, and $k$ is a typical density value. From my (small) experience, if your covariance kernel works well, a minimum value would be $k \sim 1$. – Romain Reboulleau Oct 26 '18 at 11:58
• Thanks! Please correct me if I'm wrong. I always scale my input features to a Normal distribution and assume I have 8 features. In such case, all $σ_i = 1$ and assume all $range(x_i) = 3$. So the equation above is $k^8 * 4^8 = 65536 * k^8$. If the covariance kernel works well, $k = 1$ and we need roughly $65536$ samples. If not, we would need much more, since it is $k^8$. Thank you again! – Rocco Oct 26 '18 at 20:22