15
$\begingroup$

I'm training Gaussian Process models on a relatively small data set, which have 8 input features and 75 input data.

I tried different kernels and find the following kernel (2 RBF + a white noise)works best.

from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import RBF, WhiteKernel

k1 = sigma_1**2 * RBF(length_scale=length_scale_1) 
k2 = sigma_2**2 * RBF(length_scale=length_scale_2)  
k3 = WhiteKernel(noise_level=sigma_3**2)  # noise terms

kernel = k1 + k2 + k3

I used 10-fold cv to calculate the R^2 score and find the averaged training R^2 is always > 0.999, but the averaged validation R^2 is about 0.65.

Looks like that the models are overfitted. I'm wondering what we could do to prevent overfit in Gaussian Process.

In linear regression, we can add regularization, and in neural network we can add regularization and dropout.

What about Gaussian Process?

Improve this question
$\endgroup$
1
  • $\begingroup$ You could increase the length scales or increase the sigma_3. That should help in my opinion $\endgroup$
    – sega_sai
    Commented Oct 24, 2018 at 22:50

2 Answers 2

Reset to default
9
$\begingroup$

Gaussian processes are sensible to overfitting when your datasets are too small, especially when you have a weak prior knowledge of the covariance structure (because the optimal set of hyperparameters for the covariance kernel often makes no sense).

Also, gaussian processes usually perform very poorly in cross-validation when the samples are small (especially when they were drawn from a space-filling design of experiment).

To limit overfitting:

  • set the lower bounds of the RBF kernels hyperparameters to a value as high as reasonably possible regarding your prior knowledge
  • try increasing (progressively) the noise kernel, or use sklearn's alpha parameter in GaussianProcessRegressor (increase the value corresponding to some training points where the GPR seems to overfit the most).
Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks Romain! I will try your suggestions and see how it works. Moreover, usually, what would be a good dataset size to use gaussian process? Could you offer some suggestions on this, too? $\endgroup$
    – Rocco
    Commented Oct 25, 2018 at 20:27
  • $\begingroup$ I think it mostly depends on the typical length scales and the size of the experimental space. A typical size (not necessarily enough) for a gaussian kernel with length scales $\sigma_i$ could be: $\prod\limits_{i=1}^{p}k\cdot (\frac{range(x_i)}{\sigma_i}+1)$ where $p$ is the number of dimensions, $range(x_i)$ is the feature space size in the $i$-th dimension, and $k$ is a typical density value. From my (small) experience, if your covariance kernel works well, a minimum value would be $k \sim 1$. $\endgroup$
    – Romain Reboulleau
    Commented Oct 26, 2018 at 11:58
  • $\begingroup$ Thanks! Please correct me if I'm wrong. I always scale my input features to a Normal distribution and assume I have 8 features. In such case, all $σ_i = 1$ and assume all $range(x_i) = 3$. So the equation above is $k^8 * 4^8 = 65536 * k^8$. If the covariance kernel works well, $k = 1$ and we need roughly $65536 $ samples. If not, we would need much more, since it is $k^8$. Thank you again! $\endgroup$
    – Rocco
    Commented Oct 26, 2018 at 20:22
  • $\begingroup$ Good hint about increasing the lower bound of the hyperparameters! $\endgroup$
    – Ken Grimes
    Commented Jul 26, 2021 at 15:44
6
$\begingroup$

Gaussian processes are very flexible models and prone to overfit. For 75 samples I'd suggest that you move to a simpler model where you can better interpret the output.

If you'd like to learn about different ways to apply regularization to GPs, I'd recommend taking a look at this presentation and this related question.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you very much for the reference and suggestions! As you mentioned, 75 samples might be too small for Gaussian Process. I'm wondering, empirically speaking, how many sample are needed if we want to train a Gaussian process model with good performance, saying we have 8 features. $\endgroup$
    – Rocco
    Commented Oct 25, 2018 at 20:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.