I'm training Gaussian Process models on a relatively small data set, which have 8 input features and 75 input data.
I tried different kernels and find the following kernel (2 RBF + a white noise)works best.
from sklearn.gaussian_process import GaussianProcessRegressor from sklearn.gaussian_process.kernels import RBF, WhiteKernel k1 = sigma_1**2 * RBF(length_scale=length_scale_1) k2 = sigma_2**2 * RBF(length_scale=length_scale_2) k3 = WhiteKernel(noise_level=sigma_3**2) # noise terms kernel = k1 + k2 + k3
I used 10-fold cv to calculate the R^2 score and find the averaged training R^2 is always > 0.999, but the averaged validation R^2 is about 0.65.
Looks like that the models are overfitted. I'm wondering what we could do to prevent overfit in Gaussian Process.
In linear regression, we can add regularization, and in neural network we can add regularization and dropout.
What about Gaussian Process?