# Sklearn Gaussian Process Regressor Overfitting

I am testing a set of regression algorithms and I'm having troubles with GPR. I have a set of 60 observations x 101 variables as a predictor (X) versus a set of 60 observations x 1 variable as a response (y). Given that the 101 elements of X represent multiple wavelengths of an hyperspectral dataset, I apply PCA to reduce collinearity.

I am running the following code

scaler=preprocessing.StandardScaler().fit(X)
xscaled=scaler.transform(X)
ncomps=2
pca=PCA(n_components=ncomps)
xpca=pca.fit_transform(xscaled)


later I run the GPR as following:

kernel=ConstantKernel(1.0) * RBF(1.0)
gpr=GaussianProcessRegressor(kernel=kernel,normalize_y=False).fit(xpca,y)
y_gpr=gpr.predict(xpca)
r2g=r2_score(y,y_gpr)
mseg=mean_squared_error(y,y_gpr)
print('GPR: R2: %0.4f, MSE: %0.4f' %(r2g,mseg))


And the print result is

GPR: R2: 1.0000, MSE: 0.0000

Now that's clearly not a good result, but I do not understand how to make this better. Any suggestion would be super helpful.

• Increase the length_scale= parameter of RBF()
• Increase the alpha= parameter value in GaussianProcessRegressor()
• Again for alpha, but specify alpha= as a vector instead, where you boost it for the data points where the model seems to overfit the most.
• Thank you! I am trying to optimize this parameter with a gridsearch, but it seems that I cannot iterate over the length_scale. Even if I run it like parameters={'alpha': [0.0001, 0.001,0.1,0.5,1,1.5],'kernel__length_scale':[100,1000,100000]} gpr=GaussianProcessRegressor(kernel=RBF(length_scale_bounds="fixed")) optm=GridSearchCV(gpr,parameters) I still end up with GaussianProcessRegressor(kernel=RBF(length_scale=1)) Commented Dec 5, 2023 at 16:17
• You're welcome. I assume you run optm.fit(...)? After that, the best parameters found will be in optm.best_params_, and the best fitted model (refitted on all the data) in optm.best_estimator_. Running print(optm.best_estimator_.get_params()) or print(optm.best_params_) will report the best parameters. Commented Dec 5, 2023 at 18:00
• About the search...another way of doing it when you don't know the right value is to use RandomizedSearchCV, which works best of you tell it the distribution to search over rather than a fixed list of values, e.g. parameters={'kernel__length_scale': scipy.stats.loguniform(100, 100_000), 'alpha': scipy.stats.loguniform(1e-4, 1.5}. Your approach is valid of course, but if you needed to search more finely the randomized approach takes less time. Commented Dec 5, 2023 at 18:07