Sklearn Gaussian Process Regressor Overfitting

Question

I am testing a set of regression algorithms and I'm having troubles with GPR. I have a set of 60 observations x 101 variables as a predictor (X) versus a set of 60 observations x 1 variable as a response (y). Given that the 101 elements of X represent multiple wavelengths of an hyperspectral dataset, I apply PCA to reduce collinearity.

I am running the following code

scaler=preprocessing.StandardScaler().fit(X)
xscaled=scaler.transform(X)
ncomps=2
pca=PCA(n_components=ncomps)
xpca=pca.fit_transform(xscaled)

later I run the GPR as following:

kernel=ConstantKernel(1.0) * RBF(1.0)
gpr=GaussianProcessRegressor(kernel=kernel,normalize_y=False).fit(xpca,y)
y_gpr=gpr.predict(xpca)
r2g=r2_score(y,y_gpr)
mseg=mean_squared_error(y,y_gpr)
print('GPR: R2: %0.4f, MSE: %0.4f' %(r2g,mseg))

And the print result is

GPR: R2: 1.0000, MSE: 0.0000

Now that's clearly not a good result, but I do not understand how to make this better. Any suggestion would be super helpful.

Answer 1

Some thoughts for regularising the model:

• Increase the length_scale= parameter of RBF()
• Increase the alpha= parameter value in GaussianProcessRegressor()
• Again for alpha, but specify alpha= as a vector instead, where you boost it for the data points where the model seems to overfit the most.

A plot of sample index vs. y, overlaid on sample index vs. prediction, could be helpful for visualisation.