Gaussian Process Regression: Normalization of data worsens fit. Why? I've run into situations where using in Gaussian Process Regression doing normalization (which is recommended in many places) yields much worse fit.
For example, for the data given by x=[20, 20, 140, 140] and y=[740, 680, 1260, 1200] this is the fit I get using a RBF kernel + white noise on the raw data:

WhiteKernel(noise_level=1.8e+03) + 1e+03**2 * RBF(length_scale=224)
And this is the fit I get using normalized data :

WhiteKernel(noise_level=0.03) + 0.99**2 * RBF(length_scale=0.018)
In this case the length scale is obviously too narrow and the noise component too big.
I made a plot of the log marginal likelihood for both GPR models, I marked with a red star the optimal values encountered. Both plot seem pretty similar save for a shifting due to the scaling but in the normalized case, there is some "plateau" at the optimal noise-level=0.03 where the LML is mostly constant and one could increase the length scale (which would make the fit better) but the optimization converges to the wrong smaller value of length scale.

In case someone is interested in the scikit-learn code, check here, but I tried this with GPy as well and got basically the same results so I don't think this is implementation specific.
Do someone has an idea what is going on? How can I ake GPR more robust against scaling issues?
 A: Those four points allow too much degrees of freedom for the hyperparameters to change.

*

*In your first case, you get some Gaussian curve with a scale of 224 and the slope of the curve is able to cross through the two points. So here the interpretation is that the points originate from a very broad bump.


*In your second case, you get very sharp peaks from white noise and a Gaussian curve with a very short scale of 0.018. So here the interpretation is that the points originate from two sharp peaks.

Both situations are very good fits for the points. Possibly there are multiple optima or the convergence is not very easy. Then the optimizer is not able to choose well between the different situations and a small change in scaling, the normalization, can change the result.
Another effect
One particular effect is that the normalisation turns one set of two points negative and the other two points positive. The fit with a broad Gaussian curve of scale 224 is not possible anymore.
You see this more extreme with these data
# Training data
x_tr = np.array( [20, 20, 140, 140, 260, 260, 400, 400] ).reshape(-1, 1)
y_tr = np.array( [1025, 1027, 1716, 1723, 1986, 2000, 1515, 1509] ).reshape(-1, 1)


The normalizing changes the assumptions about the mean of the data.
