5
$\begingroup$

I've run into situations where using in Gaussian Process Regression doing normalization (which is recommended in many places) yields much worse fit.

For example, for the data given by x=[20, 20, 140, 140] and y=[740, 680, 1260, 1200] this is the fit I get using a RBF kernel + white noise on the raw data:

GPR_raw WhiteKernel(noise_level=1.8e+03) + 1e+03**2 * RBF(length_scale=224)

And this is the fit I get using normalized data :

GPR_normalized WhiteKernel(noise_level=0.03) + 0.99**2 * RBF(length_scale=0.018)

In this case the length scale is obviously too narrow and the noise component too big.

I made a plot of the log marginal likelihood for both GPR models, I marked with a red star the optimal values encountered. Both plot seem pretty similar save for a shifting due to the scaling but in the normalized case, there is some "plateau" at the optimal noise-level=0.03 where the LML is mostly constant and one could increase the length scale (which would make the fit better) but the optimization converges to the wrong smaller value of length scale.

LML

In case someone is interested in the scikit-learn code, check here, but I tried this with GPy as well and got basically the same results so I don't think this is implementation specific.

Do someone has an idea what is going on? How can I ake GPR more robust against scaling issues?

$\endgroup$
7
  • 1
    $\begingroup$ A naive point: Just to see this is not a sample size artefact, have you tried testing this with much more than 4 points? Maybe take 1000 points. $\endgroup$ Oct 8, 2021 at 9:14
  • 1
    $\begingroup$ You are using completely different kernels in both cases. If you want to compare anything, you need at least to use similar settings in both scenarios. If you change both data and hyperparameters, there is nothing to compare. $\endgroup$
    – Tim
    Oct 8, 2021 at 9:47
  • $\begingroup$ Mehmet: Well of course the theory says that if number of observations grows then regardless of kernel the underlying function can be approximated. In my application I can't arbitrarily increase the number of observations since I'm modelling the results of very expensive experiments $\endgroup$
    – Ken Grimes
    Oct 8, 2021 at 10:05
  • $\begingroup$ Tim: No in both cases it's the same kernel (White noise + constant*RBF), the hyperparameters are different because the result of the hyperparameter optimization is different. That's my point, that hyperparameter optimization works well in the non-normalized case but breaks down in the normalized case $\endgroup$
    – Ken Grimes
    Oct 8, 2021 at 10:08
  • $\begingroup$ The typical way of fitting a GP is likelihood maximisation of the hyperparameters, which (essentially) tries to summarise the distribution of candidate hyperparameters by its mode. This distribution can be more or less "peaked" (meaning the mode is a more or less effective summary) depending on a lot of different factors, but with such a small amount of data you can probably expect it to be quite flat in the sense that a well defined optimum doesn't necessarily exist (or is at least challenging to find numerically). You should try using with more data or playing with the initialisation. $\endgroup$
    – rxFt20
    Oct 8, 2021 at 11:12

1 Answer 1

1
$\begingroup$

Those four points allow too much degrees of freedom for the hyperparameters to change.

  • In your first case, you get some Gaussian curve with a scale of 224 and the slope of the curve is able to cross through the two points. So here the interpretation is that the points originate from a very broad bump.

  • In your second case, you get very sharp peaks from white noise and a Gaussian curve with a very short scale of 0.018. So here the interpretation is that the points originate from two sharp peaks.

fits

Both situations are very good fits for the points. Possibly there are multiple optima or the convergence is not very easy. Then the optimizer is not able to choose well between the different situations and a small change in scaling, the normalization, can change the result.

Another effect

One particular effect is that the normalisation turns one set of two points negative and the other two points positive. The fit with a broad Gaussian curve of scale 224 is not possible anymore.

You see this more extreme with these data

# Training data
x_tr = np.array( [20, 20, 140, 140, 260, 260, 400, 400] ).reshape(-1, 1)
y_tr = np.array( [1025, 1027, 1716, 1723, 1986, 2000, 1515, 1509] ).reshape(-1, 1)

more extreme example

The normalizing changes the assumptions about the mean of the data.

$\endgroup$
2
  • 1
    $\begingroup$ Hi Sextus Empiricus, thanks for your detailed answer. Regarding your assertion "Both situations are very good fits for the points" I would respectfully disagree: In the first example, for x=25 a prediction of y=970 is unreasonable, the non-smooth orange line is overfitting and thus not a good fit. Regarding the shift of the mean: that's very interesting that having non zero-mean data leads to a smoother and better fit. $\endgroup$
    – Ken Grimes
    Oct 8, 2021 at 16:47
  • $\begingroup$ @Ken what I meant is that both fits have the same value in the points x=20 and x=140 and the training data is fitted very well. Whether this is over-fitting and whether the models extrapolate well is another question that I did not mean to cover with that comment. $\endgroup$ Oct 8, 2021 at 17:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.