# How does the scale of the target value effect the Gaussian Process performance?

I am currently working with Gaussian Process to make a surrogate model for temperature prediction. A question arose when I was thinking to make result plots in Celsius instead of Kelvin. I suspect if the scale of the target values actually has influences on the performance of the prediction, or numerical stability in computation.

In Celsius unit, the range of target value is about [2, 12](true simulation results minus 273.15K), while in Kelvin, the range of target value is about [275, 285].
In both cases, the input variables are a pair of (x, y) with $$0 \leq x, y \leq 12$$.

Below is the comparison, in both cases, training X and testing X_s are exactly the same, only Yk is target in Kelvin and Yc in Celsius, i.e. Yc = Yk - 273.15. The RMES are calculated using true out-of-sample data, again Yc_test = Yk_test - 273.15, that's the only "altering" I made to the true data. The optimal hyperparams were determined by sklearn's fit, I assume it also takes into account the noise?

The comparison using a Matern 5/2 kernel:

Opt hyperparams K: 164**2 * Matern(length_scale=30.4, nu=2.5)
Opt hyperparams C: 2.25**2 * Matern(length_scale=4.66, nu=2.5)

RMSE of GP k is:  0.5708
with mean
[278.0877772  278.04188218 278.34190474 278.52007792 278.30711453
277.92538269 278.26296211 278.09418015 277.96937866 278.31043331
278.43649371 278.1461776  278.28837775 278.15507878 278.0700446
278.09371288 278.55638764 278.38255591 277.98571652 278.84694022
277.7481693  278.03451848 277.99577411 278.25475644 275.95379841
278.15831804 278.29354604 278.08891763 278.14547972 278.21558574]
and std
[0.02753081 0.06158417 0.16654261 0.06007996 0.18874988 0.24601083
0.14980955 0.17353778 1.2442803  0.12232569 0.23858323 0.1568298
0.17946806 0.06186052 0.1470862  0.51832778 0.23957539 0.10343477
0.24336809 0.15834222 1.48455365 0.24805243 0.25254461 0.17170461
1.00108457 0.20223195 0.02500552 0.44764142 0.03171011 0.24490125]

RMSE of GP c is:  0.4448
with mean
[2.94357222 2.89137106 3.20838123 3.37955134 3.14854234 2.91082754
3.10040522 3.03193313 2.82377777 3.16726111 3.47845897 2.96088007
3.11604222 2.99387894 2.91385656 2.97304756 3.40067683 3.23475982
2.86215504 3.59861003 2.63974466 2.98580884 2.79380584 3.09148033
1.83852795 2.99547885 3.14866498 2.89539608 2.99505726 3.05531382]
and std
[0.0310449  0.07625099 0.2146962  0.05278573 0.23432427 0.24231991
0.19390313 0.20282269 0.75912637 0.14482788 0.18460642 0.18520402
0.21742137 0.07523845 0.17770176 0.3992286  0.25272089 0.13543541
0.22858217 0.17797118 0.90329155 0.29803099 0.28306299 0.21466026
0.58521239 0.24682408 0.03197601 0.41992849 0.04055883 0.239456  ]


The difference in performance is even obvious if using a Gaussian kernel instead of Matern 5/2:


Opt hyperparams K: 144**2 * RBF(length_scale=6.17)
Opt hyperparams C: 2.34**2 * RBF(length_scale=2.63)

RMSE of GP k is:  2.5652
RMSE of GP c is:  0.6711



It is expected that the hyperparameters are going to be different for K and C because they need to be optimized for the target values of each case.

But I do notice that based on RMSE (the root mean square error) and standard deviation, small target values seems to perform better than large values.

I was not sure if such conclusion was valid. I tried to look into the entries of the covariance matrix:

• Kelvin case has much larger entries because of large $$\sigma = 164$$, in the entries it even needs to be squared, so is $$164^2$$.
• although the entries are large, it is not ill-conditioned

Hence "technically" I would expect that two groups of target values should perform equally well, but it is not.

The only possible aspect is the noise = 0.01^2 for both case, I am not sure if the noise is the cause. There is a similar question but the cause is the hyperparameters for noise was not optimized, which is not exactly the confusion part of mine.

I couldn't find out a reason to explain it nor to convince myself that the prediction performance are in fact equally good.

Is there any idea or hint? Thank you!

=======

In summary, what I notice is that larger scale target values cause larger (pair of) hyperparameters. Is it true that usually a reasonable smaller length scale is more favorable than a larger one? since it makes the close-by points more relevant?

In this case, specifically, the range of input variables are $$[0, 12]$$, can I conclude that using Celsius model returns a more reliable prediction than using Kelvin?? Thank you.

Following the comments, I found this answer explaining the setting normalize_y=True: normalize_y in GPR

the real problem here is that the prior is for data with unit variance, but the data doesn't have unit variance. The solution would be to normalise the data so that it has unit variance after it 'enters' the GP, conduct the GP analysis, and then 'unnormalise' it back again at the end.

• Sorry, just to clarify this, do we, or do we not estimate/optimise the noise variance too? If both, models have the same variance but one is dealing with a response variable on a large scale that model "must" do a better job in modelling the mean response. Also while you do not explicitly comment on this, how is the fit validated? Are these RMSE values in-sample or out-of-sample estimates? Especially if they are in-sample estimates, it is likely they are over-fitted and suffer from optimism bias. Nov 4, 2022 at 23:46
• Hi @usεr11852 thank you for your comment and sorry for the confusion. Soon after I posted my question I realized that the RMSE has to be calculated based on "true target". I then used true data(input, target) to train and random variables to predict but compare this predict of random again with true out-of-sample targets... it doesn't make any sense. Thanks for pointing it out. Now I run the comparison again and update the results in my post. As you can see, the RMSE by Celsius is still better than Kelvin, though marginally better. Can we safely conclude that they perform equally well? Thanks.
– Ann
Nov 5, 2022 at 10:05
• So the comparison in the post is no longer a "toy comparison" partially using random input to predict. The data are from 60 true simulations: 30 used for training and 30 for testing. The only altering I did is minus 273.15, so in unit the targets are in different scale.
– Ann
Nov 5, 2022 at 10:19
• Hi @JohnMadden thank you! I tried your suggestion, indeed, it works, both approaches now have the same RMSE :) also exactly the same hyperparams, both RBF and Matner
– Ann
Nov 5, 2022 at 21:14
• Hi @usεr11852 thank you in advance! Sure, only at the moment, I am working against a ddl, I'll do it a few days later :)
– Ann
Nov 17, 2022 at 9:02

$$bar[f_*] = k_*^T (K + \sigma_n^2 I)^{-1} y$$ (Equation 2.25)
$$var[f_*]=𝑘(𝑥_∗,𝑥_∗)−𝑘^𝑇_∗(K+𝜎^2_𝑛 𝐼)^{−1}𝑘_∗$$ (Equation 2.26)
One solution is to set normalize_y = True mentioned by @John Madden in the comment. The hyperparams will be same, so does the RMSE.