2
$\begingroup$

I am currently working with Gaussian Process to make a surrogate model for temperature prediction. A question arose when I was thinking to make result plots in Celsius instead of Kelvin. I suspect if the scale of the target values actually has influences on the performance of the prediction, or numerical stability in computation.

In Celsius unit, the range of target value is about [2, 12](true simulation results minus 273.15K), while in Kelvin, the range of target value is about [275, 285].
In both cases, the input variables are a pair of (x, y) with $ 0 \leq x, y \leq 12$.

Below is the comparison, in both cases, training X and testing X_s are exactly the same, only Yk is target in Kelvin and Yc in Celsius, i.e. Yc = Yk - 273.15. The RMES are calculated using true out-of-sample data, again Yc_test = Yk_test - 273.15, that's the only "altering" I made to the true data. The optimal hyperparams were determined by sklearn's fit, I assume it also takes into account the noise?

The comparison using a Matern 5/2 kernel:

Opt hyperparams K: 164**2 * Matern(length_scale=30.4, nu=2.5)
Opt hyperparams C: 2.25**2 * Matern(length_scale=4.66, nu=2.5)

RMSE of GP k is:  0.5708 
with mean 
[278.0877772  278.04188218 278.34190474 278.52007792 278.30711453
 277.92538269 278.26296211 278.09418015 277.96937866 278.31043331
 278.43649371 278.1461776  278.28837775 278.15507878 278.0700446
 278.09371288 278.55638764 278.38255591 277.98571652 278.84694022
 277.7481693  278.03451848 277.99577411 278.25475644 275.95379841
 278.15831804 278.29354604 278.08891763 278.14547972 278.21558574] 
and std 
[0.02753081 0.06158417 0.16654261 0.06007996 0.18874988 0.24601083
 0.14980955 0.17353778 1.2442803  0.12232569 0.23858323 0.1568298
 0.17946806 0.06186052 0.1470862  0.51832778 0.23957539 0.10343477
 0.24336809 0.15834222 1.48455365 0.24805243 0.25254461 0.17170461
 1.00108457 0.20223195 0.02500552 0.44764142 0.03171011 0.24490125]

RMSE of GP c is:  0.4448 
with mean 
[2.94357222 2.89137106 3.20838123 3.37955134 3.14854234 2.91082754
 3.10040522 3.03193313 2.82377777 3.16726111 3.47845897 2.96088007
 3.11604222 2.99387894 2.91385656 2.97304756 3.40067683 3.23475982
 2.86215504 3.59861003 2.63974466 2.98580884 2.79380584 3.09148033
 1.83852795 2.99547885 3.14866498 2.89539608 2.99505726 3.05531382] 
and std 
[0.0310449  0.07625099 0.2146962  0.05278573 0.23432427 0.24231991
 0.19390313 0.20282269 0.75912637 0.14482788 0.18460642 0.18520402
 0.21742137 0.07523845 0.17770176 0.3992286  0.25272089 0.13543541
 0.22858217 0.17797118 0.90329155 0.29803099 0.28306299 0.21466026
 0.58521239 0.24682408 0.03197601 0.41992849 0.04055883 0.239456  ]

The difference in performance is even obvious if using a Gaussian kernel instead of Matern 5/2:


Opt hyperparams K: 144**2 * RBF(length_scale=6.17)
Opt hyperparams C: 2.34**2 * RBF(length_scale=2.63)

RMSE of GP k is:  2.5652
RMSE of GP c is:  0.6711  

It is expected that the hyperparameters are going to be different for K and C because they need to be optimized for the target values of each case.

But I do notice that based on RMSE (the root mean square error) and standard deviation, small target values seems to perform better than large values.

I was not sure if such conclusion was valid. I tried to look into the entries of the covariance matrix:

  • Kelvin case has much larger entries because of large $\sigma = 164$, in the entries it even needs to be squared, so is $164^2$.
  • although the entries are large, it is not ill-conditioned

Hence "technically" I would expect that two groups of target values should perform equally well, but it is not.

The only possible aspect is the noise = 0.01^2 for both case, I am not sure if the noise is the cause. There is a similar question but the cause is the hyperparameters for noise was not optimized, which is not exactly the confusion part of mine.

I couldn't find out a reason to explain it nor to convince myself that the prediction performance are in fact equally good.

Is there any idea or hint? Thank you!

=======

In summary, what I notice is that larger scale target values cause larger (pair of) hyperparameters. Is it true that usually a reasonable smaller length scale is more favorable than a larger one? since it makes the close-by points more relevant?

In this case, specifically, the range of input variables are $[0, 12]$, can I conclude that using Celsius model returns a more reliable prediction than using Kelvin?? Thank you.

======= links for future readers =======

Following the comments, I found this answer explaining the setting normalize_y=True: normalize_y in GPR

the real problem here is that the prior is for data with unit variance, but the data doesn't have unit variance. The solution would be to normalise the data so that it has unit variance after it 'enters' the GP, conduct the GP analysis, and then 'unnormalise' it back again at the end.

$\endgroup$
7
  • $\begingroup$ Sorry, just to clarify this, do we, or do we not estimate/optimise the noise variance too? If both, models have the same variance but one is dealing with a response variable on a large scale that model "must" do a better job in modelling the mean response. Also while you do not explicitly comment on this, how is the fit validated? Are these RMSE values in-sample or out-of-sample estimates? Especially if they are in-sample estimates, it is likely they are over-fitted and suffer from optimism bias. $\endgroup$
    – usεr11852
    Nov 4, 2022 at 23:46
  • $\begingroup$ Hi @usεr11852 thank you for your comment and sorry for the confusion. Soon after I posted my question I realized that the RMSE has to be calculated based on "true target". I then used true data(input, target) to train and random variables to predict but compare this predict of random again with true out-of-sample targets... it doesn't make any sense. Thanks for pointing it out. Now I run the comparison again and update the results in my post. As you can see, the RMSE by Celsius is still better than Kelvin, though marginally better. Can we safely conclude that they perform equally well? Thanks. $\endgroup$
    – Ann
    Nov 5, 2022 at 10:05
  • $\begingroup$ So the comparison in the post is no longer a "toy comparison" partially using random input to predict. The data are from 60 true simulations: 30 used for training and 30 for testing. The only altering I did is minus 273.15, so in unit the targets are in different scale. $\endgroup$
    – Ann
    Nov 5, 2022 at 10:19
  • 1
    $\begingroup$ Hi @JohnMadden thank you! I tried your suggestion, indeed, it works, both approaches now have the same RMSE :) also exactly the same hyperparams, both RBF and Matner $\endgroup$
    – Ann
    Nov 5, 2022 at 21:14
  • 1
    $\begingroup$ Hi @usεr11852 thank you in advance! Sure, only at the moment, I am working against a ddl, I'll do it a few days later :) $\endgroup$
    – Ann
    Nov 17, 2022 at 9:02

1 Answer 1

0
$\begingroup$

It is observed that the RMSE of the Kelvin model is higher than that of the Celsius model. At first glance, however, it may be expected that the two groups of target values should perform equally well. According to the equations from GPML:

$bar[f_*] = k_*^T (K + \sigma_n^2 I)^{-1} y $ (Equation 2.25)

$var[f_*]=𝑘(𝑥_∗,𝑥_∗)−𝑘^𝑇_∗(K+𝜎^2_𝑛 𝐼)^{−1}𝑘_∗$ (Equation 2.26)

The mean involves a multiplication and the variance is independent from the observed targets (also refer to question: scale of variance in Gaussian process). However, the hyperparameters are learned during the training process, whose value do depend on the observed targets. Larger target values lead to larger variance and larger entries in the covariance matrix. In the example, the condition number of the Kelvin model is about five orders of magnitude higher than that of the Celsius model. It may explain why the RMSE is different from each other.

One solution is to set normalize_y = True mentioned by @John Madden in the comment. The hyperparams will be same, so does the RMSE.

Also thanks to @usεr11852 for hints and encouragements.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.