14
$\begingroup$

I am performing Gaussian process regression (GPR) and optimizing over hyper-parameters. I am using minFunc to perform all optimizations. My question is should we (or rather, can we) standardize the data before giving it to the objective function? If we do standardize, then the hyper-parameters will be learned according to the standardized data. However, at test time, assuming we get samples one-by-one, it won't be possible to standardize each sample indepdently, right? (Unless, we use some standardizing factors from the training data). If it matters, all the elements in my data are between -1 to 1, however, some columns may have a very small mean and variance as compared to the other columns.

So my question is, should we normalize the data while doing GPR?

P.S. Actually, I observe some weird behavior if I don't standardize my data. For example, minFunc suddenly gives me step direction is illegal error. Some online reading led me to believe that there is either a problem in your gradient calculation or your data is not standardized. I am sure about my gradient function calculation, I have also check it with the DerivativeCheck option. So, that leaves the possibility of data not being standardized.

$\endgroup$
5
  • 3
    $\begingroup$ To your question about processing later data that is out-of-sample relative to the data used for model calibration, I think it would be very important to use the same parameters from the calibration data to pre-process or transform any subsequent data passed through the model. If this isn't done, you would be mixing or confounding any drift in the new information with noise. $\endgroup$ Oct 28, 2015 at 14:04
  • 1
    $\begingroup$ @DJohnson So, you mean I should normalize my test data by using the same scaling parameters obtained from the training data? $\endgroup$
    – Autonomous
    Oct 28, 2015 at 22:07
  • 3
    $\begingroup$ Yes...that's exactly what I mean. To normalize any new data with information drawn from that new data would not be consistent with the model parameters developed from the normalized calibration data. $\endgroup$ Oct 28, 2015 at 22:50
  • $\begingroup$ Sadly I cant comment on @zilver's answer (not enough reputation). I applied the same procedure (zero mean and normalizing with the standard deviation) for my implementation to the predicted the input space. After the prediction I want to revert the standardization by multiplying and adding the mean back to the values. That works great for the predicted mean but not for the predicted variance. Is there a possibility to apply it the the predictive variance as well? As far as I understand the equations you cant just apply the same formula since it results in complete nonsense, right? Best $\endgroup$
    – ChKl
    Jun 26, 2019 at 10:52
  • $\begingroup$ I have a counterexample where it doesn't work: stats.stackexchange.com/questions/547490/… $\endgroup$
    – Ken Grimes
    Oct 8, 2021 at 12:33

2 Answers 2

18
$\begingroup$

Yes, it is desirable to standartize the data while learning Gaussian processes regression. There are a number of reasons:

  1. In common Gaussian processes regression model we suppose that output $y$ has zero mean, so we should standartize $y$ to match our assumption.
  2. For many covariance function we have scale parameters in covariance functions. So, we should standartize inputs to get better estimation of parameters of covariance functions.
  3. Gaussian processes regression is prone to numerical problems as we have to inverse ill-conditioned covariance matrix. To make this problem less severe, you should standartize your data.

Some packages do this job for you, for example GPR in sklearn has an option normalize for normalization of inputs, while not outputs (http://scikit-learn.org/stable/modules/generated/sklearn.gaussian_process.GaussianProcess.html)

$\endgroup$
7
  • $\begingroup$ I understand all the three points. However, my question is: if we learn the hyper-parameters on the normalized training data, would they work on test data, which I cannot normalize (apart from using scaling factors from training data). One of comments by @DJohnson (I think) suggests the same. $\endgroup$
    – Autonomous
    Oct 28, 2015 at 22:10
  • $\begingroup$ For test sample you should apply the normalization similar to that used for training sample, as suggested above in the comments. $\endgroup$ Oct 29, 2015 at 7:11
  • $\begingroup$ If we don't use normalization for test data, than the model is wrong, as normalized training sample differs from not transformed test sample. $\endgroup$ Oct 29, 2015 at 7:13
  • $\begingroup$ @AlexeyZaytsev sorry to interject, am I right in interpreting your answer in saying that we not only need to normalise the inputs, but also the output for the training set? $\endgroup$
    – KRS-fun
    Feb 10, 2017 at 10:48
  • 1
    $\begingroup$ @KRS-fun I suggest you to do normalise outputs to improve numerical stability of the technique, while the right course of actions always depends on your data. Also, I expect that a benefit (model accuracy, robustness and so on) of the normalization of outputs can be much smaller than that of the normalization of inputs. $\endgroup$ Feb 10, 2017 at 12:53
6
$\begingroup$

I agree with Alexey Zaytsev's answer (and the discussion of that answer) that standardising is good to do for various reasons also on the outputs. However, I just want to add an example of why it can be important to standardise the outputs.

An example of where normalizing/standardizing the output is important is for small and noisy output values. The following figure illustrates it on a very simple example. MATLAB code below figure.

The blue dots represent noisy samples from a sin() function. The orange/red circles represent the predicted Gaussian Process values.

  1. In the top subplot the amplitude is unity (with noise).
  2. In the second subplot the output values have been scaled with 1e-5 and we can see that the Gaussian Process model (with default settings) predicts a constant model. The default settings optimise on the noise parameter and have a fairly high lower bound.
  3. In the third subplot the noise parameter is set to zero and not optimised over. The model over-fits in this case.
  4. The fourth subplot shows the standardised outputs and a model fitted on that data.
  5. In the last subplot the outputs are scaled back from the standardisation operation (not scaled back to the original values) and also the predicted values are scaled. Note that the predicted values are scaled using the mean and standard deviation from the training data standardisation.

Simple example

function importance_normgp()
% small outputs

% data
x = 0:0.01:1; x = x(:);
xp = linspace(0.1, 0.9, length(x)); xp = xp(:);

% noise free model
y = sin(2*pi*x) + 5e-1*randn(length(x), 1);

% train and predict gp model
mdl = fitrgp(x, y);
yp = predict(mdl, xp);

figure
subplot(5, 1, 1)
plot(x, y, '.')
hold on
plot(xp, yp, 'o')
title('original problem')

%% make outputs small (below noise lower bound)
ym = y/1e5;

% train and predict gp model
mdlm = fitrgp(x, ym);
ypm = predict(mdlm, xp(:));

subplot(5, 1, 2)
plot(x, ym, '.')
hold on
plot(xp, ypm, 'o')
title('small outputs')

%% outputs small and set sigma = 0

% train and predict gp model
mdlm1 = fitrgp(x, ym, 'Sigma', 1e-12, 'ConstantSigma', true, 'SigmaLowerBound', eps);
ypm1 = predict(mdlm1, xp(:));

subplot(5, 1, 3)
plot(x, ym, '.')
hold on
plot(xp, ypm1, 'o')
title('small outputs and sigma = 0')

%% normalise/standardise
nu = mean(ym);
sigma = std(ym);
yms = (ym - nu)/sigma;

% train and predict gp model
mdlms = fitrgp(x, yms);
ypms = predict(mdlms, xp(:));

subplot(5, 1, 4)
plot(x, yms, '.')
hold on
plot(xp, ypms, 'o')
title('standardised outputs')

% rescale
ypms2 = ypms*sigma + nu;

subplot(5, 1, 5)
plot(x, ym, '.')
hold on
plot(xp, ypms2, 'o')
title('scaled predictions')

legend('true model', 'prediction', 'Location', 'best')
end
$\endgroup$
1
  • $\begingroup$ (+1) Helpful edit and useful answer. Thanks, and welcome to the site! $\endgroup$
    – mkt
    Jun 24, 2019 at 14:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.