Does it make sense to expect zero mean and standard deviation 1 for errors from a gaussian process model?

Question

ML newbie here. If whatever information I have provided is not sufficient feel free to let me know what more I need to add. Now, the question:

I am working with multi-task Gaussian processes. I have 3 dimensional real vectors as inputs and 3 dimensional real vectors as outputs. The training data has been cleaned of NaNs and anomalies. The training data is normalized by subtracting the mean and standard deviation along each dimension of the data.

For testing, I am feeding the model individual test samples (and NOT batches). But before the test sample is sent in, I subtract the mean of the training data from this test sample and divide it by the training data's standard deviation.

I compute the error of the output. When I collect these errors I find that their mean is [-0.12, -0.08, -0.14] and that their standard deviation is [0.62, 0.20, 0.34].

What surprises me is that the mean of the errors is not zero and the standard deviation along any dimension is pretty far from being 1.

Am I expecting something completely unjustified? If yes, why? If no, what do you think I can try doing to better understand the issue?

Some more information:

As a next step, I whitened each error vector using the predictive covariance matrix for it and then found that the resulting set of whitened errors does not form a spherical Gaussian - again, something that I was not expecting.

I am using GPyTorch for the multi-task Gaussian process implementation.

Answer 1

The discrepancy between train and test error may depend on how much your model overfitts, in such case test error will be much higher than train error. Another reason may be that your test set is very different from train set, so things learned on train set do not apply to the test set. To simplify the argument, let's assume that your model is not overfitting and train and test set have same distribution.

Variance of the data is defined as

$$ \frac{1}{n} \sum_{i=1}^n (y_i - \mu)^2 $$

Mean squared error is defined as

$$ \frac{1}{n} \sum_{i=1}^n (y_i - \hat y_i)^2 $$

So if you observe mean squared error equal to the variance it means that your model is not better than predicting mean for every sample. If you are able to replace your model with predicting a constant instead, this means your model is very poor. Actually, predicting mean for every sample is a good benchmark to start with, so you know the lower bound for mean squared error below which your model should not go.

As about mean not equal to zero, you would not expect sample mean to be equal to the population mean. Sample mean gets closer to population mean as your sample size grows. With sample size approaching infinity, it would converge, but otherwise they rather won't be equal.