# What does it mean to have Covariance > 1 in Gaussian Processes? (Or Cov(x, x) != 1?)

The sum of two kernels is a kernel. [. . .] The product of of two kernels is a kernel. - Gaussian Processes for Machine Learning, Section 4.2.4

I can quite easily see how the product would work: For identical elements, both kernels evaluate to 1, so they stay at one. All other elements are in the [0, 1)-interval, which is closed under multiplication. It's even quite easy to see that the property of positive-semidefinite-ness will not be violated: The diagonals keep their value, while all others get smaller.

What happens when adding? Naively, I would assume you could add two kernels and divide them by two, keeping everything cleanly in the [0, 1]-range. However, this is not stated in the chapter. Also, some kernel definitions I've found use a multiplying parameter, as seen here (From Lloyd/Duvenaud: Automatic Baysean Covariance Discovery):

Maybe I'm just particularly dense, but in multiple complete readings of the paper I could not find a definition of the sigma, forcing me to conclude that sigma is just some number in R. Obviously, this will lead to entries in the covariance matrix much larger than 1,or smaller than 1 along the diagonals.

As for testing it myself, I fail to really see the significance. Curves defined by cos(x), 2*cos(x) and 0.5*cos(x) are equally well explained by a periodic kernel of the proper period, no "amplitude" argument required.

However, if I just multiply the kernels by random values, it still works fine and seems to affect nothing but the standard deviation. For example, a cos-curve explained by the same kernel three times, but scaled by 0.01, 1 and 2 respectively. Mean stays the same, standard deviation changes.

So, what am I missing? Is it completely fine to multiply kernels by arbitrary values, as long as they are positive? Is it just another hyperparameter? Does it ever do anything other than changing standard deviation? Why does it change the standard deviation?

• Assessing relative to 1 don't mean a thing, if it ain't got that scaling thing. (What Duke Ellington would have said if he had been a statistician.) – Mark L. Stone Jun 21 '18 at 15:31
• This doesn't help me at all, but it still cheers me up quite a bit. +1 – Kjeld Schmidt Jun 21 '18 at 15:37
• I just realized that the linear kernel in the picture is not just able to exceed 1 by itself, but even reach negative values. My confusion rises. – Kjeld Schmidt Jun 21 '18 at 15:43
• Covariance of x with itself is just the variance of x; that can be any positive number.. Your question appears to confuse covariance with correlation (which must be in $[-1,1]$.) – Glen_b Jun 22 '18 at 3:34
• That seems quite true.If you'd expand that into an answer, I would probably accept it. – Kjeld Schmidt Jun 22 '18 at 12:30