I was watching a lecture on Gaussian Process and when the covariance matrix was introduced, the tutor explained that the matrix is $(n \times n)$ because every point is covered twice - we include the information about the covariance of $(x_1, x_2)$ and about the covariance of $(x_2, x_1)$. She then said: that's because your covariance can vary in different directions.

How is it possible that the covariances can vary in different directions inside the GP covariance matrix? Could you give me an example of when that could be the case?


After giving it some thought, I realized that it is not $cov(x_1, x_2)$ or $cov(x_2, x_1)$ (as computed from the definition of covariance) that go into the GP covariance matrix, but instead (as was shown in the lecture as well), the covariance matrix is populated by a covariance kernel $k(x, y)$ that acts/is interpreted as a covariance, but it is some function of the distance between $x$ and $y$.

I could therefore imagine, that we might have a covariance kernel that is a function of $(x - y)^p$ where $p$ is an odd power. In such instance, it would indeed make $k(x_1, x_2) \neq k(x_2, x_1)$. But would this be a valid kernel?

Could you clarify if my thinking about the covariance kernel is reasonable? Could you explain if the covariance matrix in Gaussian Process can be non-symmetric? If yes, could you give an example of a dataset where it would make sense to make covariance different in different directions, i.e. where we would like $k(x_1, x_2)$ to be different from $k(x_2, x_1)$?

  • $\begingroup$ You are right, they are never different. The lady is wrong. $\endgroup$ – g g Nov 4 '18 at 17:11
  • $\begingroup$ I am giving it some thought and I have a possible explanation why it might make sense. The covariance matrix does not really contain $cov(x, y)$ but instead, it is populated by a function $k(x, y)$ that "acts" as a covariance (it is later interpreted as a covariance between $x$ and $y$, however, the function can indeed take a different value for $k(x, y)$ and $k(y, x)$). Since in the lecture it was said that the covariance kernel is a function of distance between $x$ and $y$, it might indeed make sense for kernels where there is $x - y$ raised to an odd power. Does my reasoning make sense? $\endgroup$ – kamilazdybal Nov 4 '18 at 19:26
  • $\begingroup$ In a gaussian process, the matrix $K(x_1, x_2)$ is always used as a covariance matrix for a multivariate normal, hence, it must be symmetric. $\endgroup$ – adityar Nov 5 '18 at 10:12
  • $\begingroup$ @InfProb These comments appear to be applying two completely different senses of "symmetric." The first sense is that $K(x_i,x_j)$ is a symmetric $n\times n$ matrix. Specifically, $K(x_i,x_j)_{rs}=K(x_j,x_i)_{sr}$ for all $1\le r,s\le n.$ As you point out it must be, because it's a covariance matrix. The second sense--which I believe is the one used in this question--is that $K(x_i,x_j)=K(x_j,x_i)$ for all $i,j.$ This is decidedly not implied by the defining properties of a Gaussian process. $\endgroup$ – whuber Dec 19 '18 at 15:57
  • $\begingroup$ @whuber Could you please post an example of such an assymetric function? Don't kernel functions have to be symmetric as in this wikipedia article? $\endgroup$ – adityar Feb 1 '19 at 9:34

Can the covariance matrix in a Gaussian Process be non-symmetric?

Every valid covariance matrix is a real symmetric non-negative definite matrix. This holds regardless of the underlying distribution. So no, it can't be non-symmetric. If the lecturers are making an argument for using some non-symmetric matrix (e.g., using a non-symmetric kernel) in a way that "acts/is interpreted as a covariance" somehow, then the onus is on them to explain how far this analogy holds, given that the matrix is not a valid covariance matrix.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.