7
$\begingroup$

I was watching a lecture on Gaussian Process and when the covariance matrix was introduced, the tutor explained that the matrix is $(n \times n)$ because every point is covered twice - we include the information about the covariance of $(x_1, x_2)$ and about the covariance of $(x_2, x_1)$. She then said: that's because your covariance can vary in different directions.

How is it possible that the covariances can vary in different directions inside the GP covariance matrix? Could you give me an example of when that could be the case?

Update:

After giving it some thought, I realized that it is not $cov(x_1, x_2)$ or $cov(x_2, x_1)$ (as computed from the definition of covariance) that go into the GP covariance matrix, but instead (as was shown in the lecture as well), the covariance matrix is populated by a covariance kernel $k(x, y)$ that acts/is interpreted as a covariance, but it is some function of the distance between $x$ and $y$.

I could therefore imagine, that we might have a covariance kernel that is a function of $(x - y)^p$ where $p$ is an odd power. In such instance, it would indeed make $k(x_1, x_2) \neq k(x_2, x_1)$. But would this be a valid kernel?

Could you clarify if my thinking about the covariance kernel is reasonable? Could you explain if the covariance matrix in Gaussian Process can be non-symmetric? If yes, could you give an example of a dataset where it would make sense to make covariance different in different directions, i.e. where we would like $k(x_1, x_2)$ to be different from $k(x_2, x_1)$?

Improve this question
$\endgroup$
12
  • 1
    $\begingroup$ I am giving it some thought and I have a possible explanation why it might make sense. The covariance matrix does not really contain $cov(x, y)$ but instead, it is populated by a function $k(x, y)$ that "acts" as a covariance (it is later interpreted as a covariance between $x$ and $y$, however, the function can indeed take a different value for $k(x, y)$ and $k(y, x)$). Since in the lecture it was said that the covariance kernel is a function of distance between $x$ and $y$, it might indeed make sense for kernels where there is $x - y$ raised to an odd power. Does my reasoning make sense? $\endgroup$
    – kamilazdybal
    Commented Nov 4, 2018 at 19:26
  • 1
    $\begingroup$ In a gaussian process, the matrix $K(x_1, x_2)$ is always used as a covariance matrix for a multivariate normal, hence, it must be symmetric. $\endgroup$
    – adityar
    Commented Nov 5, 2018 at 10:12
  • 1
    $\begingroup$ @InfProbSciX Don't overlook Wikipedia's qualifier "For most applications..." There's no inherent mathematical necessity that kernels be symmetric. For instance, an exponentially weighted moving average (EWMA) of a time series is a windowed mean using an asymmetric kernel. $\endgroup$
    – whuber ♦
    Commented Feb 1, 2019 at 15:12
  • 2
    $\begingroup$ @Mathtick You seem to confuse two separate concepts: In a Gaussian process, all finite-dimensional marginal distributions are Gaussian; but the kernel describes how those distributions vary with the spatial configuration. That's a separate issue. $\endgroup$
    – whuber ♦
    Commented Oct 1, 2021 at 14:05
  • 2
    $\begingroup$ @mathtick This might be a matter of what community one works within. In the spatial statistics literature such asymmetric "kernels" are contemplated, but often the simplifying assumption of symmetry is quickly made. Spatio-temporal modeling is a natural area of applications where asymmetric kernels can be expected, so ruling them out would just be unconstructive. $\endgroup$
    – whuber ♦
    Commented Oct 1, 2021 at 16:47

2 Answers 2

Reset to default
11
$\begingroup$

Can the covariance matrix in a Gaussian Process be non-symmetric?

Every valid covariance matrix is a real symmetric non-negative definite matrix. This holds regardless of the underlying distribution. So no, it can't be non-symmetric. If the lecturers are making an argument for using some non-symmetric matrix (e.g., using a non-symmetric kernel) in a way that "acts/is interpreted as a covariance" somehow, then the onus is on them to explain how far this analogy holds, given that the matrix is not a valid covariance matrix.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ Although this (correctly) answers the question in the title, it is not relevant to the question as expressed in the body of the post. $\endgroup$
    – whuber ♦
    Commented Oct 1, 2021 at 16:52
  • $\begingroup$ Just a side note for novice readers, non-symmetric Kernel (covariance functions en.wikipedia.org/wiki/Gaussian_process#Covariance_functions ) and non-symmetric covariance matrix are not the same thing. $\endgroup$
    – patagonicus
    Commented Oct 5, 2021 at 13:01
  • $\begingroup$ @MehmetSüzen There is no non-symmetric covariance matrix. I think the semantics are getting even more confusing surrounding this question. And covariance function should really be some bilinear symmetric form or whatever right? $\endgroup$
    – safetyduck
    Commented Oct 5, 2021 at 13:08
  • 1
    $\begingroup$ @mathtick Definitely, you are right. Semantics could get more confusing, specially if it applied in cross disciplinary fields. $\endgroup$
    – patagonicus
    Commented Oct 5, 2021 at 13:49
1
$\begingroup$

Answering below in order to be able to post a screen shot in response to a comment, this is not an answer to the question.

Kevin Murphy

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.