I was watching a lecture on Gaussian Process and when the covariance matrix was introduced, the tutor [explained][1] that the matrix is $(n \times n)$ because every point is covered twice - we include the information about the covariance of $(x_1, x_2)$ and about the covariance of $(x_2, x_1)$. She then said: *that's because your covariance can vary in different directions*.

From the definition of the covariance I see no reason for these to be different: $cov(X, Y) = E[(X - E(X))(Y - E(Y)]$.

How is it possible that the *covariances can vary in different directions* inside the GP covariance matrix? Could you give me an example of when that could be the case?

**Update:**

After giving it some thought, I realized that it is not $cov(x_1, x_2)$ or $cov(x_2, x_1)$ (as computed from the definition of covariance) that go into the GP covariance matrix, but instead (as was shown in the lecture as well), the covariance matrix is populated by a covariance kernel $k(x, y)$ that acts/is interpreted **as a covariance**, but it is some function of the distance between $x$ and $y$.

I could therefore imagine, that we might have a covariance kernel that is a function of $(x - y)^p$ where $p$ is an odd power. In such instance, it would indeed make $k(x_1, x_2) \neq k(x_2, x_1)$. **But would this be a valid kernel?**

*InfProbSciX* and *g g* mentioned in the comments that they cannot be different and the covariance matrix should be symmetric.

Could you clarify if my thinking about the covariance kernel is reasonable? Could you explain if the covariance matrix in Gaussian Process can be non-symmetric? If yes, could you give an example of a dataset where it would make sense to make covariance different in different directions, i.e. where we would like $k(x_1, x_2)$ to be different from $k(x_2, x_1)$?

  [1]: https://www.youtube.com/watch?v=UpsV1y6wMQ8&feature=youtu.be&t=1130