In probability theory covariance matrix denote how each variable relates to other in a pairwise manner. So 1 would mean they are identical and 0 would mean they are independent and are not related. Is this concept similar to Jacobian matrix where such relation between multiple variables are denoted by partial derivative?
$\begingroup$
$\endgroup$
8
-
5$\begingroup$ Because covariance matrices represent quadratic forms (which is why they are always symmetric and positive semi-definite) and Jacobians represent arbitrary linear transformations, about the only thing they have in common is that they are square matrices. Your question sounds like you are asking whether apples and squid are similar. They are in the sense that they are derived from living things, but there's little to recommend the comparison. $\endgroup$– whuber ♦Mar 5, 2020 at 16:21
-
$\begingroup$ allright makes sense! +1 for creative analogy $\endgroup$– GENIVI-LEARNERMar 5, 2020 at 16:28
-
1$\begingroup$ In general, whuber is right. But there are actually some similarities. And in some cases, the covariance matrix is the jacobian matrix -- e.g. when a variable transformation is performed; to be more precise, it would need to be a linear transformation, otherwise it's only an approximation. In general it helps to think of the Jacobian simply as the derivative in spaces with dimension larger one. But I'd also be careful to associate covariance with "how a variable relates to another"; it's just the expectation value of the product. $\endgroup$– cherubMar 6, 2020 at 15:19
-
1$\begingroup$ @cherub much of what you write is confusing. In particular, any covariance matrix can be considered a linear transformation--that's standard, but in a statistical application it is important to describe what it might be transforming. A Jacobian cannot possibly be a covariance matrix unless it is square, symmetric, and positive-semidefinite, which in more than one dimension would be a rare circumstance, and even then it's unclear what it would be the covariance of. $\endgroup$– whuber ♦Mar 7, 2020 at 13:57
-
1$\begingroup$ @whuber: I'm sorry that I added to the confusion. Yes, it is a special circumstance that the Jacobian is square. When I'm teaching about coordinate transformations (I'm a physicist) this actually comes up frequently. In most cases the transformation isn't linear (e.g. cartesian to spherical), so it's just the first order approximation. But sometimes (e.g. simple translations or scaling -- linear), it is true. I tried to add a little practical info, but didn't intend to confuse. $\endgroup$– cherubMar 11, 2020 at 13:48
|
Show 3 more comments
2 Answers
$\begingroup$
$\endgroup$
3
I'd rather say that the covariance matrix is similar to the Hessian matrix .. Furthermore, the Hessian tends to be proportional to the precision matrix (which is the inverse of the covarinace matrix). The intuition behind is that the higher the curvature in a given direction is, the lower the auto-covariance (the variance) in that same direction will be.
-
$\begingroup$ Hi, could you please provide more details in the answer. In terms of how parwise relation in Jacobian matrix conveys something. Also how is covariance matrix similar to Hessian matrix where single element is partial second derivative? $\endgroup$ Nov 11, 2020 at 13:06
-
$\begingroup$ Assuming a linear relationship between Y and X in the form $Y=aX+b$, the first derivative (Jacobian) will be a constant equal to the regression coefficient $a$ which is proportional to $Cov(Y,X)$ , whereas the second derivative (Hessian) will be 0. In general, pairwise, a null Hessian coefficient means there is a collinearity or a correlation between variables, the degree of intensity of this correlation is conveyed by the Jacobian coefficient. But, the Jacobian coefficient must be constant, otherwise the relationship would be of a higher order than a simple collinearity. $\endgroup$– bigInnerNov 11, 2020 at 16:53
-
$\begingroup$ alright. this is the worked example I was looking for. Also why is it that "higher the curvature in a given direction is, the lower the auto-covariance (the variance) in that same direction will be"? $\endgroup$ Nov 18, 2020 at 11:33
$\begingroup$
$\endgroup$
9
Consider the direction of $x$ axis in the figure below. At a fixed $y$ ($y = 15$ in the example) the sample of points $x_i$ drawn from the blue plot has a higher variance than the red sample. However, in the same direction $x$, the red plot has a higher curvature $.6$ compared to the blue one $.2$.
-
2$\begingroup$ Could you explain how any of these plots determine how to draw samples?? $\endgroup$– whuber ♦Nov 20, 2020 at 14:01
-
$\begingroup$ @whuber could this be that he meant to show normal distribution, so blue would have higher variance and red with lower variance and the in the x-axis line the red/blue points are the draws? I am curious to see how this particular example can be worked out to show how plots determine the sample draws. $\endgroup$– metronNov 20, 2020 at 15:45
-
$\begingroup$ @metron I don't see anything resembling a Normal distribution in this plot. $\endgroup$– whuber ♦Nov 20, 2020 at 16:07
-
$\begingroup$ We can just draw a uniform sample from the interval $[0,15]$ on the $y$ $axis$, then map each sample to its projection on the $x$ $axis$ via one of the plots. Using the same original uniform sample, we get the visualized blue samples when the projection occurs via the blue plot and the red ones when it occurs via the red plot. $\endgroup$– bigInnerNov 20, 2020 at 20:05
-
1$\begingroup$ @GENIVI-LEARNER Nothing about that "explanation" makes much sense, beginning with the fact that the y-axis does not map to the x-axis (there's no objective way to determine the sign of the result) and the red and blue smears on the x-axis clearly do not correspond, even qualitatively, with the result of such an approach. $\endgroup$– whuber ♦Nov 22, 2020 at 18:08