1) What is meant by linear dependence?

2) How can I convince myself that covariance measures linear dependence?

3) How I can convince myself that non-linear dependence is not measured by covariance?

  • 2
    $\begingroup$ I realize we avoid links but this seemed relevant...a visual depiction of the limitations: autodeskresearch.com/publications/samestats $\endgroup$ – SecretAgentMan Dec 9 '18 at 20:17
  • $\begingroup$ @SecretAgentMan: Thank you for the link. It offers good insights related to my questions. However, it does not answer them. $\endgroup$ – ColorStatistics Dec 9 '18 at 20:25
  • 1
    $\begingroup$ I agree. It does not answer your question which is why I left it as a comment. I trust others with more talent than myself will answer your question. Cheers. $\endgroup$ – SecretAgentMan Dec 9 '18 at 20:29
  • 1
    $\begingroup$ A while ago I answered these questions geometrically in a post at stats.stackexchange.com/a/71303/919. To see whether it might be of any interest to you, skip to the conclusions at the bottom. $\endgroup$ – whuber Dec 9 '18 at 21:41
  • $\begingroup$ @whuber: Thank you for the link. Wow, that is one long, thorough answer. However, having read most of it, I am not sure that I've seen there direct answers to my questions above. $\endgroup$ – ColorStatistics Dec 10 '18 at 2:50

A1) Say two variables X and Y are linearly dependent, then $X = \alpha Y + c$ for some $\alpha,c \in \mathbb{R}$.

A2) The formula for covariance is:

$$COV(X,Y) = E([X-E(X)][Y-E(X)]) = E(XY)-E(Y)E(X)$$

From A1, consider some linear relationship $X = \alpha Y + c$, but all we have is the data from individual points in each variable. How do we get the value of $\alpha$? Well, it turns out we can instead ask the question, "how do we draw a line between these points so as to minimise the sum of squared differences between each point and the line?". And when we do this analysis for two variables, we get a closed form equation that looks like this:

$$\alpha = \dfrac{E(XY) -E(Y)E(X)}{E(X^2) - E(X)^2}$$

Please note that the numerator is the covariance. I.e.

$$ \alpha = \dfrac{COV(X,Y)}{E(X^2) - E(X)^2}$$

Correlation (e.g. Pearson) is often a measure of the covariance normalised against something to give it a comparable value. So you see the entire measure precedes from the analysis of how to fit a line to some data.

A3) Covariance doesn't measure non-linear relationships for the exact same reason it measures linear ones. Namely, that you can basically think of it as the slope in a linear equation (e.g. $X=\alpha Y + c$), so when you try and fit a line to a curve, the sum of square differences between the points and the line may be large. Here is a good diagram illustrating the implications. The numbers indicate Pearson's correlation coefficient, whilst the diagrams show the corresponding scatter plots.

enter image description here

  • 4
    $\begingroup$ Could you point to a particular definition that makes your assertion in (2) apparent? This is key to the entire answer, because the rest of it merely establishes a qualitative relationship between $\rho$ and apparent linearity. $\endgroup$ – whuber Dec 9 '18 at 21:43
  • 1
    $\begingroup$ @usgroup: note that the phrase you use "linearly correlated" is precisely the tautology this question explores :) It is not clear to me from your answer why covariance measures linear dependence only. Why is "dividing the product of the difference of each variable from it's mean (i.e. the covariance) by the product of the standard deviations of each variable" a measure of linear dependence only? $\endgroup$ – ColorStatistics Dec 10 '18 at 3:07
  • $\begingroup$ @colorstatistics you're right . i'll edit the answer to address covariance. $\endgroup$ – usgroup Dec 10 '18 at 9:35
  • 1
    $\begingroup$ @whuber how my edits have made it better :) $\endgroup$ – usgroup Dec 10 '18 at 14:27
  • $\begingroup$ @usgroup: thank you, this is much improved and quite insightful. I was with you until in A3, where you write "you can basically think of it [covariance] as the slope in a linear equation". Row 2 of your graphical depiction shows sets of very differently looking x and y variables that have correlation=1, and the magnitude of the slope doesn't seem to matter. I am hoping we can close in on this difference between slope and covariance/correlation to understand it better. $\endgroup$ – ColorStatistics Dec 10 '18 at 16:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.