1) What is meant by linear dependence?

2) How can I convince myself that covariance measures linear dependence?

3) How I can convince myself that non-linear dependence is not measured by covariance?

  • 2
    $\begingroup$ I realize we avoid links but this seemed relevant...a visual depiction of the limitations: autodeskresearch.com/publications/samestats $\endgroup$ Dec 9 '18 at 20:17
  • $\begingroup$ @SecretAgentMan: Thank you for the link. It offers good insights related to my questions. However, it does not answer them. $\endgroup$ Dec 9 '18 at 20:25
  • 1
    $\begingroup$ I agree. It does not answer your question which is why I left it as a comment. I trust others with more talent than myself will answer your question. Cheers. $\endgroup$ Dec 9 '18 at 20:29
  • 1
    $\begingroup$ A while ago I answered these questions geometrically in a post at stats.stackexchange.com/a/71303/919. To see whether it might be of any interest to you, skip to the conclusions at the bottom. $\endgroup$
    – whuber
    Dec 9 '18 at 21:41
  • 1
    $\begingroup$ Might be of intetest: stats.stackexchange.com/q/229667/3277. (My own opinion expressed there too, is that covariance coefficient is not a measure of the amount of just linear relationship in the sense correlation coefficient is. It is improper to say "linear covariance coefficient".) $\endgroup$
    – ttnphns
    Sep 16 '20 at 7:40

A1) Say two variables X and Y are linearly dependent, then $X = \alpha Y + c$ for some $\alpha,c \in \mathbb{R}$.

A2) The formula for covariance is:

$$COV(X,Y) = E([X-E(X)][Y-E(Y)]) = E(XY)-E(X)E(Y)$$

From A1, consider some linear relationship $X = \alpha Y + c$, but all we have is the data from individual points in each variable. How do we get the value of $\alpha$? Well, it turns out we can instead ask the question, "how do we draw a line between these points so as to minimise the sum of squared differences between each point and the line?". And when we do this analysis for two variables, we get a closed form equation that looks like this:

$$\alpha = \dfrac{E(XY) -E(Y)E(X)}{E(X^2) - E(X)^2}$$

Please note that the numerator is the covariance. I.e.

$$ \alpha = \dfrac{COV(X,Y)}{E(X^2) - E(X)^2}$$

Correlation (e.g. Pearson) is often a measure of the covariance normalised against something to give it a comparable value. So you see the entire measure precedes from the analysis of how to fit a line to some data.

A3) Covariance doesn't measure non-linear relationships for the exact same reason it measures linear ones. Namely, that you can basically think of it as the slope in a linear equation (e.g. $X=\alpha Y + c$), so when you try and fit a line to a curve, the sum of square differences between the points and the line may be large. Here is a good diagram illustrating the implications. The numbers indicate Pearson's correlation coefficient, whilst the diagrams show the corresponding scatter plots.

enter image description here

  • 4
    $\begingroup$ Could you point to a particular definition that makes your assertion in (2) apparent? This is key to the entire answer, because the rest of it merely establishes a qualitative relationship between $\rho$ and apparent linearity. $\endgroup$
    – whuber
    Dec 9 '18 at 21:43
  • 1
    $\begingroup$ @usgroup: note that the phrase you use "linearly correlated" is precisely the tautology this question explores :) It is not clear to me from your answer why covariance measures linear dependence only. Why is "dividing the product of the difference of each variable from it's mean (i.e. the covariance) by the product of the standard deviations of each variable" a measure of linear dependence only? $\endgroup$ Dec 10 '18 at 3:07
  • $\begingroup$ @colorstatistics you're right . i'll edit the answer to address covariance. $\endgroup$
    – usgroup
    Dec 10 '18 at 9:35
  • 1
    $\begingroup$ @whuber how my edits have made it better :) $\endgroup$
    – usgroup
    Dec 10 '18 at 14:27
  • $\begingroup$ @usgroup: thank you, this is much improved and quite insightful. I was with you until in A3, where you write "you can basically think of it [covariance] as the slope in a linear equation". Row 2 of your graphical depiction shows sets of very differently looking x and y variables that have correlation=1, and the magnitude of the slope doesn't seem to matter. I am hoping we can close in on this difference between slope and covariance/correlation to understand it better. $\endgroup$ Dec 10 '18 at 16:37

Before moving to the interesting stuff, let's dispose of two of the questions. Covariance is too complicated for our purposes because it depends on three things: the magnitude of variable $X,$ the magnitude of variable $Y,$ and their correlation. (See the discussion at the end of https://stats.stackexchange.com/a/18200/919 for the details.)

The individual magnitudes tell us nothing about the relationship between the variables. Therefore we take them out of the picture by turning the covariance into the correlation: this is the quantity we need to pay attention to. Thus, questions (2) and (3) are meaningless because they are misguided: covariance does not measure linear dependence, nor does it tell us anything (by itself) about "non-linear" dependence.

Question (1) remains, though: what really is "linear dependence"?

This answer adduces five simple general principles (highlighted below) to identify a class of possible ways to quantify departures from linear dependence (between two variables $X$ and $Y$). Within that class, the correlation coefficient $\rho(X,Y)$ and its square $R^2$ emerge as being among the mathematically simplest choices.

In effect, I am proposing that the concept of "linear dependence" is deeply and inextricably associated with these five principles: to understand one is to understand the other.

It may be insightful to ground an answer in fundamental, widely general principles. How would we want to characterize "linear"? Let's hold that in abeyance and first consider the kinds of simplifications we might permit ourselves to make in such situations. I propose beginning with the concept of invariance to changes of units.

This means that

any relationship between two variables $X$ and $Y$ that is considered to have some degree of "linearity," no matter what that might be, must have exactly the same degree of linearity when these variables are expressed in other units.

For instance, if $X$ is a temperature and $Y$ is a mass, the linearity of their relationship ought to be the same no matter whether we express $X$ in degrees C, F, R, or K; and no matter whether we express $Y$ in g, Kg, lb, oz, metric tons, or whatever.

This form of invariance permits us freely to adopt initial "normalizations" of the variables. One possible normalization chooses units in which (a) the average of each variable is zero and (b) the variance of each variable is $1.$ In effect, this uses the standard deviation as a natural measure of the "size" of a variable: a metric. It's not the only possibility, but it's an extremely useful one (as suggested by the Central Limit Theorem: see my discussion at https://stats.stackexchange.com/a/3904/919).

At this point I can suggest two avenues of investigation. One is to view all variables as random variables and to analyze their joint distributions by means of the joint cumulant generating function (which always exists and provides full information about the distribution). After these preliminary normalizations, the c.g.f. (which is a function of two variables $(s,t)$) takes the form

$$\psi_{X,Y}(s,t) = -\frac{1}{2}t^2 - \frac{1}{2}s^2 - \kappa_{11} s t + o(|(s,t)|^2).$$

See, for instance, Stuart & Ord, Kendall's Advanced Theory of Statistics Vol. I (5th Ed.), sections 3.28 and 3.29. From this perspective, the first, simplest, and most dominant scalar indicator of any bivariate distribution is its standardized bivariate cumulant $\kappa_{11} = \operatorname{Cor}(X,Y)=\rho_{XY}$ (the correlation coefficient). We might then define the concept we are pursuing as

"linearity" of an association (between either random variables or data vectors) is the degree to which their bivariate distribution is well approximated by the terms in their c.g.f. through second order.

In particular, the Bivariate Normal distribution equals the foregoing expression, and therefore is the "most linear" of all possible distributions; consequently its "degree of linearity" must be solely a function of the correlation coefficient $\rho.$

However, to those not well accustomed to thinking (rather physically) in terms of moments and (rather abstractly) in terms of cumulants, this characterization might be of little help. Let's therefore return to the program at hand, of exploring where basic invariance principles might lead.

No matter whether your "variable" is a finite dataset or a random variable, it "lives" in a natural vector space (wherein variables may be rescaled and added to each other). Normalization places all variables on the unit sphere in this vector space. When two variables (before normalization) differ by an affine function (one is a scaled, shifted version of the other), then after normalization they will either coincide or be diametrically opposite points on this sphere. We may therefore conceive of the amount of nonlinearity as being some function of distances on this sphere.

A "nice" distance will vary smoothly. Such distances are given by Riemann metrics.. Although that tells us little in general, it has an interesting implication for "very nearly linear" relations: that is, when the distance between $X$ and $Y$ is small. A basic (and elementary) result of Riemannian geometry is that locally (that is, to a very good approximation for nearby points), the squared distance between $X$ and $Y,$ written $\delta(X,Y)^2,$ is a quadratic function of the vector difference $X-Y.$ When things are written out in coordinates with $X=(x_1,x_2,\ldots,x_n)$ and $Y=(y_1,y_2,\ldots,y_n),$ this means there exist numbers $g_{i}$ which may vary with $X$ for which

$$\delta(X,Y)^2 \approx \sum_{i=1}^n g_i(X) (x_i - y_i)^2.$$

There are many possible candidates for such a distance. But we may exploit another form of invariance. Because (in this context) "linear" refers to some general relationship holding among matched pairs of data, the order in which we list those pairs should not matter. Mathematically this means

the degree of linearity between $X$ and $Y$ must be the same when the components of $X$ and $Y$ are permuted (in parallel).

This implies all the $g_i(X)$ must be equal and permits us to drop the subscript "$i.$"

There remains one more almost trivial form of invariance it seems natural to demand of any linear relationship:

the degree of linearity between $X$ and $Y$ is the same as the degree of linearity between $Y$ and $X.$

Thus, $X$ is not privileged as the "base point" in the distance calculation, so we may equally well think of the Riemannian metric elements as varying with $Y:$

$$\delta(X,Y)^2 \approx \sum_{i=1}^n g(Y) (x_i - y_i)^2.$$

Now, because this is an approximation, it does not follow that $g(X) = g(Y).$ But it does indicate that $g$ itself ought to vary smoothly and slowly over the sphere.

Since everything is relative--we are discussing relative degrees of "linearity," not any absolute sense of it--we may rescale the function $g$ to make it equal $1/2$ at some point $X.$ Near this point, simple algebra (and our preliminary normalizations) show

$$\delta(X,Y)^2 \approx g(X) \sum_{i=1}^n (x_i-y_i)^2 = \frac{1}{2}\left[\sum_i x_i^2 + \sum_i y_i^2 - 2 \sum_i x_i y_i\right] = 1 - \rho_{XY}.$$

Generally, then, at least for "nearby" vectors $X$ and $Y,$ linearity must be measured as some multiple of $1-\rho_{XY}$ (where the multiple itself could depend on $X$). This often is called the "cosine distance" or "cosine similarity".

Obviously the simplest such multiple is the constant function $1,$ because it just disappears from the formula. But I wish to re-emphasize the point that this is not a unique solution: it's a convention. Maybe it could be useful in some applications to break it. This would amount to weighting $\delta$ according to some relevant pattern of variation among the components of the (standardized!) vectors $X$ and $Y.$ However, as a general proposition, any such weighting would appear arbitrary: it could be suitable only in particular circumstances. As such it doesn't seem to qualify as any part of the concept of linear dependence.

Finally, remember that this characterization of $\delta$ is appropriate only for "nearly linear" associations, where already $X\approx Y.$ It pays to remember that $\rho$ therefore is truly justified in characterizing "linear dependence" only in cases where that dependence is clear and strong--that is, $X$ and $Y$ approximate each other--and in all other cases, $\rho$ may be of little value. Be prepared for $\rho$ to fool you when the relationship between $X$ and $Y$ gets really complicated! Visit our posts on Anscombe's quartet for a well-known set of examples.

On the Riemannian sphere there will be points that get as far from $X$ as possible and, if our distance is any good, those points ought to be close to $-X,$ which is diametrically opposite (at a cosine distance of $1-(-1)=2.$ Because we want to view $X$ as being closely linearly related to vectors near $-X$ (as well as near $X$), we may invoke one more invariance principle:

The degree of linearity between $X$ and $Y$ must be the same as the degree of linearity between $X$ and $-Y.$

Since $\rho(X,-Y) = -\rho(X,Y),$ this last principle implies the degree of linearity--whatever it might mean or be--should be measured by some decreasing function of $|\rho(X,Y)|.$ This explains why $R^2(X,Y) = \rho(X,Y)^2$ is commonly compared to its maximum possible value of $1:$ we are really using $1-R^2$ as a measure of departure from linearity. Clearly this is one of the simplest possible formulas (and the presence of the square simplifies its mathematical analysis).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.