Please explain what is the difference between if two variables are linearly dependent or linearly correlated.

I looked up the wikipedia article but didn't get a proper example. Please explain it with example.


Two variables are linearly dependent if one can be written as a linear function of the other. If two variable are linearly dependent the correlation between them is 1 or -1. Linearly correlated just means that two variables have a non-zero correlation but not necessarily having an exact linear relationship. Correlation is sometimes called linear correlation because the Pearson product moment correlation coefficient is a measure of the strength of the linearity in the relationship between the variables.

| cite | improve this answer | |
  • 3
    $\begingroup$ +1. Though, I'd rather say the Pearson coef. "is a measure of strength of linear relationship" instead of is a measure of the degree of linearity in [= of?] the relationship $\endgroup$ – ttnphns Jun 28 '12 at 9:59
  • $\begingroup$ @ttnphns Okay that sounds more appropriate. $\endgroup$ – Michael R. Chernick Jun 28 '12 at 10:51
  • $\begingroup$ Perhaps $\rho^2$ rather than $\rho$ would be a better measure since we don't need to hassle with $\rho$ close to $-1$ meaning a strong linear relationship (albeit with negative slope). Also, consider how much variance is explained versus non-explained, and that $\rho = 0.51$ does not provoke the statistician into turning cartwheels and doing handstands in celebration whereas $\rho^2 > 1/\sqrt{2} \approx 70\%$ is much better evidence of a positive (read, publishable) result. $\endgroup$ – Dilip Sarwate Jun 28 '12 at 20:22

In $\mathbf{R}^2$ linear dependence implies that one vector is a linear function of the other: $$ \textbf{v}_{1}=a\textbf{v}_2. $$ It's clear from this definition that the two variables would move in lock-step, implying a correlation of $1$ or $-1$ depending on the value of $a$. To more fully understand the differences and connections between the concepts, however, I think it's beneficial to consider the geometry involved.

The graph below shows an example of the formula for linear dependence. You can see that the vectors are linearly dependent because one is simply a multiple of the other. enter image description here

This is in contrast to linear independence, which in $\mathbf{R}^2$ is described by: $$ \textbf{v}_{1}\neq a\textbf{v}_2 $$ for vectors $\textbf{v}_1, \textbf{v}_2 \neq \textbf{0}.$ An example of linear independence can be seen in the graphic below. enter image description here

The most extreme version of linear independence is orthogonality, defined for vectors $\textbf{v}_1, \textbf{v}_2$ as: $$ \textbf{v}_{1}^T \textbf{v}_{2} = 0. $$ When graphed in $\mathbf{R}^2$, orthogonality corresponds to the vectors $\textbf{v}_{1}$ and $\textbf{v}_2$ being perpendicular to one another:

enter image description here

Now, consider Pearson's correlation coefficient: $$ \rho_{\textbf{v}_{1}\textbf{v}_{2}} = \frac{(\textbf{v}_{1}-\bar{v}_{1}\textbf{1})^T(\textbf{v}_{2}-\bar{v}_{2}\textbf{1})}{\sigma_{\textbf{v}_{1}}\sigma_{\textbf{v}_{2}}}. $$

Note that if the vectors $(\textbf{v}_{1}-\bar{v}_{1}\textbf{1})$ and $(\textbf{v}_{2}-\bar{v}_{2}\textbf{1})$ are orthogonal then the numerator of Pearson's coefficient is zero, implying that the variables $\textbf{v}_{1}$ and $\textbf{v}_{2}$ are uncorrelated. This illustrates an interesting connection between linear independence and correlation: linear dependence between the centered versions of the variables $\textbf{v}_{1}$ and $\textbf{v}_{2}$ corresponds to a correlation of $1$ or $-1$, non-orthogonal linear independence between the centered versions of $\textbf{v}_{1}$ and $\textbf{v}_{2}$ corresponds to a correlation between $0$ and $1$ in absolute value, and orthogonality between the centered versions of $\textbf{v}_{1}$ and $\textbf{v}_{2}$ corresponds to a correlation of $0$.

Thus, if two vectors are linearly dependent the centered versions of the vectors will also be linearly dependent, i.e. the vectors are perfectly correlated. When two linearly independent vectors (orthogonal or not) are centered the angle between the vectors may or may not change. Thus for linearly independent vectors the correlation may be positive, negative, or zero.

| cite | improve this answer | |

Let f(x) and g(x) be functions.

For f(x) and g(x) to be linearly independent we must have

a*f(x) + b*g(x) = 0 if and only if a=b=0.

In other words there is no c such that a or b is not zero but

a*f(c) + b*g(c) = 0

If there is such a c, then we say that f(x) and g(x) are linearly dependent.


f(x) = sin(x) and g(x) = cos(x) are linearly independent

f(x) = sin(x) and g(x) = sin(2x) are not linearly dependent (Why?)

| cite | improve this answer | |
  • 2
    $\begingroup$ With the definition you're using there, there can be a $c$ such that $a f(c) + b g(c) = 0$; they're only linearly dependent if it happens for all $x$ in the considered domain; for example, consider your second example, with $c=\pi/3$. (Also, I think there's a problem with your first example) $\endgroup$ – Glen_b Nov 15 '13 at 0:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.