6
$\begingroup$

I'm trying to understand better the difference between covariance and correlation, besides the fact that the correlation coefficient is a dimensional and has values between $-1$ and $1$.

One unclear point is the following: the correlation coefficient can highlight only a linear relation between two variables, but does covariance only work for linear relations too? Supposing that the relation is not linear, is the covariance not zero? Is it bigger than in the case in which the relation is linear?

Cauchy inequality says $$|\sigma_{xy}|<\sigma_{x} \sigma_{y}$$

Which seems to say that the covariance is maximum when there is a perfect linear relation.

So I do not get what does happen if the relation is not linear, and, if covariance is not a suitable parameter in that cas I don't get the reason for that, since simply looking at the definition:

$$\sigma_{xy}=\frac{\sum(x-\bar{x})(y-\bar{y})}{N}$$

it looks like that this should be non zero for any kind of relation (like parabolic).

So can this be a difference between the use of covariance and correlation coefficient? (The first is useful for any kind of relation, the second one only for linear ones).

$\endgroup$
  • 4
    $\begingroup$ Correlation is just scaled covariance, otherwise they are the same. $\endgroup$ – Richard Hardy Aug 13 '16 at 14:29
  • $\begingroup$ @RichardHardy If you expand this a little it would be a good answer. Or should I do it for you? $\endgroup$ – Peter Flom Aug 13 '16 at 14:57
  • $\begingroup$ @PeterFlom, feel free to do it. I thought it was probably a duplicate but did not have the energy to look up the other thread. $\endgroup$ – Richard Hardy Aug 13 '16 at 15:01
  • $\begingroup$ These questions are thoroughly answered at stats.stackexchange.com/search?q=Anscombe+quartet and stats.stackexchange.com/questions/tagged/…. The only reason a single duplicate cannot be found is because of the multiple questions asked here, so I have arbitrarily nominated one relevant thread as a duplicate. I warmly recommend the answer by @chl. $\endgroup$ – whuber Aug 13 '16 at 17:10
  • 1
    $\begingroup$ @ttnphns, +1 but I might need to think deeper to really get your point. Probably it is a matter of perspective. $\endgroup$ – Richard Hardy Jun 26 '19 at 10:57
4
$\begingroup$

As @RichardHardy points out in his comment, correlation is simply scaled covariance. So, they are useful for exactly the same types of relationships, but correlations are comparable across different relationships and correlations will not be affected by choice of units, while covariances will.

set.seed(123)
htin <- rnorm(100,68,3)
wtpound <- htin*2.5 + rnorm(100,0,5)
htm <- htin*0.0254
wtkg <- wtpound/2.2

cor(htin,wtpound) #0.81
cov(htin,wtpound) #18.09

cor(htm,wtkg) #0.81
cov(htm,wtkg) #0.21

If you have a perfect U shaped relation, both cov and corr will be 0:

x <- seq(-4,4,by = 0.1)
y <- x^2
cor(x,y) #1.63*10^-16
cov(x,y) #1.89*10^-15
| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ this doesn't answer the question of whether covariance only picks up linear relationships, or if it also reflects non-linear ones $\endgroup$ – develarist Aug 20 at 19:29
4
$\begingroup$

Unlike Pearson correlation, covariance itself is not a measure of the magnitude of linear relationship. It is a measure of co-variation (which could be just monotonic). This is because covariance depends not only on the strength of linear association but also on he magnitude of the variances. In order for covariance to be only the measure of linear association the variances must be controlled for somehow, without that control covariance might occur stronger under nonlinear underlying relationship than under linear one.

Example: let there be completely linearly tied variables X and Y. Without touching Y move quite apart two polar utmost values of X. Now the relationship is only monotonic, but due to widening the range of X the covariance has enhanced.

But covariance has theoretical upper limit equal $\sigma_X \sigma_Y$ which is attainable only under exact linear relationship. In the example, if we back-rescale the widened X data to its original variance the newer value of covariance will drop lower, not higher, than the very initial value. And this is because we had abandoned the linear relationship for monotonic one. Linearity coefficient, the Pearson $r=cov_{XY}/(\sigma_X \sigma_Y)$ is nothing else than the covariance relative that its upper limit.

But the limiting fact that - under controlling the variances (such as standardizing them) - covariance is maximized when the bond is linear, does not make covariance the measure of the magnitude of linear association. It would be improper to call the covariance coefficient the "linear covariance coefficient" like we say it "linear correlation coefficient".

However, covariance is often used in place of Pearson correlation in analyses which assume linear models. For example, you can do factor analysis based on covariance matrix rather than correlation matrix. While covariance can tap not just linearity among the manifest variables, latent factors yet effect the variables but linearly (Pt 2) according to the model, therefore accounting or taking responsibility only for linear bonds between them.

Covariance the higher the...

  • more monotonic is the association (i.e. the fewer are the instances of inversions in the data
  • greater is the combined variability $\sigma_X^2+\sigma_Y^2$
  • more equal are the two variabilities
  • more equal or proportional are the variables' values: under condition that $\sigma_X=\sigma_Y$ cov will be maximal when $X_i=Y_i$ (considering already centered variables), or, equivalently, under $\sigma_X \ne \sigma_Y$ cov will be maximal when $X_i=kY_i$. Linearity.
| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.