8
$\begingroup$

I'm trying to understand better the difference between covariance and correlation, besides the fact that the correlation coefficient is a dimensional and has values between $-1$ and $1$.

One unclear point is the following: the correlation coefficient can highlight only a linear relation between two variables, but does covariance only work for linear relations too? Supposing that the relation is not linear, is the covariance not zero? Is it bigger than in the case in which the relation is linear?

Cauchy inequality says $$|\sigma_{xy}|<\sigma_{x} \sigma_{y}$$

Which seems to say that the covariance is maximum when there is a perfect linear relation.

So I do not get what does happen if the relation is not linear, and, if covariance is not a suitable parameter in that cas I don't get the reason for that, since simply looking at the definition:

$$\sigma_{xy}=\frac{\sum(x-\bar{x})(y-\bar{y})}{N}$$

it looks like that this should be non zero for any kind of relation (like parabolic).

So can this be a difference between the use of covariance and correlation coefficient? (The first is useful for any kind of relation, the second one only for linear ones).

$\endgroup$
6
  • 7
    $\begingroup$ Correlation is just scaled covariance, otherwise they are the same. $\endgroup$ Aug 13, 2016 at 14:29
  • $\begingroup$ @RichardHardy If you expand this a little it would be a good answer. Or should I do it for you? $\endgroup$
    – Peter Flom
    Aug 13, 2016 at 14:57
  • $\begingroup$ @PeterFlom, feel free to do it. I thought it was probably a duplicate but did not have the energy to look up the other thread. $\endgroup$ Aug 13, 2016 at 15:01
  • $\begingroup$ These questions are thoroughly answered at stats.stackexchange.com/search?q=Anscombe+quartet and stats.stackexchange.com/questions/tagged/…. The only reason a single duplicate cannot be found is because of the multiple questions asked here, so I have arbitrarily nominated one relevant thread as a duplicate. I warmly recommend the answer by @chl. $\endgroup$
    – whuber
    Aug 13, 2016 at 17:10
  • 1
    $\begingroup$ @ttnphns, +1 but I might need to think deeper to really get your point. Probably it is a matter of perspective. $\endgroup$ Jun 26, 2019 at 10:57

3 Answers 3

5
$\begingroup$

As @RichardHardy points out in his comment, correlation is simply scaled covariance. So, they are useful for exactly the same types of relationships, but correlations are comparable across different relationships and correlations will not be affected by choice of units, while covariances will.

set.seed(123)
htin <- rnorm(100,68,3)
wtpound <- htin*2.5 + rnorm(100,0,5)
htm <- htin*0.0254
wtkg <- wtpound/2.2

cor(htin,wtpound) #0.81
cov(htin,wtpound) #18.09

cor(htm,wtkg) #0.81
cov(htm,wtkg) #0.21

If you have a perfect U shaped relation, both cov and corr will be 0:

x <- seq(-4,4,by = 0.1)
y <- x^2
cor(x,y) #1.63*10^-16
cov(x,y) #1.89*10^-15
$\endgroup$
1
  • 1
    $\begingroup$ this doesn't answer the question of whether covariance only picks up linear relationships, or if it also reflects non-linear ones $\endgroup$
    – develarist
    Aug 20, 2020 at 19:29
7
$\begingroup$

Unlike Pearson correlation, covariance itself is not a measure of the magnitude of linear relationship. It is a measure of co-variation (which could be just monotonic). This is because covariance depends not only on the strength of linear association but also on the magnitude of the variances. In order for covariance to be only the measure of linear association the variances must be controlled for somehow, without that control covariance might occur stronger under nonlinear underlying relationship than under linear one.

Example: let there be completely linearly tied variables X and Y. Without touching Y move quite apart two polar utmost values of X. Now the relationship is only monotonic, but due to widening the range of X the covariance has enhanced.

But covariance has theoretical upper limit equal $\sigma_X \sigma_Y$ which is attainable only under exact linear relationship. In the example, if we back-rescale the widened X data to its original variance the newer value of covariance will drop lower, not higher, than the very initial value. And this is because we had abandoned the linear relationship for monotonic one. Linearity coefficient, the Pearson $r=cov_{XY}/(\sigma_X \sigma_Y)$ is nothing else than the covariance relative that its upper limit.

But the limiting fact that - under controlling the variances (such as standardizing them) - covariance is maximized when the bond is linear, does not make covariance the measure of the magnitude of linear association. It would be improper to call the covariance coefficient the "linear covariance coefficient" like we say it "linear correlation coefficient".

However, covariance is often used in place of Pearson correlation in analyses which assume linear models. For example, you can do factor analysis based on covariance matrix rather than correlation matrix. While covariance can tap not just linearity among the manifest variables, latent factors yet effect the variables but linearly (Pt 2) according to the model, therefore accounting or taking responsibility only for linear bonds between them.

Covariance the higher the...

  • more monotonic is the association (i.e. the fewer are the instances of inversions in the data
  • greater is the combined variability $\sigma_X^2+\sigma_Y^2$
  • more equal are the two variabilities
  • more equal or proportional are the variables' values: under condition that $\sigma_X=\sigma_Y$ cov will be maximal when $X_i=Y_i$ (considering already centered variables), or, equivalently, under $\sigma_X \ne \sigma_Y$ cov will be maximal when $X_i=kY_i$. Linearity.
$\endgroup$
5
  • $\begingroup$ I take issue with this. You are correct that a weak correlation and be present between variables with a huge covariance just because each variable individually has an enormous variance, but either way, the measure of relationship between the variables has to do with a linear relationship, and both correlation and covariance can catch nonlinear relationships (such as $Y=X^3$). $\endgroup$
    – Dave
    Nov 28, 2022 at 2:35
  • $\begingroup$ @Dave, I could not quite understand your point. In Y=X^3, the curvature is accounted an error term, a deviation from straight line, by Pearson r. So r measures how much small this deviation is: it measures linear "portion" or "approximation" of the relationship. Covariance, on the other hand, does not postulate a straight line as a model relationship. $\endgroup$
    – ttnphns
    Nov 28, 2022 at 4:53
  • $\begingroup$ Covariance is proportional to the correlation, so I’m not sure why you say that covariance does not postulate a straight line as a model relationship. Could you please explain? $\endgroup$
    – Dave
    Nov 28, 2022 at 4:56
  • $\begingroup$ If it were "proportional to the correlation" we wouldn't need covariance as a separate coefficient at all. My whole answer tried to demonstrate it that covariance coefficient per se contains no notion of linearity in it. We could add such an assumption later though (find the paragraph on "linear models"). $\endgroup$
    – ttnphns
    Nov 28, 2022 at 5:06
  • $\begingroup$ In my first example in the answer, I take a strictly linear relationship (Y=X, r=1) and then make it closer to Y=X^3 by making the pair of utmost observations more extreme, farther from the mean, on Y (but not on X). So now the line is curved (here, broken), r has diminished, but cov has enlarged. $\endgroup$
    – ttnphns
    Nov 28, 2022 at 5:23
1
$\begingroup$

Well, I think that a good measure of dependence of two random variables must be scale invariant. The philosophy is simple, if I know how random variables $X$ and $Y$ qualitatively depends on each other then this qualitative dependence should not change if we made a change of scale, only the magnitude of the dependence. This is why we normalize the covariance by the product of the standard deviations. The motivation to do that specifically normalization comes from linear algebra, at the end, the correlation is the angle between $X$ and $Y$.

$\endgroup$
1
  • 1
    $\begingroup$ Scale invariance certainly is part of it. For an extended rumination on this topic, see stats.stackexchange.com/a/513608/919. My first two paragraphs there are essentially the same as your answer here. $\endgroup$
    – whuber
    Dec 15, 2022 at 21:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.