Difference between correlation and covariance: is covariance only useful if the relation is linear?

Question

I'm trying to understand better the difference between covariance and correlation, besides the fact that the correlation coefficient is a dimensional and has values between $-1$ and $1$.

One unclear point is the following: the correlation coefficient can highlight only a linear relation between two variables, but does covariance only work for linear relations too? Supposing that the relation is not linear, is the covariance not zero? Is it bigger than in the case in which the relation is linear?

Cauchy inequality says $$|\sigma_{xy}|<\sigma_{x} \sigma_{y}$$

Which seems to say that the covariance is maximum when there is a perfect linear relation.

So I do not get what does happen if the relation is not linear, and, if covariance is not a suitable parameter in that cas I don't get the reason for that, since simply looking at the definition:

$$\sigma_{xy}=\frac{\sum(x-\bar{x})(y-\bar{y})}{N}$$

it looks like that this should be non zero for any kind of relation (like parabolic).

So can this be a difference between the use of covariance and correlation coefficient? (The first is useful for any kind of relation, the second one only for linear ones).

Answer 1

As @RichardHardy points out in his comment, correlation is simply scaled covariance. So, they are useful for exactly the same types of relationships, but correlations are comparable across different relationships and correlations will not be affected by choice of units, while covariances will.

set.seed(123)
htin <- rnorm(100,68,3)
wtpound <- htin*2.5 + rnorm(100,0,5)
htm <- htin*0.0254
wtkg <- wtpound/2.2

cor(htin,wtpound) #0.81
cov(htin,wtpound) #18.09

cor(htm,wtkg) #0.81
cov(htm,wtkg) #0.21

If you have a perfect U shaped relation, both cov and corr will be 0:

x <- seq(-4,4,by = 0.1)
y <- x^2
cor(x,y) #1.63*10^-16
cov(x,y) #1.89*10^-15

Answer 2

Unlike Pearson correlation, covariance itself is not a measure of the magnitude of linear relationship. It is a measure of co-variation (which could be just monotonic). This is because covariance depends not only on the strength of linear association but also on the magnitude of the variances. In order for covariance to be only the measure of linear association the variances must be controlled for somehow, without that control covariance might occur stronger under nonlinear underlying relationship than under linear one.

Example: let there be completely linearly tied variables X and Y. Without touching Y move quite apart two polar utmost values of X. Now the relationship is only monotonic, but due to widening the range of X the covariance has enhanced.

But covariance has theoretical upper limit equal $\sigma_X \sigma_Y$ which is attainable only under exact linear relationship. In the example, if we back-rescale the widened X data to its original variance the newer value of covariance will drop lower, not higher, than the very initial value. And this is because we had abandoned the linear relationship for monotonic one. Linearity coefficient, the Pearson $r=cov_{XY}/(\sigma_X \sigma_Y)$ is nothing else than the covariance relative that its upper limit.

But the limiting fact that - under controlling the variances (such as standardizing them) - covariance is maximized when the bond is linear, does not make covariance the measure of the magnitude of linear association. It would be improper to call the covariance coefficient the "linear covariance coefficient" like we say it "linear correlation coefficient".

However, covariance is often used in place of Pearson correlation in analyses which assume linear models. For example, you can do factor analysis based on covariance matrix rather than correlation matrix. While covariance can tap not just linearity among the manifest variables, latent factors yet effect the variables but linearly (Pt 2) according to the model, therefore accounting or taking responsibility only for linear bonds between them.

Covariance the higher the...

• more monotonic is the association (i.e. the fewer are the instances of inversions in the data
• greater is the combined variability $\sigma_X^2+\sigma_Y^2$
• more equal are the two variabilities
• more equal or proportional are the variables' values: under condition that $\sigma_X=\sigma_Y$ cov will be maximal when $X_i=Y_i$ (considering already centered variables), or, equivalently, under $\sigma_X \ne \sigma_Y$ cov will be maximal when $X_i=kY_i$. Linearity.

Answer 3

Well, I think that a good measure of dependence of two random variables must be scale invariant. The philosophy is simple, if I know how random variables $X$ and $Y$ qualitatively depends on each other then this qualitative dependence should not change if we made a change of scale, only the magnitude of the dependence. This is why we normalize the covariance by the product of the standard deviations. The motivation to do that specifically normalization comes from linear algebra, at the end, the correlation is the angle between $X$ and $Y$.