Intuition on the definition of the covariance

Question

I was trying to understand the Covariance of two random variables better and understand how the first person that thought of it, arrived at the definition that is routinely used in statistics. I went to wikipedia to understand it better. From the article, it seems that good candidate measure or quantity for $Cov(X,Y)$ should have the following properties:

1. It shoukd have a positive sign when two random variables are similar (i.e. when one increases the other one does to and when one decreases the other one does too).
2. We also want it to have a negative sign when two random variables are oppositely similar (i.e. when one increases the other random variable tends to decrease)
3. Lastly, we want the this covariance quantity to be zero (or extremely small probably?) when the two variables are independent of each other (i.e. they don't co-vary with respect to each other).

From the above properties, we want to define $Cov(X,Y)$. My first question is, it is not entirely obvious to me why $Cov(X,Y) = E[(X-E[X])(Y-E[Y])]$ satisfies those properties. From the properties we have, I would have expected more of a "derivative"-like equation to be the ideal candidate. For example, something more like, "if the change in X positive, then the change in Y should also be positive". Also, why is taking the difference from the mean the "correct" thing to do?

A more tangential, but still interesting question, is there a different definition that could have satisfied those properties and still would have been meaningful and useful? I am asking this because it seems no one is questioning why we are using this definition in the first place (it kind of feels like, its "always been this way", which in my opinion, is a terrible reason and it hinders scientific and mathematical curiosity and thinking). Is the accepted definition the "best" definition that we could have?

---

These are my thoughts on why the accepted definition makes sense (its only going to be an intuitive argument):

Let $\Delta_X$ be some difference of for variable X (i.e. it changed from some value to some other value at some time). Similarly for define $\Delta_Y$.

For one instance in time, we can calculate if they are related or not by doing:

$$sign(\Delta_X \cdot \Delta_Y)$$

This is somewhat nice! For one instance in time, it satisfies the properties we want. If they both increase together, then most of the time, the above quantity should be be positive (and similarly when they are oppositely similar, it will be negative, because the $Delta$'s will have opposite signs).

But that only gives us the quantity we want for one instance in time, and since they are r.v. we might overfit if we decide to base the relationship of two variables based on only 1 observation. Then why not take the expectation of this to see the "average" product of differences.

$$sign(E[\Delta_X \cdot \Delta_Y])$$

Which should capture on average what the average relationship is as defined above! But the only problem this explanation has is, what do we measure this difference from? Which seems to be addressed by measuring this difference from the mean (which for some reason is the correct thing to do).

I guess the main issue I have with the definition is taking the difference form the mean. I can't seem to justify that to myself yet.

---

The interpretation for the sign can be left for a different question, since it seems to be a more complicated topic.

Answer 1

Imagine we begin with an empty stack of numbers. Then we start drawing pairs $(X,Y)$ from their joint distribution. One of four things can happen:

1. If both X and Y are bigger then their respective averages we say the pair are similar and so we put a positive number onto the stack.
2. If both X and Y are smaller then their respective averages we say the pair are similar and put a positive number onto the stack.
3. If X is bigger than its average and Y is smaller than its average we say the pair are dissimilar and put a negative number onto the stack.
4. If X is smaller than its average and Y is bigger than its average we say the pair are dissimilar and put a negative number onto the stack.

Then, to get an overall measure of the (dis-)similarity of X and Y we add up all the values of the numbers on the stack. A positive sum suggests the variables move in the same direction at the same time. A negative sum suggests the variables move in opposite directions more often than not. A zero sum suggests knowing the direction of one variable doesn't tell you much about the direction of the other.

It's important to think about 'bigger than average' rather than just 'big' (or 'positive') because any two non-negative variables would then be judged to be similar (e.g. the size of the next car crash on the M42 and the number of tickets bought at Paddington train station tomorrow).

The covariance formula is a formalisation of this process:

$\text{Cov}(X,Y)=\mathbb E[(X−E[X])(Y−E[Y])]$

Using the probability distribution rather than monte carlo simulation and specifying the size of the number we put on the stack.

Answer 2

Here is my intuitive way at looking at it without any equations.

1. Its a generalization of the variance to higher dimensions. The motivation probably came from trying to describe how data behaves. To the first order, we have its location - the mean. To the second order, we have the scatter - the covariance.

   I guess the main issue I have with the definition is taking the difference form the mean. I can't seem to justify that to myself yet.

   the scatter is evaluated relative to the center of the distribution. The most basic definition of the variance is the 'mean deviation from the mean'. hence, you have to substract the mean also in the case of the Covariance.

2. Another prime motivation that comes to mind is the need to define a way to measure distance between random variables. The Mahalanobis distance and the Covariance come hand in hand: Given a Gaussian distribution and two other samples that have an equal Euclidean distance to the distribution mean. If I would ask you which of the samples is more likely to be an outlier that was not drawn from the gaussian distribution, the Euclidean distance will not do. The Mahalanobis distance has a single notable difference from Euclidean distance: it takes into account the scatter (Covariance) of the distribution. This allows you to generalize distance to random variables.

Answer 3

3. Lastly, we want the this covariance quantity to be zero (or extremely small probably?) when the two variables are independent of each other (i.e. they don't co-vary with respect to each other).

OK, let us consider two independent Bernoulli$\left(\frac 12\right)$ random variables $X$ and $Y$. If you are agreeable to allowing $E[XY]$ to mean covariance but are gagging on the subtraction of the mean, then we can readily compute that $E[XY] = \frac 14$ which is pretty small. But what about independent random variables $\hat{X}=1000X$ and $\hat Y = 1000Y$ for which $E[\hat X \hat Y] = 250,000$? So the covariance is not zero (or perhaps just small) as you want it to be for independent random variables. On the other hand, the standard definition cov$(X,Y) = E[(X-E[X])(Y-E[Y])]$ suffers from no such defects, giving zero as the covariance value in both pf the simple cases described above.

2. We also want it to have a negative sign when two random variables are oppositely similar (i.e. when one increases the other random variable tends to decrease)

So now, consider $X$ as before but define $Y = 1-X$. It is very clear that as one variable increases, the other decreases. But, $E[XY]=0$ whereas the standard definition cov$(X,Y) = E[(X-E[X])(Y-E[Y])]$ gives a negative value just as you want it to.

1. It should (sic) have a positive sign when two random variables are similar (i.e. when one increases the other one does to and when one decreases the other one does too).

Once again, let $X$ be as before but now define $Y = X-1$. It is very clear that as one variable increases, so does the other increase. But, $E[XY]$ is negative instead of being positive the way you want it to, whereas the standard definition cov$(X,Y) = E[(X-E[X])(Y-E[Y])]$ gives a positive value just as you want it to.

Finally, the standard definition of covariance simplifies to the definition of variance when $X = Y$.

Answer 4

I was wondering about the same question, and the intuition given by conjectures helped me. To visualise the intuition, I took two random normal vectors, x and y, plotted the scatterplot and coloured each point by the product of their deviations from their respective means (blue for positive values, red for negative).

As is clear from the plot, the product is most positive in the upper-right and bottom-left quadrants, while it is most negative in the bottom-right and upper-left quadrants. The effect of summing the products would result in 0, as the blue points cancel out the red ones.

But you can see that if we removed the red points, the remaining data exhibit a positive relationship with each other, which is validated by the positive sum of products (i.e. sum of the blue points).

Answer 5

in the vector space of random variables it is reasonable to define the square of distance between two random variable x and y with E{(x-y)^2} now with respect to this definition of distance dot product or relation of random variables will be E{x.y} which is so similar to the definition of covariance except the terms -E{x} and -E{y} which are for kind of normalization.