# Intuition on the definition of the covariance

I was trying to understand the Covariance of two random variables better and understand how the first person that thought of it, arrived at the definition that is routinely used in statistics. I went to wikipedia to understand it better. From the article, it seems that good candidate measure or quantity for $Cov(X,Y)$ should have the following properties:

1. It shoukd have a positive sign when two random variables are similar (i.e. when one increases the other one does to and when one decreases the other one does too).
2. We also want it to have a negative sign when two random variables are oppositely similar (i.e. when one increases the other random variable tends to decrease)
3. Lastly, we want the this covariance quantity to be zero (or extremely small probably?) when the two variables are independent of each other (i.e. they don't co-vary with respect to each other).

From the above properties, we want to define $Cov(X,Y)$. My first question is, it is not entirely obvious to me why $Cov(X,Y) = E[(X-E[X])(Y-E[Y])]$ satisfies those properties. From the properties we have, I would have expected more of a "derivative"-like equation to be the ideal candidate. For example, something more like, "if the change in X positive, then the change in Y should also be positive". Also, why is taking the difference from the mean the "correct" thing to do?

A more tangential, but still interesting question, is there a different definition that could have satisfied those properties and still would have been meaningful and useful? I am asking this because it seems no one is questioning why we are using this definition in the first place (it kind of feels like, its "always been this way", which in my opinion, is a terrible reason and it hinders scientific and mathematical curiosity and thinking). Is the accepted definition the "best" definition that we could have?

These are my thoughts on why the accepted definition makes sense (its only going to be an intuitive argument):

Let $\Delta_X$ be some difference of for variable X (i.e. it changed from some value to some other value at some time). Similarly for define $\Delta_Y$.

For one instance in time, we can calculate if they are related or not by doing:

$$sign(\Delta_X \cdot \Delta_Y)$$

This is somewhat nice! For one instance in time, it satisfies the properties we want. If they both increase together, then most of the time, the above quantity should be be positive (and similarly when they are oppositely similar, it will be negative, because the $Delta$'s will have opposite signs).

But that only gives us the quantity we want for one instance in time, and since they are r.v. we might overfit if we decide to base the relationship of two variables based on only 1 observation. Then why not take the expectation of this to see the "average" product of differences.

$$sign(E[\Delta_X \cdot \Delta_Y])$$

Which should capture on average what the average relationship is as defined above! But the only problem this explanation has is, what do we measure this difference from? Which seems to be addressed by measuring this difference from the mean (which for some reason is the correct thing to do).

I guess the main issue I have with the definition is taking the difference form the mean. I can't seem to justify that to myself yet.

The interpretation for the sign can be left for a different question, since it seems to be a more complicated topic.

• The very starting point may be the concept or intuition of cross-product (covariance is only the extension of it). If we have two series of numbers X and Y of the same length, and we define the summed cross-product as Sum(Xi*Yi), then it is maximized if both series were sorted in the same order, and is minimized if one series were sorted ascendingly and the other descendingly. – ttnphns May 17 '14 at 19:03
• The difference from the mean is not the fundamental issue. It is just the magnitude which matters, the difference from the origin; for some reasons it is naturally and convenient to put the origin into the mean. – ttnphns May 17 '14 at 19:07
• @ttnphns are you saying that if they covary together then then the covariance should be "maximized" and if they covary opposite it should be as negative as it can? (i.e. minimized) Why isn't it defined then as the expectation of the cross-product? – Charlie Parker May 17 '14 at 21:33
• Covariance is natural for variables without inherent origin. We then compute mean as the origin (mean have nice properties not relating to the theme of association, so it is typically chosen). If the origin is inherent and is meaningful, it is reasonable to stick to it, then "covariance" (co-outburst) won't be symmetric, but who cares? – ttnphns May 17 '14 at 22:22
• This answer provides a very nice piece of intuition relating to covariance. – Glen_b May 18 '14 at 1:24

Imagine we begin with an empty stack of numbers. Then we start drawing pairs $(X,Y)$ from their joint distribution. One of four things can happen:

1. If both X and Y are bigger then their respective averages we say the pair are similar and so we put a positive number onto the stack.
2. If both X and Y are smaller then their respective averages we say the pair are similar and put a positive number onto the stack.
3. If X is bigger than its average and Y is smaller than its average we say the pair are dissimilar and put a negative number onto the stack.
4. If X is smaller than its average and Y is bigger than its average we say the pair are dissimilar and put a negative number onto the stack.

Then, to get an overall measure of the (dis-)similarity of X and Y we add up all the values of the numbers on the stack. A positive sum suggests the variables move in the same direction at the same time. A negative sum suggests the variables move in opposite directions more often than not. A zero sum suggests knowing the direction of one variable doesn't tell you much about the direction of the other.

It's important to think about 'bigger than average' rather than just 'big' (or 'positive') because any two non-negative variables would then be judged to be similar (e.g. the size of the next car crash on the M42 and the number of tickets bought at Paddington train station tomorrow).

The covariance formula is a formalisation of this process:

$\text{Cov}(X,Y)=\mathbb E[(X−E[X])(Y−E[Y])]$

Using the probability distribution rather than monte carlo simulation and specifying the size of the number we put on the stack.

• Wow, this a very good answer. Just one last thing, do you mind adding more details on the justification on why it has to be the difference form the mean? Why not some other value? Why does it make sense? I think thats the main thing that is getting me stuck about fully internalizing this definition. Thanks btw! – Charlie Parker May 17 '14 at 18:54
• Thanks. Suppose there are two big trucks in two different countries. Now big trucks tend to carry big loads. If we added a positive number to the stack every time each truck carried a big load we'd end up having to say that the behaviour of the two trucks was very similar. But actually the size of the load carried by one truck isn't related to the size of the load carried by the other at any particular time. They just happen to both be big trucks. So our measure of similarity wouldn't be useful. That's why we have to think about 'bigger than average'. – conjectures May 17 '14 at 19:27
• Sorry this is a little late, but I decided to review this topic and I still have a question about why its the difference from the mean. Is the difference from their respective means important because each random variable X and Y might come from different scales? i.e. To have a sense of what "big" is, its different depending on what base scale they are. So to overcome this issue of scale we compare it to their respective means? – Charlie Parker Mar 11 '15 at 16:57

Here is my intuitive way at looking at it without any equations.

1. Its a generalization of the variance to higher dimensions. The motivation probably came from trying to describe how data behaves. To the first order, we have its location - the mean. To the second order, we have the scatter - the covariance.

I guess the main issue I have with the definition is taking the difference form the mean. I can't seem to justify that to myself yet.

the scatter is evaluated relative to the center of the distribution. The most basic definition of the variance is the 'mean deviation from the mean'. hence, you have to substract the mean also in the case of the Covariance.

2. Another prime motivation that comes to mind is the need to define a way to measure distance between random variables. The Mahalanobis distance and the Covariance come hand in hand: Given a Gaussian distribution and two other samples that have an equal Euclidean distance to the distribution mean. If I would ask you which of the samples is more likely to be an outlier that was not drawn from the gaussian distribution, the Euclidean distance will not do. The Mahalanobis distance has a single notable difference from Euclidean distance: it takes into account the scatter (Covariance) of the distribution. This allows you to generalize distance to random variables.

1. Lastly, we want the this covariance quantity to be zero (or extremely small probably?) when the two variables are independent of each other (i.e. they don't co-vary with respect to each other).

OK, let us consider two independent Bernoulli$\left(\frac 12\right)$ random variables $X$ and $Y$. If you are agreeable to allowing $E[XY]$ to mean covariance but are gagging on the subtraction of the mean, then we can readily compute that $E[XY] = \frac 14$ which is pretty small. But what about independent random variables $\hat{X}=1000X$ and $\hat Y = 1000Y$ for which $E[\hat X \hat Y] = 250,000$? So the covariance is not zero (or perhaps just small) as you want it to be for independent random variables. On the other hand, the standard definition cov$(X,Y) = E[(X-E[X])(Y-E[Y])]$ suffers from no such defects, giving zero as the covariance value in both pf the simple cases described above.

1. We also want it to have a negative sign when two random variables are oppositely similar (i.e. when one increases the other random variable tends to decrease)

So now, consider $X$ as before but define $Y = 1-X$. It is very clear that as one variable increases, the other decreases. But, $E[XY]=0$ whereas the standard definition cov$(X,Y) = E[(X-E[X])(Y-E[Y])]$ gives a negative value just as you want it to.

1. It should (sic) have a positive sign when two random variables are similar (i.e. when one increases the other one does to and when one decreases the other one does too).

Once again, let $X$ be as before but now define $Y = X-1$. It is very clear that as one variable increases, so does the other increase. But, $E[XY]$ is negative instead of being positive the way you want it to, whereas the standard definition cov$(X,Y) = E[(X-E[X])(Y-E[Y])]$ gives a positive value just as you want it to.

Finally, the standard definition of covariance simplifies to the definition of variance when $X = Y$.

I was wondering about the same question, and the intuition given by conjectures helped me. To visualise the intuition, I took two random normal vectors, x and y, plotted the scatterplot and coloured each point by the product of their deviations from their respective means (blue for positive values, red for negative).

As is clear from the plot, the product is most positive in the upper-right and bottom-left quadrants, while it is most negative in the bottom-right and upper-left quadrants. The effect of summing the products would result in 0, as the blue points cancel out the red ones.

But you can see that if we removed the red points, the remaining data exhibit a positive relationship with each other, which is validated by the positive sum of products (i.e. sum of the blue points). in the vector space of random variables it is reasonable to define the square of distance between two random variable x and y with E{(x-y)^2} now with respect to this definition of distance dot product or relation of random variables will be E{x.y} which is so similar to the definition of covariance except the terms -E{x} and -E{y} which are for kind of normalization.