I'm just learning about Covariance and encountered something I don't quite understand.

Assume we have two random variables X and Y, where the respective joint-probability function assigns equal weights to each event.

According to wikipedia the Cov(X, Y) can then be caluculated as follows:

$${\displaystyle \operatorname {cov} (X,Y)={\frac {1}{n}}\sum _{i=1}^{n}(x_{i}-E(X))(y_{i}-E(Y)).}$$

What confuses me is the fact, that they sum only over $x_i$ and $y_i$; $i=1,...,n$ , but not $x_i$ and $y_j; \;i=1,...,n$ $j=1,...,m$ , thus many possible combinations are not calculated. In other words if we look at each calculated combination individually, we only get a $n*1$ matrix, instead of a $n*m$ matrix.

Can anyone explain this (I suppose it's rather obvious but I just don't see the reason at the moment).

  • 5
    $\begingroup$ Maybe this helps. Lets say you are interested in the relationship between height and weight in people. Covariance could give an indication of this relationship. In this case I am interested if someone with a larger height weighs more, or less. This means that I am interested in the relationship between the height and weight of each individual person, over all persons. I am not interested in the relationship between person A s weight and person B height. Hence, in the calculation both features should stem from the same observational unit, a person. The $i$ designates a particular person. $\endgroup$ – spdrnl Mar 11 '17 at 17:05
  • $\begingroup$ I was typing something along these lines, but I erased it now, because the prior comment summarizes it well. $\endgroup$ – Antoni Parellada Mar 11 '17 at 17:06

The idea is that the possible outcomes in your sample are $i=1, \ldots, n$, and each outcome $i$ has equal probability $\frac{1}{n}$ (under the probability measure that assigns equal probability to all outcomes that you appear to be using). You have $n$ outcomes, not $n^2$.

To somewhat indulge your idea, you could compute:

$$ \operatorname{Cov}(X,Y) = \sum_i \sum_j (x_i - \mu_x) (y_j - \mu_y) P(X = x_i, Y = y_j )$$


  • $P(X = x_i, Y = y_j) = \frac{1}{n}$ if $i=j$ since that outcome occurs $1/n$ times.
  • $P(X = x_i, Y = y_j) = 0 $ if $i \neq j$ since that outcome doesn't or didn't occur.

But then you'd just have:

$$ \sum_i \sum_j (x_i - \mu_x) (y_j - \mu_y) P(X = x_i, Y = y_j ) = \sum_i (x_i - \mu_x) (y_i - \mu_y) P(X = x_i, Y = y_i ) $$

Which is what the original formula is when $P(X = x_i, Y = y_i ) = \frac{1}{n}$.

You intuitively seem to want something like $P(X = x_i, Y = y_j ) = \frac{1}{n^2}$ but that is seriously wrong.

Simple dice example (to build intuition):

Let $X$ be the result of a roll of a single 6 sided die. Let $Y = X^2$.

Recall that a probability space has three components: a sample space $\Omega$, a set of events $\mathcal{F}$, and a probability measure $P$ that assigns probabilities to events. (I'm going to hand wave away the event stuff to keep it simpler.)

$X$and $Y$ are functions from $\Omega$ to $\mathcal{R}$. We can write out the possible values for $X$ and $Y$ as a function of $\omega \in \Omega$

$$ \begin{array}{rrr} & X(\omega) & Y(\omega) \\ \omega_1 & 1 & 1\\ \omega_2 & 2 & 4 \\ \omega_3 & 3 & 9 \\ \omega_4 & 4 & 16 \\ \omega_5 & 5 & 25 \\ \omega_6 & 6 & 36 \end{array} $$

We don't have 36 possible outcomes here. We have 6.

Since each outcome of a die is equally likely, we have $P( \{ \omega_1) \}) = P( \{ \omega_2) \}) = P( \{ \omega_3) \}) = P( \{ \omega_4) \}) = P( \{ \omega_5)\}) = P( \{ \omega_6) \}) = \frac{1}{6}$. (If your die wasn't fair, these numbers could be different.)

What's the mean of $X$?

\begin{align*} \operatorname{E}[X] = \sum_{\omega \in \Omega} X(\omega) P( \{ \omega \} ) &= 1 \frac{1}{6} + 2\frac{1}{6} + 3 \frac{1}{6} + 4 \frac{1}{6} + 5 \frac{1}{6} + 6 \frac{1}{6}\\ &= \frac{7}{2} \end{align*}

What's the mean of $Y$?

\begin{align*} \operatorname{E}[Y] = \sum_{\omega \in \Omega} X(\omega) P( \{ \omega \} ) &= 1 \frac{1}{6} + 4\frac{1}{6} + 9 \frac{1}{6} + 16 \frac{1}{6} + 25 \frac{1}{6} + 36 \frac{1}{6}\\ &= \frac{91}{6} \end{align*}

What's the covariance of $X$ and $Y$?

\begin{align*} \sum_{\omega \in \Omega} \left(X(\omega) - \frac{7}{2}\right)\left( Y(\omega) - \frac{91}{6}\right) P( \{ \omega \} ) &= \left( 1 - \frac{7}{2} \right)\left( 1 - \frac{91}{6} \right) P(\{\omega_1\}) + \left( 2 - \frac{7}{2} \right)\left( 4 - \frac{91}{6} \right) P(\{\omega_2\}) + \left( 3 - \frac{7}{2} \right)\left( 9 - \frac{91}{6} \right) P(\{\omega_3\}) + \left( 4 - \frac{7}{2} \right)\left( 16 - \frac{91}{6} \right) P(\{\omega_4\}) + \left( 5 - \frac{7}{2} \right)\left( 25 - \frac{91}{6} \right) P(\{\omega_5\}) + \left( 6 - \frac{7}{2} \right)\left( 36 - \frac{91}{6} \right) P(\{\omega_6\}) \\ &\approx 20.4167 \end{align*}

Don't worry about the arithmetic. The point is that to calculate $\operatorname{Cov}\left(X , Y\right) = \operatorname{E}\left[(X -\operatorname{E}[X])(Y - \operatorname{E}[Y]) \right] = \sum_{\omega \in \Omega} \left(X(\omega) - \operatorname{E}[X]\right)\left( Y(\omega) - \operatorname{E}[Y]\right) P( \{ \omega \} ) $ you sum over the 6 possible outcomes $\omega_1, \ldots, \omega_6$.

Back to your situation...

The possible outcomes in your sample are $i=1\, \ldots, n$. Those are the outcomes you should sum over.

  • $\begingroup$ Thanks a lot, I think that really helped. I messed up the concept of probability space. As I was somehow thinking of one probability space per random variable instead of realizing that the joint probability has its own probability space. Now the notation makes sense. $\endgroup$ – meow Mar 11 '17 at 22:51
  • $\begingroup$ @meow So one common situation that occurs is if you have two independent random variables $X$ and $Y$. Then you can construct the sample space for the joint distribution as the Cartesian product of the two sample spaces. The joint probability $P(X=x,Y=y)$ would equal to the product of the individual probabilities $P(X=x)P(Y=y)$. Then you would get a $\sum_i \sum_j$ type situation where $P(X,Y) = P(X)P(Y)$. (Of course, in this situation, the covariance would automatically be zero because $X$ and $Y$ are independent.) $\endgroup$ – Matthew Gunn Mar 12 '17 at 0:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.