# A trivial question about Covariance

I'm just learning about Covariance and encountered something I don't quite understand.

Assume we have two random variables X and Y, where the respective joint-probability function assigns equal weights to each event.

According to wikipedia the Cov(X, Y) can then be caluculated as follows:

$${\displaystyle \operatorname {cov} (X,Y)={\frac {1}{n}}\sum _{i=1}^{n}(x_{i}-E(X))(y_{i}-E(Y)).}$$

What confuses me is the fact, that they sum only over $$x_i$$ and $$y_i$$; $$i=1,...,n$$ , but not $$x_i$$ and $$y_j; \;i=1,...,n$$ $$j=1,...,m$$ , thus many possible combinations are not calculated. In other words if we look at each calculated combination individually, we only get a $$n*1$$ matrix, instead of a $$n*m$$ matrix.

Can anyone explain this (I suppose it's rather obvious but I just don't see the reason at the moment).

• Maybe this helps. Lets say you are interested in the relationship between height and weight in people. Covariance could give an indication of this relationship. In this case I am interested if someone with a larger height weighs more, or less. This means that I am interested in the relationship between the height and weight of each individual person, over all persons. I am not interested in the relationship between person A s weight and person B height. Hence, in the calculation both features should stem from the same observational unit, a person. The $i$ designates a particular person. – spdrnl Mar 11 '17 at 17:05
• I was typing something along these lines, but I erased it now, because the prior comment summarizes it well. – Antoni Parellada Mar 11 '17 at 17:06

The idea is that the possible outcomes in your sample are $i=1, \ldots, n$, and each outcome $i$ has equal probability $\frac{1}{n}$ (under the probability measure that assigns equal probability to all outcomes that you appear to be using). You have $n$ outcomes, not $n^2$.

To somewhat indulge your idea, you could compute:

$$\operatorname{Cov}(X,Y) = \sum_i \sum_j (x_i - \mu_x) (y_j - \mu_y) P(X = x_i, Y = y_j )$$

Where:

• $P(X = x_i, Y = y_j) = \frac{1}{n}$ if $i=j$ since that outcome occurs $1/n$ times.
• $P(X = x_i, Y = y_j) = 0$ if $i \neq j$ since that outcome doesn't or didn't occur.

But then you'd just have:

$$\sum_i \sum_j (x_i - \mu_x) (y_j - \mu_y) P(X = x_i, Y = y_j ) = \sum_i (x_i - \mu_x) (y_i - \mu_y) P(X = x_i, Y = y_i )$$

Which is what the original formula is when $P(X = x_i, Y = y_i ) = \frac{1}{n}$.

You intuitively seem to want something like $P(X = x_i, Y = y_j ) = \frac{1}{n^2}$ but that is seriously wrong.

## Simple dice example (to build intuition):

Let $X$ be the result of a roll of a single 6 sided die. Let $Y = X^2$.

Recall that a probability space has three components: a sample space $\Omega$, a set of events $\mathcal{F}$, and a probability measure $P$ that assigns probabilities to events. (I'm going to hand wave away the event stuff to keep it simpler.)

$X$and $Y$ are functions from $\Omega$ to $\mathcal{R}$. We can write out the possible values for $X$ and $Y$ as a function of $\omega \in \Omega$

$$\begin{array}{rrr} & X(\omega) & Y(\omega) \\ \omega_1 & 1 & 1\\ \omega_2 & 2 & 4 \\ \omega_3 & 3 & 9 \\ \omega_4 & 4 & 16 \\ \omega_5 & 5 & 25 \\ \omega_6 & 6 & 36 \end{array}$$

We don't have 36 possible outcomes here. We have 6.

Since each outcome of a die is equally likely, we have $P( \{ \omega_1) \}) = P( \{ \omega_2) \}) = P( \{ \omega_3) \}) = P( \{ \omega_4) \}) = P( \{ \omega_5)\}) = P( \{ \omega_6) \}) = \frac{1}{6}$. (If your die wasn't fair, these numbers could be different.)

### What's the mean of $X$?

\begin{align*} \operatorname{E}[X] = \sum_{\omega \in \Omega} X(\omega) P( \{ \omega \} ) &= 1 \frac{1}{6} + 2\frac{1}{6} + 3 \frac{1}{6} + 4 \frac{1}{6} + 5 \frac{1}{6} + 6 \frac{1}{6}\\ &= \frac{7}{2} \end{align*}

### What's the mean of $Y$?

\begin{align*} \operatorname{E}[Y] = \sum_{\omega \in \Omega} X(\omega) P( \{ \omega \} ) &= 1 \frac{1}{6} + 4\frac{1}{6} + 9 \frac{1}{6} + 16 \frac{1}{6} + 25 \frac{1}{6} + 36 \frac{1}{6}\\ &= \frac{91}{6} \end{align*}

### What's the covariance of $X$ and $Y$?

\begin{align*} \sum_{\omega \in \Omega} \left(X(\omega) - \frac{7}{2}\right)\left( Y(\omega) - \frac{91}{6}\right) P( \{ \omega \} ) &= \left( 1 - \frac{7}{2} \right)\left( 1 - \frac{91}{6} \right) P(\{\omega_1\}) + \left( 2 - \frac{7}{2} \right)\left( 4 - \frac{91}{6} \right) P(\{\omega_2\}) + \left( 3 - \frac{7}{2} \right)\left( 9 - \frac{91}{6} \right) P(\{\omega_3\}) + \left( 4 - \frac{7}{2} \right)\left( 16 - \frac{91}{6} \right) P(\{\omega_4\}) + \left( 5 - \frac{7}{2} \right)\left( 25 - \frac{91}{6} \right) P(\{\omega_5\}) + \left( 6 - \frac{7}{2} \right)\left( 36 - \frac{91}{6} \right) P(\{\omega_6\}) \\ &\approx 20.4167 \end{align*}

Don't worry about the arithmetic. The point is that to calculate $\operatorname{Cov}\left(X , Y\right) = \operatorname{E}\left[(X -\operatorname{E}[X])(Y - \operatorname{E}[Y]) \right] = \sum_{\omega \in \Omega} \left(X(\omega) - \operatorname{E}[X]\right)\left( Y(\omega) - \operatorname{E}[Y]\right) P( \{ \omega \} )$ you sum over the 6 possible outcomes $\omega_1, \ldots, \omega_6$.

The possible outcomes in your sample are $i=1\, \ldots, n$. Those are the outcomes you should sum over.
• @meow So one common situation that occurs is if you have two independent random variables $X$ and $Y$. Then you can construct the sample space for the joint distribution as the Cartesian product of the two sample spaces. The joint probability $P(X=x,Y=y)$ would equal to the product of the individual probabilities $P(X=x)P(Y=y)$. Then you would get a $\sum_i \sum_j$ type situation where $P(X,Y) = P(X)P(Y)$. (Of course, in this situation, the covariance would automatically be zero because $X$ and $Y$ are independent.) – Matthew Gunn Mar 12 '17 at 0:18