3
$\begingroup$

We define the covariance of two random variables $X$ and $Y$ as $Cov(X,Y) = E[(X - \mu_X)(Y - \mu_Y)]$. The covariance measures the "linear dependence" between the two r.v s.

But in a lot of places, on the internet as well as in a course I am taking, I see the covariance defined for a set of paired data points $(x_1, y_1), (x_2, y_2), ..., (x_N, y_N)$ as $$Cov(X, Y) = \frac{\sum_{t=1}^N (x_t - \mu_X)(y_t - \mu_Y)}{N-1},$$which looks like an average has been taken. Does this assume that $(X-\mu_X)(Y-\mu_Y)$ follows a uniform distribution?

$\endgroup$

2 Answers 2

10
$\begingroup$

The two formulas aren't the same. The first formula is the formula for population covariance, denoted by $Cov(X,Y)$ (and also $\sigma_{XY}$):

$Cov(X,Y) = E[(X - \mu_X)(Y - \mu_Y)]$

Suppose if the population contains exactly $N$ pairs of $(x, y)$, i.e. $(x_1, y_1), (x_2, y_2), ..., (x_N, y_N)$, and you know the values of all those pairs, then it turns out to be equal to:

$$ {Cov}(X, Y) = \frac{\sum_{t=1}^N (x_t - \mu_X)(y_t - \mu_Y)}{N}. $$

However, this is usually not the case because normally the population size is very large. So you can only sample a part of its data, i.e. collect some number of pairs $(x, y)$ but not all of them.

In such case, the second formula you wrote is used and it is called the sample covariance, denoted by $\widehat{Cov}(X, Y)$ (and also $S_{XY}$). It's an estimate of $Cov(X,Y)$ because it's used to estimate (or predict) the value of the population covariance.

So for example, if you collect a sample data of size $N$, $(x_1, y_1), (x_2, y_2), ..., (x_N, y_N)$, from the population (but the population size is much greater than $N$), then the sample covariance is:

$$ \widehat{Cov}(X, Y) = \frac{\sum_{t=1}^N (x_t - \mu_X)(y_t - \mu_Y)}{N-1}. $$

For your question: There's no assumption on the distribution of $(X-\mu_X)(Y-\mu_Y)$. However, you may be able to calculate this distribution if you know the distribution of $X$ and $Y$.

Note: The formula you saw for sample covariance is an unbiased estimate of the $Cov(X,Y)$. If you see $N$ in the denominator instead of $N-1$ then it's a biased estimate.

$\endgroup$
4
  • $\begingroup$ Hi, in your first part (where the population contains exactly N pairs), how can you equate the expectation to the arithmetic average without making an assumption about distributions? $\endgroup$ Oct 28, 2023 at 14:53
  • $\begingroup$ From the very first formula for $Cov(X, Y)$, just expand the formula for expectation (in the discrete case), do some algebra (to match the result of the arithmetic average) and you'll see that it works for any distributions of $X$ and $Y$. $\endgroup$
    – Tran Khanh
    Oct 28, 2023 at 16:49
  • $\begingroup$ @insipidintegrator : I had the same intuition as you regarding needing some assumption about the distribution but it turns out not to be the case. Here are some nice proofs that show that one does not need any additional distributional assumptions. math.stackexchange.com/questions/2019122/… $\endgroup$
    – mlofton
    Oct 29, 2023 at 4:02
  • $\begingroup$ Thanks a lot @TranKhanh and @mlofton! $\endgroup$ Nov 6, 2023 at 9:36
3
$\begingroup$

It does not assume a uniform distribution.

It assumes that you sampled the data $x_j$ and $y_j$ so that these occur in your samples with a frequency according to the probability distribution of the corresponding data generating processes $X$ and $Y$.

For example: If $X$ is a standard normal r.v. then many of your $x_j$'s would be close to 0. If you did plot them in a histogram then it would look similar to a standard normal distribution.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.