Covariance for Random Variables vs. Sample Data

Question

In my textbook, it says that the formula for finding covariance between two random variables is:

$Cov(X,Y)=E((X-EX)(Y-EY))$

With $EY$ and $EX$ being the mathematical expectation for the random variable Y and X respectively.

How does this formula translate into:

$Cov(X,Y) = \frac{\sum (x-\bar x)(y-\bar y)}{n-1}$

For when we are calculating with real data (sampled data)?

Let's say I want to calculate the covariance between two stock prices in a given month. Of course, I will resort to the 2nd formula to find the covariance. However, the fundamental question I want to ask is, for the first formula we are talking in the context of random variables, we assume that we know the underlying distributions of X and Y (as is with the examples in my textbook). However, in practical applications such as above, when I want to calculate covariance between two stock prices, I don't know the underlying distribution of the two stock prices data that I've sampled.

I understand how to apply the first formula, but only if I know the distribution of the random variable (be it $N(0,1)$ or any other common distributions shown in most textbooks). But what is the intuitive approach when dealing with real, sampled, data of which we don't know the distribution?

Answer 1

The second one is an estimate of covariance, i.e. $\widehat{\operatorname{cov}(X,Y)}$. A typical estimate of a joint moment is $$\widehat{E[f(X,Y)]}=\frac{1}{n}\sum_{i=1}^n f(x_i,y_i)$$ where the covariance estimate formula is based on. The value we divide for averaging operation is chosen to be $n-1$ instead of $n$ to make it an unbiased estimator (Bessel's correction). So, this is not a theoretical calculation as the first one.

Answer 2

The simplest approach when you have a sample but do not know from what distribution, or are not willing to assume a particular distribution as a model, is to use the empirical distribution. That is, the probability of observing the value $x$ is set to be the observed proportion in the sample, so if there are $k$ observations equal to $x$ and the sample has size N:

$$P[X=x] = k/N$$

Any values that you do not observe in your sample are given probability zero. You can check that these probabilities sum to 1 and that this is a valid distribution.

Now, as you know, the sample mean is defined like this:

$$\bar{x} = \frac{1}{N}\sum_{i=1}^N x_i$$

If you group together the observations that have the same value, denoting each distinct possible value as $x^{(1)}, ... , x^{(M)}$ (where $M \leq N$ since you can have values appear more than once in your sample), then this is the same as:

$$\bar{x} = \frac{1}{N}\sum_{j=1}^M (k_j \cdot x^{(j)}) = \sum_{j=1}^M \frac{k_j}{N}x^{(j)} = \sum_{j=1}^M P[X=x^{(j)}] x^{(j)} = E(X)$$

That is, you can consider the sample mean as the usual mathematical expectation of a random variable which is computed under the empirical distribution. That is one useful interpretation for the link between those two concepts.

The second formula you show (the sample covariance) can be interpreted similarly: it can be derived from the first one by assuming the empirical distribution, except for the small detail that it is then multiplied by $\frac{N}{N-1}$. For a large sample, this is close to 1, so does not make a big difference. This estimator uses a correction for bias, as pointed out in another answer. This is a technical detail that doesn't change the intuition behind the formula.