# Covariance of data versus distribution

$\newcommand{\Cov}{\operatorname{Cov}}$I am getting hung up on what is probably a very basic point. We know that covariance is defined as $$\Cov(X,Y) = E(XY)-E(X)E(Y)$$ Now if we look at the definition of expected value, assuming $X,Y$ are discrete with mean 0 $$\Cov(X,Y) = \sum_{(x_i,y_i)}x_iy_iP(X=x_i,Y=y_i)$$ Now, go read any book on statistics and they will say that $$\Cov(X,Y) = \sum_i x_i y_i$$ without any mention of the distribution. Why do we no longer care about $P(X=x_i,Y=y_i)$? Is it because we observed $x_i,y_i$in the data so it happens with probability 1?

• Is it possible that there is a factor $\frac{1}{n}$ missing in the last formula? Mar 3 '17 at 20:11

The probability doesn't disappear; it's the $\frac{1}{n}$ term.

• Let $P$ be the true probability measure.
• Let $Q$ be the the empirical measure, the probability measure implicitly defined by your random sample.

### What is the empirical measure Q?

If you have $n$ independent observations and a set of unique, observed outcomes $\mathcal{A}$, that is, $\mathcal{A} = \left\{x_1, x_2, \ldots, x_n \right\}$

$$Q(X = x) = \left\{\begin{array}{rl}\frac{1}{n}: & \text{if x belongs to the sample \mathcal{A}}\\ 0 :& \text{otherwise}\end{array} \right.$$

Note: If observations occur multiple times, then the probability would be $\frac{k}{n}$ where $k$ is the number of times that observation $x$ appeared in the sample.

### Application to calculating the mean:

What is the mean according to empirical measure $Q$?

\begin{align*} \operatorname{E}^Q\left[X\right] &= \sum_{x \in \mathcal{A}} x \,Q(X = x) \\ &= \sum_{x \in \mathcal{A}} x \frac{1}{n}\\ &= \frac{1}{n} \sum_{x \in \mathcal{A}} x \end{align*}

Note also that $\sum_{x \in \mathcal{A}} x$ is just another way to write $\sum_{i=1}^n x_i$ since $\mathcal{A} = \left\{x_1, x_2, \ldots, x_n \right\}$.

The formula $\displaystyle \frac 1 n \sum_{i=1}^n x_i y_i$ is valid if $\Pr((X,Y)=(x_i,y_i)) = \dfrac 1 n.$ That is the empirical distribution.