$\newcommand{\Cov}{\operatorname{Cov}}$I am getting hung up on what is probably a very basic point. We know that covariance is defined as $$\Cov(X,Y) = E(XY)-E(X)E(Y)$$ Now if we look at the definition of expected value, assuming $X,Y$ are discrete with mean 0 $$\Cov(X,Y) = \sum_{(x_i,y_i)}x_iy_iP(X=x_i,Y=y_i)$$ Now, go read any book on statistics and they will say that $$\Cov(X,Y) = \sum_i x_i y_i$$ without any mention of the distribution. Why do we no longer care about $P(X=x_i,Y=y_i)$? Is it because we observed $x_i,y_i$in the data so it happens with probability 1?

  • 4
    $\begingroup$ Is it possible that there is a factor $\frac{1}{n}$ missing in the last formula? $\endgroup$
    – JohnK
    Mar 3 '17 at 20:11

The probability doesn't disappear; it's the $\frac{1}{n}$ term.

  • Let $P$ be the true probability measure.
  • Let $Q$ be the the empirical measure, the probability measure implicitly defined by your random sample.

What is the empirical measure Q?

If you have $n$ independent observations and a set of unique, observed outcomes $\mathcal{A}$, that is, $\mathcal{A} = \left\{x_1, x_2, \ldots, x_n \right\}$

$$Q(X = x) = \left\{\begin{array}{rl}\frac{1}{n}: & \text{if $x$ belongs to the sample $\mathcal{A}$}\\ 0 :& \text{otherwise}\end{array} \right.$$

Note: If observations occur multiple times, then the probability would be $\frac{k}{n}$ where $k$ is the number of times that observation $x$ appeared in the sample.

Application to calculating the mean:

What is the mean according to empirical measure $Q$?

\begin{align*} \operatorname{E}^Q\left[X\right] &= \sum_{x \in \mathcal{A}} x \,Q(X = x) \\ &= \sum_{x \in \mathcal{A}} x \frac{1}{n}\\ &= \frac{1}{n} \sum_{x \in \mathcal{A}} x \end{align*}

Note also that $\sum_{x \in \mathcal{A}} x$ is just another way to write $\sum_{i=1}^n x_i$ since $\mathcal{A} = \left\{x_1, x_2, \ldots, x_n \right\}$.


The formula $\displaystyle \frac 1 n \sum_{i=1}^n x_i y_i$ is valid if $\Pr((X,Y)=(x_i,y_i)) = \dfrac 1 n.$ That is the empirical distribution.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.