3
$\begingroup$

Given the general definition of Covariance between two random variables $x$ and $y$: \begin{equation*} \text{Cov}(x,y)=\frac{1}{n}\sum_{i=1}^n(x_i-\bar{x})(y_i-\bar{y}) \end{equation*} Does the above definition implicitly assume that every bivariate observation $(x_i,y_i)$ has the same relative frequency, namely equal to $\frac{1}{n}$?

I would expect that for bivariate observations with different relative frequency, the above definition (given that the value set of component random variable $x$ has $r$ values while that of $y$ has $s$ values) would become equal to: \begin{equation} \text{Cov}(x,y)=\sum_{i=1}^r\sum_{j=1}^sf_{x,y}(x_i,y_i)(x_i-\bar{x})(y_i-\bar{y}) \end{equation} with $f_{x,y}(x_i,y_i)$ denotes the relative frequency of the pair $(x_i, y_i)$.

Is my reasoning correct or am I wrong? Why?

$\endgroup$
6
  • 2
    $\begingroup$ To make things easier, you could ask the same question about the ordinary mean ... $\endgroup$ Commented Oct 14, 2021 at 13:45
  • $\begingroup$ OK, your answer seems to me like "yes, it depends on what you are dealing with: either with equally-weighted observations (first definition) or differently-weighted observations (second definition)", correct? @BigBendRegion $\endgroup$ Commented Oct 14, 2021 at 13:47
  • 1
    $\begingroup$ I'm having hard time understanding what you "different relative frequency." To calculate we need paired observations of x and y. Thus, by definition there is no difference in frequency. I imagine I am misunderstanding your point. Can you clarify? $\endgroup$ Commented Oct 14, 2021 at 13:48
  • 2
    $\begingroup$ No, it is just easier to answer. The unweighted formula and the weighted formula are equivalent. If weights are needed, they reflect repeats in the data of certain observations, and this is implicitly taken care of by the unweighted formula. Just make up a small data set with some repeats, calculate the the average, and you will see. $\endgroup$ Commented Oct 14, 2021 at 13:50
  • $\begingroup$ See en.wikipedia.org/wiki/Covariance#Discrete_random_variables $\endgroup$
    – Sergio
    Commented Oct 14, 2021 at 14:10

3 Answers 3

9
$\begingroup$

You are confusing the estimator for covariance with covariance itself. Covariance is defined as

$$ \operatorname{cov}(X, Y) = \operatorname{E}{\big[(X - \operatorname{E}[X])(Y - \operatorname{E}[Y])\big]} $$

Where $X$ and $Y$ are random variables and by calculating expected values we do consider their distributions and relative probabilities when calculating the integrals to obtain expected values

$$ \operatorname{E}[X] = \int x f(x)\, dx $$

As noticed by @BigBendRegion in the comment, it's easier to show for mean as an estimator for expected value. For discrete data, the expected value is defined as

$$ E[X] = \sum_i x_i p_i $$

If you want to estimate it from the sample, the problem is that you don't know the $P(X = x_i) = p_i$ probabilities. You can estimate them from the sample by looking at the relative frequencies $\tfrac{n_i}{n}$ where $n_i$ is the count of how many times did you observe $x_i$'th value (notice: this is not $i$-th sample, but $i$-th distinct value) in your sample, with $n = \sum_i n_i$. In such a case,

$$ \sum_i x_i \tfrac{n_i}{n} = \sum_i \,x_i n_i \tfrac{1}{n} $$

it is the same as if you repeated the $\tfrac{1}{n} x_i$ operation $n_i$ times in the summation. So using arithmetic mean is the same as if you calculated the relative frequencies $\hat p_i = \tfrac{n_i}{n}$ and used them to calculate $\sum_i x_i \hat p_i$. Exactly the same thing happens for sample covariance, but the math is just a little bit more cluttered.

$\endgroup$
1
  • 4
    $\begingroup$ @Paul Can you elaborate? $\endgroup$ Commented Oct 14, 2021 at 21:50
7
$\begingroup$

I interpret this to mean that you are unhappy if you have a data set like this.

$$ X, Y\\ 1,1\\ 2,3\\ 1,1\\ 0, -1 $$

In this case, the $(1,1)$ is repeated, so you want to weight it double. However, that is covered by the formula.

$$ cov(X, Y) = \frac{1}{4}\sum_{i = 1}^4 (X_i - \bar X)(Y_i-\bar Y)\\ =\dfrac{(1-1)(1-1) + (2 - 1)(3 - 1) + (1 - 1)(1 - 1) + (0 - 1)(\text{-}1 - 1) }{4} $$

The values get repeated in the sum, so there is no need to weight observations according to how many times they appear.

$\endgroup$
0
$\begingroup$

Use of the Roman alphabet rather than Greek implies that we are talking about sample observations rather than the population. In a sample, each observation does in fact have the same relative frequency (if the same pair of values appears more than once, they will still be different observations, and each of those observations appears once). However, if the mean is being estimated by $\bar x$ rather than being known, then $\frac 1{N-1}$ rather than $\frac 1 N$ is used.

For the population covariance, the formula is $\int \int p(x,y)(x-\mu_x)(y-\mu_y)dxdy$, which for a discrete distribution reduces to $\sum \sum p(x,y)(x-\mu_x)(y-\mu_y)$

$\endgroup$
2
  • 1
    $\begingroup$ If variance is any indication, $\frac{1}{N}$ gives a maximum likelihood estimator (at least for a Gaussian), while $\frac{1}{N-1}$ an unbiased estimator (not just for a Gaussian). Even if that is false for covariance (I’m pretty sure it’s true), the $\frac{1}{N-1}$ variant could be a perfectly defendable estimator, even if it is less common. $\endgroup$
    – Dave
    Commented Oct 14, 2021 at 22:17
  • $\begingroup$ In my last sentence, I meant the $\frac{1}{N}$ variant. $\endgroup$
    – Dave
    Commented Oct 15, 2021 at 22:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.