36
$\begingroup$

The denominator of the (unbiased) variance estimator is $n-1$ as there are $n$ observations and only one parameter is being estimated.

$$ \mathbb{V}\left(X\right)=\frac{\sum_{i=1}^{n}\left(X_{i}-\overline{X}\right)^{2}}{n-1} $$

By the same token I wonder why shouldn't the denominator of covariance be $n-2$ when two parameters are being estimated?

$$ \mathbb{Cov}\left(X, Y\right)=\frac{\sum_{i=1}^{n}\left(X_{i}-\overline{X}\right)\left(Y_{i}-\overline{Y}\right)}{n-1} $$

$\endgroup$
  • 15
    $\begingroup$ If you did that, you would have two conflicting definitions for the variance: one would be the first formula and the other would be the second formula applied with $Y=X$. $\endgroup$ – whuber Mar 19 '15 at 14:19
  • 3
    $\begingroup$ A bi/multivariate mean (expectation) is one, not 2 parameters. $\endgroup$ – ttnphns Mar 19 '15 at 15:20
  • 14
    $\begingroup$ @ttnphns That's not true: the bivariate mean is obviously two parameters because it requires two real numbers to express it. (Indeed it is a single vector parameter, but saying so only disguises the fact it has two components.) This shows up explicitly in the degrees of freedom for pooled-variance t-tests, for instance, where $2$ is subtracted, not $1$. What's interesting about this question is how it reveals just how vague, unrigorous, and potentially misleading is the common "explanation" that we subtract $1$ from $n$ because one parameter has been estimated. $\endgroup$ – whuber Mar 19 '15 at 16:31
  • $\begingroup$ @whuber, You are right at that. If it were only $n$ (independent observations) which matters we wouldn't spend more df in multivariate tests than in univariate ones. $\endgroup$ – ttnphns Mar 19 '15 at 16:41
  • 3
    $\begingroup$ @whuber: I would perhaps say that it shows that what counts as "a parameter" depends on the situation. In this case variance is computed over $n$ observations and so each observation -- or the total mean -- can be seen as one parameter, even if it is a multivariate mean, as ttnphns said. However, in other cases when e.g. a test considers linear combinations of dimensions, each dimension of each observation becomes "a parameter". You are right that this is a tricky issue. $\endgroup$ – amoeba says Reinstate Monica Mar 19 '15 at 21:21
31
$\begingroup$

Covariances are variances.

Since by the polarization identity

$$\newcommand{\c}{\text{Cov}}\newcommand{\v}{\text{Var}} \c(X,Y) = \v\left(\frac{X+Y}{2}\right) - \v\left(\frac{X-Y}{2}\right),$$

the denominators must be the same.

$\endgroup$
20
$\begingroup$

A special case ought to give you an intuition; think about the following:

$$\hat{\mathbb{Cov}}\left(X, X\right)= \hat{\mathbb{V}}\left(X\right)$$

You are happy that the latter is $\frac{\sum_{i=1}^{n}\left(X_{i}-\overline{X}\right)^{2}}{n-1}$ due to the Bessel correction.

But replacing $Y$ by $X$ in $\hat{\mathbb{Cov}}\left(X, Y\right)$ for the the former gives $\frac{\sum_{i=1}^{n}\left(X_{i}-\overline{X}\right)\left(X_{i}-\overline{X}\right)}{\text{mystery denominator}}$, so what do you now think might best fill in the blank?

$\endgroup$
  • 1
    $\begingroup$ OK. But the OP might ask "why to consider cov(X,X) and cov(X,Y) to be in one line of logic? Why are you replacing Y by X in cov() flippantly? Maybe cov(X,Y) is a different situation?" You didn't avert that, while the answer (highly upvoted) ought to have, in my impression :-) $\endgroup$ – ttnphns Dec 20 '16 at 19:20
7
$\begingroup$

A quick and dirty answer... Let’s consider first $\text{var}(X)$; if you had $n$ observations with known expected value $E(X) = 0$ you would use ${1\over n}\sum_{i=1}^n X_i^2$ to estimate the variance.

The expected value being unknown, you can transform your $n$ observations into $n-1$ observations with known expected value by taking $A_i = X_i - X_1$ for $i = 2, \dots,n$. You will get a formula with a $n-1$ in the denominator — however the $A_i$ are not independent and you’d have to take this into account; at the end you’d find the usual formula.

Now for the covariance you can use the same idea: if the expected value of $(X,Y)$ was $(0,0)$, you’d had a ${1\over n}$ in the formula. By subtracting $(X_1,Y_1)$ to all other observed values, you get $n-1$ observations with known expected value... and a ${1\over n-1}$ in the formula — once again, this introduces some dependence to take into account.

P.S. The clean way to do that is to choose an orthonormal basis of $\big\langle (1, \dots, 1)' \big\rangle^{\perp}$, that is $n-1$ vectors $c_1, \dots, c_{n-1} \in \mathbb R^n$ such that

  • $\sum_j c_{ij}^2 = 1$ for all $i$,
  • $\sum_j c_{ij} = 0$ for all $i$,
  • $\sum_j c_{i_1j} c_{i_2j} = 0$ for all $i_1 \ne i_2$.

You can then define $n-1$ variables $A_i = \sum_j c_{ij} X_j$ and $B_i = \sum_j c_{ij} Y_j$. The $(A_i,B_i)$ are independent, have expected value $(0,0)$, and have same variance/covariance than the original variables.

All the point is that if you want to get rid of the unknown expectation, you drop one (and only one) observation. This works the same for both cases.

$\endgroup$
6
$\begingroup$

Here is a proof that the p-variate sample covariance estimator with denominator $\frac{1}{n-1}$ is an unbiased estimator of the covariance matrix:

$ x' = (x_1,...,x_p) $.

$\Sigma= E((x-\mu)(x-\mu)') $

$S = \frac{1}{n} \sum (x_i - \bar{x})(x_i - \bar{x})'$

To show: $E(S) = \frac{n-1}{n}\Sigma$

Proof: $S= \frac{1}{n}\sum x_ix_i' - \bar{x}\bar{x}'$

Next:

(1) $ E(x_ix_i') = \Sigma + \mu\mu'$

(2) $E(\bar{x}\bar{x}') = \frac{1}{n} \Sigma+ \mu\mu' $

Therefore: $E(S) = \Sigma + \mu\mu' - (\frac{1}{n} \Sigma+ \mu\mu') = \frac{n-1}{n} \Sigma $

And so $S_u = \frac{n}{n-1}S $, with the final denominator $\frac{1}{n-1}$, is unbiased. The off-diagonal elements of $S_u$ are your individual sample covariances.

Additional remarks:

  1. The n draws are independent. This is used in (2) to calculate the covariance of the sample mean.

  2. Step (1) and (2) use the fact that $Cov(x)= E[xx']-\mu\mu'$

  3. Step (2) uses the fact that $Cov(\bar{x})= \frac{1}{n}\Sigma$

$\endgroup$
  • $\begingroup$ The difficulty being in step 2 ! :) $\endgroup$ – Elvis Mar 19 '15 at 14:54
  • $\begingroup$ @Elvis It's messy. One needs to apply the rule Cov(X+Y,Z)=Cov(X,Z) + Cov(Y,Z) and recognize that the different draws are independent. Then it's basically summing up the covariance n times and scaling it down by 1/n² $\endgroup$ – statchrist Mar 19 '15 at 15:13
4
$\begingroup$

I guess one way to build intuition behind using 'n-1' and not 'n-2' is - that for calculating co-variance we do not need to de-mean both X and Y, but either of the two, i.e.

$\endgroup$
  • $\begingroup$ Could you elaborate on how this bears on the question of what denominator to use? The algebraic relation in evidence derives from the fact that the residuals relative to the mean sum to zero, but otherwise is silent about which denominator is relevant. $\endgroup$ – whuber Nov 24 '15 at 14:14
  • 5
    $\begingroup$ I came here because I had the same question as the OP. I think this answer gets at the nub of the point @whuber pointed out above: that the rule of thumb is that df ~= n - (parameters estimated) can be "vague, unrigorous, and potentially misleading." This points out the fact that though it looks like you need to estimate two parameters (xbar and ybar), you really only estimate one (xbar or ybar). Since the df should be the same in both cases, it must be the lower of the two. I think that is the intent here. $\endgroup$ – mpettis Dec 11 '15 at 20:30
1
$\begingroup$

1) Start $df=2n$.

2) Sample covariance is proportional to $\Sigma_{i=1}^n(X_i-\bar{X})(Y_i-\bar{Y})$. Lose two $df$; one from $\bar{X}$, one from $\bar{Y}$ resulting in $df=2(n-1)$.

3) However, $\Sigma_{i=1}^n(X_i-\bar{X})(Y_i-\bar{Y})$ only contains $n$ separate terms, one from each product. When two numbers are multiplied together the independent information from each separate number disappears.

As a trite example, consider that

$24=1*24=2*12=3*8=4*6=6*4=8*3=12*2=24*1$,

and that does not include irrationals and fractions, e.g. $24=2\sqrt{6}*2\sqrt{6}$, so that when we multiply two number series together and examine their product, all we see are the $df=n-1$ from one number series, as we have lost half of the original information, that is, what those two numbers were before the pair-wise grouping into one number (i.e., multiplication) was performed.

In other words, without loss of generality we can write

$(X_i-\bar{X})(Y_i-\bar{Y})=z_i-\bar{z}$ for some $z_i$ and $\bar{z}$,

i.e., $z_i=X_iY_i-\bar{X}Y_i-X_i\bar{Y}$, and, $\bar{z}=\bar{X}\bar{Y}$. From the $z$'s, which then clearly have $df=n-1$, the covariance formula becomes

$\Sigma_{i=1}^n\frac{z_i-\bar{z}}{n-1}=$

$\Sigma_{i=1}^n\frac{[(X_i-\bar{X})(Y_i-\bar{Y})]}{n-1}=$

$\frac{1}{n-1}\Sigma_{i=1}^n(X_i-\bar{X})(Y_i-\bar{Y})$.

Thus, the answer to the question is that the $df$ are halved by grouping.

$\endgroup$
  • $\begingroup$ @whuber How on earth did I get the same thing posted twice and deleted once? What gives? Can we get rid of one of them? For future reference, is there any way to permanently delete such duplicates? I have a few hanging around and it's annoying. $\endgroup$ – Carl Dec 16 '16 at 22:38
  • $\begingroup$ As far as I can tell, you reposted your answer from the duplicate to here. (Nobody else has the power to post answers in your name.) The system strongly discourages posting identical answers in multiple threads, so when I saw that, it convinced me these two threads are perfect duplicates and I "merged" them. This is a procedure that moves all comments and answers from the source thread to the target thread. I then deleted your duplicate post here in the target thread. It will remain permanently deleted, but will be visible to you as well as to people of sufficiently high reputation. $\endgroup$ – whuber Dec 16 '16 at 23:10
  • $\begingroup$ @whuber I didn't know what happens in a merge, that a merge was taking place or what many of the rules are, despite looking things up constantly. It takes time to learn, be patient, BTW, would you consider taking stats.stackexchange.com/questions/251700/… off of Hold? $\endgroup$ – Carl Dec 16 '16 at 23:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.