6
$\begingroup$

I was trying to derive the equations from page 109 in "elements of statistical learning" (image below) enter image description here

To be honest, I am not sure how the covariance $\Sigma$ is estimated (the third bullet point in image). Can someone kindly show me and how $\hat{\Sigma}$ is derived (in particular I am not sure how $N - K$ appear in the denominator) ? Thanks

UPDATE: I still cannot figure why $N-K$ (where $K$ is the number of parameters) shows up in the denominator, but I suspect it has to do with making the covariance matrix estimator unbiased. This is similar to $\frac{1}{N} \sum_i (x_i - \bar{x})^2$ is biased, while $s^2 = \frac{1}{N-1} \sum_i (x_i - \bar{x})^2$ is not. Please CORRECT ME IF I AM WRONG.

$\endgroup$
3
  • $\begingroup$ Yes, N-K (K is the number of classes) is the proper degrees of freedom. You see the same in ANOVA. But why do you call Sigma_hat covariance? In this instance, it is variance. $\endgroup$ – ttnphns Mar 19 '14 at 20:46
  • $\begingroup$ @ttnphns: In this quote it is a covariance matrix (because $x_i$ are vectors). $\endgroup$ – amoeba Mar 20 '14 at 14:39
  • $\begingroup$ stats.stackexchange.com/questions/3931/… but you have estimated $K$ means $\endgroup$ – user83346 Feb 16 '16 at 6:05
6
$\begingroup$

You are right. The equation for the shared variance-covariance matrix comes from Pooled Variance

The shared covariance matrix $\Sigma$ is taken as a weighted average of individual covariance matrices, weigted by $n_i-1$

So $\Sigma = \frac{(n_1-1) * \Sigma_1 + (n_2-1) * \Sigma_2 + .... + (n_k-1) * \Sigma_k}{(n_1-1)+(n_2-1)+...+(n_k-1)}$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.