0
$\begingroup$

I am reading both LDA lecturer notes and follow the example, both having different covariance calculation.

In LDA lecturer note 1 slide 19,

Did not multiple by number of samples when calculating Covaraince matrix S1, S2.

enter image description here

In LDA lecture note 2' slide 20

She calculate covariance using by multiple the number of samples.

enter image description here

$\endgroup$

1 Answer 1

1
$\begingroup$

The confusion is most likely due to the n and (n - 1) difference in variance estimators.

$$V_{biased} = \dfrac{\sum_{i=1}^{n} (z_i - \mu_z)^2}{n}$$ $$V_{unbiased} = \dfrac{\sum_{i=1}^{n} (z_i - \mu_z)^2}{n - 1}$$

On slide 9 you define scatter as biased variance estimator without dividing by n: $$S = \sum_{i=1}^{n} (z_i - \mu_z)^2 = n ((\sum_{i=1}^{n} (z_i - \mu_z)^2) / n) = n V_{biased}[z]$$

You can write an equivalent formula based on unbiased variance estimator: $$S = \sum_{i=1}^{n} (z_i - \mu_z)^2 = (n - 1) ((\sum_{i=1}^{n} (z_i - \mu_z)^2) / (n - 1)) = (n - 1) V_{unbiased}[z]$$

On slide 20 you use the scatter $S_1$ based on the covariance matrix $cov(c_1)$ that is calculated by some software that uses the $V_{unbiased}$ formula:

$$S_1 = (n_1 - 1) V_{unbiased}[z_1] = 5 * \left[ {\begin{array}{cc} 10 & 8 \\ 8 & 7.2 \\ \end{array} } \right] $$

You can check that it all adds up in any computer language or manually, e.g. here is R:

> c1 = t(matrix(c(1, 2, 2, 3, 3, 3, 4, 5, 5, 5), nrow = 2))
> (nrow(c1) - 1) * var(c1)
     [,1] [,2]
[1,]   10  8.0
[2,]    8  7.2
> # First element of the scatter matrix calculated manually
> sum((c1[, 1] - mean(c1[, 1])) ^ 2)
[1] 10
$\endgroup$
2
  • $\begingroup$ thanks. So which is correct actually? $\endgroup$
    – aan
    Commented Jun 25, 2020 at 5:42
  • $\begingroup$ They are completely equivalent. The difference is that on slide 9 you calculate the scatter from a biased (unadjusted) sample variance estimator and on slide 20 you calculate the scatter from an unibiased (adjusted) sample variance estimator. If you use some software (e.g. R, MATLAB) to calculate the sample variance, it gives you the unbiased sample variance and you should use the scatter formula from slide 20 (with n-1 in the denominator) $\endgroup$
    – eillasti
    Commented Jun 25, 2020 at 9:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.