16
$\begingroup$

I was fiddling with PCA and LDA methods and I am stuck at a point, I have a feeling that it is so simple that I can't see it.

Within-class ($S_W$) and between-class ($S_B$) scatter matrices are defined as:

$$ S_W = \sum_{i=1}^C\sum_{t=1}^N(x_t^i - \mu_i)(x_t^i - \mu_i)^T $$

$$ S_B = \sum_{i=1}^CN(\mu_i-\mu)(\mu_i-\mu)^T $$

Total scatter matrix $S_T$ is given as:

$$ S_T = \sum_{i=1}^C\sum_{t=1}^N(x_t^i - \mu)(x_t^i - \mu)^T = S_W + S_B $$

where C is number of classes and N is number of samples $x$ are samples, $\mu_i$ is ith class mean, $\mu$ is overall mean.

While trying to derive $S_T$ I came up to a point where I had:

$$ (x-\mu_i)(\mu_i-\mu)^T + (\mu_i-\mu)(x-\mu_i)^T $$

as a term. This needs to be zero, but why?


Indeed:

\begin{align} S_T &= \sum_{i=1}^C\sum_{t=1}^N(x_t^i - \mu)(x_t^i - \mu)^T \\ &= \sum_{i=1}^C\sum_{t=1}^N(x_t^i - \mu_i + \mu_i - \mu)(x_t^i - \mu_i + \mu_i - \mu)^T \\ &= S_W + S_B + \sum_{i=1}^C\sum_{t=1}^N\big[(x_t^i - \mu_i)(\mu_i - \mu)^T + (\mu_i - \mu)(x_t^i - \mu_i)^T\big] \end{align}

$\endgroup$
2
  • 2
    $\begingroup$ The answer is that you are summing the deviations of values around their mean and that sum is zero. But what, precisely, are $x$, $m$, and $m_i$? How are $m$ and $m_i$ related to $\mu$ and $\mu_i$? The quality of answers will depend on how accurately we guess but you're forcing us to do an awful lot of guessing! $\endgroup$
    – whuber
    Mar 22 '11 at 17:35
  • $\begingroup$ @whuber: You are totally right, I revised my question. $\endgroup$
    – nimcap
    Mar 23 '11 at 8:46
9
$\begingroup$

If you assume

$$\frac{1}{N}\sum_{t=1}^Nx_t^{i}=\mu_i$$

Then

$$\sum_{i=1}^C\sum_{t=1}^N(x_t^i-\mu_i)(\mu_i-\mu)^T=\sum_{i=1}^C\left(\sum_{t=1}^N(x_t^i-\mu_i)\right)(\mu_i-\mu)^T=0$$

and formula holds. You deal with the second term in the similar way.

$\endgroup$
4
  • 3
    $\begingroup$ (+1) The second term, being the transpose of the first, must also be zero :-). $\endgroup$
    – whuber
    Mar 23 '11 at 15:57
  • $\begingroup$ @whuber, yes, that too :) $\endgroup$
    – mpiktas
    Mar 23 '11 at 17:16
  • $\begingroup$ Hi,i don't get why the assumption holds?Can someone explain that? $\endgroup$ Jan 7 '19 at 9:48
  • 1
    $\begingroup$ @Mvkt It is not so much an assumption as the definition of $\mu_i$ I suppose. That is to say: $\mu_i$ is the mean of the observations in group $i$. I expect the answer uses 'assume' because the OP doesn't explain the notation, so we have to guess that the group mean is meant by $\mu_i$. $\endgroup$
    – Vincent
    Jan 21 '19 at 12:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.