I am trying to decide between / reconcile four formulations for class scatter matrices.
The first from Duda et al. (2012), p.544 has (with symbols modified):
$$m_i = \frac{1}{n_i} \sum_{x\in \mathcal{D}_i}x$$
$$m = \frac{1}{n}\sum_{\mathcal{D}}x$$
$$S_W = \sum_{i=1}^C \sum_{x \in\mathcal{D}_i}(x -m_i)(x-m_i)^T$$
$$S_B = \sum_{i=1}^Cn_i(m_i - m)(m_i - m)^T$$
$$S_T = \sum_{x \in \mathcal{D}}(x-m)(x-m)^T$$
where $\mathcal{D}_i$ is the $i$th class, $m_i$ is the class mean, $m$ the overall mean, $C$ the number of classes, $n_i$ the number of items in class $\mathcal{D}_i$. $\mathcal{D}$ is the set of all classes.
The second formulation, from this course page, which is equivalent to the one in Webb (2002), p.375:
$$P_i = n_i / n$$
$$S_W = \frac{1}{n} \sum_{i=1}^C \sum_{x \in\mathcal{D}_i}(x -m_i)(x-m_i)^T$$
$$S_B = \sum_{i=1}^{C} P_i (m_i - m)(m_i - m)^T$$
$$S_T = \frac{1}{n} \sum_{x \in \mathcal{D}}(x-m)(x-m)^T$$
where $P_i$ is the a priori probability for class $\mathcal{D}_i$.
A third formulation is from Johnson and Wichern (2007), p.623:
$$\bar{m} = \frac{1}{C}\sum_{i=1}^C m_i$$
$$S_W = \sum_{i=1}^C \sum_{x \in\mathcal{D}_i}(x -m_i)(x-m_i)^T$$
$$S_B = \sum_{i=1}^{C} (m_i - \bar{m})(m_i - \bar{m})^T$$
$$S_T = S_B + S_W$$
where $\bar{m}$ is the mean of class means. $S_T$ is not given directly.
And finally, a fourth formulation, proposed in this post:
$$\bar{n} = \frac{\sum n_i}{C}$$
$$S_W = \bar{n} \sum_{i=1}^C \frac{1}{n_i} \sum_{x \in\mathcal{D}_i}(x -m_i)(x-m_i)^T$$
$$S_B = \bar{n} \sum_{i=1}^{C} (m_i - \bar{m})(m_i - \bar{m})^T$$
$$S_T = \bar{n} \sum_{i=1}^{C} \frac{1}{n_i} \sum_{x \in \mathcal{D}}(x-\bar{m})(x-\bar{m})^T$$
where $\bar{n}$ is the mean number of points per class, and $\bar{m}$ is the mean of class means.
As far as I can tell, the main difference is in the choice of covariance matrix. It looks to me like Duda is based on scatter matrices, Webb on the maximum likelihood estimate of the covariance matrix and Johnson on sample scatter matrices.
Can anyone detail the difference/effects? What is the use case for each? Has anyone seen formulation 4 in a more authoritative source?
n
at any place (except in mean computation) should take place. Formulas (1) are correct. Sw is called pooled within class scatter matrix. (Word "pooled" means in statistics either "summative" or "weightedly averaged".) $\endgroup$ – ttnphns Feb 22 '16 at 18:44