2
$\begingroup$

Hang Nguyen writes the following in "Machine learning basics (part 14): Linear Discriminant Analysis":

If there are two classes then the LDA draws one hyperplane and projects the data onto this hyperplane in such a way as to maximize the separation of the two categories. This hyperplane is created according to the two criteria considered simultaneously:

  • Maximizing the distance between the means of two classes;
  • Minimizing the variation between each category.

Is that correct?

Improve this question
$\endgroup$
4
  • $\begingroup$ It has been a while since I learned LDA. What I have kept in my memory is that LDA can be viewed as a special case of the Bayes classifier. There, the focus is on priors, likelihoods and posteriors instead of optimizing means and variances. I suppose there can be alternative representations of the same optimization problem, so I wonder if this one is correct or not. $\endgroup$
    – Richard Hardy
    Commented Mar 2, 2023 at 9:24
  • 1
    $\begingroup$ In Fisher's LDA the ratio of the between-group sum of squares to the within-group sum of squares is maximized. But there are different discriminant rules (ML, Bayes, ...). $\endgroup$
    – statmerkur
    Commented Mar 2, 2023 at 10:01
  • $\begingroup$ @statmerkur, so I take it to mean the quoted interpretation is incorrect? $\endgroup$
    – Richard Hardy
    Commented Mar 2, 2023 at 10:15
  • 1
    $\begingroup$ Yes. For 2 groups Fisher's method amounts to maximizing the ratio of the squared Euclidean distance between the group means to the sum of the two within-group "variations" (of the projected data). $\endgroup$
    – statmerkur
    Commented Mar 2, 2023 at 14:30

2 Answers 2

Reset to default
3
$\begingroup$

Fisher's approach was ingenious and intuitively clear: separate the populations $\Pi_i$ based on the linear function $\mathbf a^\top \mathbf x$ that maximizes the ratio of between-groups of sum of squares to within-group sum of squares:

Consider $\mathbf y=\mathbf X\mathbf a, ~\mathbf X$ being the data matrix. Total sum of squares $\mathbf y^\top \mathbf H\mathbf y=\mathbf a^\top \mathbf X^\top\mathbf H\mathbf X\mathbf a$ can be decomposed into sum of within-groups sum of squares $\sum\mathbf y_i^\top\mathbf H_i\mathbf y_i=\sum \mathbf a^\top \mathbf X_i^\top\mathbf H_i\mathbf X_i\mathbf a:= \mathbf a^\top \mathbf W\mathbf a, $ (here $\mathbf H_i$ being the centering matrices) and between-groups sum of squares $\sum_i n_i(\bar y_i-\bar y) :=\mathbf a^\top\mathbf B\mathbf a. $

We seek to find $\mathbf a$ that maximizes the ratio (as remarked by statmerkur) $$ \frac{\mathbf a^\top\mathbf B\mathbf a}{\mathbf a^\top\mathbf W\mathbf a}$$ which would be nothing but the largest eigenvalue's eigenvector of $\mathbf W^{-1}\mathbf B. $

Coming to the stated blog: while I won't say it is outright wrong, it is indeed problematic:

$\bullet$ The author defined $\mathbf S_b$ in terms of $\boldsymbol\mu_i,$ the population means. But then used sample within groups matrix $\sum_i (n_i-1) \mathbf S_i=\sum_i\sum_{j=1}^{n_i}(\mathbf x_{ij} -\bar{\mathbf x}_i) (\mathbf x_{ij} -\bar{\mathbf x}_i)^\top.$ The former is used when the population parameters $(\boldsymbol\mu_i, \mathbf\Sigma) $ are known. Otherwise, in place of $\mathbf S_b, ~\sum_i(\mathbf{\bar x}_i-\mathbf{\bar x}) (\mathbf{\bar x}_i-\mathbf{\bar x}) ^\top$ has to be substituted. The author is plausibly confusing things using Fisher's sample LDs.

$\bullet$

Maximizing the distance between the means of two classes;

Now they are talking about two classes instead of $g$ classes. For $g=2, $ the interpretation would be to find $\mathbf a$ such that maximum separation is achieved between $\bar y_1$ and $\bar y_2, $ measured in standard deviation units.

References:

$\rm [I]$ Applied Multivariate Statistical Analysis, Wolfgang Karl Härdle, Léopold Simar, Springer-Verlag, $2015, $ sec. $14.2, $ pp. $418-419.$

$\rm [II]$ Applied Multivariate Statistical Analysis, Richard A. Johnson, Dean A. Wichern, Pearson, $2013, $ sec. $11.6, $ pp. $622-623; ~590.$

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thank you for the answer! If the LDA does not involve maximization of the distance between the means of the two classes, then the statement is simply incorrect, is it not? $\endgroup$
    – Richard Hardy
    Commented Mar 2, 2023 at 14:44
  • $\begingroup$ I would say we are seeking maximization of the separation of the two means measured in standard deviation units; in other words maximize the ratio. So, yes, the statement provided is misleading. I think the author wanted to argue like since we are seeking maximization of $a/b, $ it can be achieved by maximizing $a$ and minimizing $b.$ But this is too oversimplification and can be misleading. $\endgroup$
    – User1865345
    Commented Mar 2, 2023 at 14:56
  • $\begingroup$ Overall the whole blog seemed to, though in good spirit, be outright confusing and dubious. $\endgroup$
    – User1865345
    Commented Mar 2, 2023 at 15:01
  • 1
    $\begingroup$ Thanks! That answers my question. $\endgroup$
    – Richard Hardy
    Commented Mar 2, 2023 at 16:08
2
$\begingroup$

In "The use of multiple measurements in taxonomic problems" Fisher asked the question

What linear function of the four measurements $$X=\lambda_1x_1 + \lambda_2x_2 + \lambda_3x_3 + \lambda_4x_4 $$ will maximize the ratio of the difference between the specific means to the standard deviations within species?

So you can see LDA as finding the linear combination of the measured variables that maximizes the F-ratio in an ANOVA test.

example iris

If there are two classes then the LDA draws one hyperplane and projects the data onto this hyperplane in such a way as to maximize the separation of the two categories. This hyperplane is created according to the two criteria considered simultaneously:

  • Maximizing the distance between the means of two classes;
  • Minimizing the variation between each category.

The quote is more or less correct.

It is more precisely about maximizing the ratio of the 'distance between the means of two classes' to 'the variation within each category'.

The projection is onto a line, not onto a hyperplane. Although the line defines a range of hyperplanes that can be used in categorisation.

LDA can be viewed as a special case of the Bayes classifier.

If you estimate the distributions as multivariate normal distributions with the same covariance, then the Bayes classifier is a hyperplane perpendicular to the LDA-axis.

An example of the hyperplane is below (which also shows the hyperplane for qda, which relaxes the assumption of equal covariance matrices)

example

Improve this answer
$\endgroup$
4
  • $\begingroup$ Amazing plots, as always! Regarding the last quote, does your answer support or contradict it? $\endgroup$
    – Richard Hardy
    Commented Mar 16, 2023 at 10:41
  • $\begingroup$ The image is a copy from stats.stackexchange.com/a/501450 it actually doesn't show the point in the best way as it draws the distributions assuming that the covariance is different. But, if you would have the same covariance, then the line of LDA will flow through the intersections of the iso-lines, where the distributions are equal (currently the QDA line does this). $\endgroup$
    – Sextus Empiricus
    Commented Mar 16, 2023 at 10:52
  • 1
    $\begingroup$ You could say in a way that LDA works as a Bayes classifier. It can draw the same boundary. (You would have to apply the correct weights between the prior groups, and the Bayes needs to assume equal covariance) $\endgroup$
    – Sextus Empiricus
    Commented Mar 16, 2023 at 10:55
  • 1
    $\begingroup$ It is more precisely about maximizing the ratio of the 'distance between the means of two classes' to 'the variation within each category. - Exactly, this is the correct statement. As I said earlier too, the blog is not wrong but imo, it is a bit confusing and then there was bit of hotchpotch in terminologies. $\endgroup$
    – User1865345
    Commented Mar 16, 2023 at 11:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.