Suppose I have a 3D data set (3 features) and $c=2$ classes. If we perform LDA, we will only get 1 LD since the maximum number of dimensions is $c-1$. I understand from the math derivation of why this is the case, but still can't visualize why we can't have a 2D plane where we can project the 3D data and have maximum separation of the classes.

Imagine the picture below with only 2 classes (2 spheres) instead of 3 classes. Can't we find a plane for LDA instead of a line in this specific case?

enter image description here


Roughly speaking, this is the mental picture that I get so far:

  • Imagine the above picture with only 2 spheres
  • Imagine a Linear Discriminant line in the 2D plane on the left
  • Then, all the 2D planes that I was looking for are those planes with the LD line as the axis of rotation. So, there are infinite number of those planes. As @ttnphns mentioned, the planes are thus unnecessary as the line can separate the 2 classes.
  • 1
    $\begingroup$ 1. Can we find a plane for LDA instead of a line in this specific case? Sure. You can even stay in 3D space :-). But the single line will suffice. If you have g balls (equal sized) in p dimensional space it is suffice to draw min(p,g-1) lines, axes, to distinguish linearly the balls as well as the whole pD space does it. $\endgroup$ – ttnphns Dec 20 '16 at 21:34
  • 1
    $\begingroup$ 2. I am not sure about plotting biplot for LDA By biplot we a mean scatterplot with both data cases and variables shown as points, while the "latent" dimensions (reducing the space) are the axes. It is possible to do it with LDA, though more often a scatterplot of cases + boundaries, so called territorial map, is done with LDA. Also, one can plot discriminants as lines in the original space. $\endgroup$ – ttnphns Dec 20 '16 at 21:45
  • $\begingroup$ @ttnphns: Regarding #1: Could you provide me with some info on how to create the 2D plot? $\endgroup$ – Lam Dec 20 '16 at 22:02
  • $\begingroup$ @ttnphns Regarding #2: Thanks for the info on territorial map and discriminants. Going back to the biplot, just to make sure that I understand it correctly, to get the variables shown as points, I can simply multiply the transformation matrix with the identity matrix of size 3 (for 3D data set)? $\endgroup$ – Lam Dec 20 '16 at 22:05
  • $\begingroup$ @ilanman: thanks for the edit :) i should have done that :) $\endgroup$ – Lam Dec 20 '16 at 22:06