Suppose I have a 3D data set (3 features) and $c=2$ classes. If we perform LDA, we will only get 1 LD since the maximum number of dimensions is $c-1$. I understand from the math derivation of why this is the case, but still can't visualize why we can't have a 2D plane where we can project the 3D data and have maximum separation of the classes.
Imagine the picture below with only 2 classes (2 spheres) instead of 3 classes. Can't we find a plane for LDA instead of a line in this specific case?
Roughly speaking, this is the mental picture that I get so far:
- Imagine the above picture with only 2 spheres
- Imagine a Linear Discriminant line in the 2D plane on the left
- Then, all the 2D planes that I was looking for are those planes with the LD line as the axis of rotation. So, there are infinite number of those planes. As @ttnphns mentioned, the planes are thus unnecessary as the line can separate the 2 classes.