I am trying to get a feel for how LDA 'fits' within other supervised learning techniques. I have already read some of the LDA-esque posts on here about LDA. I am already familiar with the perceptron, but just learning LDA now.

How does LDA 'fit' into the family of supervised learning algorithms? What might be its drawbacks vs those other methods, and what might it be used better for? Why use LDA, when one could just use, say, the perceptron for instance?

  • 1
    $\begingroup$ I think you may be confused about what supervised learning is. K-means is an unsupervised learning clustering algorithm. Perceptron is a supervised learning classification algorithm that attempts to find a hyperplane that separates negative from positive observations. LDA is a method that can be used for supervised classification but is more commonly used for supervised feature selection. See @AdamO's answer for assumptions of the LDA classifier. $\endgroup$ – Bitwise Jul 22 '13 at 18:05
  • $\begingroup$ @Bitwise Oops! I dont know why I put K-means in there. Yes, it is an unsupervised algorithm. I will remove it in an edit. $\endgroup$ – Creatron Jul 22 '13 at 18:07
  • $\begingroup$ @Bitwise Regarding what you said about LDA and Perceptron, yes, that is what is confusing me. LDA tries to find a hyperplane unto which to project your data, such that it maximizes intercluster variance, while minimizing the intra-cluster variance. Then on the border, you have a classifier. Perceptron does something similar, in that it also tries to find an optimal hyperplane to disjoint the labelled data. So why use one over the other? $\endgroup$ – Creatron Jul 22 '13 at 18:10

As AdamO suggests in the above comment, you can't really do better than read Chapter 4 of The Elements of Statistical Learning (which I will call HTF) which compares LDA with other linear classification methods, giving many examples, and also discusses the use of LDA as a dimension-reduction technique in the vein of PCA which, as ttnphns points out, is rather popular.

From the point of view of classification, I think the key difference is this. Imagine that you have two classes and you want to separate them. Each class has a probability density function. The best possible situation would be if you knew these density functions, because then you could predict which class a point would belong to by evaluating the class-specific densities at that point.

Some kinds of classifier operate by finding an approximation to the density functions of the classes. LDA is one of these; it makes the assumption that the densities are multivariate normal with the same covariance matrix. This is a strong assumption, but if it is approximately correct, you get a good classifier. Many other classifiers also take this kind of approach, but try to be more flexible than assuming normality. For example, see page 108 of HTF.

On the other hand, on page 210, HTF warn:

If classification is the ultimate goal, then learning the separate class densities well may be unnecessary, and can in fact be misleading.

Another approach is simply to look for a boundary between the two classes, which is what the perceptron does. A more sophisticated version of this is the support vector machine. These methods can also be combined with adding features to the data using a technique called kernelization. This does not work with LDA because it does not preserve normality, but it is no problem for a classifier which is just looking for a separating hyperplane.

The difference between LDA and a classifier which looks for a separating hyperplane is like the difference between a t-test and some nonparamteric alternative in ordinary statistics. The latter is more robust (to outliers, for example) but the former is optimal if its assumptions are satisfied.

One more remark: it might be worth mentioning that some people might have cultural reasons for using methods like LDA or logistic regression, which may obligingly spew out ANOVA tables, hypothesis tests, and reassuring things like that. LDA was invented by Fisher; the perceptron was originally a model for a human or animal neuron and had no connection with statistics. It also works the other way; some people might prefer methods like support vector machines because they have the kind of cutting-edge hipster-cred which twentieth-century methods just can't match. It doesn't mean that they're better. (A good example of this is discussed in Machine Learning for Hackers, if I recall correctly.)

  • $\begingroup$ " some people might prefer methods like support vector machines because they have the kind of cutting-edge hipster-cred which twentieth-century methods just can't match." LOL! So true. Btw you have a knack for explaining things very clearly and accurately. Thank you! I needed a 'map' on how things fit together and you provided it. $\endgroup$ – Creatron Jul 23 '13 at 15:33

For intuition, consider this case:

enter image description here

The line represents the "optimal boundary" between the two classes o and x.

LDA tries to find a hyperplane that minimizes the intercluster variance and maximize the intracluster variance, and then the takes the boundary to be orthogonal to that hyperplane. Here, this will probably not work because the clusters have large variance in the same direction.

A perceptron, on the other hand, may have a better chance of finding a good separating hyperplane.

In the case of classes that have a Gaussian distribution, though, the LDA will probably do better, since the perceptron only finds a separating hyperplane that is consistent with the data, without giving guarantees about which hyperplane it chooses (there could be an infinite number of consistent hyperplanes). However, more sophisticated versions of the perceptron can choose a hyperplane with some optimal properties, such as maximizing the margin between the classes (this is essentially what Support Vector Machines do).

Also note that both LDA and perceptron can be extended to non-linear decision boundaries via the kernel trick.


One of the biggest differences between LDA and the other methods is that it's just a machine learning technique for data which are assumed to be normally distributed. That can be great under the case of missing data or truncation where you can use the EM algorithm to maximize likelihoods under very strange and/or interesting circumstances. Caveat emptor because model misspecifications, such as multimodal data, can lead to poor performing predictions where K-means clustering would have done better. Multimodal data can also be accounted for with EM to detect latent variables or clustering in LDA.

For instance, suppose you are looking to measure probability of developing a positive diagnosis of AIDS in 5 years based on CD4 count. Suppose further that you don't know the value of a specific biomarker that greatly impacts CD4 counts and is associated with further immunosuppression. CD4 counts under 400 are below lower limit of detection on most affordable assays. The EM algorithm allows us to iteratively calculate the LDA and biomarker assignment and the means and covariance for CD4 for the untruncated DF.

  • $\begingroup$ Thanks Adam, although I find myself more confused now. :-) How is LDA better/worse that say, the Perceptron, or other supervised learning technique? Regarding the EM algo, you are using it in a scope of saying that you can solve for the LDA, using an EM algo, correct? $\endgroup$ – Creatron Jul 22 '13 at 18:13
  • 1
    $\begingroup$ @AdamO, I'd want to add for clarity that LDA as data reduction technique doesn't rely on normality, just like PCA doesn't. Normality within LDA is an assumption for 1) statistical testing (Box's M test etc.), 2) classification. $\endgroup$ – ttnphns Jul 22 '13 at 18:41
  • $\begingroup$ @ttnphns assuming normality means that LDA is an ML technique. ML is a good thing. The specific caveats in the example I mentioned utilize ML to solve difficult problems. Those solutions would only be possible with sophisticated simulation and/or BUGS. $\endgroup$ – AdamO Jul 22 '13 at 20:37
  • $\begingroup$ @TheGrapeBeyond LDA maximizes Mahal distance btn two groups. SLP (single layer perceptron, or nnet) draws the hyperplane in the feature space that creates the maximum classification accuracy... I think. A good starting place is reading the Tibs/Hastie book. I may need to brush up on that one myself. $\endgroup$ – AdamO Jul 22 '13 at 20:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.