LDA vs. perceptron I am trying to get a feel for how LDA 'fits' within other supervised learning techniques. I have already read some of the LDA-esque posts on here about LDA. I am already familiar with the perceptron, but just learning LDA now. 
How does LDA 'fit' into the family of supervised learning algorithms? What might be its drawbacks vs those other methods, and what might it be used better for? Why use LDA, when one could just use, say, the perceptron for instance?
 A: For intuition, consider this case:

The line represents the "optimal boundary" between the two classes o and x.
LDA tries to find a hyperplane that minimizes the intercluster variance and maximize the intracluster variance, and then the takes the boundary to be orthogonal to that hyperplane. Here, this will probably not work because the clusters have large variance in the same direction.
A perceptron, on the other hand, may have a better chance of finding a good separating hyperplane.
In the case of classes that have a Gaussian distribution, though, the LDA will probably do better, since the perceptron only finds a separating hyperplane that is consistent with the data, without giving guarantees about which hyperplane it chooses (there could be an infinite number of consistent hyperplanes). However, more sophisticated versions of the perceptron can choose a hyperplane with some optimal properties, such as maximizing the margin between the classes (this is essentially what Support Vector Machines do).
Also note that both LDA and perceptron can be extended to non-linear decision boundaries via the kernel trick.
A: As AdamO suggests in the above comment, you can't really do better than read Chapter 4 of The Elements of Statistical Learning (which I will call HTF) which compares LDA with other linear classification methods, giving many examples, and also discusses the use of LDA as a dimension-reduction technique in the vein of PCA which, as ttnphns points out, is rather popular.
From the point of view of classification, I think the key difference is this. Imagine that you have two classes and you want to separate them. Each class has a probability density function. The best possible situation would be if you knew these density functions, because then you could predict which class a point would belong to by evaluating the class-specific densities at that point.
Some kinds of classifier operate by finding an approximation to the density functions of the classes. LDA is one of these; it makes the assumption that the densities are multivariate normal with the same covariance matrix. This is a strong assumption, but if it is approximately correct, you get a good classifier. Many other classifiers also take this kind of approach, but try to be more flexible than assuming normality. For example, see page 108 of HTF. 
On the other hand, on page 210, HTF warn:

If classification is the ultimate goal, then learning the separate
  class densities well may be unnecessary, and can in fact be
  misleading.

Another approach is simply to look for a boundary between the two classes, which is what the perceptron does. A more sophisticated version of this is the support vector machine. These methods can also be combined with adding features to the data using a technique called kernelization. This does not work with LDA because it does not preserve normality, but it is no problem for a classifier which is just looking for a separating hyperplane.
The difference between LDA and a classifier which looks for a separating hyperplane is like the difference between a t-test and some nonparamteric alternative in ordinary statistics. The latter is more robust (to outliers, for example) but the former is optimal if its assumptions are satisfied.
One more remark: it might be worth mentioning that some people might have cultural reasons for using methods like LDA or logistic regression, which may obligingly spew out ANOVA tables, hypothesis tests, and reassuring things like that. LDA was invented by Fisher; the perceptron was originally a model for a human or animal neuron and had no connection with statistics. It also works the other way; some people might prefer methods like support vector machines because they have the kind of cutting-edge hipster-cred which twentieth-century methods just can't match. It doesn't mean that they're better. (A good example of this is discussed in Machine Learning for Hackers, if I recall correctly.)
A: One of the biggest differences between LDA and the other methods is that it's just a machine learning technique for data which are assumed to be normally distributed. That can be great under the case of missing data or truncation where you can use the EM algorithm to maximize likelihoods under very strange and/or interesting circumstances. Caveat emptor because model misspecifications, such as multimodal data, can lead to poor performing predictions where K-means clustering would have done better. Multimodal data can also be accounted for with EM to detect latent variables or clustering in LDA.
For instance, suppose you are looking to measure probability of developing a positive diagnosis of AIDS in 5 years based on CD4 count. Suppose further that you don't know the value of a specific biomarker that greatly impacts CD4 counts and is associated with further immunosuppression. CD4 counts under 400 are below lower limit of detection on most affordable assays. The EM algorithm allows us to iteratively calculate the LDA and biomarker assignment and the means and covariance for CD4 for the untruncated DF.
