How do I test whether I can properly apply LDA?

Question

I have some data which works nicely with JMP's canned linear discriminant analysis (LDA), but after reading about LDA I'm not sure if the analysis is valid. The Wiki article notes a fundamental assumption of LDA is that independent variables are normally distributed. Does this also mean that I cannot use LDA if my variables have any correlation whatsoever? Also, supposing I have or transform to independent variables-- then I must also test if they are normally distributed (say, Shapiro-Wilk test) and only use those that pass in the LDA?

Thanks

Answer 1

The normality assumption is only a criteria for optimality in the sense that if the class conditional densities for the feature vector are normally distributed with known mean vectors and known equal covariance matrices the linear discriminant function is the Bayes rule (optimal) when the two types of errors have equal loss weights. The actual Fisher linear discriminant function is an approximation to the idea Bayes rule when the above mentioned assumptions hold but the parameters of the normal distribution are unknown and estimated from the training data set. So even when the normal assumption "holds" LDA only approximates the Bayes rule (closely if the training set is large). Now the rule is valid under other assumptions. In many case it may have good performance charateristics even though the noramlity assumption fails. But what it does do is open the door that some other algorithm such as QDA, kernel discrimination or other method could have better performance.

Answer 2

I would also add that LDA is usually preceded by a variable selection step, which basically removes variables that are too correlated (using criterias such as Wilks' lambda/ Fisher's F, Rao's V, etc.). LDA itself is then performed in the selected variable subset.

It is also common to perform PCA to compute a higher-level set of variables that you can give to LDA. Before PCA, you usually center-reduce your variables so that the mean is 0 and variance is 1 so they become normalized. PCA then removes the noise added by the correlated variables.