Logistic regression vs. LDA as two-class classifiers

Question

I am trying to wrap my head around the statistical difference between Linear discriminant analysis and Logistic regression. Is my understanding right that, for a two class classification problem, LDA predicts two normal density functions (one for each class) that creates a linear boundary where they intersect, whereas logistic regression only predicts the log-odd function between the two classes, which creates a boundary but does not assume density functions for each class?

Answer 1

It sounds to me that you are correct. Logistic regression indeed does not assume any specific shapes of densities in the space of predictor variables, but LDA does. Here are some differences between the two analyses, briefly.

Binary Logistic regression (BLR) vs Linear Discriminant analysis (with 2 groups: also known as Fisher's LDA):

• BLR: Based on Maximum likelihood estimation.
  LDA: Based on Least squares estimation; equivalent to linear regression with binary predictand (coefficients are proportional and R-square = 1-Wilk's lambda).

• BLR: Estimates probability (of group membership) immediately (the predictand is itself taken as probability, observed one) and conditionally.
  LDA: estimates probability mediately (the predictand is viewed as binned continuous variable, the discriminant) via classificatory device (such as naive Bayes) which uses both conditional and marginal information.

• BLR: Not so exigent to the level of the scale and the form of the distribution in predictors.
  LDA: Predictirs desirably interval level with multivariate normal distribution.

• BLR: No requirements about the within-group covariance matrices of the predictors.
  LDA: The within-group covariance matrices should be identical in population.

• BLR: The groups may have quite different $n$.
  LDA: The groups should have similar $n$.

• BLR: Not so sensitive to outliers.
  LDA: Quite sensitive to outliers.

• BLR: Younger method.
  LDA: Older method.

• BLR: Usually preferred, because less exigent / more robust.
  LDA: With all its requirements met, often classifies better than BLR (asymptotic relative efficiency 3/2 time higher then).

Answer 2

Let me add some points to @ttnphns nice list:

• The Bayes prediction of the LDA's posterior class membership probability follows a logistic curve as well.
  [Efron, B. The efficiency of logistic regression compared to normal discriminant analysis, J Am Stat Assoc, 70, 892-898 (1975).]

• While that paper shows that the relative efficiency of LDA is superior to LR if the LDA's assumtions are met (Ref: Efron paper above, @tthnps' last point), according to the Elements of Statistical Learning in practice there is hardly any difference.
  [Hastie, T. and Tibshirani, R. and Friedman, J. The Elements of Statistical Learning; Data mining, Inference andPrediction Springer Verlag, New York, 2009]

• That hugely increased relative efficiency of LDA mostly happens in asymptotic cases where the absolute error is practically negligible anyways.
  [ Harrell, F. E. & Lee, K. L. A comparison of the discrimination of discriminant analysis and logistic regression under multivariate normality, Biostatistics: Statistics in Biomedical, Public Health and Environmental Sciences, 333-343 (1985).]

• Though I have in practice encountered high dimensional small sample size situations where the LDA seems superior (despite both the multivariate normality and the equal covariance matrix assumptions being visibly not met).
  [Beleites, C.; Geiger, K.; Kirsch, M.; Sobottka, S. B.; Schackert, G. & Salzer, R. Raman spectroscopic grading of astrocytoma tissues: using soft reference information., Anal Bioanal Chem, 400, 2801-2816 (2011). DOI: 10.1007/s00216-011-4985-4]

• But note that in our paper the LR is possibly struggling with the problem that directions with (near) perfect separability can be found. The LDA on the other hand may be less severely overfitting.

• The famous assumptions for LDA are only needed to prove optimality. If they are not met, the procedure can still be a good heuristic.

• A difference that is important for me in practice because the classification problems I work on sometimes/frequently turn out actually not to be that clearly classification problems at all: LR can easily be done with data where the reference has intermediate levels of class membership. After all, it is a regression technique.
  [see paper linked above]

• You may say that LR concentrates more than LDA on examples near the class boundary and basically disregards cases at the "backside" of the distributions.

• This also explains why it is less sensitive to outliers (i.e. those at the back side) than LDA.

• (support vector machines would be a classifier that goes this direction to the very end: here everything but the cases at the boundary is disregarded)

Answer 3

I just wanted to add one more point.

• LDA works when all the independent/predictor variables are continuous (not categorical) and follow a Normal distribution.
• Whereas in Logistic Regression this is not the case and categorical variables can be used as independent variables while making predictions.