Discriminant analysis vs logistic regression

I found some pros of discriminant analysis and I've got questions about them. So:

When the classes are well-separated, the parameter estimates for logistic regression are surprisingly unstable. Coefficients may go to infinity. LDA doesn't suffer from this problem.

If number of features is small and the distribution of the predictors $X$ is approximately normal in each of the classes, the linear discriminant model is again more stable than the logistic regression model.

1. What is stability and why is it important? (If logistic regression provides a good fit that does its job, then why should I care about stability?)

LDA is popular when we have more than two response classes, because it also provides low-dimensional views of the data.

1. I just don't understand that. How does LDA provide low-dimensional views?
2. If you can name more pros or cons, that would be nice.
• You might also want to read other Q/A on this topic (lda vs logistic). Please search this site. – ttnphns Dec 28 '15 at 13:34

When the classes are well-separated, the parameter estimates for logistic regression are surprisingly unstable. Coefficients may go to infinity. LDA doesn't suffer from this problem.

If there are covariate values that can predict the binary outcome perfectly then the algorithm of logistic regression, i.e. Fisher scoring, does not even converge. If you are using R or SAS you will get a warning that probabilities of zero and one were computed and that the algorithm has crashed. This is the extreme case of perfect separation but even if the data are only separated to a great degree and not perfectly, the maximum likelihood estimator might not exist and even if it does exist, the estimates are not reliable. The resulting fit is not good at all. There are many threads dealing with the problem of separation on this site so by all means take a look.

By contrast, one does not often encounter estimation problems with Fisher's discriminant. It can still happen if either the between or within covariance matrix is singular but that is a rather rare instance. In fact, If there is complete or quasi-complete separation then all the better because the discriminant is more likely to be successful.

It is also worth mentioning that contrary to popular belief LDA is not based on any distribution assumptions. We only implicitly require equality of the population covariance matrices since a pooled estimator is used for the within covariance matrix. Under the additional assumptions of normality, equal prior probabilities and misclassification costs, the LDA is optimal in the sense that it minimizes the misclassification probability.

How does LDA provide low-dimensional views?

It's easier to see that for the case of two populations and two variables. Here is a pictorial representation of how LDA works in that case. Remember that we are looking for linear combinations of the variables that maximize separability.

Hence the data are projected on the vector whose direction better achieves this separation. How we find that vector is an interesting problem of linear algebra, we basically maximize a Rayleigh quotient, but let's leave that aside for now. If the data are projected on that vector, the dimension is reduced from two to one.

The general case of more than two populations and variables is dealt similarly. If the dimension is large, then more linear combinations are used to reduce it, the data are projected on planes or hyperplanes in that case. There is a limit to how many linear combinations one can find of course and this limit results from the original dimension of the data. If we denote the number of predictor variables by $p$ and the number of populations by $g$, it turns out that the number is at most $\min(g-1,p)$.

If you can name more pros or cons, that would be nice.

The low-dimensional representantion does not come without drawbacks nevertheless, the most important one being of course the loss of information. This is less of a problem when the data are linearly separable but if they are not the loss of information might be substantial and the classifier will perform poorly.

There might also be cases where the equality of covariance matrices might not be a tenable assumption. You can employ a test to make sure but these tests are very sensitive to departures from normality so you need to make this additional assumption and also test for it. If it is found that the populations are normal with unequal covariance matrices a quadratic classification rule might be used (QDA) instead but I find that this is a rather awkward rule, not to mention counterintuitive in high dimensions.

Overall, the main advantage of the LDA is the existence of an explicit solution and its computational convenience which is not the case for more advanced classification techniques such as SVM or neural networks. The price we pay is the set of assumptions that go with it, namely linear separability and equality of covariance matrices.

Hope this helps.

EDIT: I suspect my claim that the LDA on the specific cases I mentioned does not require any distributional assumptions other than equality of the covariance matrices has cost me a downvote. This is no less true nevertheless so let me be more specific.

If we let $\bar{\mathbf{x}}_i, \ i = 1,2$ denote the means from the first and second population, and $\mathbf{S}_{\text{pooled}}$ denote the pooled covariance matrix, Fisher's discriminant solves the problem

$$\max_{\mathbf{a}} \frac{ \left( \mathbf{a}^{T} \bar{\mathbf{x}}_1 - \mathbf{a}^{T} \bar{\mathbf{x}}_2 \right)^2}{\mathbf{a}^{T} \mathbf{S}_{\text{pooled}} \mathbf{a} } = \max_{\mathbf{a}} \frac{ \left( \mathbf{a}^{T} \mathbf{d} \right)^2}{\mathbf{a}^{T} \mathbf{S}_{\text{pooled}} \mathbf{a} }$$

The solution of this problem (up to a constant) can be shown to be

$$\mathbf{a} = \mathbf{S}_{\text{pooled}}^{-1} \mathbf{d} = \mathbf{S}_{\text{pooled}}^{-1} \left( \bar{\mathbf{x}}_1 - \bar{\mathbf{x}}_2 \right)$$

This is equivalent to the LDA you derive under the assumption of normality, equal covariance matrices, misclassification costs and prior probabilities, right? Well yes, except now that we have not assumed normality.

There is nothing stopping you from using the discriminant above in all settings, even if the covariance matrices are not really equal. It might not be optimal in the sense of the expected cost of misclassification (ECM) but this is supervised learning so you can always evaluate its performance, using for instance the hold-out procedure.

References

Bishop, Christopher M. Neural networks for pattern recognition. Oxford university press, 1995.

Johnson, Richard Arnold, and Dean W. Wichern. Applied multivariate statistical analysis. Vol. 4. Englewood Cliffs, NJ: Prentice hall, 1992.

• (I am not the user who downvoted). To try to reconcile your answer with Frank Harell's, It seems to me that one still needs to assume that all the variables are continuous (otherwise, I think the maximum of the Rayleigh quotient would not be unique). – user603 Dec 28 '15 at 14:44
• @user603 I have not seen anywhere this condition. The solution is only determined up to a constant anyway. – JohnK Dec 28 '15 at 14:52
• John, Imagine that there are only 2 classes (and so, only one discriminant line) having identical, symmetrical (ellipsoidal) distributions, and equal prior probabilities. Then we in fact need not assume specifically normal distribution because we don't nee any pdf to assign a case to a class. In more complex settings (such as 3+ classes) we have to use some pdf, and it is usually normal. – ttnphns Dec 28 '15 at 14:55
• @ttnphns I see your point but this is not part of the assumptions you use to derive Fisher's discriminant even in the complex cases. In those settings you work with the eigenvalues/eigenvectors of the matrix $\mathbf{W}^{-1} \mathbf{B}$ where the $\mathbf{W}$ is the within covariance matrix and $\mathbf{B}$ the between. Then you can assign to classes using the customary normal distribution but you can also use something else if you want to. This does not invalidate the approach. – JohnK Dec 28 '15 at 15:03
• John, your last comment is what about you and I concur. – ttnphns Dec 28 '15 at 15:11

LDA makes severe distributional assumptions (multivariate normality of all predictors) unlike logistic regression. Try getting posterior probabilities of class membership on the basis of subjects' sex and you'll see what I mean - the probabilities will not be accurate.

The instability of logistic regression when a set of predictor values gives rise to a probability of 0 or 1 that $Y=1$ is more or less an illusion. Newton-Raphson iterations will converge to $\beta$s that are close enough to $\pm \infty$ (e.g., $\pm 30$) so that predicted probabilities are essentially 0 or 1 when they should be. The only problem this causes is the Hauck-Donner effect in the Wald statistics. The solution is simple: don't use Wald tests in this case; use likelihood ratio tests, which behave very well even with infinite estimates. For confidence intervals use profile likelihood confidence intervals if there is complete separation.

Note that if multivariable normality holds, by Bayes' theorem the assumptions of logistic regression hold. The reverse is not true.

Normality (or at the very least symmetry) must almost hold for variances and covariances to "do the job". Non-multivariate normally distributed predictors will even hurt the discriminant extraction phase.

• To my mind, normality is needed specificially at the classification (class prediction) stage of LDA. It is not necessary at the discriminants extraction (dimensionality reduction) stage, which, however, still assumes variance-covariance homogeneity. (Interestingly that latter assumption may be somewhat released at the classification: you may use separate within-class covariances for the discriminants there.) – ttnphns Dec 28 '15 at 13:33
• It is a bit misleading to say that only an equal covariance matrix assumption is needed for part of LDA to work. You are making a leap of faith in assuming that covariances are good data summaries. Think of the special case of LDA: the 2-sample $t$-test. The $t$-test assumes that the variance is a good summary of dispersion. It is the normal distribution assumption that makes the variance a sufficient statistic and a statistic that is not destroyed by outliers. LDA doesn't have convergence problems/instability precisely because it uses oversimplified sufficient statistics. – Frank Harrell Dec 28 '15 at 14:49
• I reserve 'data reduction' and 'dimensionality reduction' to refer to unsupervised learning. Yes I'm implying that sums of squares in and of themselves may require a normal distribution to be good choices. Think again of the 2-sample $t$-test (a special case of all this) and how horrible it performs if the standard deviation were a bad choice for a dispersion measure. The SD requires symmetry plus low probability of outlying observations to perform well. – Frank Harrell Dec 28 '15 at 16:51
• Yes SD makes various assumptions and is non-robust. To a less degree the mean makes some assumptions to be meaningful. Least squares, PCA, and LDA effectively make more distributional assumptions than many people think. – Frank Harrell Dec 28 '15 at 17:35
• I am not convinced by this reasoning and I still believe the downvote was unfair but I am no authority in the matter. The references I provided will tell you the same however. – JohnK Dec 28 '15 at 22:29

When the classes are well-separated, the parameter estimates for logistic regression are surprisingly unstable. Coefficients may go to infinity. LDA doesn't suffer from this problem.

Disclaimer: What follows here lacks mathematical rigour completely.

In order to fit a (nonlinear) function well you need observations in all regions of the function where "its shape changes". Logistic regression fits a sigmoid function to the data:

In the case of well-separated classes all observations will fall onto the two "ends" where the sigmoid approaches its asymptotes (0 and 1). Since all sigmoids "look the same" in these regions, so to speak, no wonder the poor fitting algorithm will have difficulties to find "the right one".

Let's have a look at two (hopefully instructive) examples calculated with R's glm() function.

Case 1: The two groups overlap to quite some extent:

and the observations distribute nicely around the inflexion point of the fitted sigmoid:

These are the fitted the parameters with nice low standard errors:

Coefficients:
Estimate Std. Error z value Pr(>|z|)
(Intercept) -17.21374    4.07741  -4.222 2.42e-05 ***
wgt           0.35111    0.08419   4.171 3.04e-05 ***


and the deviance also looks OK:

    Null deviance: 138.629  on 99  degrees of freedom
Residual deviance:  30.213  on 98  degrees of freedom


Case 2: The two groups are well separated:

and the observations all lie on the asymptotes practically. The glm() function tried its best to fit something, but complained about numerically 0 or 1 probabilities, because there are simply no observations available to "get the shape of the sigmoid right" around its inflexion point:

You can diagnose the problem by noting that the standard errors of the estimated parameters go through the roof:

Coefficients:
Estimate Std. Error z value Pr(>|z|)
(Intercept)   -232.638 421264.847  -0.001        1
wgt              5.065   9167.439   0.001        1


and at the same time the deviance looks suspiciously good (because the observations do fit the asymptotes well):

    Null deviance: 1.3863e+02  on 99  degrees of freedom
Residual deviance: 4.2497e-10  on 98  degrees of freedom


At least intuitively it should be clear from these considerations why "the parameter estimates for logistic regression are surprisingly unstable".

• Look at the answer by @Frank Harrell which clearly disagrees with you! And study its links and references ... – kjetil b halvorsen Jul 25 '19 at 13:43
• @kjetilbhalvorsen My main point is an intuitive illustration of the "surprisingly unstable" fit. I removed the last sentence referring to the LDA. – Laryx Decidua Jul 25 '19 at 14:08