5
$\begingroup$

I have some experience with both FLD and LR for classification.

On most data sets, I get very similar results, which raises the question - are FLD and LR related in some why?

An idea, for example, if I assume a normal distribution on the likelihood function instead a of a logistic function, does the maximization come down to the FLD criteria?

Please note that I'm asking about FLD and NOT LDA.

Also, I'm looking for a mathematical way to show that FLD and LR are equivalent under some constraints. I had some ideas but nothing worked out. I'm sure it holds some way.

$\endgroup$
2
$\begingroup$

Just (maybe redundant) elaborate on Frank's answer. LDA is based on assumptions of multivariate normality and equality of covariance matrices of the 2 groups (in population); it is also irritable to outliers and to unbalanced n's; the predictors should normally be interval scale. All that is not required by LR which is therefore more universally robust. Probability of classification is estimated in LDA indirectly by formulas such as Bayes' but directly in LR. Still, when all assumptions for LDA are nicely met LDA is somewhat superior to LR from a statistical perspective.

$\endgroup$
  • $\begingroup$ Thanks for the answer but LDA and FLD are not the same. FLD does not assume assume multivariate normality. Again - I'm looking for the similarities between FLD and LR, mainly in the mathematical development. Thanks! $\endgroup$ – Ran Aug 23 '11 at 16:03
  • 3
    $\begingroup$ @Ran: If I am not mistaken, FLD is the same as LDA under the assumption of normality. Under normality assumptions, there is a close connection between LDA and logistic regression where the latter can be viewed as a conditional version of the former. As ttnphns alludes to, Efron showed that the ARE of logistic regression with respect to LDA is about 70% under the normality assumption. When normality breaks down the LDA procedure is still the same as FLD, but logistic regression can dominate performance-wise. $\endgroup$ – cardinal Aug 23 '11 at 16:55
  • 1
    $\begingroup$ Please take a careful look at the literature. For optimality, linear discriminant analysis does assume multivariate normality with a common covariance matrix across classes. To get accurate posterior probabilities of class membership from discriminant analysis you definitely need multivariate normality. $\endgroup$ – Frank Harrell Aug 23 '11 at 16:57
  • $\begingroup$ @carindal: Thanks! Efron paper shows how FLD and LR are equivalent when each class is distributed according to the normal distribution. $\endgroup$ – Ran Aug 24 '11 at 10:13
  • 2
    $\begingroup$ @Ran: LDA uses the full likelihood, while LR discards the marginal component, using only the conditional part. This is what causes it to lose some efficiency in the multivariate-normal case. $\endgroup$ – cardinal Aug 25 '11 at 18:15
6
$\begingroup$

For many reasons, classification is not a good goal for most problems; prediction is. Logistic regression (LR) is a more direct probability model to use for prediction, with fewer assumptions. Linear discriminant analysis (LDA) assumes that X has a multivariate normal distribution given Y. Using Bayes' rule to get Prob(Y|X) you get a logistic model. So if assumptions of LDA hold, assumptions of LR automatically hold. The reverse is not true, hence LR is more robust (e.g., X's can be dichotomous, far from normal). It is interesting that logistic regression is, in a sense, more related to the normal distribution that is probit regression.

$\endgroup$
  • $\begingroup$ FLD doesn't assume a normal distribution. My question is more about the relations and similarity between FLD and LR and not which one is better. Thanks. $\endgroup$ – Ran Aug 23 '11 at 15:18
  • 1
    $\begingroup$ You persist in not reading the answers above. $\endgroup$ – Frank Harrell Jan 2 '15 at 13:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.