6
$\begingroup$

I have some experience with both FLD and LR for classification.

On most data sets, I get very similar results, which raises the question - are FLD and LR related in some why?

An idea, for example, if I assume a normal distribution on the likelihood function instead a of a logistic function, does the maximization come down to the FLD criteria?

Please note that I'm asking about FLD and NOT LDA.

Also, I'm looking for a mathematical way to show that FLD and LR are equivalent under some constraints. I had some ideas but nothing worked out. I'm sure it holds some way.

Improve this question
$\endgroup$
2

2 Answers 2

Reset to default
2
$\begingroup$

Just (maybe redundant) elaborate on Frank's answer. LDA is based on assumptions of multivariate normality and equality of covariance matrices of the 2 groups (in population); it is also irritable to outliers and to unbalanced n's; the predictors should normally be interval scale. All that is not required by LR which is therefore more universally robust. Probability of classification is estimated in LDA indirectly by formulas such as Bayes' but directly in LR. Still, when all assumptions for LDA are nicely met LDA is somewhat superior to LR from a statistical perspective.

Improve this answer
$\endgroup$
8
  • 1
    $\begingroup$ Thanks for the answer but LDA and FLD are not the same. FLD does not assume assume multivariate normality. Again - I'm looking for the similarities between FLD and LR, mainly in the mathematical development. Thanks! $\endgroup$
    – Ran
    Commented Aug 23, 2011 at 16:03
  • 3
    $\begingroup$ @Ran: If I am not mistaken, FLD is the same as LDA under the assumption of normality. Under normality assumptions, there is a close connection between LDA and logistic regression where the latter can be viewed as a conditional version of the former. As ttnphns alludes to, Efron showed that the ARE of logistic regression with respect to LDA is about 70% under the normality assumption. When normality breaks down the LDA procedure is still the same as FLD, but logistic regression can dominate performance-wise. $\endgroup$
    – cardinal
    Commented Aug 23, 2011 at 16:55
  • 1
    $\begingroup$ Please take a careful look at the literature. For optimality, linear discriminant analysis does assume multivariate normality with a common covariance matrix across classes. To get accurate posterior probabilities of class membership from discriminant analysis you definitely need multivariate normality. $\endgroup$
    – Frank Harrell
    Commented Aug 23, 2011 at 16:57
  • 1
    $\begingroup$ @carindal: Thanks! Efron paper shows how FLD and LR are equivalent when each class is distributed according to the normal distribution. $\endgroup$
    – Ran
    Commented Aug 24, 2011 at 10:13
  • 2
    $\begingroup$ @Ran: LDA uses the full likelihood, while LR discards the marginal component, using only the conditional part. This is what causes it to lose some efficiency in the multivariate-normal case. $\endgroup$
    – cardinal
    Commented Aug 25, 2011 at 18:15
7
$\begingroup$

For many reasons, classification is not a good goal for most problems; prediction is. Logistic regression (LR) is a more direct probability model to use for prediction, with fewer assumptions. Linear discriminant analysis (LDA) assumes that X has a multivariate normal distribution given Y. Using Bayes' rule to get Prob(Y|X) you get a logistic model. So if assumptions of LDA hold, assumptions of LR automatically hold. The reverse is not true, hence LR is more robust (e.g., X's can be dichotomous, far from normal). It is interesting that logistic regression is, in a sense, more related to the normal distribution that is probit regression.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ FLD doesn't assume a normal distribution. My question is more about the relations and similarity between FLD and LR and not which one is better. Thanks. $\endgroup$
    – Ran
    Commented Aug 23, 2011 at 15:18
  • 1
    $\begingroup$ You persist in not reading the answers above. $\endgroup$
    – Frank Harrell
    Commented Jan 2, 2015 at 13:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.