46

Summary: PCA can be performed before LDA to regularize the problem and avoid over-fitting. Recall that LDA projections are computed via eigendecomposition of $\boldsymbol \Sigma_W^{-1} \boldsymbol \Sigma_B$, where $\boldsymbol \Sigma_W$ and $\boldsymbol \Sigma_B$ are within- and between-class covariance matrices. If there are less than $N$ data points (...


36

It sounds to me that you are correct. Logistic regression indeed does not assume any specific shapes of densities in the space of predictor variables, but LDA does. Here are some differences between the two analyses, briefly. Binary Logistic regression (BLR) vs Linear Discriminant analysis (with 2 groups: also known as Fisher's LDA): BLR: Based on Maximum ...


29

Multicollinearity means that your predictors are correlated. Why is this bad? Because LDA, like regression techniques involves computing a matrix inversion, which is inaccurate if the determinant is close to 0 (i.e. two or more variables are almost a linear combination of each other). More importantly, it makes the estimated coefficients impossible to ...


27

Tijl De Bie wrote an interesting chapter "Eigenproblems in Pattern Recognition" which talks about exactly these from a primal/dual perspective. The three tables at the end summarise really nicely from an optimisation perspective:


27

The most standard linear method of supervised dimensionality reduction is called linear discriminant analysis (LDA). It is designed to find low-dimensional projection that maximizes class separation. You can find a lot of information about it under our discriminant-analysis tag, and in any machine learning textbook such as e.g. freely available The Elements ...


24

Here is a short tale about Linear Discriminant Analysis (LDA) as a reply to the question. When we have one variable and $k$ groups (classes) to discriminate by it, this is ANOVA. The discrimination power of the variable is $SS_\text{between groups} / SS_\text{within groups}$, or $B/W$. When we have $p$ variables, this is MANOVA. If the variables are ...


24

As I've noted in the comment to your question, discriminant analysis is a composite procedure with two distinct stages - dimensionality reduction (supervised) and classification stage. At dimensionality reduction we extract discriminant functions which replace the original explanatory variables. Then we classify (typically by Bayes' approach) observations to ...


23

"Fisher's Discriminant Analysis" is simply LDA in a situation of 2 classes. When there is only 2 classes computations by hand are feasible and the analysis is directly related to Multiple Regression. LDA is the direct extension of Fisher's idea on situation of any number of classes and uses matrix algebra devices (such as eigendecomposition) to compute it. ...


23

This particular figure in Hastie et al. was produced without computing equations of class boundaries. Instead, algorithm outlined by @ttnphns in the comments was used, see footnote 2 in section 4.3, page 110: For this figure and many similar figures in the book we compute the decision boundaries by an exhaustive contouring method. We compute the decision ...


21

In a nutshell Both one-way MANOVA and LDA start with decomposing the total scatter matrix $\mathbf T$ into the within-class scatter matrix $\mathbf W$ and between-class scatter matrix $\mathbf B$, such that $\mathbf T = \mathbf W + \mathbf B$. Note that this is fully analogous to how one-way ANOVA decomposes total sum-of-squares $T$ into within-class and ...


21

Let me add some points to @ttnphns nice list: The Bayes prediction of the LDA's posterior class membership probability follows a logistic curve as well. [Efron, B. The efficiency of logistic regression compared to normal discriminant analysis, J Am Stat Assoc, 70, 892-898 (1975).] While that paper shows that the relative efficiency of LDA is superior to LR ...


20

I take it that the question is about LDA and linear (not logistic) regression. There is a considerable and meaningful relation between linear regression and linear discriminant analysis. In case the dependent variable (DV) consists just of 2 groups the two analyses are actually identical. Despite that computations are different and the results - regression ...


20

Update: Thanks to this discussion, scikit-learn was updated and works correctly now. Its LDA source code can be found here. The original issue was due to a minor bug (see this github discussion) and my answer was actually not pointing at it correctly (apologies for any confusion caused). As all of that does not matter anymore (bug is fixed), I edited my ...


16

Classification in LDA goes as follows (Bayes' rule approach). [About extraction of discriminants one might look here.] According to Bayes theorem, the sought-for probability that we're dealing with class $k$ while observing currently point $x$ is $P(k|x) = P(k)*P(x|k) / P(x)$, where $P(k)$ – unconditional (background) probability of class $k$; $P(x)$ – ...


16

Discriminants are the axes and the latent variables which differentiate the classes most strongly. Number of possible discriminants is $min(k-1,p)$. For example, with k=3 classes in p=2 dimensional space there can exist at most 2 discriminants such as on the graph below. (Note that discriminants are not necessarily orthogonal as axes drawn in the original ...


15

While "The Elements of Statistical Learning" is a brilliant book, it requires a relatively high level of knowledge to get the most from it. There are many other resources on the web to help you to understand the topics in the book. Lets take a very simple example of linear discriminant analysis where you want to group a set of two dimensional data points ...


15

As AdamO suggests in the above comment, you can't really do better than read Chapter 4 of The Elements of Statistical Learning (which I will call HTF) which compares LDA with other linear classification methods, giving many examples, and also discusses the use of LDA as a dimension-reduction technique in the vein of PCA which, as ttnphns points out, is ...


14

Discriminant analysis assumes a multivariate normal distribution because what we usually consider to be predictors are really a multivariate dependent variable, and the grouping variable is considered to be a predictor. This means that categorical variables that are to be treated as predictors in the sense you wish are not handled well. This is one reason ...


13

The credit for this answer goes to @ttnphns who explained everything in the comments above. Still, I would like to provide an extended answer. To your question: Are the LDA results on standardized and non-standardized features going to be exactly the same? --- the answer is Yes. I will first give an informal argument, and then proceed with some math. ...


13

When the classes are well-separated, the parameter estimates for logistic regression are surprisingly unstable. Coefficients may go to infinity. LDA doesn't suffer from this problem. If there are covariate values that can predict the binary outcome perfectly then the algorithm of logistic regression, i.e. Fisher scoring, does not even converge. If you are ...


12

I think that Multi-class LDA classifier always (well, in most practical tasks) out-performs 2 class LDA. And I will try to describe why. Have a look at the example dataset: You have three classes here. And let's say you want to build one-vs-other classifier with LDA for the blue class. The estimated mean for class "blue" is zero, but the estimated mean ...


12

As I seem to think gui11aume has given you a great answer, I want to give an example from a slightly different angle that might be illuminating. Consider that a covariate in your discriminant function looks as follows: $X_1= 5X_2 +3X_3 -X_4$. Suppose the best LDA has the following linear boundary: $X_1+2X_2+X_3-2X_4 =5$ Then we can substitute $...


11

Here is a reference to one of Efron's papers: The Efficiency of Logistic Regression Compared to Normal Discriminant Analysis, 1975. Another relevant paper is Ng & Jordan, 2001, On Discriminative vs. Generative classifierers: A comparison of logistic regression and naive Bayes. And here is an abstract of a comment on it by Xue & Titterington, 2008, ...


11

GDA, is a method for data classification commonly used when data can be approximated with a Normal distribution. As first step, you will need a training set, i.e. a bunch of data yet classified. These data are used to train your classifier, and obtain a discriminant function that will tell you to which class a data has higher probability to belong. When ...


11

Here is what Hastie et al. have to say about it (in context of two-class LDA) in The Elements of Statistical Learning, section 4.3: Since this derivation of the LDA direction via least squares does not use a Gaussian assumption for the features, its applicability extends beyond the realm of Gaussian data. However the derivation of the particular ...


11

Following the paper you linked to (Mika et al., 1999), we have to find the $\mathbf{w}$ which maximizes the so called generalized Rayleigh quotient, $$\frac{\mathbf{w}^\top \mathbf{S}_B \mathbf{w}}{\mathbf{w}^\top \mathbf{S}_W \mathbf{w}},$$ where for means $\mathbf{m}_1, \mathbf{m}_2$ and covariances $\mathbf{C}_1, \mathbf{C}_2$, \begin{align} \mathbf{S}...


11

You are missing something deeper: PCA isn't a classification method. PCA in machine learning is treated as a feature engineering method. When you apply PCA to your data you are guaranteeing there'll be no correlation between the resulting features. Many classification algorithms benefit from that. You always have to keep in mind algorithms might have ...


10

I will first provide a verbal explanation, and then a more technical one. My answer consists of four observations: As @ttnphns explained in the comments above, in PCA each principal component has certain variance, that all together add up to 100% of the total variance. For each principal component, a ratio of its variance to the total variance is called the ...


10

I will provide only a short informal answer and refer you to the section 4.3 of The Elements of Statistical Learning for the details. Update: "The Elements" happen to cover in great detail exactly the questions you are asking here, including what you wrote in your update. The relevant section is 4.3, and in particular 4.3.2-4.3.3. (2) Do and how the two ...


10

LDA makes severe distributional assumptions (multivariate normality of all predictors) unlike logistic regression. Try getting posterior probabilities of class membership on the basis of subjects' sex and you'll see what I mean - the probabilities will not be accurate. The instability of logistic regression when a set of predictor values gives rise to a ...


Only top voted, non community-wiki answers of a minimum length are eligible