# What is the difference between SVM and LDA?

What is the difference between Support Vector Machines and Linear Discriminant Analysis?

• Do you think all SVMs are linear ?
– user83346
Commented Nov 3, 2016 at 6:48
• Possible duplicate of Help me understand Support Vector Machines Commented Nov 3, 2016 at 7:27
• LDA tries to maximise the distance between the means of the two groups, while SVM tries to maximise the margin between the two groups.,
– user83346
Commented Nov 3, 2016 at 9:27

LDA: Assumes: data is Normally distributed. All groups are identically distributed, in case the groups have different covariance matrices, LDA becomes Quadratic Discriminant Analysis. LDA is the best discriminator available in case all assumptions are actually met. QDA, by the way, is a non-linear classifier.

SVM: Generalizes the Optimally Separating Hyperplane(OSH). OSH assumes that all groups are totally separable, SVM makes use of a 'slack variable' that allows a certain amount of overlap between the groups. SVM makes no assumptions about the data at all, meaning it is a very flexible method. The flexibility on the other hand often makes it more difficult to interpret the results from a SVM classifier, compared to LDA.

SVM classification is an optimization problem, LDA has an analytical solution. The optimization problem for the SVM has a dual and a primal formulation that allows the user to optimize over either the number of data points or the number of variables, depending on which method is the most computationally feasible. SVM can also make use of kernels to transform the SVM classifier from a linear classifier into a non-linear classifier. Use your favorite search engine to search for 'SVM kernel trick' to see how SVM makes use of kernels to transform the parameter space.

LDA makes use of the entire data set to estimate covariance matrices and thus is somewhat prone to outliers. SVM is optimized over a subset of the data, which is those data points that lie on the separating margin. The data points used for optimization are called support vectors, because they determine how the SVM discriminate between groups, and thus support the classification.

As far as I know, SVM doesn't really discriminate well between more than two classes. An outlier robust alternative is to use logistic classification. LDA handles several classes well, as long as the assumptions are met. I believe, though (warning: terribly unsubstantiated claim) that several old benchmarks found that LDA usually perform quite well under a lot of circumstances and LDA/QDA are often goto methods in the initial analysis.

LDA can be used for feature selection when $p>n$ with sparse LDA: https://web.stanford.edu/~hastie/Papers/sda_resubm_daniela-final.pdf. SVM cannot perform feature selection.

In short: LDA and SVM have very little in common. Luckily, they are both tremendously useful.

• Fisher's discriminant Analysis which is specific kind of LDA does not require normal distribution over the datasets or same cooveriance. Commented Jul 18, 2018 at 23:17
• A linear SVM gives a separating hyperplane, and projecting on the normal of that hyperplane is a feature reduction in the same sense that projecting on the lines found by LDA is.
– Erik
Commented Dec 6, 2022 at 9:47

# Short and sweet answer:

### The answers above are very thorough, so here is a quick description of how LDA and SVM work.

Support vector machines find a linear separator (linear combination, hyperplane) that separates the classes with the least error, and chooses the separator with the maximum margin (the width that the boundary could be increased before hitting a data point).

E.g. which linear separator best separates the classes?

The one with the maximum margin:

Linear discriminant analysis finds the mean vectors of each class, then finds projection direction (rotation) that maximizes separation of means:

It also takes into account within-class variance to find a projection which minimizes overlap of distributions (covariance) while maximizing the separation of means:

• Thanks a lot! In case of SVMs, is there a number for the required support vectors?
– Ben
Commented Jun 30, 2021 at 10:54
• @Ben I'm not sure if I understand your question, but the idea is to find a single linear separator and use that for classification. Commented Jul 1, 2021 at 10:02
• and the line separator aka hyperplane is constructed by using the so-called support vectors, or?
– Ben
Commented Jul 1, 2021 at 11:35
• I believe that's the correct terminology, yes: the support vectors are used to identify the linear separator with the maximum margin. Note that "linear separator" != "line" because it can be multidimensional. Commented Jul 2, 2021 at 11:44
• The explanation is slightly unclear as to how the within-class variance is incorporated with the mean separation objective. Commented Jan 5, 2022 at 11:50

SVM focuses only on the points that are difficult to classify, LDA focuses on all data points. Such difficult points are close to the decision boundary and are called Support Vectors. The decision boundary can be linear, but also e.g. an RBF kernel, or an polynomial kernel. Where LDA is a linear transformation to maximize separability.

LDA assumes that the data points have the same covariance and the probability density is assumed to be normally distributed. SVM has no such assumption.

LDA is generative, SVM is discriminative.