3
$\begingroup$

I went to a short talk and the speaker quickly mentioned something like 'LDA (linear discriminant analysis) is more likely to be overfitted than SVM (support vector machine)'. Is this true? And why?

$\endgroup$
7
$\begingroup$

I do not believe it is the inverse either. Both approaches are very different.

LDA is optimal (in a Bayes sense) whenever the assumptions under which it is derived are met, namely: data is generated from two multivariate Gaussians with equal covariance matrices. This assumptions are very restrictive.

Linear SVM on the other side, makes no assumptions on the distribution of the data, and has parameters which allow one to control the number of outliers directly. This seems a quite more flexible approach.

Further, LDA has numerical problems in high dimensions. SVMs are more robust in that setting. So, in general, yes, SVMs will behave better.

As an example (more like a starting point to experiment with) you can take a look at this notebook in R. There you can see how in a general case (where assumptions are no longer warranted and for a higher dimensional problem) linear SVMs usually perform better.

| cite | improve this answer | |
$\endgroup$
5
$\begingroup$

I suspect the reason for the claim that LDA is more likely to overfit than the SVM is because the SVM incorporates regularisation (i.e. a penalty term) that allows you to control the complexity of the model, and hence avoid over-fitting. However, there are also regularised versions of LDA (the Least-Squares SUpport Vector Machine is equivalent to one form of regularised LDA and uses the same form of regularisation as the SVM). So strictly speaking the claim is probably incorrect.

As to non-linear SVMs, as mentioned by Jacques, there are non-linear variants of LDA as well (e.g. Kernel Fisher Discriminant analysis), so for a fair comparison, you would compare SVM and KFD with Gaussian kernels, and in that case I'd say that neither was a-priori more likely to overfit, as both have the same form of regularisation to avoid over-fitting.

| cite | improve this answer | |
$\endgroup$
2
$\begingroup$

I think it is the reverse.

SVM specially with a Gaussian kernel have a much higher probability of overfitting the data than LDA which is LINEAR on the data space!. SVM with a Gaussian Kernel will map the data into a infinite dimension space, and in that space the data is linearly separable. If one uses a very high C, the cost of errors (placing data in the wrong side of the margin) will be unbearable, and the SVM will find the hyperplane that separates the classes (since there is one - they are linearly separable!). That plane (linear on the infinite dimension space) will be a very convoluted manifold in the data space, that separates the classes exactly - the very definition of overfitting!!

For SVM with a linear kernel, I dont know, but if the data is generated by two Gaussians of the same shape (same covariance matrix) the LDA is the optimal solution, which would mean the least overfitting. A Linear SVM can at most match the LDA! For other cases, I dont know.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ (-1) I am sorry for the downvote, but this answer is currently the top one in this thread and I am afraid it can be misleading. As others replies have already pointed out, in high dimensions LDA tends to overfit badly, because its computations rely on inversion of the within-class covariance matrix that is very difficult to estimate reliably unless $N>>k$. See my illustrations here. Of course one can use a regularized version of it, but "vanilla" LDA does not use any regularization, whereas SVM always does. $\endgroup$ – amoeba Jul 30 '14 at 10:21
  • $\begingroup$ @amoeba: in contrast, you'd say that vanilla SVM have a soft margin, though? I think one could argue that the plainest thinkable SVM is hard margin. Which I'd guess to have at least as much overfitting problems in high dimensions as LDA (possibly more: unregularized LDA gives equal weight to all available cases, whereas the [hard margin] SVM focuses on few cases at the boundary) $\endgroup$ – cbeleites unhappy with SX Jul 30 '14 at 12:17
  • $\begingroup$ @cbeleites: No, I guess vanilla SVM is the one with hard margin, I agree with you here. But I don't know enough about its behaviour in high dimensions to comment on overfitting. I think, though, that if the classes are perfectly separated, then hard margin SVM is unlikely to overfit (because maximum margin itself acts as a regularizer), but LDA still might. In a more realistic case of non-perfect separation, I do not know. In any case, my primary objection to the answer above was that LDA does not result in (quote) "the least overfitting". $\endgroup$ – amoeba Jul 30 '14 at 13:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.