Is linear discriminant analysis (LDA) more likely to overfit than support vector machine (SVM)? I went to a short talk and the speaker quickly mentioned something like 'LDA (linear discriminant analysis) is more likely to be overfitted than SVM (support vector machine)'. Is this true? And why?
 A: I do not believe it is the inverse either. Both approaches are very different.
LDA is optimal (in a Bayes sense) whenever the assumptions under which it is derived are met, namely: data is generated from two multivariate Gaussians with equal covariance matrices. This assumptions are very restrictive.
Linear SVM on the other side, makes no assumptions on the distribution of the data, and has parameters which allow one to control the number of outliers directly. This seems a quite more flexible approach.
Further, LDA has numerical problems in high dimensions. SVMs are more robust in that setting. So, in general, yes, SVMs will behave better.
As an example (more like a starting point to experiment with) you can take a look at this notebook in R. There you can see how in a general case (where assumptions are no longer warranted and for a higher dimensional problem) linear SVMs usually perform better.
A: I suspect the reason for the claim that LDA is more likely to overfit than the SVM is because the SVM incorporates regularisation (i.e. a penalty term) that allows you to control the complexity of the model, and hence avoid over-fitting.  However, there are also regularised versions of LDA (the Least-Squares SUpport Vector Machine is equivalent to one form of regularised LDA and uses the same form of regularisation as the SVM).  So strictly speaking the claim is probably incorrect.
As to non-linear SVMs, as mentioned by Jacques, there are non-linear variants of LDA as well (e.g. Kernel Fisher Discriminant analysis), so for a fair comparison, you would compare SVM and KFD with Gaussian kernels, and in that case I'd say that neither was a-priori more likely to overfit, as both have the same form of regularisation to avoid over-fitting.
A: I think it is the reverse. 
SVM specially with a Gaussian kernel have a much higher probability of overfitting the data than LDA which is LINEAR on the data space!. SVM with a Gaussian Kernel will map the data into a infinite dimension space, and in that space the data is linearly separable. If one uses a very high C, the cost of errors (placing data in the wrong side of the margin) will be unbearable, and the SVM will find the hyperplane that separates the classes (since there is one - they are linearly separable!). That plane (linear on the infinite dimension space) will be a very convoluted manifold in the data space, that separates the classes exactly - the very definition of overfitting!!
For SVM with a linear kernel, I dont know, but if the data is generated by two Gaussians of the same shape (same covariance matrix) the LDA is the optimal solution, which would mean the least overfitting. A Linear SVM can at most match the LDA! For other cases, I dont know.
