I have a original dataset with 70 samples, each sample with 96 features. The samples are labeled as 0 or 1. So I use linear discriminant analysis (LDA) to reduce the dimensionality of all the dataset, generating a samples with only one feature.

My results with all 96 features is 83% of accuraccy, with the projeted samples i have a 100% of accuracy. I'm using a svm for classification, with a split of 80% for training and 20% for test.

So my questions is : LDA is known as a supervised method of classification, but often used as a dimensionality reduction technique. The usage of LDA in all samples is doing a pre-training in the data? If yes, did I have to split the data before use LDA for reduction and later use the transformation to project the test data?

  • 1
    $\begingroup$ That's as with any modeling. If you want to test on a test dataset you train (model, extract; then classify if you want) the discriminant functions on the train dataset. Then, having their coefficients you compute the functions' values in the test dataset and perform classification by them there. You can then compare classification accuracy in train and test sets. $\endgroup$ – ttnphns Feb 25 '16 at 7:40
  • $\begingroup$ You say you are using a svm for classification. But what classification you are using in LDA? LDA uses gaussian linear classifier (a Bayes classifier). You should not mix the methods. If you are comparing classifications as done by all the features and by just discriminants you should use one type of classifier in both cases. Another question to you: I wonder how you managed to run LDA on n=70 < p=96 singular data? $\endgroup$ – ttnphns Feb 25 '16 at 7:48
  • $\begingroup$ I'm not using LDA for classification. I'm using it for dimension reduction only. $\endgroup$ – Caio Belfort Feb 25 '16 at 21:36

LDA used as a dimensionality-reducing technique can be seen as a "supervised PCA", so it will redistribute your data in a new space (of lesser dimension) where classes should be better separated (based on the labels you provided).

The projection matrix is made of the first eigen vectors (of positive eigen values) given by LDA to project your test data into that new feature space, then input that vector into your SVM.

Note that you should use a non-linear kernel in your SVM (e.g. RBF), otherwise you'll have a linear transformation on top of another linear transformation, which will not improve discrimination. SVM and LDA are pretty much equivalent when it comes to linear classification.

  • $\begingroup$ em... but LDA classify using the cluster center of each class while SVM uses support vectors which are the boundary samples. $\endgroup$ – Xiaoxiong Lin Oct 30 '18 at 13:15
  • $\begingroup$ @XiaoxiongLin for classification, yes. But LDA can also be used to reduce the number of dimensions, by maximizing the distance between clusters centers $B$ while minimizing the intra-cluster covariance $W$ (so to speak. i.e. minimizing $W/B$ ratio). $\endgroup$ – Matthieu Oct 30 '18 at 15:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.