1
$\begingroup$

I have a original dataset with 70 samples, each sample with 96 features. The samples are labeled as 0 or 1. So I use linear discriminant analysis (LDA) to reduce the dimensionality of all the dataset, generating a samples with only one feature.

My results with all 96 features is 83% of accuraccy, with the projeted samples i have a 100% of accuracy. I'm using a svm for classification, with a split of 80% for training and 20% for test.

So my questions is : LDA is known as a supervised method of classification, but often used as a dimensionality reduction technique. The usage of LDA in all samples is doing a pre-training in the data? If yes, did I have to split the data before use LDA for reduction and later use the transformation to project the test data?

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ That's as with any modeling. If you want to test on a test dataset you train (model, extract; then classify if you want) the discriminant functions on the train dataset. Then, having their coefficients you compute the functions' values in the test dataset and perform classification by them there. You can then compare classification accuracy in train and test sets. $\endgroup$
    – ttnphns
    Commented Feb 25, 2016 at 7:40
  • $\begingroup$ You say you are using a svm for classification. But what classification you are using in LDA? LDA uses gaussian linear classifier (a Bayes classifier). You should not mix the methods. If you are comparing classifications as done by all the features and by just discriminants you should use one type of classifier in both cases. Another question to you: I wonder how you managed to run LDA on n=70 < p=96 singular data? $\endgroup$
    – ttnphns
    Commented Feb 25, 2016 at 7:48
  • $\begingroup$ I'm not using LDA for classification. I'm using it for dimension reduction only. $\endgroup$
    – Caio Belfort
    Commented Feb 25, 2016 at 21:36
  • $\begingroup$ Can I know that in the context of dimensionality reduction using LDA/FDA. LDA/FDA can start with n dimensions and end with k dimensions, where k < n. Is that correct? Or The output is c-1 where c is the number of classes and the dimensionality of the data is n with n>c. $\endgroup$
    – aan
    Commented May 6, 2020 at 21:23
  • $\begingroup$ @aan please ask a separate question instead of asking by commenting all posts... $\endgroup$
    – Matthieu
    Commented May 7, 2020 at 11:09

1 Answer 1

Reset to default
2
$\begingroup$

LDA used as a dimensionality-reducing technique can be seen as a "supervised PCA", so it will redistribute your data in a new space (of lesser dimension) where classes should be better separated (based on the labels you provided).

The projection matrix is made of the first eigen vectors (of positive eigen values) given by LDA to project your test data into that new feature space, then input that vector into your SVM.

Note that you should use a non-linear kernel in your SVM (e.g. RBF), otherwise you'll have a linear transformation on top of another linear transformation, which will not improve discrimination. SVM and LDA are pretty much equivalent when it comes to linear classification.

Improve this answer
$\endgroup$
9
  • $\begingroup$ em... but LDA classify using the cluster center of each class while SVM uses support vectors which are the boundary samples. $\endgroup$
    – Xiaoxiong Lin
    Commented Oct 30, 2018 at 13:15
  • $\begingroup$ @XiaoxiongLin for classification, yes. But LDA can also be used to reduce the number of dimensions, by maximizing the distance between clusters centers $B$ while minimizing the intra-cluster covariance $W$ (so to speak. i.e. minimizing $W/B$ ratio). $\endgroup$
    – Matthieu
    Commented Oct 30, 2018 at 15:18
  • $\begingroup$ @Matthieu Can I know that in the context of dimensionality reduction using LDA/FDA. LDA/FDA can start with n dimensions and end with k dimensions, where k < n. Is that correct? Or The output is c-1 where c is the number of classes and the dimensionality of the data is n with n>c. $\endgroup$
    – aan
    Commented May 6, 2020 at 21:24
  • $\begingroup$ @aan both are correct because $c < n$. Just mind that eigenvalues for dimensions $> c-1$ will be $0$ or imaginary, so you won't get more than $c-1$ dimensions really... $\endgroup$
    – Matthieu
    Commented May 7, 2020 at 11:11
  • $\begingroup$ @Matthieu Thanks. Let say my original dataset has 2 classes, the output will be 1 dimensionality ( 2 – 1 =1), likewise, if my original dataset has 5 classes, the output will be 4 dimensionality. $\endgroup$
    – aan
    Commented May 7, 2020 at 15:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.