3
$\begingroup$

I am new to machine learning and I was reading about dimensional reduction algorithms like LDA(linear discriminant analysis) and PCA. Currently I am using LDA to find the optimal dimensions that separate my classes. But once I do that and get some new data to classify, I reproject it using LDA and then use KNN to define to which cluster my data corresponds to the most. So how Classification can be done with LDA alone?

$\endgroup$
  • $\begingroup$ LDA is a composite procedure. It consists of two distinct stages: 1) dimensionality reduction (extraction of distriminants); 2) classification (usually by Bayes approach). When there are just two classes ("Fisher's LDA") both stages can be mathematically managed as "two in one" because LDA then is theoretically equivalent to linear regression with normal assignment. $\endgroup$ – ttnphns Jul 24 '15 at 15:39
5
$\begingroup$

LDA on its own can be used to classify, you do not need to use KNN. In LDA you are modeling the data as a set of multivariate normal distributions, with a common covariance matrix $\Sigma$ but different mean vectors $\mu_k$ for $k$ classes. You simply use the estimates of $\Sigma$ and $\mu_k$ to compute log ratios of the density for one class vs another $$d(c_1) = log\frac{P(Class = c_1|X)}{P(Class = c_2|X)} $$ which results in linear discriminant functions thanks to taking the log and the fact that we use the same covariance matrix for all classes. You then classify an observation to whichever discriminant function is highest. You also need estimates for the marginal probabilites $P(Class = c_k)$ which can simply be $\frac{N_i}{N}$ or you can experiment with your own values so long as they sum to 1

$\endgroup$
  • 1
    $\begingroup$ How is this probability estimated? what do you mean by N_i/N ?? LDA tutorials always stop at the fitting part and don't talk about the classification. $\endgroup$ – Mehdi Dec 26 '15 at 16:03
0
$\begingroup$

LDA is a dimensionality reduction method, not a classifier. In SKlearn, LinearDiscriminantAnalysis seems to be a naive bayes classifier after LDA, see docs.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.