Why is LDA considered to be a classifier?

Question

I am new to machine learning and I was reading about dimensional reduction algorithms like LDA(linear discriminant analysis) and PCA. Currently I am using LDA to find the optimal dimensions that separate my classes. But once I do that and get some new data to classify, I reproject it using LDA and then use KNN to define to which cluster my data corresponds to the most. So how Classification can be done with LDA alone?

Answer 1

LDA on its own can be used to classify, you do not need to use KNN. In LDA you are modeling the data as a set of multivariate normal distributions, with a common covariance matrix $\Sigma$ but different mean vectors $\mu_k$ for $k$ classes. You simply use the estimates of $\Sigma$ and $\mu_k$ to compute log ratios of the density for one class vs another $$d(c_1) = log\frac{P(Class = c_1|X)}{P(Class = c_2|X)} $$ which results in linear discriminant functions thanks to taking the log and the fact that we use the same covariance matrix for all classes. You then classify an observation to whichever discriminant function is highest. You also need estimates for the marginal probabilites $P(Class = c_k)$ which can simply be $\frac{N_i}{N}$ or you can experiment with your own values so long as they sum to 1

Answer 2

LDA is a dimensionality reduction method, not a classifier. In SKlearn, LinearDiscriminantAnalysis seems to be a naive bayes classifier after LDA, see docs.