Linear Discriminant Analysis Assumptions

Question

I'm trying to learn about LDA and so i'm gathering information from different places. One thing which strikes me is on some occasions it's been explained that $\pi_{k}f_{k}(X=x)$ is normally distributed and that there are assumptions on the distribution of the data. From what i've seen these sources don't really dive deep into the geometry of dimensionality reduction. In other cases it seems that LDA is more of just an exercise in dimensionality reduction and there is very little that goes into explaining the underlying assumptions of distributions of the data. What is the 'correct' approach? Does anyone have a source which explains both? Often i can only seem to find an explanation detailing one approach.

Answer 1

I can't tell you much about LDA applied in dimensionality reduction since I didn't realize it can be applied for such case until I just stumbled upon your question. But, I can provide a concise explication of LDA applied in binary classification.

Let's start with vital concepts of LDA, which are multivariate gaussian distributions and maximum likelihood estimations.

Suppose you observe normal distributions of two classes: survived and died. You encounter an individual with X known condition and assess the likelihood of survival or death. How might you estimate the probabilities?

Using known patient data, you create two separate normal distribution models, one for death and another one for survival, using the maximum likelihood estimation to estimate the variance and mean. So essentially, each model (or normal distribution) would be equipped with different variance and mean.

Finally, with the two models fitted on past patient data, you now input the X features of the new patient in each of the two models to get the probabilities of survival and death.

Based on how the model is constructed, you can see why in LDA you'd want to avoid applying the model on non-normally distributed data.

Answer 2

Assumption of LDA

•Each feature/column in the dataset is Gaussian distribution in simple words data points are normally distributed having bell-shaped curves.

•Independent variables are normal for each level of the grouping variable.

•Predictive power can decrease with an increase in correlation between variables.

•All instances are assumed to be randomly sampled and scores on one variable are assumed to be independent.

It is observed that linear discriminant analysis is relatively robust to a slight variation on all of the above assumptions.

It is sometimes recommended to apply dimensionality reduction before using LDA (I suppose, it concerns PCA as unsupervised, - though it seems strange for me). But LDA itself is a supervised classification technique in Machine Learning as so as ->

Under LDA we basically try to address which set of parameters can best describe the association of groups for a class, and what is the best classification model that separates those groups.

as a result, starting from population/sample with multiple features, after applying LDA you're getting several groups rather similar by certain feature_values. Nota bene: it is Supervised (targeting classes are known beforehand) as opposed to PCA. -> here under title 'What is the difference between LDA and PCA for dimension reduction' is comparison, reference and reminding "that LDA makes assumptions about normally distributed classes and equal class covariances (at least the multiclass version)"

So, yes it can be considered to be dimensionality reduction method, but as always in statistics you should care about the distribution of samples, nature of features (& targets if exists) [numerical or categorical] & purpose of this analysis application. See wiki LDA: continuous independent variables and a categorical dependent variable (i.e. the class label)

But ML, in general care little about the shape of distribution - but of course it decreases learning scores (With the Normality of data distribution - you're getting better scores). Also ML do not care (as I've noticed) about Heteroscedasticity -- but this can be explained by the use of maximum-likelihood estimator in ML (that takes into consideration weights - thus not pure OLS used for getting mean, but more resembles WLS-like estimations.Though nevertheless ML often requires scaling/standartization anyway. For such a case better to see manual for the library - though it is not rich for details)