# Two-class LDA with multiple variables: can we separate classes and know the weight of variables?

So far I have performed an LDA having two classes, but I'm really struggling with certain aspectos of the analysis. I am working with biomedical data in which I would like to classify in k=2 groups: CG and IG based on p = 11 variables. Theorically, as far as I know you're gonna get discriminant functions for k-1 dimmensions and N individuals ( N = 59 in my case).

1. From the perspective of separation, I've seen graphs explaining two-class discrimination ellipses, I guess coming from centroids. Is this possible? On other posts, I have just seen a density function with a cutoff point discriminating both groups. As I said I am not quite sure about the rule of the two-category group and its display.
2. Once you perform the lda is there any way from the transformed matrix to infer the weight of the classifiers (p=11).

• “Once you perform the lda is there any way from the transformed matrix to infer the weight of the classifiers” Do you mean by 'weights' something else than the parameters in the discriminant function? Jan 17 at 11:12
• The influence of the variables. I've read that standardizing variables and then running model. The scaling parameter gives a notion of the influence of variables to discriminate data2<- data data2[,2:p_variables] <- scale(data2[,2:p_variables]) lda(class~ . , data2)\$scaling Am I right about it? Jan 17 at 11:26
• Why do LDA rather than logistic regression? I'm not saying your choice is wrong, but I'm wondering about it, since LDA has more assumptions than logistic reg. does (e.g that the data are normally distributed). Jan 17 at 11:43
• Data is normally distributed. Not sure about it, I'm comparing it with PCA, and as far as I know both PCA and LDA seem to complement each other, plus I was interested in observing if the set of variables selected could be able to classify correctly, and I kind of understood this is more about LDA. To be honest I don't totally understand difference between the 2 methods to give a coherent answer on why picking one over the other Jan 17 at 11:51
• LDA leads to a logistic function for class probabilities: stats.stackexchange.com/questions/549508 However it is not the same procedure as logistic regression. Logistic regression can occur in other ways as well (e.g. as approximation for a probit model with completely different distributions than the multivariate normal distribution). The fitting procedure differs. LDA fits the the underlying latent model (the multivariate normal distributions). I imagine that LDA might perform better than logistic regression when the assumptions are true (because it uses more information). Jan 17 at 12:02

I've seen graphs explaining two-class discrimination ellipses, I guess coming from centroids.

...

On other posts, I have just seen a density function with a cutoff point discriminating both groups.

Both situations apply to a two-class LDA.

### Bayes classifier

You can consider LDA as a Bayes classifier, assuming multivariate Gaussian distributions for the classes.

An example image that shows how this works is:

(I have to find a better one, this image here fits distributions with different covariance, and the QDA curve is the result, the LDA curve has been added for comparison)

You can see the elipses which are the fitted densities for the two distributions. and the points where they have the same value P(class 1) = P(class 2) is the classifier boundary.

### One dimensional viewpoint

For the LDA the classifier is a single p-1 dimensional plane and we can consider a coordinate that is along the normal vector on that plane.

An analogues switch to a 1 dimensional vector is made here for logistic regression: Probability threshold in ROC curve analyses

We have some distribution of variables for two classes. Below this is shown for two variables, such that we can plot it, but the principle is the same in higher dimensions.

To this we fit a linear classifier that represents several parallel iso-planes in the space. At each plane we predict a certain odds or probability ratio P(class 1)/P(class 2)

This data can be projected onto a single dimension that is perpendicular to those planes, and then we have a 1-dimensional view like below.

The picture above is a typical image for logistic regression, plotting the single observed points, but the points can also be represented as a density (using a histogram or some estimation of the density)

An example is the graph below (now with three classes):

From "The use of multiple measurements in taxonomic problems" Annals of Eugenics, Vol VII, Pt. II, op. 179-188, 136

Another graph that might help to see the connection between the multidimensional centroids and the 1-dimensional density graphs is:

The histograms show the distribution of the two classes along different 1 dimensional directions.

• I don't truly understand which parameters are you going to use for depicting the ellipses. The origin is not methodologically specified. The standardized coefficients? I am just getting one column. The posterior probabilities? About the image similar to logit regression, the coefficients you get assure 100% discrimination such as depicted? Probability class 1 completely distinguished from class 0? I guess this has something to do with the graph of isolines which I don't know its meaning either. Jan 17 at 14:42
• "I don't truly understand which parameters are you going to use for depicting the ellipses" In the example I used 2 parameters, such that it can be plotted, but more generally these ellipses are multidimensional and use all parameters. Jan 17 at 19:55
• What does it mean all parameters? Scores? Standardized coefficientes? Jan 18 at 8:22
• @JavierHernando sorry I mix these terms up often. I mean variables. In the example of the answer I depicted a scatter plot with 2 variables. In computations this can be all of your 11 variables. Jan 18 at 11:47