# Qualitative implications of Linear Discriminant Analysis (LDA)

I'm a beginner to LDA, and my question is about its qualitative implications. Say I have two classes of medical data, already classified as:

$C_1=\{x_{11},x_{12}, ...x_{1n_1}\}, C_2=\{x_{21}, x_{22} ...x_{2n_2}\}$. Let's assume that each data lies in $\mathbb{R}^m$, corresponding to $m-$ number of features from a patient, say height, weight, arm-length etc. Assume that the patients in $C_1$ have a certain disease, and the ones in $C_2$ do not. No distributional assumptions are given, and we also don't assume that the covariance matrices for these two classes are the same. (Apprantly, in this note http://www.di.univr.it/documenti/OccorrenzaIns/matdid/matdid437773.pdf that I just studied, they didn't have this assumption, although the wiki page mentions it.) But one can use central limit theorems if necessary. Now, we use linear discriminant analysis (LDA) and find a line or direction $w$ that maximizes the class separation between $C_1, C_2.$

What will that imply from the view point of the disease, or medically?

I previously worked on classification of biomedical data.

The implication of a LDA model is that there's a discriminant hyperplane given by the linear combination of your features which, given the projection of your data on this subspace, maximizes the interclass variance and minimizes the intraclass variance. It's dimension is $\min\left(N,C-1\right)$, $N$ is the feature space dimension and $C$ is the number of classes. And that's it, no explicit medical implications.

Or, in other words, the class centroid will be the furthest apart they can be while the instances from the same class will be the nearest possible given the LDA conditions.

It's of your best interest to have the features be dimensionless, and most probably normalized aswell. Also, in your case, the hyperplane is actually a single dimension.

Now, it could be used to infer a rule to classify new unlabelled cases, if you can show that it's reliable and accurate given your goals. It's a single value given by the linear combination of other values, and the possibility of a parametric solution for classification makes it interesting.

LDA itself doesn't give you such a rule, as it's only a dimensionality reduction method. It gives you the weights to the discriminating subspace, where you could apply any rule to the classification. A common approach is to define a rule based on the distance to the centroids of the classes in this hyperplane, but any other rule can be applied.

In this case, the projection is given by (Derivations in this lecture). $$Y = W^{T}X$$

The weights $W$ are found through the maximization of the Fisher Criterion $J\left(W\right)$ $$W^{T}=\underset{W}{\operatorname{argmax}} \{J(W)\}=\underset{W}{\operatorname{argmax}} \left\{\frac{W^{T}S_{B}W}{W^{T}S_{W}W}\right\}$$

See $J(W)$ depends of the between classes scatter $S_{B}$ and the within classes scatter $S_{W}$. If you add one more dimension, a new feature, to the problem, $S_{W}$ necessarily increases. The only way to make your new $J(W)$ bigger than the other is if your new $S_{B}$ increases the criterion just as much $S_{W}$ diminished it, or in other words, if you obtain a bigger $J(W)$ with more features it's because said features increase the class separation.

If the rule you created in the discriminant subspace is as accurate as the medical diagnosis, congratulations. If it isn't it can still be (probably is) useful, as it's a summary of all the information, and that would allow you to develop, say, a Decision Support System.

• thanks for the answer, upvoted. But I couldn't follow some of the stuffs, if you could please explain for me. – Mathmath May 9 '16 at 23:29
• @Mathmath What exactly you didn't understand? – Firebug May 9 '16 at 23:36
• Sorry the sentence went incomplete somehow. I think my main question is this: okay I understand the dimension of the separating hyperplane, and also that the hyperplane maximizes interclass variability and minimizes the intraclass variability. But then: what can we do next, or what can we infer from the hyperplane about our medical data at hand? – Mathmath May 10 '16 at 1:52
• "Now, it could be used to infer a rule to classify new unlabeled cases, if you can show that it's reliable and accurate given your goals." How exactly can we infer such a rule? – Mathmath May 10 '16 at 1:54
• One another thing that comes to mind is: suppose you've two different kind of data on patients, as follows: say in both cases the clusters you've are with disease or without, but the $m-$ features that you've on the training set are different for these two data sets. Say for the first training data there're $m=2$ features, age and height, and for the second set of data, you've $m=3$, and the features are arm-length, eye-opening and length of fingers. Suppose that I compute the Fisher's linear discriminant $J$ in both cases and in the second case, maximum value of $J$ is much more (contd...) – Mathmath May 10 '16 at 2:00