3
$\begingroup$

Does linear discriminant analysis always project the points to a line? Most of the graphical illustrations of LDA that I see online use an example of 2 dimensional points which are projected onto a straight line y=mx+c. If the points were each a 10-dimensional vector, does LDA still project them to a line?

Or would it project them to a hyperplane with 9 dimensions or less.

ANother question about projections: If I have a vector Y=[a,b,c,d]. The projection of this vector onto a given line is the product of the direction vector V of the line and the vector Y. This is equivalent to a dot product given by transpose(V).Y, and gives just one number (a scalar).

This seems to be the way how LDA works. So, if I may ask, does LDA map a full n-dimensional vector onto a scalar (a singe number)?

Apologies in advance for my newbie question.

Improve this question
$\endgroup$
5
  • $\begingroup$ If you have n dimensional data, LDA reduces it to at most n-1. In 2D example, the only reduction is 1D, hence a line. $\endgroup$
    – Vladislavs Dovgalecs
    Commented Aug 18, 2015 at 15:17
  • $\begingroup$ @xeon: sorry, but this is wrong - see my answer. $\endgroup$
    – cbeleites
    Commented Aug 18, 2015 at 16:10
  • $\begingroup$ minaj, welcome to cross validated. $\endgroup$
    – cbeleites
    Commented Aug 18, 2015 at 16:10
  • $\begingroup$ @cbeleites Yes, you are right! This about classes of the data, not the dimensionality! $\endgroup$
    – Vladislavs Dovgalecs
    Commented Aug 18, 2015 at 16:13
  • $\begingroup$ @VladislavsDovgalecs Can I know that in the context of dimensionality reduction using LDA/FDA. LDA/FDA can start with n dimensions and end with k dimensions, where k < n. Is that correct? Or The output is c-1 where c is the number of classes and the dimensionality of the data is n with n>c. $\endgroup$
    – aan
    Commented May 6, 2020 at 21:25

2 Answers 2

Reset to default
3
$\begingroup$

LDA seeks to reduce dimensionality while preserving as much of the class discriminatory information as possible. Assume we have a set of $d$-dimensional observations $X$, belonging to $C$ different classes. The goal of LDA is to find an linear transformation (projection) matrix $L$ that converts the set of labelled observations $X$ into another coordinate system $Y$ such that the class separability is maximized. The dataset is transformed into the new subspace as:

\begin{equation} Y = XL \end{equation}

The columns of the matrix $L$ are a subset of the $C-1$ largest (non-orthogonal) eigenvectors of the squared matrix $J$, obtained as:

\begin{equation} J = S_{W}^{-1} S_B \end{equation}

where $S_W$ and $S_B$ are the scatter matrices within-class and respectively between-classes.

When it comes to dimension reduction in LDA, if some eigenvalues have a significantly bigger magnitude than others then we might be interested in keeping only those dimensions, since they contain more information about our data distribution. This becomes particularly interesting as $S_B$ is the sum of $C$ matrices of rank $\leq 1$, and the mean vectors are constrained by $\frac{1}{C}\sum_{i=1}^C \mu_i = \mu$ \cite{c.radhakrishnarao1948}. Therefore, $S_B$ will be of rank $C-1$ or less, meaning that there are only $C-1$ eigenvalues that will be non-zero (more info here). For this reason, even if the dimensionality $k$ of the sub-space $Y$ can be arbitrarily chosen, it does not make any sense to keep more than $C-1$ dimensions, as they will not carry any useful information. In fact, in \ac{lda} the smallest $d - (C-1)$ dimensions have magnitude zero, and therefore the subspace $Y$ should have exactly $k = C-1$ dimensions.

Improve this answer
$\endgroup$
13
  • $\begingroup$ is that correct that: Let say my original dataset has 2 classes, the output will be 1 dimensionality ( 2 – 1 =1 ), likewise, if my original dataset has 5 classes, the output will be 4 dimensionality. $\endgroup$
    – aan
    Commented May 8, 2020 at 16:28
  • $\begingroup$ If you choose L to contain only the non-zero eigenvectors (meant as those eigenvectors whose corresponding eigenvalue is non-zero), yes, correct. $\endgroup$
    – Renthal
    Commented May 11, 2020 at 8:05
  • 1
    $\begingroup$ Your definition is imprecise. You can yes choose any output you want (provided it is smaller or equal to $d$, with the notation above), however, the linear separability of a space with dimension $x$ such that $C-1 < x \leq d$ is not going to be any better than another space with dimension $y = C-1$. The eigervectors with zero (not sure where you get the imaginary part into play?) eigenvalue are in the $J$ matrix, not in in the final space $Y$. Hope it helps. $\endgroup$
    – Renthal
    Commented May 12, 2020 at 11:52
  • 1
    $\begingroup$ Because you have only $C-1$ non-zero eigenvalues in matrix $J$. $\endgroup$
    – Renthal
    Commented May 14, 2020 at 12:27
  • 1
    $\begingroup$ The Utilization of Multiple Measurements in Problems of Biological Classification, C. Radhakrishna Rao, Journal of the Royal Statistical Society. Series B (Methodological), 1948, jstor.org/stable/2983775 $\endgroup$
    – Renthal
    Commented May 25, 2020 at 7:53
2
$\begingroup$

LDA project to (at most) $n_{classes} - 1$ dimensions, so binary (2-class) LDA reduces to 1D (= onto line).
10 classes would lead to a 9D projection (as long as X is at least 9D, of course).

  does LDA map a full n-dimensional vector onto a scalar (a singe number)? Not always, see above.

For more details on what the projection step does, see e.g. https://stats.stackexchange.com/a/87509/4598

(Obviously, if you code your classes as numbers then the final class prediction will be a single number)

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.