In this article , the author links linear discriminant analysis (LDA) to principal component analysis (PCA). With my limited knowledge, I am not able to follow how LDA can be somewhat similar to PCA.
I have always thought that LDA was a form of classification algorithm, similar to logistic regression. I will appreciate some help in understanding how LDA is similar to PCA, i.e how is it a dimensionality reduction technique.
As I've noted in the comment to your question, discriminant analysis is a composite procedure with two distinct stages - dimensionality reduction (supervised) and classification stage. At dimensionality reduction we extract discriminant functions which replace the original explanatory variables. Then we classify (typically by Bayes' approach) observations to the classes using those functions.
Some people tend to fail to recognize this clear-cut two-stage nature of LDA simply because they have got acquainted themselves only with LDA with 2 classes (called Fisher's discriminant analysis). In such analysis, only one discriminant function exists and classification is straightforward, and so everything can be explained in a textbook in a single "pass" without inviting concepts of space reduction and Bayes classification.
LDA is closely related to MANOVA. The latter is a "surface and broad" side of the (multivariate) linear model while the "depth and focused" picture of it is canonical correlation analysis (CCA). The thing is that the correlation between two multivariate sets of variables is not uni-dimensional and is explained by a few pairs of "latent" variables called canonical variates.
As a dimensionality reduction, LDA is theoretically a CCA with two sets of variables, one set being the correlated "explanatory" interval variables and the other set being the $k-1$ dummy (or other contrast coded) variables representing the $k$ groups, the classes of observations.
In CCA, we consider the two correlated variable sets X and Y as equal in rights. Therefore we extract canonical variates from both sides, and they form pairs: variate 1 from set X and variate 1 from set Y with canonical correlation between them maximal; then variate 2 from set X and variate 2 from set Y with a smaller canonical correlation, etc. In LDA, we usually are not interested numerically in canonical variates from the class set side; we however take interest in the canonical variates from the explanatory set side. Those are called canonical discriminant functions or discriminants.
The discriminants are what correlate maximally with the "lines" of separateness among the groups. Discriminant 1 explains the major portion of separateness; discriminant 2 picks some of the separeteness left unexplained due to the orthogonality to the previous separateness; descriminat 3 explains yet some remnant of separateness orthogonal to the previous two, etc. In LDA with $p$ input variables (dimensions) and $k$ classes the possible number of discriminants (reduced dimensions) is $min(k-1,p)$ and when the assumptions of LDA hold this number of them completely discriminate between classes and are able to fully classify the data to the classes (see).
To repeate, this is actually CCA in its nature. LDA with 3+ classes is even called "canonical LDA". Despite that CCA and LDA are typically implemented algorithmically somewhat differently, in views of program efficiency, they are "same" enough so that it is possible to recalculate results (coefficients etc.) obtained in one procedure onto those obtained in the other. Most of the LDA specificity lies in the domain of coding the categorical variables representing groups. This is that same dilemma which is observed in (M)ANOVA. Different coding schemes lead to different ways of interpretation of the coefficients.
Since LDA (as dimensionality reduction) can be understood as a particular case of CCA, you definitely have to explore this answer comparing CCA with PCA and regression. The main point there is that CCA is, in a sense, closer to regression than to PCA because CCA is a supervised technique (a latent linear combination is drawn out to correlate with something external) and PCA is not (a latent linear combination is drawn to summarize the internal). These are two branches of dimensionality reduction.
When it comes to math you might find that while the variances of the principal components correspond to the eigenvalues of the data cloud (the covariance matrix between the variables), the variances of the discriminants are not so clearly related to those eigenvalues that are produced in LDA. The reason is that in LDA, eigenvalues do not summarize the shape of the data cloud; rather, they pertain to the abstract quantity of the ratio of the between-class to the within-class variation in the cloud.
So, principal components maximize variance and discriminants maximize class separation; a simple case where a PC fails to discriminate between classes well enough but a discriminant can is these pictures. When drawn as lines in the original feature space discriminants do not usually appear orthogonal (being uncorrelated, nevertheless), but PCs do.
Footnote for meticulous. How, in their results, LDA is exactly related to CCA. To repeat: if you do LDA with p variables and k classes and you do CCA with Set1 as those p variables and Set2 as k-1 indicator dummy variables representing groups (actually, not necessarily indicator variables - other types of contrast variables, such as deviation or Helmert - will do), then the results are equivalent in regards to the canonical variates extracted for Set1 - they directly correspond to the discriminant functions extracted in the LDA. What is the exact relationship, though?
Algebra and terminology of LDA is explained here, and algebra and terminology of CCA is explained here. Canonical correlations will be the same. But what about coefficients and "latents"'s values (scores)? Consider a $j$th discriminant and correspondent ($j$th) canonical variate. For them,
$\frac {\text {CCA standardized coefficient}}{\text {LDA raw coefficient}} = \frac {\text {CCA canonical variate value}}{\text {LDA discriminant value}} = \sqrt \frac {\text {pooled within class variance in the variate }}{\text {pooled within class variance in the discriminant}}$
"Pooled within class variance" is the weighted average of the group variances with weight = n-1 in a group. In discriminant, this quantity is $1$ (read in LDA algebra link), and so the coefficient of proportionality to switch onto CCA results from LDA results is simply $$\sqrt {\text {pooled within class variance in the variate}}$$. But because the canonical variate is standardized in the whole sample, this coefficient is equal to the $\text {st. deviation of the discriminant}$ (which is standardized within groups). So, just divide the LDA results (coefficients and scores) by the discriminant's $\sigma$ to get the CCA results.
The difference between CCA and LDA is due to that LDA "knows" that there are classes (groups): you directly indicate the groups to compute the within and between scatter matrices. That makes it both the computations faster and results more convenient for subsequent classification by discriminants. CCA, on the other hand, isn't aware of classes and process the data as if they all were continuous variables - which is more general but a slower way of computation. But the results are equivalent, and I've shown how.
So far it was implied that the k-1 dummies are entered CCA the typical way, i.e. centered (like the variables of Set1). One might ask, is it possible to enter all k dummies and do not center them (to escape singularity)? Yes, it is possible, albeit probably less convenient. There will appear a zero-eigenvalue additional canonical variate, coefficients for it should be thrown away. Other results remain valid. Except the dfs to test the significance of canonical correlations. Df for the 1st correlation will be p*k which is wrong and the true df, as in LDA, is p*(k-1).
