Why does doing LDA on a outer product of input matrix gives same result as doing PCA of the input matrix?

Why does doing Linear Discrimenant Analysis (LDA) on a outer product of input matrix gives same result as doing Principal Component Analysis (PCA) of the input matrix?

Not exactly sure where this came from and why it works, but the outer product of the input matrix $X^T X$

In PCA they use it to make the matrix square, so they can perform linear discrimant analysis on that matrix.

It's also used in Normal equation as the first step, for solving the regression coefficient in linear regression. A geometric interpretation would be very helpful.

• Do you mean geometric interpretations like those offered in stats.stackexchange.com/questions/2691/…?
– whuber
Feb 22 '14 at 3:24
• Show an example here. What you are writing about seems to be a coincident. Feb 22 '14 at 7:54
• Your question is quite confusing and as stated does not make sense. What does "LDA on a outer product of input matrix" mean? Please clarify. Feb 22 '14 at 17:03

(This is for data with cases in rows and variates in columns)

In a way the answer is: it doesn't, you probably overlooked two small points.

• The centering of the input is different: PCA centers on the overall mean, LDA on the class means.
• The projections differ by whether the eigenvalues end up in the scores (PCA) or in the loadings/coefficients (LDA)
As this is just a squeezing of each individual axis, it is easy to overlook this difference in plots unless the axis scaling is fixed.

• input vs. outer product: that is because of the connections between SVD on input and eigen decomposition of the outer product.

Here's the long explanation

PCA

First of all, the underlying big principle in PCA is a singular value decomposition $\mathbf X = \mathbf U \mathbf \Sigma \mathbf V^T$ in order to find the major axes of the variance-covariance ellipsoid.

Remember that $\mathrm{COV} (\mathbf X) = \frac{1}{n - 1} \mathbf X^T \mathbf X$ for mean centered $\mathbf X$ .

There are several equivalent ways of doing this. In particular, you can use the eigen decomposition of $\mathbf X^T \mathbf X = \mathbf V \mathbf \Lambda \mathbf V^T$ or of $\mathbf X \mathbf X^T = \mathbf U \mathbf \Lambda \mathbf U^T$. (The two $\mathbf \Lambda$s are diagonal matrices, their first $r$ diagonal elements are the same ($r$ = rank) and they are also the sqare of the diagonal elements of $\Sigma$ (also a diagonal matrix), the remaining elements are 0.)

Now PCA uses $\mathbf V$ for projection. This is a rotation that aligns the major axes of the variance-covariance ellipsoid with the new coordinate axes. The PC scores are then $\mathbf U \mathbf \Sigma$.

LDA

Recall that LDA maximises the between-class variance : within-class variance. While it is not immediately obvious which directions do this in general, this is easy in the special case where the within-class variance has a spherical structure (i.e. is proportional to the identity matrix). In that case, the directions between the class means maximise the between-class variance.

• The difference between within-class variance-covariance matrix and the overall-variance covariancee matrix above is that the centering occurs within each class.

• The $\mathbf U$ of the SVD above has the requested identity covariance structure. So for LDA, we go one step further in the projection compared to the CA and squeeze the new axes so that the variance-covariance matrix of the scores is the identity matrix.
So all we need to do is to multiply $\mathbf \Sigma$ into the loadings (coefficients) instead of into the scores.

• In that LDA score space, the directions between the class means are normal vectors for the decision planes. Of course we can project it back into the original space, where the directions are tilted according to the within-class variance-covariance structure.

Here's a picture: • (+1) Maybe you could add somewhere in the PCA part that you assume that in $\mathbb{X}$ columns are variables/features and rows are observations/data-points (as often the convention is opposite). Feb 22 '14 at 16:56
• @amoeba: I added the info. But can you point me to where the opposite convention is the default? So far, the closest I've met to that was when both Q- and R-mode analyses were considered (but the data convention was still cases in rows). Feb 22 '14 at 17:00
• Sorry, I got confused when I wrote my comment. The convention you used is indeed more common (at least in the textbooks I know and use). However, I definitely did see multiple machine learning papers using the opposite convention. I don't have any examples at hand, but I could search for some if you are really curious. Feb 22 '14 at 22:02
• @amoeba: Ahh, now that you say that I remember seeing some algorithm descriptions that were "transposed" - I somehow had an idea that the caos stems rather from the different "coordinate systems" for matrices (down then right), cartesian (right then up) and images (right then down)... Feb 22 '14 at 22:06
• @cbeleites: The pic says LDA discrimination plane is orthogonal to the line through the class means. Sorry for asking. Does it mean that the discriminant (its axis) itself is that line connecting the 2 centres on the right? And on the left, this axis would remain, perpendicular to the "plane" (the thick line), so that it doesn't connect the 2 centres anymore? (See e.g. my pic here). Feb 23 '14 at 18:39