Why does doing LDA on a outer product of input matrix gives same result as doing PCA of the input matrix?

Question

Why does doing Linear Discrimenant Analysis (LDA) on a outer product of input matrix gives same result as doing Principal Component Analysis (PCA) of the input matrix?

Not exactly sure where this came from and why it works, but the outer product of the input matrix $X^T X$

In PCA they use it to make the matrix square, so they can perform linear discrimant analysis on that matrix.

It's also used in Normal equation as the first step, for solving the regression coefficient in linear regression. A geometric interpretation would be very helpful.

Answer 1

First of all, these are guesses what your question could be about, so please comment/update your question.

(You are not asking about kernel LDA, are you?)

(This is for data with cases in rows and variates in columns)

In a way the answer is: it doesn't, you probably overlooked two small points.

• The centering of the input is different: PCA centers on the overall mean, LDA on the class means.

• The projections differ by whether the eigenvalues end up in the scores (PCA) or in the loadings/coefficients (LDA)
  As this is just a squeezing of each individual axis, it is easy to overlook this difference in plots unless the axis scaling is fixed.

• input vs. outer product: that is because of the connections between SVD on input and eigen decomposition of the outer product.

Here's the long explanation

---

PCA

First of all, the underlying big principle in PCA is a singular value decomposition $\mathbf X = \mathbf U \mathbf \Sigma \mathbf V^T$ in order to find the major axes of the variance-covariance ellipsoid.

Remember that $\mathrm{COV} (\mathbf X) = \frac{1}{n - 1} \mathbf X^T \mathbf X$ for mean centered $\mathbf X$ .

There are several equivalent ways of doing this. In particular, you can use the eigen decomposition of $\mathbf X^T \mathbf X = \mathbf V \mathbf \Lambda \mathbf V^T$ or of $\mathbf X \mathbf X^T = \mathbf U \mathbf \Lambda \mathbf U^T$. (The two $\mathbf \Lambda$s are diagonal matrices, their first $r$ diagonal elements are the same ($r$ = rank) and they are also the sqare of the diagonal elements of $\Sigma$ (also a diagonal matrix), the remaining elements are 0.)

Now PCA uses $\mathbf V$ for projection. This is a rotation that aligns the major axes of the variance-covariance ellipsoid with the new coordinate axes. The PC scores are then $\mathbf U \mathbf \Sigma$.

LDA

Recall that LDA maximises the between-class variance : within-class variance. While it is not immediately obvious which directions do this in general, this is easy in the special case where the within-class variance has a spherical structure (i.e. is proportional to the identity matrix). In that case, the directions between the class means maximise the between-class variance.

• The difference between within-class variance-covariance matrix and the overall-variance covariancee matrix above is that the centering occurs within each class.

• The $\mathbf U$ of the SVD above has the requested identity covariance structure. So for LDA, we go one step further in the projection compared to the CA and squeeze the new axes so that the variance-covariance matrix of the scores is the identity matrix.
  So all we need to do is to multiply $\mathbf \Sigma$ into the loadings (coefficients) instead of into the scores.

• In that LDA score space, the directions between the class means are normal vectors for the decision planes. Of course we can project it back into the original space, where the directions are tilted according to the within-class variance-covariance structure.

Here's a picture:

Citation is: Krafft, C.; Steiner, G.; Beleites, C. & Salzer, R.: Disease recognition by infrared and Raman spectroscopy., Journal of Biophotonics, 2, 13-28 (2009). DOI: 10.1002/jbio.200810024