How to calculate the variance-covariance matrix of the principal components from the variance-covariance matrix of the original data?

Question

This is a PCA question. I am reading about the mathematics behind PCA on this website. I understand that Y = XA is the matrix notation of the transformation of the original variables to the principal components where X is the feature vector and rows of the A matrix represent the eigenvectors and within each row we have the loading.

Then using the A matrix and the Sx matrix (the var-covar matrix of the original data), we can derive the var-covar matrix of the PC.

i.e.

I am not sure what linear algebra method this is called and it derives the var-covar matrix of the PCs, which is called Sy and the elements in the diagonal of this matrix are the eigenvalues and that is the variance explained by each principal component. If that is the case, we will expect the very first element of the matrix to be the variance of the first component which should be the largest.

If this is not the way, then how do we calculate the variance-covariance matrix of the principal components and what does this matrix tell us?

Answer 1

Yes. Unrotated PCs are orthogonal, so Sy is diagonal, with declining variances on the diagonal.

Answer 2

Are you familiar with the eigendecomposition of a square matrix? If not, you should start there.

Going forward from there, we first notice that our data matrix $X$ is not square. To get a square matrix we can simply multiply by it by its own transpose: $X^{\top}X$. This square matrix gives us one other nice thing, it is the covariance of the features in $X$ (assuming features are along the rows). Now call this matrix $S_X$, i.e. $S_X=X^{\top}X$. Principal components are now sought which are each orthogonal to each other. Some more linear algebra: if we notice that a $X^{\top}X$ is a real, symmetric matrix, we can know something nice about its eigendecomposition. The eigenvalues are real-valued and the eigenvectors are orthogonal. Thus, the eigenvectors can intuitively provide a change of basis to represent our data as a diagonal matrix, i.e. $S_Y = V^{-1}S_XV$ where the variances in $S_Y$ tell us how much variance in the data that each principal component in the columns of $V$ capture.

I wrote this hastily. Let me know if something isn't clear/needs explaining.