In principal component analysis (PCA), what is the covariance matrix in the new space of reduced dimensionality, i.e. the covariance matrix after multiplying the data by the eigenvector matrix?
I know it should be a diagonal matrix because they are orthogonal, but what is the formula to calculate the variances (or the whole covariance matrix)?
For this argument I am going to assume that your variables follow the multivariate normal distribution, although this assumption is not required for the extraction of the principal components. Assume then that
$$\mathbf{X} \sim \mathcal N_{m} \left( \boldsymbol{\mu}, \boldsymbol{\Sigma} \right)$$
For ease of notation, I am also going to stick to the population level. The ideas are readily extended to the sample level, though.
The way I like to think about PCA is that we project the data onto orthogonal directions of maximum variability in order to achieve some dimensionality reduction. It can be shown that these directions are given by the eigenvectors of $\mathbf{\Sigma}$, so let's stack all these eigenvectors in a matrix $\mathbf{\Gamma}$
$$\boldsymbol\Gamma = \begin{bmatrix} \mathbf{e}_1 | \mathbf{e}_2| \ldots |\mathbf{e}_m \end{bmatrix}$$
By the orthonormality of the eigenvectors, this is an orthogonal matrix, i.e. $\mathbf{\Gamma}^{\prime} \mathbf{\Gamma} = I_m$. And now let's project our (centered) data onto these directions. Define
$$\mathbf{Z} = \boldsymbol{\Gamma}^{\prime} \left( \mathbf{X} - \boldsymbol{\mu} \right)$$
The transformation is linear so $\mathbf{Z}$ is still normally distributed. My question to you is, what is the mean and covariance matrix of $\mathbf{Z}$? Surely, since we have centered the data the mean of the orthogonal transformation is zero, no problems there. As for the covariance matrix, recall the spectral decomposition of $\boldsymbol{\Sigma}$
$$\boldsymbol{\Sigma} = \boldsymbol{\Gamma} \boldsymbol{\Lambda} \boldsymbol{\Gamma}^{\prime}$$
where $\Lambda$ is a diagonal matrix with the eivenvalues in the diagonal. Now, by the sandwhich theorem, the covariance of $\mathbf{Z}$ is given by
$$\operatorname{Cov}(\mathbf{Z}) = \boldsymbol{\Gamma}^{\prime} \boldsymbol{\Sigma} \boldsymbol{\Gamma} = \boldsymbol{\Gamma}^{\prime} \boldsymbol{\Gamma} \boldsymbol{\Lambda} \boldsymbol{\Gamma}^{\prime} \boldsymbol{\Gamma} = \boldsymbol{\Lambda} $$
the diagonal matrix with the eigenvalues and these eigenvalues are exactly the variances of these linear combinations. Further, since the off-diagonal elements are zero, the variables are independent, as we had hoped. I believe this answers your question then.
As a final remark, note that nowhere above did we really need the normality. Without it, you just cannot assert that the linear combinations are independent and you have to say that they are uncorrelated. This is not very important though. Indeed, you can apply the spectral decomposition and the sandwhich theorem regardless of whether your data is normally distributed. In the case of normality, however, we have the neat interpretation of the principal components as axes of the ellipsoid of constant density (recall the exponent of the multivariate normal distribution).
