What is this projection matrix doing?

Question

Let’s say we have a $m\times d$ zero mean multivariate Gaussian matrix $X$. Its covariance matrix is $X^{T}X$. Let $V$ be the $d\times d$ matrix of eigenvectors of $X^{T}X$, with the columns sorted in descending order of eigenvalues.

Let’s say we know that the last $k$ eigenvectors correspond to noise. We zero out the first $d-k$ eigenvectors, call this matrix $W$, then construct a projection matrix $P = W^{T}W$.

My question is, what exactly is $P$ doing? If we want to analyse only the noise component of $X$, why don’t we just multiply $X$ by $W$? Why is the projection matrix necessary?

Answer 1

Multiplying $X$ by $W$ gives you transformed data. Multiplying $X$ by $P$ gives you $X$ again ($P$ is the identity matrix because the eigenvectors are orthogonal).

Observe that $$ \text{Var}\left[W^T X\right] = W^T\text{Var}\left[X\right]W \approx W^T\frac{1}{n}X^TXW $$

You're writing the spectral decomposition as

$$ X^TX = V \begin{bmatrix} \lambda_1 & \cdots & 0\\ \vdots &\ddots & \vdots \\ 0 & \cdots & \lambda_d \end{bmatrix} V^T $$ which implies that $$ V^TX^TX V = V^TV \begin{bmatrix} \lambda_1 & \cdots & 0\\ \vdots &\ddots & \vdots \\ 0 & \cdots & \lambda_d \end{bmatrix} V^TV = \begin{bmatrix} \lambda_1 & \cdots & 0\\ \vdots &\ddots & \vdots \\ 0 & \cdots & \lambda_d \end{bmatrix} $$ so $$ \text{Var}\left[W^T X\right] \approx n^{-1} \begin{bmatrix} \lambda_1 & \cdots & 0\\ \vdots &\ddots & \vdots \\ 0 & \cdots & \lambda_{d-k} \end{bmatrix}. $$