Why is sum of squares equal to eigenvalue in PCA? We fit a line or a hyperplane to a set of points. We project the points onto the hyperplane. The sum of squared distances of the projected points to origin is equal to the eigenvalue. Why is that?
 A: Because that eigenvalue $\lambda_i$ is the variance of a zero-centred variable $\xi_i$, $\xi_i$ being the principal component score for the $i$ eigen-component. By definition the variance of a variable $X$ is $\text{Var}(X) = E[(X-E[X])^2]$. If $E[X]=0$, then $\text{Var}(X) = E[(X)^2]$ which is just the scaled sum of squares.
A: Let's presume, without loss of generality, that the data is centered, i.e. the mean is equal to zero.
First point:
Since the dataset $X = \{\mathbf{x}_i\}_{i=1}^n$ is centered, we have:
$$
n\sigma_k^2 = n \cdot \frac{1}{n}\sum_{i=1}^n x_{ik}^2 = \sum_{i=1}^n x_{ik}^2,
$$
where $x_{ik}$ is the $k$-th coefficient of the $i$-th data point. Now consider some unit vector $\mathbf{v}$. The sum of squared distances of the data points in the direction of $\mathbf{v}$ is:
$$
\sum_{i=1}^n \langle \mathbf{v}, \mathbf{x}_i \rangle^2 = \sum_{i=1}^n \sum_{k=1}^d v_k^2 x_{ik}^2 = \sum_{k=1}^d v_k^2 \sum_{i=1}^n x_{ik}^2 = \sum_{k=1}^d v_k^2 n\sigma_k^2 = n\langle \mathbf{v}, \Sigma \mathbf{v} \rangle,
$$
where $\langle \cdot, \cdot \rangle$ is the inner product.
Furthermore, for an $m$-dimensional linear subspace $V$ spanned by the orthonormal vectors $(\mathbf{v}_1, \ldots, \mathbf{v}_m)$, the squared distance of the projection of some $\mathbf{x}_i$ to this subspace is $\sum_{r=1}^m \langle\mathbf{v}_r, \mathbf{x}_i\rangle^2$, so we can generally state: The sum of squared norms of the projections of the dataset $X$ to some subspace $V$, denoted by $SSP_V(X)$, is:
$$
SSP_V(X) = \sum_{i=1}^n \sum_{r=1}^m \langle\mathbf{v}_r, \mathbf{x}_i\rangle^2 = n\sum_{r=1}^m \langle\mathbf{v}_r, \Sigma \mathbf{v}_r\rangle.
$$
Second point:
PCA finds the $V$ that maximizes $SSP_V(X)$. But this is a quadratic optimization problem with quadratic constraints (because the $\mathbf{v_i}$ are unit vectors), which can be solved e.g. by using Lagrange multipliers. The result is, that the linear subspace $V$ is the one spanned by the eigenvectors $(\omega_r)_{r=1}^m$ of $\Sigma$ that belong to the $m$ largest eigenvalues. And, if you enter an orthonormal selection of those into the expression of $SSP_V(X)$ above, you obtain:
$$
\begin{align}
SSP_V(X) &= n\sum_{r=1}^m \langle\mathbf{\omega}_r, \Sigma \mathbf{\omega}_r\rangle\\
         &= n\sum_{r=1}^m \langle\mathbf{\omega}_r, \lambda_r \mathbf{\omega}_r\rangle\\
         &= n\sum_{r=1}^m \lambda_r.
\end{align}
$$
Thus, the sum of squared distances of the projected points to the origin is equal to the size $n$ of the dataset $X$ times the sum of the first $m$ eigenvalues of $\Sigma$.
