# Relevant Dimension Estimation: Showing that $\sum_{i=1}^N s_i^2 = ||y||^2$

In relevant dimension estimation, we are given a Kernel Matrix $$K \in \mathbb{R}^{n \times n}$$, where $$K_{ij} = k(x_i, x_j)$$. We then compute the kernel eigenvector from the multiple solutions of the eigenvalue Problem: $$K \mu_i = \lambda_i \mu_i$$ (where $$\lambda_1 \geq \lambda_n \geq 0$$ are the sorted eigenvalues and $$\mu_i$$ is the eigenvector associated wih $$\lambda_i$$). Considering for each data point $$x_i$$ a label $$y_i \in \mathbb{R}$$, we can define the RDE coefficients: $$\forall_{i=1}^N s_i = \mu_i^T \textbf{y}$$ where $$\textbf{y} =(y_i)_{i=1}^N$$

I now need to show that: $$\sum_{i=1}^N s_i^2 = ||\textbf{y}||^2$$

Unfortunately I have no idea how to show this. My guess is that I would reorder the sums and then at some point use that $$\mu_i^2 = 1$$ or $$<\mu_i, \mu_j> = 0$$ for $$i \neq j$$. How exactly do I arrive at the desired result?

• When one refers to "the" eigenvector, it is presumed those vectors have been (a) normalized to unit length and (b) made orthogonal. The result is then immediate from (a) and (b). – whuber Jul 5 '19 at 13:33
• I am having trouble understanding why the result follows from (a) and (b). Could you elaborate? – CryptoThomas Jul 5 '19 at 16:02
• The linear transformations that preserve Euclidean distance are the orthogonal ones. The matrix representation of an orthogonal transformation consists of a collection of orthonormal vectors. That's precisely what the eigenvectors are, QED. – whuber Jul 5 '19 at 16:19

To write out whuber's answer in the comments in a little more detail: $$\sum_i s_i^2 = \sum_i (\mu_i^T y)^2 = \sum_i y^T \mu_i \mu_i^T y = y^T \left( \sum_i \mu_i \mu_i^T \right) y = y^T y = \lVert y \rVert^2 .$$ The key bit here is that $$M := \sum_i \mu_i \mu_i^T$$ is just the identity matrix. One way to see this is because if $$Q$$ is the matrix with the eigenvectors $$\mu_i$$ as its columns, $$\sum_i \mu_i \mu_i^T = Q Q^T$$; the matrix of orthonormal eigenvectors for a real symmetric matrix is always orthogonal, $$Q Q^T = Q^T Q = I$$, because its columns are an orthonormal basis for $$\mathbb R^n$$.