Let $X$ be $n \times p$ with rank$(X)=k$, $\epsilon \sim N(0,I_p)$ be a vector with i.i.d. Show $E\|\ X\epsilon \|_2^2 = \sum_{j=1}^k \sigma_j^2$

Let $X$ be an $n \times p$ matrix with rank$(X)=k$. Let $\epsilon \sim N(0,I_p)$ be a vector with the i.i.d. Gaussian components.

(a) Show that $$E\|\ X\epsilon \|_2^2 = \|\ X \|_F^2 = \sum_{j=1}^k \sigma_j^2$$ where $\sigma_1 \geq \sigma_2 \geq \dots \geq \sigma_k > 0$ are positive signular values of $X$ and $\|\ X \|_F$ is the Frobenius norm of $X$.

(b) Show that if $X$ is a projection matrix, i.e. $X=A(A^TA)^{-1}A^T$ for some matrix $A : n \times p$ with rank$(A)=p$, then $$E \|\ X\epsilon \|_2^2 = \sigma^2p.$$

By the singular value decomposition theorem we know that $X$ can be decomposed in the form: $$X=U\Sigma V^H,$$ where $U$ is an $n \times n$ unitary matrix, $\Sigma$ is a diagonal $n \times p$ matrix with non-negative real numbers on the diagonal, and $V$ is an $p\times p$ unitary matrix, and $V^H$ denotes the conjugate transpose of $V$. The diagonal entries $\sigma_i$ of $\Sigma$ are known as the singular values of $M$, in other words $\Sigma$ has $\left\{\sigma_1, \dots, \sigma_k\right\}$ on the diagonal, as rank$(X)=k$. Now, we have that from definition of Frobenius norm, \begin{align*} \|\ X \|_F^2 &= \operatorname{Tr}(XX^H) \\ &= \operatorname{Tr}((U\Sigma V^H)(U\Sigma V^H)^H) \\ &= \operatorname{Tr}(U\Sigma V^H V \Sigma^H U^H) \\ &= \operatorname{Tr}(U\Sigma\Sigma^H U^H) \\ &= \operatorname{Tr}(\Sigma\Sigma^H)\\ &= \sum_{j=1}^k \sigma_j^2. \end{align*} since $V^HV=I$ and $U^HU=I$, where we also utilized the cyclic property of trace. All that remains to be shown is that $E\|\ X\epsilon \|_2^2 = \sum_{j=1}^k \sigma_j^2$. I am not sure about this part of (a) and, hoping that once I figure that part out of (a), I'll have a better idea of (b). Any help would be appreciated!

• Have you considered just writing out $||X\epsilon||^2$? It is the sum of $Y_{ij}\epsilon_i\epsilon_j$ where $Y_{ij}$ is the $ij$ entry of $X^\prime X.$ Taking expectations reduces everything to (very simple) algebraic properties of $Y.$ If you know them you're done and otherwise you have some exercises in linear algebra. – whuber Sep 18 '18 at 21:48
• @whuber Is my approach correct? – Maxtron Sep 18 '18 at 22:55

I think you have almost solved it. $\mathbb{E}( ||X\epsilon||^2_2 ) = \mathbb{E}[ \mathrm{Tr} ( X \epsilon \epsilon^H X^H) ] = \mathrm{Tr} [ \mathbb{E} ( X \epsilon \epsilon^H X^H)]$ since trace and expectation are interchangeable.