Let $X$ be an $n \times p$ matrix with rank$(X)=k$. Let $\epsilon \sim N(0,I_p)$ be a vector with the i.i.d. Gaussian components.

(a) Show that $$E\|\ X\epsilon \|_2^2 = \|\ X \|_F^2 = \sum_{j=1}^k \sigma_j^2$$ where $\sigma_1 \geq \sigma_2 \geq \dots \geq \sigma_k > 0$ are positive signular values of $X$ and $\|\ X \|_F$ is the Frobenius norm of $X$.

(b) Show that if $X$ is a projection matrix, i.e. $X=A(A^TA)^{-1}A^T$ for some matrix $A : n \times p$ with rank$(A)=p$, then $$E \|\ X\epsilon \|_2^2 = \sigma^2p.$$

By the singular value decomposition theorem we know that $X$ can be decomposed in the form: $$X=U\Sigma V^H,$$ where $U$ is an $n \times n$ unitary matrix, $\Sigma$ is a diagonal $n \times p$ matrix with non-negative real numbers on the diagonal, and $V$ is an $p\times p$ unitary matrix, and $V^H$ denotes the conjugate transpose of $V$. The diagonal entries $\sigma_i$ of $\Sigma$ are known as the singular values of $M$, in other words $\Sigma$ has $\left\{\sigma_1, \dots, \sigma_k\right\}$ on the diagonal, as rank$(X)=k$. Now, we have that from definition of Frobenius norm, \begin{align*} \|\ X \|_F^2 &= \operatorname{Tr}(XX^H) \\ &= \operatorname{Tr}((U\Sigma V^H)(U\Sigma V^H)^H) \\ &= \operatorname{Tr}(U\Sigma V^H V \Sigma^H U^H) \\ &= \operatorname{Tr}(U\Sigma\Sigma^H U^H) \\ &= \operatorname{Tr}(\Sigma\Sigma^H)\\ &= \sum_{j=1}^k \sigma_j^2. \end{align*} since $V^HV=I$ and $U^HU=I$, where we also utilized the cyclic property of trace. All that remains to be shown is that $E\|\ X\epsilon \|_2^2 = \sum_{j=1}^k \sigma_j^2$. I am not sure about this part of (a) and, hoping that once I figure that part out of (a), I'll have a better idea of (b). Any help would be appreciated!

  • $\begingroup$ Have you considered just writing out $||X\epsilon||^2$? It is the sum of $Y_{ij}\epsilon_i\epsilon_j$ where $Y_{ij}$ is the $ij$ entry of $X^\prime X.$ Taking expectations reduces everything to (very simple) algebraic properties of $Y.$ If you know them you're done and otherwise you have some exercises in linear algebra. $\endgroup$ – whuber Sep 18 '18 at 21:48
  • $\begingroup$ @whuber Is my approach correct? $\endgroup$ – Maxtron Sep 18 '18 at 22:55

I think you have almost solved it. $\mathbb{E}( ||X\epsilon||^2_2 ) = \mathbb{E}[ \mathrm{Tr} ( X \epsilon \epsilon^H X^H) ] = \mathrm{Tr} [ \mathbb{E} ( X \epsilon \epsilon^H X^H)]$ since trace and expectation are interchangeable.

\begin{align} \mathrm{Tr} [ \mathbb{E} ( X \epsilon \epsilon^H X^H)] = \mathrm{Tr}[ X \mathbb{E} \{\epsilon \epsilon^H \} X^H ] = \mathrm{Tr} [X I_p X^H] = \sum_{j=1}^k \sigma_j^2 \end{align}

  • 1
    $\begingroup$ I was not aware that trace and expectation are commutative operators. Thank you! $\endgroup$ – Dragonite Sep 19 '18 at 11:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.