1
$\begingroup$

For a Multivariate Gaussian Distribution The formula of Multivariate Normal Distribution The geometric properties

I am a bit confused about the derivation of sigma Derivation of sigma

The deduction of sigma from Christopher Bishop's book- Pattern recognition and machine learning

I am confused that I still cannot get the details of the derivation of sigma.

Could anyone help me with the process from having known $\Sigma u_i=\lambda_i u_i$ and $u_i $is orthonormal, to the result of $\Sigma=\sum^D_i \lambda_i u_i u_i^T$?

$\endgroup$
3
$\begingroup$

From $\Sigma u_i = \lambda_i u_i$, we can get by multiply $u_i^T$ on the right sides $$ \Sigma u_i u_i^T = \lambda_i u_iu_i^T, \quad i=1, \ldots, D. $$ Now, let us sum up $D$ equations above, which leads to $$\Sigma \sum_i u_i u_i^T = \sum_i \lambda_i u_iu_i^T.$$ Let $U=[u_1, \ldots, u_D]$, a column-binded matrix from $\{u_i\}_{i=1}^D$. If $u_i$'s are orthonormal (i.e. $\sum_i u_i u_i^T=U U^T = I$), we now conclude $$\sum_i \lambda_i u_iu_i^T = \Sigma \sum_i u_i u_i^T=\Sigma.$$

As whuber suggested, the eigenvectors should satisfy orthonormality, not just orthogonality. OP would be better to check this condition.

Hope this helps.

$\endgroup$
9
  • $\begingroup$ +1 -- but your argument is a little incomplete. Orthonormality of the $u_i$ does not (in itself) imply $UU^\prime = I:$ you also need that there are $D$ of the $u_i.$ I realize this is implicit in your argument, but making it explicit may assist the reader in understanding the key ideas. $\endgroup$
    – whuber
    Jan 31 '20 at 23:23
  • $\begingroup$ @whuber Agreed. I made an edit based on your suggestion. Thanks! $\endgroup$
    – inmybrain
    Feb 1 '20 at 3:23
  • $\begingroup$ @inmybrain Thank you for your prompt and useful answer. I think ui(i=1,...,D) is the elements of the vector U. Thus, the transpose of ui is equal to ui,because ui itself is a scalor not a vector. Could you please tell me whether my thoughts is correct? U is the mean so it is a D*1 vector, and vector x as observation, is also a D*1 vector, the sigma covariance matrix is a D*D matrix, is that right? $\endgroup$
    – Cici Yang
    Feb 1 '20 at 15:05
  • $\begingroup$ @CiciYang I'm afraid to say it is not. $u_i$ is a $D$-dimensional vector, as $x$ is. $\Sigma$ is a $D \times D$ matrix. $\endgroup$
    – inmybrain
    Feb 1 '20 at 23:20
  • $\begingroup$ @@inmybrain Can you please explain why (sum uiui(transpose)=UU(transpose)? While using matrix multiplication I can get UU(transpose)=I, I still can not get why sum uiui(transpose)=UU(transpose) exists. I assume that U is a D*D matrix with D column and observations. Is that right? Is there an intuitive way to explain why the sum of the uiui(T)=UU(T)? Thank you. $\endgroup$
    – Cici Yang
    Feb 2 '20 at 6:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.