7
$\begingroup$

Spectral Decomposition

Let $\mathbf{A}$ be a $k\times k$ positive definite matrix with the spectral decomposition $\mathbf{A}=\sum_{i=1}^{k}\lambda_{i}\mathbf{e}_{i}\mathbf{e}_{i}^{\prime}$. Let the normalized eigenvectors be the columns of another matrix $\mathbf{P}=\begin{bmatrix}\mathbf{e}_{1}, & \mathbf{e}_{2}, & \ldots, & \mathbf{e}_{k}\end{bmatrix}$. Then

$ \mathbf{A}=\sum_{i=1}^{k}\lambda_{i}\mathbf{e}_{i}\mathbf{e}_{i}^{\prime}=\mathbf{P}\Lambda\mathbf{P}^{\prime} $

where $\mathbf{P}\mathbf{P}^{\prime}=\mathbf{P}^{\prime}\mathbf{P}=\mathbf{I}$ and $\Lambda$ is the diagonal matrix

$ \Lambda=\begin{bmatrix}\lambda_{1} & 0 & \ldots & 0\\ 0 & \lambda_{2} & \ldots & 0\\ \vdots & \vdots & \ddots & \vdots\\ 0 & 0 & \ldots & \lambda_{k} \end{bmatrix}\textrm{ with }\lambda_{i}>0. $

R Code

A <- matrix(data=c(1, 0, 1, 3), nrow=2, ncol=2, byrow=TRUE)
eigen(A)
eigen(A)$vectors %*% diag(eigen(A)$values) %*% t(eigen(A)$vectors)

Output

$values
[1] 3 1

$vectors
     [,1]       [,2]
[1,]    0  0.8944272
[2,]    1 -0.4472136

     [,1] [,2]
[1,]  0.8 -0.4
[2,] -0.4  3.2

I don't know what I'm missing here. I'm not able to prove $\mathbf{A}=\sum_{i=1}^{k}\lambda_{i}\mathbf{e}_{i}\mathbf{e}_{i}^{\prime}=\mathbf{P}\Lambda\mathbf{P}^{\prime} $ with $ \mathbf{A}=\begin{bmatrix}1 & 0\\ 1 & 3 \end{bmatrix}$. I also got the same results with hand calculations. I'd highly appreciate if you guide me what I'm missing here. Thanks in advance for your help and time.

$\endgroup$
  • $\begingroup$ i am somewhat new hear but if I am learning correctly this problem sounds like it might fall into the homework problem category and hence be designated as such. Also although you use the key word multivariate analysis and computational statistics and these result could be applied to some of problems in these fields it really looks like a question more appropriate for the mathematics site. $\endgroup$ – Michael R. Chernick May 12 '12 at 21:21
  • $\begingroup$ @MichaelChernick: Thanks for your notice and comment. This is not a homework problem. $\endgroup$ – MYaseen208 May 12 '12 at 21:26
  • 5
    $\begingroup$ Hint: What is the definition of symmetric? $\endgroup$ – cardinal May 12 '12 at 21:30
  • $\begingroup$ Thanks @cardinal for your comment. As I understood from your comment only symmetric matrices would be positive definite. Am I right? If so please change your comment to answer so that I can accept it for future users. Thanks $\endgroup$ – MYaseen208 May 12 '12 at 21:37
  • 1
    $\begingroup$ @MichaelChernick: A symmetric matrix need not be positive semidefinite either. Just think of a diagonal matrix with at least one negative entry on the diagonal. $\endgroup$ – cardinal May 12 '12 at 22:46
12
$\begingroup$

In the particular example in the question, the properties of a symmetric matrix have been confused with those of a positive definite one, which explains the discrepancies noted.

A brief tour of symmetry and positive semidefiniteness

Symmetric positive (semi)definite matrices play an important role in statistical theory and applications, making it useful to briefly explore some of their properties and from whence they arise. The proofs can be made reasonably short, making it feasible to do this exploration here.

A word on notation. Let $\newcommand{\reals}{\mathbb R}A$ be an $n \times n$ matrix with real-valued entries. Often, this is denoted $A \in M_n(\reals)$. In what follows, $\lambda$ will be an eigenvalue of $A$ and $v$ will be a corresponding unit eigenvector, i.e., $A v = \lambda v$ and $\|v\|_2 = 1$. Even though $A$ is real-valued, $\lambda$ and $v$ both might be complex-valued, which is an important point to keep in mind. The notation $v^T$ denotes the transpose of $v$ if it is real-valued and $v^*$ denotes the conjugate transpose. The conjugate of $z \in \mathbb C$ is denoted $\bar z$.

Spectral decomposition and symmetry

For this part, we'll assume $A$ is symmetric, that is, $A = A^T$.

Theorem 1 The eigenvalues of $A$ are real.
Proof: We have $$ \lambda = \lambda v^* v = v^* A v = \sum_i a_{ii} v_i \bar v_i + \sum_{i < j} a_{ij} (v_i \bar v_j + \bar v_i v_j) \>,$$ where the last equality follows from the fact that $a_{ij} = a_{ji}$. Note that all of the terms in each sum on the right-hand side are real. Hence, $\lambda$ is real.

The result above gives our first hint at what any potential eigendecomposition must look like. To get a further hint, consider the following.

Theorem 2 The eigenvectors corresponding to distinct eigenvalues are orthogonal.
Proof: Let $\lambda_1 \neq \lambda_2$ with corresponding $v_1$ and $v_2$. Then $$ \lambda_1 v_1^* v_2 = v_1^* A^* v_2 = v_1^* A v_2 = \lambda_2 v_1^* v_2 \>, $$ so, $(\lambda_1 - \lambda_2) v_1^* v_2 = 0$ and since $\lambda_1 \neq \lambda_2$, we must have $v_1^* v_2 = 0$.

So now we know that distinct eigenvalues yield orthogonal eigenvectors. But, we still don't know that the eigenvectors themselves are (i.e., can be chosen to be) real-valued.

Theorem 3 The eigenvectors of each eigenvalue can be chosen to be real-valued.
Proof: Decompose $v = u + i w$ where $u$ and $w$ are real-valued vectors. Then, $$ \lambda u + i \lambda w = \lambda v = A v = Au + i A w \>, $$ and $\lambda$ and $A$ are both real-valued, hence $A u = \lambda u$ and $A w = \lambda w$. This shows that if $v$ is an eigenvector of $A$, so is $u$ (and $w$, for that matter).

So, we've got some real eigenvalues and some real eigenvectors that are orthogonal whenever the eigenvalues are distinct. Now we handle the case where a particular eigenvalue has multiple linearly independent eigenvectors.

Theorem 4 The (real) eigenvectors of a common eigenvalue form a vector subspace of $\reals^n$.
Proof: If $A v_1 = \lambda v_1$ and $A v_2 = \lambda v_2$, then for any $\alpha$ and $\beta$, we have $$ A (\alpha v_1 + \beta v_2) = \alpha A v_1 + \beta A v_2 = \lambda (\alpha v_1 + \beta v_2) \>, $$ and so the set of vectors $\mathcal V = \{v: A v = \lambda v\}$ is a linear subspace.

Since every finite-dimensional real-valued linear subspace of $\reals^n$ has an orthogonal basis, this is enough to conclude that any symmetric $A$ satisfies $A V = V \Lambda$ where $V$ is an orthogonal matrix and $\Lambda$ is a real-valued diagonal matrix. Note that if it happens that there are less than $n$ nonzero eigenvalues, that we can "fill out" the columns of $V$ with an orthogonal basis of the remaining subspace and place zeros along the diagonal of $\Lambda$ in the corresponding locations.

Hence, there is some orthogonal $V$ and some real-valued diagonal $\Lambda$ such that $$ A = V \Lambda V^T \>. $$ We can also see that the converse is trivially true. If $A = V \Lambda V^T$, then $A^T = (V \Lambda V^T)^T = V \Lambda V^T = A$ and so $A$ is symmetric.

But, what about positive semidefiniteness?

So far, we've said nothing about positive definiteness; all of the properties we have derived deal purely with symmetry. Now, we'll try to briefly develop the properties of positive definite matrices and make some connections to symmetry.

A matrix $A$ (not necessarily symmetric!) is called positive semidefinite if for all $x \in \reals^n$, we have $x^T A x \geq 0$. If $x \neq 0$ further implies that $x^T A x> 0$, then $A$ is called positive definite. This arises frequently in statistics in the study of quadratic forms.

Theorem 5 Let $A \in M_n(\reals)$. Then, there exists a symmetric matrix $B$ such that $x^T A x = x^T B x$ for all $x \in \reals^n$.
Proof: $ x^T A x = (x^T A x)^T = x^T A^T x$ so choose $B = \frac{1}{2}(A + A^T)$.

Notice that there is nothing in the statement of the theorem about $A$ being positive semidefinite; it's completely general. What this says, though, is that when considering quadratic forms, we can always implicitly assume that $A$ is symmetric. This motivates the fact that oftentimes authors will assert that $A$ is symmetric immediately in the definition of positive semidefiniteness.

Theorem 6 A symmetric positive semidefinite matrix $A$ has nonnegative eigenvalues.
Proof: If $v$ is a unit eigenvector corresponding to $\lambda$, then $$ \lambda = \lambda v^T v = v^T A v \geq 0 \>. $$

As an exercise, you should prove that if $A$ is symmetric positive definite, then all eigenvalues must be strictly positive.

A counterexample

While the above shows that we can conclude that a symmetric $A$ having real positive eigenvalues is positive definite, this does not hold if we drop the symmetry requirement. Let $$ A = \left(\begin{array}{rr}1 & -4 \\ 0 & 1 \end{array}\right) \>. $$ Then $A$ has an eigenvalue of 1 (with algebraic multiplicity of 2), but by taking $x$ to be a vector of ones we see that $A$ is not positive semidefinite. The motivation for this example can be found in this question and answer.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.