If all the elements of a positive-definite covariance matrix are positive, how can I prove that the coefficients [elements] of the first principal component [first eigenvector] are all of the same sign, and the coefficients [elements] of all other principal components [eigenvectors] cannot all be of the same sign?

This is Problem 11.11 from "Multivariate Statistical Analysis" by Anderson, 3rd ed. I have tried to prove it for several days...

  • 1
    $\begingroup$ Hint (for the edited version): For a covariance matrix $\mathbb{A}$, the first principal component $\mathbb{x}$ maximizes the norm of $\mathbb{Ax}$ over all unit vectors $\mathbb{x}$. Suppose two components of $\mathbb{x}$ had different signs: could you then demonstrate that $\mathbb{A x}$ cannot have maximal norm? (This is where you're likely to need the assumption of positive-definiteness of $\mathbb{A}$.) $\endgroup$ – whuber Oct 5 '12 at 1:18
  • $\begingroup$ Yap I was thinking abt that before. As from geometric view if and only if they have the same sign, it will give the maximum magnitude hence maximize the variation. But by saying that is that enough for the proof? I try to find some theory to support my thought but there is no result... $\endgroup$ – Grapy Oct 5 '12 at 1:24
  • $\begingroup$ I see that you have added a new information that was not in the question earlier: "If all of the coefficients of the matrix are positive". Then the result is true, see my earlier comment below. $\endgroup$ – kjetil b halvorsen Oct 5 '12 at 1:34

With your new information, that all the components of the positive-definite matrix are positive, it becomes easy. While it follows directly from the Perron-Frobenius theorem (which is valid for square matrices with non-negative elements, symmetric or not), in the symmetric case it is much easier.

Let the positive-definite matrix be $S$. The eigenvector corresponding to the largest eigenvector is the vector $x$ obtaining the maximum in the following problem: $$ \lambda_{\mathrm{max}} = \mathrm{max}_{\{x \colon \| x\|=1\}} x^T S x $$(that is, the "argmax") where $\lambda_{\text{max}}$ is the largest eigenvalue.

Suppose to get a contradiction that $x_1$ is negative, while the other components of $x$ are non-negative. We can write $$ x^T S x = x_1 S_{11} x_1+2x_1 \sum_{j=2}^m s_{1j} x_j + \sum_{i=2}^m \sum_{j=2}^m x_i s_{ij} x_j $$ Note that the first and third terms are positive while the second term is negative, and we can get a strictly larger value by switching the sign of $x_1$, which respects the restriction on norm. That gives the contradiction you need. A similar argument can be written for any other pattern of negative/positive sign.

  • $\begingroup$ This is the right idea but the demonstration is incomplete--the "similar argument" hides a messy detail. Suppose $\mathbf{x}$ has multiple positive and negative signs. How do you choose which component to switch, and will you be switching a negative to a positive or positive to a negative value? Remember, whenever $\mathbf{x}$ is a solution, so is $-\mathbf{x}$! $\endgroup$ – whuber Oct 5 '12 at 3:21
  • $\begingroup$ I this kjetil is talking abt sign of x1 only...coz you can see x1 as a component of vector x. So when u switch you actually switch the sign of x1 only. $\endgroup$ – Grapy Oct 5 '12 at 3:32

If the signs of the coefficients are all of the same sign it indicates that they are all measuring the first component in the same direction. If you reverse one of the variables (e.g. by multiplying it by -1) you ought get something different. The signs of the first component do not all need to be positive; indeed, when I run this example from R prcomp

prcomp(USArrests, scale = TRUE)

all the signs are negative (but that could be different when you run it)

If I then run

prcomp(~ Murder + Assault + Rape, data = USArrests, scale = TRUE)

all the signs are again negative on PC1 but if I modify it:

rapeinv <- Rape*-1
prcomp(~ Murder + Assault + rapeinv, scale = TRUE)

then two signs are negative but the one for rapeinv is positive.

Often, in practice (as here) the first component is capturing something general about the data (here, crime rate) so it is frequently the case that the first PC will have all its signs the same, but it is not necessarily the case.

  • $\begingroup$ Hmmm...but at this stage, the thing is i have to prove that the coefficient of the first principal component are all of the same sign and coefficients of each other pricipal component cannot be all of the same sign... $\endgroup$ – Grapy Oct 3 '12 at 15:10
  • 1
    $\begingroup$ It is not the case that the coefficients of the first PC have to be of the same sign, as my example showed, so you can't prove it, at least, not without some additional conditions. Nor is it the case that the coefficients of the other PCs cannot be of the same sign. $\endgroup$ – Peter Flom Oct 3 '12 at 15:12
  • 2
    $\begingroup$ But it is the case that if one of the principal components have coefs all of the same sign, then this cannot be true for any of the other principal components. 'Maybe thats what you asked to prove? (hint: If two principal components both have all coefs the same sign, how can their interior producyt be zero? $\endgroup$ – kjetil b halvorsen Oct 3 '12 at 15:26
  • 1
    $\begingroup$ There are also some cases where what the OP asks for will happen: If ALL the correlations are positive, then you can apply the Perron-Frobenius theorem and get the result. $\endgroup$ – kjetil b halvorsen Oct 3 '12 at 15:27
  • $\begingroup$ This is a question from a book called an introduction of multivariate statistical analysis 3ed by Anderson. Problem 11.11. The only condition stated there is all the elements in the covariance matrix are positive then ask u to prove the coefficient... I can't really figured out...please help... $\endgroup$ – Grapy Oct 3 '12 at 22:10

The solutions to the two questions follow in a straightforward manner from the definitions, but some care is needed in the analysis. I offer this post to fill in some gaps in the previous ones, to make the solution self-contained (without relying on any advanced or specialized theorems), and to provide a solution to the second question, which so far has not been offered.

Let $\mathbb A$ be the covariance matrix, of dimensions $n$ by $n$. It is symmetric and positive-definite by assumption. Therefore (these are standard results in the study of such matrices) there exists a basis of $n$ nonzero eigenvectors $E=(e_1, e_2, \ldots, e_n)$ for which

  1. The $e_i$ all have real (not merely complex) coefficients;

  2. $\mathbb A e_i = \lambda_i$ for non-negative real (not merely complex) numbers $\lambda_i$, the eigenvalues;

  3. We may therefore order the eigenvectors so that $\lambda_1 \ge \lambda_2 \ge \cdots \ge \lambda_n \gt 0$;

  4. The eigenvectors are mutually orthogonal: $e_i^\prime e_j = 0$ whenever $i\ne j$; and

  5. We may normalize the eigenvectors (by dividing each one by $\sqrt{e_i^\prime e_i}$ if necessary) to make them all of unit length.

These are the basic facts worth remembering, because they (greatly) simplify our understanding and analysis of such matrices, which are ubiquitous in statistical theory and practice. The remainder of this post exploits these properties to address the two questions.

Because $E$ is a basis, any arbitrary vector $x$ has a unique expansion as a linear combination of eigenvectors,

$$x = x_1 e_1 + x_2 e_2 + \cdots x_n e_n$$

for real numbers $x_1, x_2, \ldots, x_n$ determined by $x$. Facts (4) and (5) let us calculate that

$$|x|^2 = x_1^2 + x_2^2 + \cdots + x_n^2$$

and property (2) implies

$$|\mathbb A x|^2 = \lambda_1^2 x_1^2 + \lambda_2^2 x_2^2 + \cdots + \lambda_n^2 x_n^2.$$

It is clear--and is an easily proven elementary inequality--that when $|x|^2=1$, the latter is maximized when $x_j=0$ for all $j$ where $\lambda_j \lt \lambda_1$. (Provided $\lambda_1$ is unique-- that is, $\lambda_i \lt \lambda_1$ for $i=2, 3, \ldots, n$--there are exactly two solutions: $x_1=\pm 1$ (and $x_i=0$ for $i=2, 3, \ldots, n$), whence $x = \pm e_1$.)

The first question concerns the original coordinates in which the matrix and the eigenvectors were originally written. Writing $e_1 = (e_{11}, e_{12}, \ldots, e_{1n})$ in those coordinates, suppose there exist indexes $j$ for which $e_{1j}\lt 0$. Let $f$ be the vector obtained by negating all such $e_{1j}$. Because $e_{1j}^2 = (-e_{1j})^2$, this does not change the norm, whence $|f|=1$ (by fact (5)). However, this process increases $|\mathbb A e|$ because--by assumption--multiplication by $\mathbb A$ consists of taking linear combinations with positive coefficients and the change from $e_1$ to $f$ has actually turned what were subtractions of positive values into additions of positive values. Since $|\mathbb A e|$ was maximal, we conclude that $|\mathbb A f|$ is maximal and $f$ is an eigenvector with eigenvalue $\lambda_1$. We may therefore take $e_1$ to be $\pm f$, but either way all its components will have the same sign.

As an (important) aside, note that all the components of $e_1$ must be positive: none can be zero. This is because (a) $e_1$ is nonzero, whence it has at least one nonzero component and (b) in computing the product $\mathbb A e_1 = \lambda_1 e_1$ (fact (2)) all the products being added up are sums of nonnegative numbers and at least one (obtained from a nonzero component of $e_1$) is nonzero. That shows all the components of $\lambda_1 e_1$ are nonzero, but since $\lambda_1 \gt 0$ (fact (3)), all components of $e_1$ must be nonzero, too.

The second question asserts that the remaining eigenvectors, $e_2, e_3, \ldots, e_n$, must have some negative components when written in the original basis. Consider one of them, say $e_j$ and write it as $e_j = (e_{j1}, e_{j2}, \ldots, e_{jn})$ in the original basis. Then from fact (4) $e_j$ is orthogonal to $e_1$:

$$0 = e_1^\prime e_j = e_{11}e_{j1} + e_{12}e_{j2} + \cdots + e_{1n}e_{jn}.$$

Since--as we showed in the aside--all the $e_{1i} \gt 0$, the only way this linear combination can equal zero is for at least one $e_{ji} \lt 0$.

A more delicate version of these results can be obtained when the components of $\mathbb A$ are merely assumed to be nonnegative. What changes is that the first principal component may have zeros (as originally expressed) and some of the other principal components may also have entirely nonnegative entries, too. As an example, take $\mathbb A$ to be the $2\times 2$ identity matrix and let the first two principal components (which are not unique!) be $e_1=(1,0)$ and $e_2=(0,1)$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.