Suppose that a two-dimensional random variable $X$ has a covariance matrix given by

$$ \Sigma = \pmatrix {1 & -2\\ -2 & 4}$$

One of the three linear combinations below corresponds to the first principal component. Without performing an eigen-decomposition of the covariance matrix, identify which linear combination corresponds to the first principal component, and justify your answer.

Linear combination #1: $-0.447 X1 + 0.894 X2$

Linear combination #2: $-0.894 X1 - 0.447 X2$

Linear combination #3: $1.789 X1 + 1.789 X2$

I think I can safely eliminate #3 because the square of the coefficients needs to sum to 1. Both #1 and #2 have this property.

As the covariance between the two variables is negative, does this mean that the coefficients would have opposite signs i.e. #1?


  • $\begingroup$ I think that your logic is correct; you actually answered your question. Your first statement (sum to 1) is actually unnecessary: it is true if the "coefficients" are defined as eigenvector elements and may not be true otherwise. The actual answer is in your last statement: if the covariance is negative, i.e. the variables as vectors has >90 degrees angle between them from geometric point of view, then the 1st PC which is the line along the maximal joint variance should reflect that fact, by taking on coordinates of different sign. $\endgroup$ – ttnphns Oct 5 '14 at 9:26
  • $\begingroup$ ...One can observe also that the 3rd linear combination is wrong because its coefficients are of equal magnitude. It is not possible since variances are unequal X2 is 4 times more variable and hence must be "stronger" in defining PC1. $\endgroup$ – ttnphns Oct 5 '14 at 9:50

As the determinant is zero we have only one factor. The variables are negatively correlated so their loadings must have different sign. So without checking the actual numbers, solution #1 is the only candidate. (The latter two arguments are equal to yours. The first argument is helpful here as we have a case, in which a further argument, that the alternating loadings are always in the first principal factor, is not needed. Such an argument needed reference to the eigendecomposition which should be avoided by the original question)

[update]: The sentence below was my first reasoning which implied an unnecessesary caution; I leave it here for reference to the first comment:

  The first argument is needed because if we had two factors it could possibly 
  happen that the first factor/first pc has equal signs on both variables and
  the second alternating signs)               

The first argument is fine here, but the arguing for its importance is wrong.
It should go like this: having two items only the pc-factorization guarantees, that on each factor the squares of the item's loadings are equal, so their covariance is $r = a \cdot -a + b \cdot b$ with loadings of absolute values $a$ and $b$. To have negative correlation $-r$ by this sum, it is required that $a \gt b$ But then also $a^2+a^2 \gt b^2+b^2$ and thus the factor containing $a, -a$ is the first principal axis.
Unfortunately, this argument uses implicitely the knowledge of the eigenvalue-decomposition but which should be avoided by the requirements of the question....

  • 1
    $\begingroup$ +1, but... if we had two factors it could possibly happen that the first factor/first pc has equal signs on both variables makes me doubt. Can you show a 2x2 negative covariance matrix with both variables having the same sign in the first eigenvector? $\endgroup$ – ttnphns Oct 5 '14 at 8:41
  • $\begingroup$ You're correct; I can't find such a matrix. So this argument should be replaced by one which shows why such a case cannot happen. (I should have done the actual factorization before claiming) $\endgroup$ – Gottfried Helms Oct 5 '14 at 8:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.