I'm stuck on something in the derivation of the Principal Components.
We have random vectors $X$ of dimension $p$. We want to find linear combinations of $X$, $a'X,$ where $a \in \mathbb{R}^{p}$ that satisfy the constraints:

(1) $\max_{a_{1}\in\mathbb{R}^{p}} \big(Var(a_1' X) \big) $ subject to $a_1'a_1 =1$

(2) $\max_{a_{2}\in\mathbb{R}^{p}} \big(Var(a_2' X )\big)$ subject to $a_2'a_2 =1$ and $Cov(a_1'X, a_2'X) = 0$

and so on.

Letting the covariance matrix of vector $X$ be denoted $\Sigma$, the proof proceeds to satisfy the first constraint by recognizing

$$\max_{a \ne 0} \big(a'\Sigma a \big) = \lambda_1 $$,

where $\lambda_1$ is the largest eigenvalue of $\Sigma$. Since this maximum is known to be achieved by the eigenvector $e_1$ associated with $\lambda_1$, this gives us $a_1 = e_1$. Fine, I understand that. It allows us to find a (unit-length) vector that satisfies the first equation.

But, then it proceeds like this:

$$ max_{a \perp e_1}\big( a'\Sigma a \big) = \cdots$$

Why are we introducing that perpendicular sign? How do we know that the space of all $\{a : a \perp e_1 \}$ is the largest possible space that can provide a maximum variance while satisfying the "uncorrelated with first principal component" constraint? Is there some relationship between orthogonal coeficcients to a linear combination and correlation? I have spent an hour or two failing to derive any such relationship.

(edit based on answer: I don't think there's a relationship between covariance and orthogonality of the coefficients in general, but here $a_1$ is an eigenvector, which gives such a relationship )

  • $\begingroup$ Intuitively, you want to diagonalize your covariance matrix (which is symmetric, hence the Spectral decomposition we sometimes see), such that the off-diagonals (where the covariance lives) are 0. This occurs when two vectors are perpendicular, or orthogonal. $\endgroup$
    – ilanman
    Oct 5 '16 at 22:21

$$\mbox{Cov}(a'_1X,a'_2X)=E[a'_1XX'a_2]=a'_1\Sigma a_2=(\Sigma a_1)'a_2,$$

where we used $\Sigma'=\Sigma$. So if $a_1=e_1$, then,

$$\mbox{Cov}(a'_1X,a'_2X)=(\lambda_1a_1)'a_2=\lambda a_1'a_2,$$

since $a_i$ is deterministic. It follows that if you want the covariance to be zero, $a_1,a_2$ must be orthogonal.

  • $\begingroup$ First, thanks a lot! I understand now. Second, I think you have too many expectation operators. I think $E(a_1'XX'A_2) = a_1'\Sigma a_2$, and so on. Third, for future readers, notice that Alex R assumes that X has mean zero, which is allowed WLOG since $cov(X-\mu) = cov(X)$. Hence any argument can be done on centered variables $\endgroup$
    – RMurphy
    Oct 6 '16 at 0:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.