# PCA: Cannot Understand one part of Derivation of Principal Comonents

I'm stuck on something in the derivation of the Principal Components.
We have random vectors $X$ of dimension $p$. We want to find linear combinations of $X$, $a'X,$ where $a \in \mathbb{R}^{p}$ that satisfy the constraints:

(1) $\max_{a_{1}\in\mathbb{R}^{p}} \big(Var(a_1' X) \big)$ subject to $a_1'a_1 =1$

(2) $\max_{a_{2}\in\mathbb{R}^{p}} \big(Var(a_2' X )\big)$ subject to $a_2'a_2 =1$ and $Cov(a_1'X, a_2'X) = 0$

and so on.

Letting the covariance matrix of vector $X$ be denoted $\Sigma$, the proof proceeds to satisfy the first constraint by recognizing

$$\max_{a \ne 0} \big(a'\Sigma a \big) = \lambda_1$$,

where $\lambda_1$ is the largest eigenvalue of $\Sigma$. Since this maximum is known to be achieved by the eigenvector $e_1$ associated with $\lambda_1$, this gives us $a_1 = e_1$. Fine, I understand that. It allows us to find a (unit-length) vector that satisfies the first equation.

But, then it proceeds like this:

$$max_{a \perp e_1}\big( a'\Sigma a \big) = \cdots$$

Why are we introducing that perpendicular sign? How do we know that the space of all $\{a : a \perp e_1 \}$ is the largest possible space that can provide a maximum variance while satisfying the "uncorrelated with first principal component" constraint? Is there some relationship between orthogonal coeficcients to a linear combination and correlation? I have spent an hour or two failing to derive any such relationship.

(edit based on answer: I don't think there's a relationship between covariance and orthogonality of the coefficients in general, but here $a_1$ is an eigenvector, which gives such a relationship )

• Intuitively, you want to diagonalize your covariance matrix (which is symmetric, hence the Spectral decomposition we sometimes see), such that the off-diagonals (where the covariance lives) are 0. This occurs when two vectors are perpendicular, or orthogonal. Oct 5 '16 at 22:21

$$\mbox{Cov}(a'_1X,a'_2X)=E[a'_1XX'a_2]=a'_1\Sigma a_2=(\Sigma a_1)'a_2,$$
where we used $\Sigma'=\Sigma$. So if $a_1=e_1$, then,
$$\mbox{Cov}(a'_1X,a'_2X)=(\lambda_1a_1)'a_2=\lambda a_1'a_2,$$
since $a_i$ is deterministic. It follows that if you want the covariance to be zero, $a_1,a_2$ must be orthogonal.
• First, thanks a lot! I understand now. Second, I think you have too many expectation operators. I think $E(a_1'XX'A_2) = a_1'\Sigma a_2$, and so on. Third, for future readers, notice that Alex R assumes that X has mean zero, which is allowed WLOG since $cov(X-\mu) = cov(X)$. Hence any argument can be done on centered variables Oct 6 '16 at 0:12