2
$\begingroup$

Let $X$ be a $p$-dimensional random vector with $p$ principal components $y_1, y_2, \dots, y_p$. By definition, a restriction put on the second principal component $y_2 = a_2'X$ is

$$ \text{cov}(y_1, y_2) = \text{cov}(a_1' X, a_2' X) = 0 $$

It is also known that $a_i$ are simply eigenvectors, so we also have

$$ \forall i \ne j, a_i \cdot a_j = 0 $$

I wonder if it's a coincidence in the context of PCA. In other words, does $\text{cov}(a_1' X, a_2' X) = 0$ imply $a_1 \cdot a_2 = 0$? I tried to prove it, but didn't get very far

$$ \begin{equation} \begin{aligned} 0 &= \text{cov}(a_1' X, a_2' X) \\ &= \text{cov}(\sum_{k=1}^p a_{1,k} X_k, \sum_{m=1}^p a_{2,m} X_m) \\ &= \sum_{k=1}^p a_{1,k} \sum_{m=1}^p a_{2,m} \text{cov}(X_k, X_m) \end{aligned} \end{equation} $$

Can I get $a_1 \cdot a_2 = 0$ from here?

$\endgroup$
  • 1
    $\begingroup$ Geometrically, you are asking whether two vectors remain orthogonal after an arbitrary linear transformation. In light of this it's easy to find counterexamples in just $p=2$ dimensions. In fact, you are almost certain to find a counterexample just by randomly selecting any $2\times 2$ covariance matrix. $\endgroup$ – whuber May 12 at 16:08
4
$\begingroup$

Consider a random walk model, $X_t = X_{t-1} + Z_t$ where $Z_t \overset{iid}{\sim} \text{Normal}(0,1)$. Also, assume that $X_1 = Z_1$ and has variance $1$. Take the random vector $X = (X_1,X_2,X_3)'$, it has a covariance matrix $$ \Sigma = \left[\begin{array}{ccc} 1 & 1 & 1 \\ 1 & 2 & 2 \\ 1 & 2 & 3 \end{array}\right]. $$ Now take $a_1 = (-1,1,0)'$ and $a_2 = (0,-1,1)$. You can see that $$ \text{Cov}(a_1'X, a_2'X) = \text{cov}(Z_2, Z_3) = 0 $$ but $a_1'a_2 = -1$.

Edit: just for completeness, consider the example that your mind might go to first. Consider a vector $X$ such that its covariance matrix is proportional to the identity matrix (i.e. $\sigma^2 \mathbf{I}$ with $\sigma^2 >0$). Then $\text{Cov}(a_1'X, a_2'X) = \sigma^2 a_1' a_2$ is zero if and only if $a_1' a_2 = 0$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.