3
$\begingroup$

I found a very nice PCA proof over here PCA_proof and I'm trying to understand it (I don't know what Langrange multipliers are so I'm trying my best).

From the second page of the previous link, finding the first principal component $a_1$ comes to studying :

$L(a_1)={a_1}^T \Sigma a_1 - \lambda ({a_1}^T a_1 -1)$

So correct me please if I'm wrong

  • Acoording to Wikipedia, studying the Langrangian $L(x,\lambda)=f(x) - \lambda (g(x) -c)$ is useful when we want to maximize a function, say $f(x)$ when the variables are constrained by a certain condition, say $g(x)=c$.

  • In the first equation, ${a_1}^T \Sigma a_1$ is the $f(x)$ and it says : "I want the first principal component to contain the maximum possible of the data variance " and the second part represents $g(x)$ as ${a_1}^T a_1=1$ ($c$ is the $1$)

So here come the questions :

  • Am-I understanding this correctly ?
  • How does the ${a_1}^T \Sigma a_1$ translate the variance projection on the first component $a_1$ ? I know how to project a vector on an axis using a scalar product but I have no idea how It's done with matrices.

Thanks in advance for all your answers.

$\endgroup$
2
  • 3
    $\begingroup$ The answer to your first question is yes (and you do realize that $a_1^\top a_1 = 1$ means that the vector $a_1$ has length $1$, right?). To answer your second question, denote the centered data matrix $X$ (rows are samples, columns are variables). Then projection of the data onto the vector $a_1$ is given by $Xa_1$. The variance of this projection is given by $$\frac{1}{n-1}(Xa_1)^\top (Xa_1) = \frac{1}{n-1}a_1^\top X^\top X a_1 = a_1^\top \left(\frac{X^\top X}{n-1}\right)a_1 = a_1^\top \Sigma a_1,$$ QED. $\endgroup$
    – amoeba
    Jan 24, 2015 at 23:41
  • $\begingroup$ Doesn't the relation $\operatorname{Var}(a)=a^\prime\Sigma a$ for all vectors $a$ define the covariance matrix $\Sigma$? $\endgroup$
    – whuber
    Apr 21, 2023 at 15:31

1 Answer 1

0
$\begingroup$

I am reproducing @amoeba's comment as an answer:

The answer to your first question is yes (and you do realize that $a_1^\top a_1 = 1$ means that the vector $a_1$ has length $1$, right?). To answer your second question, denote the centered data matrix $X$ (rows are samples, columns are variables). Then projection of the data onto the vector $a_1$ is given by $Xa_1$. The variance of this projection is given by $$ \begin{align}\frac{1}{n-1}(Xa_1)^\top (Xa_1) &= \frac{1}{n-1}a_1^\top X^\top X a_1\\ &= a_1^\top \left(\frac{X^\top X}{n-1}\right)a_1 \\ &= a_1^\top \Sigma a_1, \end{align}$$ QED.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.