I found a very nice PCA proof over here PCA_proof and I'm trying to understand it (I don't know what Langrange multipliers are so I'm trying my best).

From the second page of the previous link, finding the first principal component $a_1$ comes to studying :

$L(a_1)={a_1}^T \Sigma a_1 - \lambda ({a_1}^T a_1 -1)$

So correct me please if I'm wrong

  • Acoording to Wikipedia, studying the Langrangian $L(x,\lambda)=f(x) - \lambda (g(x) -c)$ is useful when we want to maximize a function, say $f(x)$ when the variables are constrained by a certain condition, say $g(x)=c$.

  • In the first equation, ${a_1}^T \Sigma a_1$ is the $f(x)$ and it says : "I want the first principal component to contain the maximum possible of the data variance " and the second part represents $g(x)$ as ${a_1}^T a_1=1$ ($c$ is the $1$)

So here come the (may be stupid) questions :

  • Am-I understanding this correctly ?
  • How does the ${a_1}^T \Sigma a_1$ translate the variance projection on the first component $a_1$ ? I know how to project a vector on an axis using a scalar product but I have no idea how It's done with matrices.

Thanks in advance for all your answers.

  • 3
    $\begingroup$ The answer to your first question is yes (and you do realize that $a_1^\top a_1 = 1$ means that the vector $a_1$ has length $1$, right?). To answer your second question, denote the centered data matrix $X$ (rows are samples, columns are variables). Then projection of the data onto the vector $a_1$ is given by $Xa_1$. The variance of this projection is given by $$\frac{1}{n-1}(Xa_1)^\top (Xa_1) = \frac{1}{n-1}a_1^\top X^\top X a_1 = a_1^\top \left(\frac{X^\top X}{n-1}\right)a_1 = a_1^\top \Sigma a_1,$$ QED. $\endgroup$ – amoeba Jan 24 '15 at 23:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.