so the problem of PCA basically boils down to finding the extremum of $$\psi(q) = q^TRq \tag 1$$ subject to constraint $$q^Tq = 1 \tag 2$$

How i would go about solving it is by differentiating the Lagrangian i.e. solving

$$\frac{d(q^TRq - {\lambda}(q^Tq - 1))}{dq} = 0 $$

which leads to $$Rq - {\lambda}q = 0$$

which is the eigenvalue problem to the covariance matrix $R$ as it should have appeared and the things are right into place. But then i watched this lecture and the professor goes about with the first principle kind of way and i think that he messes it up big time but manages to get to the right answer. here goes what he does:

at extremum:

$${\psi}(q+{\delta}q) = {\psi}(q)$$ $$(q+{\delta}q)^TR(q+{\delta}q) = q^TRq$$ $$q^TRq +{\delta}q^TRq + {\delta}q^TR{\delta}q + q^TR{\delta}q = q^TRq$$ removing ${\delta}q^TR{\delta}q$ as insignificant and cancelling $q^TRq$ from RHS $${\delta}q^TRq + q^TR{\delta}q = 0$$ Now as R is a covariance matrix and hence symmetric and for a symmetric matrix we have $a^TRb = b^TRa$ hence we get $${\delta}q^TRq = 0 \tag 3$$ and using eq 2 we also have $$((q+{\delta}q)^T(q+{\delta}q)) = 1$$ eliminating ${\delta}q^T{\delta}q$ and substituting $q^Tq = 1$ $${\delta}q^Tq = 0 \tag 4$$

till now everything is okay. But now the professor does something which i cant seem to square with. He proceeds as follows:

at 41:30 he claims - combining eq3 and eq 4 we get: $${\delta}q^TRq - {\lambda}({\delta}q^Tq) = 0 \tag 5$$ which i dont think is a healthy way to combine the two equations as it introduces new solutions which do not satisfy the eq4 and eq5 but we should be okay as long as we keep that fact in mind (Any comments on this are welcome). now he proceeds as: $${\delta}q^T(Rq - {\lambda}q) = 0 $$ and now the bummer, he claims that since ${\delta}q \neq 0$ we have $$Rq - {\lambda}q = 0 $$ thus taking us to the right final eigen value problem. which is non-sensical. ${\delta}q$ could very well be orthogonal to $Rq - {\lambda}q$

Please confirm if i am right in my criticism

Instead, How one could have gone about solving it is:

eq3 and eq4 tell me that ${\delta}q$ is orthogonal to the plane containing $q$ and $Rq$

 .  .  .

question: How to move ahead and eliminate all the solutions but the ones where $q$ is an eigenvector of $R$ i.e Reconcile this to the actual solution and collapse it to just containing $q$ such that $Rq = {\lambda}q$

The problem of PCA basically boils down to finding the extremum of $$\psi(q) = q^TRq \tag 1$$ subject to constraint $$q^Tq = 1 \tag 2.$$

How I would go about solving it, is by differentiating the Lagrangian i.e. solving

$$\frac{d(q^TRq - {\lambda}(q^Tq - 1))}{dq} = 0 $$

which leads to $$Rq - {\lambda}q = 0$$

which is the eigenvalue problem to the covariance matrix $R$ as it should have appeared and the things are right into place. 

But then I watched this lecture and the professor goes about with the first principle kind of way and I think that he messes it up big time but manages to get to the right answer. Here goes what he does:

At extremum:

$${\psi}(q+{\delta}q) = {\psi}(q)$$ $$(q+{\delta}q)^TR(q+{\delta}q) = q^TRq$$ $$q^TRq +{\delta}q^TRq + {\delta}q^TR{\delta}q + q^TR{\delta}q = q^TRq$$ removing ${\delta}q^TR{\delta}q$ as insignificant and cancelling $q^TRq$ from RHS $${\delta}q^TRq + q^TR{\delta}q = 0$$ Now as R is a covariance matrix and hence symmetric and for a symmetric matrix we have $a^TRb = b^TRa$ hence we get $${\delta}q^TRq = 0 \tag 3$$ and using eq 2 we also have $$((q+{\delta}q)^T(q+{\delta}q)) = 1$$ eliminating ${\delta}q^T{\delta}q$ and substituting $q^Tq = 1$ $${\delta}q^Tq = 0. \tag 4$$

Till now everything is okay. But now the professor does something which I cant seem to square with. He proceeds as follows:

At 41:30 he claims - combining eq3 and eq 4 we get: $${\delta}q^TRq - {\lambda}({\delta}q^Tq) = 0 \tag 5$$ which i don't think is a healthy way to combine the two equations as it introduces new solutions which do not satisfy the eq4 and eq5 but we should be okay as long as we keep that fact in mind (Any comments on this are welcome). Now he proceeds as: $${\delta}q^T(Rq - {\lambda}q) = 0 $$ and now the bummer, he claims that since ${\delta}q \neq 0$ we have $$Rq - {\lambda}q = 0 $$ thus taking us to the right final eigenvalue problem. Which is non-sensical. ${\delta}q$ could very well be orthogonal to $Rq - {\lambda}q$.

Please confirm if I am right in my criticism.

Instead, how could one have gone about solving it? Eq3 and eq4 tell me that ${\delta}q$ is orthogonal to the plane containing $q$ and $Rq$. How to move ahead and eliminate all the solutions but the ones where $q$ is an eigenvector of $R$ i.e. to reconcile this to the actual solution and collapse it to just containing $q$ such that $Rq = {\lambda}q$?