Deriving PCA solution using infinitesimals

Question

The problem of PCA basically boils down to finding the extremum of $$\psi(q) = q^TRq \tag 1$$ subject to constraint $$q^Tq = 1 \tag 2.$$

How I would go about solving it, is by differentiating the Lagrangian i.e. solving

$$\frac{d(q^TRq - {\lambda}(q^Tq - 1))}{dq} = 0 $$

which leads to $$Rq - {\lambda}q = 0$$

which is the eigenvalue problem to the covariance matrix $R$ as it should have appeared and the things are right into place.

But then I watched this lecture and the professor goes about with the first principle kind of way and I think that he messes it up big time but manages to get to the right answer. Here goes what he does:

At extremum:

$${\psi}(q+{\delta}q) = {\psi}(q)$$ $$(q+{\delta}q)^TR(q+{\delta}q) = q^TRq$$ $$q^TRq +{\delta}q^TRq + {\delta}q^TR{\delta}q + q^TR{\delta}q = q^TRq$$ removing ${\delta}q^TR{\delta}q$ as insignificant and cancelling $q^TRq$ from RHS $${\delta}q^TRq + q^TR{\delta}q = 0$$ Now as R is a covariance matrix and hence symmetric and for a symmetric matrix we have $a^TRb = b^TRa$ hence we get $${\delta}q^TRq = 0 \tag 3$$ and using eq 2 we also have $$((q+{\delta}q)^T(q+{\delta}q)) = 1$$ eliminating ${\delta}q^T{\delta}q$ and substituting $q^Tq = 1$ $${\delta}q^Tq = 0. \tag 4$$

Till now everything is okay. But now the professor does something which I cant seem to square with. He proceeds as follows:

At 41:30 he claims - combining eq3 and eq 4 we get: $${\delta}q^TRq - {\lambda}({\delta}q^Tq) = 0 \tag 5$$ which i don't think is a healthy way to combine the two equations as it introduces new solutions which do not satisfy the eq4 and eq5 but we should be okay as long as we keep that fact in mind (Any comments on this are welcome). Now he proceeds as: $${\delta}q^T(Rq - {\lambda}q) = 0 $$ and now the bummer, he claims that since ${\delta}q \neq 0$ we have $$Rq - {\lambda}q = 0 $$ thus taking us to the right final eigenvalue problem. Which is non-sensical. ${\delta}q$ could very well be orthogonal to $Rq - {\lambda}q$.

Please confirm if I am right in my criticism.

Instead, how could one have gone about solving it? Eq3 and eq4 tell me that ${\delta}q$ is orthogonal to the plane containing $q$ and $Rq$. How to move ahead and eliminate all the solutions but the ones where $q$ is an eigenvector of $R$ i.e. to reconcile this to the actual solution and collapse it to just containing $q$ such that $Rq = {\lambda}q$?

Answer 1

Here is a slightly different way to carry out this argument. Consider equations (3) and (4) together: $$\delta q ^\top R q = 0 \\ \delta q ^\top q = 0$$

Eq. (4) says that $\delta q$ is orthogonal to $q$; indeed, this makes intuitive sense: $q$ is constrained to have unit length and so lies on the surface of the hyper-sphere. If $q$ is on its surface already, then only the infinitesimal movements in the orthogonal directions will remain on the surface.

Eq. (3) says that $\delta q$ is orthogonal to $Rq$. Note that infinitesimal vector $\delta q$ can be arbitrary, as long as satisfies the above constraint of being orthogonal to $q$. For any such $\delta q$ it must be that it is also orthogonal to $Rq$. The one and only way this can be true, is if $Rq$ is parallel to $q$.

Which makes $q$ an eigenvector of $R$ by definition.