# Why is computation of scores in Sparse PCA different from T=XP?

I am recently learning Sparse PCA. From a lately published paper All sparse PCA models are wrong, but some are useful. Part I: Computation of scores, residuals and explained variance I learned that

since the loadings of sPCA are not orthogonal, we should use the correction formula below to compute the scores, where the superscript '+' indicates the Moore-Penrose inverse:

$$\mathbf{T}=\mathbf{X}\mathbf{P}\left(\mathbf{P}^\text{T}\mathbf{P}\right)^+$$

Actually I cannot figure out the reason behind it. As far as I know previously, $$\mathbf{P}$$ contains coefficient vectors which can construct Principal Component directly, so $$\mathbf{T}=\mathbf{X}\mathbf{P}$$ already stands for the scores.

I've tried to turn to Google, but didn't find any related topics. Could you please tell me why we should add $$\left(\mathbf{P}^\text{T}\mathbf{P}\right)^+$$ to the right side of the formula just because the loadings are not orthogonal?

• I didn't read the article, and my presupposition is as follows. $P$ (which you incorrectly call loadings and would better called eigenvectors) is the matrix of cosines of rotation. In classic PCA, it is an orthogonal rotation matrix ($P'P=I$), so it turns cartesian coordinates ($X$) into cartesian coordinates ($T$); frame axes remaining orthogonal. – ttnphns Mar 11 '20 at 11:42
• In your case though, $P'P \neq I$ and it rotates cartesian coordinates ($X$) into perpedicular coordinates $T$ of the oblique axes frame (see pic here) - where $P'P$ is the matrix of angles (cosines) between the axes. To convert perpendicular coordinates into the corresponding skew ones in the oblique frame, you have to postmultiply $T$ by $(P'P)^{-1}$, or, in general case if $P'P$ is singular, $P'P^+$. Skew coordinates express scores of different components as "independent" of each other. (This was just my quick guess, I can be wrong.) – ttnphns Mar 11 '20 at 11:42
• Thanks a lot for your kindly help! I've checked your theory with a simple example and what you said is right. To make sure I truly understand it - – Terence Mar 12 '20 at 10:36
• adding $(P^TP)^+$ to the right transforms $P$ from the structure loading to the pattern loading? (here I use the term that appears in the graph here) – Terence Mar 12 '20 at 10:39
• Pattern and structure are argor from factor analysis. Skew and perpendicular coordinates are the simple terms I am using in general setting. Math's argot for these are covariant and contravariant coordinates (or components). – ttnphns Mar 12 '20 at 11:36

$$P^TP$$ Is the inner product of the coefficient vectors (PCs, loadings ...) and returns their covariance matrix. This calculates the non-orthogonality of the coefficient vectors .
$$(P^TP)^+$$
Here's some graphs from supporting information for a paper I have submitted that illustrate this. The data is a simulated dataset that has had no noise added. The first shows the correlation between equally ranked NIPALS and SVD against PC rank in the orange dots. You can see that initially the correlation between the two algorithms is high, but from 19 on it drops dramatically. However, vector identity is not an arbitrary rank label, but its shape and orientation, so we should match by closest correlation. We see that even with this correction significant divergence emerges. So next I show how the rank of PC (identity based on correlation to SVD PCs, not rank) changes between the two algorithms. At first the ranks increase in tandem but become more disrupted. This is due to the accumulation of computation errors. Of particular note is the blip at rank 19 in the NIPALs model. This matches rank 1 in SVD, despite rank 1 in NIPALS already matching that! Here we now compare PC 1 and 19 in the NIPALS algorithms - you can see they are indeed very similar apart from PC19 having significant contribution from computational noise. finally now we compare the PC1 between the two algorithms - the difference is tiny (scale is $$10^{-14}$$ compared with $$10^{-1}$$ for the actual PCs), but as we accumulate successive small errors they grow unless renormalised.