27
$\begingroup$

I have a set of data that has $n$ samples described by $m$ variables. I do a PCA to reduce it to just 2 dimensions so I can make a nice 2D plot of the data. I understand that the $x,y$ coordinates (i.e., the PCA scores) for the plot are calculated by basically summing the products of the original data (after centering) by the loadings for each variable, so:

$$\mathrm{PC}_1 = X_1L_1 + X_2L_2 + ... + X_mL_m.$$

My question is, if I pick an arbitrary point in the PCA space (i.e. a value for $\mathrm{PC}_1$ and $\mathrm{PC}_2$, or $x$ and $y$ in my plot), is there a convenient way to translate that back to a set of the original values (i.e., $X_1,X_2,\dots,X_m$)?

Note 100% reversal is obviously not expected (since I'm only using 2 PCs), so a decent approximation is fine.

| cite | improve this question | |
$\endgroup$
  • 4
    $\begingroup$ Read the first chapter of Michael Greenacre's Biplots in Practice. Then read the rest of it when you understand that much! It is in essence a demonstration of how to do the type of interpretation from a biplot that you are asking for. $\endgroup$ – Andy W Aug 20 '12 at 18:03
  • 2
    $\begingroup$ Hi Matt. Some variants of this question are repeatedly being asked on our forum and I have eventually decided to try to write a "canonical" thread answering it: stats.stackexchange.com/questions/229092. We closed a whole bunch of older questions as duplicates. Now I see that your question from 2012 is almost the same, but unfortunately the accepted answer here is very brief and would leave many readers puzzled (that's partly why I wrote my own answer). The question is, how to consolidate our two threads. [cont.] $\endgroup$ – amoeba Aug 12 '16 at 22:35
  • 1
    $\begingroup$ [cont.] We can close this one as a duplicate of mine, because the latter thread is more comprehensive. What do you think? Or perhaps I should've posted my answer here; maybe it can still be moved somehow. I did not want to vote to close before discussing it with you, as your thread has a respected four-years-long history. Cheers. $\endgroup$ – amoeba Aug 12 '16 at 22:36
  • 2
    $\begingroup$ @amoeba: I'm completely agnostic about it. Whatever you think would be most helpful for the community. $\endgroup$ – Matt Burland Aug 13 '16 at 1:33
29
$\begingroup$

Yes. Basically, what you did was to do: $$\mathrm{PC}=\mathbf{V}X,$$ where $\mathrm{PC}$ are the principal components, $X$ is your matrix with the data (centered, and with data points in columns) and $\mathbf{V}$ is the matrix with the loadings (the matrix with the eigenvectors of the sample covariance matrix of $X$). Therefore, you can do: $$\mathbf{V}^{-1}\cdot\mathrm{PC}=X,$$ but, because the matrix of loadings is orthonormal (they are eigenvectors!), then $\mathbf{V}^{-1}=\mathbf{V}^{T}$, so: $$\mathbf{V}^T\cdot\mathrm{PC}=X.$$ Note that this gives you exactly the same equation you cite for the recovery of the PCs, but now for the data, and you can retain as many PCS as you like.

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Ok, but does that apply for any arbitrary point? i.e. a point that wasn't in the original dataset? $\endgroup$ – Matt Burland Aug 20 '12 at 19:25
  • 4
    $\begingroup$ Yes, but it can only be used for interpolation. Suppose datapoint $x_{ab}$ is missing and that the data matrix is centered. The problem is to find $\mathbf{V}$. You can estimate the sample covariance matrix as: $$\hat{\Sigma}_{ij}=\alpha_{ij}\sum_{k}x_{ik} x_{jk},$$ where $$\alpha_{ij}=\left(N-1-\sum_{k}[\delta_{ab}(i,k)+\delta_{ab}(j,k)]\right)^{-1},$$ and $\delta_{ab}(i,k)$ is one when $i=a$ and $k=b$ and zero otherwise. Given this, the transformation matrix $\mathbf{V}$ is straightforward to obtain (you just have to estimate the SVD decomposition of this sample covariance matrix). $\endgroup$ – Néstor Aug 20 '12 at 20:22
  • $\begingroup$ Well, by theory $V^TV$ should be identity matrix, but in practice, it is quite off. The off-diagonal is ~0.0005 and some of the diagonal entries are as small as 0.5. May I ask why? $\endgroup$ – Sibbs Gambling Apr 23 '15 at 14:20
  • $\begingroup$ That is strange. Typical algorithms have maximum errors on the order of ~1e-16 (see mathematica.stackexchange.com/questions/46283/…); maybe you should change the method by which you are obtaining the eigenvectors and eigenvalues? $\endgroup$ – Néstor Apr 26 '15 at 18:39
2
$\begingroup$

I have a doubt about the above answers. Since after dimension reduction, we only know 2 principal components, and the rest principal components are abandoned. The projection matrix V is not a square matrix (not completely orthonormal, it is a semi-orthogonal matrix). Suppose n is the number of samples and m is the number of variables. $X$ is a $m$-by-$n$ matrix, $V$ is a 2-by-m matrix (whose rows are the top 2 eigenvectors of the covariance matrix of $X$), PC is a 2-by-n matrix. Then we have PC = VX. Then $VV^T$ is an identity matrix, but $V^TV$ is not. Thus $V^TPC=V^TVX$ cannot give us the exact original matrix $X$, since $V^TV$ is not an identity matrix.

| cite | improve this answer | |
$\endgroup$
  • 3
    $\begingroup$ That's because there is no way to perfectly reconstruct $X$ from $V$ if you kept only $2<m$ principal components. One can, however, do a best possible reconstruction; it is given by $V^\top VX$. It's not equal to $X$. $\endgroup$ – amoeba Mar 23 '16 at 20:34

Not the answer you're looking for? Browse other questions tagged or ask your own question.