Using the SVD on matrix X (column as features)

We have X = U\sigmaV* where V contains the PCs and U\sigma would be the transformed data

PCA with linear regression

The application of PCA with linear regression is called Principal component regression (PCR), in which we use PCs as new predictors. With gene expression data, those PCs would be called eigengenes which could represent major expression patterns of genes with similar expression patterns across samples.(PCR has its own problem that PCs with low variation could have a significant effect on the response)

My question is: Why don't we use U\sigma, the transformed data as predictors? In what situations, could we use transformed data as predictors? What do those predictors mean?

  • $\begingroup$ U instead of V contains the PCs for X'X $\endgroup$ – purod Nov 11 '19 at 3:25

If X is n by p matrix, then the absolutely standard interpetation is that one has n observations of p variables. Therefore $\Sigma V^*$ is also a nxp matrix, that is what one has done is created new variables that are linear combinations of the previous variables. That makes good sense. If I have two variables x,y, then $x+y$ and $x-y$ also make sense as variables. What you suggest $U\Sigma$, is taking linear combinations of the observations. One can do it, but for what purpose. The point of SVD is to find a new basis for the set of variables one is observing that is 1) Orthogonal and 2) listed in 'order of importance' as measured by the diagonal entries of $\Sigma$. The usual reason for doing so is to throw away variables that are less important. In many situations it may turn out to be the case that a small number of variables in the new basis capture most of the information in the data as measured by variance. Two examples of this come to mind right away. mp3 encoding of music is exactly that. Variables are frequencies measured and the observations are the values at various points in time. Similarly if one represents a picture (black and white for simplicity), the the picture can be thought of a matrix where each pixel is a matrix entry between 0 and 1 where 0 = white and 1 = black. Then pca gives one a way to capture the picture with less information. Google is your friend for more details on either topic. Oh and a third. If one has time series data of various stocks, the doing a pca will almost always produce a situation in which the first principle component is about 60%-80% of the variability and 3-5 components will capture 95% or more of the variance. Incidentally the first component is easily seen to be market direction.

  • $\begingroup$ When I did the PCA, I though V would be the PC matrix. When X is a matrix with n samples and m features, U should be the PC matrix. I forgot to check meaning of rows and columns. $\endgroup$ – purod Nov 11 '19 at 3:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.