2
$\begingroup$

I have a large data matrix that I'm trying to reduce to a reasonably sized basis set. The original matrix is 916x225, and I need to reduce the number of variables (its columns) to around 50, but I want to select those that are the most representative of the complete matrix.

Specifically, I want to find a subset S of size - say - 50 variables from all, which leave the least unexplained variance in a regression of all the other variables on S ("most representative").

My current approach is to perform PCA (prcomp in R), and get the individual columns that are most associated with each principal component. I assume that the original variable with the largest absolute value for its loading (i.e., the largest absolute value in the rotation matrix for each variable), is thus most representative or most correlated with that PC.

Am I interpreting this correctly? If not, any additional guidance is appreciated.

Update: From the comments below, I wanted to add this clarifying point in order to help focus any discussion on my intent. I apologize that I did not convey it well in the original question.

Essentially I'm looking for a subset S of size - say - L=50 variables from all, which leave the least unexplained variance in a regression of the other variables on S ("most representative"). My hope was that by using PCA, I could find how many PCs are need for, say, 90% of the variance, then choose the variables that are most correlated with each PC.

I thought of brute force search, too, but haven't tried that since I have 225 variables in my original matrix, and 225 choose 50 comes to about 3*e+50. That might take a very long time to compute all those linear models.

$\endgroup$
  • 2
    $\begingroup$ You (along with prcomp in R) incorrectly applies the word "loading" for the eigenvector (rotation) matrix. As for your question about an assosiation, it is answered here. $\endgroup$ – ttnphns Sep 16 '16 at 3:36
  • $\begingroup$ In prcomp, the return value provides a matrix called 'rotation'. Is that not a loadings matrix, but rather an eigenvalue matrix? $\endgroup$ – KirkD_CO Sep 16 '16 at 4:18
  • $\begingroup$ Yes, it is so... $\endgroup$ – ttnphns Sep 16 '16 at 4:39
  • 1
    $\begingroup$ Hmm, I might misunderstand the question. But I think, you are looking (combinatorically) for a subset S of size - say - L=50 variables from all, which leave the least unexplained variance in a regression of the other variables on S ("most representative") ? If I got this correctly then a brute force checking all combinations of subsets should be "sufficient". I do not see immediately, how a PCA could be used to reduce the effort of the brute-force solution, but perhaps this is possible - before thinking deeper about it I'd like to know whether I got your optimization target correctly? $\endgroup$ – Gottfried Helms Sep 20 '16 at 14:23
  • 1
    $\begingroup$ Yes, you have the interpretation exactly: a subset S of size - say - L=50 variables from all, which leave the least unexplained variance in a regression of the other variables on S ("most representative"). My hope was that by using PCA, I could find how many PCs are need for, say, 90% of the variance, then choose the variables that are most correlated with each PC. I thought of brute force, too, but haven't tried that since I have 225 variables in my original matrix, and 225 choose 50 comes to about 3*e+50. That might take a very long time to compute all those linear models. $\endgroup$ – KirkD_CO Sep 20 '16 at 15:12
0
$\begingroup$

You're interpreting PCA incorrectly, I believe. The output of PCA is an orthogonal basis set of all the data observations, where the first principal component contributes the most variance, the second PC the second most, and so on so forth.

However, I'll take 'finding the column vectors most correlated with each PC' to mean 'I want to reduce my dimensionality/number of features. Selecting the variables most correlated with the PCs will not do this. But, there is a way! You can transform your observation matrix to the new basis found by your PCA, achieving just this. This is done by $T_{L} = X~W_{L}$, where $X$ is your observation matrix, $W$ is the loading matrix, $T$ is the transformed matrix, and $L$ is the number of principal components you want to keep. So, if you let $L=225$ (the maximum L, there cannot be more PCs than the original number of features), then nothing has changed. But, if you let $L=50$, then you're effectively projecting your data onto the 50th-dimension hyperplane, in the new principal component feature space. This is key! You are not SELECTING 50 features, but projecting your data onto the 50 most important PCs. So, your features will be different from the original ones, but they will be 50 features that are linear combinations (/projections) of the old ones.

Of course, there's also the fact that you have no idea if your dataset is linearly separable; you might want to check that out first before pursuing PCA.

$\endgroup$
  • $\begingroup$ Your answer seems to be explaining how to compute PC scores. How does that answer the Q about the association between the variables and the components? $\endgroup$ – ttnphns Sep 16 '16 at 4:01
  • $\begingroup$ Thank you for the detail. I understand what PCA is doing and that each PC is a linear combination of the original columns of the data matrix. What I need to do is reduce the original space to about 50 columns in an effort to reproduce the entire space. In other words, I'm looking for 50 columns from the original space which will allow me to reproduce the entire original matrix with the least amount of error. Let's say I had a set of 50 columns and then developed linear models for each of the columns from the original matrix. From that, I would like the 50 columns that give the best predictions $\endgroup$ – KirkD_CO Sep 16 '16 at 4:16

Not the answer you're looking for? Browse other questions tagged or ask your own question.