# Using PCA for finding features most affecting a given label [duplicate]

I have a training test with 3000 samples, 100 features per sample.

For each sample, there is also a label column with the sample "price" - continuous variable with infinite values possible (think of price of a house, for example)

My end goal is to build a model, based on the training set, which predicts a new sample "price".

In order to do so, I would first like to perform some dimensionality reduction, such as PCA, but I want to perform it so it would give me as a result, manipulated data set, with samples containing only the most affecting features in regard to the price label.

e.g. get transformed data set, with only X features that actually have an impact on the specific "price" column value

How can I achieve that in R?

• "How can I achieve something in R" is off-topic here, but your general question about using PCA for feature selection is on-topic. I think it's duplicate and voted to close as such. – amoeba Jan 30 '17 at 13:57

PCA will give you features, but which will be a linear combinations of the original 100 features. As such, you will (potentially) need all of the 100 to create the principal components. This is not the proper tool if you cast your problem as a regression of $y$ based on $\mathbf{X}$.
The maximization problem that PCA solves is $\max_{\mathbf{p}_1 \in \mathbb{R}^{100}} \mathsf{Var}(\mathbf{p}_1^\top\mathbf{X})$ subject to $\mathbf{p}_1^\top\mathbf{p}_1=1$, i.e. explain most of the variance of the design matrix. The other principal components $\mathbf{p}_j, j=2, \ldots$ are obtained by imposing the further constraint that $\mathbf{p}_i^\top\mathbf{p}_j=0$ for $i=1, \ldots, j$, so that the principal components are orthogonal. An alternative that will also give a regression based on linear combinations of $\mathbf{X}$-features is PLS, which aims at maximizing $\max_{\mathbf{w}}\mathsf{Cov}(\boldsymbol{y}, \mathbf{X}\mathbf{w})$ subject to $\|\mathbf{w}\|=1$ and orthogonality of the weights vectors.
In principle, obtain the principal components, choose the number $K$ you want to keep (based on e.g. a scree plot diagnostic) and obtain your new regressors as $\{\mathbf{p}_i\mathbf{X}\}_{i=1}^K$.
pca <- prcomp(X)
X%*%pca$rotation[,1:K]  If you want to do variable selection in say a linear regression, you could use LASSO to shrink some coefficients in the regression to zero and only keep the features of interest. • That's a good answer, but this Q is most likely going to be closed as a duplicate of stats.stackexchange.com/questions/27300. Consider posting your answer there, I think it can be useful for many people in the future, especially if you expand a little bit on your last paragraph (lasso recommendation). – amoeba Jan 30 '17 at 14:18 • Thank you for the elaborate answer. Could you please explain what the second line of code you posted does exactly? – Adiel Feb 1 '17 at 7:42 • It returns the "transformed data set", meaning the new matrix of regressors consisting of$K$orthogonal features that are linear combinations of the original matrix of inputs,$[\mathbf{X}\mathbf{p}_1, \ldots, \mathbf{X}\mathbf{p}_K]\$, in the particular case of PCA. Note that you may also wish to perform PCA on the correlation matrix rather than the covariance matrix if your features have different variance. – lbelzile Feb 1 '17 at 11:37