0
$\begingroup$

I have a training test with 3000 samples, 100 features per sample.

For each sample, there is also a label column with the sample "price" - continuous variable with infinite values possible (think of price of a house, for example)

My end goal is to build a model, based on the training set, which predicts a new sample "price".

In order to do so, I would first like to perform some dimensionality reduction, such as PCA, but I want to perform it so it would give me as a result, manipulated data set, with samples containing only the most affecting features in regard to the price label.

e.g. get transformed data set, with only X features that actually have an impact on the specific "price" column value

How can I achieve that in R?

Improve this question
$\endgroup$
1
  • $\begingroup$ "How can I achieve something in R" is off-topic here, but your general question about using PCA for feature selection is on-topic. I think it's duplicate and voted to close as such. $\endgroup$ – amoeba Jan 30 '17 at 13:57
1
$\begingroup$

PCA will give you features, but which will be a linear combinations of the original 100 features. As such, you will (potentially) need all of the 100 to create the principal components. This is not the proper tool if you cast your problem as a regression of $y$ based on $\mathbf{X}$.

The maximization problem that PCA solves is $\max_{\mathbf{p}_1 \in \mathbb{R}^{100}} \mathsf{Var}(\mathbf{p}_1^\top\mathbf{X})$ subject to $\mathbf{p}_1^\top\mathbf{p}_1=1$, i.e. explain most of the variance of the design matrix. The other principal components $\mathbf{p}_j, j=2, \ldots$ are obtained by imposing the further constraint that $\mathbf{p}_i^\top\mathbf{p}_j=0$ for $i=1, \ldots, j$, so that the principal components are orthogonal. An alternative that will also give a regression based on linear combinations of $\mathbf{X}$-features is PLS, which aims at maximizing $\max_{\mathbf{w}}\mathsf{Cov}(\boldsymbol{y}, \mathbf{X}\mathbf{w})$ subject to $\|\mathbf{w}\|=1$ and orthogonality of the weights vectors.

In principle, obtain the principal components, choose the number $K$ you want to keep (based on e.g. a scree plot diagnostic) and obtain your new regressors as $\{\mathbf{p}_i\mathbf{X}\}_{i=1}^K$.

pca <- prcomp(X)
X%*%pca$rotation[,1:K]

If you want to do variable selection in say a linear regression, you could use LASSO to shrink some coefficients in the regression to zero and only keep the features of interest.

Improve this answer
$\endgroup$
6
  • $\begingroup$ That's a good answer, but this Q is most likely going to be closed as a duplicate of stats.stackexchange.com/questions/27300. Consider posting your answer there, I think it can be useful for many people in the future, especially if you expand a little bit on your last paragraph (lasso recommendation). $\endgroup$ – amoeba Jan 30 '17 at 14:18
  • $\begingroup$ Thank you for the elaborate answer. Could you please explain what the second line of code you posted does exactly? $\endgroup$ – Adiel Feb 1 '17 at 7:42
  • $\begingroup$ It returns the "transformed data set", meaning the new matrix of regressors consisting of $K$ orthogonal features that are linear combinations of the original matrix of inputs, $[\mathbf{X}\mathbf{p}_1, \ldots, \mathbf{X}\mathbf{p}_K]$, in the particular case of PCA. Note that you may also wish to perform PCA on the correlation matrix rather than the covariance matrix if your features have different variance. $\endgroup$ – lbelzile Feb 1 '17 at 11:37
  • $\begingroup$ @amoeba While the specific question pertaining to R is off-topic, the question above asked how to obtain the matrix of regressors for PCA, while stats.stackexchange.com/questions/27300/… asks about the relative benefits of PCA for feature selection, but not necessarily in a prediction problem. $\endgroup$ – lbelzile Feb 1 '17 at 11:44
  • $\begingroup$ @lbelzile Not necessarily, but at least 90% of people who come here asking about using PCA for feature selection have a prediction problem in mind. We are closing all of them as duplicates of stats.stackexchange.com/questions/27300, so the more useful answers there are over there, the better for the community. Also note that that question asked about "information contained" in the features, which kind of assumes that there is some output variable with respect to which one can talk about "information". $\endgroup$ – amoeba Feb 1 '17 at 11:48

Not the answer you're looking for? Browse other questions tagged or ask your own question.