# How to use PCA in regression?

I'm currently reading in the Applied Predictive Modeling book about PCA for dimensionality reduction. I've read the following:

If the predictive relationship between the predictors and response is not connected to the predictors’ variability, then the derived PCs will not provide a suitable relationship with the response. In this case, a supervised technique, like PLS, will derive components while simultaneously considering the corresponding response.

I don't understand the meaning of this. Could somebody explain this in better words or in an example (when to use PLS instead of PCA)? How can I decide in a real dataset if I should use PCA or PLS? Predictors is the same as features.

Second, I'm using Matlab and I'm a bit unsure about the transformation. Let's assume that $X$ is my data matrix where the rows are the data points and the columns are the features (predictors). In Matlab the function call is

[coeff,score,latent,tsquared,explained,mu] = pca(X)


Should I now use coeff or score instead of X or should I do X*coeff or X*score? I also don't understand what the difference is between coeff and score.

Say your predictor matrix is $X$ and your response vector is $y$. PCA is concerned only with the (co)variance within the predictor matrix $X$ itself, while a regression model is (also) concerned with the covariance between $X$ and the response $y$. If there is no relationship between these concepts, dimension reduction by PCA can be harmful to your regression by screening out those predictors in $X$ that are correlated with the response.

Here is a simple example, in R, as I don't have access to matlab. Suppose I create some random gaussian data

x_1 <- rnorm(10000, mean = 0, sd = 1)
x_2 <- rnorm(10000, mean = 0, sd = .1)
X <- cbind(x_1, x_2)


And set up a situation where the a response is correlated with only the smaller varaince component

y <- x_2 + 1


The principal components of $X$ are just $x_1$ and $x_2$, given the way I set it up

> cor(X)
x_1          x_2
x_1 1.0000000000 0.0004543833
x_2 0.0004543833 1.0000000000


If PCA is used to select one component, we get $x_1$, as this has the highest variance. Regressing $y$ on $x_1$ is useless

> lm(y ~ x_1)

Call:
lm(formula = y ~ x_1)

Coefficients:
(Intercept)          x_1
1.001e+00    4.544e-05


The coefficient of $x_1$ here is zero, the model has no more predictive power than an intercept only model. On the other hand, if I select the lower variance component

> lm(y ~ x_2)

Call:
lm(formula = y ~ x_2)

Coefficients:
(Intercept)          x_2
1            1


I get back a much more predictive model.

Because PCA ignores the relationship of $y$ to $X$, there is no reason to believe that its dimension reduction is reasonable in the context of your regression problem.

On the other hand using the relationship between $y$ and $X$ to do pre-regression dimension reduction and variable selection is a very good way to overfit your model to your training data. In some ways, PCAs ignorance of the relationship between $X$ and $y$ is a blessing, because, while it can be harmful in the way outlined above, it cannot overfit the relationship in your training data in the same way that peeking at $y$ can.

As for your more practical question, the matlab documentation says

coeff = pca(X) returns the principal component coefficients, also known as loadings, for the n-by-p data matrix X. Rows of X correspond to observations and columns correspond to variables. The coefficient matrix is p-by-p. Each column of coeff contains coefficients for one principal component, and the columns are in descending order of component variance. By default, pca centers the data and uses the singular value decomposition (SVD) algorithm.

This says to me, that to do PCA dimension reduction in matlab, you need to:

• Center the columns of your $X$ matrix.
• Select the first $N$ columns of the coef matrix, where $N$ is the number of non-intercept regressors you want in your model.
• Create a new data matrix as center(X) * coef[, 1:N].
• Use the columns in the new matrix as regressors in your dimensional reduced regression.

Again, I am far from fluent in matlab, so the syntactic details of how to preform these steps is unknown to me.

• +1. Matlab's score output is already the result of X*coeff. – amoeba says Reinstate Monica Dec 11 '15 at 22:57
• Thank you so much for your great help. I have three following up questions which I will write in separate comments. By the way, I'm doing classification and not regression. First, if X is not correlated with y, PCA should not be used (and PLS instead). Is this correct? How can I test this in reality? Should I just calculate the correlation between each data point and y or the correlation between each predictor (feature) vector and y? For me it is absolutely unclear how to check if PCA should not be used. – machinery Dec 11 '15 at 23:30
• Second, X is automatically centered by the pca function in Matlab. Do you mean that I should select the first N columns of coef? You wrote, first N rows. With this I'm getting a mismatch in dimension with X * coef(1:N,:). – machinery Dec 11 '15 at 23:33
• Third, should I use X * score(:,1:N) instead of X * coeff(:,1:N)? Score holds the PCs in the columns and coeff holds the loadings. By the way, for me X * coeff does not give the same as X * score. – machinery Dec 11 '15 at 23:35
• Correction to my last comment. I mean X * coeff is not the same as score. I don't know why. If I understand correctly X * coeff should result in a matrix where the columns are the principal components. In the matlab documentation it is written that the columns of score are the principal components. For me it is unclear why X * coeff does not give score. – machinery Dec 11 '15 at 23:55