(Dis)advantages of a PLS regression over PCR

Question

I've read a lot of sources about Partial Least Squares (PLS) Regression and, based on my readings, it seems that it has some advantages over a Principal Component Regression (PCR). Different sources indicate that a PLS regression takes into account the variability of the dependent variables (while PCR doesn't). Why is this aspect so important and why it is considered to be an advantage over PCR?

Also, are there any other concrete advantages of conducting a PLS regression instead of a PCR?

I conducted both types of analyses on different datasets and I computed the corresponding MSEs. I remarked two things:

1) The MSE of a PLSR was lower than the MSE of a PCR;

2) PLSR extracts more components than the PCA (a PCA is done as a part of the PCR).

I think that the MSE of a PLSR is lower because the optimal number of extracted components is higher. However, I don't consider that extracting a higher number of components is an advantage of PLSR over PCR. Am I right to say that this would be a disadvantage of PLSR over PCR? Mainly, for both methods, we are interested in extracting a small number of components...

Lastly, I remarked that when I perform a PLSR and I work with a small number of predictors, the optimal number of extracted components is approximately equal to the no. of predictors. How this may be explained?

Answer 1

Different sources indicate that a PLS regression takes into account the variability of the dependent variables (while PCR doesn't). Why is this aspect so important and why it is considered to be an advantage over PCR?

You may have confounders that contribute large variance to $\mathbf X$, but as they are confounders that variance does not help but rather hinder your prediction. Moreover such variance may be uncorrelated to the predictions, i.e. it isn't even anything that helps to correct predictions for some other effects.

PCA will have confounders and important influencing factors alike sorted accoring to the $\mathbf X$-variance they contribute. PLS downweights confounders as that variance is not correlated with variance in $\mathbf Y$.

Also, are there any other concrete advantages of conducting a PLS regression instead of a PCR?

Can't think of any right now - but if needed, downweighting confounding variance is a pretty solid advantage.

PLSR extracts more components than the PCA

This would make me suspicious of overfitting. Sure, this may happen. But typically, PLS needs fewer latent variables/components than PCR.

Am I right to say that this [more components] would be a disadvantage of PLSR over PCR?

This disadvantage is usually formulated as PLSR being more prone to overfitting.

How did you determine the number of latent variables/components? You need to make sure the models aren't overfit for both PCA and PLSR, but for PLS it is even more crucial than for PCA.

 Lastly, I remarked that when I perform a PLSR and I work with a small number of predictors, the optimal number of extracted components is approximately equal to the no. of predictors. How this may be explained?

Your system may be of full rank, or your procedure to determine the number of latent variables may be overfitting. Without further details (in particular also on the appliation and data) it is impossible to say which.

Answer 2

An Introduction to Statistical Learning (Hastie et al):

In practise it often performs no better than ridge regression or PCR. While supervised dimension reduction of PLS can reduce bias, it also has the potential to increase variance, so that the overall benefit of PLS relative to PCR is a wash.