0
$\begingroup$

I've read a lot of sources about Partial Least Squares (PLS) Regression and, based on my readings, it seems that it has some advantages over a Principal Component Regression (PCR). Different sources indicate that a PLS regression takes into account the variability of the dependent variables (while PCR doesn't). Why is this aspect so important and why it is considered to be an advantage over PCR?

Also, are there any other concrete advantages of conducting a PLS regression instead of a PCR?

I conducted both types of analyses on different datasets and I computed the corresponding MSEs. I remarked two things:

1) The MSE of a PLSR was lower than the MSE of a PCR;

2) PLSR extracts more components than the PCA (a PCA is done as a part of the PCR).

I think that the MSE of a PLSR is lower because the optimal number of extracted components is higher. However, I don't consider that extracting a higher number of components is an advantage of PLSR over PCR. Am I right to say that this would be a disadvantage of PLSR over PCR? Mainly, for both methods, we are interested in extracting a small number of components...

Lastly, I remarked that when I perform a PLSR and I work with a small number of predictors, the optimal number of extracted components is approximately equal to the no. of predictors. How this may be explained?

$\endgroup$
1
  • $\begingroup$ Kuhn and Johnson state that, "Based on our experience, the number of components retained via cross-validation using PCR is always equal to or greater than the number of components retained by PLS. This is due to the fact that dimensions retained by PLS have been chosen to be optimally related to the response, while those chosen with PCR are not." (Applied Predictive Modeling, pp.115-116) $\endgroup$ – Todd Burus Mar 9 '20 at 16:09
1
$\begingroup$

Different sources indicate that a PLS regression takes into account the variability of the dependent variables (while PCR doesn't). Why is this aspect so important and why it is considered to be an advantage over PCR?

You may have confounders that contribute large variance to $\mathbf X$, but as they are confounders that variance does not help but rather hinder your prediction. Moreover such variance may be uncorrelated to the predictions, i.e. it isn't even anything that helps to correct predictions for some other effects.

PCA will have confounders and important influencing factors alike sorted accoring to the $\mathbf X$-variance they contribute. PLS downweights confounders as that variance is not correlated with variance in $\mathbf Y$.

Also, are there any other concrete advantages of conducting a PLS regression instead of a PCR?

Can't think of any right now - but if needed, downweighting confounding variance is a pretty solid advantage.

PLSR extracts more components than the PCA

This would make me suspicious of overfitting. Sure, this may happen. But typically, PLS needs fewer latent variables/components than PCR.

Am I right to say that this [more components] would be a disadvantage of PLSR over PCR?

This disadvantage is usually formulated as PLSR being more prone to overfitting.

How did you determine the number of latent variables/components? You need to make sure the models aren't overfit for both PCA and PLSR, but for PLS it is even more crucial than for PCA.

 Lastly, I remarked that when I perform a PLSR and I work with a small number of predictors, the optimal number of extracted components is approximately equal to the no. of predictors. How this may be explained?

Your system may be of full rank, or your procedure to determine the number of latent variables may be overfitting. Without further details (in particular also on the appliation and data) it is impossible to say which.

$\endgroup$
1
  • $\begingroup$ Thank you for your detailed answer. For PLS regression, I determined the no. of latent components using SAS (via the SVD algorithm) including cross-validation. Concerning your statement: "You need to make sure the models aren't overfit for both PCA and PLSR...", how do you recommend to do this? Should a specific hypothesis test be attempted or should I check the values of some parameters? $\endgroup$ – muffin Mar 9 '20 at 18:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.