Partial Least Squares: adding unrelated variables improves fit

Question

I am performing some simulations of partial least squares. In particular, I have 30 observations split into 20 which are for training and 10 which are for testing. I also have 23 independent variables (all unrelated to each other), the first 18 of which are related to the dependent variable while the last 5 are unrelated to the dependent variable. Then I perform two regressions on the training sample:

1. A PLS regression with all 23 independent variables using 18 principal components.
2. An OLS regression with only the 18 variables that are related to the dependent variable.

Surprisingly, when I test the two models using the testing sample, the model estimated using 1. consistently produces substantially lower RMSE than the model estimated using 2. It appears that adding variables that are unrelated to the dependent variable improves the model's predictions. The same occurs when I use principal components regression instead of PLS.

What is the cause of this? Is this expected or is it likely due to some mistake in my simulations?

(I also tried estimating 2. using PLS (with 18 components) in order to confirm that the PLS function in R indeed returns the same result as OLS when the number of independent variables is lower than the number of observations.)

Answer 1

It probably means PLS outperforms OLS even in the presence of unrelated variables.

I would definitely check regression coefficients of PLS model to see whether the the unrelated variables have relatively low absolute magnitudes. However, if the unrelated variables are related to each other, then the regression coefficients can be deceiving.

The lower RMSE of PLS may be explained by this multicollinearity. Even small amounts of dependency (not only within variables pairs but also within multiple variables) can benefit from regularization provided by PLS. Since generally the low sample count combined with multicollinearity of variables results in high RMSE for OLS.

Thus I would also perform dimension reduction techniques such as PCA to observe how much the eigenvalues decreases. If you have single or multiple almost 0 eigenvalues it indicates your variables are in fact related.