RMSE behaviour during cross-validation of a PLS model - What does it mean?

Question

In PLS regression, one of the approaches in selecting the number of components (Latent Variables, LV) is to perform cross-validation over a range of LV, and select the one with lower Root Mean Squared Error ($RMSE_{CV}$). I have tried this approach and until now, when I plotted $RMSE_{CV-train}$ and $RMSE_{CV-test}$, there is a number of LVs where $RMSE_{CV-test}$ reach a minimum and then start increasing again, while $RMSE_{CV-train}$ keeps decreasing. This implies the overfitting of the model. In the previous implementations I have done, the value of LVs for the minimum $RMSE_{CV-test}$ is between 5 and 15.

Now, I am testing the same in a new dataset and when plotting the same graph for 4 different models (and dependent variables), found some "unusual" behaviours:

• In 2 of the cases, the minimum $RMSE_{CV-test}$ is using just 2 Latent variable (Prop. 0 and Prop. 3)
• In the other 2 cases, $RMSE_{CV-test}$ start increasing from the beginning (Prop. 1 and Prop. 2)

I have never seen this kind of evolution in the RMSE of a PLS model based on the number of Latent Variables used.

This dataset consists of 128 features (columns) and 59 samples (rows), and there are 4 output variables (Prop 0 to Prop. 3 in the graph)

What does this behaviour mean or what could be its cause?

Answer 1

There are many possible reasons, but it seems like you may not have enough rows to estimate the model accurately. You have enough degrees of freedom to vastly overfit, and because PLS regression finds the latent space that best models the covariance between the regressors and the target, it will find a space that overfits the data. As you expand the model capacity, you're likely exacerbating the problem, so as the number of latent features grows, the validation error goes up.

It'd be easier to reach a better conclusion if you included a baseline model (the mean of the target?) so it'd be clear if you're beating the baseline with your PLS.