0
$\begingroup$

Using the seatpos dataset from the faraway package in R, I wanted to do PLS regression models with up to eight components, choose the one with lowest RMSE as optimal model, and make predictions with it. So I first created the models and computed the RMSEs for each of them: 

> plsmod <- plsr(hipcenter ~ .,data=seatpos,ncomp=8,validation='CV')
> plsCV <- RMSEP(plsmod,estimate='CV')

Plotting RMSEP vs number of components, it can be seen that the model with lowest RMSE has three components: 

> plot(plsCV,main='')

enter image description here

The lowest RMSEP thus corresponds to the model with three components, which corresponds to plsCV[4]:

> plsCV
(Intercept)      1 comps      2 comps      3 comps      4 comps      5 comps       
      60.45        45.90        40.54        38.62        42.36        43.54    
6 comps      7 comps     8 comps  
    45.05        45.12        45.33

Looking to the plot, the output for plsCV, and the documentation of plsr() I would specify ncmop = 3 in predict(), but if I followed the same way of analysis as for PCR in the book of Faraway, then I should use the output for which.min(plsCV$val), i.e. 4 in my case, for ncomp.

So, when using predict() in my PLS model, do I need to specify ncomp = 3 or ncomp = 4?

Improve this question
$\endgroup$
8
  • $\begingroup$ Is there something against testing both in generating the predictions? I mean it seems that you have a theoretical justification for using 4 components, based on Faraway, and a more practical justification for using 3 components based on your results. For the latter you could argue that it is sensitive to idiosyncrasies of the data and therefore prefer the theoretical justification of 4 components. Btw I am not familiar with this type of modeling so your guess is as good as mine. $\endgroup$ – horseoftheyear May 31 '20 at 8:55
  • $\begingroup$ When testing i see that ncomp=4 gives lower RMSE: > rmse <- function(x,y) sqrt(mean((x-y)^2)) > rmse(predict(plsmod,ncomp = 3),seatpos$hipcenter) [1] 34.17719 > rmse(predict(plsmod,ncomp = 4),seatpos$hipcenter) [1] 33.19577 But how to concile that with the plot and plsCV values? $\endgroup$ – ForEverNewbie May 31 '20 at 11:53
  • $\begingroup$ The first element in the array is the intercept, which is probably constant prediction as mean. So your 4th element in the array is already corresponding to RMSEP of ncomp=3. And since you need to provide ncomp, NOT the index in the array, ncomp=3 is the way to go. $\endgroup$ – theGD May 31 '20 at 15:19
  • $\begingroup$ in other words, which.min(plsCV$val) returns the index of the minimum in the array. Maybe at the release day of book, RMSEP for intercept was not present. $\endgroup$ – theGD May 31 '20 at 15:20
  • $\begingroup$ @theGD that is what i thought, yet ncomp=4 gave a smaller RMSE as i showed above. So how can this be? $\endgroup$ – ForEverNewbie Jun 1 '20 at 3:11
0
$\begingroup$

With cross-validation the basic steps for your case is: Starting with ncomp=1

  1. Remove a fixed portion of data from training set
  2. Build a model with ncomp = 1 with the remaining data
  3. Predict the removed samples
  4. Compare the prediction with the actual values of removed samples (again for removed samples ofc.)
  5. Put the removed samples back into training set
  6. If all samples are removed once, calculate an average error, otherwise go to step 2

By doing so, an error estimate is calculated for ncomp=1. Similarly, for each ncomp the error is calculated. This is to somewhat simulate the performance of each ncomp on unseen data.

On the other hand, autoprediction errors, that is calculating RMSE for each ncomp using entire training set each time, can provide a minimum RMSE for THAT DATA SET. In fact, for PLS, if you use all possible components you may end up with a nearly zero RMSE for training set while performing very bad on validation set or on any other unseen data.

99% of the time, CV is better for finding optimum parameter (ncomp) because it provides a good estimate for a higher predictive performance on unseen data or in other words for avoiding overfitting.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you very much for your explanation. I am too new to R and PLS, so it is very possible I am missing/misunderstanding something. I thought that using CV with the training set for each ncomp would allow to see which ncomp is best, i.e. yields lowest RMSE, to build the model, as shown in the graph above. Since the graph shows the lowest RMSE corresponds to ncomp=3, I thought that the smallest RMSE for the predictions using the whole training set would correspond to ncomp=3 as well. However the lowest RMSE for the predictions corresponds to ncomp=4,as seen in the output shown in the comments. $\endgroup$ – ForEverNewbie Jun 2 '20 at 12:27
  • $\begingroup$ So is it clear now? $\endgroup$ – theGD Jun 2 '20 at 15:39
  • $\begingroup$ So even though the RMSE of the prediction is smaller when ncomp=4 as which.min(plsCV$val) suggests, in order to build the model i need to use ncomp=3 as both the graph and plsCV indicate, correct? $\endgroup$ – ForEverNewbie Jun 3 '20 at 2:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.