0
$\begingroup$

I know there are plenty of questions on this topic on the site, but I still can't work out how to choose the optimal number of components for the particularities of my PLS model.

I have 1800 observations of 150 variables in my X-matrix and I am trying to predict a single continuous Y-variable. After model training in the pls package (ie the default 10-fold cross validation, starting with 10 components), RMSEP is minimised by 10 components, although when the "one SE" rule is applied, one component only is judged to be the best model (see plot below). In addition, neither of these are more than one SE away from a zero component model. Given the complex nature of the data, I would expect more than one component to be necessary to predict Y. Therefore, is this cross-validation the best approach? How should I proceed?

enter image description here

My eventual aim is to predict Y for an independent dataset. The pls package also allows you to specify which components are used for the predictions. So, as a follow-up question, why would I want to use less components than in my cross-validated model?

Help would be greatly appreciated.

$\endgroup$
2
$\begingroup$

Some questions that may help digging down to what the actual issue is. I don't think cross validation itself is the problem here - it's probably just exposing problems in the model.

  • All your tentative models do not show much improvement over the 0 component model: even at 10 latent variables, $RMSE_{CV}$* is still withing 95 % of the $RMSE_{CV}$ always guessing the average $y$ (0 latent variables).
    => What is % explained Y-variance?
    => What is the corresponding training RMSE? => How are the errors distributed?

    * In my field, usually an RMSE estimated from cross validation is denoted $RMSE_{CV}$, $RMSE_P$ is more-or-less reserved for predictions on a really separate set of samples/cases.

  • Where does the variance of the RMSE estimates come from: model instability or high variance in the prediction error for different cases?

  • Keep in mind: it is of course possible that PLSR cannot give a meaningful answer to your problem / is plainly unsuitable for that data. PLSR is a powerful tool, but it is no magic recourse in case there's almost no signal and that drowned in lots of noise.

  • Does your data allow to check the loadings/latent variables in a way that is independent of RMSE optimization?
    E.g., I work with spectra which have high correlation between neighbour variates. If the loadings or coefficients look noisy, I know for sure that the model is overfitting. The opposite is of course not true: they may look good and are spectroscopically perfectly sensible but extract "chemical noise", i.e. variation in the data that is only accidentally related to the analyte (Y) I'm after.
    I've certainly had data where I had to say: "We may be able to do something in terms of meaningful predictions for the application with more complex models - and due to the nature of the application complex models will be needed to get there. But this data set is so noisy that we cannot get a stable estimate even for the first latent variable."

Given the complex nature of the data, I would expect more than one component to be necessary to predict Y.

  • You could try to go for many latent variables, i.e. up to models so complex that they are clearly overfitting.
    Plot training and cross validation error to see this.

  • As explained above, it's good to formulate an idea of what complexity could be suitable for the application. But unfortunately that doesn't guarantee that any given data set contains sufficient information to actually train a sensible model of that complexity.


The pls package also allows you to specify which components are used for the predictions. Why would I want to use less components than in my cross-validated model

Probably the most common use of this feature is to not needing to re-fit the model in order to get predictions with the first n components during optimization of the number of latent variables.

During this optimization, you basically try out models with 1, 2, 3, ..., 10 latent variables. The implementation by pls needs to fit the model only once for 10 latent variables and then extracts the predictions for the less complex models, saving 90 % of (totally redundant) computation over the naive approach of fit 1 lv => predict, fit 2 lvs => predict, ...

Exposing this interface is certainly handy whenever you want to have more fine-grained access than just getting the overall (pooled) performance after running pls' cross valiadation procedure.

$\endgroup$
1
  • $\begingroup$ Many thanks for this comprehensive answer. I'll follow your suggested diagnostics to try and work out whether this approach is worth pursuing. Perhaps as you say PLSR is simply unsuitable for this data. $\endgroup$ – Joe Sep 5 '19 at 9:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.