I understand that the permutation test on PLS can help to detect overfitting of the PLS model. Usually if the p-value is greater than a criterion, say 0.05, it means that the model is overfitting and I won't use the PLS model to predict because of overfitting.

If I am interested in variable selection using PLS (using Variable Importance in Projection (VIP) scores), can I still use the PLS model for it even though the permutation test shows overfitting?

My thought is that even though my PLS model is overfitting, at least I can select variables that are relevant to the dependent variable.

I have only one binary dependent variable (control and treatment) and 1000 independent variables. Sample size is 25 per group. I use cross-validation to choose the number of PLS-DA components, but the subsequent permutation test shows lack of significant prediction power. The concept of permutation test that I am talking about is described in Szymańska et al., 2012, Double-check: validation of diagnostic statistics for PLS-DA models in metabolomics studies. (In a permutation test the labels of the samples are randomly permuted and a new classification model is calculated.)

  • 3
    $\begingroup$ I am afraid that to get high-quality answer you would have to edit your question to be more clear and more precise. $\endgroup$ – Tim Feb 2 '16 at 15:57
  • $\begingroup$ Hi amoeba, thank you for your commend. I am interested in feature selection using the VIP score returned by the PLS model. I think the VIP score is the average VIP scores across all the components. The number of components is determined by cross validation. $\endgroup$ – WCMC Feb 2 '16 at 23:08
  • $\begingroup$ Frank, this is a bit of a side-issue, but why did you want to use "the average VIP score across all components"? I looked up what VIP (Variable Importance in the Projection) means (e.g. see last section here) and it seems that VIP is directly computed from all [used] PLS components. No need to compute it separately per components and then average. Does it make sense or am I misunderstanding something? $\endgroup$ – amoeba Feb 3 '16 at 13:57
  • 1
    $\begingroup$ amoeba, thanks for pointint it out. Actually I completely agree with you. VIP is developed in paper Wold et al. 1993 with the same formula you mentioned. However, in my research field metabolomics, which is part of omics analysis, they often use vip in package mixOmics in R. This vip function somehow return VIP on of each variable on each component. So according to metaboanalyst.ca they often average the VIP on each component. $\endgroup$ – WCMC Feb 3 '16 at 17:32

For the benefit of other readers I will briefly explain what the permutation test is in this context.

In this specific example there is a binary dependent variable $y$, a large number of independent variables $X$ with $n\ll p$, and a specific algorithm (PLS-DA) to predict $y$ from $X$ with one hyperparameter (number $k$ of PLS components). To find the optimal value of $k$ one uses cross-validation. To estimate the performance of this model-building procedure one uses nested cross-validation. Imagine that nested cross-validation is done and the performance is, say, $43\%$ error rate or $R_\mathrm{CV}^2=0.2$. Question: are these numbers significantly different from chance? Is 43% significantly less than 50%? Is 0.2 significantly larger than 0.0? Answer: Shuffle the labels and repeat the entire procedure; do this many times to obtain the null distribution of performance; compare your actual performance with this null distribution to get the $p$-value.

What does insignificant $p$-value mean here?

Saying that it means that "the model is overfitting" is a bit of an understatement. The model includes regularization and cross-validation is performed to choose the optimal value of the regularization parameter (here, $k$). So the model is as non-overfitting as it could possibly be. So what is going on?

  1. It might be that the regularization is too weak and the model still does overfit. One can try another model altogether, allowing stronger regularization (see below).
  2. It might be that there is no or little overfitting, but the $y$ is not related to $X$ (at least not linearly) so the model cannot predict anything. You get 43% and not 50% due to finite sampling and random fluctuations.
  3. It might be that the model works fine but has little predictive power and you don't have enough data to actually be sure that predictive power is non-zero.

As you cannot be sure in #3 (that's the outcome of the permutation test), you have to consider that #2 is a real possibility. In other words: the model is absolutely useless or at least you failed to show otherwise.

Using the absolutely useless model for variable selection

There is clearly no sense in doing that. You could as well select variables at random. Or cherry pick variables that are most correlated with $y$.

I should add that PLS, PLS-DA and related approaches are popular in some applied fields but practically unknown in the mainstream statistics and machine learning. The most standard approach for binary classification together with variable selection would be logistic regression with elastic net (ridge+lasso) regularization. It might be worth considering.

Disclaimer: I have no practical experience with PLS or PLS-DA models.

  • $\begingroup$ Could you suggest any references mentioning using permutation tests to obtain "performance p-values"? $\endgroup$ – Tim Oct 18 '17 at 15:26
  • $\begingroup$ @Tim, no, sorry, I don't know a good reference, but a procedure is fairly obvious so I would be fine using it without giving any references. There is some black-box procedure outputting classification accuracy (in this answer I describe a nested cross-validation); we want to obtain its distribution under the null. The null is that there is no relationship between $X$ and $y$. So we use shuffling to sample under the null. $\endgroup$ – amoeba Oct 18 '17 at 21:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.