Combining PLS-DA with PCA dimension reduction

Question

I am implementing the PLS-DA method presented here on a data set and I am trying to understand the procedure; and whether there is anything conceptually wrong in my steps.

• I start with $188 \times 528 $ matrix $X$, where I have 188 samples and 465 features. I have a response variable $Y$ of length 188 comprised of ones and twos corresponding to class labels.
• I do PCA analysis on the dataset and find that there are only 187 non-zero eigenvalues. Next I project the data (i.e $X$) down into this 187 dimensional space using the loading vectors as a dimension reduction step to get $X'$.
• Now I use the Wilks lambda criterion to select 20 of these transformed features (out of 187 since $X'$ has size $188 \times 187$) that are most discriminatory (with respect to class membership) and then delete the other 167 columns. This gives me $\tilde X$.
• Now I split the data set into half for training, and half for testing, the do the PLS-DA routine.

I get really good results here following the procedure that I have not yet been able to achieve with other methods (SVMs,kernel-SVM, etc).

My concern here arises because both PLS and PCA are used for dimension reduction I am not sure if I am doing something wrong by mixing these techniques. Particularly, I started investigating whether using Wilks criterion on the whole data set was what was giving very good results, so I started using the criterion on just the training set (and then deleting the unwanted columns from both training and testing sets). The results markedly dropped; so indeed the Wilks criterion was overfitting to the structure of the complete data set.

But what was more interesting is that the PLS algorithm started failing when I started increasing the size of my training set (as a proportion of the complete data) to about 80%. Further it fails only when using Wilks criterion for feature selection; and it does not always fail, in say 3 tries out of 10, the script goes through.

Can someone (1) confirm that there is nothing theoretically wrong with using PCA for dimension reduction followed by PLS-DA on the transformed features for classification, (2) confirm that using Wilks criterion for feature selection in the transformed space is a valid approach (3) hint/speculate as to why PLS-DA fails when used in conjunction with Wilks criterion, in some cases.

Answer 1

PCA and PLS-DA are mostly similar yet fundamentally different methods.

PCA provides dimension reduction by penalizing directions of low variance. What is meant by that is you provide no class information whatsoever and deal only with variance in the independent variables.

PLS-DA, on the other hand, again penalizes directions, but this time the directions are about covariance between independent and dependent variables(see this very good link for more on that). Oh, and in PLS-DA your dependent variables are simply class information.

PCA + PLS-DA is quite uncommon(at least in my area, chemometrics). There are few justifications I can think of:

1. The small direction of variances which are omitted via PCA can actually be useful in classification
2. PLS-DA already provides dimension reduction and deals with multicollinearity in a very similar manner (if you have time, see the lovely NIPALS algorithm for PCA and PLS. It worked better than reading words for me)

So, there might be instances where PCA + PLS-DA provides a good model but I suspect that is quite rare. Same applies to regression case; there is Principle Components Regression(PCR) and PLS alone, but I have never seen PLS after PCA.

All in all, although not particularly wrong, I think using PCA prior to PLS-DA will probably introduce a risk of decreasing model performance while providing no significant advantage and using PLS-DA only is a better option.

I don't know about Wilks lambda criterion, yet there are other methods for component selection which aim to yield highest prediction performance with minimal overfitting such as Leave-N-Out-Cross-Validation, Monte Carlo Cross Validation etc.. These methods are quite common and if you think your component selection is problematic, you can give these methods a try.