I am implementing the PLS-DA method presented here on a data set and I am trying to understand the procedure; and whether there is anything conceptually wrong in my steps.
- I start with $188 \times 528 $ matrix $X$, where I have 188 samples and 465 features. I have a response variable $Y$ of length 188 comprised of ones and twos corresponding to class labels.
- I do PCA analysis on the dataset and find that there are only 187 non-zero eigenvalues. Next I project the data (i.e $X$) down into this 187 dimensional space using the loading vectors as a dimension reduction step to get $X'$.
- Now I use the Wilks lambda criterion to select 20 of these transformed features (out of 187 since $X'$ has size $188 \times 187$) that are most discriminatory (with respect to class membership) and then delete the other 167 columns. This gives me $\tilde X$.
- Now I split the data set into half for training, and half for testing, the do the PLS-DA routine.
I get really good results here following the procedure that I have not yet been able to achieve with other methods (SVMs,kernel-SVM, etc).
My concern here arises because both PLS and PCA are used for dimension reduction I am not sure if I am doing something wrong by mixing these techniques. Particularly, I started investigating whether using Wilks criterion on the whole data set was what was giving very good results, so I started using the criterion on just the training set (and then deleting the unwanted columns from both training and testing sets). The results markedly dropped; so indeed the Wilks criterion was overfitting to the structure of the complete data set.
But what was more interesting is that the PLS algorithm started failing when I started increasing the size of my training set (as a proportion of the complete data) to about 80%. Further it fails only when using Wilks criterion for feature selection; and it does not always fail, in say 3 tries out of 10, the script goes through.
Can someone (1) confirm that there is nothing theoretically wrong with using PCA for dimension reduction followed by PLS-DA on the transformed features for classification, (2) confirm that using Wilks criterion for feature selection in the transformed space is a valid approach (3) hint/speculate as to why PLS-DA fails when used in conjunction with Wilks criterion, in some cases.