As far as predictive modelling is concerned, there are many dimensionality reduction methods (PCA, Kernel PCA, ICA, etc.) that can be applied to reduce the number of predictors prior to modelling. I am trying different predictive models to solve a classification problem with many variables. I have applied different dimensionality reduction methods to the whole matrix of predictors (training+validation+test set). Since I've used unsupervised dimensionality reduction techniques, there should be no problem to apply the methods to the whole matrix of predictors because the response is not used for the construction of the components. So, now I have many transformed matrix of predictors: one with PC scores, one with kernel PC scores with polynomial kernel, another kernel PC scores with radial kernel etc. I have then split each dataset according to the 50/25/25 rule in a training, validation and test set. I'm trying different different predictive models (LDA, QDA, KNN, SVM, etc..). For each predictive model, I need to determine a good choice for the number of components to use. I consider this as part of model selection, so I'm using the validation set for this. Let's take LDA as an example. I try the model with 20, 30 and 40 PCs. Suppose that 30 PCs are needed to have a good classification error for LDA on the validation set. Then I try to optimize the number of kernel components for LDA. For example, I try LDA with 20, 30 and 40 kernel PCs. Suppose that on the validation set the best number of kernel PCs for LDA appears to be 50. I do the same for QDA, KNN, SVM, etc.
In the end, for each predictive model I know the best number of components for each dimensionality reduction method I used. Now the question is: which predictive models should I now evaluate on the test set? Should I evaluate $k$ LDA models, $k$ QDA models, etc. where $k$ is the number of different dimensionality reduction method I used? (This means to evaluate both the LDA model with 30 PCs and the LDA model with 50 kernel PCs on the test set). Or should I just evaluate one LDA model, one QDA model and so on? (This means to choose for evaluation the model between the LDA with 30 PCs and the LDA with 50 kernel PCs with the lowest classification error on the validation set.)
