0
$\begingroup$

As far as predictive modelling is concerned, there are many dimensionality reduction methods (PCA, Kernel PCA, ICA, etc.) that can be applied to reduce the number of predictors prior to modelling. I am trying different predictive models to solve a classification problem with many variables. I have applied different dimensionality reduction methods to the whole matrix of predictors (training+validation+test set). Since I've used unsupervised dimensionality reduction techniques, there should be no problem to apply the methods to the whole matrix of predictors because the response is not used for the construction of the components. So, now I have many transformed matrix of predictors: one with PC scores, one with kernel PC scores with polynomial kernel, another kernel PC scores with radial kernel etc. I have then split each dataset according to the 50/25/25 rule in a training, validation and test set. I'm trying different different predictive models (LDA, QDA, KNN, SVM, etc..). For each predictive model, I need to determine a good choice for the number of components to use. I consider this as part of model selection, so I'm using the validation set for this. Let's take LDA as an example. I try the model with 20, 30 and 40 PCs. Suppose that 30 PCs are needed to have a good classification error for LDA on the validation set. Then I try to optimize the number of kernel components for LDA. For example, I try LDA with 20, 30 and 40 kernel PCs. Suppose that on the validation set the best number of kernel PCs for LDA appears to be 50. I do the same for QDA, KNN, SVM, etc. In the end, for each predictive model I know the best number of components for each dimensionality reduction method I used. Now the question is: which predictive models should I now evaluate on the test set? Should I evaluate $k$ LDA models, $k$ QDA models, etc. where $k$ is the number of different dimensionality reduction method I used? (This means to evaluate both the LDA model with 30 PCs and the LDA model with 50 kernel PCs on the test set). Or should I just evaluate one LDA model, one QDA model and so on? (This means to choose for evaluation the model between the LDA with 30 PCs and the LDA with 50 kernel PCs with the lowest classification error on the validation set.)
Thank you in advance.

$\endgroup$
1
$\begingroup$

Either is permissible. Choosing the single method with the best validation-set performance and trying only that is also permissible. Using more methods makes for more final results to digest, and might perhaps raise overfitting issues, but provides some insurance against choosing an inferior method by chance.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.