Here's the question which confuses me a lot.
- Cross validation is used for model assessment, estimate the predictive error of a particular model, right? So if there are two models, M1 and M2, what's the right procedure for cross validation? Is the following steps correct? For each of the models: First, divide the training set into 10 parts. Second, train Mi on the 9 parts and test Mi on the remaining 1. Third, average the prediction error. Suppose M1 has the least average prediction error, then we should choose M1. But how can we make prediction on new dataset? Because among the 10 times validation, the coefficients are different (like 10 slightly different models). Should we train M1 on the complete training set, and use the resulting coefficients to predict on new dataset?
- But can this situation happen? For a new dataset, M2 performs better than M1? Although M1 has the least average prediction error in CV. Does that mean maybe M1 do not outperform M2?
- In PLSR, the number of the optimal number of components are decided by cross validation. If we want to compare 2 PLSR models (the same response but completely different predictors), should we perform 2 different cross validation? e.g. Split the dataset into 10 parts, for 9 of them, perform plsr and use cross validation to obtain the number of components, and use these to predict the remaining one. And then obtain the prediction error. Finally compare the prediction error of 2 models?