How to perform cross validation?

Question

Here's the question which confuses me a lot.

1. Cross validation is used for model assessment, estimate the predictive error of a particular model, right? So if there are two models, M1 and M2, what's the right procedure for cross validation? Is the following steps correct? For each of the models: First, divide the training set into 10 parts. Second, train Mi on the 9 parts and test Mi on the remaining 1. Third, average the prediction error. Suppose M1 has the least average prediction error, then we should choose M1. But how can we make prediction on new dataset? Because among the 10 times validation, the coefficients are different (like 10 slightly different models). Should we train M1 on the complete training set, and use the resulting coefficients to predict on new dataset?
2. But can this situation happen? For a new dataset, M2 performs better than M1? Although M1 has the least average prediction error in CV. Does that mean maybe M1 do not outperform M2?
3. In PLSR, the number of the optimal number of components are decided by cross validation. If we want to compare 2 PLSR models (the same response but completely different predictors), should we perform 2 different cross validation? e.g. Split the dataset into 10 parts, for 9 of them, perform plsr and use cross validation to obtain the number of components, and use these to predict the remaining one. And then obtain the prediction error. Finally compare the prediction error of 2 models?

Answer 1

• you are spot on with point # 1. Thats the way to do a cross K-fold validation. But do note that CV can be used to validate a model in itself rather than just use it to compare with other models. At times you will realize that the model is giving vastly different metrics for different sub samples of your training data set.

• the new data set is always going to give you unpredictable results and thats why the need to constantly retrain your models. For instance if your pre-real data set models gave you 40% and the incoming data is giving you like 20%, you will need to sample from the new data set to see whats so different about it and then add them to your training set. For all you know, your attribute set itself might need to be changed (using SVD / PCA / letting the algo choose). So both models will need to be retrained with the newer sample to see which one performs better now. There's no theoretical answer to this query. Just trial and error :)