Here's the question which confuses me a lot.

  1. Cross validation is used for model assessment, estimate the predictive error of a particular model, right? So if there are two models, M1 and M2, what's the right procedure for cross validation? Is the following steps correct? For each of the models: First, divide the training set into 10 parts. Second, train Mi on the 9 parts and test Mi on the remaining 1. Third, average the prediction error. Suppose M1 has the least average prediction error, then we should choose M1. But how can we make prediction on new dataset? Because among the 10 times validation, the coefficients are different (like 10 slightly different models). Should we train M1 on the complete training set, and use the resulting coefficients to predict on new dataset?
  2. But can this situation happen? For a new dataset, M2 performs better than M1? Although M1 has the least average prediction error in CV. Does that mean maybe M1 do not outperform M2?
  3. In PLSR, the number of the optimal number of components are decided by cross validation. If we want to compare 2 PLSR models (the same response but completely different predictors), should we perform 2 different cross validation? e.g. Split the dataset into 10 parts, for 9 of them, perform plsr and use cross validation to obtain the number of components, and use these to predict the remaining one. And then obtain the prediction error. Finally compare the prediction error of 2 models?

1 Answer 1

  • you are spot on with point # 1. Thats the way to do a cross K-fold validation. But do note that CV can be used to validate a model in itself rather than just use it to compare with other models. At times you will realize that the model is giving vastly different metrics for different sub samples of your training data set.

  • the new data set is always going to give you unpredictable results and thats why the need to constantly retrain your models. For instance if your pre-real data set models gave you 40% and the incoming data is giving you like 20%, you will need to sample from the new data set to see whats so different about it and then add them to your training set. For all you know, your attribute set itself might need to be changed (using SVD / PCA / letting the algo choose). So both models will need to be retrained with the newer sample to see which one performs better now. There's no theoretical answer to this query. Just trial and error :)

  • $\begingroup$ Thanks! There's 2 more stupid questions. 1. Suppose I have 1000 samples, then should I use all these samples to train my model OR separate them into a training set and a test set, then cross validate the training sample? 2. I want to get predicted values from the model. Should I split the data into 10 parts, train the model on 9 of them and predict the remaining one, and then use these predicted values as my predicted results? Or should I just separate the data into training set and test set in the beginning, train the model on training set and predict the test set? $\endgroup$
    – lilliam.C
    Sep 22, 2016 at 8:43
  • 1
    $\begingroup$ As a general observation: The more gentlemanly way of treating the results from a cross-validation is to have a training set, validation set and a final test set. Your model should only ever see the test set once. The idea is to choose between different models or choose the best parameter for whatever model you're CV'ing from the training and validation data. When you have a winner you use the test-data -that the model hasn't seen yet- and report the final error on that as the Expected Prediction Error. Naturally, you can only do that if you have enough data. If not, report validation error. $\endgroup$
    – Beyer
    Sep 22, 2016 at 9:55
  • $\begingroup$ @FBeyer brilliant answer ..i couldn't have put it better ! $\endgroup$ Sep 22, 2016 at 9:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.