Cross validation and prediction for unknown data

Question

How do we build a model, cross validate it and use it to predict for unknown data?

Say I have a known dataset of 100 points. Steps for 10 fold cross-validation are-

1. Divide the data randomly into training and test datasets in a ratio of 90:10
2. Make a model on the training dataset (90 points). (I used libSVM grid.py to optimize C and gamma)
3. Test the optimized model on the test dataset (10 points) and calculate the error.
4. Repeat steps (1,2,3) 10 times for 10-fold cross validation. Average the error from each repeat to get the average error.

Now, after repeating the steps 10 times, I will have 10 different optimized models. To predict for an unknown dataset (200 points), should I use the model which gave me minimum error OR should I do step 2 once again on the full data (run grid.py on full data) and use it as model for prediction of unknowns?

Also I would like to know, is the procedure same for other machine-learning methods (like ANN, Random Forest, etc.)

Answer 1

Now, after repeating the steps 10 times, I will have 10 different optimized models.

yes. Cross validation (like other resampling based validation methods) implicitly assumes that these models are at least equivalent in their predictions, so you are allowed to average/pool all those test results.

Usually there is a second, stronger assumption: that those 10 "surrogate models" are equvalent to the model built on all 100 cases:

To predict for an unknown dataset (200 points), should I use the model which gave me minimum error OR should I do step 2 once again on the full data (run grid.py on full data) and use it as model for prediction of unknowns?

Usually the latter is done (second assumption).

However, personally I would not do a grid optimization on the whole data again (though one can argue about that) but instead use cost and γ parameters that turned out to be a good choice from the 10 optimizations you did already (see below).

However, there are also so-called aggregated models (e.g. random forest aggregates decision trees), where all 10 models are used to obtain 10 predictions for each new sample, and then an aggregated prediction (e.g. majority vote for classification, average for regression) is used. Note that you validate those models by iterating the whole cross validation procedure with new random splits.

Here's a link to a recent question what such iterations are good for: Variance estimates in k-fold cross-validation

Also I would like to know, is the procedure same for other machine-learning methods (like ANN, Random Forest, etc.)

Yes, it can be applied very generally.

As you optimize each of the surrogate models, I recommend to look a bit closer into those results:

• are the optimal cost and γ parameters stable (= equal or similar for all models)?

• The difference between the error reported by the grid optimization and the test error you observe for the 10% unknown data is also important: if the difference is large, the models are likely to be overfit - particularly if the optimization reports very small error rates.

Answer 2

@d.putto - That's not what I'm suggesting. You want to randomly split your data into 10 (approximately) equally sized sets. You only do that once. Now on the first iteration, your training set consists of sets 2 through 10, while your test set is set 1. On the second iteration, your training set consists of sets 1 and 3 through 10, while your test set is set 2. This continues until you've gone through 10 iterations (until you've used each individual set as a test set). – Max

@Max - I think I got your point.. So 3 fold cross validation, divide data in 3 sets (say X,Y,Z). Take X as test and remaining (Y+Z) as training set. For next iteration take Y as test and (X+Z) as training set and so on... – d.putto

I saw contradiction, according to Max, at each new iteration, the test set will be included in traningg set and the size of training set tend to increase, (until using all sets as test set and according d.putto, the size of training sets still constant )