Using k-fold cross-validation to test all data

Question

Is it possible to do k-fold cross-validation to test all data, rather than using kfcv to find the optimal hypothesis as is typically done.

Example:

Say I want to use a svms on a dataset of size 1000. Could I use 900 events to train the svm for the testing of the other 100 events. Then use a separate 900 events to train the svm for a separate 100 events, repeating this process 10 times until all of the data has been tested.

Would events tested using separately trained svms be comparable? i.e. through this technique could I then use my entire dataset instead of setting aside a certain fraction for training, or is this a statistically unwise thing to do?

Thanks for any help,

Colorado

PS) Any references to why this would['nt] work is greatly appreciated, I'm discussing this technique in an academic setting.

Answer 1

As far as I understand your question, it can be formulated this way:

Instead of calculating a quality measure for each of the k validation-folds and then calculate the average, may I aggregate all folds an then calculate my quality measure, hence getting only one instead of k values?

This question requires two perspectives:
From the perspective of crossvalidation itself it is ok, because training- and testsamples still have an empty intersection etc. Since you just aggregate multiple samples drawn without replacement, the test-distribution is not spoiled.

From the perspective of the model, it depends whether the model produces comparable scores. SVM will work in my opinion, but imagine a model which min-max-normalizes the scores across the test-set (iiek), so that the calculation of a representative decision threshold across all test-samples will be quite hard.

In general, a lot of techniques which require the estimation of a parameter, which itself depends on the quality of a model, utilize this approach. A concrete example is the calculating of an operator to calibrate the scores of a classification model, e.g. Platt Scaling.

Furthermore (also this not an completely satisfying argument) the open source software Rapidminer has an operator for this approach.

PS: I want to point out, that although this approach is useful to get reliable quality measures for only small datasets, it may be hard to perform statistical tests to compare the significance of two models, since cv cannot be repeated endlessly (example: how to estimate whether the assumption of the so-often-misused t-test is satisfied if you only have 6 data points?).

PPS: Also interested, I was not able to find a paper focusing on the examination of this approach. The papers I have seen so far using this technique did not bother to reference it.

Answer 2

Yes it is; and while this is a very reliable way of reporting error, I would say it is even encouraged.

Answer 3

I'm not 100% clear on the question, but I have a few points to add:

I'm assuming that the error you are trying to estimate is the prediction error. If so, I agree that 10 fold cross validation would be good (and likely unbiased) approximation of the true prediction error IF your training sets are sufficiently large. Large in this case means that the training sets provide enough information to build a "good" SVM (one that, in a sense, captures most of the underlying relationships between between the predictors and response.) Training sets of size 900 are more than likely large enough. In fact, unless the SVM you are fitting is extremely complex, I would recommend using a 5-fold cross validation in order to get a more precise estimate of prediction error (and yes, you can average the error estimates of the 5 folds to obtain an final estimate.)

With regards to the question:

"Would events tested using separately trained svms be comparable? i.e. through this technique could I then use my entire dataset instead of setting aside a certain fraction for training, or is this a statistically unwise thing to do?"

I don't understand this question, but since the phrase "entire dataset" is in a post about CV, I just want to warn you that estimating prediction error from models fit to all available data is generally a bad idea. For cross validation to make sense, each training set/test set pair should have no points in common. Otherwise, the true error will likely be underestimated.