6
$\begingroup$

Is it possible to do k-fold cross-validation to test all data, rather than using kfcv to find the optimal hypothesis as is typically done.

Example:

Say I want to use a svms on a dataset of size 1000. Could I use 900 events to train the svm for the testing of the other 100 events. Then use a separate 900 events to train the svm for a separate 100 events, repeating this process 10 times until all of the data has been tested.

Would events tested using separately trained svms be comparable? i.e. through this technique could I then use my entire dataset instead of setting aside a certain fraction for training, or is this a statistically unwise thing to do?

Thanks for any help,

Colorado

PS) Any references to why this would['nt] work is greatly appreciated, I'm discussing this technique in an academic setting.

$\endgroup$

3 Answers 3

6
$\begingroup$

As far as I understand your question, it can be formulated this way:

Instead of calculating a quality measure for each of the k validation-folds and then calculate the average, may I aggregate all folds an then calculate my quality measure, hence getting only one instead of k values?

This question requires two perspectives:
From the perspective of crossvalidation itself it is ok, because training- and testsamples still have an empty intersection etc. Since you just aggregate multiple samples drawn without replacement, the test-distribution is not spoiled.

From the perspective of the model, it depends whether the model produces comparable scores. SVM will work in my opinion, but imagine a model which min-max-normalizes the scores across the test-set (iiek), so that the calculation of a representative decision threshold across all test-samples will be quite hard.

In general, a lot of techniques which require the estimation of a parameter, which itself depends on the quality of a model, utilize this approach. A concrete example is the calculating of an operator to calibrate the scores of a classification model, e.g. Platt Scaling.

Furthermore (also this not an completely satisfying argument) the open source software Rapidminer has an operator for this approach.

PS: I want to point out, that although this approach is useful to get reliable quality measures for only small datasets, it may be hard to perform statistical tests to compare the significance of two models, since cv cannot be repeated endlessly (example: how to estimate whether the assumption of the so-often-misused t-test is satisfied if you only have 6 data points?).

PPS: Also interested, I was not able to find a paper focusing on the examination of this approach. The papers I have seen so far using this technique did not bother to reference it.

$\endgroup$
5
  • $\begingroup$ I think you have misunderstand the question (I believe it was Can I use CV to just make error approximation instead of the whole model selection?), but +1 for relevant remarks. $\endgroup$
    – user88
    Commented Nov 24, 2010 at 12:46
  • $\begingroup$ I also had this interpretation in my mind, but I thought that this would be too easy ;), since this is exactly what cv does. Additionally, the whole "all data"-part confused me. $\endgroup$
    – steffen
    Commented Nov 24, 2010 at 13:01
  • $\begingroup$ Thanks for your response, lots of relevant information here. Just to make sure we're on the same page let me state this another way: In some papers I've noticed that people set aside a certain fraction of the dataset for training purposes. Given a dataset of size 1000 they set aside 300 events for training and use 700 events for classification and never discuss the 300 events used to train the model (i.e. they don't train a separate model and then test those 300 events). Couldn't they just use cross validation technique to classify all of the data? $\endgroup$
    – C. Reed
    Commented Nov 24, 2010 at 16:46
  • $\begingroup$ @C.Reed: Not with the same ratios ! Sometimes you want to reduce the computation time and/or control the test-to-training-ratio to study the bias-variance tradeoff. I you have some time to spend, I recommend chapter 3 of Kohavis Phd Thesis discussing this topic extensively (robotics.stanford.edu/~ronnyk/teza.pdf). But all in all, the cv (from my point of view) is the favored method for error estimation (as mbq already said) $\endgroup$
    – steffen
    Commented Nov 24, 2010 at 16:55
  • $\begingroup$ Thanks for the thesis reference, ch. 3 is quite helpful on this matter. $\endgroup$
    – C. Reed
    Commented Nov 24, 2010 at 18:49
1
$\begingroup$

Yes it is; and while this is a very reliable way of reporting error, I would say it is even encouraged.

$\endgroup$
1
$\begingroup$

I'm not 100% clear on the question, but I have a few points to add:

I'm assuming that the error you are trying to estimate is the prediction error. If so, I agree that 10 fold cross validation would be good (and likely unbiased) approximation of the true prediction error IF your training sets are sufficiently large. Large in this case means that the training sets provide enough information to build a "good" SVM (one that, in a sense, captures most of the underlying relationships between between the predictors and response.) Training sets of size 900 are more than likely large enough. In fact, unless the SVM you are fitting is extremely complex, I would recommend using a 5-fold cross validation in order to get a more precise estimate of prediction error (and yes, you can average the error estimates of the 5 folds to obtain an final estimate.)

With regards to the question:

"Would events tested using separately trained svms be comparable? i.e. through this technique could I then use my entire dataset instead of setting aside a certain fraction for training, or is this a statistically unwise thing to do?"

I don't understand this question, but since the phrase "entire dataset" is in a post about CV, I just want to warn you that estimating prediction error from models fit to all available data is generally a bad idea. For cross validation to make sense, each training set/test set pair should have no points in common. Otherwise, the true error will likely be underestimated.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.