Cross-validation - comparison of estimated erros - unexpected result

Question

I'd like to do an experiment which should prove that cross validation is better method to evaluate model then the usual hold out method. I have the following idea:

1. Create big artificial set for classification purposes (100 000 examples lets say)
2. Take small sample from the whole set. Lets say 5000.
3. Try to estimate error using hold out method with 5000 and using cross validation.
4. Compare errors taking into consideration error on the whole set which is the real error if it would be our entire space of samples.

Link to the code:

https://github.com/pawelpeksa/msCV (master branch)

I'm checking this idea using SVM. Firstly I found hyperparameter C for SVM using hyperopt package (but it shouldn't matter anyway since is just tool to evaluate the model, yup?). I did 100 runs to get some statistics. My results:

A). 5000 holdout test sample: 0.9338 +- 0.00434869073528 - accuracy on the holdout sample after training.

B). cv on holdout set: 93.4009034436 +- 2.22044604925e-14 - accuracy calculated with cv with 10 folds

C). entire dataset 0.9370653 +- 0.000283225899239 - accuracy using entire dataset (which I consider to be real error taking into consideration size of dataset).

D) train dataset: 0.934448571429 +- 0.00195385751366 - accuracy of model on data which was used to train

What is unexpected for me. CV accuracy (B) is further from real error then A. CV supposed to be better method to evaluate the model because we use more data for evaluation and because of it it should be closer to real error?. Also, accuracy of model checked with train data is lower then accuracy checked with entire dataset which seems to be ridiculous to me. What also I noticed is the fact that std of cv estimation is much smaller then in other cases. Maybe it's the thing why CV is better for us?

I can smell logic error here from the distance. Any hints?

Thanks a lot!

Answer 1

I think I figured it out.

Classic method approach:

We try different parameters and evaluate model using test data (we obtained test data after dividing all data for two parts: let's 0.7 and 0.3) - so called holdout method

When trying the best parameters based on the our result using test data, information from the test data leaks into model and because of it our evaluation is no longer objective.

We can use cross validation in this which gives us better evaluation of the model.

So why the approach in the first post was wrong? Everything was automized and we didn't care about test data during model evaluation in holdout method. So testing on test data was testing on unseen data for model. Results were just totally random. Sometimes it seemed to be ok with my predictions, other times no, no rule.

I changed code a little bit. Now I use hyperopt to optimize hyperparameters of model but I check it on train data (like in classic approach). I let train data information leak into the model. After it I evaluate model with cross validation to compare both approaches.

Hope I haven't missed anything this time but result seems to be more reasonable now.

Code available here: https://github.com/pawelpeksa/msCV/tree/experimental3 (it needs strong refactor)

The results (difference between real error and estimated error is bit smaller in case of cv):