I'd like to do an experiment which should prove that cross validation is better method to evaluate model then the usual hold out method. I have the following idea:
- Create big artificial set for classification purposes (100 000 examples lets say)
- Take small sample from the whole set. Lets say 5000.
- Try to estimate error using hold out method with 5000 and using cross validation.
- Compare errors taking into consideration error on the whole set which is the real error if it would be our entire space of samples.
Link to the code:
https://github.com/pawelpeksa/msCV (master branch)
I'm checking this idea using SVM. Firstly I found hyperparameter C for SVM using hyperopt package (but it shouldn't matter anyway since is just tool to evaluate the model, yup?). I did 100 runs to get some statistics. My results:
A). 5000 holdout test sample: 0.9338 +- 0.00434869073528 - accuracy on the holdout sample after training.
B). cv on holdout set: 93.4009034436 +- 2.22044604925e-14 - accuracy calculated with cv with 10 folds
C). entire dataset 0.9370653 +- 0.000283225899239 - accuracy using entire dataset (which I consider to be real error taking into consideration size of dataset).
D) train dataset: 0.934448571429 +- 0.00195385751366 - accuracy of model on data which was used to train
What is unexpected for me. CV accuracy (B) is further from real error then A. CV supposed to be better method to evaluate the model because we use more data for evaluation and because of it it should be closer to real error?. Also, accuracy of model checked with train data is lower then accuracy checked with entire dataset which seems to be ridiculous to me. What also I noticed is the fact that std of cv estimation is much smaller then in other cases. Maybe it's the thing why CV is better for us?
I can smell logic error here from the distance. Any hints?
Thanks a lot!