Actually, my question may just be about cross-validation in general. Here's what I'm doing: I'm trying to come up with a model using scikit-learn to learn on some data I've got. I've decided to use an SVM, using various kernels, to do the modelling. I've got about 50,000 data points from which to extract features. In an effort to make sure that my model is not over- or under-fitting, I've decided to run all of my models through cross-validation using scikit-learn's
cross_validation functionality. I'm setting aside 40% of my training data for cross-validation, and so training on 60%.
I do this iteratively until I come up with a set of features and a model that gives me a cross-validation score of about 0.96. Great! Here's the problem - when I use this model to predict results for my test data, I only get a score of about 0.79! I don't understand that result. My question is, am I misunderstanding the cross validation score? Shouldn't I be able to expect similar results for my test data when using the model cross-validated to 0.96? I even used the GridSearchCV to come up with the best parameters to use for the SVM kernel. I also made sure to train on the full set of training data when training my model before running predict.
This is my first real attempt to use machine learning for a cool project, and I'm totally confused on my expectations.