5
$\begingroup$

Actually, my question may just be about cross-validation in general. Here's what I'm doing: I'm trying to come up with a model using scikit-learn to learn on some data I've got. I've decided to use an SVM, using various kernels, to do the modelling. I've got about 50,000 data points from which to extract features. In an effort to make sure that my model is not over- or under-fitting, I've decided to run all of my models through cross-validation using scikit-learn's cross_validation functionality. I'm setting aside 40% of my training data for cross-validation, and so training on 60%.

I do this iteratively until I come up with a set of features and a model that gives me a cross-validation score of about 0.96. Great! Here's the problem - when I use this model to predict results for my test data, I only get a score of about 0.79! I don't understand that result. My question is, am I misunderstanding the cross validation score? Shouldn't I be able to expect similar results for my test data when using the model cross-validated to 0.96? I even used the GridSearchCV to come up with the best parameters to use for the SVM kernel. I also made sure to train on the full set of training data when training my model before running predict.

This is my first real attempt to use machine learning for a cool project, and I'm totally confused on my expectations.

$\endgroup$
12
  • 3
    $\begingroup$ to be clear: cross-validation uses 100% of the data. At each iteration, some p% of data is trained on, and (100-p)% is tested on, and the average score is returned. To say I'm setting aside 40% of my training data for cross-validation, and so training on 60%. sounds strange. Can you elaborate? $\endgroup$ Jun 6, 2013 at 18:09
  • $\begingroup$ I guess I'm saying that p=0.6 $\endgroup$ Jun 6, 2013 at 19:12
  • $\begingroup$ Sp typically, p is something smaller, like .2 or .1, so that one would perform 5 or 10 iterations respectively. In sklearn's cross_validation function, are you inputing 60% of the data, or 100%? Another question: how many features do you have/extract, and at what stage do you extract features? $\endgroup$ Jun 6, 2013 at 19:16
  • 1
    $\begingroup$ Can you provide the code? $\endgroup$ Jun 6, 2013 at 20:33
  • 1
    $\begingroup$ That's on another machine, but I can put some snippets up here when I get home. Thanks, Cam! $\endgroup$ Jun 6, 2013 at 20:38

3 Answers 3

3
$\begingroup$

From section 7.10.2 of Elements of Statistical Learning(free online, and it's great):

Consider a classification problem with a large number of predictors, as may arise, for example, in genomic or proteomic applications. A typical strategy for analysis might be as follows:

  1. Screen the predictors: find a subset of “good” predictors that show fairly strong (univariate) correlation with the class labels
  2. Using just this subset of predictors, build a multivariate classifier.
  3. Use cross-validation to estimate the unknown tuning parameters and to estimate the prediction error of the final model.

Is this a correct application of cross-validation? Consider a scenario with N = 50 samples in two equal-sized classes, and p = 5000 quantitative predictors (standard Gaussian) that are independent of the class labels. The true (test) error rate of any classifier is 50%. We carried out the above recipe, choosing in step (1) the 100 predictors having highest correlation with the class labels, and then using a 1-nearest neighbor classifier, based on just these 100 predictors, in step (2). Over 50 simulations from this setting, the average CV error rate was 3%. This is far lower than the true error rate of 50%.

What has happened? The problem is that the predictors have an unfair advantage, as they were chosen in step (1) on the basis of all of the samples. Leaving samples out after the variables have been selected does not cor-rectly mimic the application of the classifier to a completely independent test set, since these predictors “have already seen” the left out samples.

We selected the 100 predictors having largest correlation with the class labels over all 50 samples. Then we chose a random set of 10 samples, as we would do in five-fold cross-validation, and computed the correlations of the pre-selected 100 predictors with the class labels over just these 10 samples (top panel). We see that the correlations average about 0.28, rather than 0, as one might expect

$\endgroup$
6
  • $\begingroup$ I think I partially understand, but I'm not sure how to remedy my situation. I'm starting with a large number of AIFF files, and I need to get the data from these files into a form that can be submitted to the learning algorithm. I'm using scikit-audiolab to extract the features directly from the AIFF files, and these are what I'm passing to the cross-validation code. It sounds to me like you are saying that this is invalid. (By the way, thanks for the text reference - I'll definitely check it out!) How do I get data to the cross-validation without pre-selecting the features? $\endgroup$ Jun 6, 2013 at 20:02
  • $\begingroup$ By the way, thanks a million for the responses, Cam - I really do appreciate the guidance. I'm very new to this stuff, but it has definitely piqued my interest in a big way! $\endgroup$ Jun 6, 2013 at 20:04
  • $\begingroup$ So the answer I proposed was if you were actively selecting features, but it sounds like you are not doing this. The test data that you get .79 on: is this not involved at all in the training/cross validation part? $\endgroup$ Jun 6, 2013 at 20:07
  • $\begingroup$ Nope, that data is totally set aside. Once I have selected my model using only the training data, I then run the model to predict the results on the test data. The test data - the one where I get the score of 0.79 - is definitely in no way used to select the model and is thus not involved in the 0.96 CV score I get. $\endgroup$ Jun 6, 2013 at 20:13
  • $\begingroup$ The test data is also not used for training. $\endgroup$ Jun 6, 2013 at 20:15
1
$\begingroup$

What is the average value and standard deviation of the resulting recognition rates? If you have a high deviation, that would point out that your classifier is sensible to training data, i.e. you might be overfitting.

If the deviation is high, what if you use 70-30% or 80-20%. Do you still have the same problem?. Do you shuffle the samples before splitting them into groups?

Since you are using SVM, another possibility could be that if the distribution of samples is skewed across classes (you get by change a lot of samples from one class and very few from another) that you run into this problem too. A possible solution would be to try stratified cross validation in scikit-learn. The point is to ensure same number of samples per class.

$\endgroup$
0
$\begingroup$

I suppose your validation set is left completely untouched during GridSearchCV. If so, it would help if you described your design matrix.

Nonetheless, I personally find GridSearchCV functions tend to overfit. Therefore, I suggest you try either a Lasso linear model (and thus use LassoCV) or a ElasticNet model (and thus use ElasticNetCV).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.