0
$\begingroup$

I am trying to implement SVM and i did my parameter selection(grid search) on the whole data and used the best values of C and gamma from that search to test on the testing data. Sometimes, the cross-validation accuracy, i get is 100%. Is this method wrong? What is the correct way of doing?

How should i go about to do my parameter selection? Shld i divide the data(100%) into 10% for grid search and remaining 90% into training and testing?

Need some guidance on it.. Thanks.

$\endgroup$
1
$\begingroup$

Yes you need to perform the grid search on cross-validated scores. This is well explained in the libsvm guide.

In the libsvm tool/ folder there is a grid.py script to help automate that. Alternatively if you use scikit-learn, you should use the GridSearchCV too that does a similar job:

http://scikit-learn.org/dev/modules/grid_search.html

$\endgroup$
8
  • $\begingroup$ you are not answering my question... Parameter search on whole data or only a portion of it? $\endgroup$
    – lakshmen
    Feb 27 '12 at 15:49
  • $\begingroup$ Well it depends on the size of the data. If you can afford (small dataset or large cluster for grid search) to run on a the full data, then do it. If you have more than 200 samples per class, I would first random sample a subset of that size, perform a grid search on it. Then double the number of samples and do another grid search. If you get roughly the same optimal parameters for the two grid search, then stop there otherwise double again. $\endgroup$
    – ogrisel
    Feb 27 '12 at 17:30
  • $\begingroup$ Also you should keep some data out when doing the grid search, say 25% that you will only use for evaluation when you have finished the whole model selection procedure. Otherwise you will not be able to estimate how much variance / overfitting you have in your model. $\endgroup$
    – ogrisel
    Feb 27 '12 at 17:34
  • $\begingroup$ why is it wrong to CV the whole data set? $\endgroup$
    – lakshmen
    Feb 27 '12 at 22:09
  • 1
    $\begingroup$ If you just want to answer the question: "which hyperparameters are the best", then doing model selection on the full dataset is ok. If you want to answer the question "what is the performance / score of the model with tuned hyperparameters", then you should split the dataset into a development set and an evaluation set, run grid search with cross validation on the development set, retrain a model with the best hyperparameters on the development set and evaluate it on the evaluation set. Otherwise you might run the risk of hyperparameters overfitting. $\endgroup$
    – ogrisel
    Feb 27 '12 at 22:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.