0
$\begingroup$

Here the setup:

I have 90% of the data for training and other 10% for testing.

I am doing stratified cross-validation on the 90% Tranining. It is a 10-class dataset. I am using LibSVM for that. When doing 10 fold cross-validation for tuning the Hypermarameters (C in the C-SVM) I get accuracies of 100%. Basically something like this:

        Training with 0.03125   -  Cross Validation Accuracy = 68.5097%
        Training with 0.12500   -  Cross Validation Accuracy = 98.3%
        Training with 0.50000   -  Cross Validation Accuracy = 100%
        Training with 2.00000   -  Cross Validation Accuracy = 100%
        Training with 8.00000   -  Cross Validation Accuracy = 100%

It is ok to have 100% accuracy in the cross-validation on the TRAINING data? In this case, should I chose C = 0.5 as the best hyper-paramter?

or instead should I move away from parsmeters that ge me 100% in the cross-validation? and why? if I don't take those with 100%, should I take what 98%? 90%?

Thanks,

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

I wouldn't say that C>0.5 is necessarily that big. NEVER make any model choices based on the test set, as this would give an optimistically biased performance estimate. The best approach is to use nested cross-validation, where the outer cross-validation is used for performance estimation and the hyper-parameters are tuned independently within each fold using cross-validation. (i.e. if you use 10 fold cross-validation, you perform 10 separate cross-validations to tune the hyper-parameters).

Improve this answer
$\endgroup$
4
  • $\begingroup$ So a 100% accuracy in the cross validation is good? $\endgroup$
    – mfcabrera
    Commented Feb 27, 2013 at 11:53
  • $\begingroup$ provided the same cross-validation is not also used to tune the hyper-parameters. Essentially tuning the hyper-parameters should be treated as an integral part of fitting the model, so for performance evaluation you need to tune the hyper-parameters again every time an SVM is trained on a new sample of data. $\endgroup$
    – Dikran Marsupial
    Commented Feb 27, 2013 at 12:49
  • $\begingroup$ I am actually using the cross-validation to tune the parameters. Then I have a separate TESTING data. My question is basically wether having a 100% in the n-fold stratified cross-validation of my TRAINING data is a bad symptoms (like overfitting the training data data). Thanks :) $\endgroup$
    – mfcabrera
    Commented Feb 27, 2013 at 14:38
  • $\begingroup$ It is possible to over-fit the cross-validation statistics in tuning the model, but the extent of this over-fitting is normally much less than the extent of over-fitting in training the model itself as there are so few degrees of freedom involved. As long as you don't optimise the cross-validation criterion too aggressively (e.g. grid search) it probably isn't too much of an issue. You might want to optimise the hinge loss or the AUC rather than error rate as these are more continuous. $\endgroup$
    – Dikran Marsupial
    Commented Feb 27, 2013 at 15:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.