1
$\begingroup$

So every time I do a GridSearchCV with KFold, stratified or not, I get the same accuracy score and STDev for values of C=1,C=10, and C=100. I then did a special test without the grid search and used stratified k-fold but this time testing individual accuracy per each fold. It does not matter the amount of folds, the result is the same - same exact accuracy for different C values. Is there any reason why this could be? It doesn't happen with a shuffle split cross-validation, for example.

If it makes any difference, my classes are in order (first 40 are the same, second 40 are the same, 3rd 40 are the same in a set of 120), but even the shuffled k-fold yields the same results.

Sample:

[mean: 0.60000, std: 0.11365, params: {'C': 0.001},
 mean: 0.69167, std: 0.12472, params: {'C': 0.01},
 mean: 0.74167, std: 0.08498, params: {'C': 0.1},
 mean: 0.76667, std: 0.01179, params: {'C': 1},
 mean: 0.75833, std: 0.01179, params: {'C': 10},
 mean: 0.75833, std: 0.01179, params: {'C': 100},
 mean: 0.75833, std: 0.01179, params: {'C': 1000}]
Improve this question
$\endgroup$
2
  • $\begingroup$ You say "it doesn't happen with a shuffle split cross-validation" but then write "but even the shuffled k-fold yields the same results." How are both of these statements true? $\endgroup$
    – Sycorax ♦
    Jan 4, 2016 at 12:57
  • $\begingroup$ @user777 k-fold cross validation and shuffle split are different methods to cross validate, but I'm not exactly sure why they're different. $\endgroup$
    – rb612
    Jan 4, 2016 at 17:29

1 Answer 1

Reset to default
1
$\begingroup$

I suspect what is happening is that once C is 1, the box constraint on the Lagrange multipliers (i.e. $0 \le \alpha_i < C$) is no longer active, so you get the same model no matter what the precise value of $C$ actually is. This is not unusual, but it probably means the model is over-fitting for such large values of $C$.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks! Would it make sense that's only happening wirh a k-fold CV? $\endgroup$
    – rb612
    Jan 4, 2016 at 17:27
  • $\begingroup$ It is difficult to say, if there are a couple of outliers in the data that cause problems, it may be that a more random allocation to training and test sets is more likely to result in a problem. Having said which, a bug in your code is always a possibility... $\endgroup$
    – Dikran Marsupial
    Jan 5, 2016 at 9:35
  • $\begingroup$ Thanks Dikran. I realized actually that shuffle split generates a new random sequence every time it's called, while k-fold uses the same each time, which would explain this. It's just so strange that such a different C value every time will give the same exact model. $\endgroup$
    – rb612
    Jan 5, 2016 at 18:12
  • $\begingroup$ The way to see why this happens is to think of the hard margin SVM (i.e. $C = \infty$); if you can fit one, then a soft-margin SVM with any value of C that is larger than the largest value of alpha for the hard-margiin SVM will give the same result. $\endgroup$
    – Dikran Marsupial
    Jan 5, 2016 at 18:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.