K-fold in grid-search for linear svm C parameter giving same value?

So every time I do a GridSearchCV with KFold, stratified or not, I get the same accuracy score and STDev for values of C=1,C=10, and C=100. I then did a special test without the grid search and used stratified k-fold but this time testing individual accuracy per each fold. It does not matter the amount of folds, the result is the same - same exact accuracy for different C values. Is there any reason why this could be? It doesn't happen with a shuffle split cross-validation, for example.

If it makes any difference, my classes are in order (first 40 are the same, second 40 are the same, 3rd 40 are the same in a set of 120), but even the shuffled k-fold yields the same results.

Sample:

[mean: 0.60000, std: 0.11365, params: {'C': 0.001},
mean: 0.69167, std: 0.12472, params: {'C': 0.01},
mean: 0.74167, std: 0.08498, params: {'C': 0.1},
mean: 0.76667, std: 0.01179, params: {'C': 1},
mean: 0.75833, std: 0.01179, params: {'C': 10},
mean: 0.75833, std: 0.01179, params: {'C': 100},
mean: 0.75833, std: 0.01179, params: {'C': 1000}]

• You say "it doesn't happen with a shuffle split cross-validation" but then write "but even the shuffled k-fold yields the same results." How are both of these statements true?
– Sycorax
Jan 4, 2016 at 12:57
• @user777 k-fold cross validation and shuffle split are different methods to cross validate, but I'm not exactly sure why they're different. Jan 4, 2016 at 17:29

I suspect what is happening is that once C is 1, the box constraint on the Lagrange multipliers (i.e. $0 \le \alpha_i < C$) is no longer active, so you get the same model no matter what the precise value of $C$ actually is. This is not unusual, but it probably means the model is over-fitting for such large values of $C$.
• The way to see why this happens is to think of the hard margin SVM (i.e. $C = \infty$); if you can fit one, then a soft-margin SVM with any value of C that is larger than the largest value of alpha for the hard-margiin SVM will give the same result. Jan 5, 2016 at 18:43