I was learning about SVM using 2 sources: Andrew Ng Machine Learning course from Coursera and Stanford 'Statistical Learning' (from Trevor Hastie and Robert Tibshirani). And I encountered the following contradiction:

Andrew Ng provides this formula for SVM cost function: $$C\sum_{i=1}^m[y^{(i)}cost_1(\theta^Tx^{(i)})+(1 - y^{(i)})cost_0(\theta^Tx^{(i)})] + \frac{1}{2}\sum_{i=1}^n\theta_j^2$$ (He doesn't dive into details about $cost_1$ and $cost_0$, but says they are more or less like log-likelihood loss in logistic regression, but linear.)

Then he says that increasing $C$ leads to increased variance - and it is completely okay with my intuition from the aforementioned formula - for higher $C$ algorithm cares less about regularization, so it fits training data better. That implies higher bias, lower variance, worse stability.

But then Trevor Hastie and Robert Tibshirani say, quote:

And that means that $C$ gets bigger the more stable the margin becomes.

Also they show the following picture - the larger margins correspond to larger $C$, they say. C regularization

Stability implies higher bias, right? So they say the opposite to Andrew Ng.

So the question is who is right?


2 Answers 2


The effect of the SVM C-Parameter

While the first textbook description of an SVM always speaks of "maximizing the margin", but this is only the first step. If your data is not perfectly separable there will points on the wrong side of the separating hyperplane. To allow for such points slack variables were introduced (= soft-margin SVM). They include the problematic points into the equation and weight them using the C-Parameter. This parameter is a tradeoff between maximizing the margin and minimizing the error.

Why this?

Imagine (or draw on a paper) a perfectly separable 2D dataset with a plot similar to the above. Imagine a suitable hyperplane. Image you have a hard margin svm which does not allow for such misclassified points. Now imagine you will break the rules and place a document intentionally on the other side of the hyperplane. The hyperplane will probably change a lot and will be worse than before. If you had used a soft-margin SVM instead the old solution would still be a better one.

Your example

Increasing the value of the C-Parameter
$\iff$ Weight of misclassified points is increased
$\iff$ Margin gets smaller

And i think that is what Hastie and Tibshirani meant in terms of stable: In other words closer to the hard-margin SVM.

  • 1
    $\begingroup$ Then I believe Hastie and Tibshirani used a really unintuitive word here. For large C, as you mentioned, a single additional observation on the wrong side of the margin/decision boundary may dramatically shift/turn it. That situation is the opposite to the stability. $\endgroup$
    – Yurii
    Commented Mar 28, 2016 at 7:17
  • 1
    $\begingroup$ Stable as in "the number is misclassified points is reduced (less mix-ups between both sides of the hyperplane)". However this is only my interpretation of your quotes from the above post. (I have experience with SVMs and know the coursera course, but i have not had a look yet at the stanford course.). $\endgroup$
    – user35825
    Commented Mar 29, 2016 at 17:34

I'm not sure but I think contradictory definitions are in effect in the literature.

Introduction to Statistical Learning basically formulates C as the total "budget" for errors (i.e. violations of margins). So higher C means more errors are allowed (which will lead to lower variance).

However, in other implementations, for example this one cited by sklearn's SVM docs --


-- C is a multiplier of the sum of the errors in the cost function, so higher C means that errors are encouraged to be smaller (which will lead to higher variance).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.